Massive boost.
Hard to quantify everything.
We're almost batch/inverse design!
- All units are now correct at realization (they weren't before), and
several operations were simply wrong.
- Near-optimal "flow narrowing", which globally minimizes `DataChanged`
even in the face of complex internal property dependencies.
- Sockets now cache `FlowKind` output themselves, with invalidation
provided for and flow narrowing happening before it even enters the
node flow.
- New FlowKind (`PreviewsFlow`) w/plotting, 3D, efficient caching, with
all previewed nodes already adapted.
- Drastically prettified plot output, due to seaborn theme integration,
dict-like data transport w/better labelling, etc. .
- Deep, reliable unit/unit dimension deduction and arithmetic for
`PhysicalType` deduction using dimensional analysis on arbitrary
expression operations.
- Fourier transform w/Nyquist limits as boundary conditions (preserving original shift).
- Vastly improved math node operation checks.
- Symbol constant node, integrating domain presumptions.
- Flow integration for ExprSocket, including for flow-dynamic capabilities and color.
- Viewer node got a big update, incl. live-display of sympy types (even
matrix) and a few Tidy3D types, as well as seperating deeper options
into a "Debug" mode
- Deeply streamlined symbolic flow, including
- **Differentiable Box structure**, w/generic lazy parameter support - only the start!
We also implement `BLField` support for Blender `IDStruct` types.
There's a crash haunting us specifically with the cylinder array. Other
primitives (esp. ring) work just fine, as does previews of nested-linked
node groups. The crash triggers specifically when the file is saved and
reloaded, whereafter all the `load_post` handlers run fine (like -
extremely fine, we can perfectly access and dereference all of the node
groups, seemingly all the objects, etc.). Then, crash.
We're also discovering that `id_properties_ui` is completely useless for
after-the-fact updating of custom properties within sockets. Which is a
great surprise, and I have trouble thinking it's on purpose - the data
is stored somewhere, after all. All the forced updates/redraws/etc. in
the world don't seem to change this.
We may have to go back to the drawing board with dynamically-updated
min/max. The entire infrastructure with `SocketDef` altering sockets
after creation is entirely, _violently_ unsuited to do a static
modification. But the bare fact is, the dynamic modification methods are
falling short. It's kind of important stuff, this stuff.
We now have a single node for all temporal shapes, which is extremely
usable. Note that we found a bug where input socket caching seems to
survive changes to loose inputs / socket set-driven alterations, which
prevents output from being able to switch with the socket set. We'll
make an issue for it whenever convenient.
Work also continued very briskly with the `SimSymbol` abstraction, which
is really, really working out.
Closes#67.
Some rather foundational things were fundamentally broken, especially related to the initialization procedures of fields / cached properties.
- We completely revamped `bl_cache`, fixing many to-be-discovered bugs.
- We completely streamlined `BLField` property logic into reusable
`bl_cache.BLProp` and `bl_cache.BLPropType`.
- We implemented `BLInstance` superclass to handle ex. deterministic
persistance of dynamic enum items, and other nuanced common
functionality that was being duplicated raw.
- We implemented inter `cached_bl_property` / `BLField` dependency
logic, including the ability to invalidate dynamic enums without
@on_value_changed logic. This **greatly** simplifies a vast quantity
of nodes that were traditionally very difficult to get working due to
the sharp edges imposed by needing manual invalidation logic.
- We gave `ExprSocket` a significant usability upgrade, including
thorough parsing logic in the `SocketDef`.
It's not that existing nodes are as such broken, but their existing bugs
are now going to cause problems a lot faster.
Which is a good thing.
BREAKING CHANGE: Closes#13. Closes#16. Big work on #64. Work on #37.
It's recommended to add license headers to all files in an AGPL project, in case a file is viewed outside the context of its main repository. We're using a `pre-commit` tool to manage this, to make sure it's consistently applied to all our Python files.
I'm of the belief that the correct abstractions are now actually
available, and that most-to-all of the required functionality actually
already exists within the code base.
The art is bringing it together!
The Viz node now detects the shape of the data, and presents compatible
plot options.
Not all are implemented, but a few quite important ones are.
Additionally, a number of dataflow-related bugs were investigated and
fixed. A few were truly damaging, but many simply resulted in gross
inefficiencies - we must be careful declaring BLFields that are updated
in hot loops!
Moreover, it is exceptionally easy to add more as needed, as we analyze
more and more sims.
The only limit is `matplotlib`, which is... well, yeah.
Due to the BLField work, the dynamicness of the Viz node is quite
under control, so there will not be any critical issues there.
The plotting lags (70ms total in the hot loop), but that's actually
entirely fixeable.
It's also entirely the `managed_bl_image`'s fault.
Fixing these inefficiencies will also make Tidy3D's builtin plots
near-realtime, incidentally.
We profiled the following currently:
- 25ms: Creating `fig = plt.subplots`. We can reuse fig per-managed
image.
- 43ms: The BytesIO roundtrip, including `savefig`. We can instead use
the Agg backend, `fig.canvas.draw()`, and a `np.frombuffer` to both
plot directly to the memory location,
- ~3ms: Actual plotting functions in `image_ops`. They are seriously fast.
- ~0ms: Blitting pixels to the Blender image - this was optimized in
4.1, and it shows; the time to copy the data over is essentially nothing.
Enormously important changes to the data flow semantics and invalidation
rules. Especially significant is the way in which the node graph
produces a deeply composed function, compiles it to optimized machine
code with `jax`, and uses a seperately cached data flow to insert values
into the function from anywhere along the node graph without recompiling
the function.
A critical portion of the math system, namely the unit-aware dimensional
representation, is also finished. The `Data` node socket type now
dynamically reports the dimensional properties of the object flowing
through it, courtesy the use of a seperate data flow for information.
This allows for very high-peformance unit-aware nearest-value indexing built on binary
search.
Also, dependency management is completely ironed out. The `pip install`
process now runs concurrently, and the installation log is parsed in the
background to update a progress bar. This is the foundational work for a
similar concurrent process wrt. Tidy3D progress reporting.
The serialization routines are fast and effective.
Overall, the node graph feels snappy, and everything updates smoothly.
Logging on the action chain suggests that there aren't extraneous calls,
and that existing calls (ex. no-op previews) are fast.
There will likely be edge cases, and we'll see how it scales - but
for now, let's go with it!
We have a far more sane approach to nodeps now, which
allows us to essentially have two loggers - one that is
very useful, pretty, and clear, but requires a 'rich'
dependency, and one that is simple.
In this spirit, we factored out services/ too.
We can also set the initial console log level now when
packing the .zip.
There's still work to do with the actual flow for deps
installing / uninstalling.
But it should be far more robust now.
Finally, we have a barebones working `quartodoc`-based docs site.
It's super clever; see <https://github.com/machow/quartodoc>.
As it's "just" a quarto project with some python autodiscovery,
fleshing it out with ex. math, images, diagrams, and so forth
should be exceptionally easy.
As we develop, various linter-guided fixes are being realized.
This will be a long process, best done as we spiff everything up
in preparation for general release.
