We refactored the entire `extra_sympy_units` into a (rather decently sized)
proper package, and changed its name.
Additionally, we've seperated the operation enums from the math nodes
themselves and moved them to dedicated `math_system` package, which is a
very big breath of fresh air.
I'll need a moment to fix a few typos, but incredibly, this architecture
is kind of "just works" (TM) at this point - not like the `FlowKind`
refactoring debacle...
To go with that, we've greatly streamlined domain handling in
`SimSymbol`.
We now track a symbolic set as the domain, which is very simple and
effective, but comes with some headaches too (`sympy` is fighting me,
grumble grumble...)
We've also managed to enforce a few unit-conversions in the operate
node, and revamp the entire operation validity detection (weirdly hard).
The big pain point at the moment is determining the image of functions
we apply, on the `sp.Set` domain of `SimSymbol`s. We can express this
trivially, but `sympy` simply doesn't care to evaluate it.
One can use `SetExpr` and `AccumBounds` to "sometimes work" for some set
kinds, but nothing nearly good enough for even our relatively humble
needs.
Let alone stuff like fourier.
We've ended up deciding to hard-code this part of the process by-operation.
With domain-specific knowledge and little bit of suffering, we can
manually ensure the output domain of every operation makes it to the
output symbol.
As for "why bother", well, the entire premise of a symbolic nodal math system
that is tolerable to use, requires checking the valid domain of the input.
We do wish it were optional, but eh.
Massive boost.
Hard to quantify everything.
We're almost batch/inverse design!
- All units are now correct at realization (they weren't before), and
several operations were simply wrong.
- Near-optimal "flow narrowing", which globally minimizes `DataChanged`
even in the face of complex internal property dependencies.
- Sockets now cache `FlowKind` output themselves, with invalidation
provided for and flow narrowing happening before it even enters the
node flow.
- New FlowKind (`PreviewsFlow`) w/plotting, 3D, efficient caching, with
all previewed nodes already adapted.
- Drastically prettified plot output, due to seaborn theme integration,
dict-like data transport w/better labelling, etc. .
- Deep, reliable unit/unit dimension deduction and arithmetic for
`PhysicalType` deduction using dimensional analysis on arbitrary
expression operations.
- Fourier transform w/Nyquist limits as boundary conditions (preserving original shift).
- Vastly improved math node operation checks.
- Symbol constant node, integrating domain presumptions.
- Flow integration for ExprSocket, including for flow-dynamic capabilities and color.
- Viewer node got a big update, incl. live-display of sympy types (even
matrix) and a few Tidy3D types, as well as seperating deeper options
into a "Debug" mode
- Deeply streamlined symbolic flow, including
- **Differentiable Box structure**, w/generic lazy parameter support - only the start!
We implemented a node to load various kinds of data, notably `.npy`,
`.txt`, `.txt.gz`, and `.csv`. The `DataFileImporterNode` really should
expose some settings for setting name/mathtype/physical type/unit of
each unit, and/or treating a column from 2D data as index coordinates.
But the nuances of doing this in a manner general enough to deal with
!=2D data was a lot, and we needed similar abilities in the general math
system anyway.
So, we delved back into the `FilterMathNode` and a little into the
`TransformMathNode`. Fundamentally, a few difficult operations came out
of this:
- Filter:SliceIdx: Slice an array using the usual syntax, as baked into the
function.
- Filter:PinIdx: Pin an axis by its actual index.
- Filter:SetDim: Set the `InfoFlow` index coordinates of an axis to a specific,
loose-socket provided 1D array, and use a common symbol to set the
name+physical type (and allow specifying an appropriate unit).
- Transform:IntDimToComplex: Fold a last length-2 integer-indexed axis
into a real output type, which removes the dimension and produces a
complex output type. Essentially, this is equivalent to folding it as
a vector and treating the `R^2` numbers as real/imaginary, except this
is more explicit.
By combining all of these, we managed to process and constrain the medium data to
be a well-suited, unit-aware (**though not on the output (yet)**) `wl->C` tensor.
In particular, the slicing is nice for avoiding discontinuities.
Workflow-wise, we'll see how important these are / what else we might
want. Also, it turns out Blender's text editor is really quite nice for
light data-text viewing.
We also implemented a JIT-based `rescale` for `ArrayFlow`, where the
same function as for `LazyArrayRangeFlow` passes through and can do an
arbitrary symbolic shift/rescale/order-preserving whatever.
To make this fast for `ArrayFlow`, we utilize a "test" variable `a`,
put it through the function and rescale/strip its units, then `lambdify`
it and broadcast the function onto `ArrayFlow.values`.
It's an immensely clean way of working.
The `lambdify` seems fast, interestingly, but ideally we would of course
also cache that somehow.
Some details remain:
- Fourier Transform index bounds / length are currently not presumed
known before actually computing them; it's unclear what's best here,
as the design cross-section of physical expectation, mathematical
correctness, and ease of implementation (especially trying to keep
actual data out of the `InfoFlow` hot-path). For now, the `0..\infty`
bounds **will probably break** the `Viz` node.
- We've reverted the notion that particular `sim_symbol`s must have a
unit pre-defined, which causes a little more complexity for nodes like
`TemporalShape`. This question is going to need resolving
- The whole `InfoFlow` object really seems to be at the heart of certain
lagginess when it comes to the math system. It, together with the
index representations, would benefit greatly from a principled
refactor.
- The `Viewer` node is broken for 3D preview; see #70.
Closes#59. Progress on #54.
We now have a single node for all temporal shapes, which is extremely
usable. Note that we found a bug where input socket caching seems to
survive changes to loose inputs / socket set-driven alterations, which
prevents output from being able to switch with the socket set. We'll
make an issue for it whenever convenient.
Work also continued very briskly with the `SimSymbol` abstraction, which
is really, really working out.
Closes#67.
Driven solely by the Expr socket, we've completely replaced the
dedicated NumberConstant and PhysicalConstant nodes. We also
demonstrated symbolic variable operations w/visualization and
end-node realization, validating that this approach to Expr sockets is
key.
We prepared for a new `BLPropType`, namely dynamic lists of dataclasses,
which will represent dynamic user-adjustable lists. This is the only
proper UI design for declaring symbols directly in an Expr node.
For now, we'll do without symbols, but will be a core feature for design
space exploration (aka. batch-run a bunch of sims), and for inverse
design.
Else, a few fixes: Naming of `ManagedBLImage` was updated, the expr
socket info display was fixed to display even when only output
information is available,
A few TODOs remain in the Expr Constant, but they are less important
before dynamic symbol declarations are in place.
Sofus Albert Høgsbro Rose