IIEnv: Proposal #33

New Issue

Open

opened 2024-05-04 18:55:28 +02:00 by so-rose · 0 comments

so-rose commented

2024-05-04 18:55:28 +02:00

Owner

Proposal

Blender addons often need dependencies, but can't use their own versions of dependencies without stepping over each other's toes. It would be very nice if different parts of the same Python program could use different versions of imported modules.

With "Interpreter-Integrated Environments", you can do that. It's like venv, but in the same process.

Benefits

Having this brick in place opens the door to some staggeringly useful benefits:

Addons w/Different Dependencies:
No-Restart Addon Reinstall: Since all addon imports are namespaced, it's trivial to identify which sys.modules entries to delete after uninstalling an addon.
- Doing so without iienv would lead to unsolvable questions of what is "ok" to delete, since another addon / Blender itself might still be using it.
- This also means that Blender's Python environment is now "clean", so that installing the addon again doesn't result in some of the harsh edge cases that come with deleting modules referenced in sys.modules.
Addon Hot-Reloading: Since all addon imports are namespaced, even internally, it becomes possible to build an application that deterministically scans for file changes, finds the appropriate sys.modules entries for that addon only, and reloads not only the altered addon modules, but also updated dependencies. All without affecting Blender or other addons, while still preserving/updating module references within saved callbacks.
- Updating saved callbacks is the responsibility of the user that did the updating; the old callable should eventually be garbage collected, but at least its bound data won't become invalid while waiting for GC.

Costs

No hubris is without purgatory, and so you too must also endure a few caveats if you want to use iienv:

Replaced builtins.__import__: While iienv does its very best to keep its hands off of any import that it does not wish to manage, replacing a builtins is almost always a bad idea, no matter why it is done.
Sensitive to Python Internals: Containing a reverse-engineering import system means that iienv is very constrained to the Python interpreter and version that it is tested on. Currently, iienv is only tested on CPython 3.11 (to match Blender 4.1); do not presume that it works with anything else!
Edge Cases: The import system is full of surprising edge cases, and while iienv tries its best to be explicit about them, there will be packages that simply cannot work for reasons that may be entirely unapparent.

Status

Works with "normal" imports aka. bytecode, files, packages, etc. .
Works with extension modules on all OSes.
Tested to co-locate differently versioned numpy from two packages.
Comprehensive test suite, so it can be trusted.

The file is checked into the repo in src/, and should be considered experimental.

# Proposal Blender addons often need dependencies, but can't use their own versions of dependencies without stepping over each other's toes. It would be very nice if _different parts of the same Python program could use different versions of imported modules_. With "Interpreter-Integrated Environments", you can do that. It's like `venv`, but in the same process. ## Benefits Having this brick in place opens the door to some staggeringly useful benefits: - Addons w/Different Dependencies: - **No-Restart Addon Reinstall**: Since all addon imports are namespaced, it's trivial to identify which `sys.modules` entries to delete after uninstalling an addon. - Doing so without `iienv` would lead to unsolvable questions of what is "ok" to delete, since another addon / Blender itself might still be using it. - This also means that Blender's Python environment is now "clean", so that installing the addon again doesn't result in some of the harsh edge cases that come with deleting modules referenced in `sys.modules`. - **Addon Hot-Reloading**: Since all addon imports are namespaced, even internally, it becomes possible to build an application that deterministically scans for file changes, finds the appropriate `sys.modules` entries _for that addon only_, and reloads not only the altered addon modules, _but also updated dependencies_. All without affecting Blender or other addons, while still preserving/updating module references within saved callbacks. - Updating saved callbacks is the responsibility of the user that did the updating; the old callable should eventually be garbage collected, but at least its bound data _won't become invalid_ while waiting for GC. ## Costs No hubris is without purgatory, and so you too must also endure a few caveats if you want to use `iienv`: - **Replaced `builtins.__import__`**: While `iienv` does its very best to keep its hands off of any `import` that it does not wish to manage, replacing a `builtins` is almost always a bad idea, no matter why it is done. - **Sensitive to Python Internals**: Containing a reverse-engineering import system means that `iienv` is very constrained to the Python interpreter and version that it is tested on. Currently, `iienv` is only tested on `CPython 3.11` (to match `Blender 4.1`); do not presume that it works with anything else! - **Edge Cases**: The `import` system is full of surprising edge cases, and while `iienv` tries its best to be explicit about them, there will be packages that simply cannot work for reasons that may be entirely unapparent. # Status - [x] Works with "normal" imports aka. bytecode, files, packages, etc. . - [ ] Works with extension modules on all OSes. - [ ] Tested to co-locate differently versioned `numpy` from two packages. - [ ] Comprehensive test suite, so it can be trusted. The file is checked into the repo in `src/`, and should be considered experimental.