import - package structure

[k-dominik]

after a recent refactor there are 3 potential ways to import the v3 spec (which all come with different limitations, those are not equivalent):

1.  import bioimageio.spec.v0_3

2.  import bioimageio.spec.latest

3.  import bioimageio.spec

I can see that I would do 1. do get a specific spec, and 2. and 3. if I want the latest.

Proposal 1: let's get rid of latest :) I'm not sure what this should even mean. If you start working on v0_4, is this automatically the latest? or does latest mean current stable? Furthermore having it in the "root" of spec should be "latest" enough.

then there are issues how the imports from submodules are passed on. Let's take bioimageio/spec/__init__.py:

spec-bioimage-io/bioimageio/spec/__init__.py

Lines 1 to 4 in afff3a4

	from . import v0_1, v0_3
	from .latest import *
	__version__ = v0_3.schema.get_args(FormatVersion)[-1]

(latest __init__ will have from v0_3 import *)

with this submodule import is broken:

# doesn't work :(
from bioimageio.spec import raw_nodes

# works:
import bioimageio.spec
bioimageio.spec.raw_nodes.Something

This is because the "linking" via import * does not generate modules that can be discovered - it makes attributes available in the module namespace. There simply is no raw_nodes.py in the /spec, so it isn't discovered.
There are solutions to this:

1. live with it: doing import bioimageio.raw_nodes is not supported anymore :)
2. Write a custom finder/loader and hook into the Python import system: Honestly, until yesterday I didn't even know about this. Looks like a cool system to load modules not incarnated in the file system, e.b. a database. PMOTW has a nice example.
3. autogenerate pseudo modules in /bioimageio/spec. These are placeholder modules that import * from the relevant module in v0_3 or whatever the current (not necessarily latest...) spec is. Something like the following does the job:

# simplified, in fact `__init__.py` has to be modified, too
in_path = Path("./bioimageio/spec/v0_3")
out_path = Path("./bioimageio/spec")
for f in in_path.glob("*.py"):
    if f.name.startswith("__"):
        continue
  tmp_mod = out_path / f.name
  tmp_mod.write_text(f"# Auto-generated by gen.py - do not modify\n\nfrom .v0_3.{f.stem} import *\n")

From these solutions I have to say I favor 3:

• (1) is just ugly
• (2) is obscure, maintainability is dependent on advanced knowledge of the Python import system. I don't think preferring a non-standard file structure in a file based package justifies it.
• (3) is very easy to understand, it can be automated, checked for in CI. I'd consider maintainability high. It is a custom step, however, probably easy to miss if someone is new to the project.

Proposal 2: let's try to fix imports with (3)

CC: @FynnBe @constantinpape

P.S.: To make matters worse, we need to decide on this quickly - otherwise, I'll have to assume (1) and refactor downstream packages accordingly.

[constantinpape]

Proposal 1: let's get rid of latest :) I'm not sure what this should even mean. If you start working on v0_4, is this automatically the latest? or does latest mean current stable? Furthermore having it in the "root" of spec should be "latest" enough.

I am all in favor of this, let's get rid of latest. as soon as a spec gets merged into master and the import in the top-level init is adapted it should count as stable.

Proposal 2: let's try to fix imports with (3)

Sounds good to me 👍

[FynnBe]

yes, latest seemed like a reasonable thing while I was introducing the subversions, but it really is somewhat obsolete.

(3) is very easy to understand, it can be automated, checked for in CI. I'd consider maintainability high. It is a custom step, however, probably easy to miss if someone is new to the project.

👍 🏆

[FynnBe]

Maybe there is an obvious reason not to do this, but actually the code structure allows for symbolic links to be used to solve this as well (except for init.py, but from .latest_init import * with 'lastet_init.py -> v0_3.init.py' or the current solution would work as well / still work)

Wouldn't this be even cleaner?

I stumbled onto this when adding 'utils.py' from v0_3 to v0_1 (didn't have utils).
If we want to avoid symbolic links a utils.py copy wouldn't break anything...

[FynnBe]

yeah.. well.. never mind. PyCharm is a lost cause with symbolic links...