Resolve depends_on-chains eagerly so positions can be computed when subgroups are loaded

[SimonHeybrock]

Problem description

We have recently moved to loading individual components from NeXus files, such as NXmonitor. After such a load snx.compute_positions can then not resolve depends_on chains that loaded outside that group.

A previous implementation attempted to perform this eagerly while loading, but it causes performance issues. For example, a Bifrost file (873855_00000015.hdf, a "known" worst case) currently loads in 1 s, but takes more than 10 s if we do a naive implementation that directly follows and loads each chain.

Solution

• I tried adding a cache to a naive solution, but this still takes around 3.5 s, and it likely introduces other problems.
• Resolving only depends_on attributes that lead outside the current group brings this down to 2 s. I believe this is acceptable as most other files will see a much smaller overhead. Newer Bifrost files might also be less of an issue as it was planned to improve the chain structure (using shorter chains).

Remaining subtlety: If there is a chain with 2 jumps cut by a depend_on in the current group this implementation will not resolve the second jump. This may be rare enough to matter, so I am avoiding complications until this actually causes issues. Example:

group1/transformations/t1/depends_on = group2/transformations/t2
group2/transformations/t2/depends_on = t3  # breaks resolve
group2/transformations/t3/depends_on = group3/transformations/t4
grouped3/transformations/t4

[jl-wynen]

Can you log a message here to help with debugging if we hit this case?

[SimonHeybrock]

ScippNexus is not using logging so far. Are you suggesting to change this?

If this fail, you will get an error later when calling snx.compute_positions instead. Most likely cause for exceptions here is a dead link.

[jl-wynen]

ok

[jl-wynen]

In your description of the PR, did you mean that this line does not work whenever there is a chain that has multiple items within the same group? (group2 in your example)

Does the try-except here catch this case? Or do we need to check whether resolved is a data group?

[SimonHeybrock]

No, I mean that we will resolve a continuous series of jumps to other groups, but if there is a jump followed by a non-jump the resolving will terminate there. I'll think some more about whether we can fix this...

[SimonHeybrock]

I pushed an update, changing the mechanism. This now follows multiple hops correctly. It makes the "worst case" Bifrost load take 6 seconds.

[jl-wynen]

Why does this affect the performance so much? Does the BIFROST file contain many cases that were not properly resolved with the first version of this PR?

[SimonHeybrock]

Why does this affect the performance so much? Does the BIFROST file contain many cases that were not properly resolved with the first version of this PR?

Yes, there was a chain of hops, each to a different group. Now, ideally one would want to resolve only hops that lead outside the top-level loaded group, not outside the current subgroup being loaded. That sounds complicated with the current mechanism, so maybe some more need for rethinking this entirely?

[SimonHeybrock]

With a recent Bifrost file version the extra overhead with this PR is below the noise, a fraction of a second.

[jl-wynen]

Then I would say we can leave it as is. We should not make it more complicated than needed.