Remove dependence of ParallelInterpolation on higher-level libs

[nilsvu]

The ParallelInterpolation library currently depends on a bunch of libraries like ApparentHorizon, that are really using the interpolation framework, not the other way round. My suggestion is to disentangle these dependencies one-by-one, so we end up with a general ParallelInterpolation framework that other parallel algorithms can use. These are the libraries that the ParallelInterpolation library currently depend on, but shouldn't (not all of them are listed in the CMakeLists.txt):

• ApparentHorizons (The ApparentHorizons lib should move to ParallelAlgorithms/ and depend on ParallelInterpolation. I suggest naming it ApparentHorizonFinder, because later we want to factor out Strahlkorper/Ylm-related things.) Edit: partially addressed in Fix ApparentHorizon dependencies #5267.
• Cce and GeneralizedHarmonic (The SendGhWorldtubeData and friends should move to the CCE lib)
• GeneralRelativity (The KerrHorizon interpolation target depends on this. I'm not sure what to do with this one, maybe just leave it for now.)
• DgSubcell. Edit: Done in Remove DgSubcell from ParallelInterpolation #5274.
• Time. Edit: Done in Remove Time from ParallelInterpolation #5275.

[nilsvu]

@markscheel Do you agree with this? Would you be willing to do the ApparentHorizons lib?

[markscheel]

Here is my suggestion for ApparentHorizons:

• Move Strahlkorper, StrahlkorperFunctions, ChangeCenterOfStrahlkorper into NumericalAlgorithms/SphericalHarmonics
• Similarly, move Tags and ComputeItems that depend only on Strahlkorper to NumericalAlgorithms/SphericalHarmonics
• Create ParallelAlgorithms/ApparentHorizonFinder, and put into it
  • FastFlow
  • ObjectLabel
  • ComputeHorizonVolumeQuantities
• What to do with StrahlkorperGr, and Tags/ComputeItems that depend on both Strahlkorper and GR quantities??
• What to do with StrahlkorperInDifferentFrame? (depends on Domain and FunctionsOfTime)??

[nilsvu]

Sounds good to me!

• What do you think about moving StrahlkorperGr and friends into the GeneralRelativity library? It computes GR quantities like expansion, Ricci scalar etc.
• Move StrahlkorperInDifferentFrame to Domain? We'll probably need it for non-AH things like excision surfaces, right? It does mostly domain and coord map things, so Domain seems the most appropriate place.

[nilsdeppe]

• I don't want the GR library depending on Strahlkorper and then possibly interpolation++++
• If this only depends on domain things the maybe, but I think that's sort of weird. What's wrong with the AH finder library in ParallelAlgorithms depending on GR and domain? Naively those are dependencies I would expect it to have. Are there are things that would depend on the compute tags that aren't horizons (eg excision boundaries)? If that's the case, maybe having the compute tags in PointwiseFunctions/GeneralRelativity/SurfaceQuantities?

[nilsvu]

AH surely depends on GR, domain etc. My question above is if we need the StrahlkorperGr functions also for non-AH things like excision surfaces. Therefore:

• Moving StrahlkorperGr to PointwiseFunctions/GeneralRelativity/SurfaceQuantities and a new GrSurfaceQuantities lib sounds good to me 👍 It can depend on GR and SphericalHarmonics, but on not much more.
• StrahlkorperInDifferentFrame depends only on Strahlkorper, Domain and FoT, so moving to Domain seems most appropriate I think. It can't be in SphericalHarmonics because that can't depend on Domain, and GrSurfaceQuantities also isn't the right place because that doesn't depend on Domain and StrahlkorperInDifferentFrame doesn't need GR anyway. Domain depends on SphericalHarmonics anyway, so something that combines SphericalHarmonics and domain things seems OK in Domain.

[nilsdeppe]

1. 👍
2. Maybe this depends on whether we want spherical surfaces in a flat spacetime (presumably yes). I would guess that there are quantities like normal vectors that are also different/simpler in flat space. Maybe a PointwiseFunctions/SphericalSurface (or SurfaceQuantities) in addition to the GR lib?

[knelli2]

• Create ParallelAlgorithms/ApparentHorizonFinder, and put into it

  • FastFlow
  • ObjectLabel
  • ComputeHorizonVolumeQuantities

I was planning on moving ObjectLabel somewhere else because it isn't specific to apparent horizons and won't be if we use it for stars as well. It has to be in a very general place so any part of the code base can use it, i.e evolution, elliptic, vacuum, hydro, etc... Any suggestions on where it should go? I was thinking either DataStructures because it's just a label for things, or Domain because everything has a domain.

[nilsvu]

@knelli2 I think something like ObjectLabel can be an executable-specific struct that's passed around as template parameter (if it must be a compile-time choice, which ObjectLabel is currently used for), no need to make it super general. E.g. in a BBH exec you define struct ObjectA with name etc, and then you can use Options::name to get a string label. I'll need to think a bit more about how these things could be defined at runtime (like ExcisionSpheres are, for instance).

[wthrowe]

I agree that OptionLabel shouldn't be super general, but having every executable define its own version of this and pass it around seems like the more-unnecessarily general version to me. What is the advantage of doing it that way over a single enum (or maybe small collection of marker stucts)?

[nilsvu]

Sure, what I mean is that we don't have to try hard to share these tiny structs; they don't incur any code duplication. I have often found it easier to define these label structs per-executable so I can give them the names I want for their context, and where they show up in input files. Of course the label structs can still be shared where it helps.

[wthrowe]

The annoyance isn't duplicating the enums/structs, but adding the code to handle them generically. Everything dealing with the objects now needs more template parameters, which have to be forwarded to other functions and classes correctly. It would mean that anything tied to the binary objects would have to be moved out of cpp files into hpp files.

[nilsvu]

I see your point @wthrowe. My proposal is then: We do a simple move of ObjectLabel to a new OrbitalDynamics library, e.g. in PointwiseFunctions/. We'll need such a library anyway I think, e.g. for the BNS force balance equation etc.

[wthrowe]

Sorry, not a hydro guy. What is the "BNS force balance equation"?

[nilsvu]

Eq. (5.2) in https://arxiv.org/pdf/1408.4136.pdf.

On a branch I currently have these functions in an OrbitalDynamics library: https://github.com/nilsvu/spectre/blob/xcts/executable_backup/src/PointwiseFunctions/OrbitalDynamics/ForceBalance.hpp
Also: orbital::Tags::CenterOfMass and orbital::Tags::AngularVelocity

[wthrowe]

That's probably OK. I'm just worried about grouping this with too much other stuff because I expect it to be used in at least one of Domain or DomainCreators to identify excision surfaces, and we don't want that adding any major dependencies.

[nilsvu]

I See. Maybe it's Domain after all then 🤷‍♂️

[knelli2]

This isn't urgent at the moment so maybe we hold off moving it to Domain until we have more binary/orbital stuff we want/need to do? Am I correct in assuming that BNS stuff is still a ways off since we still have to get BBH working?

[nilsvu]

Mostly done in #5267. Remaining tasks are listed in #5282.