Subtypes of Basis, their .shape field, and infinite Hilbert spaces

[akirakyle]

Currently the docstring for the abstract type Basis states

The Basis class is meant to specify a basis of the Hilbert space of the
studied system. Besides basis specific information all subclasses must
implement a shape variable which indicates the dimension of the used
Hilbert space. For a spin-1/2 Hilbert space this would be the
vector `[2]`. A system composed of two spins would then have a
shape vector `[2 2]`.

Before getting into what this .shape field really means, let's get an overview of all the subtypes. First here's all the subtypes defined in QuantumInterface:

graph TD;
Basis-->GenericBasis;
Basis-->FockBasis;
Basis-->NLevelBasis;
Basis-->CompositeBasis;
Basis-->PauliBasis;
Basis-->SpinBasis;
Basis-->SumBasis;

Loading

Then all the subtypes defined in QuantumOpticsBase

graph TD;
Basis-->SubspaceBasis;
Basis-->ChargeBasis;
Basis-->ShiftedChargeBasis;
Basis-->ManyBodyBasis;
Basis-->ParticleBasis;
ParticleBasis-->PositionBasis;
ParticleBasis-->MomentumBasis;

Loading

Finally the only other subtype of Basis I could find in all of the dependent packages of QuantumInterface was WaveguideQED.WaveguideBasis.

One general question is whether it makes sense to try to upstream all the subtypes of Basis into QuantumInterface and discourage dependents from implementing their own subtypes? I think this would be advantageous for a few reasons. The first is simply because the the basis for a Hilbert space is independent of a given representation of the objects on that Hilbert space with respect to the given basis. Thus for QuantumInterface to provide a common interface and help dependents interoperate, I think it should strive to contain the definitions of all bases that dependents use. Then the fields of each subtype of Basis could be well documented as part of the interface. Also we can ensure the fields define only what is necessary in the mathematical notion of that basis, and nothing which secretly encodes aspects that are representation-dependent. This last point brings me to the question of the .shape field.

Looking at the Basis docstring and what is practically stored in .shape, it seems like it's mainly meant for CompositeBasis. So .shape stores the dimensions of each of the bases in the .bases field of CompositeBasis, which is understood as the basis for the tensor product space. This is consistent with defining Base.length(b::Basis) = prod(b.shape). However SumBasis also uses .shape to store the dimensions of each of the bases in its .bases field and thus must implement length(b::SumBasis) = sum(b.shape). Meanwhile QuantumSymbolics uses FockBasis(Inf,0.) in which case the .shape field is [Inf]. Finally another example of different meaning assigned to .shape is in the PositionBasis and MomentumBasis. In .shape they store the number of points at which the wavefunction is evenly sampled between the minimum and maximum cutoff values.

I don't actually think there's a universal notion of a .shape property which applies to all the various possible bases for a given Hilbert space. This point is best illustrated by the case of separable (i.e. countably infinite) Hilbert space.

There are an infinite number of irreducible representations of separable Hilbert space which are unitarily equivalent. The most common representations are fock, position/momentum wavefunction, and phase-space (sometimes called coherent state). Currently QuantumSymbolics uses FockBasis(Inf,0.) to refer to the bases for both the fock and coherent state representations. Also given that #32 will enforce that subtypes of AbstractOperator carry a basis, this will break the way Gabs repurposes those currently free types to parameterize other aspects of Gaussian operators. Since everything in Gabs is actually just working within in the coherent state basis. I think we should consider making these distinctions explicit by reworking the types intended for bases of separable Hilbert space.

So concretely, my proposal is to move all subtypes of Basis to QuantumInterface, remove the requirement for all of them to implement .shape and instead maybe only require that they implement Base.length which can be allowed to return Inf or some other sensible representation of countable infinity. Finally, I would rework the bases on separable Hilbert space to look like

graph TD;
ParticleBasis-->FockBasis;
FockBasis-->FockCutoffBasis;
ParticleBasis-->CoherentStateBasis;
CoherentStateBasis-->CoherentStateCutoffBasis;
ParticleBasis-->PositionBasis;
PositionBasis-->PositionCutoffBasis;
ParticleBasis-->MomentumBasis;
MomentumBasis-->MomentumCutoffBasis;

Loading

If all this makes sense, I can work on this on top #32!

@Krastanov @apkille

[apkille]

Hi @akirakyle, thanks for all of this work, it's greatly appreciated and will be helpful for many users! I have a few questions and comments:

• I agree that it logically makes sense to remove the .shape feature for countably infinite Hilbert space bases. It is something that was weird to deal with when developing the harmonic oscillator interface in QuantumSymbolics. I'm assuming people using QuantumOptics would have to switch over to XCutoffBasis types, which would be a pretty large breaking change? How do you envision this would work for QuantumSymbolics? I really don't have any influence in this aspect of your proposal, as I'm not a QuantumOptics maintainer, but I'm curious what your answers to these questions are.
• As you know, I'm currently rethinking how Gabs can support different symplectic representations. Right now, Gabs implicitly supports the canonical symplectic representation used in Weedbrook et. al. It would be useful to express Gaussian states and operations in different symplectic representations (and do translations between them), so I made a draft PR of how I would alter some internals of Gabs to make this happen. Currently, a GaussianState type is organized as follows:

struct GaussianState{M,V,N<:Int} <: StateVector{M,V}
    mean::M
    covar::V
    nmodes::N
end

So here, as you alluded to in this issue, I took advantage of the StateVector abstract type, treating it as a "state vector in a Hilbert space" and using it to freely parametrize the mean and covar fields. What I would like to do instead is wrap a GaussianState around mean, covar, and something that defines the symplectic form which also preserves the number of bosonic modes of the system, i.e.,

struct GaussianState{R<:SymplecticRepr,M,V} <: StateVector{M,V}
    repr::R
    mean::M
    covar::V
end

This is what is currently implemented in my draft PR. However, perhaps a workaround to your proposed changes is that Gabs uses the abstract type Basis from QuantumInterface to express symplectic bases, so according to #32, it would look something like:

struct GaussianState{B,M,V} <: StateVector{B<:Basis}
    basis::B
    mean::M
    covar::V
end

I could then define my own subtype system for symplectic bases within QuantumInterface. Similar things could be done for GaussianUnitary and GaussianChannel. This would work nicely in that regard, but it interferes with your definition of a what a Basis type should be, which is that it represents a basis for a Hilbert space. What are your thoughts on this?

[akirakyle]

Thanks for the reply and good points that you raise @apkille!

I'm assuming people using QuantumOptics would have to switch over to XCutoffBasis types, which would be a pretty large breaking change? How do you envision this would work for QuantumSymbolics?

#32 is already a significant breaking change for which I'm taking on the responsibility of fixing all dependent packages in the qojulia and QuantumSavory orgs. So I would volunteer to do the same with this change. Hopefully these changes won't be too disruptive for any end user code or packages not in these orgs. It really depends on whether they are using the higher level functions in QuantumOptics versus depending on details of lower level stuff from QuantumOpticsBase. I think it's best to try to make such breaking changes now, because further into the future they may become disruptive enough for it not be worth it.

I could then define my own subtype system for symplectic bases within QuantumInterface. Similar things could be done for GaussianUnitary and GaussianChannel. This would work nicely in that regard, but it interferes with your definition of a what a Basis type should be, which is that it represents a basis for a Hilbert space. What are your thoughts on this?

Thinking through a bit more about what the goals of Gabs may be, and what exactly the Gaussian phase space formalism is doing, I'm now thinking that one possibility is for GaussianState to simply not subtype StateVector. At the very least I think whatever reasoning and decision about this that is made for Gabs I think also apples to QuantumClifford. To elaborate, the Stabilizer and Destabilizer tableau objects are analogous to the GaussianState object. They both only represent a restricted subclass of all states of their respective Hilbert spaces. These restricted representations are then efficient with respect to the number of modes in tensor product spaces. There's a sense in which these states are defined with respect to a certain basis (the Pauli or Coherent state bases respectively), but an additional symplectic structure is required to actually interpret their raw data.

Furthermore the generic operations in QuantumOpticsBase which operate on subtypes of StateVector or AbstractOperator which have a .data field with the usual matrix representation of these objects, such as ptrace or tensor, fundamentally cannot be reused. I think it may be for these reasons thatQuantumClifford doesn't subtype StateVectoror AbstractOperator, instead opting for conversion methods defined in a package extension. I think this makes sense given that conversion can only generally happen in one direction and would similarly suggest that not subtyping StateVector or AbstractOperator also makes sense for Gabs.

However, if there's any compelling reason to subtype, then I would suggest making SymplecticRepr a subtype of Basis (and specifically CoherentStateBasis if I rework the Basis types) so that it satisfies the type parameter constraint that #32 will introduce to StateVector. I think this is still somewhat sensible in that the symplectic form does specify a basis, however only on a restricted subspace of the full hilbert space.

[Krastanov]

One general question is whether it makes sense to try to upstream all the subtypes of Basis into QuantumInterface [...]

That sounds reasonable, but not necessarily high priority. Originally all of these were part of QuantumOptics and I just moved those that I needed (for QuantumSavory) to QuantumInterface. Given that there is no-one outside of QuantumOptics that uses these bases, I am not sure that the move would be a good use of limited dev resources. But if you feel like taking care of it, please go ahead!

[...] and discourage dependents from implementing their own subtypes?

Organizationally this is good by me, but I do not think that is necessary. The current API is not necessarily good, but a good API in a multimethod language should not need to externally impose this constraint.

Also we can ensure the fields define only what is necessary in the mathematical notion of that basis, and nothing which secretly encodes aspects that are representation-dependent.

This might be at odds with how historically this happened. All of these types were created specifically for use in QuantumOptics. I contacted the devs of QuantumOptics about abstracting these things away a bit so that I do not need to create a completely separate package with these capabilities when working on QuantumSavory (and to lesser extent on QuantumClifford). As such, many of these objects probably have internal details that are focused on state-vector-specific representation capabilities.

In short, I am all on board with abstracting these away, but only if that does not increase the internal complexity of QuantumOptics.

Looking at the Basis docstring and what is practically stored in .shape, it seems like it's mainly meant for CompositeBasis. So .shape stores the dimensions of each of the bases in the .bases field of CompositeBasis, which is understood as the basis for the tensor product space. This is consistent with defining Base.length(b::Basis) = prod(b.shape). However SumBasis also uses .shape to store the dimensions of each of the bases in its .bases field and thus must implement length(b::SumBasis) = sum(b.shape). Meanwhile QuantumSymbolics uses FockBasis(Inf,0.) in which case the .shape field is [Inf]. Finally another example of different meaning assigned to .shape is in the PositionBasis and MomentumBasis. In .shape they store the number of points at which the wavefunction is evenly sampled between the minimum and maximum cutoff values.

Aren't most of these consistent? Superficially, only the SumBasis case looks suspicious to me. Maybe it would have been slightly clearer if PositionBasis was called DiscretePositionBasis and if FockBasis was called FockCutoffBasis, but their shape meaning is all simply "dimensionality of each of the subsystems". It kinda makes sense that SumBasis has a different rule as well, given that it represents the direct sum of subsystems, not their tensor product.

So concretely, my proposal is to move all subtypes of Basis to QuantumInterface, remove the requirement for all of them to implement .shape and instead maybe only require that they implement Base.length which can be allowed to return Inf or some other sensible representation of countable infinity. Finally, I would rework the bases on separable Hilbert space to look like

I do not think we should remove the .shape requirement. Sure, its content is not always useful, but in most such situations it is simply [Inf]. Removing it would be a breaking change with little benefit.

It is true that you can have different bases (including very differently structured composite/sum bases) for the same Hilbert space, but Basis represents a basis, not a space. This has indeed led to the awkwardness of Gabs and QuantumSymbolics using FockBasis(Inf) to represent a space. Introducing the notion of a space that can have multiple different basis would be neat (and a less breaking version of what you are suggesting here), but (1) it is a lot of work that I am not sure we have proven we need yet and (2) will add a lot of complexity to the codebase.

My current belief is that what you are suggesting (with potentially switching the focus from modifying Basis to inplementing a more general Space) is good elegant first-principles design, but (currently) too breaking with not enough benefit to the current code base. I can be convince otherwise though, especially if we start keeping track of issues or difficulties caused by the current design.

[Krastanov]

#32 is already a significant breaking change

This is more of a side question, but is it still "technically breaking" after the changes we discussed? I thought it greatly minimized the breakages outside of the internals of QuantumOptics and QuantumSymbolics?

I think it may be for these reasons that QuantumClifford doesn't subtype StateVectoror AbstractOperator

Not really. The actual reason was that these packages were independent. I am slowly transitioning the package to properly subtype StateVector and AbstractOperator and to reuse the generic implementations of ptrace and tensor. If something has a .data field it should probably be in QuantumOptics, but plenty of useful general implementations of things like ptrace and tensor can exist that do not depend on .data. In particular, it would be ket to be able to write tensor(S"X", Ket(GenericBasis(2), [1.0,0])) and have everything just work.

Anyway, back to the big picture. After reading the rest of the comments here, I am leaning towards not changing much about Basis but potentially starting to think about more functionality on top of basis or in parallel to Basis that let's you ask questions about spaces. I think it is fine that we use FockBasis([Inf]) as an awkward historic stand-in for "the abstract space of a harmonic oscillator" for now, and we can slowly and non-breakingly transition to some notion of Space that has the capabilities you are suggesting.

I think we should not make the type system much more complicated than it currently is. I am uncertain of the benefit of introducing more layers of abstract types.

But to be fair here, I am commenting without having had to deal with the low-level code for a while, while you are going through it right now. It is quite possible to convince me that the added complexity (and breakage) is much more necessary than my current gut feeling tells me.

Thanks for going through all of this design process, it is incredibly valuable. I am getting on a plane right now, so I can not discuss the rest of the details in the proposal, but let's continue the conversation and maybe even start some collaborative design document. What you are suggesting has to potential for a lot of downstream improvements if we can minimize the breakage.

[akirakyle]

Thanks @Krastanov for the background information on how these design choices have evolved! I've been playing with alternative abstract interfaces for the basis system in the past week. This has been a good exercise for myself to understand all the design tradeoffs and downstream impacts that might be incurred with changing the abstract interface. This has raised further issues with the current system which I've been documenting in new issues as I come across them (see #34 and #35 for the recent ones). Replying to a couple points you raise:

I do not think we should remove the .shape requirement.

Whether we remove it or not, I think we should at least enforce some uniformity on it, since it's used all over QuantumOpticsBase to essentially get the shape of the tensor product decomposition. Currently it is sometimes a vector and sometimes a tuple which I think is not great.

The actual reason was that these packages were independent. I am slowly transitioning the package to properly subtype StateVector and AbstractOperator and to reuse the generic implementations of ptrace and tensor.

I've been keeping this in mind now with my prototyping of different choices to fix these issues so that they can be "future proof" for eventual closer integration. This is I think one pressing reason to seriously consider doing some breaking changes of the interface since I don't think it's currently compatible with such integration. Specifically see #35.

is it still "technically breaking" after the changes we discussed?

Changing the number of parametric types and adding type constraints requires everywhere that such types are use (which is a lot in QuantumOpticsBase) to be fixed. Specifically going from AbstractOperator{BL, BR} to AbstractOperator{B<:OperatorBasis} necessitates a lot of updating of function type signatures.