Which data storage attributes to keep besides std::span based storage per data storage classes?

[phoeinx]

While switching from vector-based storage to std::span-based storage, we currently hold both attributes per data storage class - a vector and a span - to make the switch easier.

Precisely we have:

• DictionarySegmentIterable: _dictionary and dictionary_span
• DictionarySegment: _dictionary and dictionary_span
• FixedWidthIntegerVector: data and data_span

Ideally, we should at some point be completely span-based, but I am not sure if this achievable. (E.g. when we are first creating objects, which is mostly done by operating on vectors. Should we immediately only store spans on the data of the vectors and would that be memory-safe / would it be possible that the vector is without reference and gets garbage collected thus invalidating our span view on its data?)

This thread is for discussing which attributes to keep or delete.

[klauck]

There are two cases: in-memory version and disk version.
For both cases, the segment classes should have a span, which is used by the operators.
In addition, for both cases, we need a memory/storage object for the span, and thus also for the segment.

For the disk version, I think we aggreed that the storage manager manages this memory/storage object. (It is still open how this object is encapsulated and whether the segment should get a link (pointer/reference) to it, e.g., to know whether it is on disk or in memory).

For the in-memory version, it is unclear to me, whether the vector is managed by the segment itself or the storage manger. I think both is possible and it is more a matter of taste. But, we definitely need the vector/array somewhere. For a start, we could keep the vector in the segment, but should ensure that it is empty for disk-based segments and only used via the span for the memory-based version.

[Bouncner]

I'd vote for creating a "MemoryManager" (please don't use that name, @mweisgut is probably the most suited for naming) that is centrally located (e.g., for now within the storage manager) and can provide memory.
So memory can be requested by that "MemoryManager" and is centrally managed. There are ongoing and past projects where we would like to have a more centralized way of obtaining memory (e.g., buffer managers or just tracking the number of allocated bytes for tables). We could push into this direction if we would have a single "MemoryManager" "somewhere".

@mweisgut: what do you think?

[mweisgut]

If using std::span is the solution that we want to use in the long term (which I am not completely sure about at this moment), having a central instance that provides memory sounds good to me. However, I'm not completely sure about the naming though. A central managing instance could utilize multiple memory pools with different qualities (for example, one memory pool that is mmapped and another one that is always in memory) and provide a function to allocate/deallocate memory. MemoryManager or MemoryPoolManager sounds good to me.