Read collection of values out of backends

[lossyrob]

This is the first piece of the "Collections API", which is the ability to work with collections of (key, value) pairs in a similar way to using RDDs, without the overhead of Spark.

This task is to read a collection of tiles from the backends. We can already read in single values (e.g. Tiles), and RDDs of tiles. With the completion of this task, we will read some collection of values (e.g. Vector[(SpatialKey, Tile)]) out of the backend. This should be able to use the Query API to query the backend in order to produce the collection.

A potential interface for this type of reader is follows, which is distinctly similar to FilteringLayerReader

  abstract class CollectionLayerReader[ID] {
    def read[
      K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
      V: AvroRecordCodec: ClassTag,
      M: JsonFormat: GetComponent[?, Bounds[K]]
    ](id: ID, rasterQuery: LayerQuery[K, M], numPartitions: Int, indexFilterOnly: Boolean): Vector[(K, V)] with Metadata[M]

    def read[
      K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
      V: AvroRecordCodec: ClassTag,
      M: JsonFormat: GetComponent[?, Bounds[K]]
    ](id: ID, rasterQuery: LayerQuery[K, M], numPartitions: Int): Vector[(K, V)] with Metadata[M] =
      read(id, rasterQuery, numPartitions, false)

    def read[
      K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
      V: AvroRecordCodec: ClassTag,
      M: JsonFormat: GetComponent[?, Bounds[K]]
    ](id: ID, rasterQuery: LayerQuery[K, M]): Vector[(K, V)] with Metadata[M] =
      read(id, rasterQuery, defaultNumPartitions)

    def read[
      K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
      V: AvroRecordCodec: ClassTag,
      M: JsonFormat: GetComponent[?, Bounds[K]]
    ](id: ID, numPartitions: Int): Vector[(K, V)] with Metadata[M] =
      read(id, new LayerQuery[K, M], numPartitions)

    def query[
      K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
      V: AvroRecordCodec: ClassTag,
      M: JsonFormat: GetComponent[?, Bounds[K]]
    ](layerId: ID): BoundLayerQuery[K, M, Vector[(K, V)] with Metadata[M]] =
      new BoundLayerQuery(new LayerQuery, read(layerId, _))

    def query[
      K: AvroRecordCodec: Boundable: JsonFormat: ClassTag,
      V: AvroRecordCodec: ClassTag,
      M: JsonFormat: GetComponent[?, Bounds[K]]
    ](layerId: ID, numPartitions: Int): BoundLayerQuery[K, M, Vector[(K, V)] with Metadata[M]] =
      new BoundLayerQuery(new LayerQuery, read(layerId, _, numPartitions))
  }

This might require us to have a similar type as ContextRDD for the collection type, e.g. ContextCollection

[lossyrob]

/cc @pomadchin

[pomadchin]

@lossyrob great, thanks!

[fosskers]

This can be closed @lossyrob @echeipesh @rossbernet.