Streaming/Windowed GeoTiff Reading

[echeipesh]

The main culprit here is StreamingSegmentBytes. In order to do its job it needs to know ahead of time which segments you will read. At that point it can use the TiffTags to group them into contiguous chunks which can be fetched in on-disk order. This requires slight inversion of control between segmentBytes and GeoTiffTile where we allow the bytes to decide the order in which we see the segment. Overall it looks like this:

cfor(0)(_ < segmentCount, _ + 1) { segmentIndex =>
   val segment = getSegment(segmentIndex).bytes
   val segmentSize = segment.size
   val bandSegmentCount = segmentSize / bandCount
   val bandSegment = Array.ofDim[Byte](bandSegmentCount)
...

is replaced by

getSegments(0 until segmentCount).foreach { case (segmentIndex, geoTiffSegment) =>
    val segment = geoTiffSegment.bytes
    val segmentSize = segment.size
    val bandSegmentCount = segmentSize / bandCount
    val bandSegment = Array.ofDim[Byte](bandSegmentCount)
...

Because getSegments gives us an iterator we can zip them and use them for combine as well:

getSegments(0 until segmentCount)
  .zip(otherGeoTiff.getSegments(0 until segmentCount))
  .foreach { case ((segmentIndex, segment), (otherIndex, otherSegment)) =>
    require(segmentIndex == otherIndex, s"Segment index mismatch: $segmentIndex != $otherIndex")
    val newBytes = segment.mapWithIndex { (i, z) =>
      f(z, otherSegment.getInt(i))
    }
    arr(segmentIndex) = compressor.compress(newBytes, segmentIndex)
  }

Things that need done before this is done:

• Test on S3 and verify performance on reading fully and windowed
• Test on S3 file reading multiple crops from the same GeoTiff (verify that byteReader doesn't Reset)
• Replace all usages of getSegment with getSegments in GeoTiffMultiBandTile (some a tricksy)
• Update the StreamingSegmentBytesSpec for current code (restored LazySegmentBytes)

[pomadchin]

Took into account all comments; StreamingSegmentBytesSpec is still needs to be done.
@echeipesh @lossyrob

[lossyrob]

Where is the Option[Extent] used? Seems unused.

[lossyrob]

Have we addressed this TODO?

[echeipesh]

@lossyrob
This breaks "public" API by removing createSegment aside from having a bad name the signature of createSegment forces it to use getSegment, which breaks streaming.

Question: getSegment be added back for this PR with deprecation and a warning or is this a forgivable sin?

[echeipesh]

This had to changes from Traversable to Seq because the only way the former can tell its length is to iterate through the collection, which is obviously not great for streaming.

[echeipesh]

Here is a handy test I'm using:

import geotrellis.spark.io.s3._
import geotrellis.spark.io._
import geotrellis.spark._
import geotrellis.raster._
import geotrellis.raster.io._
import geotrellis.raster.io.geotiff._
import geotrellis.proj4._
import geotrellis.vector._

val client = S3Client.DEFAULT

import geotrellis.spark.io.s3.util._
val rr = S3RangeReader("geotrellis-test", "rf-test/356f564e3a0dc9d15553c17cf4583f21-6.tif", client)
val tiff = MultibandGeoTiff.streaming(rr)
val sube = tiff.extent.center.buffer(tiff.extent.width * 0.01).envelope
tiff.crop(sube)