Windowed Reading GeoTiffs from S3 and Hdfs #1763
Conversation
|
@jbouffard can you make sure to include the changes @pomadchin mentions here: #1758 (comment) |
|
@lossyrob Yeah, I was just about to do that. |
Fixed issue with json
jbouffard
force-pushed
the
geotiff-rdd
branch
from
ad86812 to
d54fe8c
Compare
Made another change to HadoopGeoTiffRDDSpec Yet Another revision
jbouffard
force-pushed
the
geotiff-rdd
branch
2 times, most recently
from
f393fb2 to
2eaecd5
Compare
Fixed issue that caused Travis to fail Fixed the errors for real this time
jbouffard
force-pushed
the
geotiff-rdd
branch
from
2eaecd5 to
cf6e572
Compare
jbouffard
changed the title
| @@ -51,12 +51,9 @@ object ArraySegmentBytes { | |||
| val result = Array.ofDim[Array[Byte]](offsets.size) | |||
|
|
|||
| cfor(0)(_ < offsets.size, _ + 1) { i => | |||
| byteReader.position(offsets(i).toInt) | |||
| result(i) = byteReader.getSignedByteArray(byteCounts(i).toInt) | |||
| result(i) = byteReader.getSignedByteArray(byteCounts(i), offsets(i)) | |||
There was a problem hiding this comment.
Stale doc string, byteBuffer no longer a prameter
There was a problem hiding this comment.
oldOffset is unused. Looks like the previous version of this file used it to return position, I would guess thats not a good idea and this variable needs to be removed now.
There was a problem hiding this comment.
Fixed the doc string. Yeah, oldOffset was actually always redundant since getSignedByteArray does it already.
| @@ -51,12 +51,9 @@ object ArraySegmentBytes { | |||
| val result = Array.ofDim[Array[Byte]](offsets.size) | |||
|
|
|||
| cfor(0)(_ < offsets.size, _ + 1) { i => | |||
| byteReader.position(offsets(i).toInt) | |||
| result(i) = byteReader.getSignedByteArray(byteCounts(i).toInt) | |||
| result(i) = byteReader.getSignedByteArray(byteCounts(i), offsets(i)) | |||
There was a problem hiding this comment.
This maybe a conversation that already happened but why are there ByteReaderExtensions when we control the trait we're extending? Why shouldn't getSignedByteArray and others be methods on ByteReader trait directly?
There was a problem hiding this comment.
ByteReader is meant to be a replacement for ByteBuffer, so I wanted the two to have the same methods (or at least have them be as close as possible). Since none of the methods in ByteReaderExtensions are in ByteBuffer, they were made into their own separate thing.
| @@ -17,9 +17,9 @@ import spire.syntax.cfor._ | |||
| * @param byteReader: A ByteReader that contains bytes of the GeoTiff | |||
| * @param storageMethod: The [[StorageMethod]] of the GeoTiff | |||
| * @param tifftags: The [[TiffTags]] of the GeoTiff | |||
| * @return A new instance of BufferSegmentBytes | |||
| * @return A new instance of LazySegmentBytes | |||
There was a problem hiding this comment.
@param section is out of sync with code
| @@ -173,33 +173,22 @@ trait ByteReaderExtensions { | |||
| } | |||
|
|
|||
| final def getSignedByteArray(length: Long, valueOffset: Long): Array[Byte] = { | |||
| val arr = Array.ofDim[Byte](length.toInt) | |||
|
|
|||
| // NOTE: We don't support lengths greater than Int.MaxValue yet (or ever). | |||
There was a problem hiding this comment.
This comment should be in a doc string
There was a problem hiding this comment.
Also if so, why is the length parameter Int?
There was a problem hiding this comment.
Fixed it. Why it was converted to Int? That's because you can't make an Array.ofDim using a Long.
This PR was based on another, preexisting PR: #1758
Two new features are added to Geotrellis with this PR: Reading windowed GeoTiffs from S3 and Hdfs. These features will allow for the breaking up of large GeoTiffs into smaller tiles over a distributed system. By using this method, the computation time for each GeoTiff that needs to analyzed can be shortened considerably.
To do:
SpatialGeoTiffsTemporalGeoTiffs