Code

Streaming/Windowed GeoTiff Reading#1905

Code

Merged

echeipesh merged 90 commits intolocationtech:masterlocationtech/geotrellis:masterfrom echeipesh:chunky-streamingecheipesh/geotrellis:chunky-streamingCopy head branch name to clipboardMar 10, 2017

+790-1,230

Commits

Commits on Dec 5, 2016

• Added StreamingSegmentBytes

  Show description for e1b08c1
  jbouffard

  committed
  e1b08c1

  Copy full SHA for e1b08c1

• Integrated StreamingSegmentBytes in SinglebandGeoTiffs

  Show description for ccc9a67
  jbouffard

  committed
  ccc9a67

  Copy full SHA for ccc9a67

• Integrated StreamingSegmentBytes in MultibandGeoTiffs

  Show description for fb625c6
  jbouffard

  committed
  fb625c6

  Copy full SHA for fb625c6

• Integrated StreamingSegmentBytes in GeoTiffReader

  Show description for 4bf33df
  jbouffard

  committed
  4bf33df

  Copy full SHA for 4bf33df

• Integrated StreamingSegmentBytes into Int16GeoTiffTile

  Show description for 0242cf6
  jbouffard

  committed
  0242cf6

  Copy full SHA for 0242cf6

• Integrated StreamingSegmentBytes into UByteGeoTiffTile

  Show description for cc716f7
  jbouffard

  committed
  cc716f7

  Copy full SHA for cc716f7

• Integrated StreamingSegmentBytes into Int16GeoTiffTile

  Show description for ac2bac5
  jbouffard

  committed
  ac2bac5

  Copy full SHA for ac2bac5

• Added StreamingSegmentBytesSpec

  Show description for 66c9a4d
  jbouffard

  committed
  66c9a4d

  Copy full SHA for 66c9a4d

Commits on Dec 6, 2016

• Integrated StreamingSegmentBytes into ByteGeoTiffTile

  Show description for 60fcd37
  jbouffard

  committed
  60fcd37

  Copy full SHA for 60fcd37

• Integrated StreamingSegmentBytes into UInt16GeoTiffTile

  Show description for 95b443b
  jbouffard

  committed
  95b443b

  Copy full SHA for 95b443b

• Integrated StreamingSegmentBytes into UInt32GeoTiffTile

  Show description for 5c0dde4
  jbouffard

  committed
  5c0dde4

  Copy full SHA for 5c0dde4

• Integrated StreamingSegmentBytes into Float32GeoTiffTile

  Show description for 03261f0
  jbouffard

  committed
  03261f0

  Copy full SHA for 03261f0

• Integrated StreamingSegmentBytes into Float64GeoTiffTile

  Show description for 6b4799f
  jbouffard

  committed
  6b4799f

  Copy full SHA for 6b4799f

• Added intersectingSegments to ArraySegmentBytes

  Show description for 70d3c35
  jbouffard

  committed
  70d3c35

  Copy full SHA for 70d3c35

• Removed LazySegmentBytes

  Show description for f06d71f
  jbouffard

  committed
  f06d71f

  Copy full SHA for f06d71f

• Added intersectingSegments to SegmentBytes

  Show description for 0b948bb
  jbouffard

  committed
  0b948bb

  Copy full SHA for 0b948bb

• Continuing debugging work

  Show description for 3c21091
  jbouffard

  committed
  3c21091

  Copy full SHA for 3c21091

• Continued to debug StreamingSegmentBytes

  Show description for ebc1c42
  jbouffard

  committed
  ebc1c42

  Copy full SHA for ebc1c42

• Removed LazySegmentBytes from SegmentBytesSpec

  Show description for e930661
  jbouffard

  committed
  e930661

  Copy full SHA for e930661

• Removed some debugging code and extranious methods from StreamingSegmentBytes

  Show description for ae257e8
  jbouffard

  committed
  ae257e8

  Copy full SHA for ae257e8

• Updated StreamingSegmentBytesSpec to reflect the changes to StreamingSegmentBytes

  Show description for 4c88e9e
  jbouffard

  committed
  4c88e9e

  Copy full SHA for 4c88e9e

• Moved MergeQueue from the Spark to Util

  Show description for 14a9562
  jbouffard

  committed
  14a9562

  Copy full SHA for 14a9562

• Updated the import path for MergeQueue in all files that use it

  Show description for 94428db
  jbouffard

  committed
  94428db

  Copy full SHA for 94428db

• Began work on reading consecutive segments as one chunk

  Show description for 46b46cc
  jbouffard

  committed
  46b46cc

  Copy full SHA for 46b46cc

• Removed bug that created duplicate chunks

  Show description for dbd8f42
  jbouffard

  committed
  dbd8f42

  Copy full SHA for dbd8f42

• Switched paramters around in ByteReaderExtensions

  Show description for 3f00aee
  jbouffard

  committed
  3f00aee

  Copy full SHA for 3f00aee

• Updated TiffTagsReader to reflect the changes in ByteReaderExtensions

  Show description for 3d0058c
  jbouffard

  committed
  3d0058c

  Copy full SHA for 3d0058c

• Continued to bug fix

  Show description for 011f041
  jbouffard

  committed
  011f041

  Copy full SHA for 011f041

Commits on Dec 8, 2016

• Fix streaming segment chunking

  echeipesh

  committed
  aae773a

  Copy full SHA for aae773a

• Updated the various GeoTiffTiles crop methods to reflect the changes made in StreamingSegmentBytes

  Show description for 82c97e3
  jbouffard

  committed
  82c97e3

  Copy full SHA for 82c97e3

• All tests now run in the WindowedReadingSpec

  Show description for 018490d
  jbouffard

  committed
  018490d

  Copy full SHA for 018490d

• Added readStrippedSegmentBytes and readTileSegmentBytes methods to GeoTiffTile

  Show description for 2d29425
  jbouffard

  committed
  2d29425

  Copy full SHA for 2d29425

• Changed how bytes are read in for the ByteGeoTiffTile

  Show description for 131e0bd
  jbouffard

  committed
  131e0bd

  Copy full SHA for 131e0bd

• Changed how bytes are read in for the UByteGeoTiffTile

  Show description for 3e390a9
  jbouffard

  committed
  3e390a9

  Copy full SHA for 3e390a9

• Changed how bytes are read in for the Int16GeoTiffTile

  Show description for 98226cf
  jbouffard

  committed
  98226cf

  Copy full SHA for 98226cf

• Changed how bytes are read in for the Int32GeoTiffTile

  Show description for 06a5279
  jbouffard

  committed
  06a5279

  Copy full SHA for 06a5279

• Changed how bytes are read in for the UInt32GeoTiffTile

  Show description for e3ce0ef
  jbouffard

  committed
  e3ce0ef

  Copy full SHA for e3ce0ef

• Changed how bytes are read in for the Float32GeoTiffTile

  Show description for 02d4747
  jbouffard

  committed
  02d4747

  Copy full SHA for 02d4747

• Changed how bytes are read in for the Float64GeoTiffTile

  Show description for d918773
  jbouffard

  committed
  d918773

  Copy full SHA for d918773

Commits on Dec 9, 2016

• Add .crop to GeoTiffSegmentLayout

  echeipesh

  committed
  ed04db0

  Copy full SHA for ed04db0

• Update comments in GeoTiffSegmentLayout

  echeipesh

  committed
  e0cac50

  Copy full SHA for e0cac50

• Add GeoTiffSegmentLayout.intersectingSegments

  echeipesh

  committed
  1ca071e

  Copy full SHA for 1ca071e

• Reimplement GeoTiffTile.crop using segment byte assign instead of byte range copies

  echeipesh

  committed
  ef74b86

  Copy full SHA for ef74b86

• Put GridIndexTransform into its own file

  Show description for 85bd642
  jbouffard

  committed
  85bd642

  Copy full SHA for 85bd642

• Created StripedSegmentTransform that extends GridIndexTransform

  Show description for 4818546
  jbouffard

  committed
  4818546

  Copy full SHA for 4818546

• Created TiledSegmentTransform that extends GridIndexTransform

  Show description for e812304
  jbouffard

  committed
  e812304

  Copy full SHA for e812304

• Removed GridIndexTransform from GeoTiffSegmentLayout

  Show description for 0c4a82b
  jbouffard

  committed
  0c4a82b

  Copy full SHA for 0c4a82b

• Reimplamented how segmentTransform is called in GeoTiffTile

  Show description for 8545e04
  jbouffard

  committed
  8545e04

  Copy full SHA for 8545e04

• Continued reimplamentation of segmentTransform calls in GeoTiffTile

  Show description for ed88c6f
  jbouffard

  committed
  ed88c6f

  Copy full SHA for ed88c6f

• Reimplamented how segmentTransform is called in GeoTiffMultibandTile

  Show description for a242844
  jbouffard

  committed
  a242844

  Copy full SHA for a242844

• Reimplamented how segmentTransform is called in BitGeoTiffTile

  Show description for 18d0060
  jbouffard

  committed
  18d0060

  Copy full SHA for 18d0060

• Reimplamented how segmentTransform is called in Int16GeoTiffTile

  Show description for a9499a8
  jbouffard

  committed
  a9499a8

  Copy full SHA for a9499a8

• Reimplamented how segmentTransform is called in UInt16GeoTiffTile

  Show description for 3e87239
  jbouffard

  committed
  3e87239

  Copy full SHA for 3e87239

• Reimplamented how segmentTransform is called in UInt32GeoTiffTile

  Show description for 5446eff
  jbouffard

  committed
  5446eff

  Copy full SHA for 5446eff

• Re-added the getSegmentTransform method to GeoTiffSegmentLayout

  Show description for bdd855a
  jbouffard

  committed
  bdd855a

  Copy full SHA for bdd855a

• Changed how segmentTransform is obtained

  Show description for 35c6353
  jbouffard

  committed
  35c6353

  Copy full SHA for 35c6353

• Removed the crop method from BitGeoTiffTile

  Show description for 4f78e9b
  jbouffard

  committed
  4f78e9b

  Copy full SHA for 4f78e9b

Commits on Dec 12, 2016

• reorganize GeoTiffTile around reading segments in chunks

  echeipesh

  committed
  44bafcd

  Copy full SHA for 44bafcd

• Fix NoData checking in Float32 and Float64 GeoTiff tile

  echeipesh

  committed
  834dd46

  Copy full SHA for 834dd46

• Use getSegments in MultibandGeoTiff

  echeipesh

  committed
  11727a0

  Copy full SHA for 11727a0

• Remove Intersection, no longer used

  echeipesh

  committed
  feddbfb

  Copy full SHA for feddbfb

• Move Striped and Tiled IndexTransform to trait file

  echeipesh

  committed
  e79d2a7

  Copy full SHA for e79d2a7

• StreamingSegmentBytes don't need to know the layout

  echeipesh

  committed
  aa905cd

  Copy full SHA for aa905cd

• Fix reading readSignedFractionalsTag

  echeipesh

  committed
  227c775

  Copy full SHA for 227c775

• Fix double decompression

  echeipesh

  committed
  2a409d9

  Copy full SHA for 2a409d9

• Add logger to StreamingSegmentBytes

  echeipesh

  committed
  30efae3

  Copy full SHA for 30efae3

• Use chunking read in ArraySegmentBytes apply

  echeipesh

  committed
  5c9d0fb

  Copy full SHA for 5c9d0fb

Commits on Dec 16, 2016

• Add GeoTiffSegment.deinterleave

  echeipesh

  committed
  b76f113

  Copy full SHA for b76f113

Commits on Dec 17, 2016

• Add GeoTiffSegment.deinterleaveBits

  echeipesh

  committed
  bbb6317

  Copy full SHA for bbb6317

• Factor out pixel interleave decoding

  echeipesh

  committed
  652b1de

  Copy full SHA for 652b1de

Commits on Feb 25, 2017

• Fix reading multiband bit interleave

  echeipesh

  committed
  4b4675f

  Copy full SHA for 4b4675f

• restore original name of LazySegmentBytes

  echeipesh

  committed
  0567d30

  Copy full SHA for 0567d30

Commits on Feb 27, 2017

• Fix: MultibandTile crop methods need special treatment for geotiff tiles

  echeipesh

  committed
  5a7a712

  Copy full SHA for 5a7a712

• Fetch only intersecting segments during MultibandGeoTiff crop

  echeipesh

  committed
  59a4d47

  Copy full SHA for 59a4d47

Commits on Feb 28, 2017

• Use java.lang.Float.isNaN(nd) test for performance

  echeipesh

  committed
  543361c

  Copy full SHA for 543361c

• more getSegments usage

  Show description for 6fb555f
  pomadchin

  committed
  6fb555f

  Copy full SHA for 6fb555f

• poking around mbgeotiff crop method

  Show description for 3fd8172
  pomadchin

  committed
  3fd8172

  Copy full SHA for 3fd8172

Commits on Mar 1, 2017

• fix crop function

  Show description for 6004b3d
  pomadchin

  committed
  6004b3d

  Copy full SHA for 6004b3d

Commits on Mar 2, 2017

• Merge branch 'master' of github.com:pomadchin/geotrellis into chunky-streaming

  Show description for 2e53e9c
  pomadchin

  committed
  2e53e9c

  Copy full SHA for 2e53e9c

• fix s3 unit tests

  Show description for 947131b
  pomadchin

  committed
  947131b

  Copy full SHA for 947131b

• deinterleave functions fix

  Show description for 0d48955
  pomadchin

  committed
  0d48955

  Copy full SHA for 0d48955

• improve bands and subset functions

  Show description for 71deb84
  pomadchin

  committed
  71deb84

  Copy full SHA for 71deb84

• _combine(initValueHolder: ...) function refactor

  Show description for ec4ccdc
  pomadchin

  committed
  ec4ccdc

  Copy full SHA for ec4ccdc

• combiners arity 2 refactor

  Show description for d78fd98
  pomadchin

  committed
  d78fd98

  Copy full SHA for d78fd98

• use getSegments in GeoTiffWriter

  Show description for 1024743
  pomadchin

  committed
  1024743

  Copy full SHA for 1024743

• multiband combiners improvements, moved boilerplate to use getSegments function

  Show description for 6e75560
  pomadchin

  committed
  6e75560

  Copy full SHA for 6e75560

• fix multiband combiners

  Show description for ff0ec4b
  pomadchin

  committed
  ff0ec4b

  Copy full SHA for ff0ec4b

Commits on Mar 3, 2017

• restore LazySegmentBytes tests

  Show description for e1258ec
  pomadchin

  committed
  e1258ec

  Copy full SHA for e1258ec

Commits on Mar 9, 2017

• minimize API impact of the changes

  echeipesh

  committed
  95c6a27

  Copy full SHA for 95c6a27

• Remove GeoTiffSegmenTransform.crop method until it is fully used

  echeipesh

  committed
  30f8597

  Copy full SHA for 30f8597