ERROR org.apache.spark.executor.Executor ... scala.MatchError: Some() at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo #2

aklink · 2016-10-04T14:54:34Z

When I try:

java -cp target/biggis-landuse-0.0.1-SNAPSHOT.jar
biggis.landuse.spark.examples.GeotiffToPyramid
./data/DOP_RGBI_T2.tif
new_layer
./data/pyramid

I get the following error message:

15:58:50 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)
scala.MatchError: Some() (of class scala.Some)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo(GeoTiffCSParser.scala:570)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.createGeoTiffCSParameters(GeoTiffCSParser.scala:172)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.geoTiffCSParameters$lzycompute(GeoTiffCSParser.scala:78)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.geoTiffCSParameters(GeoTiffCSParser.scala:78)
at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.model(GeoTiffCSParser.scala:80)
at geotrellis.raster.io.geotiff.tags.TiffTags.crs$lzycompute(TiffTags.scala:207)
at geotrellis.raster.io.geotiff.tags.TiffTags.crs(TiffTags.scala:205)
at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readGeoTiffInfo(GeoTiffReader.scala:315)
at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readSingleband(GeoTiffReader.scala:67)
at geotrellis.raster.io.geotiff.reader.GeoTiffReader$.readSingleband(GeoTiffReader.scala:61)
at geotrellis.raster.io.geotiff.SinglebandGeoTiff$.apply(SinglebandGeoTiff.scala:40)
at geotrellis.spark.io.hadoop.formats.GeotiffInputFormat.read(GeotiffInputFormat.scala:28)
at geotrellis.spark.io.hadoop.formats.BinaryFileInputFormat$$anonfun$createRecordReader$1.apply(BinaryFileInputFormat.scala:34)
at geotrellis.spark.io.hadoop.formats.BinaryFileInputFormat$$anonfun$createRecordReader$1.apply(BinaryFileInputFormat.scala:34)
at geotrellis.spark.io.hadoop.formats.BinaryFileRecordReader.initialize(BinaryFileInputFormat.scala:18)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.(NewHadoopRDD.scala:158)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:129)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:64)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

aklink · 2016-10-04T15:13:40Z

Issue is specific to used image.
Trying a landsat image (created with ERDAS Imagine) is ok (no issue).

Trying a different aerila image leads to different error:

15:05:03 ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 3.0 (TID 3)
geotrellis.raster.GeoAttrsError: invalid cols: 0
at geotrellis.raster.RasterExtent.(RasterExtent.scala:72)
at geotrellis.raster.GridDefinition$class.toRasterExtent(GridDefinition.scala:36)
at geotrellis.raster.GridExtent.toRasterExtent(GridExtent.scala:9)
at geotrellis.raster.reproject.ReprojectRasterExtent$.apply(ReprojectRasterExtent.scala:105)
at geotrellis.spark.reproject.TileRDDReproject$$anonfun$6$$anonfun$apply$2.apply(TileRDDReproject.scala:185)
at geotrellis.spark.reproject.TileRDDReproject$$anonfun$6$$anonfun$apply$2.apply(TileRDDReproject.scala:182)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

aklink · 2016-10-04T18:39:15Z

The first issue with "geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo" is caused by an older SRS (Gauss-Krüger Zone 3, used in Baden-Württemberg) in GeoTiff file, can be fixed by applying "gdalwarp -t_srs EPSG:25832 srcfile targetfile" to file to use a newer SRS (ETRS89 UTM32N). Geotrellis seems not to support older SRS.

The second issue with "geotrellis.raster.RasterExtent.(RasterExtent.scala:72)" is caused by a missing SRS in GeoTiff file, can be fixed by applying "gdal_translate -a_srs EPSG:25832 srcfile targetfile" in advance to add a valid SRS (on test file, not in sourcecode).

aklink · 2016-10-13T19:08:02Z

Maybe related to this issues?

aklink · 2016-10-13T19:11:24Z

Problematic CRS is EPSG:31467 in following GDAL representation:

PROJCS["DHDN_3_Degree_Gauss_Zone_3",
    GEOGCS["GCS_Deutsches_Hauptdreiecksnetz",
        DATUM["Deutsches_Hauptdreiecksnetz",
            SPHEROID["Bessel_1841",6377397.155,299.1528128,
                AUTHORITY["EPSG","7004"]],
            AUTHORITY["EPSG","6314"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",9],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",3500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]

vsimko · 2016-10-14T12:54:57Z

It works for my TIF file in EPSG_31467:

$ gdalinfo morning_EPSG_31467.tif
Driver: GTiff/GeoTIFF
Files: morning_EPSG_31467.tif
Size is 5161, 7911
Coordinate System is:
PROJCS["DHDN / 3-degree Gauss-Kruger zone 3",
    GEOGCS["DHDN",
        DATUM["Deutsches_Hauptdreiecksnetz",
            SPHEROID["Bessel 1841",6377397.155,299.1528128000008,
                AUTHORITY["EPSG","7004"]],
            TOWGS84[598.1,73.7,418.2,0.202,0.045,-2.455,6.7],
            AUTHORITY["EPSG","6314"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4314"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",9],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",3500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","31467"]]
Origin = (3445385.870000000111759,5447456.839999999850988)
Pixel Size = (5.000000000000000,-5.000000000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_IMAGEDESCRIPTION=
  TIFFTAG_RESOLUTIONUNIT=3 (pixels/cm)
  TIFFTAG_XRESOLUTION=100
  TIFFTAG_YRESOLUTION=100
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( 3445385.870, 5447456.840) (  8d15' 3.88"E, 49d 9'46.43"N)
Lower Left  ( 3445385.870, 5407901.840) (  8d15'23.00"E, 48d48'25.95"N)
Upper Right ( 3471190.870, 5447456.840) (  8d36'17.74"E, 49d 9'52.74"N)
Lower Right ( 3471190.870, 5407901.840) (  8d36'27.83"E, 48d48'32.18"N)
Center      ( 3458288.370, 5427679.340) (  8d25'48.14"E, 48d59' 9.82"N)
Band 1 Block=5161x1 Type=Byte, ColorInterp=Palette
  Color Table (RGB with 256 entries)
...

aklink · 2016-10-14T13:13:31Z

Seems like the major difference between the two CRS representations is missing
AUTHORITY["EPSG","31467"] as well as TOWGS84[598.1,73.7,418.2,0.202,0.045,-2.455,6.7]
in non-working variant (unfortunatelly all tiles in BW are using this non-working representation)

pomadchin · 2016-11-06T13:47:07Z

Is this still an issue? Can you throw us a non working example, or (as i can understand) the problem is in corrupted (or specific) geotiff metadata? Or it is locationtech/geotrellis#1415 ?

aklink · 2016-11-07T07:52:30Z

I am not sure if this exactly issue locationtech/geotrellis#1415 ...

However it is specific to SRS MetaData of DHDN_3_Degree_Gauss_Zone_3.

When looking on: http://spatialreference.org/ref/epsg/31467/
It is neither exactly http://spatialreference.org/ref/epsg/31467/ogcwkt/
nor http://spatialreference.org/ref/epsg/31467/esriwkt/ due to a slighty different string representation.

Please find attached file using DHDN_3_Degree_Gauss_Zone_3.
DOP_RGBI_T2_400x400_epsg31467.zip

pomadchin · 2016-11-07T09:09:01Z

@aklink thx for attached file, so yes, i believe it is unsupported projection, good info, think it would be fixed (or we'll try to fix it :D) in terms of locationtech/geotrellis#1415 looks like a bug in a proj4j or in tiff metadata.

aklink · 2016-11-21T14:02:52Z

@pomadchin Thanks for looking into this issue. I really appreciate it and I am looking forward to have this projection supported by Geotrellis.

pomadchin · 2017-10-03T15:30:43Z

Yo guyz, can you try this PR locationtech/geotrellis#2403 if it's not too late? :D basically it was a minor bug, sry that it took us half a year to fix it.

aklink · 2017-10-04T19:51:59Z

@pomadchin: Thanks for the update. I will have a look into it.
Anyhow, I have trouble building it using ./scripts/publish-local.sh in Win10.

Downloading sbt launcher for 0.13.16:
  From  http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.16/sbt-launch.jar
    To  /home/ak/.sbt/launchers/0.13.16/sbt-launch.jar
Invalid maximum heap size: -Xmx2g
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Is there any Maven repository with an already built test version? Or another simple option how to test it ontop of an "older" Geotrellis version?

pomadchin · 2017-10-20T08:14:13Z

@aklink sure, we have a location tech snapshot repo:

resolvers ++= Seq(
  "LocationTech GeoTrellis Snapshots" at "https://repo.locationtech.org/content/repositories/geotrellis-snapshots"
)

After that just use 1.2.0-SNAPSHOT as a GeoTrellis version in your deps.

aklink · 2017-11-15T20:23:25Z

@pomadchin
Sorry for my late reply. Thanks! I have tested it with 1.2.0-RC1 and it seems to work fine.
I did not check the georeferenced position yet if it is on the proper location, but at least the error is gone.
Layer is created properly.

aklink closed this as completed Oct 4, 2016

aklink reopened this Oct 13, 2016

pomadchin mentioned this issue Nov 21, 2016

Not possible ingest tiles in some source projections. locationtech/geotrellis#1415

Closed

vsimko added the ready label Dec 22, 2016

vsimko assigned aklink Dec 22, 2016

pomadchin mentioned this issue Oct 3, 2017

Fix CRS parser and proj4 cea projection support locationtech/geotrellis#2403

Merged

aklink closed this as completed Nov 18, 2017

aklink added a commit that referenced this issue Nov 18, 2017

Experimental: MultibandLayerResample (still non-working #2)

17ae31d

aklink mentioned this issue Jun 19, 2018

Wrong Offset Projection Gauss Krueger Zone 3 to WebMercator #17

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR org.apache.spark.executor.Executor ... scala.MatchError: Some() at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo #2

ERROR org.apache.spark.executor.Executor ... scala.MatchError: Some() at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo #2

aklink commented Oct 4, 2016 •

edited by vsimko

Loading

aklink commented Oct 4, 2016

aklink commented Oct 4, 2016

aklink commented Oct 13, 2016

aklink commented Oct 13, 2016

vsimko commented Oct 14, 2016

aklink commented Oct 14, 2016

pomadchin commented Nov 6, 2016 •

edited

Loading

aklink commented Nov 7, 2016

pomadchin commented Nov 7, 2016 •

edited

Loading

aklink commented Nov 21, 2016

pomadchin commented Oct 3, 2017

aklink commented Oct 4, 2017

pomadchin commented Oct 20, 2017

aklink commented Nov 15, 2017

ERROR org.apache.spark.executor.Executor ... scala.MatchError: Some() at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo #2

ERROR org.apache.spark.executor.Executor ... scala.MatchError: Some() at geotrellis.raster.io.geotiff.reader.GeoTiffCSParser.getEllipsoidInfo #2

Comments

aklink commented Oct 4, 2016 • edited by vsimko Loading

aklink commented Oct 4, 2016

aklink commented Oct 4, 2016

aklink commented Oct 13, 2016

aklink commented Oct 13, 2016

vsimko commented Oct 14, 2016

aklink commented Oct 14, 2016

pomadchin commented Nov 6, 2016 • edited Loading

aklink commented Nov 7, 2016

pomadchin commented Nov 7, 2016 • edited Loading

aklink commented Nov 21, 2016

pomadchin commented Oct 3, 2017

aklink commented Oct 4, 2017

pomadchin commented Oct 20, 2017

aklink commented Nov 15, 2017

aklink commented Oct 4, 2016 •

edited by vsimko

Loading

pomadchin commented Nov 6, 2016 •

edited

Loading

pomadchin commented Nov 7, 2016 •

edited

Loading