Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sparse stitch method to StitchCollectionMethods #3017

Merged
merged 2 commits into from
Jul 10, 2019

Conversation

CloudNiner
Copy link
Contributor

@CloudNiner CloudNiner commented Jul 8, 2019

Overview

Adds sparseStitch methods to SpatialTileLayoutCollectionStitchMethods. The method uses a different algorithm, and therefore it felt more correct to name it differently from the original stitch(): Raster[V]. Also added a helper so that users who want to use the collection extent don't have to know how to get it, since this is the most likely default use case.

Opened PR to get tests to run, since there's a breaking change on the type bounds of V. Initial testing in a few key places didn't raise any issues but I want the whole test suite to run. Edit: Full test suite passed, but upon further inspection it looks like we might not test stitch anywhere else.

Checklist

  • docs/CHANGELOG.rst updated, if necessary
  • docs guides update, if necessary
  • New user API has useful Scaladoc strings
  • Unit tests added for bug-fix or new feature

Notes

I want to add a unit test or two because there's nothing at all for any of these stitch functions, but that looks like its a bit of an effort since there's no easy test data constructor for the type we need to test. So submitting this for review now and moving on to a different task. I can attempt to add a test or two once we've settled on approval for the API presented here.

Closes #2903

@pomadchin pomadchin changed the title WIP: Add sparse stitch method to StitchCollectionMethods Add sparse stitch method to StitchCollectionMethods Jul 8, 2019
@CloudNiner CloudNiner force-pushed the feature/sparse-stitch#2903 branch from f804627 to f913799 Compare July 9, 2019 13:40
@CloudNiner CloudNiner requested a review from moradology July 9, 2019 14:10
Copy link
Member

@pomadchin pomadchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM actually! Only tests are required.

Also an extra question, mb it makes sense to add this method into SpatialTileLayoutRDDStitchMethods as well?

@CloudNiner
Copy link
Contributor Author

mb it makes sense to add this method into SpatialTileLayoutRDDStitchMethods

Hmm. I'm torn. I think to add that method in that class I'd have to duplicate most of the logic and then replace the self.stitch with self.collect.stitch. Given that, it seems easier, if you have a TileLayerRDD, to just do tileLayerRdd.collect.toSeq.sparseStitch(extent) which should be possible with the current changes in this PR as long as the stitch implicits are in scope. It doesn't hide the collect (which is dangerous) and is still quite clear.

Only tests are required

Done. Added a few simple ones to geotrellis.spark.stitch.CollectionStitchMethodsSpec. It's an artifact of the spark split, but its still weird that a bunch of tests for functionality defined in layer are tested in the spark package. It would be very verbose to test those layer features without some of the spark helpers to construct the collections.

@pomadchin
Copy link
Member

@CloudNiner but we already have this duplication

It is our API problem - that for the current collections API we have to duplicate everything. If you'll look more close - Collection API completely duplicates the RDD API ¯_(ツ)_/¯

I dunno what is better at the moment - to keep consistency or to add ambiguity by adding such a divergence into the API.

@CloudNiner
Copy link
Contributor Author

Ah, if we're going for 100% feature parity between RDD and Collections API, then I guess I have to add it. Bummer.

@pomadchin
Copy link
Member

Yea, I don't disagree with your point about duplication ): but it looks like at least for now it is better to add to keep everything consistent

@CloudNiner
Copy link
Contributor Author

@pomadchin updated to include duplicate methods for RDDs

@CloudNiner CloudNiner force-pushed the feature/sparse-stitch#2903 branch from f3d753b to 19ab580 Compare July 9, 2019 19:54
Copy link
Contributor

@moradology moradology left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider the use of ++ instead of :::, but otherwise LGTM

@CloudNiner CloudNiner force-pushed the feature/sparse-stitch#2903 branch from 19ab580 to a4fae4e Compare July 9, 2019 20:49
Copy link
Member

@pomadchin pomadchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, though I already feel too pedantic :/

@pomadchin pomadchin assigned CloudNiner and unassigned moradology Jul 9, 2019
Pulled in from geotrellis/geotrellis-contrib.

The method uses a different algorithm, and therefore
it feels more correct to name it differently from
the original `stitch(): Raster[V]`. Also added a helper
so that users who want to use the collection extent
don't have to know how to get it, since this is the
most likely default use case.
Code duplication not ideal here, but we aim for
100% parity in the APIs, so we can't avoid it without
larger changes.

Signed-off-by: CloudNiner <[email protected]>
@CloudNiner CloudNiner force-pushed the feature/sparse-stitch#2903 branch from a4fae4e to 370b321 Compare July 10, 2019 12:46
@CloudNiner CloudNiner assigned pomadchin and unassigned CloudNiner Jul 10, 2019
Copy link
Member

@pomadchin pomadchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging after the travis green light!

@CloudNiner CloudNiner merged commit f1335c3 into locationtech:master Jul 10, 2019
@CloudNiner CloudNiner deleted the feature/sparse-stitch#2903 branch July 10, 2019 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TileLayoutStitcher.stitch sparseness assumption
3 participants