Support for On-The-Fly GeoJson Layers and a new Mutator Transform #2095

kevinkreiser · 2022-07-13T15:45:35Z

The basic gist here is I want to add 2 features.

i want to be able to drop a new layer into a tile build and be able to reference that layer from queries. if the layer is not there on the filesystem i want the transforms that reference it to do nothing (same thing that happens when a layer isnt found). the bulk of this was accomplished over in: add support for preprocessed inline geojson layers tilequeue#414
i want a new transform that allows me, with injected python, to be able to mutate what the properties or geometry of a given feature becomes. it seemed to me we already had some of these things especially for properties but geometry was lacking.

Once I had those two features implemented I added a landmarks layer to the queries which simply allows you to drop a geojson file into your tile build and mark buildings with it and become a point feature layer.

for more information, see https://pre-commit.ci

…ernal_mutator

kevinkreiser · 2022-07-22T17:13:08Z

vectordatasource/transform.py

+# we expect it to already be in mercator projection and we only fill out the minimal structure of the feature tuple
+# there's no ID, most of the datum stuff is missing, we are mostly worried about the shape and properties
+# we return None if we couldnt load the layer for any reason
+def _load_external_layer(params, name):


this is moving over to tilequeue where the rest of the layers are loaded and ill add a new section to queries.yaml instead of having it be part of the transform

… to load inline layers

kevinkreiser · 2022-07-26T15:10:02Z

queries.yaml

+    area-inclusion-threshold: 1
+    simplify_before_intersect: false
+    simplify_start: 0,
+    layer_path: ./metatile-layers/landmarks.geojson


this property signifies an "inline" layer. when the features are loaded from file we expect them to be already processed. that is we will not try to load a landmarks.yaml and do processing on the data that is in the geojson file. we expect the tags and values to be there and the coordinates to already be in mercator projection

Could we make the name more explicit about its behavior, like pre_processed_layer_path might be a better name?

kevinkreiser · 2022-07-26T15:10:43Z

queries.yaml

@@ -542,6 +552,23 @@ post_process:
      source_layer: roads
      properties: [layer]

+  # supersede existing osm buildings with data from landmark geojson layer


landmarks mark buildings with a superseded property so that when the two overlap the downstream consumer of the tiles will be able to tell

kevinkreiser · 2022-07-26T15:14:21Z

queries.yaml

+      target_attribute: superseded
+      linear: false
+
+  # turn the landmark geojson polygons into point features


the final output of the landmarks layer is a point layer, basically landmark pois with a bunch of metadata about the landmark. the mutate transform lets us both modify the geom and the properties with python code that gets eval'd directly. pretty powerful stuff. you can reference existing properties of a feature using the {properties} tag and you can access the geometry using the {shape} tag. in the future we might want to allow more stuff to be referenced.

kevinkreiser · 2022-07-26T15:14:47Z

queries.yaml

@@ -1865,6 +1892,7 @@ post_process:
        - addr_housenumber
        - addr_street
        - osm_relation
+      exclude: ['superseded']


if a building gets marked by a landmark, we exclude it from merging with other buildings, keeping it a single entity

kevinkreiser · 2022-07-26T15:15:25Z

requirements.txt

+kdtree
 lxml==4.6.2
 mapbox-vector-tile==1.2.0
 ModestMaps==1.4.7
 protobuf==3.4.0
 psycopg2==2.7.3.2
 pyclipper==1.0.6
 pycountry==17.9.23
+pyshp==2.3.0


apparently my system thought i needed these, i didnt check to see if that was the case. i can revert

ah its because setup.py has them

kevinkreiser · 2022-07-26T15:18:56Z

vectordatasource/transform.py

+    """
+    We take a layer and we modify the geometry using the python expression in the query. The expressions have access to
+    both the existing shape and existing properties via python string format replacements {shape} and {properties}
+    respectively. Each expression is then eval'd to replace the existing feature in the layer with the result of the
+    expressions. By default the expressions are a no-op
+    """


pretty good explanation here. basically we loop over the features and eval the python in the queries.yaml to compute new properties or geometry. if you omitted one or the other (they are both optional) it just leaves the existing values unchanged.

…ernal_mutator

peitili · 2022-07-26T22:26:02Z

queries.yaml

+    area-inclusion-threshold: 1
+    simplify_before_intersect: false
+    simplify_start: 0,
+    layer_path: ./metatile-layers/landmarks.geojson


Could we make the name more explicit about its behavior, like pre_processed_layer_path might be a better name?

integration-test/2095_inline_layers.py

vectordatasource/transform.py

…t non string values

…hem downstream anyway

… take a copy

nvkelso · 2022-07-29T23:06:48Z

integration-test/2095_inline_layers.py

+            'crs': {'type': 'name', 'properties': {'name': 'urn:ogc:def:crs:EPSG::3857'}},
+            'features': [
+                {'type': 'Feature',
+                 'properties': {'name': 'null island hut', 'superseded': True, 'height': math.pi, 'origin': [1, 1], 'id': 42},


nvkelso · 2022-07-29T23:07:35Z

queries.yaml

@@ -9,6 +9,7 @@ all:
  - boundaries
  - transit
  - admin_areas
+  - landmarks


See the traffic layers for inspiration for optional layer descriptions.

ha! i was looking around for docs, it seemed like even the changelog isnt up to date. ill grep around to see if i can find them!

https://github.com/tilezen/vector-datasource/blob/master/docs/layers.md#layer-reference
and
https://github.com/tilezen/vector-datasource/blob/master/docs/layers.md#traffic-flow

The CHANGELOG is out of date. Historically that's been updated as releases are tagged – and it's past time to tag v1.9. Thanks for the nudge.

queries.yaml

kevinkreiser · 2022-08-01T15:11:18Z

the additional integration test failures in ci are just because this branch depends on the changes in this pr: tilezen/tilequeue#414

peitili · 2022-08-03T19:51:07Z

vectordatasource/transform.py

+    if end_zoom is not None and zoom >= end_zoom:
+        return None
+
+    # for the max zoom a transform needs to be re-entrant so we take a copy here


what is max zoom here?

ah sorry the last zoom in the tile pyramids that we create (z16). there is code over in tilequeue here: https://github.com/tilezen/tilequeue/blob/master/tilequeue/process.py#L613 where it basically calls into the transforms a second time (hence re-entrancy)

tgrigsby-sc · 2022-08-03T20:51:02Z

I'd remove the strikethrough in the description before merging

kevinkreiser and others added 7 commits July 1, 2022 11:08

first stab at external layers and mutator transform

038ffbb

change name to landmarks

c6ea781

resolve conflict

4f7209c

rename test as pr number

6eb7fff

[pre-commit.ci] auto fixes from pre-commit.com hooks

0afd5e9

for more information, see https://pre-commit.ci

lint my changes let others unlinted

0a6e69e

Merge remote-tracking branch 'origin/kk_external_mutator' into kk_ext…

4f34e34

…ernal_mutator

kevinkreiser commented Jul 22, 2022

View reviewed changes

move landmarks to official layer and rely on other tile queue changes…

3fb87ad

… to load inline layers

kevinkreiser mentioned this pull request Jul 26, 2022

add support for preprocessed inline geojson layers tilezen/tilequeue#414

Merged

kevinkreiser marked this pull request as ready for review July 26, 2022 15:05

Merge branch 'master' into kk_external_mutator

bd764f2

kevinkreiser commented Jul 26, 2022

View reviewed changes

kevinkreiser and others added 2 commits July 26, 2022 11:33

dont need special fixture env for landmarks testing

7927c10

Merge remote-tracking branch 'origin/kk_external_mutator' into kk_ext…

6b64b81

…ernal_mutator

peitili reviewed Jul 26, 2022

View reviewed changes

kevinkreiser added 4 commits July 27, 2022 12:09

rename layer_path, update mutator error text and change test to expec…

4c5f780

…t non string values

origin coords for landmark pois can be in list form as we dont send t…

9ed02c5

…hem downstream anyway

transforms must be re-entrant, since this one destructively modifies,…

4eed194

… take a copy

Merge remote-tracking branch 'origin/master' into kk_external_mutator

8b0249a

nvkelso reviewed Jul 29, 2022

View reviewed changes

queries.yaml Outdated Show resolved Hide resolved

match proper kinds vernacular

7dae635

more nomenclature changes

4bd5e26

peitili reviewed Aug 3, 2022

View reviewed changes

tgrigsby-sc approved these changes Aug 3, 2022

View reviewed changes

kevinkreiser merged commit 4caaedc into master Aug 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for On-The-Fly GeoJson Layers and a new Mutator Transform #2095

Support for On-The-Fly GeoJson Layers and a new Mutator Transform #2095

kevinkreiser commented Jul 13, 2022 •

edited

Loading

kevinkreiser Jul 22, 2022

kevinkreiser Jul 26, 2022

peitili Jul 26, 2022

kevinkreiser Jul 26, 2022

kevinkreiser Jul 26, 2022 •

edited

Loading

kevinkreiser Jul 26, 2022

kevinkreiser Jul 26, 2022

kevinkreiser Jul 26, 2022

kevinkreiser Jul 26, 2022

peitili Jul 26, 2022

nvkelso Jul 29, 2022

nvkelso Jul 29, 2022

nvkelso Jul 29, 2022

kevinkreiser Jul 29, 2022

nvkelso Jul 30, 2022

nvkelso Jul 30, 2022

kevinkreiser commented Aug 1, 2022 •

edited

Loading

peitili Aug 3, 2022

kevinkreiser Aug 3, 2022

tgrigsby-sc commented Aug 3, 2022

Support for On-The-Fly GeoJson Layers and a new Mutator Transform #2095

Support for On-The-Fly GeoJson Layers and a new Mutator Transform #2095

Conversation

kevinkreiser commented Jul 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinkreiser Jul 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinkreiser commented Aug 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgrigsby-sc commented Aug 3, 2022

kevinkreiser commented Jul 13, 2022 •

edited

Loading

kevinkreiser Jul 26, 2022 •

edited

Loading

kevinkreiser commented Aug 1, 2022 •

edited

Loading