Drop many "detail" tags on minor roads at mid zooms. #1710

zerebubuth · 2018-11-22T15:58:59Z

NOTE: This follows on from #1707 and #1708, so those should have been merged before this.

On minor roads (only - this doesn't apply to major roads or highways), drop the following from zoom 15 down:

colour
motor_vehicle
operator
route
route_name
state
symbol
type

And the following from zoom 14 down:

surface

And the following from zoom 13 down:

cutting
embankment

This helps a lot with road merging, as the fewer distinct combinations of road properties there are, the more roads can be merged into one another.

Also, drop most values of the access property from zoom 13 down, keeping access=no, access=private and access=official (on all road types). The no and private values are important for knowing which streets are inaccessible, but many other distinctions are unimportant at this zoom level, where streets might be a few pixels long, e.g: access=customers (if styled at all) can be made prominent as the view is zoomed in a bit more.

Further, we merge some access values together; emergency, military, restricted and forestry all map to access=official, which indicates that there is some kind of restriction, but allows for geometry merging and value compression through re-use.

Finally, drop landuse_kind on roads on zoom < 13 where it's one of residential or industrial. These are very common landuse kinds, but aren't strongly styled. At zoom 13 they're getting to be a fraction of the size of a tile and in my opinion of limited usefulness.

These measures together produced a 14% reduction in the number of minor_road features on 12/1206/1540, reflected in a similar drop in the size of the metadata and properties for the MVT tile. Overall, the roads layer was 4% smaller (about 3kB for this tile).

Connects to #1331.

Drop RCN from 12 down to 0, LCN from 15 down to 0, and any network on track paths from 12 down to zero. These properties inhibit some degree of merging at mid zooms. Context: #1331 (comment)

…ess-surface

On minor roads (only - this doesn't apply to major roads or highways), drop the following from zoom 15 down: * `colour` * `motor_vehicle` * `operator` * `route` * `route_name` * `state` * `surface` * `symbol` * `type` And the following from zoom 13 down: * `cutting` * `embankment` This helps a lot with road merging, as the fewer distinct combinations of road properties there are, the more roads can be merged into one another. Also, drop most values of the `access` property from zoom 13 down, keeping `access=no`, `access=private` and `access=official`. The `no` and `private` values are important for knowing which streets are inaccessible, but many other distinctions are unimportant at this zoom level, where streets might be a few pixels long, e.g: `access=customers` (if styled at all) can be made prominent as the view is zoomed in a bit more. Further, we merge some `access` values together; `emergency`, `military`, `restricted` and `forestry` all map to `access=official`, which indicates that there is some kind of restriction, but allows for geometry merging and value compression through re-use. These measures together produced a 14% reduction in the number of `minor_road` features on `12/1206/1540`, reflected in a similar drop in the size of the metadata and properties for the MVT tile. Overall, the `roads` layer was 4% smaller (about 3kB for this tile).

nvkelso · 2018-11-27T05:43:52Z

queries.yaml

+      start_zoom: 0
+      end_zoom: 15
+      properties:
+        - surface


FYI: It looks like surface is used to set "brown" color on line and text halo in Walkabout at some zooms, but looks like only for kind = path so we're good?

https://github.com/tangrams/walkabout-style/blob/gh-pages/layers/bike-interlay.yaml#L705-L708

nvkelso · 2018-11-27T05:50:38Z

queries.yaml

+      end_zoom: 16
+      properties:
+        - colour
+        - motor_vehicle


It looks like motor_vehicle = no could be used for minor_road starting at zoom 12:

https://github.com/tangrams/walkabout-style/blob/gh-pages/walkabout-style.yaml#L2294-L2304

nvkelso · 2018-11-27T05:53:33Z

queries.yaml

+        - cutting
+        - embankment
+      where: >-
+        kind == 'minor_road'


Could probably be for any kind (not just minor_road)...

nvkelso · 2018-11-27T05:54:00Z

queries.yaml

+    params:
+      source_layer: roads
+      start_zoom: 0
+      end_zoom: 14


nvkelso · 2018-11-27T05:55:38Z

queries.yaml

@@ -985,6 +1052,42 @@ post_process:
      properties:
        - walking_network
        - walking_shield_text
+  # drop RCN network & shield text below zoom 13.


This PR has commits from the other PR, which has requested changes...

nvkelso · 2018-11-27T05:57:00Z

queries.yaml

+      # don't want to go there", while still not being quite the same thing as
+      # access=private or access=no, which are both "you definitely don't want
+      # to go there".
+      remap:


nvkelso

Big savings!

Suggest splitting out motor_vehicle to separate statement to avoid Walkabout regression, and expanding embankment filter beyond minor_road kind, and then housekeeping around bicycle commits.

…tion is dropped.

…ties' into zerebubuth/1331-access-surface

…Strip cutting and embankment from all roads, not just minor roads.

zerebubuth · 2018-11-27T16:26:46Z

I merged the changes in #1707 into this (so ignore anything bicycle-related). I think I've addressed your comments in 502c0a6, but please take another look to be sure.

nvkelso · 2018-11-27T17:21:14Z

queries.yaml

        - all_walking_networks
        - all_walking_shield_texts
        - all_bus_networks
        - all_bus_shield_texts
+  # drop all_bicycle_* until the max zoom:
+  # https://github.com/tilezen/vector-datasource/pull/1707#discussion_r236522176


Ah, I didn't expand the diff enough in the earlier review.

It's probably better to be more consistent and just drop all the all* in the same zoom range. I'd be fine keeping that end zoom 14, but also...

To address the merging you called out for minor_road drop all all* for minor_road only thru zoom 15 (end_zoom zoom 16) max zoom?

It's a bit of a corner-case thing, since we only export all_* properties on JSON tiles. I like them for multi-shielding, but AFAIK, no one is using it for that and it does affect merging... So perhaps this is something that we come back to once we're into multi-shielding territory?

nvkelso

With a fresh set of eyes I now see what you're doing and agree more with your earlier approach, but with a modification as proposed in https://github.com/tilezen/vector-datasource/pull/1710/files#r236765704.

nvkelso

LGTM

Worth pointing out that this reverts #1252 so /cc @matkoniecz.

zerebubuth · 2018-11-28T11:17:21Z

Worth pointing out that this reverts #1252

It doesn't fully revert it - we are now dropping surface for minor_road only. So it's still available on other road kinds, including paths.

matkoniecz · 2018-11-28T12:45:15Z

I see that it is too late for a comment. Is it OK to open a new issue if it will break my map?

zerebubuth · 2018-11-28T15:42:15Z

Yes, please do! Apologies if the change breaks your map - this is part of an effort to reduce tile size, but we don't want to break anything.

We've been trying to find places where we've got unused amounts of detail in the feature properties. Please let us know what properties you are using and we'll try to accommodate them.

matkoniecz · 2018-11-28T17:41:50Z

Currently, I am using surface tag across all zoom levels to decide rendering for road (rather than typical using highway tag as - for my private cycling oriented rendering - that is why I made #1252).

But maybe for low zoom levels it is safe anyway to assume that displayed roads are well paved ones or to switch from cycling oriented display of surface to overview map.

nvkelso · 2018-11-28T17:47:57Z

@matkoniecz can you remind us of the public URL or app that shows off your cycling map? If the YAML is available that'd be a bonus, can be shared privately :)

matkoniecz · 2018-11-28T17:50:01Z

https://github.com/matkoniecz/Zazolc (personal fork of StreetComplete).

Style is at https://github.com/matkoniecz/Zazolc/blob/fork/app/src/main/assets/map_theme/fork-style.yaml

zerebubuth · 2018-11-30T13:15:55Z

I notice that you use 3 categories of surface. Would it be OK if we remapped to those values (or perhaps paved, compacted, unpaved rather than good, medium, bad), and perhaps set some defaults (e.g: kind in (highway, major_road, minor_road) ⇒ paved, maybe kind == path ⇒ unpaved)? Hopefully that would give you enough information to render the style that you want while reducing the number of distinct values for data in the tiles?

matkoniecz · 2018-11-30T13:28:03Z

perhaps set some defaults

That would be great! On my TODO list is implementing exactly this within a map style itself.

Would it be OK if we remapped to those values

Yes, though note that this mapping may be different for other purposes (grass_paver and mud go IMHO in one category for a cyclist - it would be likely different for a car driver).

nvkelso · 2018-11-30T17:11:26Z

To make it easier to follow along just in this issue:

            # paved
            good_surface: 
                filter: {surface: [asphalt, paved, wood, metal, tartan, metal_grid]}

            # compacted
            medium_surface:
                filter: {surface: [concrete, paving_stones, compacted, sett]}

            # unpaved
            bad_surface:
                filter: {surface: [unpaved, dirt, earth, fine_gravel, grass, grass_paver, gravel,
                ground, mud, pebblestone, salt, sand, woodchips, clay, cobblestone,
                cobblestone:flattened, concrete:lanes, concrete:plates, artificial_turf, decoturf]}
                draw:

@zerebubuth are you proposing to remap just at low zooms but the full surface value would still be present at high zooms (my preference, and we've talked about doing the same for landuse before)? What zoom would the cutoff be? Right now the dropping property cutoff is zoom 15 (link).

- FYI: In Walkabout `surface`: **compacted** is used [as earlier](https://github.com/tangrams/walkabout-style/blob/gh-pages/layers/bike-interlay.yaml#L1189-L1192) as zoom 8 on paths.

The full and raw values would still be available past that zoom, so I think these 3 category mappings (paved, compacted, unpaved) are fine, even for cyclist since they could zoom in and see full details.

I don't like inventing default surface data on roads that are missing the data. It's better for a style to just have a "no data" symbolization when the data isn't there.

Should we drop surface: paved all together from low zooms so it's implied? I think it's better to include it (so the set is complete and represents the true data), unless it balloons the tile size.

nvkelso · 2018-11-30T17:11:53Z

(and it's almost time to create a new issue for this!)

zerebubuth · 2018-11-30T17:24:55Z

I don't like inventing default surface data on roads that are missing the data

I don't either. But there are quite a few minor roads even in Brooklyn which get marked as surface=paved or surface=asphalt. Most roads in Brooklyn don't have a surface tag, though, so this means it's not possible to merge the paved roads with the unknown surface roads.

In Brooklyn, I think it's sensible to say that surface=paved is the default (as it is in London despite the maintenance authorities' apparent attempts to re-wild them) and at some level surface=paved subsumes the more detailed tag surface=asphalt, allowing us to merge all those minor roads together.

The savings at zoom 13 aren't huge, but it helps keep a lid on tag proliferation, as the number of distinct property sets that we have is approximately the number of distinct tag values multiplied together. It gets big pretty quickly, and even a factor of 2 can make a difference.

I'd be open to applying this to different road types separately, or coming in at different zooms. The real problem is when we have a very dense road network on a single tile, and the worst offender for that is minor_roads at zooms 13 and 14. If we're happy to do this tag merging only for minor_roads at 13 and 14, then I think we get most of the benefit without too much reduction in detail. What do you think?

nvkelso · 2018-11-30T17:35:45Z

Interesting. A case of "go big, or go home, there is no middle way", lol.

I think we should do the mapping into those 3 (plus null) general values at all low zooms for all roads layer features.

But let's try only backfilling the values for minor_road features at zooms 13 and 14 like you describe. Good idea!

nvkelso · 2018-12-03T23:15:52Z

Continued the discussion into #1716.

zerebubuth added 4 commits November 19, 2018 12:38

Drop bicycle network and shield text to match style.

9efcb8a

Drop RCN from 12 down to 0, LCN from 15 down to 0, and any network on track paths from 12 down to zero. These properties inhibit some degree of merging at mid zooms. Context: #1331 (comment)

Merge remote-tracking branch 'origin/master' into zerebubuth/1331-acc…

0b519ee

…ess-surface

Keep surface tag until zoom 15, seems like most tests expect that.

8fc5b07

zerebubuth requested a review from nvkelso November 22, 2018 15:58

nvkelso reviewed Nov 27, 2018

View reviewed changes

queries.yaml

params:

source_layer: roads

start_zoom: 0

end_zoom: 14

Copy link

Member

nvkelso Nov 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

nvkelso reviewed Nov 27, 2018

View reviewed changes

nvkelso requested changes Nov 27, 2018

View reviewed changes

zerebubuth added 4 commits November 27, 2018 13:05

Drop all_bicycle_* until the max zoom, for all network types.

2f7728a

Be more conservative about the zoom levels at which LCN & RCN informa…

324e67f

…tion is dropped.

Merge remote-tracking branch 'origin/zerebubuth/1331-drop-road-proper…

daa90c7

…ties' into zerebubuth/1331-access-surface

Keep motor_vehicle at lower zooms and whitelist to important values. …

502c0a6

…Strip cutting and embankment from all roads, not just minor roads.

nvkelso reviewed Nov 27, 2018

View reviewed changes

nvkelso requested changes Nov 27, 2018

View reviewed changes

nvkelso approved these changes Nov 28, 2018

View reviewed changes

zerebubuth merged commit b29ccd6 into master Nov 28, 2018

zerebubuth deleted the zerebubuth/1331-access-surface branch November 28, 2018 11:17

This was referenced Dec 2, 2018

OUTDATED: Add is_paved boolean for road layer features #1335

Closed

Reduce surface tag precision at mid-zooms to 3 values #1716

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Drop many "detail" tags on minor roads at mid zooms. #1710

Drop many "detail" tags on minor roads at mid zooms. #1710

zerebubuth commented Nov 22, 2018

nvkelso Nov 27, 2018 •

edited

Loading

nvkelso Nov 27, 2018

nvkelso Nov 27, 2018

nvkelso Nov 27, 2018

nvkelso Nov 27, 2018

nvkelso Nov 27, 2018

nvkelso left a comment •

edited

Loading

zerebubuth commented Nov 27, 2018

nvkelso Nov 27, 2018 •

edited

Loading

zerebubuth Nov 27, 2018

nvkelso Nov 28, 2018

nvkelso left a comment

nvkelso left a comment

zerebubuth commented Nov 28, 2018

matkoniecz commented Nov 28, 2018

zerebubuth commented Nov 28, 2018

matkoniecz commented Nov 28, 2018

nvkelso commented Nov 28, 2018

matkoniecz commented Nov 28, 2018 •

edited

Loading

zerebubuth commented Nov 30, 2018

matkoniecz commented Nov 30, 2018

nvkelso commented Nov 30, 2018

nvkelso commented Nov 30, 2018

zerebubuth commented Nov 30, 2018

nvkelso commented Nov 30, 2018

nvkelso commented Dec 3, 2018

Drop many "detail" tags on minor roads at mid zooms. #1710

Drop many "detail" tags on minor roads at mid zooms. #1710

Conversation

zerebubuth commented Nov 22, 2018

nvkelso Nov 27, 2018 • edited Loading

Choose a reason for hiding this comment

nvkelso Nov 27, 2018

Choose a reason for hiding this comment

nvkelso Nov 27, 2018

Choose a reason for hiding this comment

nvkelso Nov 27, 2018

Choose a reason for hiding this comment

nvkelso Nov 27, 2018

Choose a reason for hiding this comment

nvkelso Nov 27, 2018

Choose a reason for hiding this comment

nvkelso left a comment • edited Loading

Choose a reason for hiding this comment

zerebubuth commented Nov 27, 2018

nvkelso Nov 27, 2018 • edited Loading

Choose a reason for hiding this comment

zerebubuth Nov 27, 2018

Choose a reason for hiding this comment

nvkelso Nov 28, 2018

Choose a reason for hiding this comment

nvkelso left a comment

Choose a reason for hiding this comment

nvkelso left a comment

Choose a reason for hiding this comment

zerebubuth commented Nov 28, 2018

matkoniecz commented Nov 28, 2018

zerebubuth commented Nov 28, 2018

matkoniecz commented Nov 28, 2018

nvkelso commented Nov 28, 2018

matkoniecz commented Nov 28, 2018 • edited Loading

zerebubuth commented Nov 30, 2018

matkoniecz commented Nov 30, 2018

nvkelso commented Nov 30, 2018

nvkelso commented Nov 30, 2018

zerebubuth commented Nov 30, 2018

nvkelso commented Nov 30, 2018

nvkelso commented Dec 3, 2018

nvkelso Nov 27, 2018 •

edited

Loading

nvkelso left a comment •

edited

Loading

nvkelso Nov 27, 2018 •

edited

Loading

matkoniecz commented Nov 28, 2018 •

edited

Loading