Separately adjusting rendering over water and over land

[imagico]

By switching to water polygons for the coastlines (#3694) we are now able to better define what to show over water and can get a more consistent drawing order with ocean and polygon mapped waterbodies (like #1547).

What we so far still cannot do is separately design how things appear over water and over land. So far we use partly transparent patterns (beach, wetlands, reefs) and to a limited extent fill color transparency (tidalflats) to achieve some level of differentiation between land and water or in other words: for showing polygons crossing the water line while not completely obscuring the water line.

The price we pay for that is

• the limitation of what design differences between styling over land and water can be achieved and
• that we can't render this in the size based sorting order of the landcover layer so overlapping landcover polygons will result in confusing mixtures between base color and overlay.

The elegant solution to solve this is to use layer compositioning to render the water part separately from the land part. Currently all layers are rendered on top of what has been rendered before and the layer order is the following:

• landcover
• water-lines
• water-areas
• ocean
• landcover overlay

This would need to be modified the following way

• landcover (including any land-only designs of what is currently in the overlay)
• water-lines
• water-areas (cut out with comp-op dst-out)
• ocean (cut out with comp-op dst-out)
• landcover overlay for water only (drawn behind using comp-op dst-over)
• global backdrop to fill in water color (using comp-op dst-over)
• landcover overlay only for things that are truly supposed to be in an overlay

This gives a lot of control but is of course technically somewhat more complicated. And apart from the additional landcover layer over water and the water color backdrop you need for any differentiation of water rendering (like not rendering all water in the same color) to render the features in question twice - once to cut out and then a second time for the differentiated backdrop.

Long story short - we need to consider if we want to do that. #3851 is the current case where this has significance, #3840 is another case where this kind of control could be useful. And the current use of transparency for tidalflat rendering could be avoided as well.

[jeisenbe]

• landcover (including any land-only designs of what is currently in the overlay)
• water-lines
• water-areas (cut out with comp-op dst-out)
• ocean (cut out with comp-op dst-out)
• landcover overlay for water only (drawn behind using comp-op dst-over)
• global backdrop to fill in water color (using comp-op dst-over)
• landcover overlay only for things that are truly supposed to be in an overlay

I would actually suggest using the water polygons rather than a global backdrop; this also works with different colors for rivers and oceans.

And it's required to give us the option of using an outline for oceans, as discussed in #3695 (comment) - I hadn't mentioned this yet, but I had separately come to the conclusion that using comp-op dst-out and then comp-op dst-over would be perfect for this. I didn't think the change was worthwhile to add the ocean outline alone, but since it also allows us to fix problems with patterns over water, it's worth trying like this:

• landcover (including any land-only designs of what is currently in the overlay)
• water-lines
• water-areas (cut out with comp-op dst-out) - (including waterway=riverbank and natural=water)
• ocean (cut out with comp-op dst-out)
• landcover overlay for water only (drawn behind using comp-op dst-over)
• ocean in ocean-color or water-color (using comp-op dst-over) + optionally add coastline line/gradient in water-color / river-color
• water-areas in water-color (using comp-op dst-over) e.g. lakes
• river-areas in river-color (using comp-op dst-over)
• landcover overlay only for things that are truly supposed to be in an overlay over both water and land; eg. military danger areas.

In the rendering, this results in this order, bottom to top:

1. landcover + land-only patterns
2. water-lines
3. river-areas in river-color (water=river areas + waterway=riverbank)
4. water-areas in water-color (e.g. lakes, ponds, reservoirs, etc)
5. ocean polygons in ocean-color or water-color + optional outline / gradient
6. water-only patterns - only over areas of water
7. pattern overlay on top

Then the rest of the layers work as before.

[jeisenbe]

See how Carto-CSS does compositing operations here:
http://carto.com/help/tutorials/cartocss-compositing-operations/

Basically, the default behavior is that each new layer is added over the previous image. Each time a layer is created, a new raster image, with colors and transparence for each pixel, is created, by adding the new layer on top of the previous ones. This default is called src-over, meaning "source over destination"; that is, "the new layer gets put on top of all the previous stuff".

dst-out means "cut the source (new layer) out of the destination (old stuff). When we use dst-out, instead what happens is that the new layer is cut out of the existing rendering (created from all the previous layers), based on the shape of the polygons in the new layer. This lets us use the shape of water-areas and ocean-polygons to "cut out" all water from the previously rendered landcover and water lines.

Then, we can use dst-over, which means destination over source, or "render the new layer behind the previously rendered stuff. This means that everything in the layer will be hidden by any previously rendered layers, only showing up in transparent parts of the map.

Apparently folks who use photoshop and similar image-editing software are quite familiar with these things. We will probably need to add notes to each affected layers in project.mml and in landcover.mss to explain.

[jeisenbe]

Another advantage of this change: we could render water-lines like rivers, ditches and streams on top of all of the current overlay patterns, e.g. woodland/forest, scrub, bare_rock, so the patterns will not make it harder to see watercourses.

We could have a special fill color (blue-green?) for mangrove and saltmarsh areas that are outside of the coastline, and beaches could use a similar color to tidalflat over the ocean, which would make it possible to solve #3840

[imagico]

Yes, you can implement a water area halo this way as well.

I would not recommend completely stripping the landcover-area-symbols layer because it serves an important function to provide feedback to mappers about accurate geometries. Rendering the patterns for scrub/wood/wetland this way makes a lot of sense.

I implemented the whole idea in the ac-style:

imagico@304561b

Here a sample of what kind of options this provides visually:

Care needs to be taken when rendering certain areas land only or water only of course - the way this is shown for reefs (i.e. water only) would not be good for mapper feedback in this style because it would hide mapping inaccuracies.

[jeisenbe]

Thanks for the examples.

I wonder if this change might make it easier to render borders differently over water and land from the boundary relations only. This would provide a nice solution for #621. It also would allow us to get rid of the use of transparency, by rendering the land borders in a solid color right on top of landcover, and then using a transparent border layer later, to show borders semi-transparently over roads and waterways (if desired).

It could be done like this:

1. landcover (including land-only patterns)
2. land borders - background color (no transparency)
3. water-lines
4. water-areas (cut out with dst-out)
5. ocean (cut out with dst-out)
6. marine borders - background color (with dst-over so only over water)
7. landcover overlay over water (drawn behind using comp-op dst-over)
8. ocean fill (using dst-over) + coastline line/gradient - optional
9. water-areas in water-color (using dst-over)
10. river-areas in river-color (using dst-over)
11. landcover overlay (over both water and land)
    ... other layers
12. transparent borders, if we want them to show over roads and other features.

[sommerluk]

See how Carto-CSS does compositing operations here:
http://carto.com/help/tutorials/cartocss-compositing-operations/

Also a good guide:
https://tilemill-project.github.io/tilemill/docs/guides/comp-op/

[jeisenbe]

Re: #3883 - should we move the patterns like forest and scrub to the first layer, so that they do not render over water lines or areas, or do we still want them to render over water areas to encourage better mapping?

If we want to render the tree and scrub patterns over water areas, but not over waterway lines, this would require another layer, like this:

• landcover (+ land-only patterns)
• cut out water-areas and ocean polygons (cut out with comp-op dst-out)
• landcover pattern overlay for water only (drawn behind using comp-op dst-over)
• landcover overlay for patterns over water and over land (src-over - standard layering)
• cut out water-lines (cut out with comp-op dst-out)
• Fill in ocean, water-areas and river-areas (drawn behind using comp-op dst-over)
• Fill in water-lines (drawn behind using comp-op dst-over)

[jeisenbe]

@pnorman - usually you are the maintainer most skeptical of increased code complexity. What are your thoughts about the relayering discussed in this issue?

It's necessary to render a proper outline around the ocean polygons (See issue #3695 and PR #3065 an dissue #3895) as well as to allow properly rendering areas of beach, shoals, reefs and tidal flats differently over water without resorting to transparency (#3894, #3877, #3883, #3864, #3840 etc) and could also nicely solve the rendering problems with admin borders over water vs land (see #3102), but it would make the rendering order less intuitive.

[imagico]

We had quickly discussed that in Heidelberg and there was agreement that a working demo code based on the current code demonstrating the idea (in other words: a usable PR) would be good as a basis for deciding on this. This is on my todo list but i have not gotten around it yet. Anyone else is welcome to get ahead of me.

[pnorman]

@pnorman - usually you are the maintainer most skeptical of increased code complexity. What are your thoughts about the relayering discussed in this issue?

Adjusting layer order shouldn't impact complexity. comp-op operations might, but I don't have a good intuitive grasp on how much complexity they would add to the MSS.

[jeisenbe]

comp-op operations might, but I don't have a good intuitive grasp on how much complexity they would add

The main difference will be that instead of having to learn that layers are rendered in reverse order to their appearance in project.mml (the layer at the top is rendered at the bottom, and the layer at the bottom of project.mml is rendered on top, but with lowest priority), and that the later layers are rendered with lower priority than the earlier layers, we would need to add comments that explained how the first few layers were drawn in the reverse order.

This would require some more learning for new contributors, but adding the comments might actually make it easier to understand the current rendering for contributors who haven't worked with Mapnik/CartoCSS before.

This is on my todo list but i have not gotten around it yet

@imagico - if you can write the basic working code, I could test it and make the images and the PR, if that would save time.

[jeisenbe]

@imagico any chance of seeing a PR this month? I could give it a try, but I think I have too many open PRs already

Initially I hoped to implement #3930 and decide on the ocean color, but it now looks like there is a better chance to implement this first.

A number of other issues will be possible to solve if we can separately adjust the rendering over land and water, including: #330, #621, #3840, #3851, #3864, #3877, #3926, #3894, and #3895

[imagico]

Unlikely i find the time this month - but not impossible. I have not forgotten this is one the list. However you are welcome to do this if you like to.

[jeisenbe]

I believe this issue is being held up by waiting for consensus on PR #4128 since it would affect implementation. We need to decide first if we are going to show a difference between rivers, lakes and seas, or continue rendering them all the same.

[dch0ph]

In the context of reviewing layering, this would seem a good issue to revisit. It is linked to #4128, but it would seem relatively straightforward to make further adjustments, i.e. #4128 is not technically blocking.

Unfortunately, I don't really understand what new things it would allow us to do. (This is separate from the topic of how it works, with the various comp-op's, which I don't clearly understand anyway).

@imagico In the test rendering above can you clarify what is new / improved? Which issues would it unblock?

[imagico]

The image above is just a technical demonstration. A practically more meaningful example is shown in #4513 (comment).

With the current approach we essentially have the following options

1. rendering something only over land and not over water - that is what happens with things rendered in the landcover layer.
2. render something identically over land and water
3. use transparency, i.e. render something in a constant, pre-defined mix of the color before (land, water or whatever landcover is overlapping) and the selected fill color
4. use one of the available non-linear color mixing comp-ops (lighten, darken etc.), potentially in combination with transparency

Combining this with the use of patterns offers more choices - but is still severely constrained design wise. In particular since any use of transparency leads to a massive increase in the number of mixed colors, which impairs intuitive readability.

Now with what has been suggested here you are completely free to separately design the appearance over water and land - at the cost of rendering things more than once and of additional code complexity. This allows in particular:

• rendering something only over land and not over water
• rendering something only over water and not over land
• rendering something differently over land and water with explicit control over both and without the limitations of using transparency.

This offers much more design flexibility and thereby would allow better differentiating features over water - like wind parks or tidal zone features. See #3707, #3840, #3864, #4513.

But all of this of course comes at the prince of a significantly more complex layer stack - owed to the limitations of Mapnik in compositioning.

[dch0ph]

Many thanks for clarifying the issues involved.

So, if I've understood correctly, we can and do achieve may of the renderings here using a landcover overlay e.g. so that natural-beach has a common "sand" pattern above and below the high water mark, whereas

From what I can gather, resolving this issue will be of most relevance to features in the tidal / coastal zone? For non-tidal water areas, mappers will typically create multipolygons e.g. for islands/other landcover within water areas.

Would a useful way forward be to focus on the ocean water layer? I tried to work out how this was implemented in the AC-style, but it did seem to be quite complex, perhaps due to the need to handle intermittent/non-solid land-based water areas?

This should reduce the layers to

• landcover [as currently]
• water-lines [as currently]
• water-areas [as currently]
• ocean (cut out with comp-op dst-out)
• landcover overlay for ocean areas (drawn behind using comp-op dst-over) [new "tidal specific" areas]
• backdrop to fill in oceans (using comp-op dst-over) [using cached SQL query result]
• landcover overlay [as currently]

This should reproduce the current rendering. The next question would be a good target for a differentiated rendering?