OpenStreetMap does not display on Androids 5 and higher

[kkrawczyk123]

Software and hardware versions

Collect v1.23.x, Android v5.1, 6.0, 7.0, and 8.1, device used...

Problem description

OpenStreetMap does not display when using Geo widgets.

Steps to reproduce the problem

1. Choose in General Settings Maps/OpenStreetMap
2. Open Geopoint/GeoTrace/GeoShape widget
3. Click on Start button

Expected behavior

OpenStreetMap should load.

Other information

Other types of map display correctly on Android 5.1 and higher
Interesting is that it displays correctly on Android 4.4 and lower.

[zestyping]

When I try this, I get lots of "403 Forbidden" responses for the tile requests. Here an example of the HTTP headers I get:

Server: squid/2.7.STABLE9
Date: Fri, 26 Jul 2019 20:11:05 GMT
Content-Type: text/html
Content-Length: 1229
X-Squid-Error: ERR_ACCESS_DENIED 0
X-Cache: MISS from stormfly-02.openstreetmap.org
X-Cache-Lookup: NONE from stormfly-02.openstreetmap.org:3128
Via: 1.0 stormfly-02.openstreetmap.org:3128 (squid/2.7.STABLE9)
Connection: close
X-Android-Sent-Millis: 1564171867232
X-Android-Received-Millis: 1564171867318
X-Android-Selected-Protocol: http/1.1
X-Android-Response-Source: NETWORK 403

@kkrawczyk123 Can you check logcat while you are experiencing this problem and verify whether it's the same for you? If it's failing for the same reason for you, the messages will look like this:

07-26 20:10:18.525 1182-1299/org.odk.collect.android W/OsmDroid: Problem downloading MapTile: /10/510/511 HTTP response: Forbidden

[zestyping]

Hmm. This looks deliberate. OsmDroid includes the header User-Agent: osmdroid in its requests, and that seems to make the difference.

When I do

curl -v -O http://c.tile.openstreetmap.org/16/105025311.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.211.167.105...
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Connected to c.tile.openstreetmap.org (140.211.167.105) port 80 (#0)
> GET /16/10508/25311.png HTTP/1.1
> Host: c.tile.openstreetmap.org
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 26 Jul 2019 20:12:14 GMT
< Server: Apache/2.4.29 (Ubuntu)
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Expect-CT: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
< ETag: "a486de698cacbeedf180b24b0d93269f"
< Content-Length: 11779
< Cache-Control: max-age=20345
< Expires: Sat, 27 Jul 2019 01:51:19 GMT
< Access-Control-Allow-Origin: *
< X-TileRender: bowser.openstreetmap.org
< Content-Type: image/png
< X-Cache: MISS from stormfly-02.openstreetmap.org
< X-Cache-Lookup: HIT from stormfly-02.openstreetmap.org:3128
< Via: 1.1 stormfly-02.openstreetmap.org:3128 (squid/2.7.STABLE9)
<
{ [5792 bytes data]

I get a tile.

But when I do:

curl -v -H 'User-Agent: osmdroid' -O http://c.tile.openstreetmap.org/16/10508/25311.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 140.211.167.105...
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to c.tile.openstreetmap.org (140.211.167.105) port 80 (#0)
> GET /16/10508/25311.png HTTP/1.1
> Host: c.tile.openstreetmap.org
> Accept: */*
> User-Agent: osmdroid
>
< HTTP/1.1 403 Forbidden
< Server: squid/2.7.STABLE9
< Date: Fri, 26 Jul 2019 20:19:02 GMT
< Content-Type: text/html
< Content-Length: 1229
< X-Squid-Error: ERR_ACCESS_DENIED 0
< X-Cache: MISS from stormfly-02.openstreetmap.org
< X-Cache-Lookup: NONE from stormfly-02.openstreetmap.org:3128
< Via: 1.0 stormfly-02.openstreetmap.org:3128 (squid/2.7.STABLE9)
< Connection: close
<
{ [1229 bytes data]

I get a 403 Forbidden error.

[lognaturel]

@zestyping and I discussed more in Slack and have concluded that the osmdroid User-Agent is blocked because it violates the OpenStreetMap terms of service. Collect itself violates the terms of service and we have reached out to OpenStreetMap about how to proceed.

#3254 sets a User-Agent string for osmdroid requests because we discovered we were supposed to do that. Incidentally, it re-establishes access to the OpenStreetMap tile server. We should not merge it until we get the ok from OSM.

[lognaturel]

OSM has given us the green light to provide a custom User-Agent string.