IP Check Policy doesn't work on OpenShift #1061

duomotomo · 2019-06-05T15:50:30Z

We have following environment:

Internet -> haproxy -> 3scale(on openshift) -> echo app (on the same openshift)
echo app is from here(https://github.com/3scale/echo-api/tree/master/openshift), we use it for testing.
IP 1.1.1.1 below is machine from internet, but ip was changed for privacy reason.
I have configured test api service, corresponding to echo app, it's working, i.e. i see response from echo app.
I have also configured simple ip check policy and placed it before apicast policy.

	"policy_chain": [{
			"name": "ip_check",
			"version": "builtin",
			"configuration": {
				"error_msg": "IP address not allowed",
				"client_ip_sources": ["X-Forwarded-For"],
				"ips": ["1.1.1.1"],
				"check_type": "blacklist"
			}
		}, {
			"name": "apicast",
			"version": "builtin",
			"configuration": {}
		}

But this doesn't work (i.e. it doesn't block 1.1.1.1). One could expect that x-forwarded-for is stripped by haproxy or openshift, but at the same time in response from echo app(this app returns all headers back) it gave back all headers and it returns x-forwarded-for header also:

{
"method": "GET",
"path": "/echo",
"args": "user_key=!stripped!",
"body": "",
"headers": {
"HTTP_VERSION": "HTTP/1.1",
"HTTP_X_REAL_IP": "10.129.0.1",
"HTTP_X_3SCALE_PROXY_SECRET_TOKEN": "!striped!",
"HTTP_CACHE_CONTROL": "max-age=0",
"HTTP_UPGRADE_INSECURE_REQUESTS": "1",
"HTTP_USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
"HTTP_ACCEPT": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3",
"HTTP_ACCEPT_ENCODING": "gzip, deflate, br",
"HTTP_ACCEPT_LANGUAGE": "en,ru;q=0.9,cs;q=0.8,en-US;q=0.7",
"HTTP_X_FORWARDED_FOR": "1.1.1.1, 172.20.1.54, 172.20.4.232",
"HTTP_FORWARDED": "for=172.20.1.54;host=apicast-br.;proto=https;proto-version=, for=172.20.4.232;host=echo-api-test.;proto=http;proto-version=",
"HTTP_UBER_TRACE_ID": "71e3131832617a9e:6caeea202a2f70bd:71e3131832617a9e:1",
"HTTP_HOST": "echo-api-test.!stripped domain!",
"HTTP_X_FORWARDED_HOST": "echo-api-test.!stripped domain!",
"HTTP_X_FORWARDED_PORT": "80",
"HTTP_X_FORWARDED_PROTO": "http"
},
"uuid": "d7c9c1ad-efdc-48c8-bb87-087a96df632b"
}

So, x-forwarded-for header is definitively there, app on openshift see it, so it's not stripped, so 3scale should see it also, but it doesn't react to this header.

Mapping rules and service definition works correctly, because if i define IP Check policy "block all, but allow only defined" it blocks access to api, so it just can't read header for some reason.
How to debug this situation, debug mode in apicast wasn't useful?

Version

Latest 2.5
nginx version: openresty/1.13.6.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)
built with OpenSSL 1.1.0h 27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl/include' --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.13 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.5 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib' --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-dtrace-probes --with-stream --with-stream_ssl_module --with-http_ssl_module

[root@qeobo-vs dc]# oc describe is amp-apicast
Name: amp-apicast
Namespace: 3scale-api
Created: 3 months ago
Labels: 3scale.component=apicast
app=3scale-api-management
Annotations: openshift.io/display-name=AMP APIcast
openshift.io/generated-by=OpenShiftNewApp
openshift.io/image.dockerRepositoryCheck=2019-06-05T15:59:23Z
Docker Pull Spec: docker-registry.default.svc:5000/3scale-api/amp-apicast
Image Lookup: local=false
Unique Images: 3
Tags: 3

2.5.0 (latest)
tagged from registry.access.redhat.com/3scale-amp25/apicast-gateway

registry.access.redhat.com/3scale-amp25/apicast-gateway@sha256:a2deb838444a7d36b90b0c9ea50df225da46ad41f4986824d69babfda743f3fc
26 minutes ago
registry.access.redhat.com/3scale-amp25/apicast-gateway@sha256:92ae1bd025fa2a62beb09bb6a09ec2755bc39e938c75530e5c571eefca93101e
4 weeks ago

2.4.0
tagged from registry.access.redhat.com/3scale-amp24/apicast-gateway

registry.access.redhat.com/3scale-amp24/apicast-gateway@sha256:4f6a5ce86a55645447963e6f524d254ec9c997fae184cdddd4816b7606a039ed
3 months ago

Steps To Reproduce

install haproxy, define simple ip check policy

Current Result

Expected Result

ip check policy should filter based in ip check settings.

Additional Information

The text was updated successfully, but these errors were encountered:

eloycoto · 2019-06-07T10:06:33Z

Hi,

I've tested this and this is working correctly:

The configuration that I used is the following:

{
  "services": [
    {
      "id": 200,
      "endpoint": "http://localhost:8080",
      "backend_version": "1",
      "proxy": {
        "service_backend_version": "1",
        "hosts": [
                  "one"
              ],
        "api_backend": "https://echo-api.3scale.net:443",
        "proxy_rules": [
          { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 1 }
        ],
        "policy_chain": [
          { "name": "apicast.policy.apicast" },
          {
            "name": "ip_check",
            "version": "builtin",
            "configuration": {
              "error_msg": "IP address not allowed",
              "client_ip_sources": [
                "X-Forwarded-For"
              ],
              "ips": [
                "1.2.3.4"
              ],
              "check_type": "blacklist"
            }
          }
        ]
      }
    }
  ]
}

This is how I run apicast:

THREESCALE_CONFIG_FILE="service.json" BACKEND_ENDPOINT_OVERRIDE="https://echo-api.3scale.net:443" ./bin/apicast

Tested with curl:

bash-4.2$ curl localhost:8080?user_key=123 -H "Host:one" -H "X-Forwarded-For: 1.2.3.44" -v
* About to connect() to localhost port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /?user_key=123 HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host:one
> X-Forwarded-For: 1.2.3.44
>
< HTTP/1.1 200 OK
< Server: openresty
< Date: Fri, 07 Jun 2019 10:03:08 GMT
< Content-Type: application/json
< Content-Length: 590
< Connection: keep-alive
< Cache-control: private
< Set-Cookie: d8c1dd0e39ac4456ed39ce5889b9a5a5=497798f4606acdfb2787fcfc67ab0b88; path=/; HttpOnly
< Vary: Origin
< X-Content-Type-Options: nosniff
<
{
  "method": "GET",
  "path": "/",
  "args": "user_key=123",
  "body": "",
  "headers": {
    "HTTP_VERSION": "HTTP/1.1",
    "HTTP_HOST": "echo-api.3scale.net",
    "HTTP_ACCEPT": "*/*",
    "HTTP_USER_AGENT": "curl/7.29.0",
    "HTTP_X_REAL_IP": "127.0.0.1",
    "HTTP_X_FORWARDED_FOR": "1.2.3.44, 2.139.235.79, 10.0.103.119",
    "HTTP_X_FORWARDED_HOST": "echo-api.3scale.net",
    "HTTP_X_FORWARDED_PORT": "443",
    "HTTP_X_FORWARDED_PROTO": "https",
    "HTTP_FORWARDED": "for=10.0.103.119;host=echo-api.3scale.net;proto=https"
  },
  "uuid": "f8450232-ee46-4034-856b-c4c3de82959f"
* Connection #0 to host localhost left intact
}bash-4.2$
bash-4.2$
bash-4.2$
bash-4.2$
bash-4.2$ curl localhost:8080?user_key=123 -H "Host:one" -H "X-Forwarded-For: 1.2.3.4" -v
* About to connect() to localhost port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /?user_key=123 HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host:one
> X-Forwarded-For: 1.2.3.4
>
< HTTP/1.1 403 Forbidden
< Server: openresty
< Date: Fri, 07 Jun 2019 10:03:14 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: keep-alive
<
IP address not allowed
* Connection #0 to host localhost left intact
bash-4.2$

Also, tried with multiple uppercase values and all works:

}bash-4.2$ curl localhost:8080?user_key=123 -H "Host:one" -H "X-ForWArded-For: 1.2.3.44"
{
  "method": "GET",
  "path": "/",
  "args": "user_key=123",
  "body": "",
  "headers": {
    "HTTP_VERSION": "HTTP/1.1",
    "HTTP_HOST": "echo-api.3scale.net",
    "HTTP_ACCEPT": "*/*",
    "HTTP_USER_AGENT": "curl/7.29.0",
    "HTTP_X_REAL_IP": "127.0.0.1",
    "HTTP_X_FORWARDED_FOR": "1.2.3.44, 2.139.235.79, 10.0.103.119",
    "HTTP_X_FORWARDED_HOST": "echo-api.3scale.net",
    "HTTP_X_FORWARDED_PORT": "443",
    "HTTP_X_FORWARDED_PROTO": "https",
    "HTTP_FORWARDED": "for=10.0.103.119;host=echo-api.3scale.net;proto=https"
  },
  "uuid": "60f4657b-bd55-4307-aa18-5dd44b66cf98"
}bash-4.2$ curl localhost:8080?user_key=123 -H "Host:one" -H "X-ForWArded-For: 1.2.3.4"
IP address not allowed

So, if you can provide the full config, the request trace, and more info I'm
happy to debug a bit more.

Regards.
Eloy

duomotomo · 2019-06-07T10:40:31Z

Hello Eloy,
it seems you are not using openshift/docker version when testing.
How to debug what headers 3scale see on input?

eloycoto · 2019-06-07T12:46:16Z

Hi,

There is no way to dump incoming request on Apicast, if you validate that curl works, I would recommend to get a packet trace on the host and see what HA-proxy is sending to APIcast.

Regards

duomotomo · 2019-06-08T18:00:18Z

Hi Eloycoto,
well, i have experimented with curl and what i see:

inside apicast docker container request is working properly(i.e rejected).
when request goes through openshift router request is not working, i.e. request is not rejected.
So, openshift router changes headers, but i can't say how for for the time being.
Could you make the same tests on openshift to confirm my findings?

duomotomo · 2019-06-08T18:18:01Z

Hi,
ok, thanks to some tcpdumps i think i have some info to think about.
How request that didn't work looks for 3scale:

GET /echo?user_key=!stripped! HTTP/1.1
User-Agent: curl/7.29.0
Accept: */*
X-forwarded-for:1.1.1.1
Host: apicast-!stripped!
X-Forwarded-Host: apicast-!stripped!
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Forwarded: for=172.20.4.249;host=apicast-!strippeddomain!;proto=http;proto-version=
X-Forwarded-For: 172.20.4.249

As you could see there are 2 x-forwarded-for headers, not just one with multi-value.
And I think that 3scale just takes last x-forwarded-for into consideration, that's why request didn't work.
So, it looks like IP Check policy broken for all OpenShift installations.

duomotomo · 2019-06-08T18:19:37Z

I have changed case name a bit to reflect that problem valid for all openshift installations.

eloycoto · 2019-06-10T08:06:57Z

Ok,

I can reproduce the issue now, I'll send the patch today.

Regards

This change allows to block traffic when multiple x-forwarder-for are defined in the same request. Resty creates an array with all duplicated headers, so in case of a array only the first heaeder will be used. Also, adds the integration test to make sure that no regression in this case. Example request to make it fail: ``` curl localhost:8080?user_key=123 -H "X-Forwarded-For: 1.2.3.4" \ -H "Host:one" \ -H "X-Forwarded-For: 1.2.3.44" \ -v ``` config: ``` { "services": [ { "id": 200, "endpoint": "http://localhost:8080", "backend_version": "1", "proxy": { "service_backend_version": "1", "hosts": [ "one" ], "api_backend": "https://echo-api.3scale.net:443", "proxy_rules": [ { "pattern": "/", "http_method": "GET", "metric_system_name": "hits", "delta": 1 } ], "policy_chain": [ { "name": "apicast.policy.apicast" }, { "name": "ip_check", "version": "builtin", "configuration": { "error_msg": "IP address not allowed", "client_ip_sources": [ "X-Forwarded-For" ], "ips": [ "1.2.3.4" ], "check_type": "blacklist" } } ] } } ] } ``` Fixes 3scale#1061 Signed-off-by: Eloy Coto <[email protected]>

davidor assigned eloycoto Jun 6, 2019

duomotomo changed the title ~~IP Check Policy not working with haproxy at front~~ IP Check Policy not working on Openshift Jun 8, 2019

duomotomo changed the title ~~IP Check Policy not working on Openshift~~ IP Check Policy not working on OpenShift Jun 8, 2019

duomotomo changed the title ~~IP Check Policy not working on OpenShift~~ IP Check Policy doesn't work on OpenShift Jun 8, 2019

eloycoto mentioned this issue Jun 10, 2019

[THREESCALE-2775] [Policy] IP Check uses multiple x-forwarder-for headers. #1065

Merged

davidor closed this as completed in #1065 Jun 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IP Check Policy doesn't work on OpenShift #1061

IP Check Policy doesn't work on OpenShift #1061

duomotomo commented Jun 5, 2019 •

edited

Loading

eloycoto commented Jun 7, 2019

duomotomo commented Jun 7, 2019

eloycoto commented Jun 7, 2019 •

edited

Loading

duomotomo commented Jun 8, 2019 •

edited

Loading

duomotomo commented Jun 8, 2019 •

edited

Loading

duomotomo commented Jun 8, 2019

eloycoto commented Jun 10, 2019

IP Check Policy doesn't work on OpenShift #1061

IP Check Policy doesn't work on OpenShift #1061

Comments

duomotomo commented Jun 5, 2019 • edited Loading

Version

Steps To Reproduce

Current Result

Expected Result

Additional Information

eloycoto commented Jun 7, 2019

duomotomo commented Jun 7, 2019

eloycoto commented Jun 7, 2019 • edited Loading

duomotomo commented Jun 8, 2019 • edited Loading

duomotomo commented Jun 8, 2019 • edited Loading

duomotomo commented Jun 8, 2019

eloycoto commented Jun 10, 2019

duomotomo commented Jun 5, 2019 •

edited

Loading

eloycoto commented Jun 7, 2019 •

edited

Loading

duomotomo commented Jun 8, 2019 •

edited

Loading

duomotomo commented Jun 8, 2019 •

edited

Loading