Feature Request: Make it possible to modify query response codes with rego

[simongottschlag]

Hi again! 😁

I wrote a question in Slack but after taking a look at the code it seems like what I'm looking for isn't supported as of today.

What I'm looking for is to use OPA together with ingress-nginx in the way described in this article: https://medium.com/@ankit.wal/authenticate-requests-to-apps-on-kubernetes-using-nginx-ingress-and-an-authservice-37bf189670ee

The main point is that I would like to be able to respond with a non-200 status code even though the evaluation works. As a reference, the following issue was created around this (at least I think it's the same): #2821

When looking at the code:

https://github.com/open-policy-agent/opa/blob/main/server/server.go#L1944-L1955

I think we should be able to add something that parses the result here and grabs a value, maybe something like http_reaponse_code and if it's an int between 200-599 we'll use it instead of the normal 200.

Thoughts, questions and/or suggestions?

Thanks!

[simongottschlag]

And of course, I'd happily try to create a PR if we find a way that works for the project to do this. 😊

[ashutosh-narkar]

I think we should be able to add something that parses the result here and grabs a value, maybe something like http_reaponse_code and if it's an int between 200-599 we'll use it instead of the normal 200.

@simongottschlag are you suggesting that each query result have a field like http_reaponse_code which determines the response code ? The HTTP status code that the API returns does not have much to do with the result itself.

I'm not familiar with ingress-nginx but what if there was a service that embedded OPA as a library (for ex.) and then you could define a rule that includes the HTTP status in the policy result. You service would then take that result, manipulate it etc. and return the custom response that ingress-nginx expects.

[simongottschlag]

I'm not familiar with ingress-nginx but what if there was a service that embedded OPA as a library (for ex.) and then you could define a rule that includes the HTTP status in the policy result. You service would then take that result, manipulate it etc. and return the custom response that ingress-nginx expects.

Hi @ashutosh-narkar!

One way would absolutely be to either use an intermediate service between nginx and OPA or build a small service that leverages the lib directly.

My thought behind the FR is that it would be a nice feature to support directly in OPA to make the interoperability between it and ingress-nginx better.

The main reason behind not creating a PR directly is that I'm not sure the maintainers agree. 😊

[anderseknert]

It's a hack ( 😅 ) but you could use OPA's mechanism for authentication and authorization of OPA's own endpoints to get 401 responses... See https://www.openpolicyagent.org/docs/latest/security/#authentication-and-authorization

[tsandall]

What do the subrequests from nginx to the auth service look like? If you configure nginx to enable external auth will it just forward arbitrary requests? For example, what what does the subrequest look like for:

PUT /some/path HTTP/1.1
Host: foo.com
Content-Type: application/json

{
   "bar": "baz"
}

What would the auth service receive? From looking at the nginx docs, it sounds like they just forward the raw request and expect a standard HTTP response to indicate allow/deny (200, 401, 403, etc.) I can see the benefit of that over defining a new format like Envoy does with external authz.

If nginx does forward the raw request to the auth service, then we'd need a new server to terminate those requests (because otherwise the existing routes on the OPA server would conflict...)

[simongottschlag]

Hi!

I've created a small example: https://github.com/simongottschlag/opa-nginx-authz

From the repo, I can start nginx and opa:

./run.sh
17:02:55 [RUN] Docker clean up started.
17:02:55 [RUN] Docker clean up finished.
17:02:55 [RUN] Creating docker network
17:02:55 [RUN] Docker network created
17:02:55 [RUN] Starting opa
17:02:55 [RUN] Started opa
17:02:55 [RUN] Starting nginx
17:02:56 [RUN] Started nginx
17:02:56 [RUN] Press enter to stop.

When I curl nginx, it works as expected:

curl http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

And if I look at the logs, I see the following:

docker logs opa-test
{"addrs":[":8181"],"diagnostic-addrs":[],"level":"info","msg":"Initializing server.","time":"2021-06-10T15:02:55Z"}
{"client_addr":"192.168.208.3:40860","level":"info","msg":"Received request.","req_id":1,"req_method":"GET","req_path":"/v1/data/nginx/authz","time":"2021-06-10T15:03:39Z"}
{"client_addr":"192.168.208.3:40860","level":"info","msg":"Sent response.","req_id":1,"req_method":"GET","req_path":"/v1/data/nginx/authz","resp_bytes":15,"resp_duration":0.474706,"resp_status":200,"time":"2021-06-10T15:03:39Z"}
{"client_addr":"192.168.208.3:40862","level":"info","msg":"Received request.","req_id":2,"req_method":"GET","req_path":"/v1/data/nginx/authz","time":"2021-06-10T15:03:39Z"}
{"client_addr":"192.168.208.3:40862","level":"info","msg":"Sent response.","req_id":2,"req_method":"GET","req_path":"/v1/data/nginx/authz","resp_bytes":15,"resp_duration":0.149859,"resp_status":200,"time":"2021-06-10T15:03:39Z"}

If I curl another path, nginx goes to the same place:

curl http://localhost:8080/test/abc
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.20.1</center>
</body>
</html>

And the logs from the above request:

{"client_addr":"192.168.208.3:43062","level":"info","msg":"Received request.","req_id":3,"req_method":"GET","req_path":"/v1/data/nginx/authz","time":"2021-06-10T15:05:42Z"}
{"client_addr":"192.168.208.3:43062","level":"info","msg":"Sent response.","req_id":3,"req_method":"GET","req_path":"/v1/data/nginx/authz","resp_bytes":15,"resp_duration":0.22837,"resp_status":200,"time":"2021-06-10T15:05:42Z"}

My understanding is that if we were able to manipulate the response code with rego, then we could respond with 403 if a JWT isn't valid or something like that and have something really easy to add extra security to nginx.

[ashutosh-narkar]

I'm surprised the req_path OPA receives in same in both the cases. Anyways you could add a service that calls out to OPA and passes the method, path, token etc to OPA. It then gets a response from OPA and based on the policy result sends a 200 or 403 back to nginx. This would be one way of doing this. Similar to this example.

[tsandall]

@simongottschlag that's a useful example! I could imagine it being useful to have a version of OPA that you could hookup to any of these HTTP-based authentication/authorization extension mechanisms (e.g., Nginx is not the only one, Envoy can be configured this way as well... I'm sure there are others.)

What I'm unsure of is what happens/what to do about non-GET requests... I'd want to at least know what the story is there...

Also, it's important to think about how the HTTP request gets mapped into the input document for policy evaluation. For example, how would the sample from above that I posted work?

PUT /some/path HTTP/1.1
Host: foo.com
Content-Type: application/json

{"foo": "bar"}

I'd want input to be bound to something like...

{
   "method": "PUT"
   "host": ["foo.com"]
   "content-type": ["application/json"]
   "body": <raw body payload>
   "parsed_body": {"foo", "bar"}
   "path": "/some/path"
   "parsed_path": ["some", "path"]
   "version": "HTTP/1.1"
}

The input would contain a mix of raw and preprocessed inputs for convenience.

Note, the existing server handlers in OPA can't be made to do this...it would not be backwards compatible. Similarly, those handlers can't start interpreting values in the result as indicating HTTP response fields...

If there's some way to make this work in OPA proper that would be nice, but we could also experiment with a plugin for OPA like the opa-envoy-plugin that extends OPA with a dedicated server for this kind of thing.

[simongottschlag]

@tsandall I've updated the example with a small echo-server.

When sending the following request to nginx:

curl --data '{"foo": "bar"}' --header "Content-Type: application/json" localhost:8080/echo/test

I can see nginx sending the following to the echo server:

&http.Request{Method:"GET", URL:(*url.URL)(0xc0000a4000), Proto:"HTTP/1.0", ProtoMajor:1, ProtoMinor:0, Header:http.Header{"Accept":[]string{"*/*"}, "Connection":[]string{"close"}, "Content-Type":[]string{"application/json"}, "User-Agent":[]string{"curl/7.76.1"}, "X-Original-Uri":[]string{"/echo/test"}}, Body:http.noBody{}, GetBody:(func() (io.ReadCloser, error))(nil), ContentLength:0, TransferEncoding:[]string(nil), Close:true, Host:"echo-test:8081", Form:url.Values(nil), PostForm:url.Values(nil), MultipartForm:(*multipart.Form)(nil), Trailer:http.Header(nil), RemoteAddr:"172.20.0.4:43118", RequestURI:"/echo", TLS:(*tls.ConnectionState)(nil), Cancel:(<-chan struct {})(nil), Response:(*http.Response)(nil), ctx:(*context.cancelCtx)(0xc00009c080)}

Not sure if I can modify the body in any way to the external auth server in this case, but will try to see if there's any way.

[simongottschlag]

@tsandall I haven't been able to get anything working as I wanted with pure nginx <-> opa, but I have created a small proof of concept service that does the job as a proxy between them. This could maybe run as a sidecar or something like that: https://github.com/simongottschlag/opa-nginx-authz/tree/opa-nginx-external-auth/opa-nginx-external-auth

Will continue to write the tests for it when I have some free time and make it maintainable. I do think this will increase the latency and complexity compared to actually handling this inside of OPA directly.

Any thoughts / questions? 😊

[ashutosh-narkar]

@simongottschlag have you considered embedding OPA as a library instead of a callout from your service ? This should help with the latency.

[anderseknert]

As the request body is discarded for authentication subrequests, you will need to set the proxy_pass_request_body directive to off and also set the Content-Length header to a null string:

From step 4 here.

Unless I'm really misinterpreting things the body of the request will never be passed to OPA. So what you're left to play with is essentially the URL and the request headers - the latter which is not passed into normal policy evaluation. Did you try the hack I suggested earlier? It seems like that could at least do what you want without adding another server in between, however hacky..

[simongottschlag]

From step 4 here.

Unless I'm really misinterpreting things the body of the request will never be passed to OPA. So what you're left to play with is essentially the URL and the request headers - the latter which is not passed into normal policy evaluation. Did you try the hack I suggested earlier? It seems like that could at least do what you want without adding another server in between, however hacky..

@anderseknert yeah, gave up on hacking nginx to support it.

@simongottschlag have you considered embedding OPA as a library instead of a callout from your service ? This should help with the latency.

@ashutosh-narkar I am considering it, but would of course prefer it to be supported by OPA out of the box and not have to maintain an additional service. I did some tests in regards to performance and you can see below that it is a quite large overhead using an additional http service. I will check how much better it becomes if I embed OPA directly instead.

k6 load test

Start all services: ./run.sh

OPA

Run load test: k6 run loadtest-opa.js

This will just send an requests to OPA end expect 200 OK and that the body contains "result": true.

running (25.0s), 00/30 VUs, 130158 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     data_received..................: 23 MB  922 kB/s
     data_sent......................: 22 MB  859 kB/s
     http_req_blocked...............: avg=1.77µs  min=689ns    med=1.41µs  max=1.67ms  p(90)=1.7µs   p(95)=2.2µs  
     http_req_connecting............: avg=44ns    min=0s       med=0s      max=1.61ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=3.21ms  min=247.54µs med=2ms     max=68.57ms p(90)=7.48ms  p(95)=10ms   
       { expected_response:true }...: avg=3.21ms  min=247.54µs med=2ms     max=68.57ms p(90)=7.48ms  p(95)=10ms   
     http_req_failed................: 0.00%  ✓ 0    ✗ 130158
     http_req_receiving.............: avg=24.58µs min=8.26µs   med=18.67µs max=19.53ms p(90)=27.23µs p(95)=31.74µs
     http_req_sending...............: avg=12.13µs min=4.53µs   med=8.44µs  max=15.62ms p(90)=11.63µs p(95)=14.79µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=3.18ms  min=219.23µs med=1.97ms  max=68.55ms p(90)=7.44ms  p(95)=9.94ms 
     http_reqs......................: 130158 5206.047259/s
     iteration_duration.............: avg=3.27ms  min=287.63µs med=2.06ms  max=68.64ms p(90)=7.54ms  p(95)=10.06ms
     iterations.....................: 130158 5206.047259/s
     vus............................: 1      min=1  max=29  
     vus_max........................: 30     min=30 max=30 

opa-nginx-external-auth

Run load test: k6 run loadtest-go.js

This will send requests to the "opa proxy" which in turn sends it to opa. Will expect 200 OK.

running (25.0s), 00/30 VUs, 77709 complete and 0 interrupted iterations
default   [======================================] 00/30 VUs  25s

     data_received..................: 5.8 MB 233 kB/s
     data_sent......................: 8.4 MB 336 kB/s
     http_req_blocked...............: avg=1.88µs  min=701ns    med=1.51µs max=3.6ms   p(90)=1.86µs  p(95)=2.41µs 
     http_req_connecting............: avg=100ns   min=0s       med=0s     max=3.54ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=5.44ms  min=429.64µs med=3.82ms max=56.84ms p(90)=11.95ms p(95)=15.34ms
       { expected_response:true }...: avg=5.44ms  min=429.64µs med=3.82ms max=56.84ms p(90)=11.95ms p(95)=15.34ms
     http_req_failed................: 0.00%  ✓ 0    ✗ 77709
     http_req_receiving.............: avg=19.81µs min=6.72µs   med=17.3µs max=7.51ms  p(90)=23.39µs p(95)=27.31µs
     http_req_sending...............: avg=10.78µs min=4.48µs   med=7.88µs max=15.91ms p(90)=11.36µs p(95)=13.55µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s     max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=5.41ms  min=404.28µs med=3.79ms max=56.82ms p(90)=11.91ms p(95)=15.3ms 
     http_reqs......................: 77709  3108.189419/s
     iteration_duration.............: avg=5.49ms  min=477.9µs  med=3.87ms max=56.88ms p(90)=12ms    p(95)=15.39ms
     iterations.....................: 77709  3108.189419/s
     vus............................: 3      min=3  max=29 
     vus_max........................: 30     min=30 max=30 

nginx public

Run load test: k6 run loadtest-nginx-public.js

This will send requests directly to nginx without any authentication enabled, expects 200 OK.

running (25.0s), 00/30 VUs, 420676 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0      ✗ 420676
     checks.........................: 100.00% ✓ 420676 ✗ 0     
     data_received..................: 358 MB  14 MB/s
     data_sent......................: 36 MB   1.4 MB/s
     http_req_blocked...............: avg=2.85µs   min=608ns    med=1.15µs   max=20.87ms p(90)=1.83µs p(95)=2.4µs  
     http_req_connecting............: avg=212ns    min=0s       med=0s       max=4.67ms  p(90)=0s     p(95)=0s     
     http_req_duration..............: avg=909.33µs min=84.43µs  med=528.31µs max=35.78ms p(90)=1.99ms p(95)=2.9ms  
       { expected_response:true }...: avg=909.33µs min=84.43µs  med=528.31µs max=35.78ms p(90)=1.99ms p(95)=2.9ms  
     http_req_failed................: 0.00%   ✓ 0      ✗ 420676
     http_req_receiving.............: avg=34.62µs  min=7.74µs   med=13.66µs  max=25.25ms p(90)=33.5µs p(95)=84.12µs
     http_req_sending...............: avg=12.26µs  min=3.47µs   med=5.39µs   max=18.91ms p(90)=9.99µs p(95)=14.15µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s     p(95)=0s     
     http_req_waiting...............: avg=862.43µs min=66.43µs  med=498.74µs max=34.28ms p(90)=1.91ms p(95)=2.79ms 
     http_reqs......................: 420676  16826.312401/s
     iteration_duration.............: avg=1ms      min=130.57µs med=602.43µs max=36.01ms p(90)=2.14ms p(95)=3.12ms 
     iterations.....................: 420676  16826.312401/s
     vus............................: 0       min=0    max=29  
     vus_max........................: 30      min=30   max=30  

nginx private

Run load test: k6 run loadtest-nginx-private.js

This will send request to nginx with authentication enabled, which sends a request through the "opa proxy" to opa, expects 200 OK.

running (25.0s), 00/30 VUs, 46070 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0     ✗ 46070
     checks.........................: 100.00% ✓ 46070 ✗ 0    
     data_received..................: 39 MB   1.6 MB/s
     data_sent......................: 5.3 MB  212 kB/s
     http_req_blocked...............: avg=2.61µs  min=944ns    med=1.75µs  max=2.83ms   p(90)=2.65µs  p(95)=3.33µs 
     http_req_connecting............: avg=208ns   min=0s       med=0s      max=840.77µs p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=9.17ms  min=727.97µs med=6.84ms  max=95.53ms  p(90)=18.8ms  p(95)=24.09ms
       { expected_response:true }...: avg=9.17ms  min=727.97µs med=6.84ms  max=95.53ms  p(90)=18.8ms  p(95)=24.09ms
     http_req_failed................: 0.00%   ✓ 0     ✗ 46070
     http_req_receiving.............: avg=38.84µs min=13.19µs  med=28.89µs max=12.18ms  p(90)=45.97µs p(95)=56.51µs
     http_req_sending...............: avg=13.2µs  min=5.51µs   med=9µs     max=8.79ms   p(90)=14.61µs p(95)=17.46µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s       p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=9.12ms  min=688.63µs med=6.79ms  max=95.49ms  p(90)=18.73ms p(95)=24.01ms
     http_reqs......................: 46070   1842.692902/s
     iteration_duration.............: avg=9.27ms  min=803.01µs med=6.94ms  max=95.63ms  p(90)=18.91ms p(95)=24.22ms
     iterations.....................: 46070   1842.692902/s
     vus............................: 1       min=1   max=30 
     vus_max........................: 30      min=30  max=30 

[ashutosh-narkar]

Since you're making a http call to OPA from the proxy on every request, the added request duration is expected. I would expect embedding OPA in your proxy to give better results.

[simongottschlag]

@ashutosh-narkar the results are.. weird? I've added the ability to evaluate rego in the service directly and going directly to it is faster than going through the proxy mode. But seems to be slower when going through nginx auth...

And to be clear: I've removed console logging from the service I've built and also removed info / decision logs from OPA which makes the tests a bit faster as well.

I've created a PR here for the changes: simongottschlag/opa-nginx-authz#2

See below:

k6 load test

Start all services: ./run.sh

Restart the above before each test!

OPA

Run load test: k6 run loadtest-opa.js

running (25.0s), 00/30 VUs, 223138 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200
     ✓ result is true

     check_failure_rate.............: 0.00%   ✓ 0      ✗ 223138
     checks.........................: 100.00% ✓ 446276 ✗ 0     
     data_received..................: 27 MB   1.1 MB/s
     data_sent......................: 39 MB   1.6 MB/s
     http_req_blocked...............: avg=2.39µs  min=686ns    med=1.34µs max=8.8ms   p(90)=1.95µs  p(95)=2.47µs 
     http_req_connecting............: avg=73ns    min=0s       med=0s     max=1.99ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=1.77ms  min=126.15µs med=1.06ms max=39.54ms p(90)=4.14ms  p(95)=5.78ms 
       { expected_response:true }...: avg=1.77ms  min=126.15µs med=1.06ms max=39.54ms p(90)=4.14ms  p(95)=5.78ms 
     http_req_failed................: 0.00%   ✓ 0      ✗ 223138
     http_req_receiving.............: avg=29.73µs min=7.66µs   med=14.1µs max=19.58ms p(90)=24.68µs p(95)=38.19µs
     http_req_sending...............: avg=14.66µs min=4.75µs   med=7.69µs max=27.33ms p(90)=12.29µs p(95)=16.67µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s     max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=1.73ms  min=109.21µs med=1.03ms max=39.52ms p(90)=4.07ms  p(95)=5.69ms 
     http_reqs......................: 223138  8925.133978/s
     iteration_duration.............: avg=1.9ms   min=188.17µs med=1.18ms max=39.63ms p(90)=4.34ms  p(95)=6.03ms 
     iterations.....................: 223138  8925.133978/s
     vus............................: 1       min=1    max=30  
     vus_max........................: 30      min=30   max=30 

opa-nginx-external-auth proxy

Run load test: k6 run loadtest-go-proxy.js

running (25.0s), 00/30 VUs, 156276 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0      ✗ 156276
     checks.........................: 100.00% ✓ 156276 ✗ 0     
     data_received..................: 12 MB   469 kB/s
     data_sent......................: 18 MB   706 kB/s
     http_req_blocked...............: avg=1.92µs  min=645ns    med=1.38µs  max=6.44ms   p(90)=1.74µs  p(95)=2.25µs 
     http_req_connecting............: avg=59ns    min=0s       med=0s      max=900.18µs p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=2.65ms  min=214.85µs med=1.9ms   max=31.37ms  p(90)=5.62ms  p(95)=7.56ms 
       { expected_response:true }...: avg=2.65ms  min=214.85µs med=1.9ms   max=31.37ms  p(90)=5.62ms  p(95)=7.56ms 
     http_req_failed................: 0.00%   ✓ 0      ✗ 156276
     http_req_receiving.............: avg=18.86µs min=6µs      med=13.46µs max=12.78ms  p(90)=19.91µs p(95)=23.22µs
     http_req_sending...............: avg=10.66µs min=3.91µs   med=6.98µs  max=21.41ms  p(90)=9.97µs  p(95)=12.81µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s       p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=2.62ms  min=200.82µs med=1.88ms  max=29.53ms  p(90)=5.59ms  p(95)=7.51ms 
     http_reqs......................: 156276  6250.694014/s
     iteration_duration.............: avg=2.73ms  min=261.51µs med=1.97ms  max=31.47ms  p(90)=5.73ms  p(95)=7.67ms 
     iterations.....................: 156276  6250.694014/s
     vus............................: 1       min=1    max=30  
     vus_max........................: 30      min=30   max=30 

opa-nginx-external-auth rego

Run load test: k6 run loadtest-go-rego.js

running (25.0s), 00/30 VUs, 218135 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0      ✗ 218135
     checks.........................: 100.00% ✓ 218135 ✗ 0     
     data_received..................: 16 MB   654 kB/s
     data_sent......................: 24 MB   977 kB/s
     http_req_blocked...............: avg=2.57µs  min=617ns    med=1.26µs  max=24.04ms p(90)=1.81µs  p(95)=2.43µs 
     http_req_connecting............: avg=58ns    min=0s       med=0s      max=3.05ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=1.85ms  min=167.68µs med=1.11ms  max=35.96ms p(90)=4.21ms  p(95)=6.01ms 
       { expected_response:true }...: avg=1.85ms  min=167.68µs med=1.11ms  max=35.96ms p(90)=4.21ms  p(95)=6.01ms 
     http_req_failed................: 0.00%   ✓ 0      ✗ 218135
     http_req_receiving.............: avg=21.19µs min=5.67µs   med=10.83µs max=20.58ms p(90)=18.01µs p(95)=23.66µs
     http_req_sending...............: avg=12.72µs min=3.33µs   med=6.33µs  max=20.54ms p(90)=9.91µs  p(95)=14.3µs 
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=1.82ms  min=148.76µs med=1.08ms  max=35.68ms p(90)=4.16ms  p(95)=5.94ms 
     http_reqs......................: 218135  8725.0431/s
     iteration_duration.............: avg=1.95ms  min=223.15µs med=1.19ms  max=36.05ms p(90)=4.36ms  p(95)=6.2ms  
     iterations.....................: 218135  8725.0431/s
     vus............................: 1       min=1    max=30  
     vus_max........................: 30      min=30   max=30  

nginx public

Run load test: k6 run loadtest-nginx-public.js

running (25.0s), 00/30 VUs, 406248 complete and 0 interrupted iterations
default   [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0      ✗ 406248
     checks.........................: 100.00% ✓ 406248 ✗ 0     
     data_received..................: 345 MB  14 MB/s
     data_sent......................: 35 MB   1.4 MB/s
     http_req_blocked...............: avg=2.86µs   min=647ns    med=1.18µs   max=9.38ms  p(90)=1.86µs  p(95)=2.54µs 
     http_req_connecting............: avg=252ns    min=0s       med=0s       max=9.29ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=938.23µs min=91.19µs  med=541.67µs max=51.92ms p(90)=2.03ms  p(95)=2.97ms 
       { expected_response:true }...: avg=938.23µs min=91.19µs  med=541.67µs max=51.92ms p(90)=2.03ms  p(95)=2.97ms 
     http_req_failed................: 0.00%   ✓ 0      ✗ 406248
     http_req_receiving.............: avg=36.96µs  min=7.52µs   med=13.84µs  max=29.03ms p(90)=39.23µs p(95)=90.32µs
     http_req_sending...............: avg=12.7µs   min=3.4µs    med=5.49µs   max=32.56ms p(90)=10.92µs p(95)=15.14µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=888.55µs min=72.99µs  med=508.26µs max=51.85ms p(90)=1.95ms  p(95)=2.86ms 
     http_reqs......................: 406248  16249.182908/s
     iteration_duration.............: avg=1.04ms   min=135.41µs med=623.29µs max=52.54ms p(90)=2.2ms   p(95)=3.22ms 
     iterations.....................: 406248  16249.182908/s
     vus............................: 3       min=3    max=29  
     vus_max........................: 30      min=30   max=30  

nginx private proxy

Run load test: k6 run loadtest-nginx-private-proxy.js

running (25.0s), 00/30 VUs, 63390 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0     ✗ 63390
     checks.........................: 100.00% ✓ 63390 ✗ 0    
     data_received..................: 54 MB   2.2 MB/s
     data_sent......................: 7.7 MB  307 kB/s
     http_req_blocked...............: avg=2.38µs  min=884ns    med=1.6µs   max=3.39ms  p(90)=2.46µs  p(95)=2.92µs 
     http_req_connecting............: avg=298ns   min=0s       med=0s      max=2.14ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=6.65ms  min=496.78µs med=4.94ms  max=81.77ms p(90)=12.57ms p(95)=18.07ms
       { expected_response:true }...: avg=6.65ms  min=496.78µs med=4.94ms  max=81.77ms p(90)=12.57ms p(95)=18.07ms
     http_req_failed................: 0.00%   ✓ 0     ✗ 63390
     http_req_receiving.............: avg=32.26µs min=12.51µs  med=24.74µs max=11.04ms p(90)=42.06µs p(95)=55.61µs
     http_req_sending...............: avg=13.69µs min=4.99µs   med=8.19µs  max=31.41ms p(90)=13.69µs p(95)=16.06µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=6.6ms   min=474.84µs med=4.91ms  max=81.66ms p(90)=12.51ms p(95)=18ms   
     http_reqs......................: 63390   2535.166203/s
     iteration_duration.............: avg=6.74ms  min=548.51µs med=5.03ms  max=81.83ms p(90)=12.68ms p(95)=18.23ms
     iterations.....................: 63390   2535.166203/s
     vus............................: 1       min=1   max=29 
     vus_max........................: 30      min=30  max=30 

nginx private rego

Run load test: k6 run loadtest-nginx-private-rego.js

running (25.0s), 00/30 VUs, 42288 complete and 0 interrupted iterations
default   [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0     ✗ 42288
     checks.........................: 100.00% ✓ 42288 ✗ 0    
     data_received..................: 36 MB   1.4 MB/s
     data_sent......................: 5.1 MB  203 kB/s
     http_req_blocked...............: avg=2.62µs  min=933ns    med=1.69µs  max=6.06ms   p(90)=2.94µs  p(95)=3.42µs 
     http_req_connecting............: avg=363ns   min=0s       med=0s      max=6ms      p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=10.01ms min=428.89µs med=4.46ms  max=121.65ms p(90)=27.43ms p(95)=36.73ms
       { expected_response:true }...: avg=10.01ms min=428.89µs med=4.46ms  max=121.65ms p(90)=27.43ms p(95)=36.73ms
     http_req_failed................: 0.00%   ✓ 0     ✗ 42288
     http_req_receiving.............: avg=34.16µs min=10.92µs  med=25.95µs max=8.94ms   p(90)=50.72µs p(95)=59.79µs
     http_req_sending...............: avg=13.47µs min=4.24µs   med=8.62µs  max=15.84ms  p(90)=14.87µs p(95)=17.04µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s       p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=9.96ms  min=402.69µs med=4.41ms  max=121.55ms p(90)=27.35ms p(95)=36.68ms
     http_reqs......................: 42288   1691.377648/s
     iteration_duration.............: avg=10.1ms  min=485.79µs med=4.56ms  max=121.81ms p(90)=27.53ms p(95)=36.83ms
     iterations.....................: 42288   1691.377648/s
     vus............................: 1       min=1   max=29 
     vus_max........................: 30      min=30  max=30 

[simongottschlag]

I've done one change, went from PartialResult to PrepareForEval and got a bit better performance. With PrepareForEval, it actually seems it's performing better than using the vanilla OPA container.

I used the internal benchmarking tool to compare them:

PartialResult

Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^(BenchmarkGetResultWithOpaInput)$ github.com/simongottschlag/opa-nginx-authz/opa-nginx-external-auth

goos: linux
goarch: amd64
pkg: github.com/simongottschlag/opa-nginx-authz/opa-nginx-external-auth
cpu: Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz
BenchmarkGetResultWithOpaInput-4   	   14156	     84160 ns/op	   24800 B/op	     510 allocs/op
PASS
ok  	github.com/simongottschlag/opa-nginx-authz/opa-nginx-external-auth	2.059s

PrepareForEval

Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^(BenchmarkGetResultWithOpaInput)$ github.com/simongottschlag/opa-nginx-authz/opa-nginx-external-auth

goos: linux
goarch: amd64
pkg: github.com/simongottschlag/opa-nginx-authz/opa-nginx-external-auth
cpu: Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz
BenchmarkGetResultWithOpaInput-4   	   38095	     31035 ns/op	   12634 B/op	     247 allocs/op
PASS
ok  	github.com/simongottschlag/opa-nginx-authz/opa-nginx-external-auth	1.509s

The k6 tests also got a bit better:

opa-nginx-external-auth rego

Run load test: k6 run loadtest-go-rego.js

running (25.0s), 00/30 VUs, 306568 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0      ✗ 306568
     checks.........................: 100.00% ✓ 306568 ✗ 0     
     data_received..................: 23 MB   920 kB/s
     data_sent......................: 34 MB   1.4 MB/s
     http_req_blocked...............: avg=2.31µs  min=632ns    med=1.17µs   max=14.95ms p(90)=1.75µs p(95)=2.35µs 
     http_req_connecting............: avg=41ns    min=0s       med=0s       max=3.04ms  p(90)=0s     p(95)=0s     
     http_req_duration..............: avg=1.28ms  min=105.45µs med=783.69µs max=32.76ms p(90)=2.88ms p(95)=4.09ms 
       { expected_response:true }...: avg=1.28ms  min=105.45µs med=783.69µs max=32.76ms p(90)=2.88ms p(95)=4.09ms 
     http_req_failed................: 0.00%   ✓ 0      ✗ 306568
     http_req_receiving.............: avg=21.28µs min=5.61µs   med=9.78µs   max=19.53ms p(90)=17.2µs p(95)=29.57µs
     http_req_sending...............: avg=12.09µs min=3.84µs   med=5.94µs   max=25.31ms p(90)=9.63µs p(95)=14.75µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s       max=0s      p(90)=0s     p(95)=0s     
     http_req_waiting...............: avg=1.25ms  min=92.49µs  med=758.31µs max=32.66ms p(90)=2.83ms p(95)=4.03ms 
     http_reqs......................: 306568  12262.12259/s
     iteration_duration.............: avg=1.38ms  min=150.87µs med=863.46µs max=32.83ms p(90)=3.03ms p(95)=4.3ms  
     iterations.....................: 306568  12262.12259/s
     vus............................: 1       min=1    max=29  
     vus_max........................: 30      min=30   max=30  

nginx private rego

Run load test: k6 run loadtest-nginx-private-rego.js

running (25.0s), 00/30 VUs, 50707 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0     ✗ 50707
     checks.........................: 100.00% ✓ 50707 ✗ 0    
     data_received..................: 43 MB   1.7 MB/s
     data_sent......................: 6.1 MB  243 kB/s
     http_req_blocked...............: avg=2.28µs  min=810ns    med=1.52µs  max=1.68ms  p(90)=2.61µs  p(95)=2.97µs 
     http_req_connecting............: avg=252ns   min=0s       med=0s      max=1.47ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=8.34ms  min=360.45µs med=3.56ms  max=67.39ms p(90)=22.38ms p(95)=28.7ms 
       { expected_response:true }...: avg=8.34ms  min=360.45µs med=3.56ms  max=67.39ms p(90)=22.38ms p(95)=28.7ms 
     http_req_failed................: 0.00%   ✓ 0     ✗ 50707
     http_req_receiving.............: avg=33.82µs min=9.31µs   med=23.83µs max=21.97ms p(90)=52.27µs p(95)=59.67µs
     http_req_sending...............: avg=12.55µs min=4.36µs   med=7.68µs  max=15.2ms  p(90)=13.58µs p(95)=15.63µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=8.29ms  min=328.54µs med=3.51ms  max=67.36ms p(90)=22.32ms p(95)=28.64ms
     http_reqs......................: 50707   2028.021526/s
     iteration_duration.............: avg=8.43ms  min=420.46µs med=3.65ms  max=67.48ms p(90)=22.49ms p(95)=28.82ms
     iterations.....................: 50707   2028.021526/s
     vus............................: 1       min=1   max=29 
     vus_max........................: 30      min=30  max=30 

And to compare, using OPA behind nginx (which will always respond with OK) - the result is about the same:

nginx private opa

PLEASE OBSERVE: OPA WILL ALWAYS RESPOND WITH OK HOWEVER THE RESULT IS EVALUATED

Run load test: k6 run loadtest-nginx-private-opa.js

running (25.0s), 00/30 VUs, 51367 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0     ✗ 51367
     checks.........................: 100.00% ✓ 51367 ✗ 0    
     data_received..................: 44 MB   1.7 MB/s
     data_sent......................: 6.1 MB  244 kB/s
     http_req_blocked...............: avg=2.6µs   min=708ns    med=1.5µs   max=3.58ms   p(90)=2.66µs  p(95)=2.99µs 
     http_req_connecting............: avg=560ns   min=0s       med=0s      max=3.42ms   p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=8.22ms  min=319.62µs med=3.36ms  max=128.61ms p(90)=20.47ms p(95)=36.5ms 
       { expected_response:true }...: avg=8.22ms  min=319.62µs med=3.36ms  max=128.61ms p(90)=20.47ms p(95)=36.5ms 
     http_req_failed................: 0.00%   ✓ 0     ✗ 51367
     http_req_receiving.............: avg=32.74µs min=11.03µs  med=23.11µs max=12.56ms  p(90)=52.05µs p(95)=59.48µs
     http_req_sending...............: avg=11.88µs min=4.61µs   med=7.7µs   max=9.5ms    p(90)=13.72µs p(95)=15.89µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s       p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=8.18ms  min=288.67µs med=3.32ms  max=128.57ms p(90)=20.41ms p(95)=36.44ms
     http_reqs......................: 51367   2054.510551/s
     iteration_duration.............: avg=8.32ms  min=382.99µs med=3.46ms  max=128.67ms p(90)=20.59ms p(95)=36.58ms
     iterations.....................: 51367   2054.510551/s
     vus............................: 1       min=1   max=29 
     vus_max........................: 30      min=30  max=30 

[simongottschlag]

I did another change, added connection reuse to nginx - and the results got much better and now the rego is better than the proxy. One really weird thing happend as well - OPA seems to get issues when nginx sets proxy_set_header Connection "";. No idea why!

You can see the nginx changes here: simongottschlag/opa-nginx-authz@6806fed#diff-4adaaefa7fc959e7dfe65e8dc8dadc9ee27ddd284eee3426463b1cdc000587d5

And for documentations sake, these were the changes to the nginx.conf:

added the following inside the http block:

  upstream auth_backend {
    server opa-nginx-external-auth-test:8082;

    keepalive 50;
  }

  upstream opa_backend {
    server opa-test:8181;

    keepalive 50;
  }

Changed the auth endpoint to use the upstreams and added the following:

        proxy_http_version 1.1;
        proxy_set_header Connection "";

Full auth endpoint example:

    location = /auth/rego {
        internal;
        proxy_pass              http://auth_backend/rego;
        proxy_pass_request_body off;
        proxy_set_header        Content-Length "";
        proxy_set_header        X-Original-URI $request_uri;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }

The results with connection reuse:

nginx private proxy

Run load test: k6 run loadtest-nginx-private-proxy.js

running (25.0s), 00/30 VUs, 119911 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0      ✗ 119911
     checks.........................: 100.00% ✓ 119911 ✗ 0     
     data_received..................: 102 MB  4.1 MB/s
     data_sent......................: 14 MB   580 kB/s
     http_req_blocked...............: avg=1.99µs  min=766ns    med=1.47µs  max=2.84ms  p(90)=1.88µs  p(95)=2.39µs 
     http_req_connecting............: avg=155ns   min=0s       med=0s      max=1.06ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=3.47ms  min=357.79µs med=2.65ms  max=41.64ms p(90)=7.08ms  p(95)=9.23ms 
       { expected_response:true }...: avg=3.47ms  min=357.79µs med=2.65ms  max=41.64ms p(90)=7.08ms  p(95)=9.23ms 
     http_req_failed................: 0.00%   ✓ 0      ✗ 119911
     http_req_receiving.............: avg=28.19µs min=9.63µs   med=22.23µs max=15.88ms p(90)=31.31µs p(95)=36.38µs
     http_req_sending...............: avg=11.35µs min=4.51µs   med=7.53µs  max=15.18ms p(90)=11.13µs p(95)=13.9µs 
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=3.44ms  min=324.46µs med=2.62ms  max=41.6ms  p(90)=7.03ms  p(95)=9.16ms 
     http_reqs......................: 119911  4796.152669/s
     iteration_duration.............: avg=3.55ms  min=414.45µs med=2.73ms  max=41.71ms p(90)=7.18ms  p(95)=9.34ms 
     iterations.....................: 119911  4796.152669/s
     vus............................: 1       min=1    max=29  
     vus_max........................: 30      min=30   max=30 

nginx private rego

Run load test: k6 run loadtest-nginx-private-rego.js

running (25.0s), 00/30 VUs, 199494 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✓ status is 200

     check_failure_rate.............: 0.00%   ✓ 0      ✗ 199494
     checks.........................: 100.00% ✓ 199494 ✗ 0     
     data_received..................: 170 MB  6.8 MB/s
     data_sent......................: 24 MB   958 kB/s
     http_req_blocked...............: avg=2.55µs  min=659ns    med=1.33µs  max=10.42ms p(90)=1.84µs p(95)=2.39µs 
     http_req_connecting............: avg=312ns   min=0s       med=0s      max=3.59ms  p(90)=0s     p(95)=0s     
     http_req_duration..............: avg=2.04ms  min=202.62µs med=1.42ms  max=34.16ms p(90)=4.27ms p(95)=5.8ms  
       { expected_response:true }...: avg=2.04ms  min=202.62µs med=1.42ms  max=34.16ms p(90)=4.27ms p(95)=5.8ms  
     http_req_failed................: 0.00%   ✓ 0      ✗ 199494
     http_req_receiving.............: avg=30.62µs min=9.2µs    med=17.18µs max=20.69ms p(90)=27.2µs p(95)=36.21µs
     http_req_sending...............: avg=12.85µs min=3.84µs   med=6.65µs  max=20.59ms p(90)=9.74µs p(95)=13.98µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s      p(90)=0s     p(95)=0s     
     http_req_waiting...............: avg=1.99ms  min=184.75µs med=1.38ms  max=33.99ms p(90)=4.21ms p(95)=5.71ms 
     http_reqs......................: 199494  7979.292418/s
     iteration_duration.............: avg=2.13ms  min=251.11µs med=1.5ms   max=34.24ms p(90)=4.41ms p(95)=5.98ms 
     iterations.....................: 199494  7979.292418/s
     vus............................: 1       min=1    max=29  
     vus_max........................: 30      min=30   max=30  

nginx private opa

PLEASE OBSERVE: OPA WILL ALWAYS RESPOND WITH OK HOWEVER THE RESULT IS EVALUATED

Run load test: k6 run loadtest-nginx-private-opa.js

running (25.0s), 00/30 VUs, 33899 complete and 0 interrupted iterations
default ✓ [======================================] 00/30 VUs  25s

     ✗ status is 200
      ↳  83% — ✓ 28231 / ✗ 5668

     check_failure_rate.............: 16.72% ✓ 5668  ✗ 28231
     checks.........................: 83.27% ✓ 28231 ✗ 5668 
     data_received..................: 28 MB  1.1 MB/s
     data_sent......................: 4.0 MB 161 kB/s
     http_req_blocked...............: avg=31.17µs min=892ns    med=1.58µs  max=15.33ms p(90)=112.21µs p(95)=136.45µs
     http_req_connecting............: avg=21.64µs min=0s       med=0s      max=15.02ms p(90)=71.48µs  p(95)=87.06µs 
     http_req_duration..............: avg=12.49ms min=313.34µs med=10.37ms max=89.58ms p(90)=28.41ms  p(95)=33.98ms 
       { expected_response:true }...: avg=11.2ms  min=313.34µs med=6.33ms  max=89.58ms p(90)=27.19ms  p(95)=33.23ms 
     http_req_failed................: 16.72% ✓ 5668  ✗ 28231
     http_req_receiving.............: avg=41.98µs min=10.64µs  med=26.02µs max=13.65ms p(90)=72.94µs  p(95)=88.66µs 
     http_req_sending...............: avg=18.31µs min=4.4µs    med=8.26µs  max=14.38ms p(90)=43.34µs  p(95)=51.85µs 
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=12.43ms min=286.56µs med=10.31ms max=89.54ms p(90)=28.31ms  p(95)=33.89ms 
     http_reqs......................: 33899  1355.734075/s
     iteration_duration.............: avg=12.61ms min=377.01µs med=10.5ms  max=89.66ms p(90)=28.56ms  p(95)=34.12ms 
     iterations.....................: 33899  1355.734075/s
     vus............................: 1      min=1   max=29 
     vus_max........................: 30     min=30  max=30 

[ashutosh-narkar]

👍

Ive done one change, went from PartialResult to PrepareForEval and got a bit better performance. With PrepareForEval, it actually seems it's performing better than using the vanilla OPA container.

nginx private rego: Is this use-case where you embed OPA as a library ?

[simongottschlag]

nginx private rego: Is this use-case where you embed OPA as a library ?

@ashutosh-narkar exactly! rego in my case refers to embedding OPA while proxy refers to proxying to OPA.

Since we see the results and difference between the two, it would be interesting to understand how the maintainers would like to proceed?

I see three ways forward: (order of personal preference)

1. Update OPA to support the use case
2. Create a separate repository for the nginx support, but keeping in in the OPA organization (GitHub)
3. Create a separate repository for the nginx support, outside of the OPA organization (GitHub)

I would of course help with the first two if you wanted and if none of them are interesting then I could create something separate in my GitHub org and take care of the maintenance. In this case, I think keeping it close to everything else OPA would make it more available to the community as well as less overhead in regards to maintaining it - or at least I hope so.

How would you like to proceed?

CC @tsandall @anderseknert

[ashutosh-narkar]

@simongottschlag my recommendation is we take your service that embeds OPA for the nginx use-case and make it part of the contrib repo in the OPA organization. If we can have extensive documentation for how this new service is expected to work (for ex. see the opa-envoy-plugin), it would immensely help the community and also help drive collaboration and adoption.

Others may want to chime in as well.

[simongottschlag]

@ashutosh-narkar thank you! I will close this issue for now and create a new one in contrib when time allows. 😊👍