Elasticsearch high-level REST client doesn't send additional HTTP headers while querying for Elasticsearch version

[mikepenzo]

Elasticsearch Version

Elasticsearch high-level REST client 7.17

Installed Plugins

No response

Java Version

openjdk version "17.0.13" 2024-10-15

OS Version

Linux 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64 GNU/Linux

Problem Description

If Elasticsearch high-level REST client is used in front of an API gateway, such as Spring Cloud Gateway, it sometimes needs to be configured to send custom additional HTTP headers in order to be validated by the API gateway (these headers might contain mandatory values to submit).
We discovered that the client checks the version of Elasticsearch without sending the additional headers. It anyway ignores the problem if it gets HTTP codes 401 or 403, but not if the returned code is HTTP 400.
In my humble opinion, the root cause of this error is not sending the additional headers. Can you please fix that?

Steps to Reproduce

1. Configure Spring Cloud Gateway to stop calls without a set of mandatory HTTP headers. Be sure that it returns 400 and not 401/403 HTTP codes.
2. Configure the high-level client with the mandatory HTTP headers.
3. Run a generic call with the high-level gateway. Even if the call is "white-listed", the client will return an error anyway.

Logs (if relevant)

No response

[swallez]

Can you elaborate on "Configure the high-level client with the mandatory HTTP headers"? Is is using RequestOptions? The version check indeed doesn't send the RequestOptions that are passed with the first client request.

There is a workaround though by setting the default headers on the low level Rest client. They can also be updated later if needed (e.g. token renewal).

RestClientBuilder builder = RestClient.builder(...);

Header[] headers = new Header[] {
    new BasicHeader("SomeHeader", "some-value"),
};
builder.setDefaultHeaders(headers);

// Create HLRC, do some stuff, and at some point we need to update the header

headers[0] = new BasicHeader("SomeHeader", "some-other-value");

// Some more stuff that will use the new header value

Does this help?

[elasticsearchmachine]

Pinging @elastic/clients-team (Team:Clients)

[mikepenzo]

Hi,

The solution you recommended doesn't work, because the RequestOptions are passed in the execute call, but they are ignored by the internal call for version validation that is executed together with the intended call. It should include it also there, as there could be things like proxy requiring specific headers.
Using DefaultHeaders is an valid option if you don't need dynamically generated headers (which we do, like header with a specific timestamp).
Current workaround is to attach interceptor directly on the httpClient and generate headers there.

We configured Spring Cloud Gateway in a way similar to this:

server:
port: 4000

spring:
cloud:
gateway:
global-filter:
- LoggingGlobalFilter
- ExtraLoggingFilter
routes:
- id: elasticsearch-route
uri: http://10.42.1.144:9200
predicates:
- Path=/api/targetelk/_,/api/targetelk//**
filters:
- name: ValidateHeadersFilter
- name: RequestSize
args:
maxSize: ${maxReqSize}
- RewritePath=/api/targetelk/(?.*), /${segment}
metadata:
redirectURI: http://10.42.1.144:9200
organisations: [ "sampleOrg" ]
logging:
level:
reactor:
netty: INFO
org:
springframework:
cloud:
gateway: INFO

where the ValidateHeadersFilter is a Java class checking if HTTP mandatory headers are in place:
@component("ValidateHeadersFilter")
@slf4j
public class ValidateHeadersFilter extends AbstractGatewayFilterFactory<ValidateHeadersFilter.Config>
...
(I can't share the code, I'm sorry)

[swallez]

Closing, as I understand you found a workaround.

[mikepenzo]

Hi,

I found a workaround, but IMHO the bug is still there. Can you pls check or assign to other developers?

Best regards
Michele