clearScroll causes 403 when using elasticsearch 12.1.1 and higher

[ajkerr]

Something changed in the elasticsearch module in version 12.1.1 and up, which is causing request signing errors for the clearScroll() method.

"The request signature we calculated does not match the signature you provided."

Reverting to 12.1.0 fixes the issue.

I think that this commit is the culprit: elastic/elasticsearch-js@97706ce

[jonberke]

We are experiencing the problem but can't downgrade to 12.1.0 of the ES library because it doesn't support the ES version we are running (5.3).

We've contacted AWS support about this but they seem to blame this module for not generating the signature correctly. I can't see how this module can incorrectly generate the signature for the clearScroll request but generate it correctly for every other request.

Has anyone found a workaround for this problem that works with ES 5.3 or do you have any information I can pass to AWS to get them to look into this issue more deeply?

[ajkerr]

@jonberke We still have the problem, it it's becoming more serious because it prevents us from moving to a newer version of Elasticsearch hosted in AWS. We are currently stuck on 2.x.

I've written a couple of sample programs that show that illustrate the problem:

This first program is written for http-aws-es 1.1.3 and elasticsearch 12.1.0. I was unable to use newer versions of http-aws-es here because of a peerDependency issue.

const elasticsearch = require("elasticsearch");
const AWS = require("aws-sdk");

AWS.config.getCredentials((err) => {
  if (err) {
    console.log(`[error] credentials are not available: ${JSON.stringify(err)}`);
  }
  const amazonESProperties = {
    region: "us-west-2",
    credentials: AWS.config.credentials,
  };
  const client = new elasticsearch.Client({
    host: "https://your-es-endpoint.amazonaws.com",
    connectionClass: require("http-aws-es"),
    amazonES: amazonESProperties,
    apiVersion: "2.4",
    log: "debug"
  });

  client.clearScroll({scrollId: "12345"}, (err, res, status) => {
    if (err || status !== 200) {
      console.log(`${status} status. Could not clear scroll: ${JSON.stringify(err)}`);
    }
    console.log("res: %j", res);
  });
});

This program will fail with 400 status. Could not clear scroll, which is correct, because the scroll id is made up.

The second program is written for http-aws-es 3.1.0 and elasticsearch 13.3.1 (the latest at the time of writing):

const elasticsearch = require("elasticsearch");
const AWS = require("aws-sdk");

AWS.config.update({region: "us-west-2"});

const client = new elasticsearch.Client({
  host: "https://your-es-endpoint-es.amazonaws.com",
  connectionClass: require("http-aws-es"),
  apiVersion: "2.4",
  log: "debug"
});

client.clearScroll({scrollId: "12345"}, (err, res, status) => {
  if (err || status !== 200) {
    console.log(`${status} status. Could not clear scroll: ${JSON.stringify(err)}`);
  }
  console.log("res: %j", res);
});

In this case, it fails with a 403 status:

403 status. Could not clear scroll: {"msg":"Authorization Exception","path":"/_search/scroll","query":{},"body":"{\"scroll_id\":\"12345\"}","statusCode":403,"response":"{\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

As I mentioned when I first opened the bug, this started happening with version 12.1.1 of the elasticsearch module.

One item of interest. The original, working program, sends a request that looks like this:

Elasticsearch DEBUG: 2017-09-27T15:00:01Z
  starting request {
    "method": "DELETE",
    "path": "/_search/scroll/12345",
    "query": {}
  }

The second, broken program with the signing error, sends a request that looks like this:

Elasticsearch DEBUG: 2017-09-27T15:07:03Z
  starting request {
    "method": "DELETE",
    "path": "/_search/scroll",
    "body": {
      "scroll_id": "12345"
    },
    "query": {}
  }

[ajkerr]

Note that calling scroll() instead of clearScroll() works fine. I can't see much difference between the generated requests other than the HTTP method. Perhaps it is a bug in the AWS signing code?

@jonberke Hopefully my test programs are useful in getting AWS to help you with this.

[jonberke]

I believe the problem is either with the signing code in this library or with AWS decoding the signature when the method is DELETE and a payload (scroll id to delete) is in the body. I tried a test where I forced the scroll id onto the URL instead of the body and it worked - no error clearing the scroll.

[ajkerr]

@jonberke I'm leaning towards the latter. I found this old forum thread: https://forums.aws.amazon.com/thread.jspa?threadID=227353

It doesn't appear to have been resolved by AWS in a satisfactory way.

[ajkerr]

I've opened aws/aws-sdk-js#1733 to track this issue as well. I created a sample program that shows the issue only using the aws-sdk.

[jonberke]

Awsome! I'll include a link to that issue in my case with AWS support.

…

On Wed, Sep 27, 2017 at 1:03 PM, Andrew Kerr ***@***.***> wrote: I've opened aws/aws-sdk-js#1733 <aws/aws-sdk-js#1733> to track this issue as well. I created a sample program that shows the issue only using the aws-sdk. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPzY72Bims4SjEAZS6Fqe6ONj2GZagyks5smqoegaJpZM4Li8Yx> .

[jonberke]

Good news. Got this from AWS support:

Elasticsearch Service Team identified the root cause of the problem. The problem is with the aws-sdk-js library only.

When we send the DELETE with body, body is not sent in the request, causing the signature mismatch.

FYI - Elasticsearch Service team is doing follow-up with the sdk team regarding this.

@ajkerr I think your script with only the aws sdk did the trick. Nicely done.

[ajkerr]

@jonberke That's good to hear. Assuming that they fix the JS SDK in a timely manner, the only fix for this repo would be to update the peerDependencies in package.json to the version of the SDK with the fix.

[TheDeveloper]

Thanks for digging into this @ajkerr & @jonberke!

Happy to bump the peerDeps when fix is live. Assuming they push it as patch it should match the semver anyway.

[ajkerr]

@jonberke @TheDeveloper I've added some more info to aws/aws-sdk-js#1733 that should help them fix the problem. Adding a proper Content-Length header to the request seems to fix the issue.

[subodhkhanduri1]

The request body is not sent because of the missing Content-Length header.

I've opened #41 to fix this. Please review.

[jonberke]

@TheDeveloper I was just wondering if you had an ETA for when you expect to merge the pull request that fixes this issue? Thanks.

[ajkerr]

@TheDeveloper I tend to agree that the submitted PR is the best best for fixing this, based on comments in aws/aws-sdk-js#1733 (comment)

[jonberke]

@TheDeveloper Any chance we can get this PR integrated soon?

[TheDeveloper]

Great work. Sorry for the delay. Merged in v3.1.1