Bruno won't send ndjson body properly

[Rmaan]

Context

In Elasticsearch, when you want to do a bulk indexing, you send all documents in JSON separated by new lines (known as ndjson or jsonl format). According to standard and Elasticsearch, body should end with a new line.

Problem

In Bruno, it seems there's no way to send a body that ends with a new line, I tried JSON format or text, it doesn't matter. Seems somehow newline is removed somewhere. When I use "generate code" and copy the curl it's fine but when I run it in bruno Elasticsearch complains.

Sample .bru file:

meta {
  name: Bulk add 22
  type: http
  seq: 47
}

post {
  url: {{ES_URL}}/tmp/_bulk
  body: json
  auth: none
}

headers {
  Content-Type: application/json
}

body:json {
  { "update" : {"_id" : "6"}
  { "doc" : {"f1" : 8}, "doc_as_upsert" : true}

}

Sample cURL:

curl --request POST \
  --url http://localhost:9200/tmp/_bulk \
  --header 'Content-Type: application/json' \
  --data '{ "update" : {"_id" : "6"}
{ "doc" : {"f1" : 8}, "doc_as_upsert" : true}
'

Error returned from Elasticsearch (only when run with bru):

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "The bulk request must be terminated by a newline [\\n]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "The bulk request must be terminated by a newline [\\n]"
  },
  "status": 400
}

When I run an nc on my local, I get a weird body being sent:

➜  ~ nc -l 9200
POST /tmp/_bulk HTTP/1.1
Accept: application/json, text/plain, */*
Content-Type: application/json
request-start-time: 1708628407192
User-Agent: axios/1.5.1
Content-Length: 93
Accept-Encoding: gzip, compress, deflate, br
Host: localhost:9200
Connection: close

"{ \"update\" : {\"_id\" : \"6\"}\n{ \"doc\" : {\"f1\" : 8}, \"doc_as_upsert\" : true}\n\n\n"

BTW keep up good work 🙌 I'm waiting for my golden edition ☺️

[mjhcorporate]

When you use json as body-type, then your request should not work -- since newline-delimited-json is not json.

But using "text" as body-type absolutely should work. I encountered the same bug in my workaround here: #1768 (comment)

[mjhcorporate]

Aha! It seems this behavior only happens if you manually set the Content-Type: application/json header!
Arguably, bruno should just send the body as-is, allowing you to send malformed json, but in your case, you can just set the header to application/x-ndjson, which is more appropriate anyways.

[Rmaan]

When you use json as body-type, then your request should not work -- since newline-delimited-json is not json.

It's not standard JSON but It works with curl or any other client as Elasticsearch accepts this Content-Type. Bruno is messing with the content.

At least if it's not a proper JSON I don't think it's good to turn it into a big JSON string, that's not what user ever intends. I think modifying body should never happen, unless you want to add some goodies, e.g. I saw if I comment a JSON line with // it will be skipped in the output. This can be a nice feature. But every JSON blob, and turning invalid JSON into a JSON string is too much.

But using "text" as body-type absolutely should work. I encountered the same bug in my workaround here:

It doesn't work I tried it 😅, body type doesn't change the behavior. Text or JSON body, with or without JSON content type, it will mess up the body.

this behavior only happens if you manually set the Content-Type: application/json header

Even if I don't set any headers this will happen as well. The only solution is what you said, setting content type to application/x-ndjson.

[mjhcorporate]

Unfortunately, this behavior is (mostly) caused by the underlying axios library: https://github.com/axios/axios/blob/d844227411263fab39d447442879112f8b0c8de5/lib/defaults/index.js#L42

I tend to agree with @Rmaan : Bruno is a testing tool and should send the body exactly as I define it, even if that violates some standard. Seeing what an API does with malformed data is a core part of testing.

It is possible to remove the default transforms from axios, and we should probably do that, but this could potentially break usage in other places.

[Rmaan]

Interesting, I think we can copy the default transformer and remove the stringifySafely function. That is the weird behavior that if JSON.parse fails it will do a JSON.stringify instead.