[Filebeat] httpjson input dot notation in last response

[leehinman]

In 8.0.0 the keys for last_response have "url.params" instead of url
& params being structs with accessible fields.

Steps to Reproduce

API server

Small API server written in Flask.

#!/usr/bin/env python3

from flask import Flask, json

companies = {"id": 1, "name": "Company One"}

api = Flask(__name__)

@api.route('/companies', methods=['GET'])
def get_companies():
  return json.dumps(companies)

if __name__ == '__main__':
    api.run() 

httpjson config

- type: httpjson
  enabled: true
  interval: 5s
  request.method: "GET"
  request.url: http://localhost:5000/companies
  request.transforms:
    - set:
        target: url.params.start
        value: "[[.cursor.last_execution_datetime]]"
        default: '[[formatDate ((now).Add (parseDuration "-5s"))]]'
    - set:
        target: url.params.end
        value: '[[formatDate ((parseDate .cursor.last_execution_datetime).Add (parseDuration "5s"))]]'
        default: '[[formatDate now]]'
  cursor:
    last_execution_datetime:
       value: '[[.last_response]]'

Results

{
  "log.level": "debug",
  "@timestamp": "2022-02-18T10:57:57.058-0600",
  "log.logger": "input.httpjson-cursor",
  "log.origin": {
    "file.name": "httpjson/cursor.go",
    "file.line": 56
  },
  "message": "cursor.last_execution_datetime stored with {\"body\":{\"id\":1,\"name\":\"Company One\"},\"header\":{\"Content-Length\":[\"32\"],\"Content-Type\":[\"text/html; charset=utf-8\"],\"Date\":[\"Fri, 18 Feb 2022 16:57:57 GMT\"],\"Server\":[\"Werkzeug/2.0.3 Python/3.9.10\"]},\"page\":1,\"url.params\":{\"end\":[\"2022-02-18T16:57:57Z\"],\"start\":[\"2022-02-18T16:57:52Z\"]},\"url.value\":\"http://localhost:5000/companies?end=2022-02-18T16%3A57%3A57Z\\u0026start=2022-02-18T16%3A57%3A52Z\"}",
  "service.name": "filebeat",
  "id": "D60AF34257EE2D7B",
  "input_source": "http://localhost:5000/companies",
  "input_url": "http://localhost:5000/companies",
  "ecs.version": "1.6.0"
}

notice key names of "url.params" and "url.value"

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

I think this was fixed, but a backport for 8.0 was missed in #28695. Looks like this change fixed it, but I'm not 100% sure.

https://github.com/elastic/beats/pull/28695/files#diff-38319f0690a1641fe887515fd856de7714b72235b98094d9caee38d13dcd8bc8R180

[leehinman]

Backport of #28695 should get us to the behavior in #30477 , I just wanted to document that 7.x and 8.x both had issues around the data stored in .last_response but in different way. Just so we didn't fix one but not the other.

[marc-gr]

Did the missing backport to 8.0. #30574