Filebeat - Modules Apache - Error : Fix Client IP - missing PORT

[wixaw]

http://httpd.apache.org/docs/current/mod/core.html#errorlog

Default : ErrorLogFormat "[%t] [%l] [pid %P] %F: %E: [client %a] %M"
%a = Client IP address and port of the request

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[jsoriano]

Thanks for investigating this issue and proposing a fix!
Could you also provide a test file that reproduces this issue? This should be added to filebeat/module/apache/error/test/ directory, you can find other examples there.

[wixaw]

Hello @jsoriano
I had not tried it on production machines but for your request I did it.
Neither works. When I modify the pipeline.json and I restart filebeat (centos 7) nothing changes
Is this compiled somewhere?
Here is my log of example:
[Thu Jun 27 06:58:09.169510 2019] [include:warn] [pid 15934] [client 123.123.123.123:12345] AH01374: mod_include: Options +Includes (or IncludesNoExec) wasn't set, INCLUDES filter removed: /test.html

This is JSON view in Kibana

{
  "_index": "filebeat-7.1.1-2019.06.11-000001",
  "_type": "_doc",
  "_id": "mLRMl2sBrO6GboNYYi5E",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "hostname": "servername",
      "id": "d16d6551-9c31-483a-89dc-07521dc178cc",
      "ephemeral_id": "d20240ec-96e9-4465-acf3-518ebdff12da",
      "type": "filebeat",
      "version": "7.1.1"
    },
    "process": {
      "pid": 15934
    },
    "log": {
      "file": {
        "path": "/var/log/httpd/www_error.log"
      },
      "offset": 82805,
      "level": "warn"
    },
    "message": "[client 123.123.123.123:12345] AH01374: mod_include: Options +Includes (or IncludesNoExec) wasn't set, INCLUDES filter removed: /test.html",
    "fileset": {
      "name": "error"
    },
    "input": {
      "type": "log"
    },
    "@timestamp": "2019-06-27T06:58:09.169Z",
    "apache": {
      "error": {
        "module": "include"
      }
    },
    "ecs": {
      "version": "1.0.0"
    },
    "service": {
      "type": "apache"
    },
    "host": {
      "hostname": "servername",
      "os": {
        "kernel": "3.10.0-957.1.3.el7.x86_64",
        "codename": "Core",
        "name": "CentOS Linux",
        "family": "redhat",
        "version": "7 (Core)",
        "platform": "centos"
      },
      "containerized": false,
      "name": "servername",
      "id": "98fc39052d3c4fbc9ad38e57fd127ef0",
      "architecture": "x86_64"
    },
    "event": {
      "module": "apache",
      "dataset": "apache.error"
    }
  },
  "fields": {
    "suricata.eve.timestamp": [
      "2019-06-27T06:58:09.169Z"
    ],
    "@timestamp": [
      "2019-06-27T06:58:09.169Z"
    ]
  },
  "highlight": {
    "event.module": [
      "@kibana-highlighted-field@apache@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1561618689169
  ]
}

We see that in message the "client" part is not extracted

[jsoriano]

@wixaw thanks for testing my proposal, actually you are right and my change wouldn't be enough because there are additional processors expecting source.address to be just an ip or a host.

Neither works. When I modify the pipeline.json and I restart filebeat (centos 7) nothing changes
Is this compiled somewhere?

The pipeline needs to be installed in Elasticsearch using filebeat setup.

Here is my log of example:
[Thu Jun 27 06:58:09.169510 2019] [include:warn] [pid 15934] [client 123.123.123.123:12345] AH01374: mod_include: Options +Includes (or IncludesNoExec) wasn't set, INCLUDES filter removed: /test.html

Thanks!

[wixaw]

Yesss
Perfect it's work !!
Thanks

{
  "_index": "filebeat-7.1.1-2019.06.11-000001",
  "_type": "_doc",
  "_id": "iLivmGsBrO6GboNYAmxo",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "hostname": "servername",
      "id": "d16d6551-9c31-483a-89dc-07521dc178cc",
      "ephemeral_id": "5a212816-3cd7-4378-bfbd-79afc938de17",
      "type": "filebeat",
      "version": "7.1.1"
    },
    "process": {
      "pid": 24096
    },
    "log": {
      "file": {
        "path": "/var/log/httpd/www_error.log"
      },
      "offset": 91097,
      "level": "warn"
    },
    "source": {
      "geo": {
        "continent_name": "Europe",
        "country_iso_code": "FR",
        "location": {
          "lon": 2.xxxxxxxx,
          "lat": 4x.xxxxx
        }
      },
      "address": "123.123.123.123,
      "port": "46142",
      "ip": "123.123.123.111"
    },
    "message": "AH01374: mod_include: Options +Includes (or IncludesNoExec) wasn't set, INCLUDES filter removed: /test.html",
    "fileset": {
      "name": "error"
    },
    "input": {
      "type": "log"
    },
    "@timestamp": "2019-06-27T13:25:29.234Z",
    "apache": {
      "error": {
        "module": "include"
      }
    },
    "ecs": {
      "version": "1.0.0"
    },
    "service": {
      "type": "apache"
    },
    "host": {
      "hostname": "servername",
      "os": {
        "kernel": "3.10.0-957.1.3.el7.x86_64",
        "codename": "Core",
        "name": "CentOS Linux",
        "family": "redhat",
        "version": "7 (Core)",
        "platform": "centos"
      },
      "containerized": false,
      "name": "servername",
      "id": "98fc39052d3c4fbc9ad38e57fd127ef0",
      "architecture": "x86_64"
    },
    "event": {
      "module": "apache",
      "dataset": "apache.error"
    }
  },
  "fields": {
    "suricata.eve.timestamp": [
      "2019-06-27T13:25:29.234Z"
    ],
    "@timestamp": [
      "2019-06-27T13:25:29.234Z"
    ]
  },
  "highlight": {
    "event.module": [
      "@kibana-highlighted-field@apache@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1561641929234
  ]
}

Proposed changes addressed

[jsoriano]

@wixaw merged, thanks!