Canvas fails with timelion datasource for self-signed certificate

[kibanamachine]

Original comment by @jguay:

I am not sure if this is just a documentation bug or this would require some new settings in kibana.yml

steps to reproduce

1a- Use LINK REDACTED and login to the kibana container as root

docker exec -i -t --privileged -u root kibana /bin/bash

OR 1b- Install kibana with self signed certificate (using certgen/certutil), adding ca with this (docker env variables here replace by valid values if using kibana.yml)

      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_KEY=/usr/share/kibana/config/certs/kibana/kibana.key
      - SERVER_SSL_CERTIFICATE=/usr/share/kibana/config/certs/kibana/kibana.crt
      - SERVER_SSL_CERTIFICATEAUTHORITIES=/usr/share/kibana/config/certs/ca/ca.crt

2- Note that curl without -k fails now (also highlighted by browser)

EMAIL REDACTED kibana]# curl https://localhost:5601
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.

3- Force kibana OS to trust the CA:

1- If needed, Install the ca-certificates package: yum install ca-certificates
2- Enable the dynamic CA configuration feature: update-ca-trust force-enable
3- Add it as a new file to /etc/pki/ca-trust/source/anchors/: cp /usr/share/kibana/config/certs/ca/ca.crt /etc/pki/ca-trust/source/anchors/
4- Use command: update-ca-trust extract

4- Curl will now work

# curl https://localhost:5601
<script>var hashRoute = '/app/kibana';

5- Restart kibana for good measures
6- In Kibana add an element and change data source to timelion returns:

{
  "error": {
    "stack": "Error: self signed certificate in certificate chain\n    at TLSSocket.<anonymous> (_tls_wrap.js:1105:38)\n    at emitNone (events.js:106:13)\n    at TLSSocket.emit (events.js:208:7)\n    at TLSSocket._finishInit (_tls_wrap.js:639:8)\n    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:469:38)",
    "message": "self signed certificate in certificate chain"
  },
  "info": {
    "context": {
      "type": "filter",
      "meta": {},
      "and": []
    },
    "args": {},
    "functionName": "timelion"
  }
}

7- In my case Chrome is on Mac I also LINK REDACTED and I restart Chrome now browser has no issue to trust kibana certificate but canvas still does :

[kibanamachine]

Original comment by @jguay:

Ok I found the solution which is an env variable in kibana server :

NODE_EXTRA_CA_CERTS=/usr/share/kibana/config/certs/ca/ca.crt

Maybe just someone can review if this is already documented somewhere or is worth documenting

[kibanamachine]

Original comment by @tsullivan:

This looks to me like a valid bug, not just a documentation thing. It seems like Canvas/Timelion should be aware of the SERVER_SSL_CERTIFICATEAUTHORITIES setting and just work. I'd like to look into this and see why it doesn't

[kibanamachine]

Original comment by @tsullivan:

I won't be able to work on this right away, but I took a quick look at the docker compose file.

• Do I need to create my own certs before starting the stack?
• Do I need to set an env variable to have it use v6.2.4?

[kibanamachine]

Original comment by @jguay:

No urgency on my end even before canvas is GA, we could knowledge base this if this question/issue comes again once we officially support canvas...

Do I need to create my own certs before starting the stack?

Not if you use the LINK REDACTED, LINK REDACTED creates LINK REDACTED automatically using certutil (ca.crt, kibana.crt/key...), those certificate can be used by localhost installation if you need just this
And also I also map port 9200 so you could possibly just start "docker-compose up -d elasticsearch" and you will then have "https://localhost:9200" with elastic/changeme and use a local kibana

Do I need to set an env variable to have it use v6.2.4?

I switched to 6.3.0 yesterday, if you need 6.2.4, let me know and I can adapt the file. For reference, you need to adapt the Dockerfile basically revert my latest LINK REDACTED) of kibana but also you would need another [canvas install file], the .env file and LINK REDACTED changing "elasticsearch:${TAG}" for "elasticsearch-platinum:${TAG}" (this is because since 6.3.0 there is no longer elasticsearch-platinum docker image which was the image for ES with x-pack before we opened it)

Anything on docker side in general or related to the example I gave, do slack me (I work EMEA times) whenever you need.

[kibanamachine]

Original comment by @tsullivan:

I can reproduce this in my dev environment. In my Kibana server console, I see:

common/interpret timelion: invokeChain rejected { Error: self signed certificate in certificate chain
    at TLSSocket.<anonymous> (_tls_wrap.js:1105:38)
    at emitNone (events.js:106:13)
    at TLSSocket.emit (events.js:208:7)
    at TLSSocket._finishInit (_tls_wrap.js:639:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:469:38)
  code: 'SELF_SIGNED_CERT_IN_CHAIN',
  config:
   { adapter: [Function: httpAdapter],
     transformRequest: { '0': [Function: transformRequest] },
     transformResponse: { '0': [Function: transformResponse] },
     timeout: 0,
     xsrfCookieName: 'XSRF-TOKEN',
     xsrfHeaderName: 'X-XSRF-TOKEN',
     maxContentLength: -1,
     validateStatus: [Function: validateStatus],
     headers:
      { Accept: 'application/json, text/plain, */*',
        'Content-Type': 'application/json;charset=utf-8',
        'kbn-xsrf': 'lollerpops',
        host: 'spicy.local:5601',
        connection: 'keep-alive',
        accept: '*/*',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36',
        referer: 'LINK REDACTED
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'en-US,en;q=0.9',
        cookie: 'io=mA-PVPjjkhwfSdHCAAAD',
        authorization: null,
        'Content-Length': 153 },
     method: 'post',
     url: 'LINK REDACTED
     responseType: 'json',
     data: '{"extended":{"es":{"filter":{"bool":{"must":[]}}}},"sheet":[".es(*)"],"time":{"from":"now-1y","to":"now","interval":"auto","timezone":"America/Phoenix"}}' },
  request:
   Writable {
     _writableState:
      WritableState {
        objectMode: false,
        highWaterMark: 16384,
        finalCalled: false,
        needDrain: false,
        ending: false,
        ended: false,
        finished: false,
        destroyed: false,
        decodeStrings: true,
        defaultEncoding: 'utf8',
        length: 0,
        writing: false,
        corked: 0,
        sync: true,
        bufferProcessing: false,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        bufferedRequest: null,
        lastBufferedRequest: null,
        pendingcb: 0,
        prefinished: false,
        errorEmitted: false,
        bufferedRequestCount: 0,
        corkedRequestsFree: [Object] },
     writable: true,
     domain: null,
     _events:
      { response: [Function: handleResponse],
        error: [Function: handleRequestError] },
     _eventsCount: 2,
     _maxListeners: undefined,
     _options:
      { protocol: 'https:',
        maxRedirects: 21,
        maxBodyLength: 10485760,
        path: '/api/timelion/run',
        method: 'post',
        headers: [Object],
        agent: undefined,
        auth: undefined,
        hostname: 'spicy.local',
        port: '5601',
        nativeProtocols: [Object],
        pathname: '/api/timelion/run' },
     _redirectCount: 0,
     _requestBodyLength: 153,
     _requestBodyBuffers: [ [Object] ],
     _onNativeResponse: [Function],
     _currentRequest:
      ClientRequest {
        domain: null,
        _events: [Object],
        _eventsCount: 6,
        _maxListeners: undefined,
        output: [],
        outputEncodings: [],
        outputCallbacks: [],
        outputSize: 0,
        writable: true,
        _last: true,
        upgrading: false,
        chunkedEncoding: false,
        shouldKeepAlive: true,
        useChunkedEncodingByDefault: true,
        sendDate: false,
        _removedConnection: false,
        _removedContLen: false,
        _removedTE: false,
        _contentLength: null,
        _hasBody: true,
        _trailer: '',
        finished: false,
        _headerSent: true,
        socket: [Object],
        connection: [Object],
        _header: 'POST /api/timelion/run HTTP/1.1\r\naccept: */*\r\nContent-Type: application/json;charset=utf-8\r\nkbn-xsrf: lollerpops\r\nhost: spicy.local:5601\r\nconnection: keep-alive\r\nuser-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36\r\nreferer: LINK REDACTED gzip, deflate, br\r\naccept-language: en-US,en;q=0.9\r\ncookie: io=mA-PVPjjkhwfSdHCAAAD\r\nauthorization: null\r\nContent-Length: 153\r\n\r\n',
        _onPendingData: [Function: noopPendingOutput],
        agent: [Object],
        socketPath: undefined,
        timeout: undefined,
        method: 'POST',
        path: '/api/timelion/run',
        _ended: false,
        res: null,
        aborted: undefined,
        timeoutCb: null,
        upgradeOrConnect: false,
        parser: null,
        maxHeadersCount: null,
        _redirectable: [Circular],
        [Symbol(outHeadersKey)]: [Object] },
     _currentUrl: 'LINK REDACTED },
  response: undefined }

[kibanamachine]

Original comment by @tsullivan:

This happens because timelion is a server-side function that internally calls the /api/timelion/run Kibana HTTP API. This call uses the axios library directly, which is not an agent that knows anything about the Kibana certificate authority.

It can be fixed in the code by making timelion a public function instead. I have tested that, and it works. However, I'd like to get @rashidkpc's input on this, because the timelion function shares a library with the esdocs function, and moving timelion to public means that library should be moved to common.

Example of a timelion API call running from Canvas in the browser:

Request payload:

{
  "extended": {
    "es": {
      "filter": {
        "bool": {
          "must": []
        }
      }
    }
  },
  "sheet": [
    ".es(index=sales, q='success:true', EMAIL REDACTED split='name:3', metric='max:metric')"
  ],
  "time": {
    "from": "now-1y",
    "to": "now",
    "interval": "auto",
    "timezone": "America/Phoenix"
  }
}

The extended property is generated by the esdocs lib.

[kibanamachine]

Original comment by @tsullivan:

I talked with @rashidkpc about this, and he is -1 for moving Timelion to be a public function. A better solution would be to not call a Kibana HTTP endpoint from the server-side, meaning that the Timelion function should use a server method that exposes the run features of Timelion.

[kibanamachine]

Original comment by @rashidkpc:

Untrusted certs are an issue all over the Elastic stack. The solution here is to use a trusted cert. Calling this a documentation bug: LINK REDACTED

[rayafratkina]

Duplicate of #26308. Closing