Test results: Load dashboards from apm-contrib. 7.x viz will only show 7.x data

[kuisathaverat]

In this test we have to check that the data show in the apm-contrib dashboards for the 7.x version only show 7.x data.

Test steps

• Prepare a couple of Docker compose files one for 6.6.1 and another for 7.0.0-rc1, to make it I have used the integration test tool compose.py
  • Run ./script/compose.py start 6.6.1 --release --with-agent-go-net-http then docker-compose -f docker-compose.yml stop
  • Rename docker-compose.yml file to docker-compose-6.6.1.yml
  • Run ./script/compose.py start 7.0.0-rc1 --bc --with-agent-go-net-http then docker-compose -f docker-compose.yml stop
  • Rename docker-compose.yml file to docker-compose-7.0.0-rc1.yml
  • edit the files to use the same Docker volume
    • Change `"path.data=/usr/share/elasticsearch/data/6.6.1" to "path.data=/usr/share/elasticsearch/data/esdata" on docker-compose-6.6.1.yml
    • Change `"path.data=/usr/share/elasticsearch/data/7.0.0" to "path.data=/usr/share/elasticsearch/data/esdata" on docker-compose-7.0.0-rc1.yml
• Start 6.6.1 docker-compose -f docker-compose-6.6.1.yml up -d
• Configure APM
• Generate some data accesing to http://localhost:8080/foo-6.6.1, http://localhost:8080/foo, and http://localhost:8080/oof
• Check that the data is show in the APM UI
• Stop 6.6.1 docker-compose -f docker-compose-6.6.0.yml stop
• Wait a couple of minutes to have a time gap between the data
• Start 7.0.0-rc1 docker-compose -f docker-compose-7.0.0-rc1.yml up -d
• Generate some data accesing to http://localhost:8080/foo-7.0.0-rc1, http://localhost:8080/foo, and http://localhost:8080/oof
• Download the dashboards from https://github.com/elastic/apm-contrib/tree/master/apm-ui/7.x
• Import the dashboard on kibana
• Check that the data show in the dashboards is only from 7.0.0-rc1 version

Issues found

• On 7.0.0-rc1, when I go to Discover it said that some shards failed and I see a bunch of the following error in the logs, the solution it is to remove the index pattern and create it again.

{"type": "server", "timestamp": "2019-03-22T10:51:24,993+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "docker-cluster", "node.name": "c95aab0bc3f6", "cluster.uuid": "uH43uCTXQNKZmC3BLzyVnw", "node.id": "Ps0q-MrPR26LYNUsgy-dag",  "message": "[2053] Failed to execute fetch phase" , 
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [c95aab0bc3f6][172.31.0.2:9300][indices:data/read/search[phase/fetch/id]]",
"Caused by: org.elasticsearch.script.ScriptException: runtime error",
"at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:94) ~[?:?]",
"at org.elasticsearch.painless.PainlessScript$Script.execute(doc['context.service.name'].value:28) ~[?:?]",
"at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitsExecute(ScriptFieldsFetchSubPhase.java:67) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:177) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:539) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:379) [elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:834) [?:?]",
"Caused by: java.lang.IllegalArgumentException: No field found for [context.service.name] in mapping with types []",
"at org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.painless.PainlessScript$Script.execute(doc['context.service.name'].value:5) ~[?:?]",
"at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitsExecute(ScriptFieldsFetchSubPhase.java:67) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:177) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:539) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:379) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0-rc1.jar:7.0.0-rc1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
"at java.lang.Thread.run(Thread.java:834) ~[?:?]"] }

• APM Error Details shows data from 6.6.1
  
• APM Errors shows data from 6.6.1
  
• APM Span Details shows data from 6.6.1
  
• APM Transactions shows data from 6.6.1
  

Dashboards without issues

APM Services show the proper data

[graphaelli]

compose.py has an --elasticsearch-data-dir option to simplify the upgrade procedure, I usually just use ''

[graphaelli]

I think it would be fair to add a major_version filter to these dashboards.