Updated TSDB documentation with additional details

[agithomas]

Type of change

• Enhancement

What does this PR do?

Added useful information for integration developers when working with TSDB.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Not applicable

Related issues

• Closes [TSDB] Update TSDB guidelines document for Integration developers #5705

Screenshots

Not applicable

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-08-23T09:04:08.003+0000

• Duration: 107 min 9 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4920
Skipped	10
Total	4930

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (453/453)	💚
Files	96.65% (779/806)	👎 -3.35
Classes	96.65% (779/806)	👎 -3.35
Methods	91.863% (7530/8197)	👎 -8.137
Lines	88.319% (170879/193480)	👎 -11.681
Conditionals	100.0% (0/0)	💚

[agithomas]

/test

[lalit-satapathy]

LGTM

[lalit-satapathy]

@agithomas, Can this PR be merged?

[agithomas]

@agithomas, Can this PR be merged?

@lalit-satapathy , @jsoriano can you please approve the PR?

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[agithomas]

/test

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[jsoriano]

/test

[jsoriano]

@agithomas @lalit-satapathy should we merge this?

[lalit-satapathy]

There may be some other changes required, @agithomas do we need to add any others update here to make it up-to-date? Let's keep this doc up-to-date.

[agithomas]

@lalit-satapathy , can you do a re-review ? I have updated the guidelines with the most recent guidelines.

[lalit-satapathy]

I think we need to re-look into these two sentences:

• A field having type flattened cannot be selected as a dimension field and that paragraph.

• An enhancement request for Kibana is created to indicate the write index. Until then, refer to the index.time_series.start_time of indices and compare with the current time to identify the write index.

At the end, Can we also refer to a cloud integration also?

[agithomas]

At the end, Can we also refer to a cloud integration also?

Added link to AWS Redshift integration at the end.

[agithomas]

@lalit-satapathy , are you ok with the documentation so that I can merge?

[constanca-m]

Can you refer the reader to the testing section and link the TSDB migration test kit?

[agithomas]

@constanca-m , can we take that as a separate update to this document? May be you can add that part?

[constanca-m]

Git doesn't let me comment on lines that were not changed...

Additional comments:

1. When a package upgrade with the above mentioned change is applied, the changes are made only on the index template. This means, the user need to wait until index.time_series.end_time of the current write index before seeing the change, following a package upgrade.

Are we sure of this? I only upgrade the package and the automatic rollover happens. The only instance that did not happen was when I modified the look_ahead_time field.

2. Known issues section suggestion: Let's just reference the meta issue to avoid having to maintain this document.

3. Reference to existing package already migrated section: Can we only add the meta issue as well for this?

4. Testing section: Please let's just link the TSDB testing kit, this way of testing is not our current way of doing it.

5. Best practices section:

Use Lens as the preferred visualisation type.

Is this related to TSDB?

[agithomas]

Git doesn't let me comment on lines that were not changed...

Additional comments:

1. When a package upgrade with the above mentioned change is applied, the changes are made only on the index template. This means, the user need to wait until index.time_series.end_time of the current write index before seeing the change, following a package upgrade.

Are we sure of this? I only upgrade the package and the automatic rollover happens. The only instance that did not happen was when I modified the look_ahead_time field.

It was the case earlier. I shall re-test this to validate the relevancy of this point.

2. Known issues section suggestion: Let's just reference the meta issue to avoid having to maintain this document.
3. Reference to existing package already migrated section: Can we only add the meta issue as well for this?
   Already present. Please refer the section "Other known issues". It will take to the meta issue.
4. Testing section: Please let's just link the TSDB testing kit, this way of testing is not our current way of doing it.

@constanca-m , you may add this part in a separate PR.

5. Best practices section:

Use Lens as the preferred visualisation type.

Is this related to TSDB?

Yes, it is. Some of the errors associated with the usage of counter type can only be seen when using Lens visualistion.

[agithomas]

4. Testing section: Please let's just link the TSDB testing kit, this way of testing is not our current way of doing it.

I do not think that there exists a fault in comparing the records, especially when the comparison is easy. Record count based check still happens. TSDB testkit can be a supplementary solution.

[agithomas]

@constanca-m , i have addressed all the review comments, except those related to test kit. Can the change be merged?

[constanca-m]

Yes! I will likely open a PR then to include the testing kit, but for now the changes here seem enough.

[agithomas]

/test