Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Reporting] Use a data stream for storage #161606

Closed
wants to merge 4 commits into from
Closed

[Reporting] Use a data stream for storage #161606

wants to merge 4 commits into from

Conversation

tsullivan
Copy link
Member

@tsullivan tsullivan commented Jul 11, 2023 •
edited
Loading

Summary

Follows #156315
Closes #161608

Changes in this PR:

  • Use the .kibana-reporting data stream in Elasticsearch.
  • No longer explicitly create the weekly index for storing reports at the time the user requests a report, or check if it exists
    • xpack.reporting.queue.indexInterval is now an unused setting.
  • Add op_type: 'create' in the call the index API, and add a (unused) @timestamp field to the document body

Checklist

Delete any items that are not applicable to this PR.

Risk Matrix

Risk Probability Severity Mitigation/Notes
Unexpected behavior may happen when the cluster contains reporting data stored previous to this change. Medium Medium Upgrade tests will verify that any combination of old reports and new reports does not create unexpected behavior.

For maintainers

@tsullivan tsullivan added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead Team:SharedUX Team label for AppEx-SharedUX (formerly Global Experience) Project:Serverless Work as part of the Serverless project for its initial release labels Jul 11, 2023
@tsullivan tsullivan self-assigned this Jul 11, 2023
@tsullivan tsullivan force-pushed the reporting/data-stream-i branch 6 times, most recently from 5c6578f to e322499 Compare July 12, 2023 00:17
@tsullivan tsullivan changed the title [Reporting] Use index template declaration in Reporting [Reporting] Use a data stream for storage Jul 13, 2023
@tsullivan tsullivan force-pushed the reporting/data-stream-i branch from e322499 to d2c1c2b Compare July 13, 2023 22:56
dakrone added a commit to dakrone/elasticsearch that referenced this pull request Jul 18, 2023
@dakrone
Install data stream template for Kibana reporting
6c8c65e 
Kibana reporting is migrating to a data stream. This commit installs an index template for the
hidden data stream, as well as updating the `kibana_system` role to allow accessing the data.

Relates to elastic/kibana#161606
@dakrone dakrone mentioned this pull request Jul 18, 2023
Merged
@tsullivan tsullivan force-pushed the reporting/data-stream-i branch 2 times, most recently from 34729e5 to f5c5fa6 Compare July 24, 2023 21:47
dakrone added a commit to elastic/elasticsearch that referenced this pull request Jul 25, 2023
@dakrone @elasticmachine
Install data stream template for Kibana reporting (#97765)
6111e95 
* Install data stream template for Kibana reporting

Kibana reporting is migrating to a data stream. This commit installs an index template for the
hidden data stream, as well as updating the `kibana_system` role to allow accessing the data.

Relates to elastic/kibana#161606

* Update docs/changelog/97765.yaml

* Undo explicit reporting role index privilege

* Add csv_contains_formulas and max_size_reached fields (and alphabetize)

* Add output.warnings field to kibana template mappings

---------

Co-authored-by: Elastic Machine <[email protected]>
@tsullivan tsullivan force-pushed the reporting/data-stream-i branch from 1f53b12 to d8f76cf Compare July 25, 2023 23:39
@kibana-ci
Copy link
Collaborator

kibana-ci commented Jul 26, 2023 •
edited
Loading

💔 Build Failed

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #14 / dashboard Reporting Dashboard Reporting Screenshots Print Layout downloads a PDF file
  • [job] [logs] FTR Configs #14 / dashboard Reporting Dashboard Reporting Screenshots Sample data from Kibana 7.6 "before all" hook for "PNG file matches the baseline image"
  • [job] [logs] FTR Configs #42 / lens app - group 6 lens reporting PNG report should be able to download report of the current visualization
  • [job] [logs] FTR Configs #42 / lens app - group 6 lens reporting should not cause PDF reports to fail
  • [job] [logs] FTR Configs #46 / maps app dashboard reporting: creates a map report PNG file matches the baseline image, using sample geo data
  • [job] [logs] FTR Configs #46 / maps app dashboard reporting: creates a map report PNG file matches the baseline image, using sample geo data
  • [job] [logs] FTR Configs #63 / Reporting API Integration Tests with Security disabled Job Listing APIs Posted CSV job is visible in the job count
  • [job] [logs] FTR Configs #63 / Reporting API Integration Tests with Security disabled Job Listing APIs Posted CSV job is visible in the job count
  • [job] [logs] FTR Configs #30 / Reporting APIs Generate CSV from SearchSource exported CSV file matches snapshot
  • [job] [logs] FTR Configs #30 / Reporting APIs Generate CSV from SearchSource exported CSV file matches snapshot
  • [job] [logs] FTR Configs #34 / reporting examples Captures PNG file matches the baseline image
  • [job] [logs] FTR Configs #34 / reporting examples Captures PNG file matches the baseline image
  • [job] [logs] FTR Configs #11 / Reporting Functional Tests with Deprecated Security configuration enabled Security with reporting_user built-in role Visualize Editor: Generate Screenshot does allow user with reporting_user role
  • [job] [logs] FTR Configs #11 / Reporting Functional Tests with Deprecated Security configuration enabled Security with reporting_user built-in role Visualize Editor: Generate Screenshot does allow user with reporting_user role
  • [job] [logs] FTR Configs #29 / Reporting Functional Tests with Security disabled Polling for jobs Displays new jobs
  • [job] [logs] FTR Configs #29 / Reporting Functional Tests with Security disabled Polling for jobs Displays new jobs
  • [job] [logs] FTR Configs #32 / Reporting Functional Tests with Security enabled Security with reporting_user built-in role Dashboard: Generate Screenshot does allow PNG generation user with reporting privileges
  • [job] [logs] FTR Configs #32 / Reporting Functional Tests with Security enabled Security with reporting_user built-in role Dashboard: Generate Screenshot does allow PNG generation user with reporting privileges
  • [job] [logs] FTR Configs #15 / reporting management app Listing of Reports Confirm single report deletion works
  • [job] [logs] FTR Configs #15 / reporting management app Listing of Reports Confirm single report deletion works
  • [job] [logs] FTR Configs #58 / Visualize Visualize Reporting Screenshots PNG reports: sample data created in 7.6 TSVB Gauge: PNG file matches the baseline image
  • [job] [logs] FTR Configs #58 / Visualize Visualize Reporting Screenshots PNG reports: sample data created in 7.6 TSVB Gauge: PNG file matches the baseline image

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
reporting 90 91 +1

History

  • 💔 Build #144502 failed 1f53b12827ee607accc4f3d4b0bfba909de994a4
  • 💔 Build #144484 failed f5c5fa62b1e0ff4ccf9bfa43df89c66cc1ba0c2e
  • 💔 Build #143403 failed 34729e5359d0094da2914a7a2ac3e6943b21c29b
  • 💔 Build #142417 failed d2c1c2b909f0106ed8e699e2b119301e38cb7728

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @tsullivan

@tsullivan tsullivan mentioned this pull request Jan 31, 2024
Merged
7 tasks
@tsullivan
Copy link
Member Author

The baseline in this PR is too old. Closing this in favor of: #176022

@tsullivan tsullivan closed this Jan 31, 2024
@tsullivan tsullivan deleted the reporting/data-stream-i branch September 5, 2024 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@tsullivan tsullivan

Labels
(Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead Project:Serverless Work as part of the Serverless project for its initial release Team:SharedUX Team label for AppEx-SharedUX (formerly Global Experience)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Reporting] Use index templates and a data stream
2 participants
@tsullivan @kibana-ci