Merge r190 into out of order #2140
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o binaries. (#1759) * Extend Dockerfiles to support multiarch builds for all Go binaries. By calling any of make push-multiarch-./cmd/metaconvert/.uptodate make push-multiarch-./cmd/mimir/.uptodate make push-multiarch-./cmd/query-tee/.uptodate make push-multiarch-./cmd/mimir-continuous-test/.uptodate make push-multiarch-./cmd/mimirtool/.uptodate make push-multiarch-./operations/mimir-rules-action/.uptodate Signed-off-by: Peter Štibraný <[email protected]>
* Update to latest dskit and memberlist fork Fixes #1743 Signed-off-by: Nick Pillitteri <[email protected]> * Update changelog Signed-off-by: Nick Pillitteri <[email protected]>
Signed-off-by: Mauro Stettler <[email protected]>
* mimirtool config: Add more retained old defaults The following parameters have their old defaults retained even when `--update-defaults` is used with `mimirtool config covert`: * `activity_tracker.filepath` * `alertmanager.data_dir` * `blocks_storage.filesystem.dir` * `compactor.data_dir` * `ruler.rule_path` * `ruler_storage.filesystem.dir` * `graphite.querier.schemas.backend` (only in GEM) These are filepaths for which the new defaults don't make more sense than the old ones. In fact updating these can lead to subpar migration experience because components start using directories that don't exist. Because activity_tracker.filepath changed its name since cortex the tests needed to allow for differentiating old common options and new ones. This is something that was already there for GEM and was added for cortex/mimir too. Signed-off-by: Dimitar Dimitrov <[email protected]> * Update CHANGELOG.md Signed-off-by: Dimitar Dimitrov <[email protected]>
* dashboards: add flag to skip gateway The gateway component seems to be an enterprise component, so groups that aren't running enterprise shouldn't need the empty panels and rows in their dashboards. This patch adds a flag to drop gateway-related widgets from the mixin dashboards. Signed-off-by: Josh Carp <[email protected]> * Update CHANGELOG.md Co-authored-by: Marco Pracucci <[email protected]>
* Gracefully shutdown querier when using query-scheduler Signed-off-by: Marco Pracucci <[email protected]> * Fixed comment Signed-off-by: Marco Pracucci <[email protected]> * Added TestQueuesOnTerminatingQuerier Signed-off-by: Marco Pracucci <[email protected]> * Commented executionContext Signed-off-by: Marco Pracucci <[email protected]> * Added CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]> * Update pkg/querier/worker/util.go Co-authored-by: Peter Štibraný <[email protected]> * Fixed typo in suggestion Signed-off-by: Marco Pracucci <[email protected]> * Removed superfluous time sensitive assertion Signed-off-by: Marco Pracucci <[email protected]> * Commented newExecutionContext() Signed-off-by: Marco Pracucci <[email protected]> Co-authored-by: Peter Štibraný <[email protected]>
* Graceful shutdown querier with not using query-scheduler Signed-off-by: Marco Pracucci <[email protected]> * Updated CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]> * Improved comment Signed-off-by: Marco Pracucci <[email protected]> * Refactoring Signed-off-by: Marco Pracucci <[email protected]>
* Increase mimir-continuous-test query timeout from 30s to 60 Signed-off-by: Marco Pracucci <[email protected]> * Added PR number to CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]>
* Increased default -tests.run-interval from 1m to 5m Signed-off-by: Marco Pracucci <[email protected]> * Added PR number to CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]>
* Fix flaky tests on querier graceful shutdown Signed-off-by: Marco Pracucci <[email protected]> * Remove spurious newline Signed-off-by: Marco Pracucci <[email protected]>
* Update build-image to use golang:1.17.8-bullseye, and add skopeo to build image. Skopeo will be used in subsequent PR to push multiarch images. Signed-off-by: Peter Štibraný <[email protected]> * Update build image. Use ubuntu-latest for workflow steps. Signed-off-by: Peter Štibraný <[email protected]>
* Publish multiarch images. Signed-off-by: Peter Štibraný <[email protected]> * Tag with extra tag, if pushing tagged commit or release. Signed-off-by: Peter Štibraný <[email protected]> * Split building of docker images and archiving them into tar. Signed-off-by: Peter Štibraný <[email protected]> * When tagging with test, use --all. Signed-off-by: Peter Štibraný <[email protected]> * Only run deploy step on tags or weekly release branches. Signed-off-by: Peter Štibraný <[email protected]> * Don't tag with test anymore. Signed-off-by: Peter Štibraný <[email protected]> * Address review feedback. Signed-off-by: Peter Štibraný <[email protected]> * Fix license check. Signed-off-by: Peter Štibraný <[email protected]>
When using `K6_HA_REPLICAS > 1`, Mimir will accept all HTTP calls but a
part of those call will receive a status code `202`. The following
commit makes this status code as expected otherwise user receive the
following error:
```
reads_inat write (file:///.../mimir-k6/load-testing-with-k6.js:254:8(137))
reads_inat native executor=ramping-arrival-rate scenario=writing_metrics source=stacktrace
ERRO[0015] GoError: ERR: write failed. Status: 202. Body: replicas did not mach, rejecting sample: replica=replica_1, elected=replica_0
```
At the end of the benchmark summary display errors:
```
✗ write worked
↳ 20% — ✓ 23 / ✗ 92
```
Example of load testing:
```shell
./k6 run load-testing-with-k6.js \
-e K6_SCHEME="https" \
-e K6_WRITE_HOSTNAME="${mimir}" \
-e K6_READ_HOSTNAME="${mimir}" \
-e K6_USERNAME="${user}" \
-e K6_WRITE_TOKEN="${password}" \
-e K6_READ_TOKEN="${password}" \
-e K6_HA_CLUSTERS="1" \
-e K6_HA_REPLICAS="3" \
-e K6_DURATION_MIN="5"
```
Signed-off-by: Wilfried Roset <[email protected]>
* implement read v2 * updated CHANGELOG.md * extend maxBytesInFram comment. * addressed PR feedback * addressed PR feedback * addressed PR feedback * use indexed xor chunk function to assert stream remote read tests * updated CHANGELOG.md Co-authored-by: Miguel Ángel Ortuño <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
…e is 0 (#1783) Signed-off-by: Marco Pracucci <[email protected]>
* Print version+arch of Mimir loaded to Docker. Signed-off-by: Peter Štibraný <[email protected]> * Use debug log for distributor. Signed-off-by: Peter Štibraný <[email protected]>
…ortex_distributor_ingester_query_failures_total (#1797) * Remove unused metrics cortex_distributor_ingester_queries_total and cortex_distributor_ingester_query_failures_total Signed-off-by: Marco Pracucci <[email protected]> * Remove unused fields Signed-off-by: Marco Pracucci <[email protected]>
* Added options support to SendSumOfCountersPerUser() Signed-off-by: Marco Pracucci <[email protected]> * Renamed SkipZeroValueMetrics() to WithSkipZeroValueMetrics() Signed-off-by: Marco Pracucci <[email protected]>
… to let people install both while migrating from Cortex to Mimir (#1801) Signed-off-by: Marco Pracucci <[email protected]>
…1808) Signed-off-by: Marco Pracucci <[email protected]>
Allow customizing mimir cli flags per zone for the store gateway. Copied the same solution as we have for ingesters. Signed-off-by: György Krajcsovits <[email protected]>
…n the ring (#1806) * Add protection to store-gateway to not drop all blocks if unhealthy in the ring Signed-off-by: Marco Pracucci <[email protected]> * Added CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]> * Update CHANGELOG.md Co-authored-by: Peter Štibraný <[email protected]> Co-authored-by: Peter Štibraný <[email protected]>
…tor_ingester_append_failures_total unused metrics (#1799) Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
* Extract and test TracerTransport functionality We need to use a TracerTransport in mimir-continous-test. We have that in the frontend package, but I don't want to import frontend from the mimir-continous-test, so we extract it to util/instrumentation. Signed-off-by: Oleg Zaytsev <[email protected]> * Set up global tracer in mimir-continuous-test Signed-off-by: Oleg Zaytsev <[email protected]> * Add tracing to the client and spans to the tests Signed-off-by: Oleg Zaytsev <[email protected]> * Add jaeger-mixin to mimir-continuous test container Signed-off-by: Oleg Zaytsev <[email protected]> * make license Signed-off-by: Oleg Zaytsev <[email protected]> * Add traces to the write path Signed-off-by: Oleg Zaytsev <[email protected]> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <[email protected]>
* Removed unused Info() and advLabelSets from BucketStore Signed-off-by: Marco Pracucci <[email protected]> * Removed unused FilterConfig from BucketStore Signed-off-by: Marco Pracucci <[email protected]> * Removed unused relabelConfig from store-gateway tests Signed-off-by: Marco Pracucci <[email protected]> * Removed unused function expectedTouchedBlockOps() Signed-off-by: Marco Pracucci <[email protected]> * Removed unused recorder from BucketStore tests Signed-off-by: Marco Pracucci <[email protected]> * go mod vendor Signed-off-by: Marco Pracucci <[email protected]>
* Helm: Add golden-record build script * Helm: add test-values golden record * Add PR check for `check-helm-tests` * Add Helm setup to lint-helm action * Update generated helm tests * Fix bash linting * Update contribution guidelines * Update generated helm manifests * Helm: fix kube version Set kubeVersionOverride to generate PodDisruptionBudget API version consistently. When I ran the test, I got a diff, because my k8s is newer (1.23). Signed-off-by: György Krajcsovits <[email protected]> * Update operations/helm/tests/build.sh Co-authored-by: Dimitar Dimitrov <[email protected]> Co-authored-by: György Krajcsovits <[email protected]>
… indexheader reads. (#2019) Introduces a new experimental configuration option (`-blocks-storage.bucket-store.index-header.map-populate-enabled`). This enables the use of the `MAP_POPULATE` flag when `mmap`-ing index-header files in the store-gateway. What this flag does is advise the kernel to (synchronously) pre-fault all pages in the memory region, loading them into the file system cache. Why is this a good idea? - The initial read process of the index-header files has shown to cause hangups in the store-gateway. - By using this option, I/O is done in the mmap() syscall, which the Go scheduler can cope with. - We reduce the likelyhood of Goroutines getting stalled in major page faults. - The initial read process walks the entire file anyway, so we are not doing any more I/O. - It's a very low risk change compared to re-writing the BinaryReader (work in progress). Why is this not perfect? - The Kernel does not guarantee the pages will stay in memory, so we are only reducing the probability of major page faults. Rationale about the implementation: - I have copied the mmap utilities from Prometheus as a temporary measure, for the sake of evaluating this change.
Signed-off-by: Peter Štibraný <[email protected]>
* Update Prometheus with async chunk mapper changes. Included changes: grafana/mimir-prometheus#131 grafana/mimir-prometheus#247 These result is lower memory usage by chunk mapper. Signed-off-by: Peter Štibraný <[email protected]>
* Fix ruler config in getting started guide Signed-off-by: Marco Pracucci <[email protected]> * Added CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]>
A previous change (#2019) assumed MAP_POPULATE was available on Darwin. This fixes the build.
…#1949) * mixin: adapt alerts/playbooks to have into consideration ruler query path components. Signed-off-by: Miguel Ángel Ortuño <[email protected]> * applied PR suggestion Signed-off-by: Miguel Ángel Ortuño <[email protected]> * applied PR suggestion Signed-off-by: Miguel Ángel Ortuño <[email protected]> * restored ruler missed evaluations alert Signed-off-by: Miguel Ángel Ortuño <[email protected]> * updated CHANGELOG.md Signed-off-by: Miguel Ángel Ortuño <[email protected]>
* Return and log detailed services information on /ready This helps debug starting services more easily. Signed-off-by: Dimitar Dimitrov <[email protected]> * Only return non-running services Signed-off-by: Dimitar Dimitrov <[email protected]>
…2009) * add validation.RateLimited to error catalogue * Add validation.TooManyHAClusters to error catalogue * update docs * Apply suggestions from code review Co-authored-by: Marco Pracucci <[email protected]> * improve new MessageWithLimitConfig and add tests * Apply suggestions from code review Co-authored-by: Marco Pracucci <[email protected]> * Update from changes in code review Co-authored-by: Marco Pracucci <[email protected]>
* Add Patrick Oyarzun as Team Member * Update MAINTAINERS.md
* mimir-continuous-test: Add smoke test mode * Add PR number to CHANGELOG * Update error assertions in write_read_series_test * Fix doc formatting * Address PR feedback * Fix goimports formatting
Signed-off-by: Marco Pracucci <[email protected]>
* Make MessageWithLimitConfig accept multiple flags * Add tenant string in per-tenant error labels * Revert "Add tenant string in per-tenant error labels" This reverts commit 758ef72. * rename error too-many-ha-clusters
* ruler: report failed eval on any 5xx status Signed-off-by: Miguel Ángel Ortuño <[email protected]> * addressed PR suggestion Signed-off-by: Miguel Ángel Ortuño <[email protected]>
#2065) * Making active series custom tracker configs marshalable for runtime config * Adding unit test
Signed-off-by: Marco Pracucci <[email protected]>
* Update dskit * CHANGELOG.md
* Make namespace in build-helm-tests explicit * Use a non-default namespace Signed-off-by: Dimitar Dimitrov <[email protected]>
* Helm: release weekly no. 190 * Helm: Bump image versions for weekly Signed-off-by: György Krajcsovits <[email protected]>
During a rule group evaluation, if the remote query never completes for any reason, then the rule group will become stuck and never evaluate again.
…ut-of-order Signed-off-by: Jesus Vazquez <[email protected]>
jesusvazquez
force-pushed
the
jvp/merge-r190-into-out-of-order
branch
from
0373e6e to
465f53e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to update the out of order branch to
r190so we build an image on top of it. That way we can have r190+OOO and test that in dev until we deploy it to mimir-dedicated-20.cc @treid314 you're on PTO today but we talked about this on Friday.