Skip to content

Commit

Permalink
Remove some white noise (#947)
Browse files Browse the repository at this point in the history
* Remove some white noise

Signed-off-by: Xiang Dai <[email protected]>

* remove white noise in genflagdocs

Signed-off-by: Xiang Dai <[email protected]>
  • Loading branch information
daixiang0 authored and GiedriusS committed Mar 21, 2019
1 parent 8e0e4dc commit 51e3672
Show file tree
Hide file tree
Showing 21 changed files with 220 additions and 217 deletions.
12 changes: 6 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ We use *breaking* word for marking changes that are not backward compatible (rel
- [#851](https://github.com/improbable-eng/thanos/pull/851) New read API endpoint for api/v1/rules and api/v1/alerts.
- [#873](https://github.com/improbable-eng/thanos/pull/873) Store: fix set index cache LRU

:warning: **WARING** :warning: #873 fix fixes actual handling of `index-cache-size`. Handling of limit for this cache was
:warning: **WARING** :warning: #873 fix fixes actual handling of `index-cache-size`. Handling of limit for this cache was
broken so it was unbounded all the time. From this release actual value matters and is extremely low by default. To "revert"
the old behaviour (no boundary), use a large enough value.

Expand Down Expand Up @@ -55,14 +55,14 @@ the old behaviour (no boundary), use a large enough value.
- [#649](https://github.com/improbable-eng/thanos/issues/649) - Fixed store label values api to add also external label values.
- [#396](https://github.com/improbable-eng/thanos/issues/396) - Fixed sidecar logic for proxying series that has more than 2^16 samples from Prometheus.
- [#732](https://github.com/improbable-eng/thanos/pull/732) - Fixed S3 authentication sequence. You can see new sequence enumerated [here](https://github.com/improbable-eng/thanos/blob/master/docs/storage.md#aws-s3-configuration)
- [#745](https://github.com/improbable-eng/thanos/pull/745) - Fixed race conditions and edge cases for Thanos Querier fanout logic.
- [#745](https://github.com/improbable-eng/thanos/pull/745) - Fixed race conditions and edge cases for Thanos Querier fanout logic.
- [#651](https://github.com/improbable-eng/thanos/issues/651) - Fixed index cache when asked buffer size is bigger than cache max size.

### Changed

- [#529](https://github.com/improbable-eng/thanos/pull/529) Massive improvement for compactor. Downsampling memory consumption was reduce to only store labels and single chunks per each series.
- Qurerier UI: Store page now shows the store APIs per component type.
- Prometheus and TSDB deps are now up to date with ~2.7.0 Prometheus version. Lot's of things has changed. See details [here #704](https://github.com/improbable-eng/thanos/pull/704) Known changes that affects us:
- Prometheus and TSDB deps are now up to date with ~2.7.0 Prometheus version. Lot's of things has changed. See details [here #704](https://github.com/improbable-eng/thanos/pull/704) Known changes that affects us:
- prometheus/prometheus/discovery/file
- [ENHANCEMENT] Discovery: Improve performance of previously slow updates of changes of targets. #4526
- [BUGFIX] Wait for service discovery to stop before exiting #4508 ??
Expand All @@ -83,11 +83,11 @@ the old behaviour (no boundary), use a large enough value.
- S3 provider:
- Added `put_user_metadata` option to config.
- Added `insecure_skip_verify` option to config.

### Deprecated

- Tests against Prometheus below v2.2.1. This does not mean *lack* of support for those. Only that we don't tests the compatibility anymore. See [#758](https://github.com/improbable-eng/thanos/issues/758) for details.

## [v0.2.1](https://github.com/improbable-eng/thanos/releases/tag/v0.2.1) - 2018.12.27

### Added
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The philosophy of Thanos and our community is borrowing much from UNIX philosoph
* Write components that work together
* e.g. blocks should be stored in native prometheus format
* Make it easy to read, write, and, run components
* e.g. reduce complexity in system design and implementation
* e.g. reduce complexity in system design and implementation

## Adding New Features / Components

Expand Down
2 changes: 1 addition & 1 deletion MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
|-----------------------|------------------------|--------------------------|------------------------------------------------------------|
| Bartłomiej Płotka | [email protected] | `@bwplotka` | [@bwplotka](https://github.com/bwplotka) |
| Dominic Green | [email protected] | `@domgreen` | [@domgreen](https://github.com/domgreen) |
| Frederic Branczyk | [email protected] | `@brancz` | [@brancz](https://github.com/brancz) |
| Frederic Branczyk | [email protected] | `@brancz` | [@brancz](https://github.com/brancz) |
| Giedrius Statkevičius | [email protected] | `@Giedrius Statkevičius` | [@GiedriusS](https://github.com/GiedriusS) |

## Storage plugins maintainers
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@

## Overview

Thanos is a set of components that can be composed into a highly available metric
system with unlimited storage capacity, which can be added seamlessly on top of existing
Prometheus deployments.
Thanos is a set of components that can be composed into a highly available metric
system with unlimited storage capacity, which can be added seamlessly on top of existing
Prometheus deployments.

Thanos leverages the Prometheus 2.0 storage format to cost-efficiently store historical metric
Thanos leverages the Prometheus 2.0 storage format to cost-efficiently store historical metric
data in any object storage while retaining fast query latencies. Additionally, it provides
a global query view across all Prometheus installations and can merge data from Prometheus
a global query view across all Prometheus installations and can merge data from Prometheus
HA pairs on the fly.

Concretely the aims of the project are:
Expand Down Expand Up @@ -65,7 +65,7 @@ Contributions are very welcome! See our [CONTRIBUTING.md](CONTRIBUTING.md) for m

## Community

Thanos is an open source project and we welcome new contributers and members
Thanos is an open source project and we welcome new contributers and members
of the community. Here are ways to get in touch with the community:

* Slack: [#thanos](https://join.slack.com/t/improbable-eng/shared_invite/enQtMzQ1ODcyMzQ5MjM4LWY5ZWZmNGM2ODc5MmViNmQ3ZTA3ZTY3NzQwOTBlMTkzZmIxZTIxODk0OWU3YjZhNWVlNDU3MDlkZGViZjhkMjc)
Expand Down
4 changes: 2 additions & 2 deletions benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,14 @@ We started a prometheus instance with thanos sidecar, scraping 50 targets every
* 0.022 average query time (min: 0.018s, max: 0.073s)
* *0.812 queries per second*

This shows an added latency of *85-150%* when using thanos query. This is more or less expected, as network operations will have to be done twice. We took a profile of thanos query while under load, finding that about a third of the time is being spent evaluating the promql queries. We are looking into including newer versions of the prometheus libraries into thanos that include optimisations to this component.
This shows an added latency of *85-150%* when using thanos query. This is more or less expected, as network operations will have to be done twice. We took a profile of thanos query while under load, finding that about a third of the time is being spent evaluating the promql queries. We are looking into including newer versions of the prometheus libraries into thanos that include optimisations to this component.

Although we have not tested federated prometheus in the same controlled environment, in theory it should incur a similar overhead, as we will still be performing two network hops.

### Store performance
To test the store component, we generated 1 year of simulated metrics (100 timeseries taking random values every 15s, a total of 210 million samples). We were able to run heavy queries to touch all 210 million of these samples, e.g. a sum over 100 timeseries takes about 34.6 seconds. Smaller queries, for example fetching 1 year of samples from a single timeseries, were able to run in about 500 milliseconds.

When enabling downsampling over these timeseries, we were able to reduce query times by over 90%.
When enabling downsampling over these timeseries, we were able to reduce query times by over 90%.

### Ingestion
To try to find the limits of a single thanos-query service, we spun up a number prometheus instances, each scraping 10 metric-producing endpoints every second. We attached a thanos-query endpoint in front of these scrapers, and ran queries that would touch fetch a most recent metric from each of them. Each metric producing endpoint would serve 100 metrics, taking random values, and the query would fetch the most recent value from each of these metrics:
Expand Down
42 changes: 21 additions & 21 deletions docs/components/bucket.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,18 +36,18 @@ Flags:
--version Show application version.
--log.level=info Log filtering level.
--log.format=logfmt Log format to use.
--gcloudtrace.project=GCLOUDTRACE.PROJECT
--gcloudtrace.project=GCLOUDTRACE.PROJECT
GCP project to send Google Cloud Trace tracings to.
If empty, tracing will be disabled.
--gcloudtrace.sample-factor=1
--gcloudtrace.sample-factor=1
How often we send traces (1/<sample-factor>). If 0 no
trace will be sent periodically, unless forced by
baggage item. See `pkg/tracing/tracing.go` for
details.
--objstore.config-file=<bucket.config-yaml-path>
--objstore.config-file=<bucket.config-yaml-path>
Path to YAML file that contains object store
configuration.
--objstore.config=<bucket.config-yaml>
--objstore.config=<bucket.config-yaml>
Alternative to 'objstore.config-file' flag. Object
store configuration in YAML.

Expand Down Expand Up @@ -86,35 +86,35 @@ Flags:
--version Show application version.
--log.level=info Log filtering level.
--log.format=logfmt Log format to use.
--gcloudtrace.project=GCLOUDTRACE.PROJECT
--gcloudtrace.project=GCLOUDTRACE.PROJECT
GCP project to send Google Cloud Trace tracings to.
If empty, tracing will be disabled.
--gcloudtrace.sample-factor=1
--gcloudtrace.sample-factor=1
How often we send traces (1/<sample-factor>). If 0 no
trace will be sent periodically, unless forced by
baggage item. See `pkg/tracing/tracing.go` for
details.
--objstore.config-file=<bucket.config-yaml-path>
--objstore.config-file=<bucket.config-yaml-path>
Path to YAML file that contains object store
configuration.
--objstore.config=<bucket.config-yaml>
--objstore.config=<bucket.config-yaml>
Alternative to 'objstore.config-file' flag. Object
store configuration in YAML.
--objstore-backup.config-file=<bucket.config-yaml-path>
--objstore-backup.config-file=<bucket.config-yaml-path>
Path to YAML file that contains object store-backup
configuration. Used for repair logic to backup blocks
before removal.
--objstore-backup.config=<bucket.config-yaml>
--objstore-backup.config=<bucket.config-yaml>
Alternative to 'objstore-backup.config-file' flag.
Object store-backup configuration in YAML. Used for
repair logic to backup blocks before removal.
-r, --repair Attempt to repair blocks for which issues were
detected
-i, --issues=index_issue... ...
-i, --issues=index_issue... ...
Issues to verify (and optionally repair). Possible
values: [duplicated_compaction index_issue
overlapped_blocks]
--id-whitelist=ID-WHITELIST ...
--id-whitelist=ID-WHITELIST ...
Block IDs to verify (and optionally repair) only. If
none is specified, all blocks will be verified.
Repeated field
Expand Down Expand Up @@ -143,18 +143,18 @@ Flags:
--version Show application version.
--log.level=info Log filtering level.
--log.format=logfmt Log format to use.
--gcloudtrace.project=GCLOUDTRACE.PROJECT
--gcloudtrace.project=GCLOUDTRACE.PROJECT
GCP project to send Google Cloud Trace tracings to.
If empty, tracing will be disabled.
--gcloudtrace.sample-factor=1
--gcloudtrace.sample-factor=1
How often we send traces (1/<sample-factor>). If 0 no
trace will be sent periodically, unless forced by
baggage item. See `pkg/tracing/tracing.go` for
details.
--objstore.config-file=<bucket.config-yaml-path>
--objstore.config-file=<bucket.config-yaml-path>
Path to YAML file that contains object store
configuration.
--objstore.config=<bucket.config-yaml>
--objstore.config=<bucket.config-yaml>
Alternative to 'objstore.config-file' flag. Object
store configuration in YAML.
-o, --output="" Optional format in which to print each block's
Expand Down Expand Up @@ -184,21 +184,21 @@ Flags:
--version Show application version.
--log.level=info Log filtering level.
--log.format=logfmt Log format to use.
--gcloudtrace.project=GCLOUDTRACE.PROJECT
--gcloudtrace.project=GCLOUDTRACE.PROJECT
GCP project to send Google Cloud Trace tracings to.
If empty, tracing will be disabled.
--gcloudtrace.sample-factor=1
--gcloudtrace.sample-factor=1
How often we send traces (1/<sample-factor>). If 0
no trace will be sent periodically, unless forced
by baggage item. See `pkg/tracing/tracing.go` for
details.
--objstore.config-file=<bucket.config-yaml-path>
--objstore.config-file=<bucket.config-yaml-path>
Path to YAML file that contains object store
configuration.
--objstore.config=<bucket.config-yaml>
--objstore.config=<bucket.config-yaml>
Alternative to 'objstore.config-file' flag. Object
store configuration in YAML.
-l, --selector=<name>="<value>" ...
-l, --selector=<name>="<value>" ...
Selects blocks based on label, e.g. '-l
key1="value1" -l key2="value2"'. All key value
pairs must match.
Expand Down
18 changes: 9 additions & 9 deletions docs/components/compact.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,38 +36,38 @@ Flags:
--version Show application version.
--log.level=info Log filtering level.
--log.format=logfmt Log format to use.
--gcloudtrace.project=GCLOUDTRACE.PROJECT
--gcloudtrace.project=GCLOUDTRACE.PROJECT
GCP project to send Google Cloud Trace tracings to.
If empty, tracing will be disabled.
--gcloudtrace.sample-factor=1
--gcloudtrace.sample-factor=1
How often we send traces (1/<sample-factor>). If 0 no
trace will be sent periodically, unless forced by
baggage item. See `pkg/tracing/tracing.go` for
details.
--http-address="0.0.0.0:10902"
--http-address="0.0.0.0:10902"
Listen host:port for HTTP endpoints.
--data-dir="./data" Data directory in which to cache blocks and process
compactions.
--objstore.config-file=<bucket.config-yaml-path>
--objstore.config-file=<bucket.config-yaml-path>
Path to YAML file that contains object store
configuration.
--objstore.config=<bucket.config-yaml>
--objstore.config=<bucket.config-yaml>
Alternative to 'objstore.config-file' flag. Object
store configuration in YAML.
--sync-delay=30m Minimum age of fresh (non-compacted) blocks before
they are being processed.
--retention.resolution-raw=0d
--retention.resolution-raw=0d
How long to retain raw samples in bucket. 0d -
disables this retention
--retention.resolution-5m=0d
--retention.resolution-5m=0d
How long to retain samples of resolution 1 (5
minutes) in bucket. 0d - disables this retention
--retention.resolution-1h=0d
--retention.resolution-1h=0d
How long to retain samples of resolution 2 (1 hour)
in bucket. 0d - disables this retention
-w, --wait Do not exit after all compactions have been processed
and wait for new work.
--block-sync-concurrency=20
--block-sync-concurrency=20
Number of goroutines to use when syncing block
metadata from object storage.

Expand Down
Loading

0 comments on commit 51e3672

Please sign in to comment.