metrics: add data flow metrics #2763

overvenus · 2021-09-07T13:58:48Z

What problem does this PR solve?

Add data flow metrics.

Also, this PR adds chechkpoint derivative and TiKV traffics

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update key monitor metrics in both TiCDC document and official document

Release note

None

Signed-off-by: Neil Shen <[email protected]>

ti-chi-bot · 2021-09-07T13:58:49Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

3AceShowHand
amyangfei

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

3AceShowHand · 2021-09-09T02:48:55Z

cdc/entry/mounter.go

@@ -138,6 +139,7 @@ func (m *mounterImpl) codecWorker(ctx context.Context, index int) error {
 		pEvent.RawKV.OldValue = nil
 		pEvent.PrepareFinished()
 		metricMountDuration.Observe(time.Since(startTime).Seconds())
+		metricTotalRows.Add(1)


how about using Inc(), and shall we delete this metrics when mounter close.

3AceShowHand · 2021-09-09T02:50:44Z

cdc/processor/pipeline/sink.go

@@ -150,6 +151,7 @@ func (n *sinkNode) flushSink(ctx pipeline.NodeContext, resolvedTs model.Ts) (err

 func (n *sinkNode) emitEvent(ctx pipeline.NodeContext, event *model.PolymorphicEvent) error {
 	if event == nil || event.Row == nil {
+		log.Warn("skip emit empty rows", zap.Reflect("event", event))


zap.Any is preferred than zap.Reflect

3AceShowHand · 2021-09-09T02:53:03Z

cdc/sink/manager.go

@@ -43,16 +44,22 @@ type Manager struct {
 	flushing int64

 	drawbackChan chan drawbackMsg
+
+	metricsTableSinkTotalRows prometheus.Counter


shall we delete this when closing the sink manager?

3AceShowHand · 2021-09-09T02:53:31Z

cdc/sink/manager.go

@@ -237,6 +245,7 @@ func (b *bufferSink) run(ctx context.Context, errCh chan error) {
 	metricFlushDuration := flushRowChangedDuration.WithLabelValues(advertiseAddr, changefeedID, "Flush")
 	metricEmitRowDuration := flushRowChangedDuration.WithLabelValues(advertiseAddr, changefeedID, "EmitRow")
 	metricBufferSize := bufferChanSizeGauge.WithLabelValues(advertiseAddr, changefeedID)
+	metricTotalRows := bufferSinkTotalRowsCountCounter.WithLabelValues(advertiseAddr, changefeedID)


Signed-off-by: Neil Shen <[email protected]>

amyangfei · 2021-09-09T02:48:08Z

cdc/sink/metrics.go

+			Namespace: "ticdc",
+			Subsystem: "sink",
+			Name:      "table_sink_total_rows_count",
+			Help:      "total count of rows",


can this help text be more detailed

amyangfei · 2021-09-09T02:48:12Z

cdc/sink/metrics.go

+			Namespace: "ticdc",
+			Subsystem: "sink",
+			Name:      "buffer_sink_total_rows_count",
+			Help:      "total count of rows",


amyangfei · 2021-09-09T03:05:39Z

metrics/grafana/ticdc.json

+            {
+              "alias": "/.*-rpc/",
+              "yaxis": 2
+            }


Is this useless

No, it make .*-rpc aligned to y axis.

amyangfei · 2021-09-09T03:06:02Z

metrics/grafana/ticdc.json

+            {
+              "alias": "/.*processor.*/",
+              "yaxis": 2
+            }


amyangfei · 2021-09-09T03:07:04Z

CI failed

Signed-off-by: Neil Shen <[email protected]>

cdc/sink/metrics.go

Signed-off-by: Neil Shen <[email protected]>

…into metrics/dataflow

amyangfei · 2021-09-13T10:36:01Z

/merge

ti-chi-bot · 2021-09-13T10:36:04Z

This pull request has been accepted and is ready to merge.

Commit hash: e5f7a60

overvenus · 2021-09-14T06:35:41Z

/run-kafka-tests
/run-integration-tests

amyangfei · 2021-09-14T08:02:00Z

/run-kafka-tests
/run-integration-tests

ti-chi-bot · 2021-09-14T16:15:59Z

@overvenus: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

overvenus · 2021-09-15T06:24:06Z

/run-integration-tests

overvenus · 2021-09-15T07:10:34Z

/run-integration-tests

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2021-09-15T07:39:17Z

In response to a cherrypick label: new pull request created: #2828.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2021-09-15T07:39:53Z

In response to a cherrypick label: new pull request created: #2829.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2021-09-15T07:40:29Z

In response to a cherrypick label: new pull request created: #2830.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2021-09-15T07:41:05Z

In response to a cherrypick label: new pull request created: #2831.

overvenus added 2 commits September 7, 2021 21:44

mounter, sink, pipeline: record total rows

f3eab14

Signed-off-by: Neil Shen <[email protected]>

metrics: add data flow metrics

b647b42

Signed-off-by: Neil Shen <[email protected]>

ti-chi-bot added the release-note-none Denotes a PR that doesn't merit a release note. label Sep 7, 2021

ti-chi-bot requested review from amyangfei and liuzix September 7, 2021 13:58

ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 7, 2021

3AceShowHand self-requested a review September 9, 2021 02:39

Merge branch 'master' into metrics/dataflow

de6638d

3AceShowHand reviewed Sep 9, 2021

View reviewed changes

sink: fix compile

3227eaf

Signed-off-by: Neil Shen <[email protected]>

amyangfei reviewed Sep 9, 2021

View reviewed changes

overvenus added 6 commits September 9, 2021 13:57

metrics: add puller total events

5b99979

Signed-off-by: Neil Shen <[email protected]>

metrics: correct legends

8582344

Signed-off-by: Neil Shen <[email protected]>

address comments

85232c3

Signed-off-by: Neil Shen <[email protected]>

metrics: display bottomk 10 tables and processors

3bfa423

Signed-off-by: Neil Shen <[email protected]>

address lints

c1fd777

Signed-off-by: Neil Shen <[email protected]>

Merge branch 'master' into metrics/dataflow

5e56272

overvenus commented Sep 10, 2021

View reviewed changes

cdc/sink/metrics.go Outdated Show resolved Hide resolved

overvenus added 2 commits September 10, 2021 23:12

metrics: adjust checkpoint derivative

6474837

Signed-off-by: Neil Shen <[email protected]>

Merge branch 'metrics/dataflow' of https://github.com/overvenus/ticdc …

b353d63

…into metrics/dataflow

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 13, 2021

Merge branch 'master' into metrics/dataflow

24f99a0

ti-chi-bot added 3 commits September 14, 2021 20:35

Merge branch 'master' into metrics/dataflow

d160c6f

Merge branch 'master' into metrics/dataflow

eeebf8f

Merge branch 'master' into metrics/dataflow

70aa404

ti-chi-bot merged commit 06a674b into pingcap:master Sep 15, 2021

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Sep 15, 2021

This is an automated cherry-pick of pingcap#2763

fc88f90

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Sep 15, 2021

metrics: add data flow metrics (#2763) #2828

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Sep 15, 2021

This is an automated cherry-pick of pingcap#2763

fd46fc3

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Sep 15, 2021

metrics: add data flow metrics (#2763) #2829

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Sep 15, 2021

This is an automated cherry-pick of pingcap#2763

bb0afc8

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Sep 15, 2021

metrics: add data flow metrics (#2763) #2830

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Sep 15, 2021

This is an automated cherry-pick of pingcap#2763

66b1999

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Sep 15, 2021

metrics: add data flow metrics (#2763) #2831

Merged

nongfushanquan added the cherry-pick-approved Cherry pick PR approved by release team. label Sep 15, 2021

ti-chi-bot added a commit that referenced this pull request Sep 15, 2021

metrics: add data flow metrics (#2763) (#2829)

756dfc9

ti-chi-bot added a commit that referenced this pull request Sep 16, 2021

metrics: add data flow metrics (#2763) (#2830)

2f09673

ti-chi-bot added a commit that referenced this pull request Sep 16, 2021

metrics: add data flow metrics (#2763) (#2831)

cc8157d

ti-chi-bot added a commit that referenced this pull request Sep 17, 2021

metrics: add data flow metrics (#2763) (#2828)

667c1ff

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metrics: add data flow metrics #2763

metrics: add data flow metrics #2763

overvenus commented Sep 7, 2021 •

edited

Loading

ti-chi-bot commented Sep 7, 2021 •

edited

Loading

3AceShowHand Sep 9, 2021

3AceShowHand Sep 9, 2021

3AceShowHand Sep 9, 2021

3AceShowHand Sep 9, 2021

amyangfei Sep 9, 2021

amyangfei Sep 9, 2021

amyangfei Sep 9, 2021

overvenus Sep 9, 2021

amyangfei Sep 9, 2021

amyangfei commented Sep 9, 2021

amyangfei commented Sep 13, 2021

ti-chi-bot commented Sep 13, 2021

overvenus commented Sep 14, 2021

amyangfei commented Sep 14, 2021

ti-chi-bot commented Sep 14, 2021

overvenus commented Sep 15, 2021

overvenus commented Sep 15, 2021

ti-chi-bot commented Sep 15, 2021

ti-chi-bot commented Sep 15, 2021

ti-chi-bot commented Sep 15, 2021

ti-chi-bot commented Sep 15, 2021

metrics: add data flow metrics #2763

metrics: add data flow metrics #2763

Conversation

overvenus commented Sep 7, 2021 • edited Loading

What problem does this PR solve?

Release note

ti-chi-bot commented Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amyangfei commented Sep 9, 2021

amyangfei commented Sep 13, 2021

ti-chi-bot commented Sep 13, 2021

overvenus commented Sep 14, 2021

amyangfei commented Sep 14, 2021

ti-chi-bot commented Sep 14, 2021

overvenus commented Sep 15, 2021

overvenus commented Sep 15, 2021

ti-chi-bot commented Sep 15, 2021

ti-chi-bot commented Sep 15, 2021

ti-chi-bot commented Sep 15, 2021

ti-chi-bot commented Sep 15, 2021

overvenus commented Sep 7, 2021 •

edited

Loading

ti-chi-bot commented Sep 7, 2021 •

edited

Loading