pkg/actor(ticdc): reduce metrics overhead #4585

overvenus · 2022-02-14T05:58:46Z

What problem does this PR solve?

Issue Number: close #4584

What is changed and how it works?

Reduce metrics overhead
Add actor metrics to Grafana

# Master
go test -benchmem -run='^$' -bench '^(BenchmarkPollActor)$' github.com/pingcap/tiflow/pkg/actor

goos: linux
goarch: amd64
pkg: github.com/pingcap/tiflow/pkg/actor
cpu: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
BenchmarkPollActor/BenchmarkPollActor/1_actor(s)-40               639072              1763 ns/op              48 B/op          1 allocs/op
BenchmarkPollActor/BenchmarkPollActor/2_actor(s)-40               359673              3286 ns/op              96 B/op          2 allocs/op
BenchmarkPollActor/BenchmarkPollActor/4_actor(s)-40               240338              5152 ns/op             192 B/op          4 allocs/op
BenchmarkPollActor/BenchmarkPollActor/8_actor(s)-40               119098             11157 ns/op             384 B/op          8 allocs/op
BenchmarkPollActor/BenchmarkPollActor/16_actor(s)-40               56025             23284 ns/op             769 B/op         16 allocs/op
BenchmarkPollActor/BenchmarkPollActor/32_actor(s)-40               30073             43730 ns/op            1539 B/op         32 allocs/op
BenchmarkPollActor/BenchmarkPollActor/64_actor(s)-40               14853             89433 ns/op            3079 B/op         64 allocs/op
BenchmarkPollActor/BenchmarkPollActor/128_actor(s)-40               7292            159212 ns/op            6150 B/op        128 allocs/op
BenchmarkPollActor/BenchmarkPollActor/256_actor(s)-40               3501            317086 ns/op           12292 B/op        256 allocs/op
BenchmarkPollActor/BenchmarkPollActor/512_actor(s)-40               2161            613879 ns/op           24580 B/op        512 allocs/op
BenchmarkPollActor/BenchmarkPollActor/1024_actor(s)-40               991           1336090 ns/op           49185 B/op       1024 allocs/op
BenchmarkPollActor/BenchmarkPollActor/2048_actor(s)-40               518           2675111 ns/op           98335 B/op       2048 allocs/op
BenchmarkPollActor/BenchmarkPollActor/4096_actor(s)-40               268           4717841 ns/op          196966 B/op       4096 allocs/op
BenchmarkPollActor/BenchmarkPollActor/8192_actor(s)-40               126           9745629 ns/op          393234 B/op       8192 allocs/op
BenchmarkPollActor/BenchmarkPollActor/16384_actor(s)-40               66          19152056 ns/op          786499 B/op      16384 allocs/op
BenchmarkPollActor/BenchmarkPollActor/32768_actor(s)-40               28          38723166 ns/op         1573554 B/op      32774 allocs/op
PASS
ok      github.com/pingcap/tiflow/pkg/actor     25.878s

# This PR
go test -benchmem -run='^$' -bench '^(BenchmarkPollActor)$' github.com/pingcap/tiflow/pkg/actor 

goos: linux
goarch: amd64
pkg: github.com/pingcap/tiflow/pkg/actor
cpu: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
BenchmarkPollActor/BenchmarkPollActor/1_actor(s)-40               901522              1354 ns/op              48 B/op          1 allocs/op
BenchmarkPollActor/BenchmarkPollActor/2_actor(s)-40               356815              3095 ns/op              96 B/op          2 allocs/op
BenchmarkPollActor/BenchmarkPollActor/4_actor(s)-40               251820              4874 ns/op             192 B/op          4 allocs/op
BenchmarkPollActor/BenchmarkPollActor/8_actor(s)-40               138922              9189 ns/op             384 B/op          8 allocs/op
BenchmarkPollActor/BenchmarkPollActor/16_actor(s)-40               56138             18193 ns/op             769 B/op         16 allocs/op
BenchmarkPollActor/BenchmarkPollActor/32_actor(s)-40               32737             35717 ns/op            1539 B/op         32 allocs/op
BenchmarkPollActor/BenchmarkPollActor/64_actor(s)-40               16482             67022 ns/op            3077 B/op         64 allocs/op
BenchmarkPollActor/BenchmarkPollActor/128_actor(s)-40               8764            135848 ns/op            6149 B/op        128 allocs/op
BenchmarkPollActor/BenchmarkPollActor/256_actor(s)-40               4104            257987 ns/op           12292 B/op        256 allocs/op
BenchmarkPollActor/BenchmarkPollActor/512_actor(s)-40               2449            497959 ns/op           24577 B/op        512 allocs/op
BenchmarkPollActor/BenchmarkPollActor/1024_actor(s)-40              1270            988900 ns/op           49176 B/op       1024 allocs/op
BenchmarkPollActor/BenchmarkPollActor/2048_actor(s)-40               636           2091994 ns/op           98427 B/op       2048 allocs/op
BenchmarkPollActor/BenchmarkPollActor/4096_actor(s)-40               312           3863858 ns/op          196624 B/op       4096 allocs/op
BenchmarkPollActor/BenchmarkPollActor/8192_actor(s)-40               147           7903754 ns/op          393224 B/op       8192 allocs/op
BenchmarkPollActor/BenchmarkPollActor/16384_actor(s)-40               74          15789187 ns/op          789148 B/op      16386 allocs/op
BenchmarkPollActor/BenchmarkPollActor/32768_actor(s)-40               36          33688860 ns/op         1573152 B/op      32770 allocs/op
PASS

Check List

Tests

Manual test (add detailed scripts or steps below)

Release note

None

* Reduce metrics overhead * Add actor metrics to Grafana Signed-off-by: Neil Shen <[email protected]>

ti-chi-bot · 2022-02-14T05:58:47Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

3AceShowHand
sdojjy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

sdojjy · 2022-02-14T10:28:33Z

pkg/actor/system.go

+			actorPollDuration := now().Sub(actorPollStartTime)
+			actorPollStartTime = approximateCurrentTime
+			if actorPollDuration > slowPollThreshold {
+				// Prometheus histogram is expensive, we only recrod slow poll.


fix lint

Suggested change

// Prometheus histogram is expensive, we only recrod slow poll.

// Prometheus histogram is expensive, we only record slow poll.

sdojjy

lgtm

pkg/actor/system.go

Signed-off-by: Neil Shen <[email protected]>

asddongmen · 2022-02-15T02:53:48Z

/merge

ti-chi-bot · 2022-02-15T02:53:52Z

This pull request has been accepted and is ready to merge.

Commit hash: 198fc4a

asddongmen · 2022-02-15T03:15:53Z

tools/bin/golangci-lint run --timeout 10m0s --skip-files kv_gen --skip-dirs dm,tests

[2022-02-15T03:07:10.169Z] pkg/actor/system.go:506:51: recrod is a misspelling of record (misspell)

[2022-02-15T03:07:10.169Z] // Prometheus histogram is expensive, we only recrod slow poll.

please fix this typo.

Signed-off-by: Neil Shen <[email protected]>

…ticdc into reduce-actor-metrics-overhead

overvenus · 2022-02-15T05:56:25Z

/merge

ti-chi-bot · 2022-02-15T05:56:27Z

This pull request has been accepted and is ready to merge.

Commit hash: 21a8451

codecov-commenter · 2022-02-15T06:15:24Z

Codecov Report

Merging #4585 (21a8451) into master (9607554) will decrease coverage by 0.1584%.
The diff coverage is 58.5868%.

Flag	Coverage Δ
cdc	`60.1939% <53.7001%> (+0.2716%)`	⬆️
dm	`51.4847% <60.1863%> (-0.5442%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #4585        +/-   ##
================================================
- Coverage   55.6402%   55.4817%   -0.1585%     
================================================
  Files           494        506        +12     
  Lines         61283      62927      +1644     
================================================
+ Hits          34098      34913       +815     
- Misses        23750      24523       +773     
- Partials       3435       3491        +56

overvenus · 2022-02-15T07:44:02Z

/merge

close pingcap#4584

ti-chi-bot · 2022-02-21T13:34:51Z

In response to a cherrypick label: new pull request created: #4642.

close pingcap#4584

close #4584

pkg/actor(ticdc): reduce metrics overhead

cc56d57

* Reduce metrics overhead * Add actor metrics to Grafana Signed-off-by: Neil Shen <[email protected]>

overvenus added subject/performance Denotes an issue or pull request is related to replication performance. component/metrics-logging Metrics and logging component. area/ticdc Issues or PRs related to TiCDC. labels Feb 14, 2022

ti-chi-bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 14, 2022

sdojjy reviewed Feb 14, 2022

View reviewed changes

sdojjy approved these changes Feb 14, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 14, 2022

3AceShowHand reviewed Feb 14, 2022

View reviewed changes

pkg/actor/system.go Show resolved Hide resolved

fix grafana dashboard

1239d23

Signed-off-by: Neil Shen <[email protected]>

3AceShowHand approved these changes Feb 15, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Feb 15, 2022

Merge branch 'master' into reduce-actor-metrics-overhead

198fc4a

ti-chi-bot removed the do-not-merge/needs-triage-completed label Feb 15, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 15, 2022

overvenus added 3 commits February 15, 2022 13:54

Merge branch 'master' into reduce-actor-metrics-overhead

897dccd

Signed-off-by: Neil Shen <[email protected]>

fix typo

20b4afd

Signed-off-by: Neil Shen <[email protected]>

Merge branch 'reduce-actor-metrics-overhead' of github.com:overvenus/…

21a8451

…ticdc into reduce-actor-metrics-overhead

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Feb 15, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 15, 2022

Merge branch 'master' into reduce-actor-metrics-overhead

7c6cd57

ti-chi-bot merged commit 87cfd44 into pingcap:master Feb 15, 2022

zhaoxinyu pushed a commit to zhaoxinyu/ticdc that referenced this pull request Feb 16, 2022

pkg/actor(ticdc): reduce metrics overhead (pingcap#4585)

24f6de6

close pingcap#4584

overvenus added the needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. label Feb 21, 2022

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Feb 21, 2022

pkg/actor(ticdc): reduce metrics overhead (pingcap#4585)

0b68e3b

close pingcap#4584

ti-chi-bot mentioned this pull request Feb 21, 2022

pkg/actor(ticdc): reduce metrics overhead (#4585) #4642

Merged

sdojjy mentioned this pull request Mar 29, 2022

mounter(ticdc): remove mounter worker #5003

Merged

overvenus added a commit to ti-chi-bot/tiflow that referenced this pull request Jun 23, 2022

pkg/actor(ticdc): reduce metrics overhead (pingcap#4585)

942417f

close pingcap#4584

ti-chi-bot pushed a commit that referenced this pull request Jun 23, 2022

pkg/actor(ticdc): reduce metrics overhead (#4585)

440ab55

close #4584

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/actor(ticdc): reduce metrics overhead #4585

pkg/actor(ticdc): reduce metrics overhead #4585

overvenus commented Feb 14, 2022 •

edited

Loading

ti-chi-bot commented Feb 14, 2022 •

edited

Loading

sdojjy Feb 14, 2022

sdojjy left a comment

asddongmen commented Feb 15, 2022

ti-chi-bot commented Feb 15, 2022

asddongmen commented Feb 15, 2022

overvenus commented Feb 15, 2022

ti-chi-bot commented Feb 15, 2022

codecov-commenter commented Feb 15, 2022 •

edited

Loading

overvenus commented Feb 15, 2022

ti-chi-bot commented Feb 21, 2022

	// Prometheus histogram is expensive, we only recrod slow poll.
	// Prometheus histogram is expensive, we only record slow poll.

pkg/actor(ticdc): reduce metrics overhead #4585

pkg/actor(ticdc): reduce metrics overhead #4585

Conversation

overvenus commented Feb 14, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Feb 14, 2022 • edited Loading

sdojjy Feb 14, 2022

Choose a reason for hiding this comment

sdojjy left a comment

Choose a reason for hiding this comment

asddongmen commented Feb 15, 2022

ti-chi-bot commented Feb 15, 2022

asddongmen commented Feb 15, 2022

overvenus commented Feb 15, 2022

ti-chi-bot commented Feb 15, 2022

codecov-commenter commented Feb 15, 2022 • edited Loading

Codecov Report

overvenus commented Feb 15, 2022

ti-chi-bot commented Feb 21, 2022

overvenus commented Feb 14, 2022 •

edited

Loading

ti-chi-bot commented Feb 14, 2022 •

edited

Loading

codecov-commenter commented Feb 15, 2022 •

edited

Loading