[feature][transaction] Add a configuration to control max active transaction of coordinator #15157

congbobo184 · 2022-04-13T09:11:37Z

detail in #15133

Status: Discussion
Author: Bo Cong
Pull Request:
Mailing List discussion:
Release: 2.11

Motivation

Currently, the transaction coordinator does not limit the number of active transactions, which may cause the following problems:

A large number of active transactions will put a lot of pressure on memory
The transaction that a single TC can handle is limited, so the active transaction cannot be expanded infinitely
End transaction should wait TP or TB recover success, so a lot end request will pending in TP or TB and TC don't kown the state of the TB or TP, it will wast a lot of resource of the machine. If there have a lot of TB or TP request in pending state, it will cause the OOM

Implementation

Add config

add maxActiveTransactions into broker.conf

# The max active transactions in one transaction coordinator
maxActiveTransactionsPerCoordinator=10000

How to handle the number of active transactions reach the maxActiveTransactions?

If reach the maxActiveTransactions, return the Exception to client. It has a lot of disadvantages:

broker should add a ReachMaxActiveTxnException, if reach the max active txn exception. client need try this exception then do op. every client will handle the ReachMaxActiveTxnException.
client receive this transaction will not stop open txn, because it don't know what time the TC will be recoverd. It will retry now. When the TC can't recover, the client will keep retrying. But this op is not make sense.

Design

When this op request reach the maxActiveTransactions, coordinator don't return any response for this request. ignore this request directly. In this way, broker don't need to add any exception for this config.

Let's we can see, how does this way will affect the client?

If broker don't return the reponse for this request, the op of open txn will timeout. and in coordinator client, it has a semaphore to control the op of txn(open, add produce topic, add ack topic, end txn). In the timeout time, the coordinator client only can open the number of semaphore txns. Any other request will be block. So this design slove this two problems:

don't need to add a exception
client will not infinite retry

Worries

If you are worried that this design will affect the client-side experience, because the open transaction will always time out and other txn op will be blocked. I think your worry is superfluous, At this time, you should consider increasing the performance of the cluster or find the problematic client to repair.

flow chart

Compatibility, Deprecation, and Migration Plan

maxActiveTransactions default = 0, if maxActiveTransactions will not block open txn

Test Plan

reach maxActiveTransactions client open txn will timeout

Rejected Alternatives

If reach the maxActiveTransactions, return the Exception to client. It has a lot of disadvantages:

broker should add a ReachMaxActiveTxnException, if reach the max active txn exception. client need try this exception then do op. every client will handle the ReachMaxActiveTxnException.
client receive this transaction will not stop open txn, because it don't know what time the TC will be recoverd. It will retry now. When the TC can't recover, the client will keep retrying. But this op is not make sense.

liangyepianzhou

LGTM!

HQebupt · 2022-04-14T08:23:56Z

LGTM :)

…_active_transaction_limitation

github-actions · 2022-05-19T02:14:45Z

The pr had no activity for 30 days, mark with Stale label.

…itation # Conflicts: # conf/broker.conf # pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java # site2/docs/reference-configuration.md

…m/BewareMyPower/pulsar into bewaremypower/last-cumulative-ack # Conflicts: # pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java

…_active_transaction_limitation

...src/main/java/org/apache/pulsar/transaction/coordinator/impl/MLTransactionMetadataStore.java

gaoran10 · 2022-06-24T04:17:18Z

pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java

@@ -2592,6 +2592,12 @@ public class ServiceConfiguration implements PulsarConfiguration {
    )
    private long transactionBufferClientOperationTimeoutInMills = 3000L;

+    @FieldContext(
+            category = CATEGORY_TRANSACTION,
+            doc = "The max active transactions per transaction coordinator."


Suggested change

doc = "The max active transactions per transaction coordinator."

doc = "The max active transactions per transaction coordinator, default value 0 indicates no limit."

gaoran10 · 2022-06-24T04:18:15Z

conf/broker.conf

@@ -1432,6 +1432,9 @@ transactionBufferSnapshotMinTimeInMillis=5000
 # The max concurrent requests for transaction buffer client, default is 1000
 transactionBufferClientMaxConcurrentRequests=1000

+# The max active transactions per transaction coordinator


Suggested change

# The max active transactions per transaction coordinator

# The max active transactions per transaction coordinator, default value 0 indicates no limit.

gaoran10

LGTM

congbo added 4 commits April 13, 2022 16:43

[future][txn] Add max active txn config in transaction coordinator

056fc31

Add doc

3e5f983

Fix some code style

2e53f96

Add some test

7ad48f5

congbobo184 added area/config area/transaction component/transaction-coordinator doc-complete Your PR changes impact docs and the related docs have been already added. labels Apr 13, 2022

congbobo184 added this to the 2.11.0 milestone Apr 13, 2022

congbobo184 requested review from Technoboy-, eolivelli, codelipenghui and gaoran10 April 13, 2022 09:11

congbobo184 self-assigned this Apr 13, 2022

gaozhangmin changed the title ~~Congbobo184 add max active transaction limitation~~ [feature][transaction] Add a configuration to control max active transaction of coordinator Apr 13, 2022

RobertIndie added the type/feature The PR added a new feature or issue requested a new feature label Apr 13, 2022

liangyepianzhou approved these changes Apr 13, 2022

View reviewed changes

HQebupt approved these changes Apr 14, 2022

View reviewed changes

congbobo184 mentioned this pull request Apr 15, 2022

Max active transaction limitation for transaction coordinator streamnative/pulsar-archived#4075

Closed

Merge remote-tracking branch 'apache/master' into congbobo184_add_max…

2641c74

…_active_transaction_limitation

github-actions bot added the lifecycle/stale label May 19, 2022

congbobo184 added 5 commits May 30, 2022 18:43

Merge branch 'master' into congbobo184_add_max_active_transaction_lim…

6bfcaee

…itation # Conflicts: # conf/broker.conf # pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java # site2/docs/reference-configuration.md

fix conflic

a622555

Merge branch 'bewaremypower/last-cumulative-ack' of https://github.co…

7258151

…m/BewareMyPower/pulsar into bewaremypower/last-cumulative-ack # Conflicts: # pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java

Merge remote-tracking branch 'apache/master' into congbobo184_add_max…

a5bd06b

…_active_transaction_limitation

fix some tests

57d0c7b

gaoran10 reviewed Jun 24, 2022

View reviewed changes

fix some comment

aaa3df6

gaoran10 approved these changes Jun 24, 2022

View reviewed changes

congbobo184 merged commit 7bf7736 into apache:master Jun 27, 2022

congbobo184 mentioned this pull request Jul 15, 2022

[PIP 154] Max active transaction limitation for transaction coordinator #15133

Closed

tisonkun removed the component/transaction-coordinator label Dec 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature][transaction] Add a configuration to control max active transaction of coordinator #15157

[feature][transaction] Add a configuration to control max active transaction of coordinator #15157

congbobo184 commented Apr 13, 2022 •

edited

Loading

liangyepianzhou left a comment

HQebupt commented Apr 14, 2022

github-actions bot commented May 19, 2022

gaoran10 Jun 24, 2022 •

edited

Loading

gaoran10 Jun 24, 2022

gaoran10 left a comment

	doc = "The max active transactions per transaction coordinator."
	doc = "The max active transactions per transaction coordinator, default value 0 indicates no limit."

	# The max active transactions per transaction coordinator
	# The max active transactions per transaction coordinator, default value 0 indicates no limit.

[feature][transaction] Add a configuration to control max active transaction of coordinator #15157

[feature][transaction] Add a configuration to control max active transaction of coordinator #15157

Conversation

congbobo184 commented Apr 13, 2022 • edited Loading

Motivation

Implementation

Add config

How to handle the number of active transactions reach the maxActiveTransactions?

Design

Let's we can see, how does this way will affect the client?

Worries

flow chart

Compatibility, Deprecation, and Migration Plan

Test Plan

Rejected Alternatives

liangyepianzhou left a comment

Choose a reason for hiding this comment

HQebupt commented Apr 14, 2022

github-actions bot commented May 19, 2022

gaoran10 Jun 24, 2022 • edited Loading

Choose a reason for hiding this comment

gaoran10 Jun 24, 2022

Choose a reason for hiding this comment

gaoran10 left a comment

Choose a reason for hiding this comment

congbobo184 commented Apr 13, 2022 •

edited

Loading

gaoran10 Jun 24, 2022 •

edited

Loading