De-flake MSC3030 backfill test with Synapse worker replication #492

MadLittleMods · 2022-10-06T07:06:53Z

De-flake MSC3030 backfill test with Synapse worker replication

Spawning from matrix-org/synapse#14028 (comment)

CI failure: https://github.com/matrix-org/synapse/actions/runs/3182998161/jobs/5189731097#step:6:15343 (from discussion)

WORKERS=1 POSTGRES=1 COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestJumpToDateEndpoint/parallel/federation/can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint

Why did this test fail with workers before?

The reason they failed before was because we're fighting against stale caches across workers racing while waiting for invalidation, see matrix-org/synapse#14211

Here is what happens:

serverB has event1 stored as an outlier from previous requests (specifically from MSC3030 jump to date pulling in a missing prev_event after backfilling)

Client on serverB calls /messages?dir=b

serverB:client_reader1 accepts the request and drives things

serverB:client_reader1 has some backward extremities in range and requests /backfill from serverA

serverB:client_reader1 processes the events from backfill including event1 and puts them in the _event_persist_queue

serverB:master picks up the events from the _event_persist_queue and persists them to the database, de-outliers event1 and invalidates its own cache and sends them over replication

serverB:client_reader1 starts assembling the /messages response and gets event1 out of the stale cache still as an outlier

serverB:client_reader1 responds to the /messages request without event1 because outliers are filtered out

serverB:client_reader1 finally gets the replication data and invalidates its own cache for event1 (too late, we already got the events from the stale cache and responded)

Dev notes

Complement test run failure with relevant debug logs

msc3030_test.go:225: MatchResponse JSONCheckOff(chunk): did not see items: [$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs] - http://localhost:62409/_matrix/client/r0/rooms/%21iXKoBLEwXgfsObqmow:hs1/messages?dir=b&from=t7--8_0_0_0_0_0_0_0_0&limit=100 => {"chunk":[{"content":{"body":"Message B","msgtype":"m.text"},"origin_server_ts":1665035839169,"room_id":"!iXKoBLEwXgfsObqmow:hs1","sender":"@alice:hs1","type":"m.room.message","unsigned":{"age":1454},"event_id":"$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8","user_id":"@alice:hs1","age":1454}],"start":"t7--8_0_0_0_0_0_0_0_0","end":"t1--3_0_0_0_0_0_0_0_0"}



## hs2


master | 2022-10-06 05:57:20,747 - synapse.storage.controllers.persist_events - 395 - INFO - GET-4-$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - persist_events event_ids=['$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs'] obackfilled=True
master | 2022-10-06 05:57:20,748 - synapse.storage.controllers.persist_events - 559 - INFO - persist_events-5 - _persist_event_batch events=[('$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', True)]
master | 2022-10-06 05:57:20,754 - synapse.storage.databases.main.events - 1461 - INFO - persist_events-5 - _store_event_txn events=[('$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', True)]

client_reader1 | 2022-10-06 05:57:20,942 - synapse.handlers.federation - 258 - INFO - GET-5 - _maybe_backfill_inner: backwards_extremities=[_BackfillPoint(event_id='$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', depth=7, type=<_BackfillPointType.BACKWARDS_EXTREMITY: 1>)] insertion_events_to_be_backfilled=[]
client_reader1 | 2022-10-06 05:57:20,945 - synapse.storage.databases.main.events_worker - 610 - INFO - GET-5 - _get_events_from_cache_or_db: looking for event_ids={'$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8'}
client_reader1 | 2022-10-06 05:57:20,945 - synapse.storage.databases.main.events_worker - 621 - INFO - GET-5 - _get_events_from_cache_or_db: AFTER _get_events_from_local_cache events=[('$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8', False)]
client_reader1 | 2022-10-06 05:57:20,946 - synapse.storage.databases.main.events_worker - 610 - INFO - GET-5 - _get_events_from_cache_or_db: looking for event_ids={'$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo'}
client_reader1 | 2022-10-06 05:57:20,946 - synapse.storage.databases.main.events_worker - 621 - INFO - GET-5 - _get_events_from_cache_or_db: AFTER _get_events_from_local_cache events=[('$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo', True)]
federation_sender1 | 2022-10-06 05:57:20,955 - synapse.http.matrixfederationclient - 299 - INFO - federation_transaction_transmission_loop-0 - {PUT-O-1} [hs1] Completed request: 200 OK in 0.38 secs, got 11 bytes - PUT matrix://hs1/_matrix/federation/v1/send/1665035836652
federation_sender1 | 2022-10-06 05:57:20,955 - synapse.federation.sender.transaction_manager - 174 - INFO - federation_transaction_transmission_loop-0 - TX [hs1] {1665035836652} got 200 response
client_reader1 | 2022-10-06 05:57:20,956 - synapse.http.federation.well_known_resolver - 253 - INFO - GET-5 - Fetching https://hs1/.well-known/matrix/server
client_reader1 | 2022-10-06 05:57:21,260 - synapse.http.federation.well_known_resolver - 284 - INFO - GET-5 - Error fetching https://hs1/.well-known/matrix/server: Connection was refused by other side: 111: Connection refused.
client_reader1 | 2022-10-06 05:57:21,261 - twisted - 274 - INFO - GET-5 - b'/etc/resolv.conf' changed, reparsing
client_reader1 | 2022-10-06 05:57:21,261 - twisted - 274 - INFO - GET-5 - Resolver added ('127.0.0.11', 53) to server list
client_reader1 | 2022-10-06 05:57:21,302 - synapse.http.matrixfederationclient - 299 - INFO - GET-5 - {GET-O-1} [hs1] Completed request: 200 OK in 0.35 secs, got 3911 bytes - GET matrix://hs1/_matrix/federation/v1/backfill/%21iXKoBLEwXgfsObqmow%3Ahs1?v=%24TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs&limit=100
client_reader1 | 2022-10-06 05:57:21,312 - synapse.handlers.federation_event - 634 - INFO - GET-5 - asdf backfill: events=['$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', '$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo', '$t308g3PzVhZa8LxkX2PwGHMEWadKmV8UOYR5NfIz4fs', '$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', '$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', '$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4']


client_reader1 | 2022-10-06 05:57:21,491 - synapse.handlers.federation_event - 828 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - Processing pulled event <FrozenEventV3 event_id=$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs, type=m.room.message, state_key=None, outlier=False>
client_reader1 | 2022-10-06 05:57:21,491 - synapse.storage.databases.main.events_worker - 610 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - _get_events_from_cache_or_db: looking for event_ids={'$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs'}
client_reader1 | 2022-10-06 05:57:21,491 - synapse.storage.databases.main.events_worker - 621 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - _get_events_from_cache_or_db: AFTER _get_events_from_local_cache events=[]
client_reader1 | 2022-10-06 05:57:21,494 - synapse.handlers.federation_event - 861 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - _process_pulled_event: De-outliering event $TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs
client_reader1 | 2022-10-06 05:57:21,500 - synapse.storage.databases.main.events_worker - 610 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - _get_events_from_cache_or_db: looking for event_ids={'$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', '$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', '$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4'}
client_reader1 | 2022-10-06 05:57:21,500 - synapse.storage.databases.main.events_worker - 621 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - _get_events_from_cache_or_db: AFTER _get_events_from_local_cache events=[('$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', True), ('$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', True), ('$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4', True)]
client_reader1 | 2022-10-06 05:57:21,500 - synapse.storage.databases.main.events_worker - 610 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - _get_events_from_cache_or_db: looking for event_ids={'$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', '$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', '$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4'}
client_reader1 | 2022-10-06 05:57:21,501 - synapse.storage.databases.main.events_worker - 621 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - _get_events_from_cache_or_db: AFTER _get_events_from_local_cache events=[('$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', True), ('$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', True), ('$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4', True)]
master | 2022-10-06 05:57:21,504 - synapse.replication.http.federation - 144 - INFO - POST-12 - Got 1 events from federation
master | 2022-10-06 05:57:21,505 - synapse.storage.controllers.persist_events - 395 - INFO - POST-12 - persist_events event_ids=['$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs'] obackfilled=True
master | 2022-10-06 05:57:21,505 - synapse.storage.controllers.persist_events - 559 - INFO - persist_events-12 - _persist_event_batch events=[('$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', False)]
master | 2022-10-06 05:57:21,508 - synapse.storage.databases.main.events - 1408 - INFO - persist_events-12 - _update_outliers_txn: Updating state for ex-outlier event $TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs
master | 2022-10-06 05:57:21,509 - synapse.storage.databases.main.events - 1437 - INFO - persist_events-12 - _update_outliers_txn setting outlier=False for event=$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs
master | 2022-10-06 05:57:21,512 - synapse.storage.databases.main.cache - 223 - INFO - persist_events-12 - _invalidate_caches_for_event START event_id=$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs
master | 2022-10-06 05:57:21,512 - synapse.storage.databases.main.cache - 267 - INFO - persist_events-12 - _invalidate_caches_for_event END event_id=$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs
master | 2022-10-06 05:57:21,515 - synapse.access.http.9093 - 459 - INFO - POST-12 - 127.0.0.1 - 9093 - {None} Processed request: 0.010sec/0.001sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 19B 200 "POST /_synapse/replication/fed_send_events/EGjlSBsHpT HTTP/1.1" "Synapse/1.69.0rc1" [0 dbevts]
client_reader1 | 2022-10-06 05:57:21,515 - synapse.http.client - 446 - INFO - GET-5-$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs - Received response to POST http://127.0.0.1:9093/_synapse/replication/fed_send_events/EGjlSBsHpT: 200
client_reader1 | 2022-10-06 05:57:21,516 - synapse.storage.databases.main.stream - 1379 - INFO - GET-5 - paginate_room_events START room_id=!iXKoBLEwXgfsObqmow:hs1 direction=b limit=100 from_key=RoomStreamToken(topological=7, stream=-8, instance_map=frozendict.frozendict({})) to_key=None
client_reader1 | 2022-10-06 05:57:21,518 - synapse.storage.databases.main.events_worker - 610 - INFO - GET-5 - _get_events_from_cache_or_db: looking for event_ids={'$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', '$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', '$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', '$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8', '$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo', '$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4', '$t308g3PzVhZa8LxkX2PwGHMEWadKmV8UOYR5NfIz4fs'}
client_reader1 | 2022-10-06 05:57:21,518 - synapse.storage.databases.main.events_worker - 621 - INFO - GET-5 - _get_events_from_cache_or_db: AFTER _get_events_from_local_cache events=[('$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', True), ('$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', True), ('$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', True), ('$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8', False), ('$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo', True), ('$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4', True), ('$t308g3PzVhZa8LxkX2PwGHMEWadKmV8UOYR5NfIz4fs', True)]
client_reader1 | 2022-10-06 05:57:21,518 - synapse.storage.databases.main.stream - 1402 - INFO - GET-5 - paginate_room_events GOT EVENTS room_id=!iXKoBLEwXgfsObqmow:hs1 direction=b limit=100 events=[('$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8', False), ('$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', True), ('$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo', True), ('$t308g3PzVhZa8LxkX2PwGHMEWadKmV8UOYR5NfIz4fs', True), ('$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', True), ('$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', True), ('$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4', True)]
client_reader1 | 2022-10-06 05:57:21,519 - synapse.handlers.pagination - 541 - INFO - GET-5 - get_messages AFTER FILTER events=['$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8', '$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs', '$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo', '$t308g3PzVhZa8LxkX2PwGHMEWadKmV8UOYR5NfIz4fs', '$yN88ZdlJv7NJKVGac5oLXmIPC2_rZdQNoaidAjRQBzs', '$foMnyAdYtC4-pZaq7HuQfcPj4VjPIOLe0Txba2q4hHk', '$LzJod63lU5fAUarmjmSF7sXyDGNtR0cJpdd6Gj1uop4']
client_reader1 | 2022-10-06 05:57:21,519 - synapse.storage.databases.main.events_worker - 610 - INFO - GET-5 - _get_events_from_cache_or_db: looking for event_ids={'$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo'}
client_reader1 | 2022-10-06 05:57:21,520 - synapse.storage.databases.main.events_worker - 621 - INFO - GET-5 - _get_events_from_cache_or_db: AFTER _get_events_from_local_cache events=[('$gKKRZsbXvPZYNJVeWhQs75OTxgZDR1Xt6nvcIl80Ivo', True)]
client_reader1 | 2022-10-06 05:57:21,520 - synapse.handlers.pagination - 553 - INFO - GET-5 - get_messages AFTER FILTER EVENTS FOR CLIENT events=['$otTGXeQQujiLXtnVfkJGHabzkjSawmBHCwgacD-DZP8']

client_reader1 | 2022-10-06 05:57:21,521 - synapse.access.http.18020 - 459 - INFO - GET-5 - ::ffff:127.0.0.1 - 18020 - {@charlie:hs2} Processed request: 0.594sec/0.000sec (0.054sec, 0.018sec) (0.015sec/0.071sec/45) 356B 200 "GET /_matrix/client/r0/rooms/%21iXKoBLEwXgfsObqmow:hs1/messages?dir=b&from=t7--8_0_0_0_0_0_0_0_0&limit=100 HTTP/1.0" "Go-http-client/1.1" [1 dbevts]

client_reader1 | 2022-10-06 05:57:21,609 - synapse.storage.databases.main.cache - 223 - INFO - process-replication-data-87 - _invalidate_caches_for_event START event_id=$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs
client_reader1 | 2022-10-06 05:57:21,609 - synapse.storage.databases.main.cache - 267 - INFO - process-replication-data-87 - _invalidate_caches_for_event END event_id=$TAChAndk84Pb08Wvc_7IFceFvg79EzbnODgRDFETrOs

Spawning from matrix-org/synapse#14028 (comment) The reason they failed before was because we're fighting against stale caches across workers racing while waiting for invalidation, see matrix-org/synapse#13185 (comment) > Here is what happens: > > 1. `serverB` has `event1` stored as an `outlier` from previous requests (specifically from MSC3030 jump to date pulling in a missing `prev_event` after backfilling) > 1. Client on `serverB` calls `/messages?dir=b` > 1. `serverB:client_reader1` accepts the request and drives things > 1. `serverB:client_reader1` has some backward extremities in range and requests `/backfill` from `serverA` > 1. `serverB:client_reader1` processes the events from backfill including `event1` and puts them in the `_event_persist_queue` > 1. `serverB:master` picks up the events from the `_event_persist_queue` and persists them to the database, de-outliers `event1` and invalidates its own cache and sends them over replication > 1. `serverB:client_reader1` starts assembling the `/messages` response and gets `event1` out of the stale cache still as an `outlier` > 1. `serverB:client_reader1` responds to the `/messages` request without `event1` because `outliers` are filtered out > 1. `serverB:client_reader1` finally gets the replication data and invalidates its own cache for `event1` (too late, we already got the events from the stale cache and responded)

squahtx · 2022-10-06T16:52:25Z

Fixes matrix-org/synapse#13944?

MadLittleMods · 2022-10-06T16:54:44Z

tests/msc3030_test.go

@@ -203,6 +203,16 @@ func TestJumpToDateEndpoint(t *testing.T) {
 				contextResResBody := client.ParseJSON(t, contextRes)


Fixes matrix-org/synapse#13944?

@squahtx I don't think this fixes that issue. The error there is:

msc3030_test.go:200: CSAPI.MustDoFunc response return non-2xx code: 403 Forbidden - body: {"errcode":"M_FORBIDDEN","error":"You don't have permission to access that event."}

which indicates we also need to retry the /context request above until it works

Fixed in a separate PR: matrix-org/synapse#14096

tests/msc3030_test.go

DMRobertson

Is the fact that we have to do this a bug in Synapse? From the description it sounds like this is all happening within a single request to the client_reader.

Are there any spec guidelines on how /messages is supposed to behave WRT backfill?

MadLittleMods · 2022-10-11T17:55:28Z

Is the fact that we have to do this a bug in Synapse? From the description it sounds like this is all happening within a single request to the client_reader.

I think this is more a discussion to be had on matrix-org/synapse#13185. The current Synapse behavior doesn't feel quite right but since there is no spec/guidelines, this could be assumed to be correct as well.

Are there any spec guidelines on how /messages is supposed to behave WRT backfill?

I don't think so. The spec just says this:

Since users in that room are able to request the history by the /messages client API endpoint, it’s possible that they might step backwards far enough into history before the homeserver itself was a member of that room.

To cover this case, the federation API provides a server-to-server analog of the /messages client API, allowing one homeserver to fetch history from another. This is the /backfill API.

-- https://spec.matrix.org/v1.1/server-server-api/#backfilling-and-retrieving-missing-events

…rker See #492 (comment)

DMRobertson

LGTM!

tests/msc3030_test.go

See #492 (comment)

MadLittleMods · 2022-10-13T09:11:38Z

Thanks for the review @squahtx and @DMRobertson 🐦

MadLittleMods mentioned this pull request Oct 6, 2022

Enable MSC2716 Complement tests for workers matrix-org/synapse#14074

Closed

7 tasks

MadLittleMods marked this pull request as ready for review October 6, 2022 08:01

MadLittleMods requested review from a team as code owners October 6, 2022 08:01

MadLittleMods removed the request for review from a team October 6, 2022 08:02

squahtx linked an issue Oct 6, 2022 that may be closed by this pull request

Complement can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint is flakey(?) matrix-org/synapse#13944

Closed

MadLittleMods commented Oct 6, 2022

View reviewed changes

squahtx removed a link to an issue Oct 6, 2022

Complement can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint is flakey(?) matrix-org/synapse#13944

Closed

squahtx reviewed Oct 7, 2022

View reviewed changes

tests/msc3030_test.go Outdated Show resolved Hide resolved

DMRobertson reviewed Oct 10, 2022

View reviewed changes

Extra description on persisting worker not the same as /messages wo…

5aebe26

…rker See #492 (comment)

MadLittleMods requested a review from DMRobertson October 11, 2022 22:30

MadLittleMods mentioned this pull request Oct 12, 2022

Include guidelines how to backfill with /messages and /context matrix-org/matrix-spec#1281

Open

DMRobertson approved these changes Oct 12, 2022

View reviewed changes

tests/msc3030_test.go Outdated Show resolved Hide resolved

DMRobertson mentioned this pull request Oct 12, 2022

Refactor MSC3030 /timestamp_to_event to move away from our snowflake pull from destination pattern matrix-org/synapse#14096

Merged

5 tasks

Return condition result directly

dcf74db

See #492 (comment)

MadLittleMods merged commit e91a70f into main Oct 13, 2022

MadLittleMods deleted the madlittlemods/deflake-msc3030-backfill-test-with-workers branch October 13, 2022 09:11

MadLittleMods mentioned this pull request Oct 17, 2022

Race condition with replication means /messages backfill lacks read-after-write consistency between workers matrix-org/synapse#14211

Open

matrixbot mentioned this pull request Dec 21, 2023

Race condition with replication means /messages backfill lacks read-after-write consistency between workers element-hq/synapse#14211

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

De-flake MSC3030 backfill test with Synapse worker replication #492

De-flake MSC3030 backfill test with Synapse worker replication #492

MadLittleMods commented Oct 6, 2022 •

edited

Loading

squahtx commented Oct 6, 2022

MadLittleMods Oct 6, 2022 •

edited

Loading

MadLittleMods Oct 7, 2022

DMRobertson left a comment

MadLittleMods commented Oct 11, 2022

DMRobertson left a comment

MadLittleMods commented Oct 13, 2022

		@@ -203,6 +203,16 @@ func TestJumpToDateEndpoint(t *testing.T) {
		contextResResBody := client.ParseJSON(t, contextRes)

De-flake MSC3030 backfill test with Synapse worker replication #492

De-flake MSC3030 backfill test with Synapse worker replication #492

Conversation

MadLittleMods commented Oct 6, 2022 • edited Loading

Why did this test fail with workers before?

Dev notes

squahtx commented Oct 6, 2022

MadLittleMods Oct 6, 2022 • edited Loading

Choose a reason for hiding this comment

MadLittleMods Oct 7, 2022

Choose a reason for hiding this comment

DMRobertson left a comment

Choose a reason for hiding this comment

MadLittleMods commented Oct 11, 2022

DMRobertson left a comment

Choose a reason for hiding this comment

MadLittleMods commented Oct 13, 2022

MadLittleMods commented Oct 6, 2022 •

edited

Loading

MadLittleMods Oct 6, 2022 •

edited

Loading