Federation catchup for a destination can block indefinitely on un-partial stating

[squahtx]

What we think happened

1. A user faster joined a remote room
2. Some events were sent into the room, but not transmitted successfully
3. The user left the room before it became full state
4. The federation catchup loop tried to transmit events in the room, but blocked on full state. This also blocked *all* catchup transmission for the destination.

Stack trace

  File "/home/synapse/src/synapse/metrics/background_process_metrics.py", line 242, in run
    return await func(*args, **kwargs)
  File "/home/synapse/src/synapse/federation/sender/per_destination_queue.py", line 321, in _transaction_transmission_loop
    await self._catch_up_transmission_loop()
  File "/home/synapse/src/synapse/federation/sender/per_destination_queue.py", line 544, in _catch_up_transmission_loop
    new_pdus = await filter_events_for_server(
  File "/home/synapse/src/synapse/visibility.py", line 650, in filter_events_for_server
    event_to_memberships = await _event_to_memberships(
  File "/home/synapse/src/synapse/visibility.py", line 746, in _event_to_memberships
    event_to_state_ids = await storage.state.get_state_ids_for_events(
  File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "/home/synapse/src/synapse/storage/controllers/state.py", line 267, in get_state_ids_for_events
    event_to_groups = await self.get_state_group_for_events(
  File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "/home/synapse/src/synapse/storage/controllers/state.py", line 374, in get_state_group_for_events
    await self._partial_state_events_tracker.await_full_state(event_ids)
  File "/home/synapse/src/synapse/logging/opentracing.py", line 914, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "/home/synapse/src/synapse/storage/util/partial_state_events_tracker.py", line 82, in await_full_state
    logger.info(

[DMRobertson]

Workedaround in ab629c1, but I think we ought to fix this properly for the next release.

[DMRobertson]

Of note: we only reach that far down into the stack if the room's history visibility is restricted:

synapse/synapse/visibility.py

Lines 649 to 676 in b5b5f66

	# for any with restricted vis, we also need the memberships
	event_to_memberships = await _event_to_memberships(
	storage,
	[
	e
	for e in events
	if event_to_history_vis[e.event_id]
	not in (HistoryVisibility.SHARED, HistoryVisibility.WORLD_READABLE)
	],
	target_server_name,
	)
	to_return = []
	for e in events:
	erased = is_sender_erased(e, erased_senders)
	visible = check_event_is_visible(
	event_to_history_vis[e.event_id], event_to_memberships.get(e.event_id, {})
	)
	if e in partial_state_invisible_events:
	visible = False
	if visible and not erased:
	to_return.append(e)
	elif redact:
	to_return.append(prune_event(e))
	return to_return

[DMRobertson]

Fixed by #15248.