Experimental Sync Speedup

[ShadowJonathan]

This PR adds a speedup to /sync when the user is in many rooms with many members, the devicelist checking part of these sync requests can take up a while (1 second flat per sync) on single-user homeservers / lower-end setups.

This PR makes it so that /sync will only check for devicelist changes for users in encrypted rooms, thus this fixes #7524.

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
• Pull request includes a sign off
• Code style is correct (run the linters)

Signed-off-by: Jonathan de Jong <[email protected]>

[ShadowJonathan]

I tested this by applying the changes onto the release/v1.31.0 branch and building a docker image from that, it works, i didn't experience any immediate breakages.

Performance comparisons

(Using the main process for a second, unpatched with this change.)

(Utilizing the patched synchotron, note the difference in database calls)

[ShadowJonathan]

There are some things wrong with this change, which need scrutiny and review;

• I'm fairly sure I did caching incorrectly, or that this change causes caching to go incorrectly.
• I faintly remember an edgecase I inwardly noticed regarding state-setting m.room.encrypted in a room. Between the from_token and now_token.
• This needs a good test.
• Element's devs would need to be asked on how they currently rely on the devicelist updates, for anything outside of encryption, to see if this change will not silently break that functionality.
• A devicelist update needs to be fired when the user joins an encrypted room, when every user in that room did not have a devicelist update in-between the local user getting invited, and joining.

[ShadowJonathan]

Also, a lot of sytest tests seem to fail because of this change, because sytest expects the behaviour that devicelist changes are still propagated even if you dont share an encrypted room with that user.

I can't do much about these tests, since this change directly either invalidates them, or they invalidate this change, so I need some input on this change, and a direction to go forward with.

[erikjohnston]

If we're only sending devices down for users we share encrypted rooms with we need to ensure that we proactively send down the device list for a user if either a) a room gets upgraded to an encrypted room or b) we join an encrypted room with the user. I'm not sure if that's the case already?

[erikjohnston]

We'd need to invalidate the cached result if the encryption state of the room change. I think we'd probably want to change _filter_encrypted_rooms to be a get_encryption_state_for_rooms and add a cache on that, so that we can on_invalidate=cache_context.invalidate when we call it, and so it'll "do the right thing".

[ShadowJonathan]

If we're only sending devices down for users we share encrypted rooms with we need to ensure that we proactively send down the device list for a user if either a) a room gets upgraded to an encrypted room or b) we join an encrypted room with the user. I'm not sure if that's the case already?

That's not the case with the current code, but i have both on this checklist.

[richvdh]

I'm going to close this as @ShadowJonathan says he has abandoned it.

[ShadowJonathan]

Yeah, I don't think that I'll be working on this any time soon, the PR was more a reference on how-to-do this with some todo points for y'all.