Perf improvement to getting auth chains

[erikjohnston]

Let's see if this helps.

--

Regression introduced in #17044

Fixes: #17129

Replaces: #17154

[csett86]

@erikjohnston I applied this patch on 1.106.0 and unfortunately it does not really improve. Very long get_auth_chain_difference_chains are back:

[heftig]

The old code also removed the chains that _materialize adds to the chains from the chains_to_fetch. Is this behavior covered again?

[heftig]

This comment no longer applies.

[strutztm]

We deployed this on top of v1.114.0 yesterday and it seems to fix the issue. On the most affected server, our transaction times for get_auth_chain_difference_chains are between 10 seconds and 2 minutes instead of quickly increasing to hours.

Is that a normal range? That's about the same as we measured after rolling back from v1.106.0 to v1.104.0, but all other transaction times are in the milliseconds range.

[jaywink]

We've been testing this in EMS. There were encouraging results for 3 hosts suffering badly from #17129 that benefited from this patch. However, further testing with more hosts was not as successful. Several hosts seem to be performing worse, once grinding to a halt with the RDS instance totally pinning itself on CPU. Removing the patch immediately resolved the RDS CPU being hammered.

Host names in the linked internal issue.