Fix performance bug with many tilesets

[ggetz]

Description

Pointed out on the forum. This was an oversight on my part in #11837.

Issue number and link

N/A

Testing plan

1. Verify before and after performance of OP's Sandcastle
2. Verify collision still works for google tilesets

Author checklist

• I have submitted a Contributor License Agreement
• I have added my name to CONTRIBUTORS.md
• I have updated CHANGES.md with a short summary of my change
• I have added or updated unit tests to ensure consistent code coverage
• I have update the inline documentation, and included code examples where relevant
• I have performed a self-review of my code

[github-actions]

Thank you for the pull request, @ggetz!

✅ We can confirm we have a CLA on file for you.

[ggetz]

@jjspace Would you mind taking a look? This branch has already been validated for fixing the original issue by the forum poster.

[jjspace]

Code looks good, confirmed the change improves performance on the sandcastle from the forums

Edit: I just pushed a fix for the merge conflict, will merge when green again