Performance regressions following pydantic update

[ml-evs]

Already running into two teething issues with #1745.

1. Tests now take significantly longer to run (15 mins per Python version) -- will investigate this on a per-test basis.
2. RAM usage has increased to the point where we can no longer use the Fly.io free-tier (above 256 MB). Will try to debug this too...

[ml-evs]

After playing around retriggering various old test runs, here's a solid example:

• https://github.com/Materials-Consortia/optimade-python-tools/actions/runs/6510594357 ran to completion in 10.5 mins, 6.5 mins and 5 mins for Python's 3.10, 3.11 and 3.9 respectively
• https://github.com/Materials-Consortia/optimade-python-tools/actions/runs/6718080498 ran to completion in 12.5 minutes, 14.1 minutes and 21 minutes for 3.10, 3.11 and 3.9 respectively...

These both ran today so don't think it can be a GH actions perf regression, but I cannot easily explain away the difference between Python versions

[JPBergsma]

It seems like some of the steps are skipped for python 3.10 and 3.11 for the new version(there is (/) infront of the task rather than (✓). For example the step: "Run the optimade client cli" is executed for python 3.9 but not for python 3.10 and 3.11 in the new version.
In the old version it was not executed for python 3.9 and 3.11 but it did get executed for python 3.10. This would explain the timing differences. But I am not sure why some tests get executed one time but not the other time.

[ml-evs]

Another reason is that we are using the fly deployment in our client tests (unfortunately down due to the memory usage described above), which by default take a while to timeout!

[ml-evs]

Closing this after fixing Fly by using a more lightweight builder with significantly lower mem usage (just enough to creep back into the free tier of Fly)