Stabilize the CI a little bit (especially the MSRV test)

[nagisa]

Once in a while our test that tests MSRV breaks, usually due to dependency changes, and currently the best action to resolve this is to test a higher version.

We probably should check in a lock file that's used for testing of the MSRV only...