Rampfit test intermittently failing

[WilliamJamieson]

The fix in #725 does not quite work all of the time, on the main branch I am getting intermittent failures in the

romancal/ramp_fitting/tests/test_ramp_fit.py::test_one_group_small_buffer_fit_ols

test cases. Iterative running of this test locally (it is skipped in CI) reproduces the error locally for me. For example I get the error:

E           AssertionError: 
E           Not equal to tolerance rtol=1e-06, atol=0
E           
E           Mismatched elements: 1 / 1 (100%)
E           Max absolute difference: 1.06613152e-11
E           Max relative difference: 1.06613152e-06
E            x: array(-1.000001e-05, dtype=float32)
E            y: array(-1.e-05)

I believe the root of this error is:

romancal/romancal/ramp_fitting/tests/test_ramp_fit.py

Lines 32 to 36 in 2396bc7

	data = u.Quantity(
	(np.random.random(shape) * 0.5).astype(np.float32), u.DN, dtype=np.float32
	)
	err = u.Quantity(
	(np.random.random(shape) * 0.0001).astype(np.float32), u.DN, dtype=np.float32

The intermittent failure is caused by the fact that the data/err change from run-to-run of the tests. Instead, the random number generator should be seeded so that every test is exactly reproducible among different test runs.

[WilliamJamieson]

Note that this does not happen very often locally, but I can get it to reappear I think the issue is the very tight tolerances meaning larger fluctuations due to randomness occasionally escape the tolerances (notice how tiny the differences are above).

In any case, the randomness should be seeded as it is just used as a convenient way to generate data.

[schlafly]

I stumbled on this issue as well. This test originally tested against the one pixel that was forced to have a value of 10, but in #725 we changed that to point to a different pixel. I think the real fix to 725 is a little different---the change to stcal meant that group_time started getting used, which is -99999 and causes an issue here. I'll put in a fix for that soon, but I agree that seeding makes sense too.

[WilliamJamieson]

Yes, I am working through all of the romancal tests so that it uses the numpy random generators: using an arbitrarily fixed seed.

[WilliamJamieson]

This does intermittently fail in the regression suite as well, see: https://plwishmaster.stsci.edu:8081/job/RT/job/Roman-Developers-Pull-Requests/268/testReport/junit/romancal.ramp_fitting.tests/test_ramp_fit/_stable_deps__test_one_group_small_buffer_fit_ols_none_/