Chunk merging issue

[FaroutYLq]

The following runs seem to be problematic in cuts computation. Example error looks like below. [45501, 44405, 46721, 47083, 47287, 47296, 47298, 46731, 45437, 45451, 045462, 045501]

ValueError: Cannot merge chunks with different number of items: [[046731.peak_positions_cnn: 1659907969sec 424508440 ns - 1659907990sec 899344920 ns, 22238 items, 0.0 MB/s], [046731.peak_positions_mlp: 1659907969sec 424508440 ns - 1659907990sec 899344920 ns, 22234 items, 0.0 MB/s], [046731.peak_positions_gcn: 1659907969sec 424508440 ns - 1659907990sec 899344920 ns, 22238 items, 0.0 MB/s]]

More details you can find logs at /project/lgrandi/xenonnt/data_management_reprocessing/job_logs. Already used --notlazy in straxer, and I am requiring single core at this point.

[FaroutYLq]

Trying to build on midway2 notebook, and same problems show up.

runs = ['051851', '050841', '050831', '050773', '047874', '047298', '047296', '047287', '047083', '046731', '046721', '046494', '046493', '045501', '045462', '045451', '045437', '044405']
st.make(runs, 'event_ms_naive')

ValueError: Cannot merge chunks with different number of items: [[045437.peaklets: 1656799988sec 619275800 ns - 1656800031sec 568948760 ns, 88236 items, 8.6 MB/s], [045437.peaklet_classification: 1656799988sec 619275800 ns - 1656800031sec 568948760 ns, 88235 items, 0.0 MB/s]]

[dachengx]

(For a record, it is discussed here: https://xenonnt.slack.com/archives/C016DM0JPK9/p1690013077769009)

I highly suspect that the reason for this error is deep in the algebra library of different nodes. Similar thing https://xenonnt.slack.com/archives/C016DM0JPK9/p1683044901706209 has been observed before. So if we re-process these three plugins on the same node, like xenon1t, we will get no error.

The quick fix would be just re-reprocess them. But for a long-term solution, we should test the output on different nodes, even on the same node multiple times to test the randomness of this phenomenon.