What does uproot.concatenate really do?

[MoAly98]

Hi experts,

Let's say I am trying to access multiple trees from the same set of files which can fit into memory, call it files. The documentation suggests that for reading sets of files, uproot.concatenate is the way to go. There are multiple ways I can think to do this:

1. Simply make two sets of file lists files_tree1 and files_tree2 where files_treeX = [f+":treeX" for f in files], pass each list to uproot.concatenate in a separate call:

uproot.concatenate(files_tree1, filter_name = tree1_branches)
uproot.concatenate(files_tree2, filter_name = tree2_branches)

This I think is inefficient, since the same files wil be opened twice. Maybe caching would help, but feels like it can be done better.

2. [Naively] make a combined files list: files_all = files_tree1 + files_tree2 and combined branches list: all_branches = tree1_branches + tree2_branches] and pass these to one uproot call, hoping uproot magic will know what to do. This surprisngly (at least to me) did not crash. Uproot just produced an awkward array which is a union of 2 awkward arrays tree1_branches (N entries) and tree2_branches (M entries), with ak.type(data) giving :

N+M * union[{"tree1_branch1": var * float32, "tree1_branch2": var * float32}, {"tree2_branch1": int32}]

so calling data[-1] gives the last entry from tree2 and data[0] gives the first entry of tree1. I guess it can be expected behaviour from the global_index that uproot.concatenate() seems to keep track of (or maybe I'm completely off)?

Anyway, I think we still open each file twice, which is non-ideal.

3. I can do a manual loop over the files and call uproot.open on each of them, then access the keys from the structure we get back. This way each file is opened once.

My questions are:

• What does uproot.concatenate do in the background that makes it more performant (if that's even true) than uproot.open inside a loop over files? What I can see quickly from a skim over the source code is that concatenate loops over the files one by one, opening them as ReadOnlyFile then grabbing the data, but I am probably missing something subtle in the steps.
• What do you recommend as best practice to ready multiple trees from many files?

[jpivarski]

Here's the implementation of uproot.concatenate: https://github.com/scikit-hep/uproot4/blob/6fea96e3e61814cdeb21a3527c215f1911759d9a/src/uproot/behaviors/TBranch.py#L343-L382

It doesn't do much, just iterates over the files, reads the requested data, and calls ak.concatenate to concatenate them. I can see how what you tried ends up producing a union array (concatenate two arrays with different types and you'll get a union array of data that is first all the first type, then all the second type). You very likely don't want the union array because it can make downstream processing harder: even methods that can deal with union arrays at all have to take a conservative approach about what's in them.

If you want to avoid opening the same files multiple times, you'd have to forsake the convenience of uproot.concatenate, loop over the set of files yourself, reading the two TTrees from each and building up a list to concatenate. Or maybe this is a blessing: if your analysis can work on smaller chunks, you can do it with smaller overall memory use.

It doesn't make sense to try to cover a lot of use-cases in uproot.concatenate because that would complicate its interface, and then it would not be a very good convenience function anymore. The reason it exists in the first place is because some people were using uproot.lazy just to get a multi-file interface, but they were suffering from the performance problems of lazy arrays. uproot.concatenate has the interface of uproot.lazy, but reads eagerly (good when you have enough memory!).