Listing a Directory with Many files takes a long time

[jag3773]

Story

As a user I don't have to have to wait a long time for a file listing when I click on a directory in my repository so that I don't waste time or get frustrated.

Notes

Opening the https://git.door43.org/Door43/en_tw/src/master/bible/other takes a while.

[ethantkoenig]

Relevant upstream issue: go-gitea/gitea#502

From poking around, the bottleneck seems to be with looking up the latest commit for each file in the directory; a separate git log -1 <filename> command is run for each file. These commands are already run in parallel, and I can't find an alternative way to look up the latest commits in batches.

[ethantkoenig]

Actually, I think I have a way to speed things up; it'll require making a change to an upstream dependency (https://github.com/go-gitea/git).

[jag3773]

This same bottleneck is why we couldn't use the elastic file system at AWS, but had to go with an EBS volume. I thought storing this data in the database might help.

What did you have in mind for a fix?

[ethantkoenig]

The general idea is to get all commit hashes affecting any of the directory's entries (git log path/to/dir), then for each commit hash list the affected files (git ls-tree <hash>). We start with the latest commit, and go backwards until we've found a commit for each entry.

A proof-of-concept test I ran locally showed a ~20x improvement (15 vs 0.7 seconds) for the example repo.