Idea to speed up the loading of the Homework Sets Editor

[taniwallach]

In the discussion at #1585:

• @pstaabp mentioned "Since you mention speed up, I've noticed that the Homework Sets Editor still loads slowly compared to others" (Improve database usage by consolidating and removing database calls #1585 (comment))
• @dlglin replied that "Every time the Homework Sets Editor loads it combs through the entire templates directory tree looking for def files, including the OPL and Contrib. Depending on how fast your storage is this can greatly slow down the page load." (Improve database usage by consolidating and removing database calls #1585 (comment))

Maybe this can be fixed by no longer generating the list "live" and by not providing the full list of set definition files each time the Homework Sets Editor page is requested.

Instead, we could try to:

1. Dynamically load the list (and create the items in the select element) only when the "import" tab is activated.
   • I assume an approach similar to what works in the library browser to populate the "Chapter" select element when the "Subject" is changed can be used.
2. Except for the "Import from where" select element, everything else could be ready immediately, so the "tab" would initially load quickly, and just need to report that "the list of set definition files is currently being loaded".
3. If feasible, store a cached version of the list of set definition files in the database (course level), and periodically update that list in the back-end when it is considered "old".
4. Provide a "rescan for new set definition files" button to request a "new full scan" immediately.
5. Determine whether the "scan" could be split between "local" content and "library" content.
   • At present $ce->{options}->{useOPLdefFiles} only effects whether paths under templates/Library get searched.
   • templates/Contrib could be added to that conditional handling.
   • However, maybe the course environment could provide an array of "top level" items under templates which should be treated as libraries to be conditionally searched. (Some courses may use additional special/shared directories" which need not be rescanned often.)
6. If scanning can be split by local vs. library directories, uploading files via the File Manager could be used to trigger a "local rescan".

[drgrice1]

Generally, scanning local directories does not take much time. It is the scan of the OPL that makes this take so long. So if we could just load the set definitions that are in the OPL from a pre-generated file, and always scan the local directories and exlcude Library and Contrib from that scan, that would solve the problem.

[dlglin]

There is exactly one .def file in the OPL, and a bunch more in Contrib.

I don't think that .def files belong in the same directories as the pg problems. If we want to accept contributed def files to the OPL, we could create a dedicated folder for them in the OPL repository, but I think it's a bad idea to have them commingled with the pg files.

I think it would also be cleaner to have a dedicated place for .def files within the course directory, but that's a separate (and less important) change, since recursing through the templates folder is less of a problem.

[taniwallach]

Implemented by @drgrice1 in #1653.
Made a "Closed" discussion category and moving this there.