Performance problem - Pages are too slow to load

[Didayolo]

Codabench is too slow to load.

• Home page of competitions are loading everything at once. We need a lazy loading.

• When a loading is long, it is decreasing the performance for everybody else (this is normal I guess).

• Competitions list is slow to load too, despite the small logos (Add thumbnail logo to competitions to increase loading speed of the list #1290). Code of this page: https://github.com/codalab/codabench/blob/f83f94c1e492da524fef5b2f6ccf26846fa88b69/src/static/riot/competitions/public-list.tag

TODO

• Increase the CPU and RAM of the main VM
• Increase the CPU and RAM of the main VM (again)
• Competition page
  • Trying to remove "entries" column from leaderboard, which may be the one that is slow to load (Removed num_entries becasue the count was slowing down the platform #1673)
  • Put back "entries" in an efficient way
  • Pagination for "Submissions" tab (and lazy loading)
  • "Show more" on leaderboard results tab. By default show 30 rows, and add more when clicking on show more. If the participants is not on the first 30, it could still be shown at the bottom of the table after some "[...]" row
• Competition list
  • Optimize the code for loading the competitions list (@bbearce) (CompetitionSerializer Simple vs Detail #1376) (Public competition filtering #1381)
  • Public competition filtering #1381 improves the query by directly filter public competitions. We need to improve the filter to take into account the dates and page of the public list we are displaying.
• General performance and home page
  • Remove temporarily participants and submissions counters to accelerate the competition loading URGENT - removed count from competition query to fix the codabench #1645
  • Participants and submissions counters in an efficient way (Optimization PR#2 - Submissions and Participants Count #1669)
  • Featured competitions manually set (Optimization PR#1 - Featured competitions #1678)
  • Home page counters (Optimization PR#3 - Home page counters #1699)
    • Update the counts somewhere in a file or in the db with a script that is run once a day
    • Update home page to fetch the counts from the file/db to make them dynamic instead of hardcoded

[ihsaan-ullah]

• Optimize the code for loading a competition page

This needs proper planning as code cannot be optimized with current structure of the web page. It is a big update

[Didayolo]

Still very slow after merging #1343

[bbearce]

Tested and think I have a fix:
Docs for automation to create large amounts of data

Essentially we used the CompetitionDetailSerializer to load competitions and I switched to CompetitionSerializerSimple. The detail one serializes phases and tasks and I believe that is time consuming. The Simple one has everything we need and it loads things faster on my local when I create lots of data. using selenium.

[Didayolo]

@bbearce #1376 is deployed on production, let's see if it helps!

[Didayolo]

We need to make the loading of competition page lazy.

codabench/src/apps/api/views/competitions.py

Line 160 in c2cd638

qs = qs.prefetch_related(

codabench/src/apps/api/views/submissions.py

Line 31 in c2cd638

class SubmissionViewSet(ModelViewSet):

[ihsaan-ullah]

What do you think if we separate the admin procedures from the competition detail page? Similar to Edit page we can separate the submissions, participants etc in separate pages

[Didayolo]

That may be a good idea, especially for "Submissions" which can be heavy. However this will accelerate the loading only for organizers and admins.

[Didayolo]

We removed temporarily the submissions and participants counters to unblock the platform (#1646)

[ihsaan-ullah]

Here is what we want to do:

Featured Competitions on home page (Solved by: #1678)

1. Update the competition modal to have field/attribute is_featured that controls if a competition is featured or not
2. Use this field in the home page to show real featured competitions

Submissions and Participants (Solved by: #1669)

1. Put submissions and participants count in the competition model (update these counts when a new submission/ participant is added or a submission is deleted)
2. For all current competitions fill these fields with correct values
3. In the competition detail page, use these values in the UI (currently we show nothing)
4. In the home page make the popular competiitons dynamic again using the new participants count field
5. In the home page use the participants count field in featured competitions

Home page counters (Solved by: #1699)

1. Update the counts somewhere in a file or in the db with a script that is run once a day
2. Update home page to fetch the counts from the file/db to make them dynamic instead of hardcoded

Long term solution for competition details page

• We should redo the apis used in the competition detail page to simplify and make them light
• We should not fetch data unless we want to use it