Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize use of channel in unpublished_changes query #3839

Merged
merged 2 commits into from
Nov 23, 2022

Conversation

bjester
Copy link
Member

@bjester bjester commented Nov 23, 2022

Summary

Description of the change(s) you made

  • Reverts the reverted ordering changes to the channel list query
  • Resolves query performance issues handling references to the channel in the unpublished_changes annotation query

Manual verification steps performed

  1. Manually tweaked the query to use the outermost channel reference
  2. Tested tweaked query in production for user who's had issues
  3. Verified the query was using a better query plan and completed (much) faster
  4. Reverted the removal of ordering channels by modified date
  5. Repeated above testing of the query plan
  6. Verified the API response in local dev server also remained unchanged

Updated query plan

                                                                                                    QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=78366.37..78366.37 rows=2 width=936) (actual time=57.145..57.152 rows=29 loops=1)
   Sort Key: ((SubPlan 5)) DESC
   Sort Method: quicksort  Memory: 54kB
   ->  Nested Loop Left Join  (cost=14.54..78366.36 rows=2 width=936) (actual time=12.653..57.058 rows=29 loops=1)
         ->  Nested Loop  (cost=13.98..30.33 rows=2 width=689) (actual time=0.086..0.360 rows=29 loops=1)
               ->  Unique  (cost=13.69..13.70 rows=2 width=32) (actual time=0.075..0.120 rows=29 loops=1)
                     ->  Sort  (cost=13.69..13.70 rows=2 width=32) (actual time=0.074..0.102 rows=29 loops=1)
                           Sort Key: ((u1.channel_id)::text)
                           Sort Method: quicksort  Memory: 29kB
                           ->  Nested Loop  (cost=0.57..13.68 rows=2 width=32) (actual time=0.024..0.043 rows=29 loops=1)
                                 ->  Index Only Scan using contentcuration_user_pkey on contentcuration_user u0  (cost=0.29..4.30 rows=1 width=4) (actual time=0.013..0.014 rows=1 loops=1)
                                       Index Cond: (id = 28085)
                                       Heap Fetches: 1
                                 ->  Index Scan using contentcuration_channel_editors_e8701ad4 on contentcuration_channel_editors u1  (cost=0.29..9.36 rows=2 width=36) (actual time=0.009..0.023 rows=29 loops=1)
                                       Index Cond: (user_id = 28085)
               ->  Index Scan using contentcuration_channel_id_be7411c1_like on contentcuration_channel  (cost=0.29..8.30 rows=1 width=689) (actual time=0.007..0.007 rows=1 loops=29)
                     Index Cond: ((id)::text = (u1.channel_id)::text)
         ->  Index Scan using contentcuration_contentnode_pkey on contentcuration_contentnode  (cost=0.56..8.55 rows=1 width=47) (actual time=0.017..0.017 rows=1 loops=29)
               Index Cond: ((contentcuration_channel.main_tree_id)::text = (id)::text)
         SubPlan 1
           ->  Nested Loop  (cost=0.70..12.75 rows=1 width=0) (never executed)
                 ->  Index Only Scan using contentcuration_user_pkey on contentcuration_user u0_1  (cost=0.29..4.30 rows=1 width=4) (never executed)
                       Index Cond: (id = 28085)
                       Heap Fetches: 0
                 ->  Index Scan using contentcuration_channel_editors_channel_id_a6a01379_like on contentcuration_channel_editors u1_1  (cost=0.41..8.43 rows=1 width=4) (never executed)
                       Index Cond: ((channel_id)::text = (contentcuration_channel.id)::text)
                       Filter: (user_id = 28085)
         SubPlan 2
           ->  Nested Loop  (cost=0.57..13.68 rows=2 width=32) (actual time=0.005..0.014 rows=29 loops=1)
                 ->  Index Only Scan using contentcuration_user_pkey on contentcuration_user u0_2  (cost=0.29..4.30 rows=1 width=4) (actual time=0.002..0.003 rows=1 loops=1)
                       Index Cond: (id = 28085)
                       Heap Fetches: 1
                 ->  Index Scan using contentcuration_channel_editors_e8701ad4 on contentcuration_channel_editors u1_2  (cost=0.29..9.36 rows=2 width=36) (actual time=0.002..0.009 rows=29 loops=1)
                       Index Cond: (user_id = 28085)
         SubPlan 3
           ->  Nested Loop  (cost=0.56..12.61 rows=1 width=0) (never executed)
                 ->  Index Only Scan using contentcuration_user_pkey on contentcuration_user u0_3  (cost=0.29..4.30 rows=1 width=4) (never executed)
                       Index Cond: (id = 28085)
                       Heap Fetches: 0
                 ->  Index Scan using contentcuration_channel_viewers_e8701ad4 on contentcuration_channel_viewers u1_3  (cost=0.28..8.30 rows=1 width=4) (never executed)
                       Index Cond: (user_id = 28085)
                       Filter: ((channel_id)::text = (contentcuration_channel.id)::text)
         SubPlan 4
           ->  Nested Loop  (cost=0.56..12.61 rows=1 width=32) (actual time=0.007..0.007 rows=0 loops=1)
                 ->  Index Only Scan using contentcuration_user_pkey on contentcuration_user u0_4  (cost=0.29..4.30 rows=1 width=4) (actual time=0.001..0.001 rows=1 loops=1)
                       Index Cond: (id = 28085)
                       Heap Fetches: 1
                 ->  Index Scan using contentcuration_channel_viewers_e8701ad4 on contentcuration_channel_viewers u1_4  (cost=0.28..8.29 rows=1 width=37) (actual time=0.006..0.006 rows=0 loops=1)
                       Index Cond: (user_id = 28085)
         SubPlan 5
           ->  Limit  (cost=19103.36..19103.37 rows=1 width=8) (actual time=0.920..0.920 rows=1 loops=29)
                 ->  Sort  (cost=19103.36..19113.85 rows=4196 width=8) (actual time=0.918..0.918 rows=1 loops=29)
                       Sort Key: u0_5.modified DESC
                       Sort Method: top-N heapsort  Memory: 25kB
->  Index Scan using contentcuration_contentnode_656442a0 on contentcuration_contentnode u0_5  (cost=0.56..19082.38 rows=4196 width=8) (actual time=0.012..0.851 rows=563 loops=29)
                             Index Cond: (tree_id = contentcuration_contentnode.tree_id)
         SubPlan 6
           ->  Limit  (cost=28.48..28.48 rows=1 width=21) (actual time=0.021..0.021 rows=0 loops=29)
                 ->  Sort  (cost=28.48..28.48 rows=1 width=21) (actual time=0.020..0.020 rows=0 loops=29)
                       Sort Key: u0_6.token DESC
                       Sort Method: quicksort  Memory: 25kB
                       ->  Nested Loop  (cost=4.72..28.47 rows=1 width=21) (actual time=0.015..0.016 rows=0 loops=29)
                             ->  Bitmap Heap Scan on contentcuration_channel_secret_tokens u1_5  (cost=4.43..11.84 rows=2 width=4) (actual time=0.007..0.007 rows=1 loops=29)
                                   Recheck Cond: ((channel_id)::text = (contentcuration_channel.id)::text)
                                   Heap Blocks: exact=12
                                   ->  Bitmap Index Scan on contentcuration_channel_secret_tokens_channel_id_23ae1e4d_like  (cost=0.00..4.43 rows=2 width=0) (actual time=0.005..0.005 rows=1 loops=29)
                                         Index Cond: ((channel_id)::text = (contentcuration_channel.id)::text)
                             ->  Index Scan using contentcuration_secrettoken_pkey on contentcuration_secrettoken u0_6  (cost=0.29..8.30 rows=1 width=25) (actual time=0.003..0.004 rows=0 loops=24)
                                   Index Cond: (id = u1_5.secrettoken_id)
                                   Filter: is_primary
                                   Rows Removed by Filter: 0
         SubPlan 7
           ->  Aggregate  (cost=19100.71..19100.72 rows=1 width=8) (actual time=0.596..0.596 rows=1 loops=29)
                 ->  Index Scan using contentcuration_contentnode_656442a0 on contentcuration_contentnode u0_7  (cost=0.56..19092.87 rows=3136 width=33) (actual time=0.009..0.555 rows=466 loops=29)
                       Index Cond: (tree_id = contentcuration_contentnode.tree_id)
                       Filter: (((kind_id)::text <> 'topic'::text) OR (kind_id IS NULL))
                       Rows Removed by Filter: 97
         SubPlan 9
           ->  Index Scan using contentcuration_change_channel_id_35b185d1 on contentcuration_change v0  (cost=889.59..2035.41 rows=96 width=0) (actual time=0.390..0.390 rows=0 loops=29)
                 Index Cond: ((channel_id)::text = (contentcuration_channel.id)::text)
                 Filter: ((created_by_id IS NOT NULL) AND (user_id IS NULL) AND (server_rev > COALESCE($9, '0'::bigint)))
                 Rows Removed by Filter: 367
                 InitPlan 8 (returns $9)
                   ->  Limit  (cost=889.17..889.17 rows=1 width=8) (actual time=1.189..1.190 rows=1 loops=5)
                         ->  Sort  (cost=889.17..889.17 rows=1 width=8) (actual time=1.187..1.187 rows=1 loops=5)
                               Sort Key: u0_8.server_rev DESC
                               Sort Method: quicksort  Memory: 25kB
                               ->  Bitmap Heap Scan on contentcuration_change u0_8  (cost=18.69..889.16 rows=1 width=8) (actual time=0.596..1.171 rows=2 loops=5)
                                     Recheck Cond: ((channel_id)::text = (contentcuration_channel.id)::text)
                                     Filter: ((NOT errored) AND (change_type = 6))
                                     Rows Removed by Filter: 2123
                                     Heap Blocks: exact=1126
                                     ->  Bitmap Index Scan on contentcuration_change_channel_id_35b185d1_like  (cost=0.00..18.69 rows=303 width=0) (actual time=0.234..0.234 rows=2166 loops=5)
                                           Index Cond: ((channel_id)::text = (contentcuration_channel.id)::text)
 Planning time: 2.031 ms
 Execution time: 57.478 ms

References

#3835


Contributor's Checklist

PR process:

  • If this is an important user-facing change, PR or related issue the CHANGELOG label been added to this PR. Note: items with this label will be added to the CHANGELOG at a later time
  • If this includes an internal dependency change, a link to the diff is provided
  • The docs label has been added if this introduces a change that needs to be updated in the user docs?
  • If any Python requirements have changed, the updated requirements.txt files also included in this PR
  • Opportunities for using Google Analytics here are noted
  • Migrations are safe for a large db

Studio-specifc:

  • All user-facing strings are translated properly
  • The notranslate class been added to elements that shouldn't be translated by Google Chrome's automatic translation feature (e.g. icons, user-generated text)
  • All UI components are LTR and RTL compliant
  • Views are organized into pages, components, and layouts directories as described in the docs
  • Users' storage used is recalculated properly on any changes to main tree files
  • If there new ways this uses user data that needs to be factored into our Privacy Policy, it has been noted.

Testing:

  • Code is clean and well-commented
  • Contributor has fully tested the PR manually
  • If there are any front-end changes, before/after screenshots are included
  • Critical user journeys are covered by Gherkin stories
  • Any new interactions have been added to the QA Sheet
  • Critical and brittle code paths are covered by unit tests

Reviewer's Checklist

This section is for reviewers to fill out.

  • Automated test coverage is satisfactory
  • PR is fully functional
  • PR has been tested for accessibility regressions
  • External dependency files were updated if necessary (yarn and pip)
  • Documentation is updated
  • Contributor is in AUTHORS.md

@bjester bjester requested a review from rtibbles November 23, 2022 18:37
Copy link
Member

@rtibbles rtibbles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes make sense and the endpoint has coverage. The new query performance speaks for itself.

@bjester bjester merged commit 2b1f10b into learningequality:hotfixes Nov 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants