Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Pizza oven database optimizations #39

Merged
merged 1 commit into from
Sep 1, 2023

Conversation

jpmcb
Copy link
Member

@jpmcb jpmcb commented Aug 30, 2023

Description

This optimizes the database calls within the pizza oven.

Instead of iterating and inserting each author and commit, this uses methods to bulk insert those records:

  • Each author is added to a transaction. When that transaction is executed and if there is a conflict, the transaction for that record "does nothing". This takes 1 call to the database.
    • The commit authors are re-queried in order to build a map of inserted authors emails and IDs. These IDs are used when bulk inserting into the commits table.
  • Each commit is added to a transaction. When the transaction is executed, each commit is inserted.

On some preliminary testing using the freecodecamp/freecodecamp repo (which as about ~30,000 commits), each with a fresh local database (so network latency should be negligible):

Using main:

❯ time pizza bake -e http://localhost:8080 https://github.com/freecodecamp/freecodecamp --wait
Resp body:
pizza bake -e http://localhost:8080  --wait  0.01s user 0.02s system 0% cpu 1:17.42 total

Using this branch:

❯ time pizza bake -e http://localhost:8080 https://github.com/freecodecamp/freecodecamp --wait
Resp body:
pizza bake -e http://localhost:8080  --wait  0.01s user 0.02s system 0% cpu 3.322 total

which ends up being about a 24x faster.

What type of PR is this? (check all applicable)

  • 🍕 Feature
  • 🐛 Bug Fix
  • 📝 Documentation Update
  • 🎨 Style
  • 🧑‍💻 Code Refactor
  • 🔥 Performance Improvements
  • ✅ Test
  • 🤖 Build
  • 🔁 CI
  • 📦 Chore (Release)
  • ⏩ Revert

Related Tickets & Documents

N/a - related to doing some scaling and deployments after integrating with some of our internal services.

Mobile & Desktop Screenshots/Recordings

N/a

Added tests?

  • 👍 yes
  • 🙅 no, because they aren't needed
  • 🙋 no, because I need help

Added to documentation?

  • 📜 README.md
  • 📓 docs.opensauced.pizza
  • 🍕 dev.to/opensauced
  • 📕 storybook
  • 🙅 no documentation needed

[optional] Are there any post-deployment tasks we need to perform?

[optional] What gif best describes this PR or how it makes you feel?

This optimizes the database calls within the pizza oven. Instead
of iterating and inserting each author and commit, one after another,
this uses methods to bulk insert both commit authors and commits.

Signed-off-by: John McBride <[email protected]>
@jpmcb jpmcb force-pushed the pizza-optimizations branch from 7b40f46 to 6f9639f Compare August 31, 2023 00:50
@jpmcb jpmcb marked this pull request as ready for review August 31, 2023 00:51
Copy link
Member

@bdougie bdougie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for another great internal demo. It was helpful in grokking what is going on here.

Copy link

@brandonroberts brandonroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blazingly fast 🔥

@jpmcb jpmcb merged commit 80d7f88 into open-sauced:main Sep 1, 2023
@jpmcb jpmcb deleted the pizza-optimizations branch September 1, 2023 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants