Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed #1258

Merged
merged 16 commits into from
Sep 11, 2019
Merged

Speed #1258

merged 16 commits into from
Sep 11, 2019

Conversation

greenape
Copy link
Member

Towards #1256 but doesn't solve it.

I have:

  • Formatted any Python files with black
  • Brought the branch up to date with master
  • Added any relevant Github labels
  • Added tests for any new additions
  • Added or updated any relevant documentation
  • Added an Architectural Decision Record (ADR), if appropriate
  • Added an MPLv2 License Header if appropriate
  • Updated the Changelog

Description

Several small tweaks which speed up creating query objects.

  • Runs available_dates when first connecting to FlowDB to pre-create Table objects
  • Runs available_dates periodically as a background thread in the server to keep that fresh
  • Skips creating the state machine transitions if the state machine already exists
  • Skips fast forwarding Table state machines if they're already complete
  • Skips dumping a pickle if the object is already in cache
  • Removes an inline import of __version__ in write_cache_metadata - originally necessary to avoid a circular import, but not now
  • Removes a call to pg_notify that isn't used

@greenape greenape added FlowMachine Issues related to FlowMachine performance ready-to-merge Label indicating a PR is OK to automerge labels Sep 11, 2019
@cypress
Copy link

cypress bot commented Sep 11, 2019



Test summary

54 0 0 0


Run details

Project FlowAuth
Status Passed
Commit 3277b5a
Started Sep 11, 2019 1:55 PM
Ended Sep 11, 2019 1:58 PM
Duration 02:42 💡
OS Linux Debian - 8.11
Browser Electron 61

View run in Cypress Dashboard ➡️


This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@codecov
Copy link

codecov bot commented Sep 11, 2019

Codecov Report

Merging #1258 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1258      +/-   ##
==========================================
- Coverage   94.16%   94.15%   -0.02%     
==========================================
  Files         154      154              
  Lines        7478     7493      +15     
  Branches      698      702       +4     
==========================================
+ Hits         7042     7055      +13     
- Misses        330      331       +1     
- Partials      106      107       +1
Flag Coverage Δ
#flowapi_unit_tests 82.55% <100%> (ø) ⬆️
#flowauth_unit_tests 93.65% <ø> (ø) ⬆️
#flowclient_unit_tests 78.11% <ø> (ø) ⬆️
#flowetl_unit_tests 96.63% <ø> (ø) ⬆️
#flowkit_jwt_generator_unit_tests 100% <ø> (ø) ⬆️
#flowmachine_unit_tests 90.7% <85.71%> (-0.13%) ⬇️
#integration_tests 66.88% <86.36%> (+0.06%) ⬆️
Impacted Files Coverage Δ
flowmachine/flowmachine/core/connection.py 93.07% <ø> (-0.06%) ⬇️
flowmachine/flowmachine/core/table.py 90.8% <100%> (+0.1%) ⬆️
flowmachine/flowmachine/core/query_state.py 100% <100%> (ø) ⬆️
flowapi/flowapi/main.py 100% <100%> (ø) ⬆️
flowmachine/flowmachine/core/cache.py 96.31% <100%> (+0.01%) ⬆️
flowmachine/flowmachine/core/init.py 100% <100%> (ø) ⬆️
flowmachine/flowmachine/core/server/server.py 89.58% <100%> (+1.48%) ⬆️
flowmachine/flowmachine/core/logging.py 94.73% <0%> (-5.27%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fec104e...3277b5a. Read the comment docs.

@@ -275,9 +275,6 @@ def _known_dates(self, table, schema):
if x.isnumeric()
)

@cached(
TTLCache(1024, 120)
) # Many dates to cache, two minutes seems reasonable to balance db access against speed boost
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for my understanding, what's the reason for removing this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to record the reasoning - this function is only called by available_dates, which is itself cached. The periodic update function added to server.py skips the cache for available_dates so it will always hit the db and create any table objects needed. That isn't guaranteed if this method is also cached.

Copy link
Member

@jc-harrison jc-harrison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One typo. Otherwise this looks good.

flowapi/tests/unit/test_access_control.py Outdated Show resolved Hide resolved
Copy link
Member

@jc-harrison jc-harrison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@mergify mergify bot merged commit bd5ddee into master Sep 11, 2019
@mergify mergify bot deleted the speed branch September 11, 2019 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FlowMachine Issues related to FlowMachine performance ready-to-merge Label indicating a PR is OK to automerge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants