Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup CPU profiling and optimize code #421

Open
arostamianfar opened this issue Nov 26, 2018 · 0 comments
Open

Setup CPU profiling and optimize code #421

arostamianfar opened this issue Nov 26, 2018 · 0 comments

Comments

@arostamianfar
Copy link
Contributor

We have not really spent too much time optimizing our code, but it seems like there may be opportunities to gain performance. For instance, a small change in PR #417 resulted in 50% speedup in one of the PTransforms (~10% overall speedup; or more depending on the size of the data).

However, rather than trying 'random' things, it's better to run a CPU profiler on our code to determine the places where it makes sense to optimize (e.g. I have found that sanitizing the BQ fields also takes a nontrivial amount of time). Beam has some flags to setup profiling (see thread).

We can also consider using Cython once we know the areas to optimize.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant