Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mpy size actions #197

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Mpy size actions #197

wants to merge 8 commits into from

Conversation

FoamyGuy
Copy link
Contributor

@FoamyGuy FoamyGuy commented Aug 18, 2022

This change adds two actions task inside the generated library repo that will post a comment on PRs with information about mpy file sizes.

This is an example of the comments that it posts:
FoamyGuy/Adafruit_CircuitPython_SI1145#1 (comment)

Some outstanding things to consider:

  • The script that finds the sizes and compiles them into a suitable string is housed in this repo currently: https://github.com/FoamyGuy/CircuitPython_Memory_Tools/blob/main/measure_size.py it could be moved to adabot, build-tools, or somewhere else, maybe a new repo under adafruit? IT could also maybe be refactored into a proper github actions that can be run with uses: instead of launched via command line as it is now.
  • Do we want to change the wording any? I wrote the current ones, but there may be room for improvement. Do we want any other stats in the comment? The difference between sizes of each version perhaps?
  • Should it print any of the size checking or strings output into the actions output? Currently there are minimal prints that make it to actions, most things are in the comment rather than the actions output.
  • Should the comment be conditional on sizes being a big enough difference?
  • Probably would be good to test measure_size.py against more libraries. I've done one of each: package and module, but not much else. There may be library structures that confuse it.

@FoamyGuy FoamyGuy requested a review from a team August 18, 2022 00:55
@tekktrik
Copy link
Member

Nice! I have a couple of comments:

  • I think conditional statements would be good. I could see this become an issue if people push single commits, and it breaks up comments.
  • Alternatively, using the something like the annotations that the problem matchers uses would allow you to publish the information for reviewers without crowding the comments: https://github.com/adafruit/Adafruit_CircuitPython_LIFX/actions/runs/2664918272
  • I could also see it printing it only in the Actions for reviewers to see. It's worth parsing through if reviewers see things that would indicate memory changes.
  • I think notifying (however its done) about the amount of memory taken by strings is good, because that's a common memory improvement point, potentially.

@@ -54,6 +54,17 @@ jobs:
pre-commit run --all-files
- name: Build assets
run: circuitpython-build-bundles --filename_prefix {% raw %}${{ steps.repo-name.outputs.repo-name }}{% endraw -%} {% if cookiecutter.target_bundle == 'Community' %} --package_folder_prefix {% if cookiecutter.library_prefix -%} {{ cookiecutter.library_prefix | lower | replace(' ', '_') }}_ {%- else %}""{%- endif -%}{%- endif %} --library_location .
- name: Check Sizes
run: |
git clone https://github.com/FoamyGuy/CircuitPython_Memory_Tools.git
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should that move to https://github.com/circuitpython ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do like that idea. I'll cookiecut a new one to get the relavent boilerplate stuff and move this code into it, then put it in the circuitpython org. Probably sometime tomorrow I can work on that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still need to update this PR to point to the new one. Will make a new commit with that once I've got the conditions mentioned in the other comment worked out.

@FoamyGuy
Copy link
Contributor Author

  • I think conditional statements would be good. I could see this become an issue if people push single commits, and it breaks up comments.

The comment task does have a condition that makes it run only for the "pull request" event. So a direct commit that hasn't been made into a PR yet won't trigger it. Individual commits within the PR do trigger it to comment for each of them if they are all pushed individually.

What other conditions could we use to limit it? I think we want to see the changes caused by new commits so re-commenting each time allows that, although it could make the threads quite long if there are lots of commits.

  • Alternatively, using the something like the annotations that the problem matchers uses would allow you to publish the information for reviewers without crowding the comments: https://github.com/adafruit/Adafruit_CircuitPython_LIFX/actions/runs/2664918272
  • I could also see it printing it only in the Actions for reviewers to see. It's worth parsing through if reviewers see things that would indicate memory changes.

We discussed a few options during weeds discussion in the meeting and consensus arrived to at that time was to have it comment instead of only output into the actions log.

@tekktrik
Copy link
Member

We discussed a few options during weeds discussion in the meeting and consensus arrived to at that time was to have it comment instead of only output into the actions log.

Gotchya, that makes sense. In that case I would love to see it comment only when a few things happen:

  • The memory used changes by a specific amount from existing (like 5% change in either direction for memory)
  • If the baseline existing is already at a certain level (like if its above X bytes it always prints)
  • If the string percentage is too high (e.g., 50% is strings)

That would keep it from printing too much but doing it's job when needed. Those numbers above are just examples, we'd need to baseline them across different repos if we went that route.

I also think have a small tool that people can run locally (similar to pre-commit) would be useful for being able to test this if these conditions aren't met but people want to test locally.

@FoamyGuy
Copy link
Contributor Author

The latest commits move this to use the size_tools repo in the circuitpython org as well as add the conditions mentioned.

The initial values were a bit of a guess and we may want to tweak them some before making it active. They're defined here:

https://github.com/circuitpython/CircuitPython_Org_size_tools/blob/79b7603670587d8a710855584a63754868be08f1/size_tools.py#L124-L126

I have done the majority of my testing on a single library (SI1145) it's a bit over the 1500 byte size that I set the baseline value to, so it does get commented on that repo. Here is the latest comment: FoamyGuy/Adafruit_CircuitPython_SI1145#1 (comment)

The comment format is unchanged, but there is now logic to create the output file based on the value of the conditions, and if no file is output then it won't make the comment.

@tekktrik
Copy link
Member

This is fantastic!!!

if: github.event_name == 'pull_request' && steps.check_sizes_file.outputs.files_exists == 'true'
uses: machine-learning-apps/pr-comment@master
env:
GITHUB_TOKEN: {% raw %}${{ secrets.GITHUB_TOKEN }}{% endraw %}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest commit removes the extra space here. This was copied from another similar usage of the token which has now been removed by #202

@tekktrik
Copy link
Member

I really, really like this idea, I think it's worth rolling out and patching to the libraries. I can help with generating an adabot patch when it gets to that point!

@FoamyGuy
Copy link
Contributor Author

Feedback from discussion during "in the weeds" of the meeting on 9/19/22 raised the idea to make the "current size" values get pulled from main branch code specifically instead of the mpy files inside the published bundles.

This would make it print information relevant to each specific PR in cases where there are multiple PRs included within a single release cycle. I.e in this scenario:

  • PR 1 adds 1k size, it gets reviewed and merged. The actions would comment with the size change.
  • PR 2 adds 8 bytes and occurs after PR 1 is merged, but before a release as been made.

With current logic the comment on PR 2 would indicate that size changed by 1.08k since it's referencing the released mpy file rather than the previous version of main which would have included PR 1.

With proposed new logic, this would output +1k for PR 1 and would only try to comment +8 bytes on PR 2 which more accurately reflects the change specifically, based on the other existing logic it will decide if the change is "too small" to comment so may end up not posting depending on what those thresholds are configured to.

I will add further commits with this change.

@FoamyGuy
Copy link
Contributor Author

The latest commits change this to use the methodology mentioned in the prior comment.

It will now clone and build main branch to get the values to compare changed branch to, instead of using the published mpy files from the bundle.

I've also added a "summary" section to the comment that shows differences for the measured values to make the size impact easier to see at a glance.

An example of the latest version of the comment is available here: FoamyGuy/Adafruit_CircuitPython_SI1145#1 (comment)

@FoamyGuy FoamyGuy requested a review from a team September 24, 2022 16:45
@tekktrik
Copy link
Member

tekktrik commented Nov 8, 2022

Do we think this is ready, or do we want to try this out on a few PRs? I think it's good and approve but also think it's worth additional reviewers because it'll affect the PR process for everyone passively moving forward.

@tekktrik
Copy link
Member

Looks like this should be opened as a new PR to https://github.com/adafruit/workflows-circuitpython-libs/blob/main/build/action.yml

@dhalbert
Copy link
Contributor

Looks like this should be opened as a new PR to https://github.com/adafruit/workflows-circuitpython-libs/blob/main/build/action.yml

So should this be closed and opened as an issue on the repo linked above?

@tekktrik
Copy link
Member

Looks like this should be opened as a new PR to https://github.com/adafruit/workflows-circuitpython-libs/blob/main/build/action.yml

So should this be closed and opened as an issue on the repo linked above?

Yes! I can transfer it.

@tekktrik
Copy link
Member

Ooops this isn't an issue, but yes it should be. (@dhalbert, @FoamyGuy)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants