Gpt 2 sweep

[tomMcGrath]

Description

Adds base sweep settings for gpt-2-small. Sweeps are intentionally relatively short: most of the action that tells us whether a sweep will be obviously bad seems to happen relatively quickly. Based on Anthropic SAE replication.

W&B dashboard for this run.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility

You have tested formatting, typing and unit tests (acceptance tests not currently in use)

• I have run make check-ci to check format and linting. (you can run make format to format code if needed.)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.21%. Comparing base (bdef2cf) to head (af93303).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #140   +/-   ##
=======================================
  Coverage   72.21%   72.21%           
=======================================
  Files          17       17           
  Lines        1814     1814           
  Branches      295      295           
=======================================
  Hits         1310     1310           
  Misses        432      432           
  Partials       72       72           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hijohnnylin]

looks good! maybe i should make this the default for the ansible example run? (@jbloomAus)

also was 200_00 intentional for training steps?

SAELens/scripts/sweep-gpt2.py

Line 23 in af892ae

total_training_steps = 200_00

[tomMcGrath]

Whoops that’s a typo!

…

On Sun, 12 May 2024 at 19:49, johnny ***@***.***> wrote: looks good! maybe i should make this the default for the ansible example run? ***@***.*** <https://github.com/jbloomAus>) also was 200_00 intentional for training steps? https://github.com/jbloomAus/SAELens/blob/af892aec9e8f280aaaee09feb472274f1aafe20e/scripts/sweep-gpt2.py#L23 — Reply to this email directly, view it on GitHub <#140 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALGU43UQAH2G2CLNID5SSDZB62THAVCNFSM6AAAAABHTAOCQ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBWGM2DEMZYGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jbloomAus]

@tomMcGrath will know better if this is a reasonable example run. My guess is not because of we run this a lot paralleling by a large number will use more compute.