Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misleading removed images count. #282

Open
SmartManoj opened this issue Jan 10, 2025 · 0 comments
Open

Misleading removed images count. #282

SmartManoj opened this issue Jan 10, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@SmartManoj
Copy link
Contributor

SmartManoj commented Jan 10, 2025

Describe the bug

clean is false, previously built instances images will not be removed, but instance images built
in the current run will be removed.

The above one says it will remove instance images but it is already removed in run_instance itself.

if rm_image:
remove_image(client, test_spec.instance_image_key, logger)

Should the count be removed or should include that previous removal count using env var?

Steps/Code to Reproduce

python -m swebench.harness.run_evaluation \
    --predictions_path gold \
    --max_workers 1 \
    --run_id gold \
    --split test \
    --dataset_name princeton-nlp/SWE-bench \
    --instance_ids sympy__sympy-21612

Expected Results

Removed 1 image(s).

Actual Results

Removed 0 images.

System Information

No response

@SmartManoj SmartManoj added the bug Something isn't working label Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant