You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description:
The FAIL_TO_PASS and PASS_TO_PASS fields of some instances contain unrelated strings instead of references to unit tests.
An example is provided below. django__django-16950 has ["If form data is provided, a parent's auto-generated alternate key is"] as its FAIL_TO_PASS and some comments as its PASS_TO_PASS
I haven't checked thoroughly, but I can see django__django-15525 and django__django-14792 also have the same problem. Not sure if this will affect the actual evaluation of swebench.
I think this is the legitimate name of a test, for instance here's the PASS_TO_PASStest referenced in the image.
I'm not sure this needs fixing. Django has its own custom testing software iirc (it doesn't use pytest). From when I last ran, I think the test name + result is printed out (e.g. <test name> ... ok or <test name> ... fail.
Leaving this open for discussion. My current stance is that this doesn't need fixing + is the expected behavior for Django testing.
Describe the bug
Description:
The
FAIL_TO_PASS
andPASS_TO_PASS
fields of some instances contain unrelated strings instead of references to unit tests.An example is provided below.
django__django-16950
has ["If form data is provided, a parent's auto-generated alternate key is"] as itsFAIL_TO_PASS
and some comments as itsPASS_TO_PASS
I haven't checked thoroughly, but I can see
django__django-15525
anddjango__django-14792
also have the same problem. Not sure if this will affect the actual evaluation of swebench.Steps/Code to Reproduce
Buggy Example: django__django-16950
Expected Results
The actual unit tests
Actual Results
As described above
System Information
No response
The text was updated successfully, but these errors were encountered: