Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(evaluate): Evaluate should not error when display_table=True #990

Merged

Conversation

christeefy
Copy link
Contributor

@christeefy christeefy commented May 7, 2024

Obtained an error when display_table=True is provided for Evaluate.
See simple reproducible example below:

from dspy.evaluate.evaluate import Evaluate
from dspy.evaluate.metrics import answer_exact_match
from dspy.predict import Predict

def new_example(question, answer):
    """Helper function to create a new example."""
    return dspy.Example(
        question=question,
        answer=answer,
    ).with_inputs("question")

devset = [new_example("What is 1+1?", "2")]

program = Predict("question -> answer")

ev = Evaluate(
    devset=devset,
    metric=answer_exact_match,
    display_table=True,
)
ev(program)  # TypeError: cannot do positional indexing on RangeIndex with these indexers [True] of type bool

This stems from isinstance(True, int) == True.

This PR contains a proposed fix with updated unit tests. Feel free to re-structure the unit tests as you see fit.

@christeefy christeefy changed the title fix(evaluate): Evaluation should not error when display_table=True fix(evaluate): Evaluate should not error when display_table=True May 7, 2024
@arnavsinghvi11
Copy link
Collaborator

Thanks @christeefy ! great find!

@arnavsinghvi11 arnavsinghvi11 merged commit 32449e1 into stanfordnlp:main May 11, 2024
4 checks passed
@christeefy christeefy deleted the bug/evaluate-display-table-arg branch May 21, 2024 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants