Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature(dspy): Copro optimizer prompts are now injectable #942

Closed
wants to merge 5 commits into from
Closed

feature(dspy): Copro optimizer prompts are now injectable #942

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c3d7770
Make copro prompt injectable
tobicoveo May 1, 2024
8c8375d
Change signature injection to additional instructions + added changes…
tobicoveo May 6, 2024
6b666ef
Made signature properity methods
tobicoveo May 6, 2024
9fc4593
Added append_instructions for code simplification
tobicoveo May 6, 2024
38946af
Run ruff
tobicoveo May 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 27 additions & 6 deletions dspy/teleprompt/copro_optimizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
* results_latest: The min,max,avg,stddev of newest prompt scores for each predictor at each depth.
* total_calls: The total number of calls to the task metric.
These statistics will be returned as attributes of the best program.
* additional_instructions: Instructions appended to the generation signatures. Can be used to provide explicit details on the optimization metric.
"""


Expand Down Expand Up @@ -63,6 +64,7 @@ def __init__(
depth=3,
init_temperature=1.4,
track_stats=False,
additional_instructions=None,
**_kwargs,
):
if breadth <= 1:
Expand All @@ -74,8 +76,25 @@ def __init__(
self.prompt_model = prompt_model
self.track_stats = track_stats

self.basic_generate_instruction = (
self._get_signature(BasicGenerateInstruction).with_instructions(
" ".join([BasicGenerateInstruction.instructions, additional_instructions])
)
if additional_instructions
else BasicGenerateInstruction
)
self.generate_instruction_given_attempts = (
self._get_signature(GenerateInstructionGivenAttempts).with_instructions(
" ".join([GenerateInstructionGivenAttempts.instructions, additional_instructions])
)
if additional_instructions
else GenerateInstructionGivenAttempts
)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to update the fields, but the instructions. Unless I am mistaken, the Signature class doesn't have a method for this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry - that's what with_instructions does!


if "verbose" in _kwargs:
dspy.logger.warning("DeprecationWarning: 'verbose' has been deprecated. To see all information for debugging, use 'dspy.set_log_level('debug')'. In the future this will raise an error.")
dspy.logger.warning(
"DeprecationWarning: 'verbose' has been deprecated. To see all information for debugging, use 'dspy.set_log_level('debug')'. In the future this will raise an error."
)

def _check_candidates_equal(self, candidate1, candidate2):
for p1, p2 in zip(candidate1["program"].predictors(), candidate2["program"].predictors()):
Expand Down Expand Up @@ -153,13 +172,13 @@ def compile(self, student, *, trainset, eval_kwargs):
if self.prompt_model:
with dspy.settings.context(lm=self.prompt_model):
instruct = dspy.Predict(
BasicGenerateInstruction,
self.basic_generate_instruction,
n=self.breadth - 1,
temperature=self.init_temperature,
)(basic_instruction=basic_instruction)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These calls to Predict, and later attempts to access members of instruct place pretty strict requirements on the acceptable signatures - maybe more flexible to allow the partial-creation of the entire instruct object?

Think a protocol or the like which specifies the signature of the signature you pass in is probably a must to allow users to understand exactly what they can put in here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @mikeedjones, yeah I agree it was a little bit too loose given the requirements of the signatures. Instead of making the signature injectable, I allowed the user to add instructions and create the signatures from the base class.

Let me know if you see an issue with this approach.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does seem much more sensible - but maybe worth adding a method to dspy.Signature to allow this sort of user injection of instructions into predefined signatures?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah - that's what "with_instructions" does!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I want to preserve the initial instructions as well; it would be something more towards the line of append_to_instructions.

Copy link
Contributor

@mikeedjones mikeedjones May 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it would be more straightforward for users to specify a protocol which defines the signature of an injectable BasicGenerateInstruction and then leave it to the user to grab the original instructions from this file, if they want to append to the originals?

I think there's a wider discussion on how to manage the internal prompts of dspy and how to enable user access to them?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think down the line, it would make sense to allow the user to change the whole instructions. One could even consider some kind of meta-learning approach on the instructions of the signature used to generate the other prompts (i.e., what signatures lead to the best prompt under a few-shot learning approach).

There should be a discussion about enabling users to access these signatures safely IMO.

I am merely starting to get familiar with this library and wanted to keep the changes very scoped in the context of this PR. 😄

Copy link
Contributor

@mikeedjones mikeedjones May 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the concern would be making a small change here actually alters the interface to Copro, which then has to be supported until a breaking 3.x release. Maybe better to figure out how to more modify the internal prompts more generally, and then apply that here?

else:
instruct = dspy.Predict(
BasicGenerateInstruction,
self.basic_generate_instruction,
n=self.breadth - 1,
temperature=self.init_temperature,
)(basic_instruction=basic_instruction)
Expand Down Expand Up @@ -299,19 +318,21 @@ def compile(self, student, *, trainset, eval_kwargs):
if self.prompt_model:
with dspy.settings.context(lm=self.prompt_model):
instr = dspy.Predict(
GenerateInstructionGivenAttempts,
self.generate_instruction_given_attempts,
n=self.breadth,
temperature=self.init_temperature,
)(attempted_instructions=attempts)
else:
instr = dspy.Predict(
GenerateInstructionGivenAttempts,
self.generate_instruction_given_attempts,
n=self.breadth,
temperature=self.init_temperature,
)(attempted_instructions=attempts)

if self.prompt_model:
dspy.logger.debug(f"(self.prompt_model.inspect_history(n=1)) {self.prompt_model.inspect_history(n=1)}")
dspy.logger.debug(
f"(self.prompt_model.inspect_history(n=1)) {self.prompt_model.inspect_history(n=1)}"
)
# Get candidates for each predictor
new_candidates[id(p_base)] = instr.completions
all_candidates[id(p_base)].proposed_instruction.extend(instr.completions.proposed_instruction)
Expand Down
42 changes: 38 additions & 4 deletions dspy/teleprompt/mipro_optimizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
* score: the last average evaluated score for the program
* pruned: whether or not this program was pruned
This information will be returned as attributes of the best program.
* additional_instructions: Instructions appended to the generation signatures. Can be used to provide explicit details on the optimization metric.
"""


Expand Down Expand Up @@ -142,6 +143,7 @@ def __init__(
verbose=False,
track_stats=True,
view_data_batch_size=10,
additional_instructions=None
):
self.num_candidates = num_candidates
self.metric = metric
Expand All @@ -153,6 +155,38 @@ def __init__(
self.teacher_settings = teacher_settings
self.view_data_batch_size = view_data_batch_size

self.basic_generate_instruction = (
self._get_signature(BasicGenerateInstruction).with_instructions(
" ".join([BasicGenerateInstruction.instructions, additional_instructions])
)
if additional_instructions
else BasicGenerateInstruction
)

self.generate_instruction_with_data_observations = (
self._get_signature(BasicGenerateInstructionWithDataObservations).with_instructions(
" ".join([BasicGenerateInstructionWithDataObservations.instructions, additional_instructions])
)
if additional_instructions
else BasicGenerateInstructionWithDataObservations
)

self.generate_instruction_with_examples = (
self._get_signature(BasicGenerateInstructionWithExamples).with_instructions(
" ".join([BasicGenerateInstructionWithExamples.instructions, additional_instructions])
)
if additional_instructions
else BasicGenerateInstructionWithExamples
)

self.generate_instruction_with_examples_and_observations = (
self._get_signature(BasicGenerateInstructionWithExamplesAndDataObservations).with_instructions(
" ".join([BasicGenerateInstructionWithExamplesAndDataObservations.instructions, additional_instructions])
)
if additional_instructions
else BasicGenerateInstructionWithExamplesAndDataObservations
)

def _print_full_program(self, program):
for i, predictor in enumerate(program.predictors()):
if self.verbose:
Expand Down Expand Up @@ -282,7 +316,7 @@ def _generate_first_N_candidates( # noqa: N802
instruct = None
for i in range(1, self.num_candidates):
new_instruct = dspy.Predict(
BasicGenerateInstructionWithExamplesAndDataObservations,
self.generate_instruction_with_examples_and_observations,
n=1,
temperature=self.init_temperature,
)(
Expand All @@ -302,7 +336,7 @@ def _generate_first_N_candidates( # noqa: N802
# Just data
elif view_data:
instruct = dspy.Predict(
BasicGenerateInstructionWithDataObservations,
self.generate_instruction_with_data_observations,
n=N - 1,
temperature=self.init_temperature,
)(basic_instruction=basic_instruction, observations=self.observations)
Expand All @@ -311,7 +345,7 @@ def _generate_first_N_candidates( # noqa: N802
instruct = None
for i in range(1, self.num_candidates): # Note: skip over the first example set which is empty
new_instruct = dspy.Predict(
BasicGenerateInstructionWithExamples,
self.generate_instruction_with_examples,
n=1,
temperature=self.init_temperature,
)(
Expand All @@ -329,7 +363,7 @@ def _generate_first_N_candidates( # noqa: N802
)
# Neither
else:
instruct = dspy.Predict(BasicGenerateInstruction, n=N - 1, temperature=self.init_temperature)(
instruct = dspy.Predict(self.basic_generate_instruction, n=N - 1, temperature=self.init_temperature)(
basic_instruction=basic_instruction,
)

Expand Down