feature(dspy): Copro optimizer prompts are now injectable

[tobicoveo]

What:
Allow additional instruction to be added to the signatures used to generate the prompts.

Why:
Generally speaking, it allows the users more freedom regarding the instructions used to generate the prompts.

In our case, It unluck new possibilities to guide the generating prompt. Optimization based on metrics is already a powerful approach. However, being able to add details regarding how the prompt will be evaluated would help generate prompts that fulfill the requirements that must be filled out without relying only on the LLM to understand what is essential from the metrics score.

How:

• Simply added a field to append instructions to the base one in property methods.
• The changes are backward compatible from the None initialization.

[arnavsinghvi11]

Thanks for the PR @tobicoveo !
Small clarification that this ensures the DSPy signature is injectable, not the prompts exactly.

Could you please rebase and merge this branch with the latest version on main? Seems like there was a failing test on our end (nothing wrong with this PR) that just needs an update.

Also, could you add a couple lines of documentation to the Usage Suggestions, specifying what the injectable signatures are defaulted to and what they are used for, now that they are parameters? Ready to merge after that!

As an aside, seems like this can be extended to the MIPRO optimizer, if you'd like to tag that along with this PR! :)
(small note - that would require a bit refactoring in the test as well)

[mikeedjones]

These calls to Predict, and later attempts to access members of instruct place pretty strict requirements on the acceptable signatures - maybe more flexible to allow the partial-creation of the entire instruct object?

Think a protocol or the like which specifies the signature of the signature you pass in is probably a must to allow users to understand exactly what they can put in here.

[tobicoveo]

Hey @mikeedjones, yeah I agree it was a little bit too loose given the requirements of the signatures. Instead of making the signature injectable, I allowed the user to add instructions and create the signatures from the base class.

Let me know if you see an issue with this approach.

[mikeedjones]

It does seem much more sensible - but maybe worth adding a method to dspy.Signature to allow this sort of user injection of instructions into predefined signatures?

[mikeedjones]

Ah - that's what "with_instructions" does!

[tobicoveo]

But I want to preserve the initial instructions as well; it would be something more towards the line of append_to_instructions.

[mikeedjones]

Do you think it would be more straightforward for users to specify a protocol which defines the signature of an injectable BasicGenerateInstruction and then leave it to the user to grab the original instructions from this file, if they want to append to the originals?

I think there's a wider discussion on how to manage the internal prompts of dspy and how to enable user access to them?

[tobicoveo]

I think down the line, it would make sense to allow the user to change the whole instructions. One could even consider some kind of meta-learning approach on the instructions of the signature used to generate the other prompts (i.e., what signatures lead to the best prompt under a few-shot learning approach).

There should be a discussion about enabling users to access these signatures safely IMO.

I am merely starting to get familiar with this library and wanted to keep the changes very scoped in the context of this PR. 😄

[mikeedjones]

I guess the concern would be making a small change here actually alters the interface to Copro, which then has to be supported until a breaking 3.x release. Maybe better to figure out how to more modify the internal prompts more generally, and then apply that here?

[mikeedjones]

any benefit to not just having these types as the defaults and making basic_generate_instruction optional? BasicGenerateInstruction is a type and shouldn't be getting mutated at runtime.

[tobicoveo]

To avoid mutation of BasicGenerateInstruction, I encapsulated them into properity methods.

Let me know if it is good with you!

[tobicoveo]

Thanks @arnavsinghvi11!

• I rebased and merged the main from dspy.
• I also limited the change to injecting instructions via string to avoid potential errors where the signature fields aren't the same as the base class.
• Added the changes to MIPRO.
• Added details about how to use it in the Usage Suggestions

[tobicoveo]

I don't want to update the fields, but the instructions. Unless I am mistaken, the Signature class doesn't have a method for this.

[mikeedjones]

Maybe can combine to:

Suggested change

	@property
	def basic_generate_instruction(self):
	return (self._get_signature(BasicGenerateInstruction).with_instructions(
	" ".join([BasicGenerateInstruction.instructions, self._additional_instructions])
	)
	if self._additional_instructions
	else BasicGenerateInstruction)
	@property
	def generate_instruction_given_attempts(self):
	return (
	self._get_signature(GenerateInstructionGivenAttempts).with_instructions(
	" ".join([GenerateInstructionGivenAttempts.instructions, self._additional_instructions])
	)
	if self._additional_instructions
	else GenerateInstructionGivenAttempts
	)
	def append_instructions(self, base_signature, additional_instructions):
	return self._get_signature(base_signature).with_instructions(
	" ".join([BasicGenerateInstruction.instructions, self._additional_instructions or ""])
	)
	@property
	def basic_generate_instruction(self):
	return append_instructions(BasicGenerateInstruction, self._additional_instructions)
	@property
	def generate_instruction_given_attempts(self):
	return append_instructions(GenerateInstructionGivenAttempts, self._additional_instructions)

[tobicoveo]

Added the intuitions behind your recommendation. There were a few typos, but I fixed them.

[tobicoveo]

@arnavsinghvi11 and @mikeedjones, is there anything missing to close this PR?

[arnavsinghvi11]

tagging @XenonMolecule to further confirm the behavior of COPRO and MIPRO. LGTM.

[XenonMolecule]

This looks good to me! Cool idea to allow user-defined grounding for instruction proposal!

[mikeedjones]

I'm really not sure about this PR. Making a small change here actually alters the interface to Copro, which then has to be supported until a breaking 3.x release.

I think this functionality would be better implemented by figuring out how to allow the user to modify any of the internal prompts of dspy, and then apply that same logic here. Does that make sense?

[XenonMolecule]

Sure, I see your point, Michael. Are you suggesting some config or shared interface that lets you swap out the prompts for COPRO, MIPRO, and any other DSPy internal program that uses prompts? This is a good design decision to come up with before we lock into a specific interface too soon.

One thing that we've been working on for some internal projects is refactoring MIPRO and COPRO to allow user-defined proposal programs that can include any number of details about your program. This would mean taking into account more than just the Dataset Summary, but also things like the code in your program itself, various prompt engineering tips, etc. Enabling users to plug-in from existing proposal programs or write their own might be a better interface to commit to.

[mikeedjones]

Yes - there are ~8 Signatures in mipro, any one of which might be causing a user some headache.

I think it would be possible to use something like a context manager replace associated with the Signature class which allows the user to replace or update subclasses by reference. Something like:

from dspy.teleprompt import mipro_optimizer
import dspy

class MyBasicGenerateInstruction(mipro_optimizer.BasicGenerateInstruction):
    "you are foo"
    basic_instruction = dspy.InputField(desc="The initial instructions before bar")

with mipro_optimizer.BasicGenerateInstruction.replace(MyBasicGenerateInstruction):
    print(mipro_optimizer.BasicGenerateInstruction.__doc__)

 # you are foo

which would require something like

#dspy/signatures/signature.py
class Signature(BaseModel, metaclass=SignatureMeta):
    ""  # noqa: D419

    # Note: Don't put a docstring here, as it will become the default instructions
    # for any signature that doesn't define it's own instructions.
    pass

    @classmethod
    @contextmanager
    def replace(cls, new_signature: Type["Signature"]):
        """Replace the signature with an updated version.

        This is useful for updating the internal signatures of dspy
        """
        class OldSignature(cls, Signature):
            pass

        replace_fields = ["__doc__", "model_fields", "model_extra", "model_config"]
        for field in replace_fields:
            setattr(cls, field, getattr(new_signature, field))
        cls.model_rebuild(force=True)
        yield
        for field in replace_fields:
            setattr(cls, field, getattr(OldSignature, field))
        cls.model_rebuild(force=True)

Then probably want some helper functions so the user can supply a mapping of default:new Signatures and have them all replaced? And some checks on the new_signature so nothing breaks.

[mikeedjones]

Sketched out in #1090

[tobicoveo]

Thanks @mikeedjones, I will close this PR! 🙂

[mikeedjones]

not sure everything in my PR is gold! - I would appreciate it if you could check that it satisfies your requirements and stuff?

[tobicoveo]

I already checked it out, and it looks good to me! My understanding is that I could write the custom signature with additional instructions that I want and use the context manager to replace the base signature.