DSPy Only Predict

[KCaverly]

This PR is intended to get a "DSPy Only Predict" running, ie. No dsp code running.
This is ultimately the point at which the Backend refactor becomes breaking,

TODOs, This includes a few main points:

1. Move over dsp's predict primitives:

• Greedy & Partial decoding for Templates, allowing the dspy.Template to extract each field regardless of whether all fields were generated successfully.
• Recursive "healing" for Template based generation calls. Ie. Some generate calls, often do not include all fields.
• Tests for BackendTemplates with partially completed calls.
• Ensure parameters for max_tokens, temperature and recursive workflow max_retries` are respected and exposed appropriately.

2. Introduce new DummyLanguageModel for use with the BaseLM.

• Build backend agnostic Dummy Language Model.
• Add tests for TemplateBackend generate calls without recovery.
• Add tests for TemplateBackend generate calls with recovery.

3. Update Predict

• Remove all stale attributes (lm)
• Transition away from dsp.Example and dsp.Template to dspy.Example and dspy.Template.
• Generate new predictions for Backend infrastructure.
• Expand Predict tests to ensure functionality remains consistent.

Questions

1. Do we expect the 'non-Template' backends to use recursive healing functionality.

I imagine, non-Templates would not need this functionality. I have currently included it within the BaseBackend but it could be pulled out and included within the TemplateBackend only.

2. Load/Dump State for the current dspy.Predict class will be broken with this change.

Ultimately, starting this integration across the different module classes is going to be breaking. Are there specific concerns we have regarding this, how do we want to manage this? Migration details etc. Also is it worth merging the backend implementations earlier than transitioning the Module classes?

Let me know what you think? @CyrusOfEden @okhat.

[CyrusNuevoDia]

This is fantastic man! Really appreciate you. Looks like you went through quite a concept maze to get here and the result shows it.

Pinging @okhat to explain 'stage' in your TODO, had 1 nit. lmk what you think

[CyrusNuevoDia]

@okhat can you comment here?

[KCaverly]

I decided to sneak in a fleshed out JSONBackend, will add tests such that the Predict works with either backend.

[CyrusNuevoDia]

Small bug but overall looks great!

[KCaverly]

This is done, we have feature parity with DSPy from about a week or two ago, prior to COPRO/MIPRO.
I think we should merge this with the backend-refactor branch, pull main in, and do final tests that way.

We should only be failing on tests for the signature_opt, but that has been replaced by COPRO.