Fixed issue #894 and #858 (aws_models issues) #843

drawal1 · 2024-04-16T23:49:05Z

EnsembledProgram can now be loaded/saved
~~Inference of programs (forward function) is now concurrent~~
Added checks for size = 0 or size > number of programs

okhat · 2024-04-17T19:33:08Z

This is so cool.... I need to check the parallelization, it's a bit tricky in DSPy due to dspy.settings.

In dspy.evaluate.Evaluate we do things in a bit of a special way, which has to be applied when there's threading

drawal1 · 2024-04-17T20:29:33Z

@okhat @arnavsinghvi11 - This PR now also includes a fix for issue #824. The fix is in teleprompt/random_search.py

drawal1 · 2024-04-22T18:21:25Z

@okhat - any progress on this review? I am continuing to work on intelligent ensemble routing and don't want to muddy this PR with commits related to that work

btw, I have been testing ensemble concurrency and so far it works like a charm

also, I can reverse the commit for issue #824 and submit a separate PR for that, if it would expedite this PR review

drawal1 · 2024-04-23T16:23:31Z

This is so cool.... I need to check the parallelization, it's a bit tricky in DSPy due to dspy.settings.

In dspy.evaluate.Evaluate we do things in a bit of a special way, which has to be applied when there's threading

I think I understand. I see this code in evaluate.py:

        def wrapped_program(example_idx, example):
            # NOTE: TODO: Won't work if threads create threads!
            thread_stacks = dspy.settings.stack_by_thread

So I need to test and verify the parallelization NOT in inference but as part of another optimization pipeline

drawal1 · 2024-04-24T00:46:47Z

Fixed #894 in dsp/modules/aws_models.py, in class AWSMeta(AWSModel) by moving below functionality from init function:

        max_tokens = query_args.pop("max_tokens", None)
        if max_tokens:
            query_args["max_gen_len"] = max_tokens

to the create_body function

and copying kwargs to base_args by value

drawal1 · 2024-04-25T13:40:15Z

@okhat - I have reverted the parallelization in EnsembledProgram.forward() based on your review and feedback.

The remaining fixes/improvements are targeted and should not have side-effects

arnavsinghvi11 · 2024-05-06T01:20:07Z

dspy/teleprompt/random_search.py

        for seed in range(-3, self.num_candidate_sets):
-            if (restrict is not None) and (seed not in restrict):
+            if seed < 0 and (restrict is not None) and (seed not in restrict):


why are we skipping over the zero-shot, labeled_fewshot and unshuffled_fewshot cases for all scenarios? I think even in the conditions above when a teacher is set, it is fine to run through these preliminary candidates and have it weighted in the scores.

fair enough. reverted to previous behavior

arnavsinghvi11 · 2024-05-06T01:20:19Z

dspy/teleprompt/random_search.py

@@ -96,7 +117,8 @@ def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None
                program2 = program.compile(student, teacher=teacher, trainset=trainset2)

            else:
-                assert seed >= 0, seed
+                if seed < 0:


can remove this based on decision above

assert fails Ruff check (S101 Use of assert detected). Lets keep the raise exception here. its good practice

arnavsinghvi11 · 2024-05-06T01:20:36Z

pyproject.toml

@@ -278,7 +284,14 @@ ignore = [
    "E731",
    # Sometimes we need List and Tuple
    "UP006",
+    # optimzer are using the 'compile' method which is a built-in


spelling typo

arnavsinghvi11 · 2024-05-06T01:21:07Z

pyproject.toml

+    # commented-out code should be allowed
+    "ERA001",
+    # allow pickle
+    "S301",


what are the purpose of these changes? I think the ruff has been working correctly so far but just curious

ERA001 because Ruff complains about commented out print() statements. I often find these useful for debugging

Re. S301, ensemble.py load_folder/save_folder use pickle for serialization. Ruff flags pickle as unsafe for load because it could load arbitrary objects. I suppose we could switch to https://jsonpickle.github.io/. If jsonpickle is acceptable, I can make the change and remove the S301.

@arnavsinghvi11 - what's your opinion on jsonpickle? see my comment above...

arnavsinghvi11 · 2024-05-06T01:23:37Z

Thanks @drawal1 for the PR! left a few comments. The changes to Ensemble look correct to me, but I think the ones for RandomSearch are not needed? or can be handled separately without impacting existing behavior. lmk what you think.

There's also a merge conflict to resolve and feel free to remove extraneous comments from the changes that do not help with following the code. Thanks!

tagging @okhat for reference on Ensemble/RandomSearch again (but I believe the parallelization changes are removed so should be good there).

drawal1 · 2024-06-03T15:00:55Z

@arnavsinghvi11 - I have reverted ALL the complicated changes. Hopefully now this PR is acceptable

arnavsinghvi11 · 2024-06-15T18:52:39Z

Hi @drawal1 , thanks for the changes. can you also revert the changes to BootstrapWithRandomSearch since those changes are not relevant to patching #858 .

drawal1 · 2024-06-18T14:55:07Z

@arnavsinghvi11 - BootstrapWithRandomSearch changes are now reverted

arnavsinghvi11 · 2024-06-19T22:41:40Z

Thanks @drawal1 !

drawal1 mentioned this pull request Apr 16, 2024

Error loading program compiled using the Ensemble optimizer #775

Open

drawal1 mentioned this pull request Apr 17, 2024

Optimizer issue: Using a pre-compiled module as a teacher causes an assertion crash in bootstrap.py #824

Closed

drawal1 changed the title ~~Fixed issue #775 and refactored Ensemble optimizer~~ Fixed issue #775, #824, #894 and refactored Ensemble optimizer Apr 25, 2024

This was referenced Apr 25, 2024

Feature request: Handcraft and freeze parts of a prompt, optimize other parts #905

Open

fix(dspy/modules/aws_models): properly copy kwargs so temporary changes don't propagate to base model #858

Closed

arnavsinghvi11 reviewed May 6, 2024

View reviewed changes

drawal1 closed this Jun 3, 2024

drawal1 force-pushed the main branch from be844f1 to 8e01bee Compare June 3, 2024 13:59

fix(aws_models): fix issues 894 and 858

0a7a460

drawal1 reopened this Jun 3, 2024

drawal1 changed the title ~~Fixed issue #775, #824, #894 and refactored Ensemble optimizer~~ Fixed issue #894 and #858 (aws_models issues) + #824 (crash using precompiled teacher) Jun 3, 2024

fix(optimizer): Fixed issue stanfordnlp#824

0c40d09

arnavsinghvi11 mentioned this pull request Jun 17, 2024

ValueError: Required 'max_tokens' or 'max_output_tokens' not specified in settings when using meta.llama2-13b-chat-v1 in AWS SageMaker #1156

Closed

drawal1 added 3 commits June 18, 2024 09:37

fix(optimizer): Undo fix for issue stanfordnlp#824

12a9ae6

Merge branch 'main' of https://github.com/drawal1/dspy

6874b71

fix(optimizer): Undo fix for issue stanfordnlp#824

9d582d9

drawal1 changed the title ~~Fixed issue #894 and #858 (aws_models issues) + #824 (crash using precompiled teacher)~~ Fixed issue #894 and #858 (aws_models issues) Jun 18, 2024

arnavsinghvi11 merged commit 4379ead into stanfordnlp:main Jun 19, 2024
4 checks passed

arnavsinghvi11 mentioned this pull request Jun 19, 2024

DSPY Copro Tutorial for Hotpot QA doesn't work with Bedrock Claude Sonnet Model #1173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed issue #894 and #858 (aws_models issues) #843

Fixed issue #894 and #858 (aws_models issues) #843

drawal1 commented Apr 16, 2024 •

edited

Loading

okhat commented Apr 17, 2024

drawal1 commented Apr 17, 2024

drawal1 commented Apr 22, 2024 •

edited

Loading

drawal1 commented Apr 23, 2024

drawal1 commented Apr 24, 2024 •

edited

Loading

drawal1 commented Apr 25, 2024

arnavsinghvi11 May 6, 2024

drawal1 May 7, 2024

arnavsinghvi11 May 6, 2024

drawal1 May 7, 2024

arnavsinghvi11 May 6, 2024

drawal1 May 7, 2024

arnavsinghvi11 May 6, 2024

drawal1 May 6, 2024 •

edited

Loading

drawal1 May 7, 2024

arnavsinghvi11 commented May 6, 2024

drawal1 commented Jun 3, 2024

arnavsinghvi11 commented Jun 15, 2024

drawal1 commented Jun 18, 2024

arnavsinghvi11 commented Jun 19, 2024

Fixed issue #894 and #858 (aws_models issues) #843

Fixed issue #894 and #858 (aws_models issues) #843

Conversation

drawal1 commented Apr 16, 2024 • edited Loading

okhat commented Apr 17, 2024

drawal1 commented Apr 17, 2024

drawal1 commented Apr 22, 2024 • edited Loading

drawal1 commented Apr 23, 2024

drawal1 commented Apr 24, 2024 • edited Loading

drawal1 commented Apr 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drawal1 May 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnavsinghvi11 commented May 6, 2024

drawal1 commented Jun 3, 2024

arnavsinghvi11 commented Jun 15, 2024

drawal1 commented Jun 18, 2024

arnavsinghvi11 commented Jun 19, 2024

drawal1 commented Apr 16, 2024 •

edited

Loading

drawal1 commented Apr 22, 2024 •

edited

Loading

drawal1 commented Apr 24, 2024 •

edited

Loading

drawal1 May 6, 2024 •

edited

Loading