Validator, logging and modelling improvements #127
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MartBakler
changed the title
MartBakler
changed the title
MartBakler
changed the title
MartBakler
changed the title
JackHopkins
reviewed
Jan 25, 2024
src/tanuki/function_modeler.py
Outdated
| @@ -453,6 +455,9 @@ def _check_finetuning_condition(self, func_hash): | |||
| # if havent read in the patch dataset size, read it in | |||
| patch_dataset_size = self._get_dataset_info(PATCHES, func_hash, type="length") | |||
| self.dataset_sizes[PATCHES][func_hash] = patch_dataset_size | |||
| logging.info(f"Function {function_description.__name__} [{align_dataset_size} aligns | {patch_dataset_size} runs] will be finetuned from"\ | |||
| f" {self.function_configs[func_hash].teacher_models.model_name} using {self.function_configs[func_hash].distilled_model.provider} in"\ | |||
There was a problem hiding this comment.
Isn't teacher_models a list? If so, how can you get model_name from it?
| @@ -129,20 +136,27 @@ def get_generation_case(self, args, kwargs, function_description, llm_parameters | |||
| # no examples needed, using a finetuned model. Dont save to finetune dataset | |||
| if is_distilled_model and suitable_for_distillation: | |||
| prompt = self.construct_prompt(f, args, kwargs, [], distilled_model) | |||
| if func_hash not in self.current_generators: | |||
| self.current_generators[func_hash] = {"model": "finetuned_model_placeholder", "examples": []} | |||
There was a problem hiding this comment.
Why is this a placeholder?
| return prompt, distilled_model, suitable_for_distillation, True | ||
|
|
||
| else: | ||
| aligns = self.function_modeler.get_symbolic_alignments(function_description.__hash__(), max=16) | ||
| examples = [f"Inputs:\nArgs: {align['args']}\nKwargs: {align['kwargs']}\nOutput: {align['output']}" for align in | ||
| aligns] | ||
|
|
||
| if func_hash not in self.current_generators: | ||
| self.current_generators[func_hash] = {"model": "", "examples": examples} |
There was a problem hiding this comment.
Can we change the variable name current_generators to something more intuitive?
| context_length = 128000, | ||
| instructions="You are given below a function description and input data. The function description of what the function must carry out can be found in the Function section, with input and output type hints. The input data can be found in Input section. Using the function description, apply the function to the Input and return a valid output type, that is acceptable by the output_class_definition and output_class_hint.\nINCREDIBLY IMPORTANT: Only output a JSON-compatible string in the correct response format. Use the [END] tokens to specify when the output ends.", | ||
| parsing_helper_tokens={"start_token": "[START]", "end_token": "[END]"}), | ||
| "gpt-3.5-turbo-1106-finetune": OpenAIConfig(model_name = "", context_length = 14000), |
There was a problem hiding this comment.
Why is the model_name empty?
| finetuning_response: FineTuningJob = self.client.fine_tuning.jobs.create(training_file=training_file_id, | ||
| model="gpt-3.5-turbo", | ||
| finetuning_response: FineTuningJob = self.client.fine_tuning.jobs.create(training_file=training_file_id, | ||
| model="gpt-3.5-turbo-1106", |
There was a problem hiding this comment.
Why is this model hardcoded?
JackHopkins
approved these changes
Jan 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed a bug regarding lists of pydantic objects in validator
Added a lot of logging