Convert generated dataset to messages format

[shivchander]

Currently the generated synthetic data has the q/a/context in its own columns, the new training api assumes the datasets are formatted in messages format

Will need a simple util function to run post generation to handle the conversion

Columns to have:

messages: list[dict] - with roles for system, user, assistant
metadata: json str - json string version of a dictionary which acts like a catch-all for all other columns in the dataset

[nathan-weinberg]

Eval is expecting this as well for MMLU! Tracker issue on our side: instructlab/eval#35

[russellb]

the current output is consistent with what the CLI has always done.

we can change it, but we have to coordinate it across all components, including code still in the CLI, that depends on it

[shivchander]

yes I see that it converts to messages - but we still need to handle all other columns so that it doesn't break when we concatenate with other datasets during training.

Added some more details about the schema we need in my original comment

[RobotSail]

The training library currently expects messages, but I believe legacy Linux training and MacOS training doesn't.

[oindrillac]

Where do we see this conversion happening? Would this be in _gen_train_data in generate_data.py? Will we be maintaining 2 formats, the existing one that is for qlora and the additional new format as proposed above for the full train?

[aakankshaduggal]

Yes you are spot on @oindrillac

sdg/src/instructlab/sdg/generate_data.py

Lines 80 to 98 in 45ecc73

	def _gen_train_data(logger, machine_instruction_data, output_file_train):
	train_data = []
	for synth_example in machine_instruction_data:
	logger.debug(synth_example)
	user = _get_question(logger, synth_example)
	if len(synth_example.get("context", "")) > 0:
	user += "\n" + synth_example["context"]
	train_data.append(
	{
	"system": get_sysprompt(),
	"user": _unescape(user),
	"assistant": _unescape(_get_response(logger, synth_example)),
	}
	)
	with open(output_file_train, "w", encoding="utf-8") as outfile:
	for entry in train_data:
	json.dump(entry, outfile, ensure_ascii=False)
	outfile.write("\n")

We would want to have two separate outputs, one in the format the CLI expects for legacy training, and one for the new version that would expect the messages format.

[oindrillac]

cool, and based on whether pipeline == "full" or pipeline == "simple" we can enforce the final output to be in a certain required format?

[aakankshaduggal]

Yes!
@RobotSail mentioned that we just need to pass the data_file_path so for full pipeline mode we can mention the messages file instead.

[russellb]

cool, and based on whether pipeline == "full" or pipeline == "simple" we can enforce the final output to be in a certain required format?

or we could just always produce both?

[nathan-weinberg]

Yes! @RobotSail mentioned that we just need to pass the data_file_path so for full pipeline mode we can mention the messages file instead.

For what's it worth, on the eval side we just need a path to a directory where the messages data is going to be dumped to - similar to the generate directory the CLI dumps data to currently.

[oindrillac]

It would be in the same directory and we can produce both formats as per @russellb suggestion