From 026162ee81846023e5db9ca7952ae34af0fc7692 Mon Sep 17 00:00:00 2001 From: Mollie Munoz Date: Wed, 6 Nov 2024 18:34:13 -0800 Subject: [PATCH] gc learnings doc (#227) - Add new file - Add initial thoughts/content to file. --- assistants/prospector-assistant/gc_learnings.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 assistants/prospector-assistant/gc_learnings.md diff --git a/assistants/prospector-assistant/gc_learnings.md b/assistants/prospector-assistant/gc_learnings.md new file mode 100644 index 00000000..971af5ee --- /dev/null +++ b/assistants/prospector-assistant/gc_learnings.md @@ -0,0 +1,15 @@ +# GC Learnings + +## Overview + +This doc is intended to capture our team learnings with using Guided Conversation. Learnings include best use cases, concerns, hacks, favorites aspects, and different re-design ideas to improve our use of GC and any future version of it. + +## Notes - To Be Updated + +1. Artifact accessibility - When using GC as a component conversation of a larger conversation, it would be helpful to set an artifact BEFORE starting up GC. Currently, GC takes in a schema and produces the original artifact. As a result, we are having to hack inputs via context, rules, or conversation flow. I (Mollie) have been trying the context approach to add information such as attachment filenames, if this is the first call to this particular gc within a larger conversation or a later call (see startup status below), and possibly chat history and attachment content (not just filenames). + +2. Startup status - Within a larger conversation scope, a single gc (w/ config) may be called multiple times. BUt without the correct context, that gc may think its starting a new conversation, when its not. Right now I am getting a GC saying "Hello!" every time it starts, even though it's like the 5th time I'm calling it for the sole purpose of talking to me about updating an outline. Another templated field that might be helpful is the context of where this gc falls within the broader conversation. This points back to the artifact as a template, and how I would rather set some values ahead of time within a template for GC to use than hack the context with custom prompt engineering. + +3. Completion status - Would love to get more information on why gc decides a conversation is over. Right now its just a bool in the result at the end. There is other data i'm also interested in using at the completion of a conversation "round" as I'm calling it. Currently using the artifact to access this info, and this seems to be working decently well. (This might be the better approach in general because it is customizable.) Via the artifact, rules, and conversation_flow, I can tell the gc what the conversation_status options are and when to use them (especially what to use at the end), and i'm experimenting with a user_decision in the artifact that upon completion, will tell the code what branch path to take next (kinda like the OpenAI tool selection concept if I'm understanding it correctly, but hackier). The issue is that I'm having to prompt engineer all of this and I'm making mistakes. Sometimes no final response is given even when I think I've given GC all the information it should need. + --> This all points to a more robust template that relies less on individual prompt engineering but uses well-tested prompts that fill in basic variable names, absolutes (such as MUST or NEVER), and other prompting best practices that yield consistent results. The programmer should not have to keep reinventing the prompting wheel on some basic data needs, like completion status or completion-next-action. + --> Another note, its possible that prompt instructions for setting a conversation status to "complete" does not coincide with gc thinking the "coversation_is_over". I suspect because the instructions I have given it to determine the convo is complete (via using the artifact field) uses different prompting internally than the `result.conversation_is_over` uses. As a result, I am not checking conversation_is_over at all.