add total and working time to model, tool, and subtask events #1415
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
All events currently carry a
timestamp
for the clock time when they start, with this PR we add several fields to events to track the working time at which they start as well as the duration of clock and working time consumed by events which occur over a span (ModelEvent
,ToolEvent
,SubtaskEvent
).Here are the fields and which events they apply to:
timestamp
completed
ModelEvent
,ToolEvent
,SubtaskEvent
working_start
working_time
ModelEvent
,ToolEvent
,SubtaskEvent
For
ToolEvent
andSubtaskEvent
the completed/working tracking is trivial as we start execution, create the event, and update the event all in the same stack frame.For
ModelEvent
, we start execution outside of a semaphore and then potentially severalModelEvent
instances can be added to the transcript (none of which are visible in the outer stack frame). We navigate this by searching for and updating the lastModelEvent
in the transcript from the outer stack frame.@dragonstyle We need to figure out how to display this data in the viewer.
Viewer Changes
Timing In model event (all subsection)
Event titles now include working time (model, tool calls, subtask)
Working start time for all events