PIX CPU Frame / Task API

[eylvisaker]

PIX CPU Frame/Task API

Introduction

It has long been a request from customers that PIX capture and show information about the frames used within their games. This seems simple from a naïve perspective – the present-to-present timing on the GPU provides a unambiguous signal for how long each frame takes. However, in real-world scenarios, there is actually a lot of complexity that comes from the parallel nature of game engines – the simplest example of which is that the CPU may finish its work for a frame, submit it all to the GPU, and then begin working on the next frame while the GPU renders the previous frame. Taking that a step further, some game engines might run their physics engine on some other subsystem with a frame rate or timing that’s decoupled from the GPU or the main render loop on the CPU. In the extreme case, there are a few game engines out there that run a significant portion of their CPU workloads in jobs systems which deserialize the workloads. Even if we only examine the work done on the GPU, many game engines will do postprocessing on the GPU’s compute queue for the one frame while the graphics queue has already started work on the next frame. In addition, we’ve gotten requests to support fibers in PIX, which are a cooperative multi-tasking technology distinct from threads.

All of these examples point to a shortcoming in PIX – the standard concept we use to instrument user code, PIX events, are thread-bound. Yet in many of the scenarios listed above, the conceptual information our users want about their game engines is not related to threads. Thus, we believe there is value in adding an API similar to PIX events that tracks work not pinned to a thread.

Proposal

The primary focus of this document will be CPU profiling scenarios. The main change in the PIX UI will be to include additional lanes in the timeline. These lanes will look similar to how thread lanes look today. The lanes will each be designated for a particular “timeline” – a user defined grouping of events. We will refer to all these types of events as “tasks."

The scenarios we want to address are:

• Frames
  • Support for multiple types of frames – the user will specify a timeline for each type.
• Jobs
  • States - Jobs can have states that change over time. For example, a job may be initialized in a waiting state and later advance to the executing state.
    • We will define for the API a few PIX-known states. The user can supply their own state names. These user-defined state changes will show with a marker on the event in the timeline the user can hover the mouse over to see details.
  • Child jobs - Jobs can have child jobs that start after the parent job and end before the parent job can end. This follows the existing presentation for the flame graph in our timeline, so we will present this the same.
  • Dependencies – Jobs can have dependencies that prevent them from starting or advancing in state. This will be displayed as a set of arrows on the timelines that show when jobs are blocked by other jobs. (This requirement will not be addressed in the first phase of this project).
• Fibers / other scenarios
  • The API we propose should be flexible enough that it can be used to track other types of non-thread based work, even if it’s not explicitly called out here.

Phase 1 – Tracking tasks in PIX

In the first phase, we’ll introduce three new API functions. These will be used to create event profiling data for all the scenarios described above.

• PIXBeginTask begins tracking a task in PIX.
  • The user can supply a string that specifies the initial state.
  • initialState can be nullptr – this is equivalent to supplying PIXTASKSTATE_EXEC.
• PIXSetTaskState allows the user to update the state of a task.
• PIXEndTask ends a task.

The PIX UI will show data from these API methods, as separate lanes determined by the supplied timeline parameter to PIXBeginTask. If multiple tasks are running on the same timeline, they will stack similar to how the flame graph looks for threads.

PIX API

// Creates task, returning a unique task identifier.
INT PIXBeginTask(UINT color, PCSTR timeline, PCSTR initialState, PCSTR taskNameFormatString, ...);

// Ends a task. 
void PIXEndTask(INT taskId);

// Updates the state of a task. 
void PIXSetTaskState(INT taskId, PCSTR state);

// PIX-aware task states 
const PCSTR PIXTASKSTATE_WAIT = “Waiting”; 
const PCSTR PIXTASKSTATE_EXEC = “Executing”;

API Usage

The Task API methods are expected to be called in the following sequences. Function parameters are suppressed in the examples for readability.

Basic Usage - track when a task starts and ends:

INT id = PIXBeginTask(...); 
PIXEndTask(id)

Tracking task state changes before it's complete:

INT id = PIXBeginTask(...);
PIXSetTaskState(id, ...) /*may be called multiple times */; 
PIXEndTask(id);

Invalid API usage examples

A call to PIXSetTaskState for a task that is completed will be ignored by PIX:

INT id = PIXBeginTask(...); 
PIXEndTask(id); 
PIXSetTaskState(id, ...);

Phase 2 – Job Dependency Tracking

This section is not final. Details are TBD.
In the second phase, we add the ability to connect tasks to one another through dependency tracking. There are two pieces to this:

• Display of dependencies in the PIX UI
• A PIX API that allows the user to tell PIX to track a dependency.

The canonical example of this scenario is where a job is registered with the job system and it’s dependent on other jobs to complete.
Dependencies will be displayed in the PIX UI on the task lanes as arrows that point from the end of the dependency job to the blocked state on the dependent job. Depending on UI performance, we may need to find ways to limit how many arrows are drawn.

PIX API

// Tells PIX that taskId must wait for blockingTaskId to complete before its state can advance to the blocked state value.
void PIXTaskDependency(INT taskId, PCSTR blockedState, INT blockingTaskId);

API Usage

Creating two jobs, job1 waits on job2 before it can execute.

INT job1 = PIXBeginTask(initialState: PIXTASKSTATE_WAIT); 
INT job2 = PIXBeginTask(); 
PIXTaskDependency(job1, PIXTASKSTATE_EXEC, job2);  
PIXEndTask(job2);
PIXSetTaskState(job1, PIXTASKSTATE_EXEC);
PIXEndTask(job1);

INT job1 = PIXBeginTask(initialState: PIXTASKSTATE_WAIT); 
INT job2 = PIXBeginTask(initialState: PIXTASKSTATE_WAIT); 
PIXTaskDependency(job1, PIXTASKSTATE_EXEC, job2);
PIXSetTaskState(job2, PIXTASKSTATE_EXEC);
PIXEndTask(job2);
PIXSetTaskState(job1, PIXTASKSTATE_EXEC);
PIXEndTask(job1);

Invalid API usage examples

In the following case, job 1 was marked as having its execution state blocked by job2, but job1 entered the executing state before job2 was completed. This may indicate that incorrect API usage, or it may indicate that the job system is not respecting the defined task dependency. PIX will show this with a dependency arrow that points left instead of the usual right direction. The color of the arrow will also change, and there will be a tooltip available to indicate something is wrong.

INT job1 = PIXBeginTask(initialState: PIXTASKSTATE_WAIT); 
INT job2 = PIXBeginTask(); 
PIXTaskDependency(job1, PIXTASKSTATE_EXEC, job2);
PIXSetTaskState(job1, PIXTASKSTATE_EXEC);
PIXEndTask(job2);
PIXEndTask(job1);

References

Unreal Engine’s task system: Tasks Systems in Unreal Engine | Unreal Engine 5.0 Documentation