GRPO for RL on agent trajectories

[korbinian-hoermann]

Hi,

I am new in the field of RL for LLMs, and I was wondering if it is possible to use GRPO to perform RL on LLM based Agent trajectories where the reward function is an ORM on the full trajectory (sparse)? In this case, the generated data for a task would not just be an answer that can be evaluated, but rather a multi-turn conversation, which includes the new observation in form of the next prompt.

It would look like this instead:

{
    "trajectory_id": "trajectory_1",
    "steps": [
      {
        "prompt": "task: search for cat images \naction_history: none \ncurrent_screenshot: <image_step_1.png>",
        "agent_response": "navigate to google.com"
      },
      {
        "prompt": "task: search for cat images \naction_history: navigate to google.com \ncurrent_screenshot: <image_step_2.png>"
        "agent_response": "click on the search bar"
      },
      // ... more steps needed for task completion
    ],
    "reward": 1.0  (ORM based reward)
  },

A naive approach i can think of would be:

1. For each task sample k completions
2. Filter (shortest) successful trajectories
3. Generate dataset for GRPO by converting the prompt for each timestep into a separate question
4. Perform GRPO with this dataset.

As this would, again, involve sampling n times for each question, this seems very resource intensive.

I would appreciate any input on that! :)

[August-murr]

#2723