Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] Implement GRPO, PPO and potentially other policy gradient methods to finetune LM Agents #1528

Open
1 of 2 tasks
apokryphosx opened this issue Jan 30, 2025 · 2 comments · May be fixed by #1559
Open
1 of 2 tasks

[Feature Request] Implement GRPO, PPO and potentially other policy gradient methods to finetune LM Agents #1528

apokryphosx opened this issue Jan 30, 2025 · 2 comments · May be fixed by #1559
Assignees
@apokryphosx
Labels
New Feature
Milestone
Sprint 22

Comments

@apokryphosx
Copy link

Required prerequisites

Motivation

After recent success in Deepseek-R1, leveraging synthetic data with RL is vital on the path to more capable models. Particularly interesting is the idea of rule-based reward signals. A pipeline to finetune an LM in camel with RL would be a great addition

Solution

No response

Alternatives

No response

Additional context

I would like to work on this issue, if its possible please assign it to me
@apokryphosx apokryphosx added the enhancement New feature or request label Jan 30, 2025
@GitHoobar
Copy link
Collaborator

this looks good! happy to help

@lightaime
Copy link
Member

There are some projects we can look into for implementing this:

veRL: https://github.com/volcengine/verl
OpenRLHF: https://github.com/OpenRLHF/OpenRLHF
TinyZere (they use veRL): https://github.com/Jiayi-Pan/TinyZero
simpleRL-reason (they use OpenRLHF): https://github.com/hkust-nlp/simpleRL-reason
Open-R1: https://github.com/huggingface/open-r1

@apokryphosx apokryphosx linked a pull request Feb 6, 2025 that will close this issue
Open
8 tasks
@Wendong-Fan Wendong-Fan added New Feature and removed enhancement New feature or request labels Feb 6, 2025
@Wendong-Fan Wendong-Fan linked a pull request Feb 6, 2025 that will close this issue
docs: Created a cookbook that walks you through finetuning a Model with GRPO #1559
Open
8 tasks
@Wendong-Fan Wendong-Fan added this to the Sprint 22 milestone Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@apokryphosx apokryphosx

Labels
New Feature
Projects
Project Camel
Status: No status
Milestone
Sprint 22
Development

Successfully merging a pull request may close this issue.

docs: Created a cookbook that walks you through finetuning a Model with GRPO apokryphosx/camel
4 participants
@lightaime @Wendong-Fan @GitHoobar @apokryphosx