Openrlhf 학습
원문: openrlhf-training
High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.
무엇을 하나요
OpenRLHF High Performance RLHF Training Quick start OpenRLHF is a Ray based RLHF framework optimized for distributed training with vLLM inference acceleration. Installation : PPO Training (Hybrid Engine): GRPO Training (Group Normalized Policy Optimization): Common workflows Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO) Step 1: Train reward model (DPO): Step 2: PPO training : Workflow 2: GRPO training (no critic model needed) Memory efficient alternative to PPO: Key GRPO parameters : advantage estimator group norm Enables GRPO use kl loss KL loss from GRPO paper kl estimator k3 Loss function (k2 ≈ k1) no advantage std norm Disables std normalization Workflow 3: DPO training (pref…
실행 시 본인 API 키(BYOK)로 동작하며, 모델 비용은 사용자 계정에서 직접 결제됩니다.