Distributed LLM Pretraining Torchtitan
원문: distributed-llm-pretraining-torchtitan
Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
무엇을 하나요
TorchTitan PyTorch Native Distributed LLM Pretraining Quick start TorchTitan is PyTorch's official platform for large scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs. Installation : Download tokenizer : Start training on 8 GPUs : Common workflows Workflow 1: Pretrain Llama 3.1 8B on single node Copy this checklist: Step 1: Download tokenizer Step 2: Configure training Edit or create a TOML config file: Step 3: Launch training Step 4: Monitor and checkpoint TensorBoard logs are saved to ./outputs/tb/ : Workflow 2: Multi node training with SLURM Step 1: Configure parallelism for scale For 70B model on 256 GPUs (32 no…
실행 시 본인 API 키(BYOK)로 동작하며, 모델 비용은 사용자 계정에서 직접 결제됩니다.