학습 Llms Megatron

무엇을 하나요

Megatron Core Large Scale LLM Training Quick start Megatron Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies. Installation : Simple distributed training : Common workflows Workflow 1: Train LLaMA style model with 3D parallelism Copy this checklist: Step 1: Choose parallelism configuration Model size determines parallelism strategy: | Model Size | GPUs | Tensor Parallel | Pipeline Parallel | Data Parallel | Context Parallel | | | | | | | | | 7B | 8 | 1 | 1 | 8 | 1 | | 13B | 8 | 2 | 1 | 4 | 1 | | 70B | 64 | 4 | 4 | 4 | 1 | | 405B | 128 | 8 | 8 | 2 | 2 | Step 2: Configure training hyperparameters Step 3: Launc…