Nemo Curator
원문: nemo-curator
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality tr
무엇을 하나요
NeMo Curator GPU Accelerated Data Curation NVIDIA's toolkit for preparing high quality training data for LLMs. When to use NeMo Curator Use NeMo Curator when: Preparing LLM training data from web scrapes (Common Crawl) Need fast deduplication (16× faster than CPU) Curating multi modal datasets (text, images, video, audio) Filtering low quality or toxic content Scaling data processing across GPU cluster Performance : 16× faster fuzzy deduplication (8TB RedPajama v2) 40% lower TCO vs CPU alternatives Near linear scaling across GPU nodes Use alternatives instead : datatrove : CPU based, open source data processing dolma : Allen AI's data toolkit Ray Data : General ML data processing (no curatio…
실행 시 본인 API 키(BYOK)로 동작하며, 모델 비용은 사용자 계정에서 직접 결제됩니다.