Llama C++
원문: llama-cpp
Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.
무엇을 하나요
llama.cpp Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non NVIDIA hardware. When to use llama.cpp Use llama.cpp when: Running on CPU only machines Deploying on Apple Silicon (M1/M2/M3/M4) Using AMD or Intel GPUs (no CUDA) Edge deployment (Raspberry Pi, embedded systems) Need simple deployment without Docker/Python Use TensorRT LLM instead when: Have NVIDIA GPUs (A100/H100) Need maximum throughput (100K+ tok/s) Running in datacenter with CUDA Use vLLM instead when: Have NVIDIA GPUs Need Python first API Want PagedAttention Quick start Installation Download model Run inference Server mode Quantization formats GGUF format overview | Format | Bits | Size (7B) | Spee…
실행 시 본인 API 키(BYOK)로 동작하며, 모델 비용은 사용자 계정에서 직접 결제됩니다.