Tensorrt LLM
원문: tensorrt-llm
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi
무엇을 하나요
TensorRT LLM NVIDIA's open source library for optimizing LLM inference with state of the art performance on NVIDIA GPUs. When to use TensorRT LLM Use TensorRT LLM when: Deploying on NVIDIA GPUs (A100, H100, GB200) Need maximum throughput (24,000+ tokens/sec on Llama 3) Require low latency for real time applications Working with quantized models (FP8, INT4, FP4) Scaling across multiple GPUs or nodes Use vLLM instead when: Need simpler setup and Python first API Want PagedAttention without TensorRT compilation Working with AMD GPUs or non NVIDIA hardware Use llama.cpp instead when: Deploying on CPU or Apple Silicon Need edge deployment without NVIDIA GPUs Want simpler GGUF quantization format …
실행 시 본인 API 키(BYOK)로 동작하며, 모델 비용은 사용자 계정에서 직접 결제됩니다.