Question
Clearly sourced
Expert knowledge for digital decisions
What hardware does an LLM server need?
Short answer
The critical factor is graphics memory, not computational power. As a rough estimate: a model with N billion parameters requires approximately N gigabytes of graphics memory with standard quantization, plus additional memory for context. A model with 7 to 8 billion parameters runs on a 16 GB card, while 70 billion parameters require multiple cards.
The critical factor: Graphics memory
A language model must fit entirely into the graphics card memory, otherwise it becomes unusably slow. Rough orientation with 4-bit quantization:
| Model size | Memory requirement | Realistically suitable for |
|---|---|---|
| 7–8 billion parameters | ~6–10 GB | Classification, summarization, simple answers |
| 13–14 billion | ~10–16 GB | Better text quality |
| 30–34 billion | ~20–28 GB | More demanding tasks |
| 70 billion | ~40–48 GB | Multiple cards or a large card |
Additional memory is required for context – the longer the documents, the more.
What quantization means
The model weights are stored with reduced precision, usually 4 or 8 bits instead of 16. This significantly reduces memory requirements with only minor quality loss. Without quantization, the requirement multiplies several times.
What is often overlooked
- Concurrent usage. A single card does not serve twenty users simultaneously. Multiple concurrent requests require more hardware or a queue.
- Power and cooling. Servers with multiple graphics cards require appropriate space, not just a storage room.
- First response time. Those expecting a response within a second need more reserve memory than for a batch processing run overnight.
Renting instead of buying
Often the more reasonable choice initially: a rented server with a graphics card in a European data center. Data protection law-compliant almost as good as own hardware, without purchase – and demand can be measured before investment.
Key facts
- Critical
- Graphics memory, not computational power
- Estimate
- Approximately 1 GB per billion parameters with 4-bit quantization
- Recommendation for starting
- Rent instead of buy until demand is measured
Sources
All external claims are backed by traceable sources.- 01