Expert knowledge for digital decisions
What is used to operate a language model in-house?
Short answer
The two usual approaches
Ollama. Installation in minutes, models are loaded via command. Ideal for testing and individual users. It reaches its limits with many simultaneous requests.
vLLM. Built for server operation. Bundles requests and uses graphics memory more efficiently – resulting in significantly higher throughput performance. More complex to set up.
Why the OpenAI-compatible interface is important
Both provide an interface similar to that of commercial providers. This allows an application to switch between own server and cloud without modification – helpful for comparison and as a fallback option.
The usual approach
- Test with Ollama on an existing computer
- Measure with real cases whether the quality is sufficient
- Decide only then whether a server is worthwhile
- Switch to vLLM or a rented environment for productive operation
Step 2 is often skipped – and is the most important.
What is required
A model server alone is not an application. Missing components include:
- Access control
- Logging
- Search in own documents
- A user interface
These parts account for the greater effort, not the model server.
Key facts
- For the initial setup
- Ollama
- For multi-user operation
- vLLM
- Greater effort
- Access control, logs, document search, user interface
Sources
All external claims are backed by traceable sources.- 01