How to evaluate a new language model before productive use?

The task determines the test

Before download, the use case is described: language, expected inputs, prohibited content, output format, required context length, and technical goals. Then license, model card, origin, revision, and known limitations are checked. A model name without a clear revision is not a reproducible operational basis.

The acceptance catalog contains real cases from the own process. A starting set of at least 100 representative tasks can be meaningful for a narrowly limited assistant, but it is a planning value and not a general standard. It includes normal questions, edge cases, intentionally unanswerable questions, prompt injection attempts, protected data, and long contexts. Expected answer, acceptable variants, and evaluation method are defined before model execution.

Structured tasks can be automated with metrics like Exact Match, Accuracy, or F1. Open texts require a subject-matter evaluation grid and samples assessed by qualified personnel. NIST requires systematic testing, evaluation, verification, and validation methods for generative AI. Additionally, source referencing, correct rejection, and authorization thresholds are checked; linguistic elegance must not mask factual errors.

Under the same load profile, Time to First Token, p95 end-to-end latency, error rate, token rate, and memory usage are measured. Base model, quantization, prompt, RAG status, and hardware remain documented during comparison. Acceptance thresholds are defined before results to prevent retroactive adjustments. After successful offline acceptance, only limited canary traffic initially receives the new model. Monitoring and tested rollback remain active until sufficient real-world evidence is available.