How to secure sensitive company data on an LLM server?

Security Measures for LLM Servers

Data Encryption

Sensitive data must be encrypted both at rest (on storage media) and in transit (e.g., between server and client). Acceptable algorithms include AES-256 or RSA-2048. Key management must be secure, e.g., via Hardware Security Modules (HSMs) or Key Management Services (KMS). It is important that encryption occurs at the infrastructure and network level rather than the application level to ensure protection against application-level attacks.

Access Restrictions

Access to the LLM server should only be permitted through authentication and authorization. This includes:

• Role-Based Access Control (RBAC): Users are granted only the rights necessary for their tasks.
• Multi-Factor Authentication (MFA): At least two factors (e.g., password + token) are required to enable login.
• Network Segmentation: The LLM server should operate in an isolated network segment to prevent unauthorized access.

Compliance with Regulations

Compliance with legal and industry-specific regulations is central. Relevant provisions include:

• DSGVO: Requires documentation of security measures (Article 30) and compliance with processing agreements.
• NIST AI RMF 1.0: Defines risk management frameworks for AI systems, including ensuring data integrity and confidentiality.

Security Audits and Monitoring

Regular audits and continuous monitoring are required to detect security vulnerabilities early. This includes:

• Automated Security Scans: To identify vulnerabilities in software and hardware infrastructure.
• Log Management: All access and changes to sensitive data must be logged and monitored for anomalies.
• Penetration Tests: External security services can systematically test the server for vulnerabilities.

Backup Strategy and Disaster Planning

A robust backup strategy is crucial to avoid data loss. Regular backups (e.g., daily) should be stored in multiple, geographically separate data centers. Additionally, a disaster recovery plan must be in place that describes how data can be quickly restored in the event of a security incident. This also includes training IT personnel in emergency procedures.

These measures are general in nature, as no Ouhud-specific information is available, but they align with proven best practices in information security.