Recommended controls and corresponding threats for LLM applications
The tables below outline the recommended controls to mitigate corresponding threats, grouped by applicable architecture components. These controls are not intended for compliance purposes. Refer to the controls for architecture components that are relevant for specific LLM application.
Identified threats on LLM applications by lifecycle
Architecture Component: Engineering Device (ED)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| ED-C1 | Harden Engineering Devices (Endpoints) Perform engineering activities only through Government Standard Image Build (GSIB) or equivalent; or device onboarded to Security Suite for Engineering Endpoint Devices (SEED) to mitigate the risks of security vulnerabilities exploitation through insecure engineering endpoints. Non-government organisations could implement similar requirements for hardened endpoints for developers and accompanying identity and access management solutions in order to secure the development environment. |
Dev1 to Dev 3: Foundation/Fine-tuned LLM poisoning Dev4 to Dev 6: Sensitive data leakage |
IM8 |
| ED-C2 | Use Trusted LLM Development Tools Use development tools offered on Government Commercial Cloud (GCC) as Software-as-a-Service (SaaS) (e.g., AIBots, LLM Stack, MAESTRO, SHIP-HATS, etc.) rather than third-party tools to mitigate the risks of security vulnerabilities exploitation through insecure development tools. Non-government organisations could implement similar secure continuous integration / continuous deployment tools. |
Supply Chain Management |
Architecture Component: Training Data (TD)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| TD-C1 | Control Access to Training Data and underlying Accelerated Compute Grant user access to the training data and underlying accelerated compute based on the principle of least privilege to mitigate the risks of unauthorised access to them. |
Dev4 to Dev5: Sensitive data leakage | AML.M0019 IM8 |
| TD-C2 | Right Classify Training Data and underlying Accelerated Compute Ensure that the training data stored on accelerated compute are aligned in their classification to mitigate the risks of data exfiltration and to properly assess its impact. For example, accelerated compute classified as Restricted should not store training data classified beyond Restricted. |
Dev4 to Dev5: Sensitive data leakage | IM8 |
| TD-C3 | Store Sensitive Training Data Securely Store and encrypt training data that are sensitive2, in GCC or on-premise based infrastructure to mitigate the risks of data exposure. |
Dev1 to Dev3: Foundation/Fine-tuned LLM poisoning Dev4 to Dev5: Sensitive data leakage |
Segregate Data IM8 |
| TD-C4 | Sanitise Training Data Sanitise training data classified as Sensitive-Normal and above, using tools such as Cloak; or apply other privacy enhancing technologies such as differential privacy3 or synthetic data generation4 (using tool like Mirage) to mitigate the risks of data exposure and limit its impact. |
Dev4: Sensitive data leakage (Training data) | Segregate Data AML.M0007 |
| TD-C5 | Maintain Provenance of Training Data and Validation data Record metadata of the training data such as author, origin, date, and time of creation, and maintain a record of changes to the metadata, to mitigate the risks associated with inadequate incident handling. |
Dev1: Foundation/Fine-tuned LLM Poisoning (Data sets) | Data Provenance and Lineage |
2Sensitive refers to information or data that if disclosed, accessed, or handled improperly has potential consequential impact on finance, reputation, operations, privacy, legal matters, etc.
3A mathematical framework for ensuring the privacy of individuals in datasets.
4The process of creating artificial data that mimics the statistical properties of real-world data.
Architecture Component: Foundation LLM (FM)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| FM-C1 | Verify Integrity of Foundation LLM Verify the integrity of the foundation LLM's files using methods like file hashes, digital signatures, or checksums; to mitigate the risks associated with tampered LLMs. |
Dev3: Foundation LLM Poisoning | NIST 800-53 SI-7 |
| FM-C2 | Use Benchmarked Foundation LLM Prefer the use of supplier/vendor-furnished foundation LLM that has undergone academic or industry benchmarking, demonstrating its robustness and security to mitigate the risk of security vulnerabilities exploitation. |
Dev1 to Dev2: Foundation/LLM Poisoning Dev4 to Dev5: Sensitive data leakage Dep1 to Dep2: Runtime LLM Poisoning/ Theft |
Cataloguing LLM Evaluation |
| FM-C3 | Maintain Provenance of Foundation LLM Record metadata of foundation LLM used, such as author, origin, date, and time of creation, and maintain a record of changes to metadata to mitigate the risks associated with inadequate incident handling. |
Dev3: Foundation LLM Poisoning | NIST 800-53 AU-6 |
Architecture Component: Fine-tuned LLM (TM)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| TM-C1 | Control Access to Fine-tuned LLM and underlying Accelerated Compute Grant user access to the fine-tuned LLM and underlying accelerated compute based on the principle of least privilege to mitigate the risks of unauthorised access to them. |
Dev5: Sensitive data disclosure (LLM parameters) Dep1 to Dep2: Runtime LLM Poisoning/ Theft Dep3: Indirect runtime LLM behaviour manipulation Use1: LLM behaviour manipulation |
AML.M0019 |
| TM-C2 | Right Classify Fine-tuned LLM and underlying Accelerated Compute Ensure that the training data, fine-tuned LLM and underlying accelerated compute are aligned in their classification to mitigate the risks of data exfiltration and to properly assess its impact. For example, accelerated compute classified as Restricted should not store training data and fine-tuned LLM classified beyond Restricted. |
Dev4: Sensitive data disclosure (Training data) | NIST 800-53 AC-3 |
| TM-C3 | Save Fine-tuned LLM in Secure Formats Save fine-tuned LLM in known secure formats (e.g., GGML, GGUF5, etc.) and avoid the use of insecure formats (e.g., pickle file6) to mitigate the risks of data corruption, unauthorised access, and security vulnerabilities exploitation. |
Dev1 to Dev2: Foundation / Fine-tuned LLM poisoning Dev4 to Dev5: Sensitive data disclosure Dep1 to Dep2: Runtime LLM Poisoning/ Theft |
Python Serialization Vulnerabilities - Pickle AML.M0016 AML.T0018 Secure Development Program |
| TM-C4 | Employ Adversarial Testing on Fine-tuned LLM For LLM Applications with medium risk and above, employ adversarial testing utilising tools such as Garak and Giscard, to mitigate the risk of security vulnerabilities exploitation. |
Dep1 to Dep2: Runtime LLM Poisoning/ Theft Use2: Sensitive data disclosure Use5 to Use6: LLM Theft/ Service Disruption |
AML.M0003 Generative Adversarial Reward Modelling |
| TM-C5 | Use SHIP-HATS or equivalent DevSecOps tools to integrate and deploy LLM Use SHIP-HATS or equivalent DevSecOps tools for integration and deployment of LLMs that covers: i. Software Composition Analysis (SCA), Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), Container Image Scanning, etc. in the LLM Application’s Continuous Integration (CI) pipeline to mitigate the risk using insecure dependencies and security vulnerabilities exploitation due to inadequate patches and misconfiguration. ii. Code signing in the LLM’s Continuous Deployment (CD) pipeline to mitigate the risks of using tampered LLM Applications. iii. Disallow changes on LLM to flow from runtime to development environment to mitigate the risks of unauthorised changes. Non-government organisations could implement equivalent DevSecOps tools to provide the features stated in i., ii. and iii. |
Dep1 to Dep2: Runtime LLM poisoning & theft | AML.M0016 Secure Development Program Supply Chain Manage |
5GGML and GGUF are GPT-Generated Model Language and <b?GPT-Generated Unified Format by Georgi Gerganov.
6Pickle is a Python library used for serializing and deserializing Python objects. While it’s a convenient tool for storing and retrieving data structures, it’s not considered a secure format for storing Large Language Models (LLMs) or any sensitive data.
Architecture Component: Runtime LLM (RM)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| RM-C1 | Control Access to Runtime LLM Grant user access to the runtime LLM based on the principle of least privilege to mitigate the risks of unauthorised access to it. |
Dep1 to Dep2: Runtime LLM Poisoning/ Theft | LLM Access Control AML.M0001 Weight and Bias |
| RM-C2 | Encrypt Runtime LLM Artefacts in Transit and at Rest Encrypt runtime LLM artefacts in transit (e.g., using https) to mitigate the risks of data exposure and limit its impact. |
Dep1 to Dep2: Runtime LLM Poisoning/ Theft | AML.M0012 |
| RM-C3 | Access only Whitelisted LLM Agent(s) Encrypt runtime LLM artefacts in transit (e.g., using https) to mitigate the risks of data exposure and limit its impact. |
Dep1 to Dep2: Runtime LLM Poisoning/ Theft | NIST 800-53 AC-6 |
Architecture Component: Model-as-a-Service(MS)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| MS-C1 | Disable Interaction History to MaaS Disable interaction history on MaaS, to mitigate the risk of data exposure on it. |
Use4: Sensitive data disclosure | NIST 800-53 R5 AC3, AC6 |
Architecture Component: Vector Store (VS)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| VS-C1 | Harden Vector Store Ensure vector store is hardened to mitigate the risks of security vulnerabilities exploitation due to misconfigurations. |
Dep1 to Dep2: Runtime LLM Theft & behaviour manipulation | NIST 800-53 CM-6 |
| VS-C2 | Control Access to Vector Store Grant user access to vector store based on the principle of least privilege to mitigate the risk of unauthorised access. |
NIST 800-53 AC-6 | |
| VS-C3 | Encrypt Vector Store Encrypt all indexed and embedded data stored in vector store to mitigate the risk of data exposure and limit its impact. |
Dep2: Runtime LLM behaviour manipulation | NIST 800-53 SC-13 |
| VS-C4 | Sanitise Input to Vector Store Remove sensitive information from any document before converting the documents to embeddings for storage in the vector store to mitigate the risk of data exposure and limit its impact. |
Dep2: Runtime LLM behaviour manipulation Use3: Sensitive in context learning data disclosure |
NIST 800-53 PL-2 |
Architecture Component: LLM User Interface Application (LA)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| LA-C1 | Apply Rate Limiting on User Input using LLM User Interface Application Apply rate limiting on user input via LLM user interface application to mitigate the risk of disruption of service on the LLM Applications. |
Use1 to Use2: Runtime LLM Theft & behaviour manipulation Use4 to Use6: Sensitive Data Disclosure |
NIST 800-53 AC-3 |
| LA-C2 | Validate User Input to LLM User Interface Application Validate user input (e.g., using prompt injection detection systems, web application firewall, etc.) via LLM user interface application to mitigate the risk of prompt injection on LLM and thus manipulating its behaviour. |
Dep3: Indirect runtime LLM behaviour manipulation Use1 to Use2: Runtime LLM Theft & behaviour manipulation Use4 to Use6: Sensitive Data Disclosure |
Prompt Input Validation Validate ML Model |
| LA-C4 | Conduct Regular Vulnerability Assessments and Penetration Tests Conduct vulnerability assessments every quarter and penetration test every year on LLM user interface application to mitigate the risks of security vulnerabilities exploitation. |
Dep1 to Dep2: Runtime LLM poisoning & theft | NIST 800-53 CA-2 |
Architecture Component: Retrieval and LLM Agent (RL)
| Control ID | Control Description | Threat ID(s) | References |
|---|---|---|---|
| RL-C1 | Access only Whitelisted Internet Resource Ensure that agency-owned retrieval and LLM agent accesses only whitelisted Internet-accessible resources to mitigate the risk of indirect prompt injection within untrusted Internet resources. |
Dep3: Indirect runtime LLM behaviour manipulation Use1 to Use6: LLM behaviour manipulation, sensitive data disclosure, Theft and Service Disruption |
NIST 800-53 AC-6 |
| RL-C2 | Access only Whitelisted Intranet Resource Ensure that the agency-owned retrieval and LLM agent accesses only whitelisted GEN-accessible resources (e.g. knowledge base) to mitigate the risk of indirect prompt injection and sensitive data exposure. |
Dep3: Indirect runtime LLM behaviour manipulation Use1 to Use6: LLM behaviour manipulation, sensitive data disclosure, Theft and Service Disruption |
NIST 800-53 AC-6 |
| RL-C3 | Validate Content from Internet and Intranet Resources Ensure that the agency-owned retrieval and LLM agent validates content from Internet and intranet resources before processing them to mitigate the risk of indirect prompt injection and manipulation of LLM behaviour. |
Dep3: Indirect runtime LLM behaviour manipulation Use1 to Use6: LLM behaviour manipulation, sensitive data disclosure, Theft and Service Disruption |
NIST 800-53 AC-17 |
