Associated Threats for LLM Application
We mapped the potential threats specific to LLMs, grouped by development, deployment, and utilisation stages of LLM lifecycle and outlined the details of each of the identified threats and its impact to cybersecurity objectives (i.e., confidentiality, integrity, and availability), supported by industry case studies for reference. While not explicitly mentioned here, threats applicable to traditional IT components can still apply.
Identified threats on LLM applications by lifecycle
LLM Lifecycle: Development
The development involves training and fine-tuning of the LLM using training data.
| Threat ID | Threat | Threat Scenarios | Threat Impact | Industry Case Studies |
|---|---|---|---|---|
| Dev1 | Foundation / Fine-tuned LLM poisoning (data sets) |
An attacker manipulates LLM behaviour by altering training data, including test data, during development time. | Integrity | Poison Training Data |
| Dev2 | Foundation / Fine-tuned LLM poisoning (engineering elements) |
An attacker manipulates LLM behaviour by altering the code, configuration or LLM parameters during development time. | Learning to poison LLM during instruction tuning | |
| Dev3 | Foundation LLM poisoning (transfer learning attack) |
An attacker uploads malicious LLM to third-party repository, which is subsequently downloaded by the victim during development time. | PoisonGPT How to poison LLM supply chain on Hugging Face |
|
| Dev4 | Sensitive data leakage (training data) |
An attacker gains unauthorised access to training data, including test data during development time. | Confidentiality | Arbitrary Code Execution with Google Colab |
| Dev5 | Sensitive data leakage (LLM parameters) |
An attacker gains unauthorised access to LLM parameters during development time. | ||
| Dev6 | Sensitive data leakage (source code/ configuration) |
An attacker gains unauthorised access to source code or configuration that is used to pre-process the training data, including test data and train the LLM during development time. | Uber Breach Exposes the Data of 57 Million Drivers and Users |
LLM Lifecycle: Deployment
The deployment involves deploying fine-tune LLM from development environment to runtime environment.
| Threat ID | Threat | Threat Scenarios | Threat Impact | Industry Case Studies |
|---|---|---|---|---|
| Dep1 | Runtime LLM poisoning | An attacker manipulates the behaviour of the LLM itself or its input and/or output logic by altering the LLM parameters during deployment or utilisation phase. | Confidentiality | Learning from Tay’s introduction |
| Dep2 | Runtime LLM theft | An attacker steals LLM parameters during deployment or utilisation phase. For example, by gaining access to LLM’s executables, memory, or other parameters in the deployment environment. | Stealing part of production LLM | |
| Dep3 | Indirect runtime LLM behaviour manipulation | An attacker manipulates the behaviour of the LLM by injecting indirect prompts through the compromised data source used by the LLM application, causing it to perform unintended actions. | Integrity | Indirect Prompt Injection Threats: Bing Chat Data Pirate |
LLM Lifecycle: Use
The use of LLM arise through user interaction with the LLM in the runtime environment.
| Threat ID | Threat | Threat Scenarios | Threat Impact | Industry Case Studies |
|---|---|---|---|---|
| Use1 | LLM behaviour manipulation | An attacker manipulates the behaviour of the LLM by injecting direct prompts, causing it to perform unintended actions such as gaining unauthorised privilege access to restricted parts of a system, altering system configurations or critical data records, and unauthorised purchases or fund transfers where LLM can access to an e-commerce system or financial database. | Confidentiality, Integrity, Availability | ChatGPT Plugin Privacy Leak Achieving Code Execution in MathGPT via Prompt Injection Tay Poisoning LLMjacking: Stolen Cloud Credentials Used in New AI Attack |
| Use2 | Sensitive training data disclosure | An attacker manipulates the behaviour of the LLM by injecting direct prompts, causing it to leak sensitive data from the private training set. | Confidentiality | Extracting Training Data from ChatGPT Why system prompt leaks are bad |
| Use3 | Sensitive in-context learning data disclosure | An attacker manipulates the behaviour of the LLM by injecting direct prompts, causing it to leak rich context learning data with sensitive data from non-user input. | ||
| Use4 | Unintended sensitive data disclosure | A user provides sensitive data to an LLM service, unaware that the LLM utilises the provided sensitive data to train itself. Consequently, leading to leakage of this sensitive data to other users of the LLM service. | Samsung Workers Accidentally Leaked Trade Secrets via ChatGPT | |
| Use5 | LLM theft | An attacker collects input and output of target LLM and uses those combinations to train a replica LLM. | Confidentiality | GPT-2 Model Replication Stanford researchers make a new ChatGPT with less than $600 |
| Use6 | LLM Service Disruption | An attacker provides high frequency, voluminous or malicious input to the LLM, with aim to cause denial of service. | Availability | Energy Latency Attacks On Neural Network |
