Securing Generative AI: A Practical Guide to Threat Modeling

As generative AI continues to reshape the landscape of business applications, it’s essential to address the security risks that accompany these innovations. At AWS re:Invent 2023, we dived deep into this very topic, helping hundreds of eager attendees understand how to adopt new technologies while maintaining robust security. In this blog post, we’ll explore the critical steps for effective threat modeling specifically tailored for generative AI workloads, highlighting best practices, typical deliverables, and key outcomes at each stage. We’ll also reference examples created with the open-source AWS Threat Composer tool, an invaluable resource for documenting and managing threat models at no cost.

Why Threat Modeling for Generative AI?

Every new technology presents a learning curve, especially when it comes to identifying and mitigating unique security risks. This is particularly true for generative AI workloads that leverage large language models (LLMs). LLMs can produce a wide range of outputs based on user inputs, which opens the door to potential misuse or abuse. Moreover, these workloads often access large datasets, sometimes containing sensitive internal information, raising further security concerns.

Understanding that LLMs are part of a more complex system is crucial. Threat modeling should encompass not only the risks associated with the AI itself but also traditional threats like data injections or credential compromises. The Securing Generative AI blog series offers a solid foundation for understanding these nuances and risks.

The Four Stages of Threat Modeling for Generative AI

Threat modeling is a structured approach that identifies, analyzes, and addresses security risks within an application. It’s a cornerstone of the design phase, allowing teams to implement protective measures early on. At AWS, threat modeling is a foundational aspect of our Application Security (AppSec) process, supported by Security Guardians who assist in creating these critical models.

1. What Are We Working On?

The first step aims to clarify your business context and application architecture. This foundational knowledge should already be documented as part of your system’s design. By leveraging existing documentation, teams can more effectively focus on identifying threats without redundant effort.

Example Deliverables:

• Data Flow Diagrams (DFDs) showing critical data flows within the application.
• Articulated assumptions regarding user interactions and system integrations.
• Documentation of key design decisions and their rationale.
• Contextual information about the application’s business significance and the types of data it handles.

2. What Can Go Wrong?

Here, you’ll identify potential threats to your application utilizing the contextual information gathered earlier. Resources like the OWASP Top Ten for LLMs and the MITRE ATT&CK framework can be incredibly helpful in identifying risks. Using a framework like STRIDE can systematically guide your thinking about threats.

You can frame potential risks as threat statements, using a structured formula that keeps documentation consistent:
“A [threat source] with [prerequisites] can [threat action], leading to [threat impact], negatively impacting [impacted assets].”

Example Threat Statements:

• A threat actor can inject malicious prompts that access confidential healthcare data, impacting data confidentiality.
• A threat actor may execute requests that lead to the deletion of critical data, jeopardizing system availability.

3. What Can We Do About It?

Once you’ve identified threats, it’s time to determine appropriate controls. Your decisions will be influenced by your organization’s policies and context. Define specific mitigation strategies for each identified threat, aiming to implement both preventative and detective controls.

Example Mitigations:

• M-001: Predefine SQL query structure.
• M-002: Input sanitization to filter known parameters.
• M-003: Dynamic permission checks on sensitive model actions.

4. Did We Do a Good Enough Job?

Lastly, remember that a threat model is a living document. Regularly validate both the effectiveness of your mitigations and the threat modeling process itself. Continuous testing and interaction across teams are vital for updating your threat model as new threats emerge and existing ones evolve.

Example Outputs:

• Automated test cases based on identified mitigations.
• A living document that evolves with the application and its functionality.
• A retrospective overview to capture lessons learned and opportunities for improvement.

Conclusion

In this article, we’ve examined a practical approach to threat modeling generative AI workloads. By following these clearly defined steps, organizations can not only identify and mitigate known risks but also foster a culture of security awareness, which is essential when working with powerful technologies. Such diligence ensures that your organization can fully harness the benefits of generative AI while safeguarding its data and systems.

The AI Buzz Hub team is excited to see where these breakthroughs take us. Want to stay in the loop on all things AI? Subscribe to our newsletter or share this article with your fellow enthusiasts.