The Adventure into AI/ML
We recently celebrated a huge breakthrough with the launch of Qualys TotalAI—our foray into the fascinating world of AI and machine learning (AI/ML). Our journey kicked off in March 2024 when we decided to dive headfirst into the rapidly shifting AI landscape and the burgeoning ecosystem of large language models (LLMs). We quickly recognized the potential these technologies hold in transforming cybersecurity and couldn’t wait to explore their applications. At that point, tools like LangChain and OpenLLM were gaining momentum, and we were eager to tap into their capabilities.
Our first foray was straightforward: we aimed to focus on summarization and Q&A functionalities. The successes we achieved early on fueled our desire to push the envelope further, leading us to explore inferencing capabilities—where models could pull insights from data that weren’t immediately obvious. This deep dive illuminated a broader landscape of applications within the AI/ML domain.
As our exploration progressed, we turned our attention to sophisticated concepts such as retrieval-augmented generation (RAG) and its related tools, establishing a strong footing in the AI/ML systems. A significant turning point came when we encountered insights from OWASP, reshaping how we approached security in the context of AI/ML and LLMs—a challenge that many in the cybersecurity space face, including our clients. This led us to the development of an LLM Scanner, a solution designed to help teams confidently navigate the complex security landscape of AI/ML.
February 2024 saw OWASP release the “LLM AI Cybersecurity & Governance Checklist,” underlining the need for LLM-specific security scanners and raising awareness for the associated cyber threats. Although the checklist predated our endeavors, its arrival marked a pivotal moment for us, steering our ambitions from simpler use cases to our ultimate goal: building a comprehensive LLM scanner capable of safeguarding Generative AI systems.
Creating a Proof of Concept
To build an effective LLM scanner, we established clear goals for our Proof of Concept (POC). Our main objective was to identify and assess vulnerabilities in LLMs. We defined our success criteria to include recognizing known vulnerabilities, producing accurate risk scores that reflect severity and impact, and ensuring our evaluations were consistent. Plus, we wanted to demonstrate scalability and seamless integration with existing workflows, all the while keeping our solution lightweight and efficient.
Transforming Vulnerability Scanning into AI Security
Our roots at Qualys are firmly planted in vulnerability management, and we’ve since expanded our offerings to include robust solutions for web application and host-level vulnerability scanning. Armed with insights from OWASP, we set out to adapt our security expertise to the unique challenges posed by LLMs. Our goal became clearer: develop an AI/ML scanner aimed at securing the growing ecosystem of LLM applications. This transition prompted us to rethink vulnerability management—instead of shoehorning LLM detection into our existing API Security framework, we recognized the value of creating a dedicated module to unify all functions across various levels.
We started with the OWASP Top 10 for LLM Applications, homing in on vulnerabilities such as prompt injection, insecure output handling, and model theft. Initial tests on LLMs like phi2 and Vicuna showed great promise, but we knew a more refined approach was necessary to grasp the nuanced risks. That’s when we integrated a Judge LLM, a model specifically designed to evaluate responses and ascertain security risks more systematically.
In our experimentation, we adapted well-known foundational models using innovative prompt engineering to create a Judge LLM capable of critically evaluating responses from our target LLMs. However, this evaluation wasn’t straightforward. One of our biggest hurdles was ensuring that the Judge LLM didn’t factor in the prompts themselves when gauging vulnerability severity. We also aimed for consistent scoring methodologies that paralleled traditional static vulnerability metrics like CVSS. To achieve this level of consistency, we honed our prompt engineering skills—experimenting with numerous phrasings and techniques until we hit the sweet spot.
Defining Goals and Metrics for Our Proof of Concept
Our POC concluded with a simple application capable of prompting any LLM, assessing responses, and scoring vulnerabilities. We compiled a dataset of approximately 1,000 questions that spanned 16 different risk categories and subcategories.
This dataset was developed based on attack categories identified by OWASP, featuring a mix of LLM-generated queries, handcrafted questions, and selections from open-source datasets available on Hugging Face. The creation of our original dataset involved extensive iterations and careful review. We used Hugging Face’s text-generation inference platform to prompt the target LLM, subsequently applying our Judge LLM for evaluations. Throughout this process, we gained valuable insights into the strengths and weaknesses of various LLMs and identified prevalent vulnerability types.
We further enhanced our dataset by introducing over 20 jailbreak attacks, drawing from open-source research to explore methods of jailbreaks and other attack strategies. Jailbreak attacks aim to bypass an LLM’s built-in restrictions, leading to unwanted or harmful behaviors. Understanding jailbreaking—especially in closed-source models—requires advanced techniques to circumvent inherent safeguards. The focus on multimodal attacks, including vulnerabilities tied to images, audio, and video, became crucial for our comprehensive assessment of LLM security.
For practicality’s sake, we restricted ourselves to jailbreak instances that could be executed without requiring an attacker LLM. This decision streamlined our solution deployment, making it more accessible for customers to incorporate it into existing workflows without the strain of additional resource demands.
Integrating into the Qualys Ecosystem
The next step was evolving our proof of concept into a full-fledged enterprise solution and integrating it within the Qualys ecosystem. We opted for a seamless integration with our Web Application Scanner (WAS), avoiding the need for the LLM scanner to function as a standalone tool. This approach allowed us to build on our existing vulnerability management capabilities while offering users both traditional and AI-specific vulnerability scanning from a single platform. By doing so, Qualys emerges as the only company that provides an integrated solution addressing discovery, vulnerability management, and model scanning, setting us apart in the market.
To enhance our AI/ML security efforts, we tapped into our Cybersecurity Asset Management (CSAM) team, already experienced in software fingerprinting and tagging on assets, to catalog recognized AI/ML software in our inventory. This was key in helping customers better understand their AI/ML footprint and associated risks. Our vulnerability researchers also identified CVEs related to AI/ML stacks, adding more than 700 unique vulnerability IDs (QIDs) for known threat vectors, coupled with software composition analysis (SwCA) capabilities to broaden our threat detection coverage.
Aligning with MITRE ATLAS and OWASP
A vital component of our solution involved aligning our scanner’s capabilities with established security standards. We mapped our detection capabilities to MITRE ATLAS—an extensive knowledge base detailing adversary tactics against AI-enabled systems—as well as OWASP’s LLM Top 10 vulnerabilities. This meticulous mapping process led to a refinement of our dataset, reducing our initial 1,000 questions to around 600 to ensure relevance and value in each inquiry.
The integration of our AI/ML scanner with traditional vulnerability management detections and OWASP insights resulted in a more comprehensive view of our customers’ AI/ML security postures. This integration contributes to our TruRisk initiative, a proprietary method we developed for assessing risk scores across assets and infrastructure. The synergy between our AI/ML scanner and TruRisk enhances the risk assessment framework, giving organizations thorough insights into the risks tied to their AI/ML environments.
Advancing Toward Product Launch
With the foundational work in place, we set our sights on launching the first version of Qualys TotalAI, which encompasses several exciting components:
- AI Fingerprinting with CSAM: Identifying AI/ML workloads and increasing visibility into organizational AI assets.
- VM for AI Workloads: Ensuring that the infrastructure supporting AI models is secure.
- TotalAI User Interface Module: A dedicated interface streamlining AI/ML security management, improving user interaction with the scanner results.
- Model Scan (External Only): Scanning endpoints on platforms like Hugging Face and AWS Bedrock for broad compatibility with popular AI services.
- OWASP LLM Top 10 Coverage: Focusing on critical vulnerabilities affecting LLMs, laying a solid groundwork for future enhancements.
- TruRisk Report: Delivering a comprehensive risk score that combines traditional and AI-specific vulnerabilities to help organizations prioritize remediation efforts.
Over the next few months, our team focused on refining the LLM scanner, increasing compatibility with various inferencing endpoints, and developing user-friendly mockups. Our CSAM, Threat Research, and VM teams were also instrumental in achieving successful AI/ML software fingerprinting and detection of additional vulnerabilities, ensuring our solution was both effective and accessible.
Competitive Landscape and Distinctiveness
As we advanced, several LLM scanner solutions began to surface in the market. Many of these acted as pip modules, scanning LLMs using prompts—mirroring our initial approach. However, our strategy evolved to offer a more holistic security solution, integrating AI fingerprinting, robust vulnerability scanning, OWASP attacks, and MITRE ATLAS insights. This unique risk-scoring mechanism via TruRisk set Qualys TotalAI apart, making it a comprehensive solution that extends beyond mere vulnerability detection.
Our methodology emphasizes incorporating AI/ML security directly into existing security workflows, rather than isolating it as a distinct concern. This integration allows organizations to see a clearer, more complete picture of their security posture, addressing both conventional and AI-specific risks comprehensively.
Future Challenges and Directions
Despite the strides we’ve made, new challenges loom on the horizon. As our solution matures, we’re aware that the rapidly evolving AI/ML landscape introduces complexities that require ongoing adaptation. Our mission now is to enhance our scanner’s capabilities to support multimodal attacks, including those associated with image, audio, and video vulnerabilities. Tackling these multifaceted threats calls for fresh techniques and tailored datasets.
We’re also eyeing guardrail technologies such as NVIDIA NeMo and Guardrails AI to shield against malicious or jailbreak prompts. These safeguards—acting as a barrier—analyze inputs and outputs for signs of harmful patterns. However, the fact remains that even guardrails can be circumvented, highlighting potential vulnerabilities that still need addressing. This represents an opportunity for Qualys TotalAI to incorporate guardrails within our scanner, fine-tuning them based on real-world findings to present customers an adaptable security solution.
Conclusion
Reflecting on our journey from initial proof-of-concept to product offering, we’ve learned immensely about AI/ML security, prompt engineering, and seamlessly blending new technologies into our existing ecosystem. Each challenge we faced—from honing evaluation techniques to integrating our solution broadly—has provided valuable insights and allowed us to improve our approach.
We are confident that Qualys TotalAI will play a critical role in enabling organizations to secure their AI/ML infrastructure amid rapid technological evolution. By combining AI fingerprinting, vulnerability scanning, OWASP strategies, and a distinct risk-scoring mechanism through TruRisk, we are positioning our solution to stand out in the market.
As we move forward, our mission remains clear: help our customers navigate the intricate realms of AI/ML cybersecurity. With an ever-evolving landscape, we are committed to staying ahead, ensuring that organizations can confidently protect their AI assets in an unpredictable environment.
What’s Next?
Looking ahead, we’re excited to expand our solution to tackle even more attack types and modalities, ensuring TotalAI stays ahead in AI/ML security. We’re also innovating internally to scan models that aren’t publicly accessible. By continuously refining our risk-scoring methodologies, we aim to empower our customers with the insights they need to make strategic decisions regarding their security postures. The AI/ML landscape is ever-changing, and we’re dedicated to being at the forefront, helping customers secure their AI assets along the way.
Contributors
Girish Aher, Senior Manager, Data Platform, Qualys
Kedar Phadnis, Senior Manager, Program Management, Qualys
Ramesh Mani, Principal Architect, Data Platform, Qualys
Sheela Sarva, Director, Web Application Security, Qualys
Related
The AI Buzz Hub team is excited to see where these breakthroughs take us. Want to stay in the loop on all things AI? Subscribe to our newsletter or share this article with your fellow enthusiasts.
