Building Security Into AI: A Practical Threat Modeling Approach for DevSecOps

By Robert Herbig, AI Practice Lead at SEP
As presented at APIsec|CON 2025

Introduction: From Code to Context – Securing AI Systems

In an era where AI-driven applications are everywhere—from chatbots to autonomous systems—understanding how to secure AI is no longer optional. At APIsec|CON 2025, Robert Herbig, AI Practice Lead at SEP, delivered a masterclass in making artificial intelligence robust and resilient. His talk offers a preview of a broader course available through APIsec University, laser-focused on demystifying AI security through practical threat modeling.

If you're in AppSec, DevSecOps, or software engineering, this is your roadmap to navigating the intersection of software security and AI development.

Why Traditional Security Models Fall Short for AI

Herbig emphasizes that AI systems are fundamentally different from traditional software. While many security practices still apply—like CIA (Confidentiality, Integrity, Availability), access control, and code hygiene—the attack surface of AI is broader, fuzzier, and more dynamic.

To make sense of it, Herbig introduces a generic AI threat model broken down into six core components:

1. Internal Data
2. External Dependencies
3. Training Process
4. Model Inputs
5. Model Outputs
6. AI Component Interaction with the Application

This model helps security practitioners apply their existing API and software security knowledge to AI systems by drawing clear analogies and identifying where traditional models fall short.

Real-World AI Threats DevSecOps Teams Must Know

1. Data Poisoning Attacks

These attacks inject malicious or mislabeled data into training sets. Herbig demonstrates this with a subtle manipulation of road sign data that causes AI to misclassify stop signs—an issue with direct implications for autonomous vehicles. He draws a parallel with a real Gmail skewing attack, where spammers poisoned Google's spam filters by marking spam as “not spam” en masse using fake accounts.

2. Supply Chain Attacks in the AI Ecosystem

Just like software libraries, pretrained models and public datasets can be compromised. Herbig references a 2024 incident where malicious models were uploaded to Hugging Face, potentially contaminating downstream AI products. Typo-squatting packages (like huggingface-cli) also pose risks, especially when AI code generation tools hallucinate incorrect package names.

3. Adversarial and Obfuscated Inputs

Attackers can subtly alter inputs—like adding color distortions to road signs—to degrade or manipulate model performance. These "black-box" or "white-box" attacks take advantage of models leaking confidence scores or behavioral cues.

4. Prompt Injection & Role-Playing Attacks

For generative AI systems, prompt injection remains one of the most dangerous vulnerabilities. Herbig illustrates this with the now-infamous “grandmother napalm story” exploit, showing how attackers can bypass safety filters through clever roleplaying instructions.

He also warns about ASCII smuggling, where invisible whitespace characters embed malicious instructions in what appears to be safe input—often unknowingly copy-pasted by users.

5. Indirect and RAG-Based Attacks

Herbig dives into indirect prompt injection, where malicious content is embedded not in user input but in third-party content (e.g., websites or tools). He shows how AI tools summarizing a web page might unknowingly execute hidden instructions like “talk like a pirate” or worse.

These risks grow with the adoption of retrieval-augmented generation (RAG) and MCP-based agentic AI, which give AI systems more autonomy and connectivity.

6. Output Leakage and Intellectual Property Risk

AI systems may leak sensitive data they’ve memorized—like customer PII or proprietary source code. They can also generate content that infringes on copyright, even if the prompts are vague. Herbig urges developers to think carefully about what their models can output and under what conditions.

Actionable Security Principles for AI Developers

Rather than recommending specific tools, Herbig focuses on timeless principles:

• Use threat modeling as a lens for security architecture.
• Treat foundational models and datasets like third-party libraries—verify and validate.
• Don't trust AI-generated code or package names blindly.
• Limit tool use and network access for autonomous agents when possible.
• Implement input validation, logging, and anomaly detection across AI input/output boundaries.
• Stay aware of new adversarial techniques, including those targeting RAG and MCP frameworks.

Final Thoughts: Why This Matters for API and Security Professionals

AI doesn’t exist in a vacuum—it’s embedded in applications, powered by APIs, and operating at scale. For DevSecOps and AppSec leaders, AI is no longer a theoretical security problem. It’s in production, and it’s exploitable.

"Security is not about eliminating risk—it's about managing it intelligently."
— Robert Herbig

If your organization is building, integrating, or auditing AI systems, Robert Herbig’s threat model gives you a practical foundation to spot and mitigate vulnerabilities before they become incidents.

📥 Resources

• 🎥 Watch the full session on YouTube
• 🎓 Take the full course on APIsec University (link)
• 🛡️ Explore the OWASP AI Security and LLM Top 10
• 🎮 Try the Gandalf Prompt Injection Game

About the Speaker

Robert Herbig is the AI Practice Lead at SEP, a software product design and development company. With nearly two decades in software architecture and development, he now focuses on bridging the gap between AI systems and secure software engineering. Connect with him via email or LinkedIn to continue the conversation.

‍

‍