Attackers use AI against two targets at once: the AI systems organizations are racing to deploy, and the people those organizations employ. They move faster than any periodic testing schedule, probing for the one weakness that turns into a breach.
The human attack surface is where that asymmetry hits hardest, because many security programs still test their workforce far less rigorously than their software. The cost shows up in the numbers: attackers used AI in 16% of data breaches in 2025, and a human element still factors into 60% of breaches.
This article defines AI red teaming, shows how it applies both to AI systems and to the human attack surface, and lays out what continuous AI red teaming takes to keep pace with AI-powered attacks.
Key Takeaways
- AI red teaming runs in two directions at once: probing AI systems for jailbreaks, prompt injection, and data leakage, and pointing AI at the organization's own defenses to test how employees hold up against adversarial social engineering.
- The human attack surface is the larger gap, because most security programs scan and patch software continuously while measuring workforce risk through annual videos and quarterly phishing emails that do not reflect how attackers actually operate.
- Effective red teaming of people requires three things working together: coverage across the channels attackers use (voice, SMS, email, Teams, Zoom), simulations grounded in live attacker behavior rather than generic templates, and results that route into training tied to each employee's specific failure mode.
- Doppel closes that loop on one platform, correlating real attacker infrastructure through the Threat Graph, converting detected threats into multi-channel simulations in one click, and routing failed drills into targeted training so the drills employees run match the attacks targeting the organization that week.
What Is AI Red Teaming?
Red teams attack their own systems the way an adversary would, so they find the weaknesses before a real attacker does. AI red teaming applies that discipline in two directions at once.
Red Teaming Finds Weaknesses by Imitating a Real Adversary
Red teaming works because it abandons the assumption that people will use systems as intended. A standard test confirms a feature behaves correctly. A red team asks what happens when someone deliberately tries to break it, mislead it, or turn it against the organization that built it. That is why red teaming surfaces failures outside functional testing's scope.
AI red teaming brings a structured testing effort and adversarial methods to bear on an AI system, surfacing flaws and vulnerabilities a functional test would pass over. The discipline predates AI, but the targets have multiplied.
AI Red Teaming Means Both Testing AI Systems and Using AI to Test the Human Layer
The first direction points the red team at AI systems: probing models and the applications built on them for jailbreaks, prompt injection, and data leakage. The second direction points AI at the organization's defenses, running adversarial social engineering against employees at machine speed.
These directions are converging because attackers converged first. The same move works on both sides: malicious instructions hidden inside a document an AI copilot reads, and attacker intent hidden inside a message an employee trusts. Attackers now treat your deployed AI and the people who use it as one surface.
How AI Red Teaming Tests AI Systems
Applied to AI systems, red teaming deliberately attacks models and the applications around them to surface failures standard quality testing misses. The attacks fall into two groups: those that manipulate the model through its inputs, and those that target the data and pipeline behind it.
Adversarial Prompts Expose What Standard Quality Testing Misses
Adversarial prompting attacks the model through its own input channel, and it is one of the most prominent failure classes in production AI systems. Prompt injection is one of the top risks in the OWASP Top 10 for LLM Applications. The root cause is a semantic gap: developer instructions and user input share the same format, so the model cannot reliably tell a command from data.
Jailbreaking pushes a model to disregard its safety protocols entirely. Indirect prompt injection creates a harder problem. Attackers hide instructions inside external content the model processes: a webpage, a resume, an email, even white text on a white background.
The 2025 zero-click vulnerability in Microsoft 365 Copilot let a single crafted email make Copilot access internal files and exfiltrate their contents. Red teaming a generative AI system surfaces security and safety failures at once, the kind a QA script never looks for.
Data Poisoning and Model Extraction Target the System Behind the Output
Adversarial prompts attack the model at runtime. Poisoning and extraction attack the system around it. Data poisoning corrupts what a model learns: an attacker who taints a very small share of the training data can plant failures that surface later, which keeps the cost of a large-scale attack low.
Model extraction works from the other side, querying a deployed model to reconstruct its training data or reverse-engineer the proprietary logic behind it. Data extraction and prompt extraction sit alongside poisoning as distinct attack categories. Functional tests miss all of these because the failure lives behind the visible output, which is exactly why a red team has to go looking.
Why the Human Attack Surface Is the Bigger Red Teaming Gap
The same AI attackers point at models can also be turned on people. Because many security programs test their software more rigorously than their workforce, the human attack surface becomes one of the weakest and least-measured parts of the defense.
1. AI Lets Attackers Run Social Engineering at Machine Speed and Scale
AI turns social engineering from a craft into an industrial process. Voice cloning can train on small public audio samples and run live during a call. When attackers automate phishing emails with AI, click-through rates jump to 54%, compared with 12% for standard attempts, a 4.5x increase that reflects how AI changes phishing efficacy at the campaign level.
The attacks already work at scale. In one documented incident, attackers generated every participant in a video conference, including the apparent CFO, with AI-generated deepfakes, and an employee at a global engineering firm joined the call and authorized a major multimillion-dollar transfer.
The firm's IT environment stayed intact: no malware, no intrusion, no compromised credentials. The attack ran entirely on psychology and synthetic media.
2. Most Organizations Test Their Software Harder Than Their People
Many enterprises run mature, continuous processes for software vulnerability management while handling workforce risk through periodic training and simulation. Security teams scan, patch, and re-test software on a continuous cadence.
Employees get an annual video and a quarterly phishing email, and then the program treats their risk as managed. The data says otherwise.
3. Annual Phishing Tests Leave Most Employee Risk Unmeasured
Annual phishing testing in isolation measures almost nothing that matters. A multi-month study of anti-phishing training found that completing annual awareness training did not predict whether an employee clicked a simulated phishing link, and embedded training cut failure rates only slightly. Completion rates and click rates become vanity metrics when programs treat them as proof of resilience.
They show that an employee finished a requirement or failed a narrow test, but do not indicate whether that person would resist an attacker on a live call.
What AI Red Teaming Looks Like Against People
Red teaming the human attack surface uses AI agents to run adversarial, conversational social engineering across the channels attackers use. The agent adapts to each target in real time the way a skilled human red teamer would, at a scale and frequency manual teams cannot sustain.
AI Agents Run Live, Adaptive Attacks Across Voice, Email, and Chat
AI voice agents make live outbound calls and respond to whatever the target says. They apply pressure, urgency, and appeals to authority while holding a fixed objective. That adaptiveness is exactly what attackers weaponize: when a target deflects, the agent pivots mid-call to an email or SMS follow-up and keeps the attempt alive.
This mirrors how real attacks unfold. Groups like Scattered Spider use social engineering against IT help desks through phishing, vishing, smishing, urgency, and authority cues. A red team that tests only single-step email leaves the channels attackers actually pivot through undermeasured.
Simulations Built From Real Attacks Test the Threats Employees Actually Face
A simulation becomes more useful as it gets closer to a live threat. Generic templates risk training employees to recognize last quarter's drill rather than live attacker behavior. OSINT-driven scenarios that mirror how attackers research and target employees build threat recognition that holds up when a new lure arrives.
The strongest version of this loop converts a threat detected in the wild into a drill the same week. When a cloned-voice scam targets a finance team, the most useful test uses that exact scenario, defanged and run against the people in its path, before the real call arrives.
Continuous, Automated Testing Replaces the One-Off Red Team Engagement
A point-in-time red team validates conditions at a single moment. That validation can become stale as systems, configurations, and threats change, and that gap only widens. AI systems are probabilistic, so an attack that fails on one attempt can succeed on the next. Repeated attempts can raise the success rate materially, which means a single test understates real risk and continuous testing produces a more reliable signal, for AI systems and people alike.
Manual red teaming is difficult to run continuously given the limits of time, budget, and expert availability. Continuous automated testing runs persistently and probes for the weaknesses that open between scheduled exercises. Human-led engagements stay valuable for deep, creative validation. Continuous testing keeps the picture current in between.
What Continuous AI Red Teaming Requires
Running AI red teaming against people continuously takes three things working together: coverage of the channels attackers use, simulations grounded in live attacker behavior, and a direct line from each result into targeted training.
Each one closes a gap that undermines the program if left open:
- Cover the channels attackers use most. A program tied to email measures one lane of a multi-channel attack. Modern campaigns combine ads, messaging apps, voice, and spoofed pages in coordinated funnels, and untested channels become pivot paths. Coverage has to span voice, SMS, email, and collaboration tools like Microsoft Teams and Zoom, because that is where business communication, and therefore social engineering, now happens.
- Ground tests in live attacker behavior. A simulation disconnected from real threats can teach employees to spot the drill. Grounding tests in the campaigns actually targeting the organization, including the lures, landing pages, and infrastructure attackers are using this week, produces threat recognition that holds up under a real attack.
- Turn results into targeted training. A test that ends at a click rate changes nothing. Security teams have to measure risk per employee across tested channels, then route each result into training tied to the specific failure mode, because a smaller group of users can drive concentrated risky behavior and generic training spread across everyone misses them. The loop only reduces risk when each result feeds a targeted intervention.
These three requirements describe a closed loop: detect the threats targeting the organization, run them as drills across the channels attackers use, and turn the results into training that measurably reduces risk.
How Doppel Brings AI Red Teaming to the Human Attack Surface
Most organizations run that loop with a fragmented stack: separate tools for detection, simulation, and training, none sharing a signal. Doppel is the AI-native Social Engineering Defense (SED) platform that unifies Digital Risk Protection and Human Risk Management, which lets it run continuous AI red teaming of the human attack surface against the real attacks it detects and dismantles in the wild.
The Doppel Threat Graph solves the signal problem first. It correlates domains, social, ads, telco, dark web, and crypto into campaign-level views, so security teams see the attack pattern behind the alerts. Recon AI Agents add the same OSINT layer attackers use, ingesting job postings, company announcements, and public filings to produce templates already grounded in the organization's real context. Security teams then convert any detected threat into an employee simulation in one click.
That conversion closes the realism gap. The platform clones, defangs, and wires the lure, landing page, and infrastructure into a multi-channel drill. Dynamic Simulation then runs live voice agents that pivot mid-call to email or SMS, while line-by-line sentiment analysis on the transcript helps teams harden the protocol that failed. Help desks and contact centers are where groups like Scattered Spider strike directly.
Helpdesk Mode trains voice agents to work through IVR trees, wait through hold queues, and handle line transfers, so testing reaches the surfaces a single-step email program never touches. When attackers target a CFO today, the platform turns that scenario into an org-wide drill tomorrow.
The same loop supports disruption and closes the third requirement. The platform's agentic AI correlates, prioritizes, and executes takedowns of the underlying attacker infrastructure at scale, so analysts focus on the complex escalations that require human judgment.
Per-employee risk profiles then route each failed drill into training tied to the specific failure mode. Coinbase defends one of the most-impersonated brands in crypto and has used the Doppel Platform to dismantle impersonation accounts and fraudulent domains, the same wild-caught threats that become the drills its people run.
Red-Team Your People Like You Red-Team Your Code
The organizations that pull ahead will red-team their people with the same rigor and cadence they already demand for their code, because attackers are using AI to test both. That cadence makes your organization too costly to attack.
Request a demo to get started with Doppel.