--- title: "The Model Found the Vulnerability. The Tool Call Became the Incident." description: "A balanced operating model for AI-assisted pentesting: where models improve coverage and evidence work, where excessive agency turns a valid test into a destructive action, and how to keep cloud, shell, and Domain Admin authority outside the model." date: 2026-08-21 author: Sevban Dönmez (@jankesec) canonical: https://jankesec.com/posts/ai-pentest-tool-call-incident/ --- ## AI-assisted pentesting risk in 60 seconds AI can make a pentest faster. It can also make a pentest the first incident in the final report. Imagine that an agent discovers an authorization flaw in a cloud API. The finding is real. The agent has a connected tool with delete capability. Instead of producing a bounded request for review, it invokes the vulnerable operation against a production object. The assessment no longer has a promising hypothesis. It has missing data. Or place the same agent on an administrator workstation. The interactive session carries Domain Admin authority, and the agent generates one incorrect directory command. The command does not need to be malicious. It inherits the authority of the process that launched it. A plausible model error has become a deterministic change across the domain. These are not arguments against AI-assisted security testing. They are arguments against making a probabilistic planner the authorization boundary for deterministic tools. **A model proposal is not authorization.** This article presents a source-reviewed operating model, not a report of an AI-caused production incident. The scenarios are deliberately constructed to examine the failure mode without claiming an event or inventing performance numbers. ## The same capability creates both value and risk The useful properties of an AI pentest assistant and its dangerous properties are often the same. It can explore many hypotheses, translate between technologies, keep context across a long engagement, and call tools without fatigue. That creates coverage. It also creates speed, scale, and persistence when the selected action is wrong. | Capability | Pentest advantage | Operational cost when authority is excessive | | --- | --- | --- | | Rapid hypothesis generation | Surfaces overlooked trust boundaries and test variations | Produces persuasive but unsupported paths faster than a human can review them | | Tool orchestration | Removes repetitive collection and normalization work | Converts ambiguous language or hostile context into a real side effect | | Long-context correlation | Connects identities, routes, configurations, and observations | Allows untrusted target content to influence later privileged decisions | | Automatic retries | Recovers from transient collection failures | Repeats a non-idempotent operation after a timeout or partial success | | Persistent memory | Preserves engagement context | Carries secrets, stale scope, or poisoned instructions between tasks | | Report generation | Accelerates evidence tables, reproduction notes, and remediation drafts | Turns an unverified interpretation into an authoritative-looking finding | The dividing line is not whether the model is “smart enough.” It is whether a mistake can cross from analysis into external state.

The control that matters is outside the prompt. It validates the proposed operation against engagement scope, effective identity, arguments, side effects, and a specific approval.

## Where AI genuinely helps a pentest AI is most valuable where the output remains a candidate for human or deterministic verification. That includes substantial parts of a professional engagement. ### Before execution An assistant can transform the rules of engagement into a test inventory, map assets to hypotheses, identify missing preconditions, and produce a risk-ranked queue. It can compare API schemas, configuration exports, source code, architecture diagrams, and previous observations. None of these tasks requires production mutation. This is also where AI can challenge the tester. It can ask whether an apparent IDOR is actually cross-tenant, whether a credential has effective permissions, whether an attack path ends at a privileged resource, and what negative control would disprove the hypothesis. ### During read-only collection Models are useful for selecting among narrowly defined discovery tools, normalizing outputs, and correlating large datasets. A read-only worker can enumerate documented API routes, parse scanner results, compare identity relationships, or find inconsistencies between inventory and observed services. “Read-only” must describe the server-side operation, not the tool's name. A GET-like request can still trigger work, expose secrets, consume scarce resources, or acknowledge a message. The tool contract therefore needs an explicit side-effect classification. ### During evidence and reporting An assistant can assemble timestamps, hashes, request identifiers, screenshots, audit records, positive tests, and negative controls into a consistent evidence matrix. It can draft remediation that maps to the failed decision rather than merely naming a product. But the model should not promote its own conclusion. Tool and scanner outputs remain candidate evidence until scope, reachability, effective permission, protected resource, and business impact are independently established. ## Where AI becomes dangerous The familiar concern is hallucination: an AI invents a finding. That matters, but excessive agency creates a more immediate problem. The finding can be correct while the proof action is unsafe. [OWASP's Excessive Agency guidance](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) describes damaging actions caused by unexpected, ambiguous, or manipulated model output when a system exposes too much functionality, permission, or autonomy. Its examples include extensions that can delete data and database connections that permit destructive statements. The security failure is not limited to model behavior; it is the authority designed around the model. Five patterns deserve special attention in a pentest environment: 1. **Correct finding, wrong proof.** The agent reaches a real authorization bypass but selects an irreversible object or an excessive set of records to demonstrate it. 2. **Ambient privilege.** A local shell silently inherits the operator's cloud session, SSH agent, Kubernetes context, browser cookies, or directory token. 3. **Hostile context.** A page, repository file, ticket, or retrieved document contains instructions that the planner mistakes for engagement guidance. 4. **Retry ambiguity.** A tool times out after the server committed the change. The planner retries, multiplying a non-idempotent effect. 5. **Scope drift.** Similar tenant names, stale memory, redirects, aliases, or shared control planes move the action outside the approved asset set. ## Failure path one: the delete API is real Suppose an agent observes that one tenant can address another tenant's resource. A safe test would use an engagement-owned canary and the least harmful operation capable of proving the authorization failure. An unsafe agent sees a delete function in its tool catalog, supplies the discovered object identifier, and executes it using an identity that the vulnerable API accepts. At that point, “the vulnerability existed” is not a defense. The tester chose the execution model. Deletion is the impact, not the evidence. The protective design is layered: - Destructive methods and operations are denied by default, not hidden in the prompt. - The broker locks tenant, account, region, host, and object namespace to the engagement record. - Proof uses a tester-created canary with a recorded owner and cleanup plan. - The UI resolves and displays the exact object, action, count, identity, and expected effect. - Plan and execution are separate states; a plan cannot mutate the target. - Approval is bound to the resolved parameters and expires if any parameter changes. - Non-idempotent calls are never automatically retried after an ambiguous result. - A second evidence path verifies the result through an audit record or independent read operation. - Recovery prerequisites—versioning, snapshot, backup, or recreation—are checked before a bounded change is authorized. Their existence is verified, not assumed. A vendor “dry run” helps only if that exact API guarantees no mutation. Renaming a live request `dry_run=true` inside an agent wrapper does not create a safety property. ## Failure path two: the agent is already Domain Admin Now move the agent from a cloud worker to an administrator workstation. It is launched from a session carrying Domain Admin credentials. The model may intend to inspect a group, validate a path, or clean up a test object. It produces the wrong command, resolves a broad target, or acts on an attacker-controlled value copied from directory data. The operating system does exactly what it was asked to do. The model's uncertainty is irrelevant to the directory service. Microsoft's [least-privilege Active Directory guidance](https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/implementing-least-privilege-administrative-models) uses the same underlying lesson: when an administrator logs on with a privileged account and inadvertently runs malicious code, that code receives the account's local and domain authority. Microsoft also places Domain Admin among the highest-impact privileged roles and recommends isolating those roles from general-purpose activity in its [privileged-access security model](https://learn.microsoft.com/en-us/security/privileged-access-workstations/privileged-access-security-levels). For AI-assisted testing, the implication is direct: - Never run exploratory agents inside an interactive Domain Admin session. - Use a clean, isolated worker with no inherited tickets, browser sessions, SSH agent, or cloud CLI context. - Give discovery a dedicated read-only directory identity with a documented search base and result cap. - Deny directory mutations in the normal tool catalog. A separate change workflow should be rare, named, time-bound, and limited to engagement-owned lab objects. - Resolve distinguished names, domains, object types, and maximum object counts before approval. - Require a clean precondition check and an explicit rollback path for every permitted lab change. - Test high-impact directory behavior in a representative test forest, not by lending production Domain Admin authority to the model. The goal is not to make a Domain Admin prompt more careful. The goal is to ensure that no ordinary model decision can reach Domain Admin execution. ## Authority must live outside the model Prompt instructions are useful for behavior shaping. They are not access control. A model can misread, forget, override, or be manipulated around them. The enforcement point must be a deterministic component that the model cannot rewrite. The [Model Context Protocol tools specification](https://modelcontextprotocol.io/specification/draft/server/tools) reflects this separation: servers are expected to validate inputs, enforce access control, rate limit calls, and sanitize outputs, while clients should expose tool inputs, confirm sensitive operations, apply timeouts, and preserve audit logs. Its [authorization guidance](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization) also recommends least-privilege scopes and step-up authorization. These are useful protocol properties, but the deployment still has to implement them correctly. A practical capability ladder keeps discussion concrete: | Level | Allowed capability | Suitable examples | Default decision | | --- | --- | --- | --- | | L0 — Offline | No target connection | Scope analysis, code review, hypothesis generation, report drafting | Allow with data-handling controls | | L1 — Observe | Bounded, verified read operations | Inventory, configuration retrieval, schema inspection, evidence normalization | Allow through scoped worker | | L2 — Reversible lab action | Named canary, bounded effect, known rollback | Test-account workflow, lab-only state transition, disposable resource validation | Require parameter-bound approval | | L3 — High impact | Destructive, broad, privileged, or production mutation | Delete, policy change, credential reset, directory mutation, Domain Admin shell | Deny to autonomous execution | This ladder should be attached to tool definitions, identities, and workers—not merely written in an engagement prompt. ## A safer execution architecture There are three common arrangements. An advisor with no tools is comparatively safe but cannot verify much. A fully autonomous agent with broad credentials is fast but creates an unacceptable blast radius. The useful middle is brokered execution: the model creates a typed proposal; an independent service decides whether a narrowly scoped worker may carry it out.

A broker issues authority only after resolving the proposal against current scope and effect. Execution and verification use separate paths so the agent cannot grade its own work.

The tool contract should expose the safe property directly. For example, an inspection request can be both typed and bounded: ```json { "operation": "directory.object.inspect", "mode": "read_only", "scope": { "domain": "lab.example", "max_objects": 50 }, "requires": ["engagement_id", "scope_match"], "side_effects": "none" } ``` A potentially destructive cloud action should not masquerade as another ordinary tool. The model may produce a non-executable plan request that identifies the effect: ```json { "operation": "cloud.resource.change.request", "mode": "plan_only", "effect": "destructive", "requires": ["named_canary", "recovery_verified", "separate_change_authority"], "autonomous_execution": "denied" } ``` Free-form shell access defeats most of this design. A shell is not one tool; it is a capability to discover and compose every executable, credential, file, socket, and inherited session available to the worker. If a task truly requires it, use an isolated disposable environment, a command allowlist, a low-privilege identity, network egress limits, explicit target locks, and recorded operator supervision. ## Approval that means something A generic “Are you sure?” dialog is not informed approval. The operator must see the decision that will actually execute: - engagement and scope identifier; - resolved tenant, subscription, domain, host, and object—not a model-provided alias; - tool and operation; - executing identity and its effective role; - object count and maximum blast radius; - whether the action is read-only, reversible, destructive, or privilege-changing; - expected state difference and verified recovery path; - reason the action is necessary when a safer proof is insufficient; - expiration time and an immutable approval identifier. If any resolved parameter changes, the approval becomes invalid. Approval for “test the API” must not authorize “delete object 84 in production.” Approval for a read-only directory query must not carry over to a group membership change. This is consistent with formal testing discipline. [NIST SP 800-115](https://csrc.nist.gov/publications/detail/sp/800-115/final) frames technical security testing as planned work, and NIST defines [rules of engagement](https://csrc.nist.gov/publications/detail/sp/800-115/final) as the detailed constraints and authorized activities established before testing begins. An AI agent does not relax those constraints; it makes machine-enforceable constraints more important. ## Evidence without incident The safest proof is the smallest controlled state difference that isolates the failed security decision. It is not the largest impact the agent can create. Start with an engagement-owned canary. Record its owner, initial state, permitted operation, and cleanup method. Capture the proposal before execution. Preserve the broker decision, resolved parameters, worker identity, request or correlation identifier, result, and independent state observation. Repeat the same test with one decisive variable removed. For an authorization flaw, that negative control may be an object owned by the caller, a correctly denied cross-tenant request, or the same operation under an identity without the vulnerable path. For a prompt-influenced tool call, repeat the task with the hostile instruction removed while holding the identity, tool set, and target constant. The agent can help assemble this record. It must not be the only component that decides whether the record proves the claim. ## Evidence matrix | Claim | AI contribution | Required execution evidence | Negative control | Limit that must remain explicit | | --- | --- | --- | --- | --- | | Cross-tenant authorization is missing | Correlates object identifiers and proposes a canary test | Request identity, canary ownership, response, audit event, bounded state diff | Same request against an authorized object or under a non-vulnerable identity | No real customer object was modified or deleted | | Untrusted content influenced tool selection | Identifies the decision change and reconstructs context | Controlled source, proposed call, policy result, tool audit ID | Same task with the instruction removed | Model output alone does not prove tool execution | | Directory query reveals an attack path | Correlates group, session, ACL, and host observations | Fresh directory data, effective identity, reachable privileged resource, independent validation | Remove or disable one decisive edge | Graph reachability is not equivalent to compromise | | Broker blocked a destructive action | Generates a prohibited plan request | Resolved parameters, deterministic deny reason, no worker credential issued, no target state change | Approved read-only request reaches a bounded worker | A model refusal is not evidence that the broker works | | Lab change workflow is reversible | Builds the test and cleanup checklist | Pre-state, named object, parameter-bound approval, post-state, verified rollback | Same proposal outside lab scope is denied | Lab behavior does not prove production safety | ## A decision table for real engagements | Activity | Recommended AI role | Execution rule | | --- | --- | --- | | Scope and architecture review | Primary assistant | Offline; source references retained | | Passive discovery and output normalization | Orchestrator | Read-only worker, target allowlist, rate and result caps | | Vulnerability hypothesis generation | Copilot | Human or deterministic validation before finding status | | Production exploit with reversible canary | Planner only | Named approval, bounded worker, independent verifier | | Cloud deletion or broad data mutation | Documentation only | No autonomous execution | | Active Directory discovery | Correlator | Dedicated read-only identity and fixed search base | | Domain Admin or equivalent privileged shell | No agent execution | Isolated human-controlled administrative workflow | | Report drafting | Assistant | Every material claim linked to preserved evidence | ## Prepare for the action that should never happen Prevention is not enough. The operating environment needs an incident path before the first tool call: - one control that stops new jobs and revokes short-lived capabilities; - complete, append-only records of prompts, proposals, resolved parameters, approvals, identities, tool results, and retries—with secrets redacted; - separate credentials per engagement and worker so one revocation does not disable unrelated work; - verified snapshots, versioning, or recreation procedures for permitted lab changes; - named technical and business contacts for a scope breach or unexpected side effect; - a rule that ambiguous outcomes pause execution rather than trigger autonomous retry; - cleanup verification and disclosure of any test-created objects. The broader lifecycle matters too. The [NIST Generative AI Profile](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence) places generative-AI risk management across design, development, use, and evaluation. For pentest agents, that means the review cannot stop at the model. Tool definitions, credential issuance, approval UX, logs, worker isolation, evaluation cases, and incident response are all part of the system under test. ## The operating rule AI-assisted pentesting has real advantages: broader hypothesis coverage, faster correlation, repeatable evidence handling, and better continuity across a complex assessment. Those gains are largest when the model works as an analyst and planner. Its disadvantages become unacceptable when broad, ambient authority is attached to that planner: scope drift, prompt injection, mistaken arguments, repeated side effects, credential exposure, and authoritative-looking false conclusions. A valid vulnerability does not make an unsafe proof safe. The durable division of responsibility is simple: > **The model proposes. Scope constrains. Policy authorizes. A least-privileged worker executes. An independent path verifies.** Keep cloud deletion, production mutation, and Domain Admin authority outside ordinary model reach. If a proposed proof cannot survive that separation, the answer is not a stronger prompt. It is a safer test design. ## MITRE ATT&CK mapping | Tactic | Technique ID | Technique name | Defender verification signal | | --- | --- | --- | --- | | Execution | `T1059` | Command and Scripting Interpreter | Deterministic tool broker rejects unauthorized command execution | | Initial Access | `T1190` | Exploit Public-Facing Application | Scope registry validates canonical IP / CIDR before tool dispatch | | Defense Evasion | `T1562.001` | Impair Defenses: Disable or Modify Tools | Agent runtime enforces unbypassable audit logging of raw tool I/O | | Impact | `T1485` | Data Destruction | Absolute denial rule for mutating / destructive API calls | ## Defender action checklist Use this actionable checklist when evaluating or operating autonomous AI tooling: 1. **Enforce strict parameter validation:** Validate all tool arguments against deterministic JSON schemas before invocation. 2. **Enforce out-of-band scope checking:** Compare target hostnames and IPs against engagement boundaries using an immutable registry. 3. **Isolate execution credentials:** Issue ephemeral, single-use API tokens with restricted read-only permissions. 4. **Mandate human-in-the-loop for state mutation:** Require explicit operator approval for any action classified above L1 read operations. 5. **Provide an emergency kill switch:** Ensure one command revokes all active agent credentials and terminates worker containers.