This website uses cookies

Read our Privacy policy and Terms of use for more information.

Expert Checklist: Risk Allocation for AI Agent Actions

What happens when an AI agent does something nobody authorized?

California’s Attorney General sued OpenAI this week, alleging its AI agents escaped testing and accessed other companies’ systems.

Contract teams need terms that answer the same question. To help with that, I’m sharing this expert checklist with real-world advice on how to address unauthorized AI agent actions in AI product contracts, including:

  • General language versus precise lists

  • Risk tiers for agent actions

  • Misuse, malfunction, and autonomous but unintended conduct

  • Edge cases and shared governance

I created this checklist based on the advice shared by John Pavolotsky and Olga V. Mack during a How to Contract webinar earlier this year.

Checklist Download

Here is the link to download the PDF version of this checklist:

Checklist Contents

A. Precision and General Language

“Provider shall use commercially reasonable efforts to ensure the AI Agent operates within the scope of the intended use described in the Documentation.”

  • Benefit of General Language. Consider leaving some standards general while the technology and the regulations move. Over the last 10 to 20 years, requirements in privacy and security have become considerably stricter than practitioners would have predicted. General language may capture that later standard where a fixed list would not.

  • Negative Implication of Precise Lists. Watch for the argument that a specific prohibition permits everything it does not name. A counterparty may say it followed the listed restriction and was therefore free to do other things. Balance precision against that risk rather than defaulting to maximum specificity.

  • Predictability Versus Evolving Practice. Understand that narrow scope and precision give more predictability. Accept that precision also gives up the benefit of practices that may evolve in your favor. Weigh both before you fix the standard in the contract.

B. Risk Tiers for Agent Actions

“Provider shall use commercially reasonable efforts to ensure the AI Agent operates within the scope of the intended use described in the Documentation.”

  • High-Risk Processing Under Existing Law. Start from the definitions of high risk processing already in law. The Colorado AI Act and the automated decision-making technology rules in the California Consumer Privacy Act regulations describe triggers that are fairly consistent among them and fairly well understood. Use them as the baseline for your own tiering.

  • Financial and External System Actions. Treat an agent that can execute a financial transaction as high risk. Apply the same tier to an agent that can change anything in an external system. Include changes to an internal system that have knock-on effects you cannot quite envision.

  • Obligations Tied to Risk Category. Categorize agent actions by risk level in an exhibit, separating internal actions, actions that modify records, and actions that affect third parties or create legal obligations. Define the provider’s obligations for each category. Tie the provider’s responsibility for unauthorized actions to the applicable category.

C. Misuse, Malfunction, and Autonomous Conduct

“Provider shall use commercially reasonable efforts to ensure the AI Agent operates within the scope of the intended use described in the Documentation.”

  • Three Failure Categories. Separate misuse, malfunction, and autonomous but unintended conduct before you draft. The three are used interchangeably in conversation and mean different things for agents. Decide which one you are addressing well before drafting, because the categories map different risks.

  • Misuse and Malfunction Allocation. Allocate misuse, where the customer violates constraints, to the customer unless the facts are unusual. Allocate malfunction, where the system fails technically, to provider remediation unless the facts are unusual. Know the unique facts of your own deal and of the other side.

  • Autonomous but Unintended Conduct. Address the case where the agent operates within its parameters and produces an unanticipated outcome. Most contracts are thin on this category, which has appeared mainly in smart contract agreements in the crypto context. Give it predefined notification, suspension, and root cause processes.

  • Operational Incident Framework. Build an operational incident framework rather than relying on a breach clause. Assume something will go wrong and specify how the parties handle it. Chart the first five steps in the contract, which buys time to work out the next five.

D. Edge Cases and Governance

“Provider shall use commercially reasonable efforts to ensure the AI Agent operates within the scope of the intended use described in the Documentation.”

  • Edge Cases Becoming the Case. Normalize edge cases in your risk analysis rather than discounting them. An edge case that never happens becomes the only thing anyone talks about once it does. Identify the unlikely events and the consequences that would follow.

  • Shared Interest in Governance. Treat catastrophic unintended outcomes as a shared problem rather than a matter for blaming the other side. Neither the provider nor the customer wants the reputational damage from a catastrophic result. Put guardrails, guidelines, and governance in place together, independent of who ultimately pays.

  • Agent Interactions and Reasonable Care. Treat the agent as a product for liability purposes. Consider what happens when the agent interacts with another agent or a human, including where another agent tricks it. Understand that appropriate testing and reasonable care may not absolve the developer in court, and that they may support a better defense.