10 Ways to Draft AI Warranties and Acceptance Testing Provisions: Practice Points

Start by naming the job and asking whether that job had a right answer. Laura Belmont opened by zooming out, because AI in a contract meant a lot of different things. A chatbot answering any question a user typed was not the same purchase as a narrow tool pulling specific terms out of hundreds of invoices, and the two could not warrant the same way. So she asked two questions before drafting anything. What is the job this tool does, and does that job have a right answer. Everything you can reasonably ask for downstream follows from those answers.

Split the ordinary software out from the AI part. Laura Belmont pushed back hard on the idea that a product becomes unwarrantable the moment AI goes into it. Most AI products carry a lot of ordinary software around the model, including access controls, uptime, citation behavior, audit logs, and approval steps, and none of that is probabilistic. Vendors building on a frontier model are selling that whole package. Separate the ordinary components and hold your normal expectations for them.

Keep conformance and accuracy as two different promises. Conformance warrants that the product does what the documentation says it does. Accuracy measures an output against a metric you defined. Matt and Laura Belmont both treated these as distinct, and they added a third bucket for reliability that essentially nobody was granting. A narrow, grounded tool could support both conformance and accuracy, and on a broad general purpose tool conformance was often the most you could get.

Pin down the metric, the dataset, and the threshold. Laura Belmont was direct that saying something is materially accurate does not mean anything you can measure. You need to know what metric gets tested, what data it gets tested against, and what number separates a pass from a failure. Ask whether you can test against your own customer data rather than only the vendor's. Without those three pieces you cannot identify a breach or trigger a cure.

Treat materially and substantially as words that need definitions. Matt flagged a warranty promising performance materially in accordance with documentation and outputs that were substantially accurate. Two different vague terms sitting in one provision invited an argument that they meant two different things, and neither one told either party what performance looked like. Laura Belmont made the same point from the customer chair, with an example where a tool cited its sources every time and still produced problematic answers most of the time. Decide what you actually need to be accurate and push on that specific thing.

Scope the warranty to a defined set of outputs and name who runs the test. Matt called a warranty covering all outputs a trap, because one bad output could support a breach claim. Hallucinations were not a bug to be engineered away, so he wanted the promise measured over a defined set and a defined period rather than output by output. He also wanted the benchmark testing spelled out, including the tester, the methodology, the dataset, the standard, and the window. Customers benefit from that specificity too, because vague testing language is exactly where two parties reach different results and cannot resolve them.

Read the exclusions for prompts, inputs, and configurations very carefully. The sample warranty died if the customer modified prompts, inputs, or configurations. Laura Belmont agreed a vendor should not answer for a customer rewriting code it never approved, and she drew the line at customization the tool was built for. Many of these tools are sold precisely so you can tune them to your work. Matt added that carve-outs for custom configuration were showing up on indemnity as well, which left two traps in one agreement.

Give the warranty a length and a cure that survive how much the product changes. Datasets change, models drift, and a tool trained on the world as it was will not recognize what happened since. Accepting that the product worked on the day it arrived tells you nothing about the rest of your subscription, so Laura Belmont wanted a real warranty length. She also wanted a named consequence, because a warranty without one does no work. A promise to correct at the next scheduled release means very little when nobody knows when that release lands.

Build acceptance testing that actually ends. Matt's goal was avoiding an infinite testing loop, and he got there through specificity about the tests, the datasets, the metrics, the thresholds, and the tester. He thought testing starting upon delivery was often unreasonable, since implementations take time and customers need to connect their data pipelines. On something this sophisticated he wanted written sign-off rather than silence, feature by feature where the build had distinct parts. From the other side, Laura Belmont rejected any commitment to test every deliverable, wanted sample sets instead, and wanted retesting triggered by a model change or a version rollout.

Narrow the disclaimer so it does not cancel what you just negotiated. Matt pointed out that a vendor warranting accuracy in one section and disclaiming accuracy in another leaves its own obligations unclear, and that a broad non-infringement disclaimer can collide with the IP indemnity customers fought to get. Laura Belmont said she looks to strike as-is language in a paid enterprise agreement, because a disclaimer should not take back what the product was sold to do. Laura Frederick spent a career watching disclaimers quietly gut the covenants and warranties above them, and she noted that all caps and a UCC reference make customers feel they cannot push. The standard fix says that except for the warranties set forth in the warranty section, there are no other warranties.