This checklist reviews AI vendor audit rights and compliance verification, including notice period and audit frequency, scope of the compliance audit, business hours and confidentiality, access to records and systems, and allocation of audit costs. Sample contract provisions are provided as an example and are not intended as model language.
Here is the link to download the PDF version of this checklist.
A. Notice Period and Audit Frequency
“Customer shall have the right, upon thirty (30) days’ prior written notice and no more than once per calendar year,”
Expedited Audit After an Incident. Add an expedited audit right on five business days notice for a material service failure, a data breach, or a credible indication of noncompliance. Keep that audit outside the once per year limit. The notice period and the annual cap otherwise stop the customer from responding to a suspected performance or data integrity issue.
Routine and Performance Audits. Write two audit provisions. Keep the routine audit at once per calendar year on notice, and tie a performance based audit to defined events.
Model Update Notice. Require the vendor to notify the customer of a model update within a set number of business days. Tie that notice to the right to conduct a performance based audit. Models may be updated frequently, and data pipelines may change with them.
Output Problems at Other Customers. Require the vendor to report output problems that other customers raise. Degradation may come from a general issue with the model rather than from the customer’s own use. Tie that report to a performance based audit.
Audit Rights Blocked in Practice. Check whether the customer can exercise the audit right when it needs to. Advance notice requirements, vendor-controlled scope limitations, and restrictions on third-party auditors leave the customer little practical recourse during an incident.
B. Scope of the Compliance Audit
“to audit Vendor’s compliance with its obligations under this Agreement, including its data security and data protection obligations.”
Obligations Under the Agreement. Name the obligations the audit covers and state how the parties measure compliance with them. The phrase points to no representation, warranty, or specific provision in the contract.
AI-Specific Records in Scope. Specify the records within the audit scope, including system availability logs, recovery time documentation, data backup and restoration records, model version history, data lineage records, and logs of vendor personnel access to customer data.
Discriminatory Output in High-Risk Uses. Audit for discriminatory output where the model does high risk work such as resume review or ranking. A compliance obligation of that kind does not appear in a traditional software audit provision.
The Vendor’s Obligations to Others. As a vendor, separate the data protection obligations owed to this customer from those owed to third parties, to the vendor’s own information, and to data the vendor licensed to train its systems.
Undefined Audit Terms. Define the terms used in the audit provision even where the business pushes for plain language. AI contracting has no accepted practices yet, which makes undefined terms on new concepts doubly problematic.
C. Business Hours and Confidentiality
“Any such audit shall be conducted during normal business hours in a manner that does not unreasonably disrupt Vendor’s operations, and shall be subject to Vendor’s reasonable confidentiality requirements.”
Onsite Access to the Model. Assume the customer will not get onsite access to the model. This sentence supports an argument for it, and a vendor is unlikely to make that promise. Broad access conflicts with how vendors protect the model and its weights.
Limits From the Upstream Developer. Ask whether the vendor built the model or distributes someone else’s. A distributor or deployer may not be able to grant the audit rights the customer wants because of its own limits with the upstream developer.
Confidential Material in Audit Logs. Expect confidentiality obligations to limit what an audit produces. Audit logs and the documents requested for a performance audit may contain other customers’ personal information, the vendor’s intellectual property, and the upstream developer’s intellectual property.
No Uniform Standard for Reasonable. Spell out what reasonable means in this provision, or attach metrics or technical design details. No uniform definition of reasonable or commercially reasonable applies in the AI context, whatever the case law says about other industries.
D. Access to Records and Systems
“Vendor shall provide Customer or its designated third-party auditor with reasonable access to relevant records and systems as necessary to verify Vendor’s compliance.”
Relevant Records. List the categories of records the vendor must produce. Relevant records tells the customer nothing about what kinds of data it will get, and leaves the parties with conflicting expectations.
Period Covered by the Records. State how far back the records must go. An audit is a snapshot in time, and AI model performance can degrade over time. Decide whether a bias audit covers one year, two years, or the full term.
Systems the Vendor Can Access. Confirm which systems the vendor can access. The vendor may face operational and technological limits on the systems the customer considers relevant and necessary.
Resources to Read the Output. Confirm the customer has the subject matter expertise to use what an audit produces. Logs, reports, and processing reports have little value to a customer that cannot read them. Pushing for data the customer does not know what to do with is a common negotiation mistake.
Time an Audit Takes. Build the audit timeline into the provision. An audit takes significant time, a battery of tests, and subject matter expert review. The customer’s data environment may change during that period, which affects what the vendor can show.
E. Allocation of Audit Costs
“Customer shall bear all costs of any such audit.”
Cost After a Material Breach. Require the vendor to reimburse the reasonable costs of an audit that reveals a material breach of its obligations. A flat customer-pays allocation creates a disincentive to exercise the right at all.
Cost of a Performance Audit. Where the customer accepts the cost of the routine audit, state that the vendor bears all or a significant portion of the cost of a performance based audit.
Scope Control Over Audit Cost. Define the audit scope before the vendor sets it. A vendor that picks the work can spend heavily on diligence useful to itself and charge the customer for it.
Remedy After a Failed Audit. Write the remedy the customer gets when an audit finds a problem. Tie it to termination, to a period for the vendor to cure, or to a reasonable alternative such as a different model that does not interrupt the customer’s business.
