How to Audit AI in Your Business: The B.R.A.I.N.S.™ Guide

An AI audit is a structured review of how an AI system is configured, what happens when it runs and whether its behavior produces acceptable business outcomes.

There are two layers of Audit.

Execution-level auditing records the behavior of an AI system one run at a time.

This is where Audit becomes operational.

If an AI system analyzed a sales meeting at 2:41 PM on Tuesday, you should be able to find that execution and understand what happened.

If it updated a deal incorrectly, you should be able to trace the action back to the AI run that caused it.

If costs suddenly increase, you should be able to determine which agent, process, model or workflow is responsible.

Think of execution-level auditing as an AI execution ledger.

Each meaningful AI run creates a record.

That record is the receipt.

Aggregate provider dashboards are a starting point.

Knowing:

We used 14 million AI tokens this month.

still leaves the question:

What business work did those tokens perform?

You want to be able to move from aggregate usage to operational traceability.

For example:

We ran our Customer Risk Agent 3,842 times this month.
97.4% completed successfully.
4.6% required human correction.
The average run cost $0.08.
One particular SOP generated 61% of all corrections.

That is actionable.

It requires you to track executions individually.

The first requirement is simple:

Every meaningful AI execution should have a unique identifier.

For example:

ai_exec_483f20f9

That identifier should follow the run across your systems.

Imagine this flow:

An AI execution ledger is the centralized record of what your AI systems are doing.

A ledger can stay lean. It needs enough information to answer:

What ran?

Why did it run?

What did it use?

What did it do?

What did it cost?

And did it work?

We use a custom AI Executions object in HubSpot for this purpose.

That lets us make AI activity visible inside the same operational system where much of the underlying business work already lives.

The object currently contains more than 80 properties because it supports multiple agent architectures, n8n workflows, OpenClaw sessions, HubSpot workflows, human review and historical migration.

A business recreating the model can start with a subset of those properties.

The important part is the underlying structure.

Our HubSpot AI Executions object also has its own pipeline:

AI Agent Tasks

with stages:

A useful way to think about AI risk is:

Observe → Recommend → Prepare → Execute

One feature of our AI Executions schema that I think is particularly important is that we distinguish between:

what the agent planned to do

and:

what actually happened.

That is why we track both:

plan_summary

and:

outcome_summary

along with:

plan_outcome

Plan → Approve → Execute

For any execution with real consequences, our agents work in three steps. The agent writes a plan. A person approves it. Then the agent executes.

The approval happens where people already work. Our agents post the plan in Slack, and the deal owner or reviewer replies in the thread to approve it, change it, or turn it down. Microsoft Teams, Google Chat or a HubSpot task can play the same role. What matters is that the approval becomes a recorded event with a name and a timestamp, stored next to the plan it refers to.

That is why plan_outcome has three values:

• Completed: the plan was approved and carried out.
• Failed: the plan was approved, and the execution broke.
• Rejected: a person read the plan and said no.

A rejection is a finding in its own right. A rising rejection rate tells you the SOP, the prompt or the model is producing plans people distrust, long before anything goes wrong in the CRM.

An agent can also make a perfectly reasonable plan, get it approved, and still fail operationally.

For example:

Plan: Create a follow-up task for the deal owner.

But perhaps:

• The HubSpot API failed
• The owner was missing
• Authentication expired
• The task was duplicated
• The workflow stopped midway through execution

If you only save the AI's reasoning or final response, you may never know the operational action failed.

One caveat. The outcome summary is written from what the agent could observe. A failed API call or an expired token shows up because the agent sees the error. A duplicated task only shows up if the agent checks its own work, so build that check into the SOP.

Audit both.

As AI becomes agentic, the model response is only one part of the execution.

An agent may:

1. Read a HubSpot record.
2. Search a knowledge base.
3. Query another API.
4. Call another agent.
5. Update a CRM property.
6. Create a task.
7. Write a document.
8. Send a Slack notification.
9. Schedule another workflow.

The natural-language response may say:

Done.

An audit trail needs the actual tool activity.

For important tool calls, consider capturing:

• Tool name
• Action
• Input
• Result
• Timestamp
• Status
• Error
• Record accessed
• Record changed
• Authorization used
• Approval requirement

The question that matters most:

What did the AI do?

Technical logs naturally focus on things like:

• Model
• Request
• Tokens
• Latency
• Error code

Those are useful, and businesses also need to understand why the AI ran.

That is why our schema includes fields such as:

jbtd

agent_role

sop_name

and:

sops_executed

These make it possible to analyze AI activity by business process.

Provider logs say:

GPT-5 consumed 4 million tokens.

Your ledger can eventually say:

Customer onboarding consumed 4 million tokens.

or:

Sales call classification is costing $0.11 per successful call.

That is a much more useful level of analysis.

Every execution should identify the model that ran it.

Our schema stores it in one field, ai_model_used, as free text. Store the exact identifier the provider returns, including the version or snapshot date. A marketing name such as "GPT-5" is too coarse to diagnose anything.

Model identity matters because AI behavior changes.

A prompt may perform well on one model and poorly on another.

Model providers also retire and replace models.

OpenAI explicitly notes that prompting behavior can vary between model snapshots and recommends pinned model versions and evaluations where consistency matters.

If an agent suddenly behaves differently, you want to know whether:

• The prompt changed
• The workflow changed
• The data changed
• The model changed

Without version tracking, those changes become very difficult to diagnose.

The current Mind & Metrics schema has extensive workflow version tracking, including:

• workflow_version_id
• workflow_version_counter
• workflowupdatedat
• workflowcreatedat

For a generalized AI execution schema, I would extend the same idea to explicitly include:

Prompt Version

That gives you:

Agent version

Prompt version

Workflow version

Model version

Tool version

Pick the ones that fit each system, with enough version information to identify what changed when AI performance shifts.

Token usage should be recorded at the individual-run level.

Our HubSpot AI Executions object currently tracks:

tokens_input

tokens_output

tokens_total

This gives us a provider-neutral base.

OpenAI, Gemini and Claude expose more detailed usage data, so you may choose to extend the object with additional fields such as:

• Cached Input Tokens
• Reasoning / Thinking Tokens
• Tool-Use Tokens

Those sit outside our core properties today, and they can provide useful detail as AI architectures become more complex.

OpenAI's Responses API returns structured usage information including:

• input_tokens
• input_tokens_details.cached_tokens
• output_tokens
• output_tokens_details.reasoning_tokens
• total_tokens

A straightforward mapping into a provider-neutral execution ledger looks like this:

You should also capture request identifiers.

OpenAI returns an x-request-id response header and explicitly recommends logging request IDs in production. OpenAI also lets developers supply their own X-Client-Request-Id, making it possible to send your internal execution ID with the request.

That means your trace can look like:

Google Gemini returns detailed usageMetadata with generation responses.

Current fields include:

• promptTokenCount
• cachedContentTokenCount
• candidatesTokenCount
• toolUsePromptTokenCount
• thoughtsTokenCount
• totalTokenCount

A normalized mapping could be:

Google also supports counting tokens before generation, which can be useful when evaluating context size or preventing unexpectedly large requests.

This becomes especially important with Gemini because tool use, multimodal inputs and large context windows can cause total execution usage to be much larger than the visible prompt.

Anthropic exposes model usage information including:

• Input tokens
• Output tokens
• Cache-read input tokens
• Cache-creation input tokens

and provider tool usage can also appear in usage metadata.

A normalized mapping can therefore look like:

Anthropic also returns a unique request-id for API responses, and its SDKs expose that identifier for traceability and support.

Again, connect that provider identifier to your own internal execution ID.

Claude Code deserves special attention because it creates a different governance pattern from centrally hosted business automations.

This is often described casually as:

Claude runs locally.

That description is incomplete.

Claude Code itself runs in the user's local working environment, and the Claude model is generally still accessed through Anthropic or another supported provider.

The important audit difference is where the execution environment lives.

Claude Code can work inside an employee's local development environment and interact with things such as:

• Source code
• Local files
• Shell commands
• Repositories
• Development tools
• MCP servers
• APIs
• Connected systems

Claude Code is explicitly designed as a command-line development tool and supports MCP and other integrations.

That means a company may have less centralized operational visibility than it would with an AI agent running through a centrally managed workflow platform.

That is an architectural tradeoff that needs to be considered.

Imagine ten developers using Claude Code independently on ten laptops.

Each developer may be productive.

But what can the organization answer?

• Who is using Claude?
• Which projects are using it?
• Which repositories can it access?
• Which models are being used?
• Which MCP servers are configured?
• Which tools can it call?
• What actions has it performed?
• How much does each project cost?
• Can security investigate an execution afterward?
• Can usage be attributed to a team?
• Can permissions be centrally changed?
• Can access be revoked?

This is fundamentally different from a centrally orchestrated agent where every execution naturally passes through one controlled environment.

The lesson:

Local execution changes where your controls and evidence need to live.

Local tooling can still be observable. How much the business sees depends on two decisions: which plan the seat runs on, and whether you route and export the traffic yourself.

The plan decides what Anthropic shows you

Claude Code is included in the Pro, Max, Team and Enterprise plans, and it also runs on an API key through the Claude Console, Amazon Bedrock or Google Vertex AI. Each of those gives the company a different view.

• Pro and Max (individual plans): these are consumer accounts. The company gets no dashboard, no usage export and no audit log. The only record of a session is the transcript file on the employee's laptop, kept for 30 days by default. Consumer accounts can also be used for model training unless the user turns that setting off. For a business, this is the blind spot: an employee on a personal Max plan can run agents against company systems and the company holds no record of it. Anthropic itself keeps consumer session data for 30 days, or five years when the user allows training, but that copy lives under the user's personal account. The employer can neither see it nor export it, so for audit purposes it does not exist.
• Team and Enterprise: commercial terms, so your prompts and code are excluded from training. Admins get a Claude Code analytics dashboard with daily active users, sessions, accepted lines of code and a per-user spend report. Enterprise adds audit logs and an Analytics API.
• API key through the Claude Console: usage and spend appear per user in the Console dashboard and through the Claude Code Analytics API. On Bedrock or Vertex AI, usage lands in your cloud provider's own billing and logging instead.

Two controls work on every plan

1. Telemetry export. Claude Code speaks OpenTelemetry. One environment variable turns on export of metrics (sessions, tokens, cost, lines changed) and events (each prompt, response, API call, tool call and permission decision) to a collector you run. An administrator can force those settings through the managed settings file, and Claude Code discards conflicting developer settings at startup. This is how a Max seat becomes observable: the company manages the machine, so the company owns the log.

2. An LLM gateway. Anthropic documents routing Claude Code through a gateway, either its own self-hosted Claude apps gateway or a third-party product, to centralize:

• Credentials
• Usage tracking
• Cost controls
• Audit logging
• Provider switching

Conceptually:

OpenAI, Google, Anthropic, HubSpot and your orchestration platforms all provide useful logs and usage data.

Use them.

None of them sees the complete business process.

OpenAI may know:

Request req_abc consumed 3,500 tokens.

Gemini may know:

Generation X used 900 thinking tokens.

Anthropic may know:

Request Y used 5,000 cached tokens.

n8n may know:

Workflow execution 82915 completed.

HubSpot may know:

Deal 12345 changed stage.

Each system logs its own slice, in its own format, under its own ID, inside its own login. Nobody joins them. The token count sits in the OpenAI dashboard, the workflow status sits in n8n, the record change sits in HubSpot's property history, and the reason the run happened sits in no system at all.

Why that matters

• When something goes wrong, someone reconstructs the story by hand across four logins.
• When leadership asks what the AI spend bought, the answer comes back in tokens. The question was asked in deals, tickets and hours.
• When a vendor shortens its retention window, part of your history disappears with it.
• When a model is retired, you cannot tell which runs it affected.

Your business needs to know:

The Sales Follow-Up Agent ran SOP v4 against Deal 12345, used model X, cost $0.09, called three tools, updated two records, required no human intervention and produced the intended outcome.

That requires your own audit layer.

What an audit layer is

One record per execution, in a system you own, that holds three kinds of information:

• Business context: which agent ran, which SOP, what triggered it, and which record it acted on.
• Technical facts copied from the vendors: model, tokens, cost, the provider request ID and the workflow ID.
• Outcome: workflow status, plan outcome, human review, corrections and time saved.

The vendor logs stay where they are. Your record carries their IDs, so you can drill down to the raw detail when you need it. The audit layer is the summary you can report on, join to business results, and keep for as long as the business decides.

In our case the audit layer is a custom object in HubSpot, sitting next to the deals and contacts the agents act on. A single report can join AI activity to pipeline results with no export in between.

Your schema should record why the AI ran.

Our execution_source field currently supports values including:

• AskElephant Webhook
• HubSpot Webhook
• Other Webhook
• Cron
• Slack Channel
• Slack DM
• Telegram
• Inter-Agent
• Direct Message
• Legacy
• Other

We also track execution platform separately, with values currently including:

• OpenClaw
• n8n
• Manual

The exact options will differ by business.

The principle is what matters:

Trigger and execution environment are two separate things.

A HubSpot workflow may trigger an n8n workflow that calls OpenAI.

Knowing only "OpenAI" tells you very little about why the business activity occurred.

Our AI Executions object currently uses these workflow statuses:

• Success
• Error
• Waiting
• Unknown

We separately track plan outcomes:

• Completed
• Failed
• Rejected

That separation is useful because the two fields answer different questions.

Workflow status answers: did the machinery run? It comes from the orchestrator. n8n, OpenClaw or a HubSpot workflow reports whether every step executed without an error.

Plan outcome answers: did the business get what it wanted? It comes from the plan, the approval and the result.

They diverge in both directions.

Technical success, business failure

The workflow runs end to end. The agent drafts a follow-up plan, posts it to Slack, and the deal owner rejects it because the deal closed yesterday. n8n logs a success. The business outcome is rejected.

Or every step succeeds and the agent updates the wrong deal, because two records shared a company name. Success in the orchestrator. Failure for the business.

Business success, technical failure

The model reads the transcript, correctly decides the deal should move to Proposal Sent, and the HubSpot API call times out. The plan was right. Nothing changed in the CRM. Reading the model's answer alone, the run looks complete.

If you only track workflow status, the first case counts as a win and the second as a loss. Both readings are wrong.

Why this matters beyond the engineering team

Counting successful business outcomes is what lets you put a number on the initiative that leadership recognizes. Efficiency figures such as "the agent runs in 40 seconds" or "we cut tokens per run by 30%" land poorly with executives who manage to revenue and headcount.

A count of completed outcomes, multiplied by the time a person would have spent on each, becomes hours, and hours become salary.

This agent completed 3,100 follow-ups this quarter, which is the workload of one full-time coordinator.

That is a sentence a CFO can act on. You can only say it when the ledger tells you which runs produced the intended outcome, and which merely ran without an error.

Technical success and business success can diverge.

Audit both.

Our execution object has three fields specifically designed to feed Audit into Improve:

execution_grade

execution_grade_notes

human_feedback_notes

The grade is intentionally simple:

Good

or:

Bad

Why?

Because a simple verdict is often enough.

Sometimes the most useful signal is:

Did this work?

A bad grade can then include:

• What went wrong
• What should have happened
• What the reviewer changed

Those corrections become extremely valuable during Improve.

If humans repeatedly change the same type of output, that is evidence.

Perhaps:

• The prompt is unclear
• An SOP is incomplete
• The knowledge source is wrong
• One edge case isn't covered
• The model is unsuitable
• A tool is missing
• The business rule itself is ambiguous

Human review tends to disappear into Slack conversations or silent manual edits.

Capture it.

That is why execution feedback belongs in the audit ledger.

Audit covers more than cost.

AI is often deployed to create business efficiency.

Our schema currently includes:

• time_saved
• estimated_time_saved
• calculated_time_saved

This gives us a way to ask:

What did we spend?

and:

What did we gain?

Those figures are estimates, and even reasonable estimates can help identify which AI processes create meaningful operational gains and which create novelty.

Total AI spend is a weak performance metric.

Even cost per execution can be misleading.

A better question is:

What does a successful business outcome cost?

Consider:

When an AI response is wrong, start by asking what the model received.

Was:

• The CRM record correct?
• The information current?
• The right document retrieved?
• Important context missing?
• Too much irrelevant context included?
• The wrong conversation history included?
• A tool result inaccurate?
• Conflicting instructions present?
• A required field empty?

A bad result caused by stale CRM data is a data problem.

A hallucination caused by retrieving the wrong knowledge document is a retrieval problem.

Audit gives you the evidence to diagnose the right layer.

Different AI activities require different definitions of quality.

You might evaluate:

• Accuracy
• Completeness
• Relevance
• Classification correctness
• Business-rule compliance
• Citation correctness
• Format compliance
• Tone
• Hallucination rate
• Recommendation quality
• Escalation accuracy

A single generic "AI quality score" hides the differences between processes.

A good customer email and a good lead classification are different things.

Once AI takes actions, output evaluation alone becomes insufficient.

Suppose the agent correctly concludes:

Create a follow-up task.

Now Audit needs to answer:

• Was the task created?
• Was it created once?
• Was it attached to the correct record?
• Was the correct owner assigned?
• Was the due date correct?
• Were the task instructions correct?

A model can be right while the automation fails.

And an automation can succeed technically while performing the wrong business action.

Your execution record should let you distinguish those situations.

Failures are some of the most useful data your AI system produces.

Categorize them.

The goal should increasingly become:

Capture everything. Review exceptions.

For example, alert when:

• Failure rate exceeds a threshold
• Human correction rate increases
• Input tokens suddenly spike
• Cost per run changes materially
• An agent executes multiple times against the same record
• A high-risk property is changed
• A prohibited tool is attempted
• Runtime changes dramatically
• Output is empty
• Confidence is below an acceptable level
• A required source was unavailable

This creates scalable oversight.

Different review cadences answer different questions.

Once run-level execution data exists, reporting becomes much more valuable.

Useful dashboard categories include:

Audit should be proportional to consequence.

A receipt for every execution can be far leaner than storing every prompt and every response forever.

Audit logs can themselves create privacy and security risk.

Consider:

• Customer data
• Employee information
• Personally identifiable information
• Credentials
• Financial information
• Health information
• Confidential commercial data

For some workflows, storing the complete transcript is appropriate.

For others you may store:

• Structured metadata
• A redacted transcript
• A summary
• A hash
• A secure external reference

Audit requires traceability.

Traceability rarely requires duplicating all the data.

A production AI process might look like:

We've published the structure of the AI Execution object we currently use inside Mind & Metrics so businesses using HubSpot can recreate the same general architecture.

The downloadable workbook includes:

• Our complete property schema
• Core Audit properties
• Enumeration options
• Pipeline and stages
• Suggested provider mappings
• OpenAI mappings
• Gemini mappings
• Claude mappings
• Recommended schema extensions
• Recreation checklist

You can adapt the fields to your own setup.

The objective is to create a record that answers the fundamental audit questions:

What ran?

Why did it run?

What did it use?

What did it do?

What did it cost?

And did it work?

This is why Audit sits directly before Improve in B.R.A.I.N.S.™.

During Build, you make assumptions.

During Run, the AI starts doing real work.

During Audit, you collect evidence.

Only then can you meaningfully Improve.

You may discover:

• A shorter prompt performs equally well
• Another prompt structure produces better results
• A cheaper model performs the task reliably
• A more expensive model reduces human corrections enough to justify the cost
• Retrieval is sending thousands of unnecessary tokens
• One tool causes most failures
• Human reviewers repeatedly correct the same output
• A business rule needs clarification
• A deterministic workflow should replace an AI decision
• Autonomy can safely increase
• Autonomy should decrease
• A process should not use AI at all

Those are evidence-based decisions.

Without Audit, they are guesses.

An AI process may work beautifully because one person understands all of its quirks.

A standard business process has to work without that person.

Before you Normalize, Audit should establish:

• Reliable performance
• Known failure modes
• Acceptable cost
• Appropriate oversight
• Defined ownership
• Stable configuration
• Clear business rules
• Repeatable outcomes

Only then should the process become the accepted way work gets done.

Problems that happen 1% of the time are easy to ignore when an AI workflow runs ten times.

At 100,000 executions, that is:

1,000 failures.

Scaling multiplies success.

It also multiplies failure.

Before you Scale, you need to understand:

• How often it fails
• Why it fails
• Whether you can detect failure
• Whether failure is reversible
• What failure costs
• Whether humans need to intervene
• Whether the system remains economically worthwhile at volume

That is what Audit makes visible.

Before you consider an AI system meaningfully auditable, ask:

Good auditing gives businesses enough evidence to decide confidently:

where AI can operate autonomously,

where a human should remain involved,

where controls are necessary,

where the system is underperforming,

and:

where autonomy can safely increase.

The more mature the audit layer becomes, the more oversight can be driven by exceptions.

AI increasingly sits between business systems.

It reads the meeting.

Interprets the conversation.

Queries the CRM.

Chooses a process.

Calls a tool.

Makes a recommendation.

Changes the record.

Creates a task.

Writes the email.

Triggers the next system.

That can create extraordinary efficiency.

It can also create an operational black box if those actions disappear into disconnected platforms, laptops, logs and vendor dashboards.

Audit prevents that.

It creates a record of:

what ran,

why it ran,

what it used,

what it planned,

what it did,

what it cost,

whether a human intervened,

and:

whether it worked.

That evidence is what makes the rest of B.R.A.I.N.S.™ possible.

Build. Run. Audit. Improve. Normalize. Scale.

Because before you improve AI, standardize it or scale it across the business, you need to be able to see what it is actually doing.