An AI tool that drafts a customer reply and an AI system that sends it, changes a booking and takes a payment carry very different responsibilities.
For a small business, that difference matters more than whether the software uses the newest model. A mistake in a draft is usually easy to correct. A mistake that reaches a customer or changes a financial record can take considerably more work to put right.
In his September 2026 essay, We Must Pace the Frontier, Anthropic CEO Dario Amodei argues that the companies developing the most advanced AI should slow capability improvements enough for safeguards to keep up.
His proposal concerns frontier AI development, rather than everyday business automation. But it raises a useful question for business owners: are we giving our systems more responsibility faster than we can check their work?
You do not need to stop exploring AI. You need to choose what it is allowed to do, prove that it helps, and know how to take over when it fails.
What Amodei is proposing
Amodei remains optimistic about AI’s potential benefits. His concern is that advances in capability could outpace the work needed to understand, evaluate and control increasingly powerful systems.
He proposes three measures:
- Embedded outside evaluators with ongoing access to AI companies’ safety work.
- Coordination among frontier developers in democratic countries, with government support where needed.
- International agreements where compliance can be meaningfully verified.
The essay announces Anthropic’s commitment to the first step and says an external team will be invited in the near future. That is an announced commitment, not evidence in itself that reviewers are already embedded or that the wider framework is operating.
Amodei also argues that any extra time should be used for specific improvements: operational reliability, safeguards, understanding model behaviour and better testing. Slowing down without doing that work would achieve little.
These are the views and proposals of an AI company leader with a stake in the debate. The article does not establish a new legal requirement for your business or prove that a particular supplier is safe.
Start with the business problem, not a replacement plan
Imagine a small plumbing company that regularly misses enquiries while its team is on jobs.
A useful first system might collect the customer’s contact details, postcode and description of the problem, then prepare a callback request for a member of staff. That is a narrower and more testable job than asking AI to run customer service.
Before choosing software, write down what currently goes wrong. How many enquiries go unanswered? Where are details lost? Who checks the inbox? What counts as urgent?
Sometimes the first improvement is a shared queue and clear ownership. AI may help interpret a message, but it does not repair an undefined process on its own.
Increase responsibility in steps
The following approach is a practical recommendation for small businesses, not a framework prescribed by Amodei.
First, let the system prepare work. It can categorise enquiries, draft responses or suggest next steps. A person checks the result before it affects a customer. Use synthetic or appropriately protected information while testing.
Next, allow a narrow set of low-consequence actions. Once you have evidence, you might permit an acknowledgement that confirms receipt of an enquiry. Keep the wording bounded: it should not promise a price, an arrival time or a guaranteed repair.
Consider broader actions only after testing the controls. Booking appointments, modifying customer records and handling payments require more than convincing conversation. The system needs appropriate permissions, accurate information, safeguards against duplicate actions and a way to confirm what happened.
Keep consequential or ambiguous decisions with a person unless you have deliberately established a safe, authorised process for them. Urgent safety concerns need a clear escalation route, not a chatbot improvising advice.
These steps can be taken quickly for a simple workflow. They should take longer when the possible harm is greater.
Test the awkward situations before customers encounter them
A good demonstration shows the software doing its intended job. A useful pilot also shows what happens when the conditions are wrong.
For the plumbing company, test a customer outside the service area, an unclear address, a duplicate enquiry and a request the business cannot fulfil. Disconnect the calendar. Send an urgent message. Ask the system to ignore its instructions and promise an unapproved discount.
Watch for whether it recognises its limits, preserves the enquiry and alerts someone who can act.
The owner should also know how to stop automated replies without losing incoming messages. A manual fallback can be simple, but someone must own it and know when to use it.
Ask suppliers for evidence you can use
You do not need to inspect the inside of an AI model to ask sensible questions about a proposed system:
- What information will it receive, and where will that information go?
- Which actions can it take without approval?
- How does it handle an unavailable integration or an uncertain answer?
- Can we see a record of what it did and whether the action succeeded?
- What happens when the model or connected software changes?
- Who responds to a failure, and can we continue working manually?
An external audit can be useful evidence, but check its scope. An assessment of the model provider does not automatically cover the workflow, permissions and customer data arrangements in your business.
Ask for the ongoing costs too. Model usage, connected software, monitoring, maintenance and time spent reviewing output all affect whether the system is worthwhile.
Measure the result after checking and corrections
A system may produce replies quickly while creating more work elsewhere.
For an enquiry workflow, compare response time, complete contact records, qualified callbacks and missed leads. Also record incorrect promises, duplicates, staff corrections and customer complaints. Include the time people spend reviewing and repairing the output.
Agree what would justify expanding the pilot and what would make you stop. Those criteria should fit your workload and risk; there is no universal number of successful conversations that makes a system safe.
If the pilot saves time but occasionally loses an enquiry, fix that before increasing its responsibility.
Caution has a cost too
Waiting for perfect certainty can leave avoidable problems untouched. A business that misses customer requests every day already has an operational risk. A bounded, well-run automation may improve on the current process even though it is imperfect.
The sensible comparison is between the proposed workflow and the real alternative, including their respective mistakes and costs. It is not between an imperfect AI system and an imaginary business where humans never miss anything.
Amodei’s broader proposals also raise questions about whether coordination can be verified and whether compliance costs could favour large incumbents. Those policy questions deserve scrutiny. They are separate from deciding whether a tightly controlled enquiry assistant would help your business this month.
Pick one frustrating process, establish the current baseline, and test the smallest useful change. Give it more responsibility only when you can explain the benefit, the remaining risks and how your team will recover if it goes wrong.
If you are unsure where to begin, book a free 30-minute fit call with Valdris. Bring the process that keeps getting stuck; we can start there rather than with a list of AI tools.
Source: Dario Amodei, “We Must Pace the Frontier”, September 2026. This article distinguishes his frontier-development proposals from Valdris’s practical recommendations for small-business adoption. The plumbing company is an illustrative example, not a client case study. No performance or savings claims are made for an untested deployment.