Five Questions CEOs Should Ask Before Scaling AI

Key Takeaways

• ‍Five questions should come before any large AI approval: CEOs should ask what outcome they are buying, what one successful outcome costs, how the economics hold at 10x and 100x volume, whether they are paying for intelligence they don’t need, and how they will measure AI ROI. One agent request can trigger dozens of model calls behind the scenes, so CEOs should test the economics at 10x and 100x volume before approving a large AI initiative.
• Cost per successful outcome beats cost per token: A four-cent task that succeeds 70 percent of the time and needs retries, as well as  human review, can cost more than a ten-cent task that succeeds 95 percent of the time. Compare AI options by what it costs to get the task done right, not by the price of the model.
• Match the intelligence to the task: Password resets and invoice sorting don’t need the most powerful model. Smaller models or standard automation can handle routine work, while advanced models are best saved for high-value, high-risk decisions. Paying for top-tier intelligence on routine work adds cost without adding value

As AI moves from experimentation into production, a new line item is finding its way into conversations between technology leaders, CFOs, and CEOs: tokens. How many tokens will this application consume? Which model has the lowest cost per million tokens? Should we reduce the context window? Can we use a smaller model? What happens to inference costs when we scale from a few hundred users to thousands, or from thousands of transactions to millions?

These are important questions, but they can quickly lead down the wrong path. There is a growing tendency to equate greater AI consumption with greater AI capability, where more context, more reasoning, more agents, and ultimately more tokens somehow imply a better solution. This is token maxing, and from a business perspective, token maxing is stupid.

Companies don’t create value simply because they consume ten billion tokens instead of five billion. Customers do not care how many tokens were required to answer their question, and an employee does not care how sophisticated an agent’s reasoning process was if they still have to manually correct its work. A CFO certainly should not celebrate increasing AI consumption unless that consumption can be connected to measurable economic value. Tokens are an input to intelligence, not a measure of the value that intelligence creates.

For a CEO, the more useful way to measure token economics is:

AI Economics = Business Value Created ÷ Total Cost of Producing a Successful Outcome

The denominator is where things become interesting because the true cost of AI is rarely just the model bill. It can include:

Total AI Cost = AI Consumption + Infrastructure + Human Review + Retries + Errors + Operating Cost

Imagine that one approach costs four cents to perform a task while another costs ten cents. Looking purely at AI consumption, the four-cent option appears to be the obvious choice. But suppose it successfully completes the task only 70 percent of the time and frequently requires retries or human intervention, while the ten-cent option completes it correctly 95 percent of the time with very little intervention. The more expensive option may actually produce dramatically better business economics.

Cost per successful outcome matters far more than cost per token.

The same logic works in the opposite direction. There is no reason to use the most powerful and expensive model for every task simply because it is available. Resetting a password, categorizing an invoice, summarizing a standard document, reviewing a complex legal agreement, and recommending a multimillion-dollar business decision do not require the same level of intelligence. Yet it is surprisingly easy to design an AI architecture that treats them as though they do.

 The right architecture should match the level of intelligence to the value and risk of each task. Some activities can be handled by smaller models, deterministic workflows, or traditional software. Others justify more sophisticated models, richer context, deeper reasoning, multiple agents, and human oversight. Good AI architecture makes those trade-offs deliberately.

This becomes even more important as enterprises move toward agentic AI. A chatbot might involve a relatively simple request and response, while an agent can retrieve information, call tools, query several systems, reason over the results, ask another agent for assistance, evaluate its own response, retry failed steps, and eventually return an answer to the user. To the employee, that may look like one interaction. Economically, it could represent dozens of model interactions happening behind the scenes.

That creates a potentially dangerous disconnect between user growth and consumption growth. An AI application might grow from 1,000 to 10,000 users, but token consumption could increase by much more than ten times depending on how the underlying agents behave. A pilot may look inexpensive, but costs can rise dramatically at production scale.

I propose that before approving large AI initiatives, CEOs should ask five questions.

First, what business outcome are we actually buying? Before discussing models or tokens, there should be clarity about whether the initiative is expected to increase revenue, eliminate cost, reduce cycle time, prevent leakage, improve customer experience, reduce risk, or release employee capacity. If the value cannot be articulated, optimizing the tokens is largely irrelevant.

Second, what does one successful outcome actually cost us? That means looking beyond AI consumption and incorporating infrastructure, software, retrieval, tool calls, retries, human validation, exception handling, and ongoing operations. A five-cent AI transaction that triggers three dollars of human intervention is not a five-cent transaction.

Third, what happens to these economics at scale? Every significant AI business case should show what happens at 10× and perhaps 100× the expected volume. The question is not simply whether the technology scales, but whether the economics scale with it. A solution that creates $10 of value while costing $1 is attractive. If its cost grows to $8 as complexity and usage increase, the conversation changes considerably.

Fourth, are we paying for intelligence we don’t need? This is where model selection, model routing, caching, context management, and architecture become business issues rather than simply engineering decisions. If a smaller model can reliably handle 70 percent of transactions and the most capable model is reserved for the difficult 30 percent, the economics may be substantially better without compromising the outcome.

Fifth, can the organization measure something resembling:

AI ROI = (Business Value Created − Total AI Cost) ÷ Total AI Cost

The exact formula will vary by use case, and not every benefit can be perfectly quantified, but the discipline matters. Over time, organizations should be able to demonstrate that AI is producing more value than it costs, rather than pointing to their token usage.

This shift in thinking will become an increasingly important part of operationalizing AI. The first phase of enterprise AI was largely about proving what was possible: Can the model do this? Can we build an agent? Can we automate this process? Those were necessary questions when the technology was new. Those questions were necessary when companies were still testing the technology.

Now AI is moving into workflows, and the usage is much higher. Capability alone is not enough, and leaders need to know whether the results justify the cost, and how that cost changes as usage grows. That changes the questions CEOs need to ask: Should AI perform this task? What level of intelligence does it require? What does a successful outcome cost? What happens when usage increases one hundredfold? Where should we use smaller models or deterministic automation? Where is additional reasoning worth paying for?

Engineering teams absolutely need to understand tokens. They should optimize prompts, context, caching, model selection, routing, and architecture. But CEOs need to look one level higher because optimizing token consumption without understanding business value is simply optimizing the wrong metric.

Ultimately, businesses don’t buy tokens. They buy outcomes.

‍