Photo By: Federico Beccari
Two decades of digital growth have been built around a simple idea: if a company has a business hypothesis, test it in the real world.
E-commerce companies have tested checkout pages. Media companies have tested subscription offers. Consumer brands have tested prices, promotions and advertising campaigns. A/B testing became a standard way to make product and growth decisions: come up with an idea, split customers into groups, measure what happens and keep the option that performs better.
That approach worked when experiments were relatively cheap and companies had a manageable number of decisions to make.
That equation is changing.
Modern businesses now face an enormous number of choices, from dynamic pricing and regional promotions to advertising budgets, product bundles and customer targeting. Generative AI has made it dramatically cheaper to create new ideas and variations.
The result is a new problem: companies can generate potential strategies faster than they can test them.
That is creating what could be called an experiment tax.
A/B testing is not going away. Asking real customers to answer every strategic question, however, is becoming increasingly difficult to scale.
The problem isn’t that A/B tests don’t work
Randomized trials remain one of the most reliable ways to understand real customer behavior. The issue is not that A/B tests produce bad information. Companies are simply asking them to answer an ever-growing number of questions.
A retailer might test several price points. A marketing team could compare advertising allocations. A regional promotion might require weeks of customer traffic simply to determine whether it deserves further investment.
Every test consumes resources. Engineers have to build it, teams have to wait for enough data, and customers may be exposed to an option that ultimately performs worse.
A promotion could increase sales while cutting profit margins. The company has then paid real money to discover that the strategy was not worth pursuing.
AI makes the problem bigger
Generative AI has made creating business ideas remarkably inexpensive.
An automated system can produce dozens of campaigns, hundreds of promotional structures or thousands of customer-targeting combinations. Companies still have limited customer traffic, engineering resources and time to test them.
AI can generate hypotheses faster than companies can validate them.
The challenge in growth may therefore shift from generating ideas to deciding which ones deserve real-world testing.
Simulation before experimentation
Research from Amazon Science offers one indication of where this could be heading. Researchers examined 67 historical marketing A/B tests using a framework called a Simulated Randomized Controlled Trial to determine whether AI-agent simulations could reproduce effects observed among real customers.
An off-the-shelf foundation model tended to overestimate experimental effects. When the simulation was calibrated using behavioral data from before the experiment, its squared prediction error fell by roughly 77 times.
The finding suggests that simulations become more useful when grounded in actual customer behavior.
That points to a different workflow: generate ideas, simulate them, prioritize the strongest candidates and then test those candidates with real customers.
Simulation for exploration, live testing for validation
A useful simulation requires more than asking an AI model to pretend to be a customer.
Business decisions often depend on relationships between multiple variables and questions about what would happen under conditions that have never occurred. A retailer considering a price increase needs to know whether the higher price would reduce demand enough to offset the additional revenue. An advertiser needs to know whether additional spending will generate new sales or simply reach customers who would have purchased anyway.
Those questions can require causal inference, price-elasticity models, forecasting and simulations that account for uncertainty.
AI agents can help collect and interpret market information, while specialized quantitative models evaluate the underlying business decision.
Kapnova, co-founded by CEO James Sun and CTO Shenbo Xu, is building enterprise software around this approach, combining autonomous agents with econometric and Monte Carlo models to evaluate potential decisions before companies commit capital.
A new role for A/B testing
A simulation-first approach does not eliminate uncertainty. Models can still perform poorly when market conditions change, consumer preferences shift or competitors introduce something unexpected.
Live experimentation therefore remains important. Its role becomes less about filtering every possible idea and more about validating decisions that have already survived an initial round of analysis.
The broader shift is straightforward: AI can generate opportunities, simulation can narrow the field, quantitative models can assess the trade-offs, and live experiments can validate the strongest choices.
That could change how companies measure experimentation itself. The goal may shift from running more tests to making better decisions about which tests deserve scarce customer traffic, time and capital.
The value of a growth team may increasingly depend not on how many experiments it can launch, but on how effectively it can separate promising opportunities from noise.
The A/B test is not becoming obsolete. It may simply be becoming too valuable to use on every question.