Can AI Decide Like a Pro? The Live Experiment That Puts Frontier Models to the Test

Imagine an interior designer choosing the perfect furniture—careful, honest, or maybe daringly bold. Now, picture AI models making management decisions for a real company, under pressure, with real money on the line. The question isn’t just whether AI can write well, but whether it can finish what it starts, stay honest, and make smart choices when it matters most. Welcome to a live experiment where cutting-edge AI models run a small software company through its toughest week — and the results may reshape how you think about AI in business.

The Live AI Business Wargame: More Than Just Chat

At Firmulate, we’ve created a unique experiment: four frontier AI models are managing the daily chaos of a real, money-losing software company. Every decision they make is real, every crisis is authentic, and every step is recorded for transparency. This isn’t a demo or a chatbot — it’s an operational simulation where AI models are tested on their ability to handle complex management tasks under stress.

What’s Being Tested?

• Can AI identify critical but hidden information that impacts decisions?
• Will models stick to ethical principles and refuse manipulation attempts?
• Can they close deals and make profitable decisions under pressure?

The Metrics and Results

The experiment measures success based on several key scores, from 0 to 100, with the highest indicating top performance. Notably, the top model scored 95, while the worst scored 77. All models successfully identified crises and refused manipulative tactics, such as fake CEO messages or staged reporter tricks. But only two models managed to close a €55,000 deal their own analysis had earned them, proving they could turn insight into profitable action.

The Hidden Factor: Reading the Company Files

While all models detected issues in customer interactions, the real breakthrough was in reading the company’s internal documents. The decisive advantage belonged to the model that uncovered a buried fact two document references deep in the company’s files. This insight led to the full deal closure and an extra €4,583 in monthly recurring revenue.

Behavioral Profiles of the Models

The models displayed distinct personalities. For example, Opus 4.8 was the most thorough, analyzing over 80 rules and providing deep insights, but it ultimately left the deal on the table due to discipline lapses, like writing attempts into a locked department instead of escalating. Meanwhile, Kimi K3, a newcomer, ran at a default setting and exhibited the cleanest discipline, closing the deal with a straightforward approach. Sonnet models fell somewhere in between, with some slips but ultimately successful.

How Does This Matter for You?

If AI is to become part of your business—whether managing customer relationships, supporting operations, or forecasting—it’s vital to look beyond chat quality. Can your AI finish what it starts? Will it read and understand your internal files? Will it stay honest under pressure? These are the questions real companies need answered, not just whether an AI can craft a convincing message.

Try It Yourself

Interested in how your own organization’s AI might perform? You can run the same wargame against a read-only export of your own data through our platform. No risk, no impact—just insights. Visit firmulate.com/pilot.html to learn more or contact us at contact@firmulate.com.

In a live AI management test, models proved capable of recognizing crises, refusing manipulation, and closing real deals—highlighting the importance of reading comprehension and ethical discipline in AI decision-making. The key takeaway: AI’s true value lies in its ability to finish what it starts and stay honest under pressure, not just in generating convincing chat.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI