Stop Using the Best AI Model for Every Job

Dr. Ryan Ries here. I keep getting asked the same question: “Ryan, what’s the best AI model right now?”

To which I say, “Does it matter?”

Because at almost the exact same time, I’ve noticed another weird thing happening with AI inside companies. We’re using way too much of it.

Too much model for simple jobs. Too many tokens. Too many agents. And wayyyyyy too many pages.

Ask an AI tool for a one-page executive summary and for some reason it decides to pull out a 47 page redundant report!

Those might sound like completely different problems. I don’t think they are. This week, let’s talk about why more AI isn’t necessarily better AI.

Stop Using the Best AI Model

There are so many AI models coming out now that even people who work in AI every day have trouble keeping the names straight. I’m still waiting for the day a model comes out called “RyAIn.”

The natural reaction is to look at the latest benchmark and ask which one is the smartest. But that is not how we buy any other kind of compute.

You would not use your largest GPU cluster to run Excel. You would not hire your most expensive engineer to rename 10,000 files. So why are we doing the AI equivalent?

Most of the large AI companies are effectively settling into three tiers:

• A smaller, faster model for straightforward work
• A daily driver that handles most things really well
• A top-tier model for the occasions when you actually need serious reasoning

The mistake is assuming the monster at the top should do everything. It should not.

The better question is: What is the least expensive model that can reliably accomplish this particular job?

Notice I said reliably. Cheap is not useful if I have to do everything twice. But neither is paying frontier-model prices to classify a support ticket.

Amazon Didn’t Need a Smarter Model. It Needed the Right Model.

Amazon had an internal workflow using Claude Sonnet to classify tickets and it worked. But think about the actual job. The model was not being asked to architect an application, research a market, or solve a novel problem. It was picking a label.

Amazon engineers replaced the general-purpose LLM with a specialized decision model that scores potential labels and selects the best one.

The reported modeled economics went from roughly $22,200 to $500 per million tickets. About a 44x cost difference.

The most interesting part is not the savings (though we do love a 44x cost savings!). It’s why they got the savings.

They stopped paying an LLM to write when all they needed was a decision.

I really like this story because it is a great lesson for all companies: use the most intelligence necessary, not the most intelligence available.

Beyond the Model

This industry spends an enormous amount of time debating Claude versus GPT, GPT versus Gemini, open versus closed models, and whatever model launched Tuesday versus whatever model launches Thursday.

But I have been playing with coding agents and agentic development environments enough lately that something has become obvious: the model is only part of the system.

The new Harness Leaderboard tries to measure exactly this. Hold the underlying model and reasoning effort relatively constant, then change the coding harness around it. The results can change dramatically.

Same model > different tools > different context> different planning > different verification.

Very different outcome.

This is exactly why I keep talking about AI-DLC. The architecture increasingly looks like this:

TASK → MODEL → HARNESS → TOOLS + CONTEXT → VERIFICATION → OUTCOME

If you are spending twice as much on a model but the harness is producing worse results, you have a systems-engineering problem.

Saying “Our AI strategy is Claude” is eventually going to sound a little like saying “Our application architecture is Intel.” Cool. What about everything else?

The AI Slop Machine

I have noticed a strange new phenomenon popping up with internal AI tools and agents.

Someone asks, “Can you create a one-page summary of this?” The AI says, “Absolutely!” and then apparently goes off and writes its doctoral dissertation. Nobody wants to read all of that.

I have seen agentic workflows produce 100+ pages of material. Technically, some of it is pretty good. But like I said, nobody is going to read it.

We spent the last couple of years teaching AI how to create things. Now I think we need to teach it when to stop.

A 100-page implementation specification another AI agent will consume? Maybe that is perfect. A 100-page executive briefing for a CIO? You have lost the plot.

So, here are my rules to avoid the AI slop machine.

Ryan’s Anti-Slop Rules

• Tell it the size of the box. “Maximum 500 words. Three sections. One recommendation. No appendix.”
• Tell it who is reading it. “This is for a CIO who has four minutes and needs to decide whether to fund a pilot.”
• Define what done means. State what the reader should understand and what decision the reader should be able to make.
• Make AI edit AI. “Remove anything that does not materially affect the reader’s understanding or decision. Do not summarize it again. Delete it.”
• Stop confusing volume with work. A 60-page AI-generated document may simply mean nobody told the agent when to stop.

Some Final Thoughts

Which model should do the work? How much reasoning should we purchase? How many agents should we launch? How much context should we provide? How much should those agents ultimately produce?

For the first couple of years of generative AI, the answer always seemed to be more. Bigger models. More tokens. More context. More agents. More output. More everything.

I do not think that is where we are going. The next phase of enterprise AI is going to be about enough: enough intelligence to solve the problem, enough context to understand it, enough autonomy to accomplish the work, enough verification to trust the result, and enough output to communicate the answer.

AI made generation nearly free. Now, judgment is becoming the scarce resource.

Use the small model when the small model works. Use the frontier model when the problem requires it. Build the harness around the work instead of endlessly swapping models. And if somebody asks your new AI agent for a one-page document, please make the damn thing one page.

If your team is trying to choose the right models, control agent costs, or build the architecture around AI rather than just buying another chatbot, reach out to us.

Until next time,
Ryan

Now time for this week’s AI-generated image and the prompt I used to create it.

Create a playful editorial scene of a glossy retro casino slot machine labeled “AI SLOP MACHINE.” The reels show “1 PAGE,” “12 PAGES,” and “47 PAGES.” A prompt card reads “ONE-PAGE SUMMARY” while an absurd stream of paper pours from the payout tray and an office professional pulls the lever. Satirical but polished enterprise-tech style, red, black, cream and warm gold palette, crisp magazine illustration.