AI Promised Speed. Why Are We Moving Slower?

Dr. Ryan Ries here with some interesting developments that challenge everything we thought we knew about AI productivity.

This week, I want to talk about a research study that is changing perspectives on AI's current impact and what this means for the future of work.

Quick side note: I gave a talk at the AWS Summit NYC on Delivering Measurable ROI with Generative AI on AWS. It was very well received, so I am giving this same talk on August 27th virtually. Register here if you’re interested.

I am also giving a talk on IDP on August 20th with AWS through Connected Community. We’ll be chatting about streamlining document chaos, so if this is something you need help with, this is the perfect session for you.

AI & Developer Productivity

A study just came out from METR with some surprising findings that go against everything we’ve been hearing about AI boosting productivity.

According to the study, when experienced developers used AI tools like Cursor with Claude, they actually took 19% longer to complete coding tasks compared to working without AI.

It gets even more interesting because the developers thought AI was making them faster. They expected a 24% speed increase, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.

This aligns with another survey I saw of 49,000 developers that found while AI usage is increasing, positive sentiment toward AI tools dropped 10 percentage points this year. More developers now actively distrust AI tools than trust them.

This perception vs. reality gap is huge, and it becomes even more interesting when you look at Microsoft’s recent analysis of jobs most likely to be impacted by AI. The most common theme that I spotted was a trend towards jobs that involved a lot of writing, which would make sense considering they are large language models. The one thing this list doesn’t take into account is how many of these jobs require empathy, which can be questionable from a robotic or avatar perspective.

Top 40 Occupations Impacted by AI

Microsoft just released a list of the top 40 occupations with the highest AI applicability scores - essentially jobs they believe AI will significantly impact or potentially replace. Here are some additional roles that caught my attention:

• Web Developers (AI score: 0.35) - ranked in the top 40 for AI displacement
• Computer Programmers (AI score: 0.44) - even higher on the displacement list
• Data Scientists (AI score: 0.36) - ironically, the people building AI systems

If AI is supposedly going to replace programmers, why did the METR study show it actually makes experienced developers slower?

This disconnect highlights a critical question we're not asking enough: Are we measuring AI's potential based on hype, or on actual real-world performance?

What's Really Happening Here?

The study revealed several factors contributing to this slowdown:

• “Almost right, but not quite”: This is the chief complaint from developers about AI tools from Stack Overflow’s study
• Over-optimization: Developers spent too much time trying to get AI to give them the "perfect" answer
• Context switching: Constantly moving between AI suggestions and their own thinking
• Quality overhead: More time spent reviewing and cleaning up AI-generated code
• False confidence: Believing the work was done faster when it actually wasn't

Now, the study also shows areas where AI does seem to make developers more productive:

• Prototyping and throwaway code – Quick experiments and single-use scripts
• Tasks with lower quality standards – When you don't need extensive documentation, testing, and formatting
• Benchmark-style problems – Algorithmic challenges that are difficult for humans
• Brainstorming and ideation – Many developers report AI being helpful for substantial tasks over 1+ hours

I think this tells us a lot about the “productivity sweet spot.” With better prompting, training, and iterating through AI attempts quickly, you’ll see genuine improvements in productivity.

Education Gets Smarter About AI

On another note, OpenAI just launched "study mode" in ChatGPT – a feature designed to help students learn rather than just get answers. Instead of providing solutions outright, it guides students through step-by-step problem-solving.

This is exactly the kind of thoughtful AI implementation we need. Rather than replacing human thinking, it's designed to enhance learning and understanding. 

As a father of a teenager, I’m really excited to see this. I think it’ll help kids in school learn how to use and understand AI, while still applying critical thinking skills.

My Take: The Right Tool for the Right Job

I believe we're in the "trough of disillusionment" phase of AI adoption, but that's actually revealing where AI's true value lies.

The disconnect between Microsoft's job displacement predictions and the METR study results tells us something important: We're still learning how to measure AI's real impact versus its theoretical potential.

Microsoft's analysis likely looks at tasks AI can do in theory. The METR study looked at what happens when real humans try to use AI for real work. Those are very different things.

AI isn't a universal productivity booster or job replacer. It's a specialized tool that excels in specific contexts.

For your business, this means:

1. Don't panic about job displacement - The reality may be much more nuanced than the predictions
2. Focus on augmentation, not replacement - Use AI to enhance your team's capabilities in the right contexts
3. Match AI to the right tasks – Start with prototyping, brainstorming, and lower-stakes work
4. Invest in training your people – The productivity gains seem to come after extensive usage and learning
5. Measure what matters – Track actual outcomes, not just perceived speed or theoretical capability

Looking Ahead

Studies like the METR one are helping us understand where the real value lies. Rather than AI being universally faster, we're learning it's more like having a specialized team member – incredibly powerful in the right situations, but you need to know when and how to deploy it.

The most successful AI implementations will be those that match the technology to the right use cases, rather than trying to force it into every workflow.

As always, if you want to chat about identifying the right AI opportunities for your organization – the ones where it actually speeds things up and helps you see real ROI – my team and I are here to help you navigate both the hype and the reality.

Until next time,
Ryan

Now, time for our AI-generated image and the prompt I used to create it.

Create an image of the famous story the tortoise and the hare, but make the characters in a muppet style.