How The Leading AI Companies Do Product (Part 2)

Hey Ant here 👋 I write a newsletter that won't go viral - and that's the point.

Because hot takes, clickbait and AI slop won't help you ship better products. But practical content grounded in real work will - the kind I do every day on my own products and the product leaders I coach.

Join 15,000+ product people who subscribe. Same applies to my YouTube channel!

Recent posts you might have missed:

- How The Leading AI Companies Do Product (Part 1)
- Nobody’s Actually Prioritising ‘Value’
- Stop Using Epics & User Stories

Part 1: How Leading AI Companies Do Product.

FYI everything in this post I covered in my recent livestream if you prefer putting this on in the background.

---

Part 2 deep dive into how the people behind Claude Code, Codex and Cursor do product. This post takes a look at changes in capabilities and what not to copy. See part 1 on how they work and what changes with AI.

---

Last week we took a look at how the leading ai companies work and the two fundamental shifts that sit underneath those changes.

This week I want to continue on that theme and cover:

• What different capabilities do these companies have? 

• What skills haven’t changed

• And why copying them might be a terrible idea.

Let's get into it.

Skills in the AI era

Four themes came up consistently across the 30+ podcasts and  interviews I watched.

1. Taste

2. Customer understanding

3. Judgement

4. Verification 

1. Customer understanding

Alexander, the Head of Product for Codex at OpenAI was asked by Lenny in his podcast, if you could only keep one core competency what would it be; 

"it would be really meaningful understanding of the problems that a certain customer has."

And of course there’s some bias here since he’s a product person but this has come up in dozens of interviews with people from engineers to designers too.

And it makes sense. When building is cheap, working out what's worth building becomes even more important.

2. Product Taste

I really don’t like the term ‘Product Taste’ because what does that mean?

And maybe that’s just me, I’m a simple person and I prefer to call ‘a-spade-a-spade’ but it was the term they used and it came up consistently.

So much so it’s become a meme…

But let’s define it.

‘Taste’ to me is knowing what solutions will resonate strongly with users. This is built from lots of reps. Seeing what works, what doesn’t and spending a lot of time with customers.

And in a world where everyone has access to the same LLMs and tools. It’s goint to be increasingly easy to produce the same stuff. ‘Taste’ or that creativity to do something different that resonates will no doubt become the differentiator and I think that’s what they’re calling out.

3. Judgement

Judgment, EQ and the more human skills like communication and empathy were also consistent themes.

Cat Wu, the Head of Product for Claude Code called out emotional intelligence specifically and even said that humans bring a level of “common sense” that the models currently don’t. 

And this all makes sense when you understand how LLMs work. 

Whilst this is an oversimplified definition, LLMs essentially work on probability - they’re looking for what’s the most likely response based on the input and what they’ve been trained on. That’s how you train a large language model - hence the name. It’s a very big model of words (well tokens). That model helps it work out what to say next.

It’s why when you ask it things like; “the carwash is 20m away should I walk or drive?” it’s says walk. Because that question is extremely similar to “the shop is 20m away should I walk or drive?” In almost any other scenario the most probable answer is walk.

Of course they’re expanding the models and improving it to fix these things but at a base level it’s still a LLM and the logic works the same.

What all this means is that they’re not capable of judgment or emotional intelligence, despite how it may seem. Some have argued that LLMs shouldn’t be called AI because they’re not intelligent at all, it’s a model. Others argue it’s the worlds greatest magic trick. They’ve managed to convince everyone that there’s super intelligence when it’s just a very good prediction model. 

I mean imagine if you were really lucky and all you did was guess and 90% of the time you got those guesses right. You’d come across pretty smart right even though you were just guessing.

Anyways; that was a bit of a rabbit-hole!

The problem with LLMs is the same problem with being data-driven rather than data-informed.

All data (and everything the LLMs have been trained on) is from the past. It can look like it’s trying to model the future but it’s not. There’s no critical thinking or  judgement happening.

This means the ability to make bets on the future, predict and critically think are still skills that AI can’t currently replace.

Not to mention taking responsibility for those decisions and bets - regardless of whether they work out or not. That’s something you can’t outsource to AI.

4. Verification

If you want to get more leverage out of AI you need to get agents working without your input - or at least working with less of your input. 

This means longer running sessions, less needing to prompt obvious things or answer obvious questions and having autonomous agents that can run in the background.

In order to achieve that and maintain the right level of quality you need to shift away from prompting to giving AI a clear criteria to verify itself against.

This is the core principle behind ‘loops’ and Loop Engineering (you might have heard these terms be tossed around).

This means skills like evals, tests, guardrails, acceptance criteria, etc are all becoming core skills for how to work with AI at scale and effectively.

And the leading tech companies are now looking for these skills specifically given that’s how they work.

5. Everyone is becoming technical

At first I was thinking - like most people probably - that if AI makes coding super cheap and easy then technical skills become irrelevant. But actually what I’m seeing play out over the last 2 years since ChatGPT launched is the opposite.

Technical skills have become more important.

And I’m not referring to the ability to read and write a specific coding language. I’m talking about understanding the difference between a url that says localhost:3000 vs something that’s actually deployed somewhere.

This is also known as the Jevons Paradox; which is when technological advancements lead to greater consumption, often outweighing the improvement. For example, if you make cars cheaper and more efficient to run, you end up with more people with cars, driving further = more fuel consumed not less.

The same thing is playing out with AI right now. Rather than AI removing engineers, it’s drastically lowering the cost and barrier to entry for creating software leading to an increased demand for technical skills.

Just think, how many vibecoded apps do you think are out there that are missing the fundamentals? Things like automated tests, security, proper data handling, etc.

Probably a lot. So it’s increased the demand for these skills - and if you want to be a successful founder in the future leveraging AI you’re probably going to need to learn this stuff! 

And the leading AI companies know this. It’s why they hire more technical people.

Several interviews with product leaders and design leaders backed this up, confirming that either they already knew how to code or the PMs and designers were quickly learning.

Side bar: I also think this is a nuance behind the “product builder” concept and “everyone shipping code” at these companies they’ve intentionally hired people who are more technical. They’ve also built a lot of scaffolding to make that safe for those who are less technical. But it’s much easier to have everyone shipping in the team when your PM and designers are ex-engineers.

Before you copy any of this

Take everything here (and in part 1) with a grain of salt. 

Whilst a lot of what I heard in those interviews and what I’ve shared here align to what I’m seeing on the ground with my clients, I know that what gets said in interviews vs what’s actually happening don’t always align. 

Sometimes it’s a marketing image thing - other times it’s just the fact there’s so much nuance that can’t be covered in a single question response.

But you also need to take context into account.

I already mentioned that they deliberately hire technical people but there’s a few more that I want to cover.

Your not the same company

I really wish this didn’t need to be said but unfortunately I see too often leaders hearing this stuff or coming back from trips being like “Anthropic does blah” and immediately proceeds to getting everyone to copy it without first even assessing whether it makes sense, whether it would work in their context and at their company..

For example, I mentioned this in part 1 whilst talking about how these companies are incredibly bottoms-up:

“[Alexander Embiricos] gave an honest explanation of why it works for them: they genuinely don't know what's technically feasible next, or what will resonate. Nobody's done this before. So diverging fast, throwing a lot of different ideas out there to see what sticks, is a deliberate strategy for navigating uncharted territory.”

In other words if you’re a regulated company, or in an industry that isn’t at the frontier paving the way for the rest of us, then perhaps many of what I’ve talked about in these two posts might not work for you.

Another important dimension is people and skills.

Of course these companies can attract some of the best people in the world, they can pay extremely high salaries with PM, design and engineering roles all advertised with salaries over $400k.

But it’s also the skills and capabilities you’ve hired for in the past. If you’ve recruited for a different set of skills and capabilities then don’t be surprised if you find adopting some of these practices a bit hard. 

I’m seeing this right now with a lot of my clients. They struggling to elevate engineering out of “building what they’re told” because that’s what they’ve hired for, so that’s what they have. 

Equally they want their PMs and designers to be using AI more, to have those automations and army of agents working for them. But they’re non-technical and they’ve never used a terminal before. 

And it’s not because they’ve hired bad people, it’s just different capabilities.

AI won’t fix your dysfunctions

This is a big one. 

AI won't fix your organisation's dysfunctions.

If anything, it's going to expose them.

In most enterprises coding was never the bottleneck. It was process, approvals, coordination between teams, and executive horse-trading. Making coding - or anything other than solving those problems - faster isn’t going to make an inch of difference. This is exactly why I wrote the post Fix Delivery First.

Frontier tax

Unpopular opinion but there's a cost to being early - and that’s not to say there isn’t a cost to being late either but here’s a real example.

A couple of months ago I wanted to be able to talk to claude through slack. I had already hooked up a bunch of agents to ping me in slack when they were either done with something or needed my support and I thought wouldn’t it be great if this wasn’t one way. If I could also reply in slack and ask questions. 

So I spent a good several hours wiring up customer slack bots, triggers and everything to get it mostly working - at least for my needs. But then a couple of weeks later Anthropic launches Claude Tag that completely makes everything I did redundant.

But this is the cost of being early. 

You’re one of the first to run in certain problems for the first time.

And since you and a bunch of other people are only just now discovering the these problems, there’s no solutions for them yet.

This means you’re either stuck living with the problem or it’s on you to solve them.

In most cases you choose to solve them.

But solving that problem isn’t your core job or business. Eventually someone will solve it better than you - like Anthropic did with Claude Tag to me. 

Anyways food for thought.

Of course, as I mentioned before there’s also a cost related to being late so it’s not perfect. There’s tradeoffs either way.

I hope this and last week’s posts have been insightful and helpful.

If it resonated, forward it to a product person who needs to read it also. It’ll help them and it helps this newsletter grow as the majority of subscribers come from referrals exactly like that!

And as always, if you have any questions hit reply, it comes straight to me.

And I do go into a little bit more in the stream that wasn’t in here so if you’re interested, give that a watch!

/Ant

---

FYI — I'm running a 3x hands-on AI workshops in October, covering AI for discovery, strategy and delivery. Exclusive to the Product Mentorship. More details here.

---