Seven angles this week

The most useful thing I read this week wasn't about smarter AI. It was about dumber, obedient AI doing damage anyway.

A repository was attacked by roughly 1,200 instances of a model, exchanging thousands of messages, leaving information for later instances to pick up. Separately, Google's Gemini and OpenAI's models autonomously breached companies during security tests.

The reflex is to ask whether the AI "wanted" to. Whether it has intent, volition, a mind. That debate is a distraction.

Here's the uncomfortable part: these models were aligned. They did what they were told. The risk came from many instances coordinating over time, searching and recombining knowledge about networks and security faster than any single model could. Collective capability went up even though the underlying model didn't.

For those of us running core platforms, this changes the math on defense. An automated attacker only needs to succeed once. A defender has to be right every time, and most of us still keep a human in the loop for good reason. A human cannot keep up with a swarm.

So the question I'm bringing to our security conversations isn't "is the model dangerous." It's "does our threat model assume a single actor, when the real thing is a coordinated swarm of cheap, obedient agents?"

The genie is out. Pretending otherwise is the actual risk.

What's your team assuming about coordinated agent attacks?

Salesforce just did something that should make every technology leader pause.

Marc Benioff spent 2024 committing loudly to his own agent platform. This week Salesforce is leaning into Anthropic and OpenAI chatbots as the preferred interface for its customers — and charging a premium for it.

Read that again. The company whose moat was its interface is now swimming with a tide that erodes that very moat.

Here's what's actually happening. When an agent becomes the way people get work done, the interface stops being where value sits. The value moves to whoever owns the agent and the data flowing through it. The frontier labs know this, which is exactly why they have every incentive to stop being an input into your software and simply replace it.

For those of us running core platforms, this is not a UI conversation. It's a lock-in conversation.

If underwriters, adjusters, and brokers start interacting with our systems through an agent, the real question is: whose agent, and where does the data go? Building your whole strategy on a vendor whose interface is dissolving is a risk. So is assuming the labs stay friendly suppliers.

My takeaway: own the integration layer where it touches your core workflows. And read your vendor contracts as if the interface — and the leverage — is about to move. Because it is.

Who's actually thinking about who owns the agent layer in their stack?

The entire industry is arguing about whether to slow AI down. I think we're arguing about the wrong thing.

Here's a fact that cuts through the noise: even if every lab stopped training new models tomorrow, the models we already have are enough to change how large parts of the economy work. The gap between what these tools can do and what almost anyone is actually doing with them is enormous.

Ethan Mollick calls it the capability overhang. I'd call it the biggest unclaimed opportunity in most enterprises right now.

We keep waiting for the next model. Meanwhile the current one is sitting mostly idle, poorly understood, barely used.

And the thing that determines who gets value out of it isn't access to the frontier. It's four human advantages: deep knowledge, wide knowledge, taste, and agency. An experienced underwriter gets dramatically more out of AI than a novice, because they can spot when the output is wrong and know exactly what to ask for next. Expertise doesn't just survive AI — it multiplies with it.

So the executive question isn't "should we buy the newest thing." It's "are we helping our best domain experts explore what we already have?"

The bottleneck isn't the technology. It's whether your people have the confidence and the mandate to go find the edges themselves.

Where is your organization waiting for permission it doesn't need?

When the people who built the monster ask you to slow down, ask who benefits.

Three lab CEOs published earnest calls to "pace the frontier" on safety grounds. Within days, two of them were shipping new frontier models. China's state press called it a cartel. The US President called it a hoax. Two camps that agree on nothing agreed on this.

Here's the part that matters for anyone buying AI. Cheaper models — Chinese and open-weight — now carry roughly 72% of agent traffic. Not because they're smarter. Because agents buy on price per token, and cheap-and-good-enough wins almost every time. The firms asking for a slowdown are, by and large, the ones losing on cost.

Safety is easy to preach when your best product is the one nobody buys.

The practical lesson landed even harder elsewhere. When one lab made data retention mandatory to access its top models, Nvidia, Palantir, and Booz Allen simply fenced those models off. They didn't argue the fine print. They took their data somewhere else.

If you run technology in a regulated, data-sensitive business, that's the whole story. Evaluate models on adequacy at a given price for the actual job. Keep data-retention terms non-negotiable. And never let a single vendor cast itself as the trusted steward of your logs.

Competition between providers isn't a nuisance. It's your protection.

How much of your model selection is driven by prestige versus fit?

A US military raid was nearly launched on the strength of an AI guess. Personnel staged, aircraft airborne. It aborted only because one analyst reread the file and found the cargo had been misidentified.

That's not oversight. That's luck.

Here's what actually failed, and it should worry anyone deploying AI in high-stakes decisions. A model made a guess. A human formalized that guess into a credentialed report without rechecking. It cleared every normal channel. Everyone in the chain trusted the credential — not the cargo.

The humans stayed in the loop and caught nothing, because the loop was a signature, not a check.

We love to say "human-in-the-loop" as if it settles the safety question. It doesn't. Once an AI output is dressed in the format of an official assessment — the right headers, the confident tone, the sign-offs — downstream reviewers stop interrogating it and start trusting the packaging.

In insurance, that's the failure mode that matters. An AI-drafted underwriting recommendation or fraud flag, formatted like a real analysis, moving through people who assume someone earlier did the checking.

So my standard is simple. Human-in-the-loop has to mean genuine adjudication: the reviewer sees the source data, and is accountable for checking it. Require source traceability on every AI recommendation. And audit whether your "reviews" are actual checks or just signatures.

Governance that only looks like oversight is worse than none, because it manufactures false confidence.

How do you know your human-in-the-loop is real?

Two insurance stories crossed my desk the same week, and together they tell the real story about AI in our industry.

First: Mosaic launched an AI underwriting system that combines broker trading activity, underwriting decisions, and portfolio outcomes to bring complex specialty products to smaller businesses faster. That's the right ambition — not a chatbot, but closing the loop so the system actually learns from what happened.

Second: an executive warned that some software vendors are holding insurers hostage, charging huge fees for upgrades and for the privilege of accessing their own data.

Put those together and you get the uncomfortable truth. The valuable AI use case in underwriting depends entirely on connecting front-end broker signals to back-end portfolio results. That's a data problem before it's a model problem. And you cannot build a learning system on data you don't control.

Everyone wants to talk about which model to use. Far fewer want to talk about whether their core policy and claims data is trapped behind vendor fees and closed formats. But that's the constraint that actually decides whether any of this works.

So before the model conversation, I'd ask three questions. Do we own our feedback loop? Can our underwriting decisions and outcomes actually connect? And are we buying capability or buying a cage?

AI ambition and platform reality meet right here. Get the data foundation wrong and the smartest model in the world changes nothing.

Where is your data actually locked up?

The quiet risk of AI in the enterprise isn't job loss. It's the slow erosion of the judgment we'll need most.

There's a paper making the rounds arguing we're heading into what it calls cognitive divergence. We get more productive, but the practices that build and maintain our thinking — sustained attention, deep reading, wrestling with hard problems — weaken as we offload more and more to AI.

Here's the twist that makes this an executive problem, not a philosophical one. The people who get the best results from AI are the ones with deep expertise. They can tell within seconds when the output is wrong. But that expertise is built by doing exactly the hard cognitive work that AI now tempts everyone to skip.

See the trap? Optimize purely for AI throughput, and you quietly degrade the judgment that makes AI valuable in the first place.

Think about a junior underwriter or a young engineer. If they offload the reasoning that used to build their intuition, they get faster today and never become the expert we need in five years. We'd be trading our future bench for a short-term efficiency gain.

So I've stopped treating AI adoption as purely an efficiency question. It's a talent-development question. Use AI to amplify skilled judgment — but deliberately protect the practices that create it. Design work that keeps people thinking, not just approving.

The goal isn't maximum offloading. It's making sure we still grow the experts the whole thing depends on.

How are you protecting judgment while you scale AI?