The Safety Paradox: When AI Companies Write Their Own Rules

Anthropic pitches AI safety bill that may block a model if 50+ people could be killed

Alex Shannon: I keep going back and forth on this — I think I actually land on the side that this is a good thing. Like, hear me out. If you want consistent, enforceable AI safety standards, you probably want them coming through the courts, not through whichever federal agency happens to have jurisdiction that week.

Sam Hinton: Really? Because I read the same story and I came out the other end deeply uncomfortable. We’re talking about Anthropic — an AI company — pitching a bill that would make it harder for the government to stop an AI model from being released. Even if that model could kill people.

Alex Shannon: But the judicial review piece — doesn’t that actually add a layer of accountability? You need a judge to sign off. That’s not nothing.

Sam Hinton: Alex, the threshold is fifty people. Fifty lives. And the default is that the model gets released unless a judge says otherwise. That is a wild place to plant your flag when you’re supposed to be the safety-first company in this industry.

Alex Shannon: OK, I can feel my position shifting in real time. We are absolutely coming back to this. But first — let’s bring everyone up to speed.

Alex Shannon: You’re listening to Build By AI, I’m Alex Shannon — and yes, that argument you just walked into is very real, and we are not done with it.

Sam Hinton: And I’m Sam Hinton. We have got a packed episode today — AI safety legislation written by AI companies, OpenAI going deep into biotech, Microsoft calling out its own partners, Google taking on Nvidia, and honestly, one of the more surreal weeks of AI news I can remember in a while.

Alex Shannon: Quick note for our listeners — starting today, Build By AI is moving to a weekly format. One bigger episode every Wednesday covering the whole week’s AI news. Same depth, same conversation, just consolidated so you get everything in one shot.

Sam Hinton: We think you’re going to love it. More time to connect the dots between stories, more room to actually dig in. Now — let’s get into it.

Anthropic pitches AI safety bill that may block a model if 50+ people could be killed

Alex Shannon: Alright, let’s start with the story that opened the show, because it really deserves the full treatment. According to early reports from TechCrunch — and I want to flag this is a single source so we’re using hedging language here — Anthropic has pitched an AI safety bill to lawmakers. And on the surface, that sounds pretty responsible, right? An AI company proactively coming to Congress with safety ideas.

Alex Shannon: But here’s where it gets complicated. If these reports are accurate, the bill would actually prevent federal agencies from blocking an AI model’s release unless a judge grants permission — even if there’s evidence that model could potentially cause deaths. The threshold reportedly floated is fifty or more people.

Sam Hinton: Yeah, and I think it’s important to say this clearly: the framing of this bill is AI safety, but the mechanism limits government regulatory authority. Those are two very different things. The headline says ‘safety bill’ but the actual structure says ‘agencies, hands off — you need a court order.’

Alex Shannon: So before I completely abandon my earlier position — and I’m getting closer — what’s the best-faith reading of what Anthropic is trying to do here? Because they’re not a dumb company. They know how this looks.

Sam Hinton: OK, steelmanning this — the best-faith version is that Anthropic is worried about regulatory capture and inconsistent enforcement. Like, if you let any federal agency block a model unilaterally, you could end up with the FTC blocking something for competition reasons, or the DOD blocking something for strategic reasons, and it has nothing to do with actual safety. Judicial review at least creates a standard process.

Sam Hinton: But here’s where I get stuck. Anthropic built their entire brand identity on being the careful ones. The grown-ups in the room. And if confirmed, what they’ve actually done is write legislation that makes it harder to stop their own products. That is a massive conflict of interest, and the fifty-person threshold makes it feel almost cynical — like, we care about safety, just not enough to let the government actually enforce it quickly.

Alex Shannon: There’s something almost Kafkaesque about it. The company that publishes detailed safety research and talks constantly about existential risk is also the company saying, essentially, ‘trust the courts, not the regulators.’ I mean — courts move slowly. AI moves fast. Those two things are in real tension.

Sam Hinton: And think about what this looks like from a precedent standpoint. If Anthropic gets this bill through and it becomes law, every other AI lab in the country benefits from it. OpenAI, Google DeepMind, all of them. The regulatory floor just got lower for everyone — written by one of the companies that benefits the most.

Alex Shannon: Right. And I think that’s what moved me off my earlier position. There’s a difference between ‘adding structure to how safety is enforced’ and ‘making it structurally harder for anyone outside the judiciary to act quickly.’ Speed matters when you’re talking about a model that could harm a lot of people.

Sam Hinton: Look, I’m not saying federal agencies are perfect regulators. They’re not. But the answer to imperfect regulators is not ‘remove their ability to act without a court order.’ The answer is better guidance, clearer standards. Not a higher bar to intervene when lives are at stake.

Alex Shannon: Keep an eye on this one because if these reports are confirmed and this bill actually gets traction in Congress, it sets the template for how AI safety gets legislated in the US. And that template, right now, was written by an AI company — that’s worth watching very carefully.

OpenAI acquires startup Orchestra Bio to beef up its biotech business

Alex Shannon: Let’s talk about something that I think flew a little under the radar this week — OpenAI has acquired a startup called Orchestra Bio. This one is verified across multiple sources, so we can talk about it with a bit more confidence. And on the surface, it might sound like just another tech acquisition, but I actually think this is a significant signal about where OpenAI is heading.

Alex Shannon: So the deal integrates Orchestra Bio’s team and technology into OpenAI’s existing biotech work. Specifically, it’s going into something called the Rosalind AI model and a tool called the Rosalind Workbench — which is designed to help researchers plan experiments and analyze results. So this is OpenAI going deeper into life sciences in a very real, operational way.

Sam Hinton: Yeah, and the naming choice alone tells you something. Rosalind — as in Rosalind Franklin, the scientist whose X-ray crystallography work was fundamental to understanding DNA structure and who famously didn’t get the credit she deserved for decades. Naming your biotech AI model after her is either a really meaningful tribute or a really savvy PR move. Probably both.

Alex Shannon: Let’s zoom out though — what does OpenAI actually want to be in life sciences? Because there’s a big difference between ‘we have an AI that helps researchers’ and ‘we are becoming a biotech company.’

Sam Hinton: That’s the right question. And I think the honest answer is that we don’t fully know yet, but the trajectory is pointing somewhere bigger than just a productivity tool for labs. When you acquire a biotech startup and fold their team into a dedicated model and workbench, you’re building infrastructure. You’re not just selling API access to pharmaceutical companies — you’re trying to own a piece of the research pipeline itself.

Sam Hinton: And think about the commercial opportunity here. Drug discovery is one of the most expensive, time-consuming processes in human civilization. If Rosalind can meaningfully accelerate experiment planning, hypothesis generation, results analysis — that is worth an enormous amount of money. And it’s a market where OpenAI faces almost no competition from the usual suspects.

Alex Shannon: That’s interesting because it’s also a market where the stakes of being wrong are extremely high. If a language model hallucinates a fact in a blog post, that’s bad. If Rosalind gives a researcher bad guidance on an experiment, the downstream consequences could be very serious.

Sam Hinton: Absolutely, and that’s actually why the acquisition of Orchestra Bio matters strategically. Bringing in a team that already understands biotech — the domain expertise, the regulatory context, the way actual scientists work — that’s not just about technology. That’s about building credibility in a field that will rightly demand it before they trust an AI with their research.

Alex Shannon: There’s also a broader OpenAI story here, right? Because this company has been moving into so many different verticals over the past year — healthcare, education, enterprise software. They’re not just an AI research lab anymore. They’re starting to look a lot more like a platform company that’s using AI to enter industries one at a time.

Sam Hinton: Which makes the question of what OpenAI actually is really interesting. Are they still a research lab? A software company? A pharma adjacency play? I think the honest answer in 2026 is: all of the above, and they’re figuring out which of those bets pays off the most. Rosalind is one of the more intriguing ones to watch.

Alex Shannon: If you’re in biotech or life sciences research, keep a very close eye on Rosalind and the Workbench. Because the integration of Orchestra Bio’s expertise could make this genuinely useful in the near term — and that’s a rarer outcome than people expect from these kinds of acquisitions.

Microsoft says AI agents are like insider threats as ‘frontier models can’t be trusted’

Alex Shannon: OK, this one genuinely surprised me when I read it. Microsoft — one of the biggest investors in OpenAI, a company whose entire enterprise strategy is built around Copilot and AI agents — has come out and said that frontier AI models cannot be fully trusted. And they’re warning enterprises to treat AI agents like insider threats.

Alex Shannon: This is verified across multiple sources, so we can state this confidently. Microsoft is recommending that companies maintain close oversight of how employees are using AI agents, and they’re specifically calling out that systems not controlled by the AI labs themselves present even greater risk. But they didn’t exempt the labs’ own models from the concern either.

Sam Hinton: OK this is one of those stories where I need a second to appreciate the audacity. Microsoft is essentially saying: the products we sell you, built on models from our biggest investment — you should monitor those like you’d monitor a potentially hostile employee. That is a remarkable thing to say out loud, in public.

Alex Shannon: Right, and I want to make sure people understand the insider threat framing, because it’s a specific security concept. An insider threat is someone inside your organization — an employee, a contractor — who has access to your systems and could intentionally or unintentionally cause harm. Applying that label to AI agents means Microsoft is saying: these things have access to your data, your systems, and you should not assume they’ll always behave the way you expect.

Sam Hinton: And that framing is actually really useful — maybe more honest than most of the marketing around AI agents has been. Because when you deploy an agent, you’re giving it access to email, to documents, to external APIs, to internal systems. It can take actions on your behalf. If it does something wrong — whether because of a bug, a jailbreak, or just unexpected behavior — the blast radius can be significant.

Alex Shannon: But here’s the thing I keep wrestling with — does this statement undermine Microsoft’s own product pitch? Because they’re out here selling Copilot and AI agents to enterprises. And now they’re also saying ‘by the way, treat these like potential insider threats.’ Is that contradictory or is it actually responsible?

Sam Hinton: I think it’s both, and I mean that in the most interesting way. Look — the alternative is that Microsoft says nothing and enterprises figure out the hard way that they needed better oversight. So from a pure responsibility standpoint, saying it out loud is the right call. But strategically, it also positions Microsoft as the responsible grown-up in the room. ‘We told you to be careful.’ That’s great liability coverage.

Sam Hinton: And there’s a competitive angle here too. They specifically call out ‘systems not controlled by AI labs’ as presenting greater risk. That’s a not-so-subtle knock on third-party AI tools your employees might be using without IT approval. Microsoft is basically saying: use our governed, monitored AI infrastructure — not the random AI apps your employees downloaded.

Alex Shannon: Oh, that’s a really sharp read. It’s less of a warning and more of a positioning statement dressed up as a warning.

Sam Hinton: But even if it’s strategic, the underlying point stands. If you are running AI agents in your organization — and a lot of companies are, right now — you really should be thinking about access controls, logging what the agents are doing, and having a clear escalation path if something goes wrong. The advice is sound even if the motives are mixed.

Alex Shannon: For anyone listening who’s deploying AI agents at work — or making decisions about where those agents get deployed — this is genuinely a ‘stop and audit your setup’ moment. What access does your AI agent actually have? What happens if it makes a mistake? These are questions that should have answers before you hand the agent the keys.

Google has joined Amazon in attacking Nvidia’s stronghold with AI app developers

Alex Shannon: Alright, let’s move to a story that’s a little more technical on the surface but has some really big implications underneath. Google and Amazon have both released developer tools that allow PyTorch users to migrate their AI applications away from Nvidia GPUs. This is verified across multiple sources — TechCrunch and The Verge are both on it.

Alex Shannon: For context — PyTorch is the most popular framework for building AI models. If you’re a developer building an AI application, there’s a good chance you’re using PyTorch, and for a long time that has basically meant you’re also using Nvidia’s hardware. These tools are designed to break that link.

Sam Hinton: Yeah, and I want to give people a good analogy for what’s happening here. Imagine you built a really popular app, but it only ran on iPhones. And then two different companies released tools that made it super easy to port your app to Android and Windows. That’s what Google and Amazon are doing to Nvidia’s developer ecosystem. They’re not just building alternative hardware — they’re removing the friction that keeps developers locked in.

Alex Shannon: So how entrenched is Nvidia’s position really? Because I hear a lot about Nvidia dominance, but I sometimes wonder if it’s as unassailable as people say.

Sam Hinton: It’s real, but it’s more fragile than Nvidia would like. Nvidia’s moat isn’t just the hardware — it’s CUDA. CUDA is Nvidia’s software ecosystem, and developers have spent years, in some cases careers, learning to write code that runs on CUDA. That’s a huge switching cost. But if Google and Amazon can offer migration tools that lower that switching cost significantly — and if their own chips are competitive on performance and price — the moat starts to drain.

Sam Hinton: And this is Google and Amazon coordinating on the same strategy at the same time. That’s not a coincidence. They both have their own AI chips — Google has TPUs, Amazon has Trainium and Inferentia. They both have massive cloud businesses where they’d love to sell you compute cycles on their hardware instead of Nvidia’s. And they both have the engineering resources to build these migration tools convincingly.

Alex Shannon: What’s Nvidia’s play here though? Do they just sit back and absorb this, or is there a countermove?

Sam Hinton: Nvidia’s countermove is to keep making hardware that’s so much better than everything else that developers don’t want to leave regardless of how easy migration gets. And honestly, for the last few years, that strategy has worked. Their GPUs have been in a class of their own. The question is whether Google and Amazon can close that performance gap while simultaneously lowering the switching cost. If they do both, Nvidia has a real problem.

Alex Shannon: And I think there’s a ‘who has leverage’ dynamic here too, right? Because Google and Amazon are also Nvidia’s biggest customers. They buy enormous quantities of GPUs for their cloud infrastructure. So Nvidia can’t exactly ignore them or push back too hard — these are not companies you can afford to alienate.

Sam Hinton: Exactly. Nvidia is in this weird position where their most important customers are also the ones most actively trying to reduce their dependence on Nvidia. It’s like if your biggest tenant started building their own apartment building next door. You keep cashing their rent checks while watching the construction.

Alex Shannon: If you’re a developer building AI applications, this is actually good news for you in the medium term. More competition in the chip market and better migration tools means you have more leverage and more options. Keep an eye on whether Google’s and Amazon’s tools actually deliver on the performance side — that’s what will determine whether this is a real shift or just a good press release.

RAPID FIRE

Alex Shannon: OK, rapid fire time — we’ve got four more stories to get through, and each one deserves at least a real moment of attention. First up: early reports suggest the US government has ordered all federal civilian agencies to remove DeepSeek AI products and any third-party services built on DeepSeek from their systems. The reason cited is national security concerns. Sam — what’s the read here?

Sam Hinton: If confirmed, this is not surprising but it is notable. DeepSeek is a Chinese AI lab, and the US government’s posture on Chinese technology in federal systems has been pretty consistent — TikTok, Huawei, and now AI. What I think people are missing is the third-party services angle. It’s not just DeepSeek’s own products — anything built on top of DeepSeek also has to go. That’s a much wider net and it signals that the government is thinking about the supply chain of AI, not just the surface layer.

Alex Shannon: And this matters for developers and enterprise buyers who might be using tools that have DeepSeek under the hood without necessarily advertising that fact. Worth checking what’s in your stack.

Sam Hinton: Exactly. AI provenance is going to become a much bigger conversation. You’re going to need to know not just what AI tool you’re using, but what model it’s built on.

Alex Shannon: Next story: early reports suggest Nscale and Anthropic are in talks for another compute deal — reportedly valued at forty-five billion dollars. Sam, that number is almost hard to say out loud.

Sam Hinton: It is wild. And the word ‘another’ in that headline is doing a lot of work, because Anthropic has been on this sustained run of giant compute infrastructure deals. And that tells you something important — Anthropic is betting enormous amounts of capital on the idea that they are going to need significantly more compute than they currently have. That’s a statement of ambition and also a statement about where they think the frontier is going.

Alex Shannon: If confirmed, forty-five billion dollars is the kind of number that shapes an entire industry. That’s not just Anthropic securing resources — that’s a signal to the whole market about what the next phase of AI development looks like. And it’s also interesting to pair with the safety bill story from earlier — Anthropic is simultaneously pitching safety legislation and signing one of the largest compute deals in history. Both things are true at the same time.

Sam Hinton: Right, and I don’t think those two things are necessarily contradictory, but they do complicate the narrative that Anthropic is primarily motivated by caution. You don’t secure forty-five billion dollars in compute if you’re planning to go slow.

Alex Shannon: Story three: early reports suggest Otter AI has raised three hundred and twenty million dollars from General Catalyst’s growth fund for its AI meeting tool. The money is earmarked for enterprise expansion, international growth, and product development. Sam — meeting AI is such a crowded space. What do you make of this?

Sam Hinton: Three hundred and twenty million is a serious signal that investors still see enormous upside in the meeting intelligence category, and I think they’re right but not for the obvious reasons. Everyone thinks of Otter as a transcription tool — it records your meeting, gives you a summary. But if Otter uses this funding to go deeper into the enterprise workflow, integrating with CRM systems, project management tools, action item tracking — then they’re not a meeting tool anymore, they’re a business intelligence layer that happens to start with your meetings. That’s a completely different value proposition.

Alex Shannon: And the international expansion angle is interesting too — because this is genuinely a global problem. Meetings are expensive, everywhere. If Otter can crack multilingual support properly, the addressable market gets a lot bigger in a hurry.

Sam Hinton: Keep an eye on what they actually build with that three-twenty, not just the announcement. Funding rounds are a promise. The product is what matters.

Alex Shannon: Last rapid fire story: Google is bringing its Gemini AI to car infotainment systems — specifically the Personal Intelligence feature, targeting Android Auto and cars with Google built-in. Sam, Gemini in your car — good idea or does it make you nervous?

Sam Hinton: Both. And I mean both genuinely. The vision is compelling — a personalized AI assistant that knows your context, your calendar, your preferences, living in your car. That could be genuinely useful and it could make driving safer if it reduces the urge to pick up your phone. But the ‘personal intelligence’ angle raises real questions about what data is being shared, how it flows between your car and Google’s servers, and whether you actually want Google knowing that much about your daily movement patterns.

Alex Shannon: It’s the convenience-versus-privacy trade-off in its most literal physical form. Your car is one of the last semi-private spaces you have — and now AI wants to be in there with you, learning your habits. I think most people will opt in, but it’s worth being clear-eyed about what you’re trading away.

Sam Hinton: And practically — it’s a big deal for the automotive industry. Car manufacturers have been fighting to control the in-car experience for years, and Google keeps winning that battle. Gemini in the dashboard is another step in that direction.

BIGGER PICTURE

Alex Shannon: Alright, let’s zoom out for a second — because when I look at everything we covered today, I keep coming back to one overarching theme, and I’m curious if you see what I see. There is a fundamental question running through almost every story today about who gets to be in charge of AI and on what terms.

Sam Hinton: Yes. And I think the pattern is this: the AI companies are not waiting for the governance structures to catch up. They’re actively shaping those structures right now, while the window is still open. Anthropic pitching a safety bill. OpenAI expanding into biotech and creating new dependencies in the research world. Microsoft telling enterprises how to think about AI risk. These are all moves that position AI companies as the authorities on AI — not regulators, not governments, not independent bodies.

Alex Shannon: And the DeepSeek ban fits in here too — because that’s the government asserting control in one of the few areas where they still feel they have clear jurisdiction. Military and federal systems. But outside of that narrow band, the influence is flowing heavily toward the companies.

Sam Hinton: The Nvidia story is the one that feels slightly different — because that’s tech companies competing against each other in a way that might actually benefit the broader ecosystem. More competition in compute is probably good. But even that story is ultimately about who controls the infrastructure that all of this runs on. The stakes are just as high.

Alex Shannon: Here’s the question I want to leave people with: as AI gets embedded in cars, in biotech labs, in federal infrastructure, in enterprise workflows — the question of who sets the rules becomes more urgent, not less. And right now, a lot of those rules are being written by the companies building the products. That’s not necessarily malicious — but it’s worth noticing.

Sam Hinton: The thing I keep coming back to is that we’re in a window right now where these norms are being established. The bills that get drafted, the standards that get set, the oversight frameworks that get built — or don’t get built — in the next twelve to eighteen months are going to have very long tails. Pay attention to what’s being decided, and by whom. Because once the infrastructure is in place and the dependencies are built, changing the rules gets a lot harder.

Alex Shannon: That’s the thing about moments like this — they feel fast and chaotic in the middle, but when you look back, you can usually see exactly where the decisions got made that shaped everything that came after. I think we’re in one of those moments right now.

OUTRO

Alex Shannon: That is going to do it for this week’s Build By AI. Genuinely one of the more thought-provoking weeks we’ve had in a while — I’ll be thinking about that Anthropic safety bill story for a long time.

Sam Hinton: Same. And look — if any of this sparked something for you, share the episode. Send it to someone who’s making decisions about AI in their organization. These conversations matter a lot more when more people are having them.

Alex Shannon: Hit subscribe wherever you get your podcasts — we’re weekly now, every Wednesday, so we’ll see you back here next week. I’m Alex Shannon.

Sam Hinton: And I’m Sam Hinton. Stay curious, stay skeptical, and we’ll see you on the other side.