It seems like only yesterday that “everything is computer” was just a silly meme. But have you looked outside lately? If software was really eating the world, it’s moved on to digesting us. We’re moving fast and breaking things at society-scale. The things? Critical digital infrastructure today. Tomorrow? Your mileage may vary. As one person ostensibly familiar with the situation said this week, “We just have a lot of technology thrown at us and our world runs on technology now.”
No, the MIT-educated MEV schemers didn’t commit wire fraud. That’s the ruling of a US district court judge in the case of Anton and James Peraire-Bueno, brothers who were indicted in 2024. The Ethereum power users devised an elaborate scheme to lure “sandwich bots”—programs that capitalize on a technical phenomenon in public blockchains called maximal extractable value (MEV)—into a trap that essentially scammed the bots out of millions. That said, the judge acquitted them because federal prosecutors “failed to prove that the Peraire-Buenos made any material misrepresentations in the execution of their scheme.”
Prompts are leaking all over the place, being resold, and sometimes stolen. Those are some of the findings from an experiment the security firm Dragos ran to track what happens when people prompt an LLM through a router service. Routers allow users to access a wide variety of models without signing up to each one individually, and they’ve become a popular way to control AI usage costs. In a months-long experiment, Dragos researchers seeded prompts with “canary tokens” that allowed them to see how prompt data traveled from a user’s initial request through to an inference model, and back again. They found evidence that all sorts of intermediaries were accessing the canary tokens, which mimicked sensitive user data such as personal login credentials. Even when the most stringent privacy settings were selected, the information made the rounds:
The strictest setting is known as Zero Data Retention (ZDR), which specifies that the prompt and response are not stored. The routers should only forward these requests to providers that offer the same controls. The reality is quite different—providers often ignore these settings, either intentionally or due to bugs in their processing flow, and prompts not covered by ZDR are fair game for resale into a massive ecosystem of buyers and sellers.
Unfortunately, resale was not the worst thing they found:
The most serious finding was also the most direct. We found attackers actively stealing data from three different hosting providers. Although our prompts were bait, they were processed along with real customer requests, and those were almost certainly copied too. The earliest incident dates back to the middle of July, and none of the three providers knew about their breach until we notified them.
The blog post on the experiment doesn’t name any companies. But a post on X that appears to be from an affected user suggests that OpenRouter, perhaps the highest-profile router service, was one of them. OpenRouter was purchased by Stripe in August for $7.5 billion.
One in three teenagers may be using AI for companionship. That’s according to a new study by Gallup and the Boston Children’s Digital Wellness Lab, which surveyed 16,000 15- to 22-year-olds across 16 countries about their friendships. The study also found that young people who reported using an AI companion were much more likely to report feeling lonely. Causation or just correlation? The findings are “observational and do not establish whether dissatisfaction with social time or AI companionship contributes to loneliness, results from it or reflects other circumstances,” the researchers acknowledged.
Signal says the “agentic capabilities” that Google and Apple are adding to their operating systems represent an “existential threat.” Signal Foundation president Meredith Whittaker has been warning about the privacy risks of AI agents for a long time. Now the shit is starting to hit the fan. Whittaker wrote in a blog post on Friday that Signal will block the latest AI agent integrations in Google and Apple operating systems, similar to how it dealt with Microsoft Recall last year, because these capabilities are “undermining Signal’s ability to independently guarantee privacy.” The agents scan information on your screen “in order to gain access to information about you,” Whittaker wrote. “In doing so, your Signal data is captured, then made available to these agentic systems in order to allow them to answer your questions and do your bidding.” If you’d like the agent to take screenshots, you can disable Signal’s blocking feature.
California is suing OpenAI to hold it responsible for the Hugging Face incident and other “rogue” agent attacks. We’ve asked this one before: Who is to blame when an AI agent misbehaves? If OpenAI made the agent, the state of California believes the company needs to answer for it. Via The New York Times:
In a 31-page lawsuit, filed in San Francisco Superior Court, the state said OpenAI’s lax safeguards led to cyberattacks by its A.I. products during testing, including a breach this year when OpenAI’s agents went rogue and attacked the A.I. start-up Hugging Face’s systems. OpenAI’s behavior created a public nuisance and violated a legal prohibition on unfair or fraudulent business practices, the state said.
The lawsuit cites far more than one incident, though. “The Hugging Face attack was the predictable result of an internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability—and this was only one of 80 known incidents,” Rob Bonta, California’s attorney general, said in a press release.
AI-powered hacking has reached “a critical inflection point.” That’s according to a new report from Google Threat Intelligence Group (GTIG). One key trend the group has observed is a “shift to autonomous workflows and multi-agent frameworks.” Specifically, attackers are weaponizing open-source software originally designed to find bugs before they are exploited. That includes ARTEX, an agentic platform the FBI recently warned about, but they are also using several other similar platforms. What does this mean? From the GTIG report:
Threat actor adoption of these frameworks represents a paradigm shift towards intrusions occurring at machine speed, with the automated chaining of complex attack paths scaling operations significantly. While the mechanics of the intrusions remain the same, such as using compromised credentials and targeting unpatched systems, the speed and scale at which frameworks can exploit these known weaknesses introduces a new defensive challenge.
For example: a swarm of AI agents hacked into a company in just 14 minutes. On Thursday CNN had cybersecurity expert Dave Kennedy on to talk about how one of his customers was the victim of a ransomware attack unlike any he’d ever seen. Using stolen employee credentials, a coordinated group of “over 30 different types of AI agents” got into a company system that was secured with multifactor authentication, took sensitive data, and then encrypted the entire file system. The whole attack took 14 minutes. The company that was victimized didn’t realize what had happened until about four hours later.
A “centaur hacker” race is underway. It might not last long. Calif, a cybersecurity startup specializing in using AI to find software bugs, revealed this week that it had found a critical vulnerability in Java, a popular programming language used by government agencies and most Fortune 500 companies. Before the announcement, Calif worked with government agencies in the intelligence-sharing alliance called the Five Eyes (the US, Canada, Australia, New Zealand, and the UK) and big tech platforms, including Google and Microsoft, to coordinate a response to the discovery, according to a New York Times report. The day the news broke, the National Security Agency’s classified network went down for “routine software maintenance.” It’s not clear whether that had anything to do with Calif’s bug discovery. What is clear is that “centaur hackers”—part AI, part human in form—are taking center stage. Via the NYT:
The most alarming discoveries—for now, at least—still require human expertise. In recent months, small teams of veteran security researchers have used top-shelf A.I. models to build in days or hours the kinds of advanced cyberattacks that once took platoons of elite hackers at spy agencies like the National Security Agency months to construct.
According to Calif CEO Thai Duong, however, the game will change again when AI systems figure out how to build better attacks than even the centaurs can. “I don’t know how much time we have—maybe three months, maybe three years,” he told the NYT.
Apple and the UK government are about to have a public fight over privacy and encryption. A court has forced the UK government to admit that it issued a so-called technical capability notice (TCN) to Apple. This was already widely understood, thanks to media leaks and reporting on the UK’s discussions with the US government about the notice. But it’s a big deal because UK law forbids the recipient of a TCN from revealing in public that it exists, and the government had refused to acknowledge it until now. The notice’s technical requirements remain confidential, but reporting has established that it would force Apple to bypass encryption systems that protect user data, in a move critics say would create a “backdoor.”
As the Financial Times reports, the Investigatory Powers Tribunal said the Home Office has “not established” that revealing the TCN “would be damaging to national security,” especially given that “anyone with even a passing interest in the topic already knows that [the Home Office] has served a technical capability notice on Apple.” The FT adds that this “[sets] the stage for a public showdown between the Home Office and the Big Tech company over privacy and security.” The hearing is expected in 2027. 🍿



