Here's what broke: a European nonprofit called AI Forensics tested nine of the most popular image editing Spaces on Hugging Face and found that seven of them could easily generate nonconsensual nude deepfakes of women. They then set up honeypot Spaces and tracked over 1,000 prompts in a single week. Seventy-three percent were sexual. Among those, 83 percent aimed to undress or sexualize the person in the submitted photo. Ninety-five percent of targets were women. And 6.7 percent of the sexual requests targeted apparent children. Hugging Face did not respond to WIRED's questions about its content moderation or safety practices.
The surface explanation writes itself: Hugging Face is an open-source platform, content moderation is hard at scale, bad actors will always find a way, et cetera. You've heard this one before. It's the same defense every platform has used since the mid-2000s—we just host the content, we can't police everything, our terms of service prohibit this. Hugging Face does have content policies that ban child sexual abuse material and sexual deepfakes. In theory.
Hugging Face isn't just failing to catch abuse. It's serving the tools. - The Systems Bastard
ERROR: MODERATION_BUDGET = $0; VALUATION = $4.5B
The structural problem isn't that Hugging Face can't moderate. It's that the incentive to moderate runs directly against the incentive to grow. Hugging Face was valued at $4.5 billion in its last funding round, backed by Salesforce, Google, Amazon, Nvidia, and others. It has raised roughly $395 million total. Its entire value proposition is being the GitHub of AI: an open repository where anyone can share models, datasets, and apps. The more models hosted, the more users attracted, the more enterprise customers, the more revenue. Friction kills growth. Content moderation is friction. The math here is not subtle.
This is the same structural failure that produced YouTube's child exploitation problem, Tumblr's CSAM crisis, and other cases where a platform's growth metrics were mechanically misaligned with user safety. What makes the Hugging Face case distinct is that the harmful capability isn't a bug in the platform—it's a feature of the models it hosts. These aren't users uploading illegal content; they're users running software that the platform provides. Hugging Face isn't just failing to catch abuse. It's serving the tools. Research has identified gaps in how U.S. technology law governing platform responsibility addresses cases where harmful capabilities emerge from the software itself rather than user-uploaded content. In the open-source AI world, those responsibility questions often remain unresolved.
Meanwhile, major technology companies are facing distinct pressures around intellectual property enforcement, talent retention, and safety practices. These are all expressions of the same underlying problem: the AI boom has created an incentive landscape where the rewards for building fast dwarf the penalties for building recklessly. Companies face pressure to enforce basic property rights through costly litigation. Engineering talent migrates toward better financial packages. And platforms hosting open-source models receive significant valuations while hosting software that enables nonconsensual intimate imagery. Nobody in this picture is irrational. That's the worst part.
The fix is simple and will make every open-source absolutist furious: Hugging Face should be required to run automated classification on every image submitted to its hosted Spaces—not the models themselves, but the live inference endpoints it operates—and block requests that match known abuse patterns. Not optional. Not community-flagged. Not a public discussion thread where you report a repo and hope someone reads it. Mandatory, automated, at the point of execution. If you host the compute, you own the output. That's not a novel legal theory. It's how we already regulate other services that process user-generated content at scale. The only reason it hasn't happened here is that regulators still think "open-source AI" is too complicated to touch. It isn't. The abuse is happening on Hugging Face's servers, using Hugging Face's infrastructure, generating revenue for Hugging Face's business. That's not a hard case. It's a simple one that nobody wants to bring.