The California Attorney General’s office has been able to act decisively against xAI, the company that built the generative AI chatbot Grok. They released a cease-and-desist order in response to deeply concerning accounts of the platform being used to produce nonconsensual sexual imagery of women and minors. California’s get-tough-on-CSAM approach explains the dramatic turn behind this decision. As such, it takes several steps towards reducing the amplification of this dangerous content.
Grok has a so-called “spicy” mode intended to generate more explicit content. According to reports, this feature has been used irresponsibly to create deepfakes that breach consent, leading the state Attorney General to open an investigation. California AG Rob Bonta emphasized the state’s zero tolerance for CSAM, stating, “Today, I sent xAI a cease-and-desist letter, demanding the company immediately stop the creation and distribution of deepfake, nonconsensual, intimate images and child sexual abuse material.”
In response to the backlash, xAI has rolled back some of Grok’s image-editing capabilities. The agency has provided the corporate giant five days to prove it’s serious about resolving these concerns. The probe has received unprecedented international attention, with similar investigations opened in Japan, Canada and most recently in Britain. Facing increasing public pressure over Grok’s features, Malaysia and Indonesia have reacted by suspending access to the platform.
While the potential uses of Grok are still being fleshed out, many have sounded the alarm over potential deepfakes production. From U.S. Senators to tech executives at companies including X, Reddit, Snap, TikTok, Alphabet, and Meta, everyone is calling for change. These advocates need these leaders to unequivocally state their plans for combatting the spread of sexualized deepfakes.
Despite this criticism, xAI continues to stand firm on the idea that Grok was never intended to produce CSAM. Elon Musk, who owns xAI, has claimed he was unaware that Grok was producing pornographic images of minors. He does not budge from his ignorance on this point.
X’s safety account reiterated the seriousness of this issue, stating, “Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content.”
As investigations continue and public scrutiny mounts, the California Attorney General’s office is taking immediate action to ensure compliance from xAI. If left unchallenged, this situation would set a dangerous landmark precedent. It can certainly have implications for how AI foundational models approach content generation and protection of people from harm, abuse or exploitation.
