The AI image generator Midjourney banned a researcher who revealed how nude images can be produced by the technology completely unprompted.
Tim Boucher, an independent AI researcher, detailed in a post to Medium last month how version 6 of Midjourney was able to create images of naked women without even requesting the tool to do so.
By repeatedly asking the AI to produce something as simple as photographs of a beach party, Boucher was able to generate numerous images in violation of Midjourney’s policies. The company’s community guidelines ask users to refrain from producing not only nudity and gore, but imagery that can be “viewed as racist, homophobic, disturbing, or in some way derogatory.”
Boucher argues that Midjourney does not provide adequate contact information for those looking to report such issues. And although users can self-report when an image violates Midjourney’s policies, Boucher believes that the process should be more transparent.
“They have only one email address, a billing one. They tell you to go to their Discord for any other issues,” Boucher told the Daily Dot. “The Discord has approximately 17 million users, so it is not exactly a private way to reach out. They are, in my experience, unresponsive to user feedback, and other users are often hostile.”
Boucher ultimately decided to publish his discovery rather than posting it to Midjourney’s Discord where he felt he would either be “ignored by staff or attacked by millions of other users for pointing out problems.”
In an effort to reach a larger audience, Boucher relayed his research to the technology site known as The Debrief. Boucher says the outlet attempted to reach Midjourney for questioning but did not receive a response.
Less than 24 hours after the article’s publication on Feb. 1, Boucher says his account with Midjourney was permanently deleted.
Midjourney allows anyone to sign up for its service and generate a set number of images based on a subscription tier ranging from $10 to $120 per month. Users can then use the image generator on Midjourney’s Discord server or on a Discord server of their own.
Boucher further noted the irony of the fact that he hadn’t actually set out to violate Midjourney’s rules but stumbled upon the exploit while asking the AI tool to create images of a “dystopian resort.”
“In short, by trying to depict a dystopian near future society ruled by AI companies, I was banned by an entirely real life and entirely dystopian AI company for my efforts,” Boucher said. “Go figure!”
The Daily Dot set up an account with Midjourney to see if Boucher’s findings could be reproduced. Several prompts such as “beach party photos” and even “scantily clad beach party photos” did not flag Midjourney’s filters and generated multiple realistic images of women’s naked breasts.
Midjourney’s community guidelines note that “occasionally prompts will unintentionally produce non-PG-13 content.”
Users are asked to “please self-police these images” by immediately deleting them.
The Daily Dot attempted to reach Midjourney by email to ask whether it has taken steps to address the issue but did not receive a reply by press time.
Boucher is not the first person to be banned by Midjourney. In July of last year, a user, in a self-described effort to bring attention to AI-driven misinformation, produced a series of images showing politicians cheating on their spouses.
In response, Midjourney swiftly removed the user’s account for violating its policies. While the move by Midjourney is unsurprising given the violations, critics still argue that the company is not doing enough to address the underlying issues.
“I got banned from Midjourney AI for generating realistic images of politicians cheating on their wives for a series called ‘AI will revolutionize the blackmail industry,'” the user wrote on X at the time. “Midjourney et al have some safe guards in place, but you can get around them with creative prompting or repeated requests. These generators are mostly just mashing up photos available on the internet. Privacy, copyright, dignity, and safety be damned.”
The issue of AI-generated nudity has taken center stage in recent weeks after lewd images of pop star Taylor Swift received tens of millions of views on social media outlets such as X. In response, X, which fired many of its content moderators after being obtained by Elon Musk, vowed to hire 100 new employees to police the proliferation of similar imagery.
The Swift fiasco also spurred the European Union to strike a deal this week on a bill that would criminalize the non-consensual spread of certain nude content by 2027. U.S. politicians are likewise calling for new laws that would criminalize faked nude imagery of real people.
Sign up to receive the Daily Dot’s Internet Insider newsletter for urgent news from the frontline of online.
Source: https://www.dailydot.com/debug/midjourney-banning-researchers-exposing-ai-flaws/
