In a first, OpenAI removes influence operations tied to Russia, China and Israel

Caption

OpenAI, the company behind generative artificial intelligence tools such as ChatGPT, announced Thursday that it had taken down influence operations tied to Russia, China and Iran. / AFP via Getty Images

Online influence operations based in Russia, China, Iran, and Israel are using artificial intelligence in their efforts to manipulate the public, according to a new report from OpenAI.

Bad actors have used OpenAI’s tools, which include ChatGPT, to generate social media comments in multiple languages, make up names and bios for fake accounts, create cartoons and other images, and debug code.

OpenAI’s report is the first of its kind from the company, which has swiftly become one of the leading players in AI. ChatGPT has gained more than 100 million users since its public launch in November 2022.

But even though AI tools have helped the people behind influence operations produce more content, make fewer errors, and create the appearance of engagement with their posts, OpenAI says the operations it found didn’t gain significant traction with real people or reach large audiences. In some cases, the little authentic engagement their posts got was from users calling them out as fake.

“These operations may be using new technology, but they're still struggling with the old problem of how to get people to fall for it,” said Ben Nimmo, principal investigator on OpenAI’s intelligence and investigations team.

That echoes Facebook owner Meta’s quarterly threat report published on Wednesday. Meta's report said several of the covert operations it recently took down used AI to generate images, video, and text, but that the use of the cutting-edge technology hasn’t affected the company’s ability to disrupt efforts to manipulate people.

The boom in generative artificial intelligence, which can quickly and easily produce realistic audio, video, images and text, is creating new avenues for fraud, scams and manipulation. In particular, the potential for AI fakes to disrupt elections is fueling fears as billions of people around the world head to the polls this year, including in the U.S., India, and the European Union.

In the past three months, OpenAI banned accounts linked to five covert influence operations, which it defines as “attempt[s] to manipulate public opinion or influence political outcomes without revealing the true identity or intentions of the actors behind them.”

That includes two operations well known to social media companies and researchers: Russia’s Doppelganger and a sprawling Chinese network dubbed Spamouflage.

Doppelganger, which has been linked to the Kremlin by the U.S. Treasury Department, is known for spoofing legitimate news websites to undermine support for Ukraine. Spamouflage operates across a wide range of social media platforms and internet forums, pushing pro-China messages and attacking critics of Beijing. Last year, Facebook owner Meta said Spamouflage is the largest covert influence operation it's ever disrupted and linked it to Chinese law enforcement.

Both Doppelganger and Spamouflage used OpenAI tools to generate comments in multiple languages that were posted across social media sites. The Russian network also used AI to translate articles from Russian into English and French and to turn website articles into Facebook posts.

The Spamouflage accounts used AI to debug code for a website targeting Chinese dissidents, to analyze social media posts, and to research news and current events. Some posts from fake Spamouflage accounts only received replies from other fake accounts in the same network.

Another previously unreported Russian network banned by OpenAI focused its efforts on spamming the messaging app Telegram. It used OpenAI tools to debug code for a program that automatically posted on Telegram, and used AI to generate the comments its accounts posted on the app. Like Doppelganger, the operation's efforts were broadly aimed at undermining support for Ukraine, via posts that weighed in on politics in the U.S. and Moldova.

Another campaign that both OpenAI and Meta said they disrupted in recent months traced back to a political marketing firm in Tel Aviv called Stoic. Fake accounts posed as Jewish students, African-Americans, and concerned citizens. They posted about the war in Gaza, praised Israel’s military, and criticized college antisemitism and the U.N. relief agency for Palestinian refugees in the Gaza Strip, according to Meta. The posts were aimed at audiences in the U.S., Canada, and Israel. Meta banned Stoic from its platforms and sent the company a cease and desist letter.

OpenAI said the Israeli operation used AI to generate and edit articles and comments posted across Instagram, Facebook, and X, as well as to create fictitious personas and bios for fake accounts. It also found some activity from the network targeting elections in India.

None of the operations OpenAI disrupted only used AI-generated content. “This wasn't a case of giving up on human generation and shifting to AI, but of mixing the two,” Nimmo said.

He said that while AI does offer threat actors some benefits, including boosting the volume of what they can produce and improving translations across languages, it doesn’t help them overcome the main challenge of distribution.

“You can generate the content, but if you don't have the distribution systems to land it in front of people in a way that seems credible, then you're going to struggle getting it across,” Nimmo said. “And really what we're seeing here is that dynamic playing out.”

But companies like OpenAI must stay vigilant, he added. "This is not the time for complacency. History shows that influence operations which spent years failing to get anywhere can suddenly break out if nobody's looking for them."