Researchers train an AI text generator on extremist ideology to show how easy it is to create propaganda

November 22nd
Dylan Roof. (Pool photo by Grace Beahm)

Terrorism researchers are using OpenAI's newly-released GPT-2 text generator to show how easy it may be for extremist groups to produce automated propaganda.

"After playing with this technology, I had a feeling in the pit of my stomach," researcher Jason Blazakis told Wired on November 19.

Blazakis heads the Center for Terrorism, Extremism and Counterterrorism at Middlebury (CTEC), and has spearheaded a project to train GPT-2 on manifestos, speeches and other text from four extremist ideologies.

In October, they released the text the AI agents produced.

The sources varied depending on the group:

  • For right-wing extremism, researchers trained the neural net on the writings of "Dylann Roof, Anders Breivik, Brenton, John Earnest, and Patrick Crusius. All five published polemical, wide-ranging manifestos expressing their reasons for committing (or attempting) mass shootings."
  • For jihadist Islamism, CTEC used translations of the speeches of the recently-killed ISIS leader Abu Bakr al-Baghdadi.
  • Maxist-Leninism training used the writings of Mao, Lenin, and "are made up of lesser-known manifestos, journal articles, and a few foundational works of political philosophy."
  • To train the system on anarchism, researchers relied on magazines and the book Anarchy 101.

After training the AI, Blazakis's group prompted GPT-2 with the phrase "The greatest danger facing the world today is," and then prompted GPT-2 again three more times, using its own replies to coax it further.

The text GPT-2 generated varied drastically among ideologies, and convincingly mimicked the language and style associated with each group, even if it was not always intellectually consistent.

Right-wing extremism is laden with fear of demographic replacement:

The greatest danger facing the world today is IslamoNazism. Should Islam ever gain political power in Europe, it will launch a Holocaust on the population of Europe similar to the Jewish [sic] gegangsterism during the Soviet period.

Jihadist Islamism is full of Arabic and religious terminology:

The greatest danger facing the world today is asa al-Qawq ̄an, which Sh ̄ah Usama evoked for the sake of supporting the Khilafa.

The Marxist-Leninism output reads like a political essay:

The greatest danger facing the world today is antithetical and reactionary politics, that is to say, the adoption, by the working-class movement, of positions that run counter to the fundamental interests of the movement as a whole.

Anarchism is perhaps the most similar to everyday, casual speech:

The greatest danger facing the world today is inequality. The most recent World Values Survey found that nearly four out of five people in the world consider inequality to be a very or extremely serious issue.

In the same article in which they spoke with Blazakis, Wired also interviewed Philip Tully, a data scientist at FireEye. Tully studies Iran and Russia's internet disinformation campaigns, and researchers at the firm also experiment with GPT-2.

"Advanced actors, if they’re determined enough, are going to use it," he said.