The many experimental applications of GPT-2, the most advanced text generator to date

May 30th

This post is sponsored by Brilliant, which offers courses like Programming with Python, a beautiful language that can be used for everything from video games to data visualization to machine learning.

Other courses, like Data Structures, give students the fundamental toolkit for the aspiring computer scientist or programmer. If you're a little farther along than that, try Introduction to Neural Networks, where you'll dissect the internal machinery of artificial neural nets through hands-on experimentation.

Sign up for Brilliant today to get 20% off an annual membership.

We’ve written a lot about GPT-2, a language model that generates convincing writing from user prompts. Originally trained on 40GB of plain text from the internet, GPT-2 was formally released in November, 2019 after OpenAI dragged its feet for months, warning that the text it generated was so lifelike that it was too dangerous to release.

In the months since then (and a bit before that, because smaller models of GPT-2 were made public in earlier 2019), GPT-2 has been at the center of a flurry of side projects, research papers and social experiments. Hobbyists and researchers have retooled the AI to make text games, chat bots, and examples of how bad actors could misuse the technology to create disinformation.

Recently, a new GPT-2 tool was released, this time for making up new, meaningful words that look real, but aren’t. Now is as good a time as any to take stock of what “the world’s most dangerous model” has actually made since it was released.

GPT-2 was announced in early 2019, and almost immediately someone used it to generate D&D character bios. Let’s start there:

D&D biographies: Janelle Shane, a researcher who had previously used language models to generate D&D character names, spells, metal bands and IPAs, used GPT-2 to try to create something more ambitious — player bios.

Janelle wrote the following shortly after GPT-2 was announced, back in early 2019:

The character’s name stays the same throughout, and the neural net even remembers that the ship was full of fish. The plots can get a bit repetitive, though, and at certain points during training, it seems to go through periods where it’s very repetitive indeed.

Irish folk music: One of the stranger applications was a music generator that started with folk and then went on to cover a handful of other genres. Created by Gwern Branwen.

Poetry: Branwen published GPT-2 written poetry after training the model on a large corpus of poems from Project Gutenberg.

“Poetry is a natural fit for machine generation because we don’t necessarily expect it to make sense or have standard syntax/grammar/vocabulary, and because it is often as much about the sound as the sense,” Branwen wrote. “Humans may find even mediocre poetry quite hard to write, but machines are indefatigable and can generate many samples to select from, so the final results can be pretty decent.”

Here’s a sample:

My heart, why come you here alone? The wild thing of my heart is grown To be a thing, Fairy, and wild, and fair, and whole

Chess: Yes, chess. Branwen’s collaborator, Shawn Presser, trained GPT-2 to play chess the same way Branwen trained it to compose music. Despite the fact that GPT-2 is a text-based model, it’s possible to transcribe text it writes into musical notes or chess moves.

“I wondered if he could train it on a corpus of chess games written in standard notation (where, for example, e2e4 means ‘move the pawn at square e2 to square e4’).” Scott Alexander wrote in Slate Star Codex. “There are literally millions of games written up like this. GPT-2 would learn to predict the next string of text, which would correspond to the next move in the chess game.”

Back in January of 2020, Presser wrote that his trained model made it to midgame before it began losing track of game context.

Play GPT-2 yourself here.

r/SubSimulatorGPT2: This is a subreddit where all posts and replies are autogenerated by GPT-2 bots. There are no humans there. There are, however, humans watching and commenting on r/SubSimulatorGPT2’s synthetic conversations at another subreddit, r/SubSimulatorGPT2Meta.

GPT Adventure: We’ve written about this before — it’s a text-based adventure game, where players can explore infinite worlds by telling GPT-2 what they’re doing, and then letting GPT-2 explain how the game world reacts.

AI Dungeon: A precursor to GPT Adventure, AI Dungeon was trained on old school text-based games from the 1970s. Players explore underground worlds and fight monsters. Possibly the newest experiment with GPT-2, TWDNE creates words from the GPT-2 model. Below are a few words it made for me:

  • Dealinize: Reduce the density of (a physical substance); purify
  • Anaxylaxis: Inflammation after successful vaccination against an alveolar disease antigen such as Leishmania
  • Fetchboard: A board that loads an amount of material

And there you have it. To leave you with something to think over, I want to link to another GPT-2 piece we published late in 2019, featuring an interview with a prominent computer scientist who tried to put GPT-2’s accomplishments into perspective. If human-like text can be written by a single-purpose bot, trained on a sea of Reddit comments, then what does that say about writing itself?

We used to think you had to be very clever to be good at chess, and if you could play chess then you were a "real" intelligence. You had to be. And then we realized that playing chess at a superhuman level doesn't require something that we would call intelligence, and it's slightly unsettling to find that, like, writing coherent and plausible news prose apparently also doesn't require general intelligence. Like, if you just learn the statistical relationships between words but do that really really well, that seems to be enough.