OpenAI and Google release "activation atlases" for visualizing machine learning's internal thought process


OpenAI and Google have released a visualization tool they call activation atlases, which visualize the internal interactions among neural networks' individual neurons.

The results are map-like, psychedelic visualizations that show how a neural net can differentiate between different patterns, animal species, or similar objects.

OpenAI says the tool helps shed light into machine learning's unintelligible decision making.

OpenAI also released a demo that allows users to explore atlases and discover how a neural network differentiates between a wok and a regular frying pan, for example.


From OpenAI's blog post:

For example, a special kind of activation atlas can be created to show how a network tells apart frying pans and woks. Many of the things we see are what one expects. Frying pans are more squarish, while woks are rounder and deeper. But it also seems like the model has learned that frying pans and woks can also be distinguished by food around them — in particular, wok is supported by the presence of noodles. Adding noodles to the corner of the image will fool the model 45% of the time!

Related people

Machine learning's black box problem



August 11th

OpenAI was founded as a non-profit research lab by Elon Musk and Sam Altman in 2015.

In February, 2018, Musk left, citing a conflict of interest with his work on Tesla's autopilot system.

In 2019, with Altman in charge, OpenAI formed OpenAI LP, a for-profit company it wrote will allow them "to rapidly increase our investments in compute and talent while including checks and balances to actualize our mission."

OpenAI has produced some impressive accomplishments-- in early 2019, its neural networks beat the world's best Dota 2 players. And in July, Microsoft invested $1 billion in OpenAI to pursue artificial general intelligence, an accomplishment many think is atill decades away, if not longer.