Inside the ?Black Box? of a Neural Network

On Wednesday, Carter’s team released a paper that offers a peek inside, showing how a neural network builds and arranges visual concepts.This particular line of research dates back to 2015, when Carter’s coauthor, Chris Olah, helped design Deep Dream, a program that tried to interpret neural networks by reverse-engineering them. If you were to squint a bit, you might see rows of white teeth and gums—or, perhaps, the seams of a baseball.As they browsed the images associated with whales and sharks, the researchers noticed that one image—perhaps of a shark's jaws—had the qualities of a baseball.It turns out the neural network they studied also has a gift for such visual metaphors, which can be wielded as a cheap trick to fool the system. By manipulating the fin photo—say, throwing in a postage stamp image of a baseball in one corner—Carter and Olah found you could easily convince the neural network that a whale was, in fact, a shark.By inserting a postage-stamp image of a baseball, they found they could confuse the neural network into thinking a whale was a shark.It’s true, Olah says, that the method is unlikely to be wielded by human saboteurs; there are easier and more subtle ways of causing such mayhem.