The exhibition began, like the fall of mankind did, with an apple. Or rather, it began with a glut of photographs of apples: red apples, green apples, yellow apples, sliced apples, bushels of apples, an apple with the word “Google” carved into it. These images were pinned to a wall and clustered around a label featuring a single, simple noun: “apple.”
This was From ‘Apple’ to ‘Anomaly,’ on display at the Barbican Centre in London in 2020. For the project, the American artist Trevor Paglen culled around thirty thousand images from the online database ImageNet, printed out small-scale versions, and pinned them to a lengthy curved wall, categorized by noun: “soil,” “valley,” “syringe,” “pizza,” “mascot.” The result was breathtakingly expansive: seen from a distance of a few feet, it resembled a shimmering mirage of animals, plants, minerals, and people.
The near-infinite visual library was possible because ImageNet is one of the largest publicly available libraries of images. It is also the pioneering dataset for much of the world’s image-recognition technology. Everything from Facebook’s self-tagging feature to self-driving cars to drones is trained to see the world using massive datasets like ImageNet. It was amassed over the course of nearly a decade, largely by workers who, for pennies, matched images to associated words on the task website Amazon Mechanical Turk. The result was a dataset of more than fourteen million images, cataloged and labeled so machines could learn to match an image of a rose to the word “rose.” Paglen made visible this wild, freewheeling taxonomy of everything under the sun, including the sun itself.
In 1668, an English clergyman and natural philosopher named John Wilkins unveiled a similarly vast project: he renamed the world. In his masterwork, “An Essay towards a Real Character, and a Philosophical Language,” which ran to almost seven hundred pages, Wilkins laid out a template for a universal language, one that would be so perfect in its powers of expression that it would bring humans closer to God and “not signifie words, but things and notions.”
Wilkins was writing against the backdrop of the English Civil War and the loss of Latin as a common Christian language. He was also grappling with an enduring human problem—the gap between words and their meanings. In an essay about Wilkins and his language, Jorge Luis Borges wrote, “Apart from the composed words and the derivations, all the languages in the world… are equally inexpressive.” Words bear no inherent relation, after all, to the things they name. Or, as the poet Robert Hass later wrote, “All the new thinking is about loss / In this it resembles all the old thinking” in its insistence on the notion that “because there is in this world no one thing / to which the bramble of blackberry corresponds, / a word is elegy to what
it signifies.”
Wilkins’s solution to this problem seemed fairly straightforward. First, he would separate the world into forty categories, or genuses. Then he would subdivide these categories into “differences,” and from there, divide everything further into “species.” Each genus, difference, and species would be assigned a monosyllable, which would then be strung together to form new, more perfect words. A curious young reader could then sound out not only a word, but also something of its meaning.
So it fell to Wilkins to break the world into its component parts. In the late seventeenth century, systems of taxonomy and classification were flourishing in the natural sciences, particularly at the Royal Society, of which Wilkins was a founding member. There was still not much established order of things. When it came to animals, for instance, most early ecologists used the divisions provided by Genesis—water, air, and land. Warm-blooded and cold-blooded animals were often separated. Beyond that, animals could be arranged by size, rarity, symbolic meaning, or simple alphabetical order. (Neither did everyone agree about which animals needed to be classified: early encyclopedias were still divided over whether griffins and centaurs were found in nature; Wilkins thought not.)
Wilkins’s forty genuses of things and notions range from “Transcendental” to “Elements” to “Space.” Underneath these umbrella categories is a set of “differences” and “species” that get increasingly specific, and which he charts in the next part of the essay. Under animals, we come to the section “Of fish,” in which Wilkins divides fish into the categories “viviparous” and “oviparous.” Then he denotes “viviparous oblong fish,” which are subdivided further by characteristics like “rows of very sharp teeth” (sharke, glaucus) and “thorns on their backs” (thornback dog, hog-fish). There are also viviparous cartilaginous fish, oviparous fish whose back fins are soft and flexible, oviparous fish having two fins on the back, oviparous fish having one fin on the back, oviparous fish of an oblong figure, fishes of a hard crustaceous skin, squamous river fishes, and so on. Wilkins’s named world unspools, paradoxically, toward organization and classification.
Taxonomy is essential to image-recognition technology; machines must break the world into human concepts and categories in order to perform any kind of “recognition.” Algorithms must be taught quite literally to tell apples from oranges (and green apples from red apples, and mandarins from tangerines). But this requires answering a deceptively simple question: What makes an apple an apple?
In From ‘Apple to ‘Anomaly’, Paglen acknowledged this predicament with a wink to the French surrealist René Magritte. Near the entrance to the exhibition, Paglen included a reproduction of Magritte’s Ceci n’est pas une pomme, a painting of an apple beneath the phrase “This is not an apple.” Magritte was toying with the distinction between an object and its representation, the distance between a painted apple and an edible Granny Smith. In Paglen’s updated version, Magritte’s painting was overlaid with an algorithmic tag that identified it: “red and green apple.” The machine had declared, with no ambiguity, that “ceci est une pomme,” after all.
Philosophical questions about representation and reality get flattened by algorithms. The ambition of ImageNet is to make all things knowable by splitting them into groups. This leads to some absurdity, which Paglen illustrates. From “apple,” we move to “apple tree” and “apple orchard,” which are simple enough. Then there are “valley” and “soil,” the general realm of the pastoral. And then there is “laborer.” Here we see men at work, bent over in rice paddies and irrigated fields. Notably, almost all are Black or brown. Soon, traveling along the curved gallery wall, we arrive at “investor,” a group composed almost entirely of white men in suits and ties, pointing at whiteboards or hunched over laptops. We begin, perhaps, to see the cracks in the foundation of this labeled visual universe. We recognize stereotypes about race, class, and gender, as spat back to us by machines.
As we continued through the gallery, the nouns Paglen had selected trended increasingly toward the abstract, and the labeling became increasingly chaotic. The word “segregator” was attached to images of both Barack Obama and George W. Bush. Photos associated with “wine drinker” and “alcoholic” bled together on the wall but were subtly different: a man sniffing wine was a wine drinker, while a woman with a large margarita with a salted rim was an alcoholic. Paglen visualized the consequences of a taxonomy based on collective human biases, which are then replicated exponentially by machines. He also showed how difficult it is for machines to make sense of any kind of abstraction, and for us to try to teach them what a “segregator” would look like. We begin to wonder: Is it possible to label the universe like this, after all? And even if it were possible, would we want that?
In one of the most poignant parts of his essay, amid his classification of animals, Wilkins briefly digresses. “He that looks upon the Starrs, as they are confusedly scattered up and down in the Firmament, will think them to be (as they are sometimes stiled) innumerable, of so vast a multitude, as not to be determined to any set number: but when all these Starrs are distinctly reduced into particular constellations, and described by their several places, magnitudes and names, it appears, that of those that are visible to the naked eye, there are but few more then [sic] a thousand in the whole Firmament.” The same is true, he argues, of many other things, including animals; there are fewer than we might imagine, and it is possible to number and name them all.
Wilkins then sets out to rebuff the doubters—specifically, the atheists who have argued that all the world’s animals could not possibly have fit into Noah’s ark, given the dimensions described in Genesis (three hundred cubits in length, fifty in breadth, and thirty in height). In the ensuing passage “A Digression Concerning Noah’s Ark,” Wilkins aims to prove the Bible right. In a table and a diagram, he denotes the number of each kind of animal on the ark, their proportions, the size of their stall, and their food requirements (hay for the herbivores; sheep for the carnivores). In a series of calculations—including a careful consideration of shipbuilding practice—he determines that there would have been ample space. “In this enumeration I do not mention the Mule, because ‘tis a mungrel production, and not to be rekoned as a distinct species,” he writes. “As for the Morse, Seale, Turtle, or Sea-Tortoise, Crocodile, Senembi, [sic] These are usually described to be such kind of Animals as can abide in the water, and therefore I have not taken them into the Ark, tho if that were necessary, there would be room enough for them.” Rodents would not be assigned specific stalls, but there would certainly be space for them underfoot. Thus, the ark was literally possible, the atheists were wrong, and the Bible was true.
It becomes clear, reading this section, that Wilkins has verged into a mode of rationalism so extreme as to become absurd. It may be technically possible to squeeze the animal kingdom into an ark of specific proportions. But, paradoxically, it feels as if this mathematical endeavor takes us away from the bright candle of belief, religious or otherwise, that animates the mysteries of the universe. In insisting that it is possible to count and name the stars, Wilkins has achieved a certain proximity to them, but he is in another sense farther than ever from truly perceiving them as stars—infinite, boundless, and beyond our reach.
There is quite a bit of beauty in From ‘Apple’ to ‘Anomaly.’ The side-by-side repetition of photos of fried eggs and sunsets has surprising visual resonance. Even the glitches can be serendipitous, as when a cheetah is mistakenly classified as a “honeycomb.” The project of ImageNet is awe-inspiring in its dimensions; after all, we are in the presence of a contemporary attempt to make a catalog of the physical world, the vast and expansive fields of human and nonhuman experience. But Paglen shows us how this attempt devolves, and we see not just the troubling and darker aspects of bias in machine learning, but also a more foundational absurdity—the idea that we can break the universe down into clearly defined categories in the first place.
Wilkins’s project represents an earlier failure, a more individual one, to get closer to God through classification. And yet it is also breathtaking to leaf through his tables and lists. Wilkins’s wrongheadedness is beautiful because it so clearly illustrates something fundamentally human: the quest to make everything knowable and legible, and to give it a name.
“Obviously there is no classification of the universe that isn’t arbitrary and subjective; the reason is simple: we do not know what the universe is,” Borges writes. We do know, however, watching Wilkins flail, something about the nature of being a person who perceives the universe—how badly we want to know what is, and to decode what Borges calls “God’s secret dictionary.” Reading Wilkins, we can also recognize how much he did with the language he already had at his command. He conjures for us not only a fish, but a viviparous oblong fish with rows of very sharp teeth (sharke).
One of God’s first instructions to Adam in Genesis was to name the animals. One of the first things we learn to do is name things. As we come out of the disorientation of babyhood and into the clearer fog of toddlerhood, we learn to say: “This is a flower”; “This is a dog”; “This is a tall girl.” The act of putting language to our experience of the world we encounter, thereby dividing it into parts and categories, becomes our basis for understanding it. Looking out the window at blooming white flowers on a tree on my street, my first though is: Are those dogwoods or magnolias? I ask because I am moved, on a cold spring day in the northeastern corner of the United States, by these blossoming flowers. I sense that I would be somehow closer to them if I knew their names.
And yet. This desire to know and name can quickly become perverse, an act that has to do less with love and more with a kind of conquest. This is especially true in an age when I can easily download an app that will label and tag these flowers with their Latin names. (One such app, Pl@ntNet, was partly trained using ImageNet.) I could demystify the world beyond Wilkins’s wildest dreams, and I would get no closer to God or to anything sacred. What I have learned from Wilkins, inadvertently, is to stand still and marvel at a universe that is impossible to contain within categories, and that is ultimately numberless and nameless.
