👆 January 3, 2025 – Practicing darkfield microscopy as the Nathaniel B. Palmer moves up and down through the Southern Ocean waves …
We came to look at phytoplankton.
Along with other members of the ship’s science team, I pumped thousands of samples of seawater through a digital microscope, watching Rhizoselenia and Chaetoceros fly by on a small screen, the machine outlining their edges the way a traffic cam might outline passing cars. In our moving lab, working to the sounds of crunching ice and sometimes humming along to Bob Dylan, we were on a kind of plankton surveillance duty. It was absorbing work. I loved every moment of it.
As an artist aboard, I was collecting other data as well: photographs of what we saw from the decks and through the portholes, waves and albatross and migrating whales and the icebergs that sailed with us like floating cathedrals. I recorded the creaking sounds of the ice tower, that tallest point of the ship, and then the joyful laughter of the mess hall deep in the hull. I dropped hydrophones into the sea and into the sink, listening to the tones and sighs of our boat. I harvested the ship’s data streams, piping a feed of the ship’s CCTV cameras and streams of updating information (what are our coordinates? what is the wind speed, air temperature?) straight to my computer via code. I recorded myself on the CCTV. I recorded myself with my phone too, in the library and in the ship’s little gym, trying to play with the unique feeling of the ship’s gravity as we rode the waves, like a kid trying to jump right before the elevator starts to change direction. If I jumped at the precise moment of the crest of a wave before it began to pitch down again, it felt like flying.
I returned home with this massive multimedia archive of our collective experience at sea.
What kind of information is in a photograph? What sort of memory is stored there? We take our camera and expose its rows of tiny sensors to the world, and we encode changes of color and brightness, and that recording becomes the scene. A photograph might be a form of data visualization. Poring over my photos back at home in New York, I realized that I had built up a dataset too: thousands of sequences of bursts of motion, Southern Ocean waves and birds in flight. These series of action shots reminded me of Edward Muybridge’s 19th century grids of movement. In the early days of photography, Muybridge was someone in between an artist making images and a scientist with a fancy new instrument. His curiosity and his desire to document all kinds of animals in motion eventually led us to cinema, as he pushed his camera to take images closer and closer in time. Incidentally, he was also probably one of the first people to take photographs from the deck of a moving ship.
Unlike Muybridge, I had AI-assisted focus tracking built into in my camera when spending time with seabirds in Antarctica. The phones in our pockets have this feature today too, whether we want it or not. I think a lot about the promises and pitfalls of machine learning algorithms – they aim to find patterns that are difficult for humans to see, and they offer the tantalizing possibility of predictions for us based on those patterns. I think too about ways to work with code and my archives. Could I write a program that attempts to surface information in between the frames of motion, the missing moments in time? How would a machine see patterns in my precious archive of memories from our expedition to Antarctica?
I wrote a program that calculates optical flow, feeding the program pairs of images from my series of wave actions. Optical flow gives you a field of vectors, one for every pixel in your photograph, showing where the machine thinks your pixels are heading based on the patterns of brightness between your two images. I then pushed each image’s pixels along those vector lines, challenging: “if your assumptions are correct, let’s see what you think the next moment in time looks like.” This gives an interpolated guess-image, some speculation in the archive. Then I layer this result on top of the difference between that interpolation and the next frame, the one that came from my camera, from light in Antarctica. The resulting image is a buzzing, almost-abstract artifact of the gap between prediction and reality. This dense stacking of moments and possibilities unfolded in the sea foam in a fraction of a second when I was witnessing it in real-time. I saw it, without seeing it, too.
This kind of image processing borrows from current techniques for prepping datasets to train AI models, the algorithms that are starting to seep into our daily lives. It’s also deeply related to the post-processing that the NBP cruise science archive will go through. The sheer amount of plankton images we collected would take humans many years to look through – and so, our hard-won images will be fed to an AI tool, hoping that the model will be able to identify species based on patterns in their shapes, ultimately getting an estimate of how many kilograms of shimmering diatom biomass we sailed through and clues as to why they didn’t happen to bloom in the 2024-2025 austral summer.
Each cycle of training any AI model aims to minimize something called a loss function. In our plankton example, if the model identifies the species correctly, the program gets rewarded by staying the same. If the model “fails,” the underlying math parameters get tweaked, changing in order to get closer to some “truth,” aiming for better luck with the next images as cycle repeats. Under the hood, your computer is trying to avoid failure, and avoid loss.
An archive also tries to minimize loss. We all create our datasets now, grids of scrollable memories. We take our archives and hold onto them, organize them in some way to mark that we were there, we witnessed this moment in time, whether it was an impossibly beautiful polar sunrise in the field or a gathering around the table at home. But there is always loss too, always something we cannot capture in memory, always a further slice to find between two moments in time, the same way there is an infinite number of points between two lines. All our methods are imperfect in grasping and portraying the richness of the world, of the feeling of experience and environment, as either scientists or artists.
There is so much density of information, at all scales, inside any living moment.
