The Met x Claude Code Hackathon
Using AI to augment the Asian Exhibit at The Metropolitan Museum of Art
My intro to The Met x Claude Code Hackathon was not typical. It started with the lead curator (whose exhibit we were about to augment with AI) opening with his belief that AI did not belong in museums. I was surprised and it took a second to process, but I realized I should have expected this reaction. Why? Because AI is extremely divisive for a lot of reasons. And my weekend at the Hackathon turned into an extended “My Dinner with Andre” series of conversations culminating with 8 hours of AI hacking that brought about a key epiphany in how I think about AI.
The Prompt: use Claude Code to augment the Metropolitan Museum’s Asian exhibit and create deeper engagement with the work.
The hackathon started with a tour of the Asian exhibit so we could absorb the context and current exhibitions. We noticed there was very little context and connectivity that we could understand in each of the rooms, especially for the niche and small collections like the Korean exhibit. While we didn’t focus on this exhibit, we noted that small details in design could add significant value, like understanding the calligraphy associated with the works.
Our idea: Recreate the arm and strokes of calligraphy masters based on analysis from pen ink pressure maps, pixel density, angle and curvature. You can derive a crazy amount of data especially from super high-res images. For each calligrapher we hoped to recreate a rendition of a Duchamp-like stop motion portrait to a character caught in motion. We wanted to portray the single character embodied as a series of dance moves, like a ballet, overlapped on each other.
We were quickly thwarted by context rot and hallucinations. Our idea may have been possible with extensive calligraphy model training and a large specific dataset. This is where complex ideas may be applied with the help of human experts, but only they would have answers to questions like these, despite all this data!
We ended up with a skeleton animation that was strangely tiny, as the canvas size was very small and it chose a 2D form with a static camera angle in bright RGB colors. A far cry from our intention but a genuinely interesting study in cyborg anthropology. How did it arrive at this? I clearly did not supply enough context. It used the original data analysis we ran and recreated it in a beautiful and mesmerizing way, an infinite loop of curiosities. A definite win in the space of experimenting and prototyping, but not a final product.
Things really went awry when we realized the analysis of stroke patterns we were using was partly or entirely hallucination. It invented calligraphy strokes and patterns and applied them as data interpretations. It produced skeleton animations of brush movements that looked vaguely right but were completely fabricated based on what we knew were verifiable facts. Claude also started grabbing characters from works in the archive that were from the painted imagery and not a single character.
We’d ask Claude to analyze 8 strokes per character (known number of strokes) and it would then start producing 23 strokes. We’d correct it. It would adjust the model to fit our constraint. The forced manipulations to meet our goals bred more hallucinations and each iteration drifted further from the actual work.
At some point we pivoted to the safer idea of a data visualization showing the vast networks of connections between works that shared similar characters and meanings. We had Claude map the relationships across the entire archive of 34k objects. It worked quickly and exhaustively, a genuinely impressive endeavor (again veracity is questionable). But it lost what we set out to do: create a physical embodiment of technique, to build a dance out of an ancient character. To tell a story of an artist and their expression.
We ended up with many outputs: stroke pressure maps, skeleton arm animations, a final data viz. In the end, we were happy with the interesting artifacts and great experiments, even if we didn’t end up with something production-ready.
We weren’t the only ones that ran into this. People joked that they were just there to repetitively hit enter as Claude suggested next steps, needing that constant manual affirmation. Laughing that after all the enters, they had no idea what they had made. Others celebrated the 80% prototype and moved on. It was a hackathon after all.
The Winners
The first place project was playful, clever and fun. The user identifies an instrument from a painting, could even be ancient and obscure, and the Claude API based app will recreate it for you to play yourself. I can’t say that it was accurate as most of the sounds were very digital sounding and most certainly walked a questionable cultural line of representation that would make it unlikely to ever be used. I could also see the curator cringing at this interpretation of classic works. But as a hackathon thought experiment, it was great. It wasn’t trying to take itself too seriously. It was just a fun thing that existed alongside the work, honest about its own playfulness.
Another project involved using an Arduino driven camera and screen to find fish in paintings, identify them and print a recipe. It was called Carp Diem, a definitely silly and weird interpretation. Again, a wonderfully fun prototype but not actually trying to enhance the work or play a serious role in the exhibit.
Maybe this is a good place for AI to live in museums? In the playful, weird and silly?
The personal realization
The realization wasn’t in anything that was made, it was the conversations. Between people who’d never used AI and people who’d spent years in it. Between the curator’s skepticism and the institutional push to innovate. Between what we thought we could build and what we actually built.
Anthropic provided swag when we arrived- two mini custom printed exerpts from provious online publications and a pin of Claude wearing a Cowboy had. The two books were called “Model Organisms of Misalignment” and “Machines of Loving Grace.” The first explores scary versions where AI takes over to protect a goal, lies, hides and breaks the original rulesets to do so. The second is about its power to reshape society for the better. They’re both a bit hypey. You can read the original texts online and there are plenty of criticisms of the way they set up the misalignment. The books also don’t cover how the models are actually trained, the bias baked in, the privacy implications or the labor costs.
In the end I kept coming back to the curator. What would he say about these works and what conversation would he start?
The real takeaway is not about what you build or how efficient it is. It’s about the conversations that define whether we can actually understand this technology before we deploy it. This is a start.
AI Citation: I used Claude to edit for typos and flag anything that could be improved. All the writing is my own.
