Joe McCraw
The collaboration
In 2019, Nesta’s Collective Intelligence Grants program selected Dovetail Labs to investigate public perceptions and potential risks in emotion-recognition systems. The work was interdisciplinary: research design, browser-based machine vision, user interaction, data analysis, and public engagement.
The research question
How do people experience emotion-recognition AI? Do they notice when the system fails? Do they understand the limits? Can a participatory experience reveal biases that don’t show up in laboratory settings?
The product: Emojify
Emojify was an interactive online experience that let visitors play with emotion-recognition models in real time. The interface was simple: show your face, see what emotion the model classified. But the games underneath were the research, teaching visitors how emotion-recognition engines actually perform against real, unscripted faces—and how far that performance sits from what people expect going in.
- Beat the Machine: Display each of the six basic emotions on request—happiness, sadness, fear, surprise, disgust, anger—and see whether the live classifier agrees with you, complete with its confidence score.
- Wink/Blink game: Can you fool the emotion classifier with facial expressions that resemble emotions you’re not feeling?
- Fake Smile challenge: What does the model think is happening when you exaggerate your expression?
- Observation prompts: “What did you notice about mistakes? When did the system surprise you?”
The experience collected thousands of interactions and observations, not through a survey, but through direct, embodied engagement with the technology’s actual behavior.
Results
- 15,207 unique visitors during the four-week experiment
- 40,000+ unique hits across the full program
- Participants documented moments of recognition, confusion, and discovery about how machine vision systems classify emotion
- The Internet Archive preserved the original experience at emojify.info/menu
My role
Technical advisor and contributor. I helped define technical requirements, prototyped research approaches, contributed to the Firebase deployment and React implementation, and worked on the browser-based computer vision integration.
This work connected prototype evaluation, browser-based machine vision, privacy-aware camera interaction, and the translation of research questions into an accessible, playful experience.
Status
The project is archived. The research outcomes remain relevant to discussions of AI bias, transparency, and public trust. The live experience is preserved in the Internet Archive and serves as a historical record of participatory AI research.