Synesthesia
A creative system that reads atmosphere, rhythm, and structure in one medium and constructs another, then lets a second person challenge that reading.
- Type
- Web App
- Role
- Creative Technologist
- Built
- -
- Updated
- -
- Source
- GitHub
Tech stack
- Next.js
- React
- TypeScript
- Tailwind CSS
- AI
- Generative media
- Supabase
What if a photograph could become a song, and that song could become a story?
Photo to music to language to image.
One medium per app
- Photographers get image tools.
- Musicians get audio tools.
- Writers get a page.
Each of those tools is good at its own file. None of them can hold what the file is doing - the pace, the color, the pressure - and rebuild it as something else.
Synesthesia is the translator in the gap. A photograph is read, then heard. The distance between the two is the piece.
Meaning, then a medium
A source is read for atmosphere, rhythm, color, composition, intensity, and structure. Those qualities are reconstructed in another medium.
The intermediate layer is the product. Without it, a photograph would only be stretched into a waveform.
type Reading = { atmosphere: string rhythm: string color: string structure: string intensity: number} type Transformation = { sourceId: string target: 'image' | 'audio' | 'text' reading: Reading outputId: string}Chains keep the lineage
One output can become the next input: photo, music, text, image, music again. Each step is a new interpretation, and the history stays attached to the artifact.
Photo → Music → Text → Image → Music
Two readings, one source
Two people receive the same source and make competing interpretations. The comparison is part of the work, not a scoreboard beside it.
Original
├── Person A → Interpretation A
└── Person B → Interpretation B
└── ComparisonThe hard part
There is no shared file format for a feeling
Image, sound, and language do not share a file format that preserves atmosphere. A direct conversion throws the meaning away and keeps the container.
Solution
The pipeline is media, then a semantic reading, then generation in the target medium. The reading is stored with the output, so a later step can argue with it.
Trade-offs
| Decision | Choice | Why |
|---|---|---|
| Fidelity | A visible reading | A perfect translation can be dull. The distance should be seeable. |
| Control | Some surprise | Too much control removes the interpretation. |
| Repeatability | The same source can return | And it can also surprise. |
What I’d change
- Longer chains, and rooms where more than one person works the same source.
- A control for how far the interpretation may travel.
- More than one model on the same reading, so the difference is the point.
About the translation
Can I drop in a JPEG and get a WAV?
You can start from a photograph and end with a song. What happens in between is a reading - atmosphere, rhythm, color - and the song is built from that, not from the pixels stretched into a waveform.
How long can a chain go?
Photo, then music, then text, then another image. Each step is a new interpretation, and the earlier ones stay attached so you can see where the piece drifted.
What happens when two people start from the same photograph?
They get two readings and two outputs. The comparison is the work: same source, different ear. There is no winner.