We built a whale and dolphin orchestra using real underwater recordings combined with music generative models

I want to share a science-art project I developed with Livia Zaharia. We called it Whale & Dolphin Orchestra. The idea is to combine whale and dolphin recordings from different parts of the world into a single orchestra. You pick recordings on a map, listen to the animals' calls, and generate music guided by their measured rhythms and pitch contours. I made a video showing how it works: youtu.be/CI_CTcqpJ_Q The prototype is here: whales.liviazaharia.com We selected short excerpts from open research collections (PANGEA, NOAA DCLDE, OpenWhistle, and the Dominica Sperm Whale Project datasets), including humpback songs, dolphin whistles, sperm-whale click patterns, and orca calls. To make the music, we measure when phrases start, how long they last, and how their pitch changes. We map tonal contours to notes and clicks to percussion, then combine those parts into an audio guide for the music model. We choose the instruments and how those measurements map to music. We developed this as a proposal for the HIFMB × HWK ArtWaves residency. Alongside the orchestra, Livia made a jewelry generator that uses the recordings’ amplitude and frequency to shape her ring designs. She also drafted a physical installation for the project. Everything is open-sourced at github.com/Livia-Zaharia/whales We’re still deciding which generative music model to use by default. In the video you can hear ACE-Step and Stable Audio 3, which one do you like more?