Extracting Onset (Beat) Times from Audio Files

I recently created a video for one of my sketches and wanted the animation to be synchronized with the rhythm of the background track. This is a quick tutorial explaining how I did this using the fantasticaubio library and Processing. Note that you will need to have some familiarity with using the command line to do this.

The aubio library comes with a bunch of command line tools, and the one I used was the aubioonset command for getting a list of timestamps of detected “note onsets”, a term that is used to describe the beginning of a musical note.

The basic workflow looks like the following: (i) get a list of onset times using aubioonset, (ii) translate these into specific frame numbers for Processing, and (iii) perform some action in your sketch at these times.

Detecting Note Onset Times with aubioonset

The aubioonset tool makes this very easy. First, get the library itself from Github using git at the command line:

Alternatively, you can visit the project page on Github and download and extract a zip file directly.

Next, compile the library as shown below:

This worked flawlessly on my Macbook Air running MacOS High Sierra (10.13). If you have any issues with your specific setup, you might need to Google around a bit.

The example programs (of which aubioonset is one) should now be in the build/examples subdirectory. The documentation for the aubioonset tool can be foundhere.

Since aubioonset outputs the onset time in milliseconds, I wrote a wrapper script to convert these values into Processing frame numbers, given a user-specified frame rate:

Paste that into your favorite editor, change the AUBIO_PATH variable to point to the location of your compiled aubioonset program, and save the file as onsets.sh. Also make sure to make the script runnable via:

The invocation of the script is shown below as well as the output. The script is passed the path to an audio file and the frame rate of your sketch, and it performs all the necessary conversions for you.

The script prints out two variables that can be copy-pasted into Processing. The first, totalFrames specifies the number of onset times that were detected, and the actionFrames is an array containing the frame numbers of each onset.

Using Onset Times in Processing

To use these two variables in Processing is very simple:

In the above code, we first copy-paste the two variables that are given to us by the onsets.sh script. We additionally need one variable: frame_index which is initialized to 0 (and represents the current index into the actionFrames array).

Now, in the draw() function, we check if we have reached our total number of onset times (stored in totalFrames), and if not, check if the frame number at the current index is equal to the current frame (using Processing’s frameCount variable), and if so, perform some actions.

Finally every frame is saved as an image, which can then be glued together into a video, remembering to set the correct frame rate for the video as used in Processing.

And that’s it!