Evolving Neural Networks for Cross-adaptive Audio Effects

I’m Iver Jordal and this is my first blog post here. I have studied music technology for approximately two years and computer science for almost five years. During the last 6 months I’ve been working on a specialization project which combines cross-adaptive audio effects and artificial intelligence methods. Øyvind Brandtsegg and Gunnar Tufte were my supervisors.

A significant part of the project has been about developing software that automatically finds interesting mappings (neural networks) from audio features to effect parameters. One thing that the software is capable of is making one sound similar to another sound by means of cross-adaptive audio effects. For example, it can process white noise so it sounds like a drum loop.

Drum loop (target sound):

White noise (input sound to be processed):

Since the software uses algorithms that are based on random processes to achieve its goal, the output varies from run to run. Here are three different output sounds:

These three sounds are basically white noise that have been processed by distortion and low-pass filter. The effect parameters were controlled dynamically in a way that made the output sound like the drum loop (target sound).

This software that I developed is open source, and can be obtained here:

https://github.com/iver56/cross-adaptive-audio

It includes an interactive tool that visualizes output data and lets you listen to the resulting sounds. It looks like this:

visualization-screenshot
For more details about the project and the inner workings of the software, check out the project report:

Evolving Artificial Neural Networks for Cross-adaptive Audio (PDF, 2.5 MB)

Abstract:

Cross-adaptive audio effects have many applications within music technology, including for automatic mixing and live music. The common methods of signal analysis capture the acoustical and mathematical features of the signal well, but struggle to capture the musical meaning. Together with the vast number of possible signal interactions, this makes manual exploration of signal mappings difficult and tedious. This project investigates Artificial Intelligence (AI) methods for finding useful signal interactions in cross-adaptive audio effects. A system for doing signal interaction experiments and evaluating their results has been implemented. Since the system produces lots of output data in various forms, a significant part of the project has been about developing an interactive visualization tool which makes it easier to evaluate results and understand what the system is doing. The overall goal of the system is to make one sound similar to another by applying audio effects. The parameters of the audio effects are controlled dynamically by the features of the other sound. The features are mapped to parameters by using evolved neural networks. NeuroEvolution of Augmenting Topologies (NEAT) is used for evolving neural networks that have the desired behavior. Several ways to measure fitness of a neural network have been developed and tested. Experiments show that a hybrid approach that combines local euclidean distance and Nondominated Sorting Genetic Algorithm II (NSGA-II) works well. In experiments with many features for neural input, Feature Selective NeuroEvolution of Augmenting Topologies (FS-NEAT) yields better results than NEAT.

The Analyzer and MIDIator plugins

… so with all these DAW examples “where are the plugins” you might ask. Well, the most up-to-date versions will always be available in the code repo at github. BUT, I’ve also uploaded precompiled versions of the plugins for Windows and OSX. To install them you just unzip and put them somewhere in your VST plugin search path (typically /Program Files/Vstplugins on windows and /Library/Audio/Plugins/VST on OSX). You also need to install Cabbage to make them work. You can find the latest Cabbage versions here.

Simple analyzer-modulator setup for Reaper

reaper_screen1
Simple example Reaper project with Analyzer and MIDIator. Spectral flux of the input signal will control Reverb Room Size Since we use only one input and it modulates its own reverb, the setup is not crossadaptive, but rather just an adaptive effect. The basic setup can be extended by adding more input channels (and Analyzers) as needed. We do not need to add more MIDIators until we’ve used up all of its modulator channels.

Following up on the recent Ableton Live set, here’s a simple analyzer-modulator project for Reaper. The routing of signals is simpler and more flexible in Reaper, so we do not have the clutter of an extra channel to
enable MIDI out, rather we can select MIDI hardware output in the routing dialog for the MIDIator track. In Reaper,  the MIDIator plugin will start processing all by itself (no need to open its editing window to kick it to life). You need to enable a virtual MIDI device for input (remember to also enable input for control messages) and output. This is done in Reaper preferences / Midi hardware setttings.

The analyzer does not send audio throu, so we still need to use two input tracks; one for the analyzer and one for the actual audio input for processing. The MIDIator is set up to map the spectral flux of Analyzer channel 1 to Midi controller 11, channel 1.  The audio input is sent to a Reverb channel, and we’ve mapped the MIDI controller (11 on channel 1) to the Room size control of the reverb.

Simple analyzer-modulator setup for Ableton Live

live_screen1
Simple example Live set with Analyzer and MIDIator. Spectral flux of the input signal will control Reverb Decay Time. Since we use only one input and it modulates its own reverb, the setup is not crossadaptive, but rather just an adaptive effect. The basic setup can be extended by adding more input channels (and Analyzers) as needed. We do not need to add more MIDIators until we’ve used up all of its modulator channels.

I’ve created a simple Live set to show how to configure the analyzer and MIDIator in Ableton Live. There are some small snags and peculiarities (read on), but basically it runs ok.

The analyzer will not let audio through, so we use two audio input tracks. One track for the actual audio processing (just send to a reverb in our case), and one track for the analyzer. The MIDI processing also requires two tracks in Live, this is because Live assumes that a MIDI plugin will output audio so it disables the MIDI out routing when a plugin is present on a track. This is easily solved by creating a second MIDI track, and select MIDIator as the MIDI input to that track. From this track we can route MIDI out from Live. You will want to set the MIDI out to a virtual MIDI device (e.g. loopMIDI on windows, IAC bus on OSX). Enable midi input from the virtual midi device (you can enable Track, Sync and Remote for a MIDI input. We want to enable Remote)

In our example setup, we’ve enabled one modulator on the MIDIator. This is set to receive spectral flux from Analyzer 1, and send this modulator data to midi channel 1, controller 11. We’ve mapped this to Reverb Decay Time on the effect return track. Input sounds with high flux (which means that the sound is probably a bit noisy) will have long reverb. Sounds with low flux (probably a tonal or stable sound) will have short reverb.

On my computer (windows), I need to open the MIDIator editing window to force it to start processing. Take care *not* to close the MIDIator window, as it will somehow stop processing when the window is closed (This only happens in Ableton Live, not sure why). To get rid of the window, just click on something in another track. This will hide the MIDIator window without disabling it.

Simple crossadaptive mixing template for Logic

Next week we’ll go to London for seminars at Queen Mary and De Montfort. We’ll also do a mixing session with Gary Bromham, to experiment with the crossadaptive modulation techniques in a postproduction setting. For this purpose I’ve done a simple session template in Logic (as it is a DAWs that Gary uses regularly).

logic_screen1
Logic session with two tracks, EQ being applied to track 2, controlled by pitch analysis of the signal on track 1. Reaper (running the analysis and modulator routing plugins) can be seen in the background. The jack patching can also be seen in the lower right part of the image.

To keep the Logic session as clean as  possible, we will do the signal analysis and preparation of the modulation signal in Reaper. This also allows us to use the VST versions of the plugins for analysis and modulation mapping we’ve created earlier. (It should be possible to use these as AU plugins too, but that is somewhat more experimental as of yet). Signal routing between Logic and Reaper is done via Jack. This means that both Logic and Reaper use Jack as the audio driver, while Jack communicates with the audio hardware for physical I/O. The signal to be analyzed (guitar on track 1 in our Logic session) is fed to a separate bus track in Logic, and the output from this track is sent to Reaper via Jack. We analyze the pitch of the signal in Reaper, send the pitch tracking data to the vstMIDIator plugin, and route the MIDI signal from Reaper back to Logic via the IAC bus. In Logic, it is relatively straightforward to map an incoming MIDI signal to control any parameter. You can find this in Logic Preferences/Automation, where there are Learn and Edit buttons towards the bottom of the dialog. To learn a mapping you will touch the desired destination parameter, then send the midi message to be mapped. In our example, we use an EQ on vocals (track 2), and control the frequency of a narrow band in the EQ with the MIDI modulator signal. This way the pitch of the guitar controls the EQ for the vocals.

logic_screen2
The analyzer and MIDIator plugins running in Reaper, with the Logic mixer in the background