Project start meeting in Trondheim

Kickoff

Monday June 6th we had a project start meeting with the NTNU based contributors: Andreas Bergsland, myself, Solveig Bøe, Trond Engum, Sigurd Saue, Carl Haakon Waadeland and Tone Åse. This gave us the opportunity to present the current state of affairs and our regular working methods to Solveig. Coming from philosophy, she has not taken part in our earlier work on live processing. As the last few weeks have been relatively rich in development, this also gave us a chance to bring all of the team up to speed. Me and Trond also gave a live demo of a simple crossadaptive setup where vocals control delay time and feedback on the guitar, while the guitar controls reverb size and hi-freq damping for the vocal. We had discussions and questions interspersed within each section of the presentation. Here’s a brief recounting of issues wwe touched upon.

Roles for the musician

The role of the musician in crossadaptive interplay has some extra dimensions when compared to a regular acoustic performance situation. A musician will regularly formulate her own musical expression and relate this to what the other musician is playing. On top of this comes the new mode of response created by live processing, where the instrument’s sound constantly changes due to the performative action of a live processing musician. In the cross-adaptive situation, these changes are directly controlled by the other musicians’ acoustic signal, so the musical response is two-fold: responding to the expression and responding to the change in own sound. As these combine, we may see converging or diverging flow of musical energy between the different incentives and responses at play. Additionally, her own actions will influence changes on the other musician’s sound, so the expressive is also two-fold; creating the (regular) musical statement and also considering how the changes inflicted on the other’s sound will affect both how the other one sounds and how that affects their combined effort. Indeed, yes, this is complex. Perhaps a bit more complex that we had anticipated. The question was raised if we do this only to make things difficult for ourselves. Quite justly. But we were looking for ways to intervene in the regular musical interaction between performers, to create yet unheard ways of playing together. It might appear complex just now because we have not yet figured out the rules and physics of this new situation, and it will hopefully become more intuitive over time. Solveig voiced it like we put the regular modes of perception in parenthesis. For good or for bad, I think she may be correct.

Simple testbeds

It seems wise to initially set up simplified interaction scenarios, like the vocal/reverb guitar/delay example we tried in this meeting. It puts emphasis on exploring the combinatorial parameter modulation space. Even with a simple situation of extracting two features for each sound source, controlling two parameters on each other’s sound, the challenges to the musical interaction is prominent. Controlling two features of one’s own sound, to modulate the other’s processing is reasonably manageable while also concentrating on the musical expression.

Interaction as composition

An interaction scenario can be thought of as a composition. In this context we may define a composition as something that guides the performers to play in a certain way (think of the text pieces from the 60’s for example, setting the general mood or each musician’s role while allowing a fair amount of freedom for the performer as no specific events are notated). As the performers formulate their musical expression to act as controllers just as much as to express an independent musical statement, the interaction mode has some of the same function as a composition has in written music. Namely to determine or to guide what the performers will play. In this setting, the specific performative action is freely improvised, but the interaction mode emphasizes certain kinds of action to such an extent that the improvisation is in reality not really free at all, but guided by the possibilities (the affordance [https://en.wikipedia.org/wiki/Affordance]) of the system. The intervention into the interaction also sheds light on regular musical interaction. We become acutely aware of what we normally do to influence how other musicians play together with us. Then this is changed and we can reflect on both the old (regular, unmodulated) kind of interaction and the new crossadaptive mode.

Feature extraction, performatively relevant features

Extracting musically salient features is a Big Question. What is musically relevant? Carl Haakon suggested that some feature related to the energy could be interesting. Energy, for the performer can be induced into the musical statement in several ways. It could be rhythmic activity, loudness, timbre, and other ways of expressing energetic performance. As such it could be a feature taking input from several mathematical descriptions. It could also be a feature allowing a certain amount of expressive freedom for the performer, as energy can be added by several widely different performative gestures, leaving some sort of independence from having to do very specific actions in order to trigger the control of the destination parameter. Mapping the energy feature to a destination parameter that results in a more rich and energetic sound could lead to musically convergent behavior, and conversely, controlling a parameter that makes the resulting sound more sparse could create musical and interactive tension. In general, it might be a good idea to use such higher level analysis. This simplifies the interaction for the musician, and also creates several alternative routes to inflict a desired change in the sound. The option to create the same effect by several independent routes/means, also provides the opportunity for doing so with different kinds of side effects (like in regular acoustic playing too), e.g. creating energy in this manner or that manner gives very different musical results but in general drives the music in a certain direction.
Machine learning (e.g. via neural networks) could be one way of extracting such higher level features, different performance situations, different distinct expressions of a performer. We could expect some potential issues of recalibration due to external conditions, slight variations in the signal due to different room, miking situation etc. Will we need to re-learn the features for each performance, or could we find robust classification methods that are not so sensitive to variations between instruments and performance situations?

Meta mapping, interpolations between mapping situations

Dynamic mappings, allowing the musicians to change the mapping modes during different sections of the performed piece. If the interaction mode becomes limited or “worn out” after a while of playing, the modulation mappings could be gradually changed. This can be controlled by an external musician or sound engineer, or it can be mapped to yet other layers of modulations. So, a separate feature of the analyzed sound is mapped to a modulator changing the mappings (the preset or the general modulation “situation” or “system state”) of all other modulators, creating a layered meta-mapping-modulator configuration. At this point this is just an option, still too complex for our initial investigation. It brings attention to the modulator mapping used in the Hadron Particle Synthesizer, where a simple X-Y pad is used to interpolate between different states of the instrument. Each state containing modulation routing and mapping in addition to parameter values. The current Hadron implementation allows control over 209 parameters and 54 modulators via a simple interface. This enables a simplified multidimensional control in Hadron. Maybe the cross-adaptive situation can be thought as somehow similar. The instrumental interface of Hadron behaves in highly predictable ways, but it is hardly possible to decode intellectually, one has to interact by intuitive control and listening.

Listening/monitoring

The influence of the direct/unprocessed sound; With acoustic instruments, the direct sound from the instrument will be heard clearly in addition to the processed sound. In our initial experiments, we’ve simplified this by using electric guitar and close miked vocals. We mostly hear the result of the effects processing. Still, the analysis of features is done on the dry signal. This creates a situation where it may be hard to distinguish what features controls which modulations, because the modulation source is not heard clearly as a separate entity in the sound image. It is easy to mix the dry sound higher, but then we hear less of the modulations. It is also possible to allow the modulated sound be the basis of analysis (creating the possibility for even more complex cross-adaptive feedback modulations as the signals can affect each other’s source for analysis). Then this would possibly make it even harder for the musicians to have intentional control over the analyzed features and thus the modulations. So, the current scheme is, if not the final answer, it is a reasonable starting point.

Audience and fellow musicians’ perception of the interplay

How will the audience perceive this? Our current project does not focus on this question, but it is still relevant to visit it briefly. It also relates to expectations, to schooling of the listener. Do we want the audience to know? Does knowledge of the modulation interaction impede the (regular) appreciation of the musical expression? One could argue that a common symphony orchestra concert-goer does not necessarily know the written score, or have analyzed the music, but appreciates it as an aesthetic object on its own terms. The mileage may vary, some listeners know more and are more invested in details and tools of the trade. Still, the music itself does not require knowledge of how it is made to be appreciated. For a schooled listener, and also for ourselves, we can hope to be able to play with expectations with in the crossadaptive technique. Andreas mentions that listening to live crossadaptive processing as we demonstrated it is like listening to an unfamiliar spoken language, trying to extract meaning. There might be some frustration over not understanding how it works. Also, expectations of the fantastic new interaction mode and not hearing it can lead to disappointment. Using it as just another means of playing together, another parameter of musical expression alleviates this somewhat. The listener does not have to know, but will probably get the opportunity for an extra layer of appreciation with understanding of the process. In any case, our current research project does not directly concern the audience’s appreciation of the produced music. We are currently at a very basic stage of exploration and we need to experiment at a much lower level to sketch out how this kind of musics can work before starting to consider how (or even if) it can be appreciated by the audience.

Conversation with Marije

We had a Skype meeting between me (Oeyvind) and Marije Baalman today, here’s some notes from the conversation:

First, we really need to find alternatives to Skype, the flakyness of connection makes it practically unusable. A service that allows recording the conversation would be nice too.

We talked about a number of perspectives on the project, including Marijes role as an external advisor and commentary, references to other projects that may relate to ours in terms of mapping of signals to musical parameters, possible implementation on embedded devices, and issues relating to the signal flow from analysis to mapping to modulator destination.

Marije mentioned Joel Ryan (* *) as an interesting artist doing live processing of acoustic instruments. His work seems closely related to our earlier work in T-EMP. Interesting to see and hear his work with Evan Parker. More info on his instruments and mapping setup would be welcome.

We discussed the prospect of working with embedded devices for the crossadaptive project. Marije mentioned the Bela project for the BeagleBboard Black, done at Queen Mary University. There are of course a number of nice tthigns about embedded devices, making self-contained instruments for easy of use for musicians. However, we feel that at this stage, our project is of such an experimental nature that is easier explored on a conventional computer. The technical hurdles of adapting it to an embedded device is easier tackled when the signal flow and processing needs are better defined. In relation to this we also discussed the different processes involved in our signal flow: Analysis, Mapping, Destination.

Our current implementation (based on the 2015 DAFx paper) has this division in analysis, mapping/routing and destination (the effect processing parameter being controlled). The mapping/routing was basically done at the destination in the first incarnation of the toolkit. However, lately I’ve started reworking it to create a more generic mapping stage, allowing for more freedom in what to control (i.e. controlling any effect processor or DAW parameter). Marije suggested more preprocessing to be done in the analysis stage, to create cleaner and more generically applicable control signals for easier use at the mapping stage. This coincides with our recent experiences in using low level analyses like spectral flux and crest for example; they work well on some instruments (e.g. vocals) as an indicator of balance between tone and noise, however on other instruments (e.g. guitar) they sometimes work well and sometimes less so. Many of the current high level analyses (e.g. in music information retrieval) seems in many respects to be geared towards song recognition, while we need some way of extracting higher level features as continuous signals. The problem being to define and extract what is the musical meaning of a signal, on a frame-by-frame basis. This will most probably be different for different instruments, so an source-specific preprocessing stage seems reasonable.

The Digital Orchestra and IDMIL, and more specifically the libmapper might provide inspiration for our mapping module. We make a note of inspecting it more closely. Likewise, the Live Algorithms for Music project, and also the possibly the Ircam OMAX project. Marije mentioned 3Dmin project at UDK/TU Berlin using techniques of exploration of arbitrary mappings as a method of influencing rather than directly controlling a destination parameter. This reminded me of earlier work by Palle Dahlstedt, and also, come to think of it our own modulation matrix for the Hadron synthesizer.

Later we want Marije to test our toolkit in more depth and comment more specifically on what it lacks and how it can be improved. There are probably some small technical issues in using the toolkit under Linux, but as our main tools are crossplatform (Windows/OSX/Linux), these should be relatively easy to solve.

Workflows, processing methods

As we’ve now done some initial experiment sessions and gotten to know the terrain a little bit better, it seems a good time to sum up the different working methods for cross adaptive processing in a live performance setting. Some of this is based on my previous article for DAFx-15, but extended due to the practical work we’ve done on the subject during the last months. I’ve also written about this in an earlier blog post here.

Dual input
This type of effect takes two (mono) signals as input, and apply a transformation to one signal according to the characteristics of the other signal. In this cathegory we typically find spectral morphing effects, resonators, convolvers). It is a quite practical effects type with regards to signal routing, as it can easily be inserted into any standard audio mixing signal flow (use the effect on a stereo aux channel, aux send to the effect from two sources, each source panned hard left and right).
We have some proposals/ideas for new effects in this cathegory, including streaming convolution, cross adaptive resonators, effects inspired by the modal reverberation techniques, and variations to common spectral processors adapted to the crossadaptive domain.

Sidechaining
This type of signal interaction can be found in a conventional audio production, most commonly for dynamics processing (e.g. the genre-typical kick-drum-ducking-synth-pads). Multiband variations of sidechaining can be seen in de-essing applications, where the detector signal is filtered so that dynamics processing is only applied whenever there is significant energy in a specific frequency region. The application has traditionally been limited to fixing cosmetic errors (e.g. dampening the too prounounced “s” sounds of a vocal recording with high sibliance). We’ve experimented with using multiband sidechaining with a wider selection of effects types. The technique can be used to attenuate or amplify the signal going into any effects processor, for example boosting the amount of vocal signal going into a pitch shifter in accordance with the amount of energy on the low frequencies of the guitar track. This type of workflow is practical in that it can be done with off-the-shelf commercially available tools, just applying them creatively and bending their intended use a bit. Quite complex routing scenarios can be created by combining different sidechains. As an extension of the previous example, the amount of vocal signals going into the pitch shifter increase with the low energy content of the guitar but only if we do not have significant energy in the high frequency region of the vocal signal. The workflow is limited to adjusting the amplitude of signals going into an effects processor. In all its simplicity, adjusting the amplitude of the input signal is remarkably effective and can create dramatic effects. It can not, however, directly modify the parameters of the processing effect, so the effect will always stay the same.

Envelope follower as modulation source
As a a slight extension of the sidechaining workflow, we can use envelope followers (on the full signal or a band-limited part of it) as modulation sources. This is in fact already done in the sidechaining processors, but the envelope followers are generally hardwired to control a compressor or a gate. In some DAWs (e.g. Reaper and Live), the output of the envelope follower can be routed more freely and thus used as a modulator for any processing parameter in the signal chain. This can be seen as a middle ground of crossadaptive workflows, combining significant complexity with widely available commrecial tools. The workflow is not available in all DAWs, and indeed it is limited to analyzing the signal energy (in any separate frequency band of the input signal). For complex mappings, it might become cumbersome or unwieldy, as the mapping between control signals and effects parameters is distributed, so it happens a little bit here and a little bit there in our processing chain. Still it is a viable option if one needs to conform to using only commercially available tools.

Analysis and automation
This is the currently preferred method and the one that gives the most flexibility in terms of what features can be extracted by analysis of the source signal, and how the resulting modulator signal can be shaped. The current implementation of the interprocessing toolkit allows any analysis methods available in Csound to be used, and the plugins are interfaced to the DAW as VST plugins. The current modulator signal output is in the form of Midi or OSC, but other automation protocols will be investigated. Extension of Csound’s analysis methods can be done by interfacing to excisting technologies used in e.g. music information retrieval, like VAMP plugins, or different analysis libraries like Essentia and Aubio. There is still an open question how to best organize the routing, mixing and shaping of the modulator signals. Ideally, the routing system should give an intuitive insight into the current mapping situation, provide very flexible mappping strategies, and being able to dynamically change from one mapping configuration to another. Perhaps we should look into the modulation matrix, as used in Hadron?

Mapping analyzed features to processing parameters
When using the analyzer plugin, it is also a big question how do we design the mapping of analyzed features to effect processing parameters. Here we have at least 3 methods:

  1. manual mapping design
  2. autolearn features to control signal routing
  3. autolearn features to control effects parameters

For method 1) we have to continue experimenting to familiarize ourselves with the analyzed features and how they can be utilized in musically meaningful ways to control effects parameters. The process is cumbersome, as the analyzed features behave quite differently on different types of source signals. As an example, spectral flux will quite reliably give an indicator of the balance between noise and tone in a vocal signal, and even if it does so also on a guitar signal, the indication is more ambiguous and thus less reliable for use as a precise modulation source for parameter control. On the positive side, the complex process of familiarizing ourselves with the analysis signals will also give us an intuitive relation to the material, which is one cruicial aspect of being able to use it as performers.

Method 2) has been explored earlier by Brech De Man et al in an intelligent audio switch box. This plugin learns features (centroid, flatness, crest and roll-off) of an audio source signal and uses the learned features to classify the incoming audio, to be routed to either output A or B from the plugin. We could adapt this learning method to interpolate between effects parameter presets, much in the way the different states are interpolated in the Hadron synthesizer.

Method 3 can most probably be soved in a number of ways. Our current approach is based on a more extensive use of artificial intelligence methods than the classification in 2) above. The method is currently being explored by  NTNU/IDI master student Iver Jordal, and his work-in-progress toolkit is available here. The task for the A.I is to figure out the mapping between analyzed features and effects parameters based on which sound transformations are most successful. The objective measurement of successful is obviously a challenge, as a number of aesthetic considerations normally apply to what we would term a successful transformation of a piece of audio. As a starting point we assume that in our context, a successful transformation makes the processed sound pick up some features of the sound being analyzed. Here we label the analyzed sound as sound A,  the input sound for processing is sound B, and the result of the processing is sound C. We can use a similarity measure between sound A and C to determine the degree of success for any given transformation (B is transformed to C, and C should become more similar to A than B already is). Iver Jordal implemented this idea using a genetic algorithmto evolve the connections of a neural network controlling the mapping from analyzed features to effect control parameters. Evolved networks can then later be transplanted to a realtime implementation, where the same connections be used on potentially different sounds. This means we can evolve modulation mappings suitable for different instrument combinations and try those out on a wider range of sounds. This application would also potentially gain from being able to interpolate between different modulator routings as described above.

 

Cross adaptive mixing in a standard DAW

To enable the use of these techniques in a common mixing situation, we’ve made some example configurations in Reaper. The idea is to extract some feature of the modulator signal (using common tools like EQs and Compressors rather than more advanced analysis tools), and use this signal to affect the processing of another signal.

By using sidechaining we can allow the energy level of one signal to gate/compress/expand another signal. Using EQ and/or multiband compression on the modulator signal, we can extract parts of the spectrum, for example so that the processing will be applied only if there is energy in the deep frequencies of the modulator signal. Admittedly, this method is somewhat limited as compared with a full crossadaptive approach, but it is widely available and as such has a practical value. With some massaging of the modulator signal and creative selection of effects applied to the affected signal, this method still can produce a fairly wide range of cross-adaptive effects.

By using automation techniques available in Reaper (we will investigate how to do this in other hosts too) , we can map the signal energy directly to parameter changes. This allows cross-adaptive processing in the full sense of the term. The technique (as implemented in Reaper) is limited by the kind of features one can extract from the modulator signal (energy level only) and by the control signal mapping being one-to-one, so that it is not possible to mix several modulator sources to control an effects parameter. Still this provides a method to experiment easily with cross-adaptive mixing techniques in a standard DAW with no extra tools required.

 

Example projects:

Sidechaining used in an unconventional cross-adaptive manner is demonstrated in the CA_sidechain_pitchverb Reaper project.

Signal automation based on modulator energy is demonstrated in the CA_EnvControl Reaper project

 

Terms:

Modulator signal: The signal being used to affect processing of another signal
Affected signal: The signal being modulated

 

Signal (modulator) routing considerations

Analysis happens at the source, but we may want to mix modulation signals from several analyzed sources, so where do we do this? In a separate modulation matrix, or at the destination? Mixing at the destination allows us to only activate the modulation sources actively contributing to a particular effect parameter (might be good to simplify like this), but it also clutters the destination parameter with the scaling/mapping/shaping/mixing of the modulators.

For integration into a traditional workflow in a DAW, do we want the analysis signals to be scaled/shaped/mixed at the source or the destination? For us in the project, it might seem natural to have a dedicated routing central (standalone application), or to have the routing at the destination (in the effect, like in the interprocessing toolkit of 2015). However, to allow any commercial effect to be used, this (scaling/mixing/shaping at the destination) is not feasible.  For a sound engineer it might be more intuitive to have the signal massaging at the source (e.g. “I want the vocals to control reverb mix for the drums, so I adjust this at the vocal source”).

Integration with standard control surface protocols (Mackie/Eucon) might allow the setting of an offset (from where deviations/modulations can occur), then use the analysis signals as modulators increasing  or decreasing this nominal value.  With this kind of routing we must take care to reset the nominal value on playback start, so the modulators do not accumulate in an unpredictable manner.

… which leads us back to the concept of a standalone master routing central, where the nominal value can be stored and modulations applied before sending the control signal to the effect. A standalone application might also be natural as the “one stop solution” when these kind of techniques are needed. It will allow the rest of the mix setup to remain uncluttered. The drawback being that nominal value for the automated effect must be set in the standalone, rather than at the effect (where we might normally expect to set it). Perhaps a two-way communication between the standalone and the DAW (i.e. the effects plugin) can allow the setting of the nominal value in the effect itself, then transmit this value to the standalone routing central, do modulation and send the calue back to the DAW/effect? Is there some kind of distinction between the outbound and inbound control message for control surfaces? I.e. that the outbound message is unaffected by the modulations done on the effect parameter? Unlikely…. Assume it would require two different control channels. But… there must be some kind of feedback prevention in these control surface protocols? “Local off” or similar?

Also, regarding the control surface protocol as a possible interface for signal interaction: is it fast enough? What is the latency and bandwidth?