Difference between revisions of "Synthesizer"

Latest revision as of 19:38, 15 May 2012

Introduction

In the late 1990ies I developed a graphical audio signal synthesis and analysis application called Assampler for Amiga. However it turned out that graphical programming does not scale well. Thus some years later I started to rewrite much of its functionality in plain Haskell. There is both a low-level interface and a high-level framework for automatical inference of sample rate and use of physical units in a sound processing network. This generalizes the usual restricted splitting into audio rate and control rate signals. Routines are now also suitable for real-time processing using Storable Vector library, a Stream like data type and aggressive inlining. There is an interface for using synthesized sounds for rendering Haskore music. The library uses the Numeric Prelude library and its numerical type class hierarchy.

Overview

The aim of this project is audio signal processing using pure Haskell code. The highlights are:

a basic signal synthesis backend for Haskore,
experimental structures for filter networks,
basic audio signal processing, including some hard-coded frequency filters,
advanced framework for signal processing supported by physical units, that is, the plain data can be stored in a very simple number format, even fixed point numbers, but the sampling parameters rate and amplitude can be complex types, like numbers with physical units,
Unlike other software synthesizer packages there are not two global rates, namely control and sample rate. Instead there can be many different rates. The control rate of a signal processor can be bound to the rate of the processed audio signal or it can be independent. In the latter case the internal control parameters are interpolated, because these are the ones that are expensive to compute. In case of constant interpolation and integer ratio of control and sample rate, you get the behaviour known from CSound and SuperCollider.
frameworks for inference of sample rate and amplitude, that is, sampling rate and amplitude can be omitted in most parts of a signal processing expression. They are inferred automatically, just as types are inferred in Haskell's type system. Although the inference of signal parameters needs some preprocessing, the frameworks preserve the functional style of programming and do not need Arrows and according notation.

We have checked three approaches, where the last one is the most promising.

Explicitly maintain a dictionary of signal parameters in a Reader-Writer-State monad, which must be computed completely before any signal processing takes place. This forces all signal parameters to share the same type and prohibits infinitely many signal processors to be involved (e.g., concatenation of infinitely many short noises).
Simulation of logic programming by lazy cycles of function applications (i.e., tied knots, fixed points). The main problems are quadratical computation complexity and a cumbersome and error-prone application. Namely, for each input you have to handle a parameter output, and vice versa for propagation of parameters through the network. You need combinators (infix operators) for combining these functions, but you will easily run into cases where you must plug manually, which is a nightmare.
Unify only the sample rate. Use a Reader functor/monad. Amplitude is propagated from inputs to outputs only. This is a bit conservative, but is simple and comprehensive and fulfils our needs so far.

We checked several low-level implementations in order to achieve reasonable speed. The standard list data structure is very convenient for programming but much too slow for signal processing. We try to get rid of it in several ways:
- A fusion framework based on mapAccumL and unfoldr like functions for plain Prelude lists. Since in current GHC versions (6.8-6.12) the optimisation rules do not fire reliably (e.g., rules are not specialised if a function gets specialised to a monomorphic type) we end up with intermediate list structures too often.
- A chunky list based on the StorableVector is much faster if higher order functions like map and unfoldr are inlined. However, this data structure is not elementwise lazy (a problem for feedback), and can store only values of Storable type (e.g., functions are excluded).
- A data structure analogous to the Stream framework, where a list is represented by a StateT s Maybe a which generates signal values by calling the generator function. In this approach fusion happens by inlining, and lists or other data structures can be used for sharing and feedback including sharing.
- A similar generator type based on (somehow portable) LLVM assembly code that is compiled to machine code at run-time. The code is fast by default: There is no clutter due to missing inlining, too much laziness or inefficient data structures. We can even utilize vector instructions to a fair degree. The downside is that you have to specify what parameters are baken into the compiled functions and which ones remain parameters of those functions. You also have to write the core signal functions using LLVM assembly language.

A combination of all but the first approaches seems to be a good choice so far. However, maintaining all code versions for comparison purposes led to much code duplication in the meantime.

Support for causal processes. Causal signal processes only depend on past data and thus are suitable for real-time processing (in contrast to a function like time reversal). These processes are modelled as mapAccumL like functions. Many important operations like function composition maintain the causality property. They are important in feedback loops where they statically warrant that no future data is accessed.

Packages

actively developed

Package synthesizer-core
Hackage	http://hackage.haskell.org/package/synthesizer-core
Repository	`darcs get http://code.haskell.org/synthesizer/core/`

Package synthesizer-dimensional
Hackage	http://hackage.haskell.org/package/synthesizer-dimensional
Repository	`darcs get http://code.haskell.org/synthesizer/dimensional/`

Package synthesizer-midi
Hackage	http://hackage.haskell.org/package/synthesizer-midi
Repository	`darcs get http://code.haskell.org/synthesizer/midi/`

Package synthesizer-alsa
Hackage	http://hackage.haskell.org/package/synthesizer-alsa
Repository	`darcs get http://code.haskell.org/synthesizer/alsa/`

Package synthesizer-llvm
Hackage	http://hackage.haskell.org/package/synthesizer-llvm
Repository	`darcs get http://code.haskell.org/synthesizer/llvm/`

frozen

Package synthesizer-filter
Hackage	http://hackage.haskell.org/package/synthesizer-filter
Repository	`darcs get http://code.haskell.org/synthesizer/filter/`

Package synthesizer-inference
Hackage	http://hackage.haskell.org/package/synthesizer-inference
Repository	`darcs get http://code.haskell.org/synthesizer/inference/`

Package synthesizer-fusionlist
Hackage	http://hackage.haskell.org/package/synthesizer-fusionlist
Repository	`darcs get http://code.haskell.org/synthesizer/fusionlist/`

Package synthesizer-harpy
Hackage	http://hackage.haskell.org/package/synthesizer-harpy
Repository	`darcs get http://code.haskell.org/synthesizer/harpy/`

Papers

@@ Line 1: / Line 1: @@
 == Introduction ==
-In the late 1990ies I (Henning Thielemann) developed a graphical audio signal synthesis and analysis application called [http://www.assampler.com/ Assampler] for [http://en.wikipedia.org/wiki/Commodore Amiga].
+In the late 1990ies [[User:Lemming|I]] developed a graphical audio signal synthesis and analysis application called [http://www.assampler.com/ Assampler] for [http://en.wikipedia.org/wiki/Commodore_Amiga Amiga].
 However it turned out that graphical programming does not scale well.
 Thus some years later I started to rewrite much of its functionality in plain Haskell.
@@ Line 8: / Line 8: @@
 This generalizes the usual restricted splitting into audio rate and control rate signals.
 Routines are now also suitable for real-time processing using [[Storable Vector]] library, a Stream like data type and aggressive inlining.
-There is an [http://darcs.haskell.org/haskore-synthesizer/ interface] for using synthesized sounds for rendering [[Haskore]] music.
+There is an [http://code.haskell.org/haskore/revised/synthesizer/ interface] for using synthesized sounds for rendering [[Haskore]] music.
 The library uses the [[Numeric Prelude]] library and its numerical type class hierarchy.
-== Links ==
+== Overview ==
-* [http://hackage.haskell.org/cgi-bin/hackage-scripts/package/synthesizer Hackage]
+The aim of this project is audio signal processing using pure Haskell code.
-* [http://darcs.haskell.org/synthesizer/ Darcs repository]
+The highlights are:
-* [http://dafx04.na.infn.it/WebProc/Proc/P_201.pdf early paper] on the concepts.
+* a basic signal synthesis backend for [[Haskore]],
+* experimental structures for filter networks,
+* basic audio signal processing, including some hard-coded frequency filters,
+* advanced framework for signal processing supported by physical units, that is, the plain data can be stored in a very simple number format, even fixed point numbers, but the sampling parameters rate and amplitude can be complex types, like numbers with physical units,
+* Unlike other software synthesizer packages there are not two global rates, namely control and sample rate. Instead there can be many different rates. The control rate of a signal processor can be bound to the rate of the processed audio signal or it can be independent. In the latter case the internal control parameters are interpolated, because these are the ones that are expensive to compute. In case of constant interpolation and integer ratio of control and sample rate, you get the behaviour known from [[CSound]] and [[SuperCollider]].
+* frameworks for inference of sample rate and amplitude, that is, sampling rate and amplitude can be omitted in most parts of a signal processing expression. They are inferred automatically, just as types are inferred in Haskell's type system. Although the inference of signal parameters needs some preprocessing, the frameworks preserve the functional style of programming and do not need Arrows and according notation.
+: We have checked three approaches, where the last one is the most promising.
+:* Explicitly maintain a dictionary of signal parameters in a Reader-Writer-State monad, which must be computed completely before any signal processing takes place. This forces all signal parameters to share the same type and prohibits infinitely many signal processors to be involved (e.g., concatenation of infinitely many short noises).
+:* Simulation of logic programming by lazy cycles of function applications (i.e., tied knots, fixed points). The main problems are quadratical computation complexity and a cumbersome and error-prone application. Namely, for each input you have to handle a parameter output, and vice versa for propagation of parameters through the network. You need combinators (infix operators) for combining these functions, but you will easily run into cases where you must plug manually, which is a nightmare.
+:* Unify only the sample rate. Use a Reader functor/monad. Amplitude is propagated from inputs to outputs only. This is a bit conservative, but is simple and comprehensive and fulfils our needs so far.
+* We checked several low-level implementations in order to achieve reasonable speed. The standard list data structure is very convenient for programming but much too slow for signal processing. We try to get rid of it in several ways:
+** A fusion framework based on <hask>mapAccumL</hask> and <hask>unfoldr</hask> like functions for plain Prelude lists. Since in current GHC versions (6.8-6.12) the optimisation rules do not fire reliably (e.g., rules are not specialised if a function gets specialised to a monomorphic type) we end up with intermediate list structures too often.
+** A chunky list based on the StorableVector is much faster if higher order functions like <hask>map</hask> and <hask>unfoldr</hask> are inlined. However, this data structure is not elementwise lazy (a problem for feedback), and can store only values of <hask>Storable</hask> type (e.g., functions are excluded).
+** A data structure analogous to the <hask>Stream</hask> framework, where a list is represented by a <hask>StateT s Maybe a</hask> which generates signal values by calling the generator function. In this approach fusion happens by inlining, and lists or other data structures can be used for sharing and feedback including sharing.
+** A similar generator type based on (somehow portable) [[LLVM]] assembly code that is compiled to machine code at run-time. The code is fast by default: There is no clutter due to missing inlining, too much laziness or inefficient data structures. We can even utilize [http://en.wikipedia.org/wiki/Vector_processor vector instructions] to a fair degree. The downside is that you have to specify what parameters are baken into the compiled functions and which ones remain parameters of those functions. You also have to write the core signal functions using LLVM assembly language.
+: A combination of all but the first approaches seems to be a good choice so far. However, maintaining all code versions for comparison purposes led to much code duplication in the meantime.
+* Support for causal processes. Causal signal processes only depend on past data and thus are suitable for real-time processing (in contrast to a function like time reversal). These processes are modelled as <hask>mapAccumL</hask> like functions. Many important operations like function composition maintain the causality property. They are important in feedback loops where they statically warrant that no future data is accessed.
+== Packages ==
+=== actively developed ===
+{{PackageInfoBox|name=synthesizer-core|darcs-code=synthesizer/core/}}
+{{PackageInfoBox|name=synthesizer-dimensional|darcs-code=synthesizer/dimensional/}}
+{{PackageInfoBox|name=synthesizer-midi|darcs-code=synthesizer/midi/}}
+{{PackageInfoBox|name=synthesizer-alsa|darcs-code=synthesizer/alsa/}}
+{{PackageInfoBox|name=synthesizer-llvm|darcs-code=synthesizer/llvm/}}
+=== frozen ===
+{{PackageInfoBox|name=synthesizer-filter|darcs-code=synthesizer/filter/}}
+{{PackageInfoBox|name=synthesizer-inference|darcs-code=synthesizer/inference/}}
+{{PackageInfoBox|name=synthesizer-fusionlist|darcs-code=synthesizer/fusionlist/}}
+{{PackageInfoBox|name=synthesizer-harpy|darcs-code=synthesizer/harpy/}}
+== Papers ==
+* [http://dafx04.na.infn.it/WebProc/Proc/P_201.pdf Audio processing using Haskell] at DAFx 2004
+* [http://arxiv.org/abs/1004.4796 Compiling Signal Processing Code embedded in Haskell via LLVM]
 [[Category:Music]]