TimeSide

open and fast web audio components

created by Guillaume Pellerin / @yomguy at Parisson.com

Goals

We just need a python library to:

• build an open python framework to do scalable asynchronous audio processing
• decode audio frames from any format into numpy arrays
• stream the frames in various processors and do numpy data analyzing
• create various image outputs like waveforms, spectrograms, etc.. with numpy and PIL
• transcode the processed frames in various media formats and stream it on the fly in realtime
• provide a high-level 100% HTML5 user interface to display the results on demand and play sound through the web
• metadata indexing, time marking and store everything on a web server (see Telemeta project)

Architecture

Quick processing example

Define some processors:

import timeside.decoder
import timeside.grapher
import timeside.analyzer
import timeside.encoder

decoder = timeside.decoder.FileDecoder('tests/samples/sweep.wav')
grapher = timeside.grapher.Waveform()
analyzer = timeside.analyzer.Level()
encoder = timeside.encoder.Mp3Encoder('tests/samples/sweep.mp3')
     

then, the magic pipeline:

(decoder | grapher | analyzer | encoder).run()
     

get the results:

grapher.render(output='image.png')
print 'Level:', analyzer.results()
     

Quick UI example

Documentation : UiGuide

Changelog (dev branch, 05/13)

• finally fix all decoder memory leaks! (piem)
• fix ogg vorbis and flac encoders (piem)
• add various aubio analyzers thanks to piem such as :
• pitch (f0)
• onsets
• tempo
• various spectral descriptors like : hfc, complex, phase, specdiff, kl, mkl, specflux, centroid, slope, rolloff, spread, skewness, kurtosis, decrease
• new AnalyzerResultContainer and AnalyzerResult classes with various i/o formats : xml, json, yaml, numpy (piem)
• more unit tests (piem)
• UI : rewind player after ending + various bugfixes (yomguy)
• separate hosting for test samples (yomguy)

Install for production

for version 0.4.3 on Linux (Debian Stable 7.0)

sudo apt-get update

sudo apt-get install python python-pip python-setuptools python-gobject \
                        python-gst0.10 gstreamer0.10-plugins-base gir1.2-gstreamer-0.10 \
                        gstreamer0.10-plugins-good gstreamer0.10-plugins-bad \
                        gstreamer0.10-plugins-ugly gobject-introspection \
                        python-numpy python-mutagen python-yaml python-imaging \
                        python-simplejson

sudo pip install timeside
      

Install for development 1/2

for version >= 0.5 + aubio 0.4dev on Linux (Debian Stable 7.0)

sudo apt-get update

sudo apt-get install python python-dev python-pip python-setuptools python-gobject \
                        python-gst0.10 gstreamer0.10-plugins-base gir1.2-gstreamer-0.10 \
                        gstreamer0.10-plugins-good gstreamer0.10-plugins-bad \
                        gstreamer0.10-plugins-ugly gobject-introspection python-numpy \
                        python-yaml python-imaging python-simplejson python-mutagen
                        libsndfile-dev libsamplerate-dev  libjack-jackd2-dev \
                        liblash-compat-dev libfftw3-dev \
                        docbook-to-man gcc git-core ipython \

                    

install aubio with "develop" branch

git clone git://git.aubio.org/git/aubio/

cd aubio

git checkout develop

./waf configure
./waf build
sudo ./waf install

cd python
sudo python setup.py install
      

Install for development 2/2

git clone https://github.com/yomguy/TimeSide.git

cd TimeSide

git checkout dev

export PYTHONPATH=$PYTHONPATH:`pwd`

tests/run_all_tests

      

Ready!

API

class IProcessor(Interface):
    """Common processor interface"""

    @staticmethod
    def id():
        """Short alphanumeric, lower-case string which uniquely identify this
        processor, suitable for use as an HTTP/GET argument value, in filenames,
        etc..."""

        # implementation: only letters and digits are allowed. An exception will
        # be raised by MetaProcessor if the id is malformed or not unique amongst
        # registered processors.

    def setup(self, channels=None, samplerate=None, blocksize=None, totalframes=None):
        """Allocate internal resources and reset state, so that this processor is
        ready for a new run.

        The channels, samplerate and/or blocksize and/or totalframes arguments
        may be required by processors which accept input. An error will occur if any of
        these arguments is passed to an output-only processor such as a decoder.
        """

        # implementations should always call the parent method

    def channels(self):
        """Number of channels in the data returned by process(). May be different from
        the number of channels passed to setup()"""

    def samplerate(self):
        """Samplerate of the data returned by process(). May be different from
        the samplerate passed to setup()"""

    def blocksize():
        """The total number of frames that this processor can output for each step
        in the pipeline, or None if the number is unknown."""

    def totalframes():
        """The total number of frames that this processor will output, or None if
        the number is unknown."""

    def process(self, frames=None, eod=False):
        """Process input frames and return a (output_frames, eod) tuple.
        Both input and output frames are 2D numpy arrays, where columns are
        channels, and containing an undetermined number of frames.  eod=True
        means that the end-of-data has been reached.

        Output-only processors (such as decoders) will raise an exception if the
        frames argument is not None. All processors (even encoders) return data,
        even if that means returning the input unchanged.

        Warning: it is required to call setup() before this method."""

    def release(self):
        """Release resources owned by this processor. The processor cannot
        be used anymore after calling this method."""

        # implementations should always call the parent method

     

API

class IDecoder(IProcessor):
    """Decoder driver interface. Decoders are different of encoders in that
    a given driver may support several input formats, hence this interface doesn't
    export any static method, all informations are dynamic."""

    def __init__(self, filename):
        """Create a new decoder for filename."""
        # implementation: additional optionnal arguments are allowed

    def format():
        """Return a user-friendly file format string"""

    def encoding():
        """Return a user-friendly encoding string"""

    def resolution():
        """Return the sample width (8, 16, etc..) of original audio file/stream,
           or None if not applicable/known"""

    def metadata(self):
        """Return the metadata embedded into the encoded stream, if any."""

     

API

class IAnalyzer(IProcessor):
    """Media item analyzer driver interface. This interface is abstract, it doesn't
    describe a particular type of analyzer but is rather meant to group analyzers.
    In particular, the way the result is returned may greatly vary from sub-interface
    to sub-interface. For example the IValueAnalyzer returns a final single numeric
    result at the end of the whole analysis. But some other analyzers may return
    numpy arrays, and this, either at the end of the analysis, or from process()
    for each block of data (as in Vamp)."""

    def __init__(self):
        """Create a new analyzer."""
        # implementation: additional optionnal arguments are allowed

    @staticmethod
    def name():
        """Return the analyzer name, such as "Mean Level", "Max level",
        "Total length, etc..  """

    @staticmethod
    def unit():
        """Return the unit of the data such as "dB", "seconds", etc...  """
     

API

class AnalyzerResultContainer(object):

    def __init__(self, analyzer_results = []):
        self.results = analyzer_results

    def __getitem__(self, i):
        return self.results[i]

    def __len__(self):
        return len(self.results)

    def __repr__(self):
        return self.to_json()

    def __eq__(self, that):
        if hasattr(that, 'results'):
            that = that.results
        for a, b in zip(self.results, that):
            if a != b: return False
        return True

    def add(self, analyzer_result):
        if type(analyzer_result) == list:
            for a in analyzer_result:
                self.add(a)
            return
        if type(analyzer_result) != AnalyzerResult:
            raise TypeError('only AnalyzerResult can be added')
        self.results += [analyzer_result]

    def to_xml(self, data_list = None):
        if data_list == None: data_lit = self.results
        import xml.dom.minidom
        doc = xml.dom.minidom.Document()
        root = doc.createElement('telemeta')
        doc.appendChild(root)
        for data in data_list:
            node = doc.createElement('data')
            for a in ['name', 'id', 'unit']:
                node.setAttribute(a, str(data[a]) )
            if type(data['value']) in [str, unicode]:
                node.setAttribute('value', data['value'] )
            else:
                node.setAttribute('value', repr(data['value']) )
            root.appendChild(node)
        return xml.dom.minidom.Document.toprettyxml(doc)

    def from_xml(self, xml_string):
        import xml.dom.minidom
        import ast
        doc = xml.dom.minidom.parseString(xml_string)
        root = doc.getElementsByTagName('telemeta')[0]
        results = []
        for child in root.childNodes:
            if child.nodeType != child.ELEMENT_NODE: continue
            child_dict = {}
            for a in ['name', 'id', 'unit']:
                child_dict[a] = str(child.getAttribute(a))
            try:
                child_dict['value'] = ast.literal_eval(child.getAttribute('value'))
            except:
                child_dict['value'] = child.getAttribute('value')
            results.append(child_dict)
        return results

    def to_json(self, data_list = None):
        if data_list == None: data_list = self.results
        import simplejson as json
        data_strings = []
        for data in data_list:
            data_dict = {}
            for a in ['name', 'id', 'unit', 'value']:
                data_dict[a] = data[a]
            data_strings.append(data_dict)
        return json.dumps(data_strings)

    def from_json(self, json_str):
        import simplejson as json
        return json.loads(json_str)

    def to_yaml(self, data_list = None):
        if data_list == None: data_list = self.results
        import yaml
        data_strings = []
        for f in data_list:
            f_dict = {}
            for a in f.keys():
                f_dict[a] = f[a]
            data_strings.append(f_dict)
        return yaml.dump(data_strings)

    def from_yaml(self, yaml_str):
        import yaml
        return yaml.load(yaml_str)

    def to_numpy(self, output_file, data_list = None):
        if data_list == None: data_list = self.results
        import numpy
        numpy.save(output_file, data_list)

    def from_numpy(self, input_file):
        import numpy
        return numpy.load(input_file)
     

Howto implement an analyzer plugin?

start from this template

from timeside.core import Processor, implements, interfacedoc, FixedSizeInputAdapter
from timeside.analyzer.core import *
from timeside.api import IAnalyzer

import numpy

class NewAnalyzer(Analyzer):
    implements(IAnalyzer)

    @interfacedoc
    def setup(self, channels=None, samplerate=None, blocksize=None, totalframes=None):
        super(NewAnalyzer, self).setup(channels, samplerate, blocksize, totalframes)
        # do setup things...

    @staticmethod
    @interfacedoc
    def id():
        return "new_analyzer"

    @staticmethod
    @interfacedoc
    def name():
        return "New analyzer"

    def process(self, frames, eod=False):
        # do process things...
        # and maybe store some results :
        # self.result_data = ...

        return frames, eod

    def results(self):

        result = AnalyzerResult(id = self.id(), name = self.name(), unit = "something")
        result.value = self.result_data
        container.add(result)

        # add other results in the container if needed...

        return container

     

Howto implement an analyzer plugin?

• adapt the template
• save the file in timeside/analyzer/
  for instance : timeside/analyzer/new_analyzer.py
• add it to timeside/analyzer/__init__.py like:

from level import *
from dc import *
from aubio_temporal import *
from aubio_pitch import *
from aubio_mfcc import *
from aubio_melenergy import *
from aubio_specdesc import *
from yaafe import * # TF : add Yaafe analyzer
from spectrogram import Spectrogram
from waveform import Waveform
from vamp_plugin import VampSimpleHost
from irit_speech_entropy import *
from irit_speech_4hz import *
from new_analyzer import * # << here
     

Howto implement an analyzer plugin?

then test it!

import timeside

decoder = timeside.decoder.FileDecoder('tests/samples/sweep.wav')
analyzer = timeside.analyzer.NewAnalyzer()

(decoder | analyzer).run()

analyzer.results()
     

Links

• Official website and wiki
• Source code on GitHub
• API
• Telemeta project : web audio CMS
• Aubio project
• GNU General Public License

Thanks!

by Guillaume Pellerin

guillaume@parisson.com

@yomguy

This document is released under the terms of the contract Creative Commons by-nc-sa/2.0/fr