GStreamer, FFmpeg and Spek
by Alexander Kojevnikov
Next version of Spek will use FFmpeg libraries to decode audio files. There are several reasons for the switch from GStreamer:
GStreamer is a fantastic framework for building complex multimedia pipelines, however what Spek really needs is a simple decoder and FFmpeg’s libavformat and libavcodec do just that.
To handle some audio formats (e.g. APE and DTS), GStreamer relies on FFmpeg anyway, so the switch will result in lesser dependencies. It doesn’t matter too much on GNU/Linux, but this will reduce the size of the Windows and Mac OS X installers.
Spek used GStreamer’s spectrum plugin to perform the actual spectral analysis, with FFmpeg I had to implement it myself. The code I ended up with is very compact and gives room for a lot of experimentation, from using different window functions (it’s still Hamming) and working on performance optimisations to switching to a faster FFT library.
The last bit is actually done, Spek now uses FFTW which in my tests is 1.5x to 2x faster than Kiss FFT used by GStreamer. Apart from that, FFTW can scale to multiple threads with near linear performance increase, future versions of Spek will take advantage of this.
UPDATE: As one of commenters pointed out, FFTs on small number of samples are not very parallelisable and my benchmarks confirm this. Also, I switched from FFTW to avfft which is built into FFmpeg. It’s a little bit faster than FFTW for my particular use case. Lastly, 1.5x to 2x speed up was actually caused by a faster decoder, not by a faster FFT library.
Another thing that would be hard with GStreamer is static cross-compilation using mingw-cross-env to produce a single Windows executable. Because FFmpeg doesn’t employ a plugin architecture, static linking is not an issue.
Last, but not least, the whole experience was very educational. I now remember why I loved C back in the days, the trick is to not even attempt to write any GObject code with it, that’s what Vala is for. Fast Fourier Transform and the maths behind it is much fun, and NR was very helpful here.
For the curious, the code is merged to git master and pushed to Gitorious. The next version of Spek will be released sometime in July.
If you tried to use both avfft (#include ) and FFTW, how do they compare?
Please allow the use of both gstreamer and ffmpeg, having spek as a gstreamer sink would be very nice.
Multi-threaded FFT is only useful on huge data sets with element counts approaching millions. With the FFT sizes typically used for audio analysis (around 1024 samples), the threading overhead will kill any speedup from parallelisation.
@Alexander: I haven’t tried avfft, is it used anywhere outside of libavcodec? FFTW benchmark results page [1] mentions [2] FFmpeg but they don’t seem to have any actual benchmarks for it.
@foo: There’s already a GStreamer plugin called `spectrum`, if you are interested in a visualisation plugin, feel free to write it based on either spectrum or Spek.
@Mans: That’s what I was afraid of as I’m using 2048 samples. I’m definitely going to do benchmarking and will blog about the results.
[1] http://www.fftw.org/benchfft/
[2] http://www.fftw.org/benchfft/ffts.html
avfft is used only in libavcodec, because one wouldn’t use such a big dependency as libavcodec just for its FFT.
@Alexander: Actually I found a benchmark [1], looks like FFTW is significantly faster than avfft.
[1] http://www.fftw.org/speed/CoreDuo-3.0GHz-icc/supersgj0-32.1d.scxx.p2.png
The benchmark is very old (HTTP headers say “Last-Modified: Sun, 08 Oct 2006 16:41:24 GMT”). Due to the relatively recent (summer 2008) switch to the split-radix FFT in ffmpeg, it makes sense to retry the benchmark.
Further to what Alexander E P said, the FFTW team has been accused of cooking benchmarks in their favour: http://cr.yp.to/djbfft/bench-notes.html
Alexander, Mans: Fair enough, I’ll give avfft a try. One dependency less if they are comparable.
Alex, I totally understand your motivation regarding installer size. Still if you missed features in gstreamers spectrum analyzer/fft, you should have told us or even contributed a patch.
With the exception of the issue of static linking for Windows, it sounds like it might have worked just as well to convert GStreamer to use FFTW and your new spectrum analysis code. Plus, that would let other projects that need the same functionality use your better implementation in GStreamer.
@Stefan: I did, even two of them (see bgo#619045)
@Anon: I think FFTW is quite a heavy dependency for gst. Even if gst devs wouldn’t mind adding it, I still think that gst is not a good fit for Spek, I really don’t need anything apart from the decoder. Nevertheless, GStreamer would still be my library of choice for an AV player or anything remotely complex.
Stefan, Anon: I did some benchmarking and apparently it’s the decoding which is slower in gst, not FFTs. I guess GStreamer’s buffer allocations and element synchronisations are slowing things down, all this great flexibility has its costs.
Alexander, Mans: I benchmarked avfft and it’s consistently 11% faster than FFTW for 11-bit real FFTs. That’s without taking into account that avfft does an in-place FFT, FFTW’s in-place FFTs are even slower.
Spek now uses avfft, thanks for being so persistent! :)
Alex, indeed you made patches even – please take my apologies. Regarding the decoding speed, we are aware that the mad mp3 decoder is not the fastest. There is a patch for a plugin based on mpg123 available somewhere (can’t find the link right now). That one is twice as fast, but people are reluctant to take yet another mp3 decoder in :/
Regarding the choise of the FFT, we have picked kissfft becasue of licensing, FFTW is GPL only. Hopefully we can optimize the inner butterfly op with orc at some point.
[...] versia.com Alexander Kojevnikov's blog « GStreamer, FFmpeg and Spek [...]
[...] version is about 3 times faster than 0.5 thanks to the lightning-fast FFmpeg decoders and the new multi-threaded analysis [...]