Interviewed by Christian Schaller

Towards GStreamer 1.0: An interview with Wim Taymans

Published on 23rd of November 2011

There is a lot of excitement brewing for the upcoming GStreamer 1.0 release which is expected out before the end of 2011. The buzz of course also extend to our team here at Collabora, with many of our leading multimedia and GStreamer experts contributing to this big milestone. And while many might have caught Wim Taymans's GStreamer 1.0 talk at the GStreamer Conference 2011 in Prague, we also know many people missed the opportunity. So to make sure everyone knows about all the great stuff happening around GStreamer 1.0 we have conducted this interview with Collabora lead multimedia architect and GStreamer maintainer Wim Taymans about his work on GStreamer 1.0.

For those who don't know you, can you tell us a little about yourself?

Wim Taymans: I'm a Belgian living in Barcelona with my awesomest wife Maria and my cute newborn son Ilian. I've been programming since an early age, first I learned BASIC on ZX80, then later I wrote demos and games for Commodore 64 and Amiga. We recently managed to release a new Commodore 64 game that I wrote with a friend back in 1990, called Jim Slim.

I got involved with Linux at around 1998 when I cofounded a project to write a video editor. We soon figured out that we needed a solid multimedia framework for our needs. We started looking for available options and that's how I became involved with GStreamer at around 1999.

For the last 3 years I have been working together with many of the other core GStreamer developers at Collabora, providing development and design work for customers using GStreamer all over the world.

The GStreamer 0.10.x series has been around since December 2005, which means GStreamer has been able to stay API stable for about 6 years, what were the major lessons learned from GStreamer 0.10?

Wim Taymans: It's indeed quite impressive how we managed to keep the API stable that long without hitting a wall. A lot of thought went in into the 0.10 API and we managed to assemble some of the brightest minds in one place to develop the 0.10 version.

I guess we paid a lot of attention to making every bit of the API extensible. We added, for example, a lot of padding to all object structures. This allowed us to add new fields to objects, or even a pointer to more private data. We also made sure that we never used fixed length structures for things where we didn't quite see all the future potential uses cases, like all the variables for data formats, events, messages and queries are stored in dictionaries so that we can simply add more fields when we want.

Looking back, what features of 0.10 where you the most happy about and which where you the least happy about?

Wim Taymans: I'm the most happy about the changes we did to the scheduler. It was certainly the most controversial change and a serious break with one of the original design decisions of the early GStreamer versions. We essentially went from a design where there is a central component that decides what piece of the multimedia pipeline to schedule to a design where we simply spawn threads and let the kernel decide what can be scheduled.

Since we were moving to a multi-threaded design, we needed to make sure that each and every piece of the framework was thread-safe. I also like how we managed to implement that, largely hiding the threads from the applications and integrating with the application mainloop. Where 0.8 was a threading nightmare, we rarely see any thread related problems with the 0.10 design.

One of the things that is quite ugly in the 0.10 design has to do with dynamic changes in the pipeline. Timing information and events that describe the context of a stream like tags are all lost when you dynamically change a connection. We initially thought that maintaining the state would be up to the application doing the dynamic re-connections but it turns out that this is a little too difficult in general.

At the 2010 GStreamer Conference in Cambridge, United Kingdom you announced the GStreamer 1.0 development effort, what was the motivation for kickstarting that effort?

Wim Taymans: Many reasons, really, but the main one was that we needed to stop talking about it and really do it. :) The first plans for the 1.0 effort were laid in 2009 when a large number of GStreamer core hackers met in Gran Canaria for the Desktop Summit.

We identified a bunch of things that we wanted to fix for a 1.0 release. We were also thinking at that time to simply rename some future 0.10 minor version to 1.0. The thing is that 0.10 works so well and that the ugly bits are not ugly enough to make us feel in a hurry to rewrite parts for a 1.0 version. GStreamer has a quite large code base and working on a 1.0 version would temporarily stall development on the main branch.

What would you say are the headline features of GStreamer 1.0?

Wim Taymans: First we rewrote how buffers and memory are handled. This was kind of an unplanned feature back in the GStreamer conference presentation. The main reason we decided to rewrite some of this was to handle integration with GPUs and DSPs much better. The new model is much closer to that in APIs such as GEM, VAAPI and VDPAU. Where 0.10 gave you access to a memory area by following a pointer on the buffer object, in 1.0 the memory access can only be explicitly accessed with an mmap-like API.

The second large change is the addition of arbitrary meta-data to the buffer objects. This allows us to attach extra structures to buffers that can then contain information such as video strides, cropping information, regions of interest etc. We already used this new feature to add extra video information to buffers that allows us now to specify custom strides and other bits that we could not express in 0.10.

We solved some of the problems with dynamic pipeline with what are now called sticky events. When you dynamically link elements, GStreamer will automatically make sure that the context of the stream (tags, format description, timing information, etc..) is updated on the newly added element. We also made some changes to more easily deal with the timing in a pipeline. We also improved how you can interact with the data on pads.

Lastly we cleaned up all the cruft that we accumulated over the years and made sure we have enough padding again for the next 5 years :). There are many more small neat things and improvements all over the place that it would be impossible to cover in detail for this interview.

All in all pretty boring features for the end users and, in fact, the application level API only has a very small amount of changes. The interesting bits are mostly for plug-in developers and most of the porting effort goes into porting our hundreds of plug-ins. We hope that the new changes will allow plug-in developers to eventually improve the performance of their plug-ins. It's basically about improving performance, making better abstractions and nice APIs in the end.

With the rapid growth of Linux on embedded systems one thing we know a lot of hardware vendors want is stronger integration between GStreamer and OpenMax IL. Anything new coming in GStreamer 1.0 which will improve interoperability with OpenMax IL?

Wim Taymans: Yes, the buffer metadata and the rewritten memory handling as mentioned before, has been a direct result of the requirements to improve integration with those frameworks. At the moment we don't have our OpenMax wrappers ported and adapted to 0.11 yet but we expect to have better optimized wrappers before 1.0 is released. These improvements also address integrating directly with the hardware DSPs so GStreamers performance on embedded hardware will improve significantly also without OpenMax IL.

Also as part of this work we are continuing to make it easier to integrate any kind of hardware acceleration in GStreamer, so doing things like vdpau and libva support in 1.0 will be simpler and better. In 1.0 it will be easier to avoid copying memory from the CPU to GPU and back again and again. The goal is that the application developer shouldn't need to worry if the decoder is a software or GPU one.

How big will the API changes in GStreamer 1.0 be for developers? For instance to port something like the Totem media player or the Banshee Music player, what kind of effort do you expect?

Wim Taymans: Those kinds of application would need none to very little changes. I almost bet that a simple recompile would be enough..

One exception to that rule though would be applications using language bindings. As we are switching to using GObject introspection for our bindings instead of doing bindings manually, the API changes of the bindings will be most noticeable. On the other hand it should also mean that it will be easier to have full API access even for smaller languages.

During the course of GStreamer 0.10 a lot of baseclasses for common element types such as filters, sources and sinks have been introduced, how have they affected general plug-in quality and will they have a big impact on the ease of porting plug-ins to GStreamer 1.0?

The quality of the element is certainly vastly improved. For some of the more complicated base-classes like the sinks or the audiosinks it would have been quite impossible to make all plug-ins reach the current good standard. For the basetransform based elements, we basically managed to rip half of the code from the base class with only very little changes to the subclasses.

Most of the changes that need to be done to elements are caused by the API changes for the buffer memory management. Elements that deal with video should really also be ported to use the new video APIs so that they get the strides and all the new features right. The base classes don't help you much with those changes.

One criticism of GStreamer has been that it's too low level, in GStreamer 0.10 elements such as playbin, camerabin, discoverer and encodebin has been introduced. How will GStreamer 1.0 affect these elements and will their use make porting simpler for application developers?

All those helper highlevel elements are great ideas and continue to function virtually identically to their 0.10 versions. Apps using those elements (such as Totem and Banshee) really need very few changes for this reason.

What are your plans beyond the GStreamer 1.0 release? Anything you already have planned for introduction during the 1.x lifecycle?

Wim Taymans: There are a whole lot of things that can be improved, software development is a never ending story, but we have to limit the amount of new features for 1.0 if we ever want to release it any time soon.

I hope that for post 1.0 version we can focus a bit more on cleaning up the higher level APIs instead of messing with the lowlevel infrastructure. It would be nice, for example, to improve the API to let the application select streams and how they are identified.

It is also important to note that the transition to 1.0 is not likely to happen overnight. 0.10 and 1.0 will be parallel installable and we foresee a period of 6-12 Months where most distros will be shipping both versions of GStreamer and applications getting ported to the new 1.0 API one by one over a period of time.

Back to Collabora GStreamer page.