GStreamer Typefinding and Dynamic Pipelines - Part 2
17 Dec 2014###Introduction
In this tutorial we see how to build dynamic piplines and its application towards playing a video file.
We will see how to identify the type of source stream being proceed and dynamically build a pipeline to handle the stream.
Reading the Source Stream
Let us consider a example that we want to play a WebM video file
Typically for steps in displaying an video file would be
- Reading the Source
- DeMultiplex audio and video channels
- Decode the video and audio
- Convert video into a format that can be displayed
- Display - Render the video frames on screen
Typicall a Video stream will contains individual video,audio and sub-title streams. The job of demultipler is to provide access to these individual streams so that they can be processed accordingly.
In the decoding step encoded video and audio streams are passed to video and audio decoder.
Next the decoded raw video streams are converted to a format suitable for display and finally video and audio streams are passed to respective video and audio sinks .
In this article we will only look at flow of video streams.
We define a structure to hold all the information
/* Structure to contain all our information, so we can pass it to callbacks */
typedef struct _CustomData {
GstElement *pipeline;
GstElement *source;
GstElement *typefinder;
GstElement *demux;
GstElement *decoder;
GstElement *convert;
GstElement *sink;
} CustomData;
Source Type
In the present application we are going to create GStreamer source element from URI passed by the user
data.source = gst_element_make_from_uri (GST_URI_SRC,"file:/home/pi19404/Downloads/sample.webm" ,NULL);
g_print("GStreamer Source Type %s \n",GST_ELEMENT_NAME (data.source));
Since the passed URI is a file location data.source will of GStreamer Element of type filesrc
Gstreamer Pads
The primary complexity in gstreamer applications is that different types of input streams need to be handled differently.
Even demultiplexing techniques of streams need to be handled differently.To this we need a mechanism to analyse the type of source dynamically.
Gstreams provides mechanism called as pads which are the element’s interface to the outside world.
Data streams from one element’s source pad to another element’s sink pad.
A pad type is defined by two properties: its direction and its availability.GStreamer defines two pad directions: source pads and sink pads.The elements receive data on their sink pads and generate data on their source pads.
A pad can have any of three availabilities: always, sometimes and on request.In case of some Gstreamer Elements the pads are are created during element creation/initialization and are always available.
Pads are created from Pad Templates specified for Gstreamer Elements.
we can see the pads associated with element using gst-inspect tool.
Let us look at the pads available in uridecodebin element
Pad Templates:
SRC template: 'src%d'
Availability: Sometimes
Capabilities:
ANY
We can see that element has a source pad and is only available Sometimes.
Typically such pads are created during runtime.The uridecodebin element will analyze the input stream before creating the source pads that can link to other elements.
In the present application the source is off type filesrc
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
ANY
We can see that the pad is always present can be immediately form a like to next element of the pipeline
Gstreamer TypeFind Element
The filesrc element, for example, does not know what type of file it is reading. Before you can attach an element to the pad of the filesrc, you need to determine the media type in order to be able to choose a compatible element.
To solve this problem, a plugin can provide the GStreamer core library with a typedefinition library with a typedefinition.
GStreamer uses the typefinding library to determine the video stream type.GStreamer will read data for as long as the type of a stream is unknown.During this period, it will provide data to all typefind functions for all plugins .
This typefind function will inspect a GstBuffer with data and will output a GstCaps structure describing the type. If the typefind function does not understand the buffer contents, it will return NULL.
GStreamer has a typefind element in its core elements that can be used to determine the type of a given pad.
/* create the typefind element */
data.typefind = gst_element_factory_make ("typefind", "typefind");
g_assert (data.typefind != NULL);
/* Adding Elements to List */
gst_bin_add_many (GST_BIN (data.pipeline),data.source,data.typefinder,NULL);
/*link source and typefind elements */
if (!gst_element_link (data.source, data.typefinder)) {
g_printerr ("Elements could not be linked.\n");
gst_object_unref (data.pipeline);
return -1;
}
GStreamer Signals
Gstreamer provides signalling which is a general purpose notification mechanism that can be used as tool for customization object behavior.
Signals allow the application to notified by means of a callback when something interesting has happened. Signals are identified by a name, and each GObject has its own signals.
For example when the typefind function associated with a plugin recognizes a video stream it uses signaling mechanism to notify the application . We can capture the event and run used defined code upon notification of occurrence of the event.
/* Connect to the typefind signal */
g_signal_connect (data.typefinder, "have-type", G_CALLBACK (cb_typefound), &data);
static void
cb_typefound (GstElement *typefind,
guint probability,
GstCaps *caps,
CustomData data)
{
gchar *type;
type = gst_caps_to_string (caps);
g_print ("Media type %s found, probability %d%%\n", type, probability);
g_free (type);
}
g_signal_connect function attaches a signal handler( g_signal_connect ) to an GStreamer element (data.typefinder = typefind ) to inform the application via callback function (cb_typefound) when the type of input stream been determined .
Once we determine the media type we know the elements that are required to be used in the pipeline In the example we are going to use Web Stream
The Demuxer element type is matroskademux which Demuxes Matroska/WebM streams into video/audio/subtitles.
Decoder element is vp8dec that is capable for decoding the webm format video into raw image format
we can also establish the link of typefinder with demuxer element and the decoder with converter and sink.
if (strcmp(type,"video/webm")==0)
{
data->demux=gst_element_factory_make ("matroskademux", "demux");
data->decoder=gst_element_factory_make ("vp8dec", "decoder");
if (gst_element_link (data->typefinder, data->demux) !=TRUE || gst_element_link_many (data->decoder, data->filter,sink,NULL) !=TRUE) {
g_printerr ("Elements could not be linked.\n");
gst_object_unref (data->pipeline);
return ;
}
}
Gstreamer Demultiplexer
Demuxers do not have any pads till they receive the buffers to parse. As data is available to parse, pads are dynamically added based on the streams available.
A demuxer contains one sink pad, through which the muxed data arrives, and multiple source pads, one for each stream found in the container:
The primary complexity when dealing with demuxers is that they cannot produce any information until they have received some data and have analyzed the type of stream.
Initially when the pipeline is set to run the demuxer does not have any source pads to which other units can link hence the pipeline terminates at the demuxer,
However when the demuxer has received sufficient data to know the type of streams it will create the source pads.This is a time when pipeline can be completed.
The gstreamer provides us a facility to set up a callback function at this point. The pipeline from the demux to decoder can be configured and complete pipeline can be established.
Again gstreamer provides us a singalling mechanism to reveive notification when demuxer has parsed the data and source pads are created.The signal type to be captured is “pad-added”
Thus the idea is to build the pipeline from the source down to the demuxer, and set it to run . When the demuxer has received enough information to know about the number and kind of streams in the container, it will start creating source pads.This is the right time for us to finish building the pipeline and attach it to the newly added demuxer pads.
Since only link till the typefinder as created earlier and no element is connected to the source pads of typefind elements the GStreamer pipeline sees that pipeline has terminated and enters the PAUSED state.
Thus after linking the GStreamer Demux elements the pipeline proceedes futher by changing its state from PAUSED to PLAYING
/* Connect to the pad-added signal */
g_signal_connect (data.demux, "pad-added", G_CALLBACK (pad_added_handler), data);
static void
pad_added_handler (GstElement *element,
GstPad *pad,
CustomData *data)
{
GstPad *sinkpad;
/* We can now link this pad with the vorbis-decoder sink pad */
g_print ("Dynamic pad created, linking demuxer/decoder\n");
/*Get the sink pad of the decoder element */
sinkpad = gst_element_get_static_pad (data->decoder, "sink");
/*link the newly created source pads with decoder sink pads */
gst_pad_link (pad,sinkpad);
gst_object_unref (sinkpad);
}
Debug Output
The Gstreamer provides mechanism to observe debug output messages.
The GST_DEBUG environment variable sets the logging level with 0 being lowest to 5 being the highest.
We can export the environment variable to get the detailed output
# export GST_DEBUG=3 # ./example3 0:00:00.000276094 3334 0x108a600 INFO GST_INIT gst.c:613:init_pre: Initializing GStreamer Core Library version 0.10.36 0:00:00.000474378 3334 0x108a600 INFO GST_INIT gst.c:614:init_pre: Using library installed in /usr/lib/x86_64-linux-gnu 0:00:00.000548656 3334 0x108a600 INFO GST_INIT gst.c:624:init_pre: Linux prasad-TA790GX-A3 3.11.0-15-generic #25~precise1-Ubuntu SMP Thu Jan 30 17:39:31 UTC 2014 x86_64 0:00:00.000835787 3334 0x108a600 INFO GST_INIT gstquery.c:107:_gst_query_initialize: init queries 0:00:00.002662900 3334 0x108a600 INFO GST_INIT gstmessage.c:73:_gst_message_initialize: init messages 0:00:00.003684321 3334 0x108a600 INFO GST_PLUGIN_LOADING gstplugin.c:350:_gst_plugin_initialize: registering 0 static plugins 0:00:00.004054157 3334 0x108a600 INFO GST_PLUGIN_LOADING gstplugin.c:255:gst_plugin_register_static: registered static plugin "staticelements" 0:00:00.004125636 3334 0x108a600 INFO GST_PLUGIN_LOADING gstplugin.c:257:gst_plugin_register_static: added static plugin "staticelements", result: 1 0:00:00.005255415 3334 0x108a600 INFO GST_REGISTRY gstregistry.c:1672:ensure_current_registry: reading registry cache: /home/pi19404/.gstreamer-0.10/registry.x86_64.bin 0:00:00.020256731 3334 0x108a600 INFO GST_REGISTRY gstregistrybinary.c:614:gst_registry_binary_read_cache: loaded /home/pi19404/.gstreamer-0.10/registry.x86_64.bin in 0.014912 seconds 0:00:00.020330088 3334 0x108a600 INFO GST_REGISTRY gstregistry.c:1522:scan_and_update_registry: Validating plugins from registry cache: /home/pi19404/.gstreamer-0.10/registry.x86_64.bin 0:00:00.021159862 3334 0x108a600 INFO GST_REGISTRY gstregistry.c:1634:scan_and_update_registry: Registry cache has not changed 0:00:00.021181003 3334 0x108a600 INFO GST_REGISTRY gstregistry.c:1707:ensure_current_registry: registry reading and updating done, result = 1 0:00:00.021192969 3334 0x108a600 INFO GST_INIT gst.c:805:init_post: GLib runtime version: 2.32.4 0:00:00.021205269 3334 0x108a600 INFO GST_INIT gst.c:807:init_post: GLib headers version: 2.32.0 0:00:00.021221264 3334 0x108a600 INFO GST_INIT gst.c:456:gst_init_check: initialized GStreamer successfully 0:00:00.021790040 3334 0x108a600 INFO GST_PLUGIN_LOADING gstplugin.c:859:gst_plugin_load_file: plugin "/usr/lib/x86_64-linux-gnu/gstreamer-0.10/libgstcoreelements.so" loaded 0:00:00.021820224 3334 0x108a600 INFO GST_ELEMENT_FACTORY gstelementfactory.c:376:gst_element_factory_create: creating element "filesrc" 0:00:00.021957404 3334 0x108a600 INFO GST_ELEMENT_PADS gstelement.c:728:gst_element_add_pad:<GstBaseSrc@0x1254040> adding pad 'src' 0:00:00.021992128 3334 0x108a600 INFO filesrc gstfilesrc.c:374:gst_file_src_set_location: filename : /home/pi19404/Downloads/sample.webm 0:00:00.022004967 3334 0x108a600 INFO filesrc gstfilesrc.c:375:gst_file_src_set_location: uri : file:///home/pi19404/Downloads/sample.webm 0:00:00.022058217 3334 0x108a600 INFO GST_ELEMENT_FACTORY gstelementfactory.c:374:gst_element_factory_create: creating element "typefind" named "typefind" 0:00:00.022136507 3334 0x108a600 INFO GST_ELEMENT_PADS gstelement.c:728:gst_element_add_pad:<GstTypeFindElement@0x11c72b0> adding pad 'sink' 0:00:00.022174432 3334 0x108a600 INFO GST_ELEMENT_PADS gstelement.c:728:gst_element_add_pad:<GstTypeFindElement@0x11c72b0> adding pad 'src' GStreamer Source Type filesrc0 0:00:00.022326110 3334 0x108a600 INFO GST_PLUGIN_LOADING gstplugin.c:859:gst_plugin_load_file: plugin "/usr/lib/x86_64-linux-gnu/gstreamer-0.10/libgstautodetect.so" loaded 0:00:00.022343305 3334 0x108a600 INFO GST_ELEMENT_FACTORY gstelementfactory.c:374:gst_element_factory_create: creating element "autovideosink" named "sink" 0:00:00.022450911 3334 0x108a600 INFO GST_ELEMENT_PADS gstelement.c:728:gst_element_add_pad:<GstAutoVideoSink@0x1257010> adding pad 'sink' 0:00:00.022466793 3334 0x108a600 INFO GST_ELEMENT_FACTORY gstelementfactory.c:374:gst_element_factory_create: creating element "fakesink" named "tempsink"
###Code
The file containing the code is example3.c.The compilation command for the same is as below
gcc example3.c -o example3 `pkg-config --cflags --libs gstreamer-0.10` -g ./example3
The code mentioned in the article can be found in pi19404 tutorials github repository at path
- gst/example3.c
The webM video files can be found at
- http://docs.gstreamer.com/media/sintel_trailer-480p.webm
- http://www.http://pi19404.github.io/pyVision/media/others/sample.webm