Gstreamer carrier

If your robot has cameras with high resolution, you need to compress and to stream their images.

Collaboration diagram for Gstreamer carrier:

If your robot has cameras with high resolution, you need to compress and to stream their images.

In order to achieve this, we use Gstreamer, a free framework for media applications.

This document contains a brief introduction to Gstreamer tool and explains how the high-resolution cameras stream is encoding and how a yarp application can read it.

Note

They are still work in progress and should be considered experimental. Please report any problems.

Gstreamer: brief introduction

Gstreamer is a free framework for media applications; it provides a set of plugins that let the user to build applications by connecting them as in a pipeline. It has been ported to a wide range of operating systems, compilers and processors, including Nvidia GPUs.

A Gstreamer application is composed by a chain of elements, the base construction block of a Gstreamer application. An element takes an input stream from previous element in the chain, carries out its function, like encode, and passes the modified stream to the next element. Usually each element is a plugin.

The user can develop application in two way: the first consists in write an application in c/c++, where the elements are connected using API, while the second uses the gst-launch command-line tool. In the following an example of how to use gst-launch command:

gst-launch-1.0 -v videotestsrc ! ‘video/x-raw, format=(string)I420, width=(int)640, height=(int)480’ ! x264enc ! h264parse ! avdec_h264 ! autovideosink

This command creates a source video test with the properties specified in this string “video/x-raw, format=(string)I420, width=(int)640, height=(int)480”; after it is encoded in h264, then decoded and shown. Each element of this pipeline, except the property element, is plugins dynamically loaded. The videotestsrc element lets the user to see a stream without using camera.

The previous command works on Linux, but since Gstreamer is platform independent, we can launch the same command on Windows taking care to change only hardware dependent plugin. So the same command on Windows is:

gst-launch-1.0 -v videotestsrc ! “video/x-raw, format=(string)I420, width=(int)640, height=(int)480” ! openh264enc ! h264parse ! avdec_h264 ! autovideosink

It’s important to notice that the changed element is the encoder (openh264enc), while the decoder is the same. This because the decoder belongs to the plugin that wraps libav library, a cross-platform library to convert stream in a wide range of multimedia formats. [see References chapter]

Please see Notes section about commands in this tutorial.

How to stream using h264 encoder

The server grabs images from cameras, so it needs to run on where cameras are connected. The server is a Gstreamer command pipeline, while the client could be a yarp or a Gstreamer application connected to the robot’s network.

Server side:

The server application consists in the following Gstreamer command:

gst-launch-1.0 -v v4l2src device="/dev/video1" ! ‘video/x-raw, width=1280, height=480, format=(string)I420’ ! omxh264enc ! h264parse ! rtph264pay pt=96 config-interval=5 ! udpsink host=224.0.0.1 auto-multicast=true  port=33000

• v4l2src device="/dev/video1": reads the cameras using v4l (video for linux) driver
• ‘video/x-raw, width=640, height=480, format=(string)I420’ : this is a property-element that specifies image size and format. In order to know which sizes and format are available you can use “v4l2-ctl” utility.
• omxh264enc: encodes in h264: plugin developed by Nvidia
• h264parse: signals downstream the format of the stream
• rtph264pay: puts h264 stream in rtp packets
• udpsink: this is the last element and sends out the stream. In this case we use multicast, but it is possible to send the stream using unicast in this way: udpsink host=IP_ADDRESS_OF_CLIENT port=NOT_WELL_KNOWN_PORT_NUMBER

Client side

The client can read the stream using Gstreamer native command:

gst-launch-1.0 -v udpsrc multicast-group=224.0.0.1 auto-multicast=true port=3000 caps="application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264 ! autovideosink

Some options and extra notes

• set the frame rate : in server side exist the parameter framerate=30/1, that configures the framerate to which grab images. Insert in in property element: 'video/x-raw, format=(string)I420, width=(int)640, height=(int)480, framerate=30/1'
• In some cases could be useful that server streams video at constant rate. You can achieve this adding this parameter to the encoder plugin: control-rate=2 bitrate=5000000
• You can use the native Gstreamer plugin “videocrop” to crop the video:

      gst-launch-1.0 -v udpsrc port=33000 caps="application/x-rtp, media=(string)video,  encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264 ! videocrop left=10, right=30, top=50, bottom=50 ! autovideosink

• fakevideosink plugin can be used instead of autovideosink to test the pipeline without displaying anything.
• videotestsrc followed by the format specifier can be used to generate a test image
• filesrc / filesink plugins can used to read from/write to file.
• other useful encoder plugins: x264,x265,avenc_mjpeg. Suggested decoders: avdec_h265,avdec_h265,avdec_mjpeg
• official plugins list: https://gstreamer.freedesktop.org/documentation/plugins_doc.html?gi-language=c
• another common format for video/x-raw instead of I420 is RGB

How to install Gstreamer

Currently we are using 1.24.4 version.

On Ubuntu

• Packages required to build
  • libgstreamer1.0-dev
  • libgstreamer-plugins-base1.0-dev
• Packages required to run
  • gstreamer1.0-plugins-base (for videoconvert and appsink)
  • gstreamer1.0-plugins-good (for udpsrc and rtph264depay)
  • gstreamer1.0-plugins-bad (for h264parse)
  • gstreamer1.0-libav (for avdec_h264)
• Useful packages but not required
  • gstreamer1.0-tools

sudo apt-get install libgstreamer1.0-dev \
libgstreamer-plugins-base1.0-dev \
gstreamer1.0-plugins-base \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-bad \
gstreamer1.0-libav \
gstreamer1.0-tools

On windows

You need to download both the main package and the devel package from here: https://gstreamer.freedesktop.org/data/pkg/windows/

Installation of package Gstreamer:

• choose typical installation
• redo installation adding:
  • Gstreamer 1.0 devtools
  • Gstreamer 1.0 Libav wrapper

Installation of grstreamer devel package:

• choose typical installation
• redo installation adding:
  • Gstreamer 1.0 devtools
  • Gstreamer 1.0 Libav wrapper
• Add in path environment variable the path to executable (Usually is C:\gstreamer\1.0\x86_64\bin)

Verify your installation

You can verify the installation by running a simple test application composed by a server and a client :

Server side (example on Windows)

gst-launch-1.0 -v videotestsrc ! "video/x-raw, format=(string)I420, width=(int)640, height=(int)480" ! openh264enc ! h264parse ! rtph264pay pt=96 config-interval=5 ! udpsink host=<YOUR_IP_ADDRESS> port=<A_PORT_NUMBER>

Client side

gst-launch-1.0 -v udpsrc port=<A_PORT_NUMBER> caps="application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264 ! autovideosink

Usage with yarp

If you want use a yarp application to read the stream you need to:

1. install Gstreamer (see How to install Gstreamer )
2. compile yarp with “ENABLE_yarpcar_gstreamer” option enabled.
3. Run the server as mentioned above.
4. register a fake port to the yarp server, in this example called gstreamer_src:

   yarp name register /gstreamer_src  gstreamer  <SERVER_IP_ADRESS> <SERVER_IP_PORT>
   

5. set up your decoding pipeline by setting an environment variable with a string containing a string. For example on windows:

   set GSTREAMER_ENV=udpsrc port=<A_PORT_NUMBER> caps="application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264
   

   on linux:

   export GSTREAMER_ENV="udpsrc port=15000 caps=\"application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96\" ! rtph264depay ! h264parse ! avdec_h264"
   

6. run your application (e.g. yarpview, or your own application). This must be executed after setting the environment variable, i.e. the environment variable should be accessible to the executable.

   yarpview --name /view
   

7. connect client and server port using gstreamer carrier:

   yarp connect /gstreamer_src /view gstreamer+pipelineEnv.GSTREAMER_ENV
   

Now on yarp view you can see the following image, where in the bottom right box there is snow pattern.

Notes

• On Windows the property element uses ‘ instead of “
• Another completely different way to use Yarp with Gstreamer is to use Yarp Gstreamer Plugins. They also allow to feed a yarp image inside a gstreamer pipeline. See documentation: Gstreamer plugins

References

[1] Gstreamer documentation https://gstreamer.freedesktop.org/documentation/

[2] Gstreamer plugins https://gstreamer.freedesktop.org/documentation/plugins.html

[3] Libav library documentation https://www.libav.org/index.html

[4] Gstreamer Libav plugin https://gstreamer.freedesktop.org/modules/gst-libav.html

[5] h264 yarp plugin http://www.yarp.it/classyarp_1_1os_1_1H264Carrier.html