GstInference - Bringing AI to GStreamer

AI is everywhere. Well, maybe this is not entirely true yet, but will be in a few years from now. I heard a prediction that 1,000,000,000 video cameras will be installed by 2020. At RidgeRun we've been working hard to join this revolution, and we want to share some of the technology we're developing with the community.

The truth is that the initial face-off with AI is never easy. A quick Google search reveals a handful of frameworks available to implement a neural network. For the sake of this example suppose you choose TensorFlow. You grasp the whole graph oriented processing paradigm, and finally understand and successfully run the image classification example you are trying out. Cool! This is exactly what your product needs to do. Almost.

Now to fit all the pieces together. Here's a small list of the most immediate tasks for your project:

• Modify the example to receive images from the camera, rather than PNG images.
• Scale down the 1920x1080 camera images to 224x224 needed by the neural network.
• Render a label in the image according to the network prediction.
• Save resulting images in a MP4 file.

If you are familiar with GStreamer, you immediately envisioned a nice, simple pipeline. Something like the following:

Putting it this way doesn't feel that overwhelming after all.

That's exactly where GstInference fits in this whole scenario. It's a set of GStreamer elements that execute AI models on a video stream. We've taken care of all the complexities and boilerplate code so that all the framework details are hidden, and you just link in the inference element as any other. And not only TensorFlow, but Caffe, NCSDK, TensorRT, and others as well (yet to come).

Here's how the whole pipeline looks altogether:

Note how the modular architecture of GStreamer allows you to make the most of the available hardware accelerators in your platform. The scaling could be done using the Image Signal Processor, the encoding using the Video Processing Unit and the inference using the GPU. Let's take it one step further: using the exact same pipeline you can migrate the inference processing from the GPU to the Tensor Processing Unit by simply changing the backend property in GstInference from TensorFlow to TensorRT, for example.

RidgeRun's main design goal is simplicity for the user.

We have made an early release available. Feel free to give it a try and give us your feedback. We'd love to hear from you!

https://developer.ridgerun.com/wiki/index.php?title=GstInference
https://github.com/RidgeRun/gst-inference/