This talk will take the usual bird's eye look at what's been happening in and around GStreamer in the last release cycle(s) and look forward at what's next in the pipeline.
The AMD Alveo MA35D is a powerful media accelerator that also includes a dedicated AI processor. Upcoming release of MA35 GStreamer SDK will bring its AI features also to our favourite multimedia library. On a series of example pipelines, I will show how to use the new Alveo GStreamer elements to detect regions of interest, like faces or text, in video streams, leverage the object detection...
Mixing NVIDIA Jetson memory and the raw CUDA one is not straightforward. In this talk, we present the steps we follow to develop GStreamer plugins based on CUDA frame process. In order to work with this kind of memory, several data conversions and mappings are required. These mappings are expensive, so we based our filters on a custom CUDA mapped memory pool to get a noticeable speed-up.
RTP is a well known staple of the networked multimedia ecosystem and underpins many real time technologies such as, WebRTC, RTSP, VoIP, SIP, XMPP, and IPTV. GStreamer's existing RTP implementation is extremely flexible in supporting the myriad of use cases involved and it can be overwhelming to understand all intricacies. What if we could do better?
At IPS, we offer product development services, with one of our specialties being mulitmedia streaming services. We built a steaming service with video and 2 separately sourced audio inputs to be delivered realtime and synchronized to end users.
This talk will showcase some of the challenges we had with synchronizing multiple streams without having timestamps within the same time reference,...
Streaming from a PC a decent quality 3D video resulting from capturing a simple scene in a small room using a couple of cameras, though challenging, is getting possible. Using V3C standard, V-PCC extension for point cloud coding and GStreamer is compelling enough for adaptation. Because the standard’s reference implementation is not practically usable for real-time streaming, a more adequate...
At the 2023 conference a new rust webrtc crate was introduced. Since then the crate has continued to gain adoption and contributors, in this talk I will briefly present the new developments.
GPAC is an open-source multimedia framework focused on modularity and standards compliance. GPAC provides tools to process, inspect, package, stream, playback, and interact with media content. GPAC is best known for its wide MP4/ISOBMFF/CMAF/DASH/HLS/CENC capabilities and is popular among video enthusiasts, academic researchers, standardization bodies, and professional broadcasters.
In this...
With the growing success of machine learning (ML) language and speech models over the past four years, ML systems are behaving increasingly like human brains. These brains must be be fed with data, and GStreamer is the perfect framework to do it. But how do we remove obstacles to rapid adoption ?
ML research and commercial development takes place almost exclusively in the Python world, and...
Debugging pipelines and GStreamer applications can be a challenging task. For example if you run hundreds of pipelines daily, deployed on dozens of servers. Or when a rare issue seems to happen only once in a while in production.
This talk will present various tools to improve the observability of your pipelines:
- tracing-gstreamer to...
rtspsrc2 was merged into gst-plugins-rs early this year. In this presentation I'll talk about its architecture, its purpose, what it can do, and what it is supposed to do that the existing rtspsrc can never do.
You'll also hear about my experience writing a network source element in Rust.
In GStreamer, there may be multiple Windows plugins with the same purpose which can confuse developers, and there might be Windows-specific plugins that developers were not aware of.
In this presentation, recommended plugins on Windows will be discussed for each case.
GStreamer's GstStructure is using GQuarks for the string name and field names. GQuark is an API for uniquely representing a string by an integer, and while this property has some theoretical performance advantages it also comes with various disadvantages that cause problems in practice.
In this lightning talk I will explain these problems, and present a solution for this, which...
Do you have a messy room in the background and you are about to jump on a video call? No worries, use Silhouette!
Silhouette is a desktop application that connects to your camera via PipeWire and exposes a PipeWire virtual camera device, using a GStreamer pipeline in the middle to filter the camera stream. In its initial implementation, it uses an ONNX model to remove the background and...
Fluster is a testing framework written in Python for decoder conformance. I would like to share the great work and progress made since last year and the project roadmap. I will list some improvements, including more decoders and test suites, new installation methods, and more.
This will present some new elements written in rust that are being developed to leverage skia to draw inside GStreamer.
As part of a project sponsored by the Sovereign Tech Fund (STF), new GStreamer RTP payloaders and depayloaders for all relevant formats were developed in Rust, together with new base classes and RTP packet parsing / writing implementations. These can act as drop-in replacement for the old RTP payloaders and depayloaders.
In this lightning talk I will present the results of this work,...
We have been exploring using web frameworks (like d3 and three.js) to build out compelling graphics that we can stream using cefsrc.
An update to last year's talk 'Closed Captions: What GStreamer Can Do' outlining some of the new additions in GStreamer's closed caption story.
Describe the ongoing work to develop a simple H.264 encoder base class for VA and Vulkan
This presentation discusses effects of a Gstreamer version upgrade on Yocto-based application. Discussion items include :
system behavior changes observed
defaults that changed in gstreamer
adjustments we made for optimal outcomes.
elements/components that we expected and disappeared
This lightning talk will be a very short introduction to the GStreamer validation framework and will expose the new features that have been developed in the last few years to make testing and debugging GStreamer a more pleasant task.
We will describe how we use Cerbero at Fluendo, the problems we have faced, and how we plan to overcome them by upstreaming Cerbero's testing capabilities.
Once again, I would like to share the great work and huge progress made in the V4L2 GStreamer plugin and highlight the contributors working hard behind the scene.
In this lightning talk, I will showcase a new web browser source plugin for Windows
Here we show how to use the intersink/intersrc elements for creating independent GStreamer pipelines that can be connected and manipulated live.
Here we show how to use a gstreamer plugin element for inserting id3 tags inside transport stream
Some years ago I had hacked together a mechanism to record video clips from a live stream on-demand without having to re-encode them. While there are closed source offerings by Nvidia, etc for the same; I thought it shouldn't be so difficult to build an open source version of it. I'd like to share some of my learnings from the process of developing it in this talk.
When you split a pipeline using a tee, it means resynchronization two threads afterwards. In the specific case where one branch just produces new Meta, we created an alternative. Introducing the original buffer elements.
This is an update after a year from the GstWASM: GStreamer for the web introductory talk. We will present the advances made, the challenges we faced, the ported elements, and finally, samples! - something we missed last time.
This talk will present the use of GStreamer at the Society for Arts and Technology (SAT). It will begin with a brief overview of the SAT's history and its position in the technological ecosystem before delving into the various ways GStreamer is utilized within the organization. Specific applications include [Scenic][1], Switcher, Shmdata, and C++ development, as well as the use of Cerbero as a...
Since last year's presentation on the new playback element, more changes and improvements have happened.
This talk will show what has happened since then, including peripheral new elements and libraries.
QUIC is a general purpose and connection-oriented transport protocol like TCP, however, unlike TCP it’s an application level protocol that’s built on top of UDP. It is built with the aim to improve latency, bandwidth estimation, congestion and solve the head-of-line blocking that has been typical of TCP-based connections.
Being quite new, there are some interesting challenges in realizing...
GStreamer is a constantly evolving project. The advancements we continually make in the framework can certainly improve support for your embedded Linux hardware, even if it's from a decade ago. In this presentation, Nicolas will walk you through his journey streaming a Qt6 application over the network using a H.264 hardware encoder on an i.MX6 Linux platform in a fully zero-copy fashion. This...
Raven AI Engine is a new cross-platform framework for developing AI-powered computer vision projects with full hardware acceleration across a wide range of edge and desktop devices. Raven provides ready-to-use solutions with pre-trained models for common vision tasks like tracking, background removal, or face anonymization, enabling developers to integrate these solutions into their...
The WebKit WPE and GTK ports are aiming to leverage GstWebRTC as their WebRTC backend. Over the years we made progress towards this goal both in WebKit and in GStreamer. During this talk we will present the current integration status of GstWebRTC in WebKit and the achievements accomplished since GStreamer Conference 2023. We will focus on new features such as simulcast, SFrames, and provide an...
Over the past 2-3 releases, a number of improvements have been done to plugins that use Apple-specific APIs, such as AVFoundation, CoreAudio, VideoToolbox, etc. There have also been build-system improvements, including continuous integration.
The out-of-the-box experience on macOS is much improved now, although there is still work to be done.
Attend to get an overview of the improvements...
A long time ago I joined a software agency working on a bunch of multimedia projects. Working for startups, we needed to iterate fast and deliver POCs validating their ideas. That's when the concept was born: a tool that would be powerful and customizable, yet written in a high-level language and possibly forgiving errors. That's how the [Membrane Framework][1] came to life, with API inspired...
Matching different clocks is a common problem in real-time audio processing — audio may originate from one device, and may need to be rendered to a different device driven by a different clock, possibly running at a slightly different rate. Managing these clock rate differences is an important factor in avoiding underruns and overruns in real-time audio applications.
This problem is...
Building robust audio policies on embedded systems that integrate well with upstream APIs and components such as PipeWire and PulseAudio has been a challenging task for a long time. In many implementations, custom hardware DSP management components are built, bypassing existing upstream functionality, making integration and maintenance harder than they need to be. In this talk, George will...
It has been 5 years now since the last update at the conference about the status of Rust usage in GStreamer. Since then a lot has happened: a lot of new plugins and applications were written in Rust, some Rust code has been added to GStreamer core, the Rust plugins are shipped with the GStreamer binaries for all platforms, and even about 25% of all commits during the 1.24 release cycle were in...
GStreamer's cross-platform support and robust RTP stack make it an excellent choice for building audio streaming applications. We leveraged these strengths to develop an AES67-compliant streaming application on Windows, designed to interoperate seamlessly with Dante devices.
We came across a number of interesting challenges in the process, some specific to being able to send packets every...
In this presentation we will dive into new unixfdsrc/unixfdsink elements and GstMeta serialization, how they are used to build zero-copy multi-process pipelines for Netflix's testing tools.
State of the art of the present and coming Vulkan video elements. We'll talk about architecture, codecs and challenges to achieve vulkan support ...
Urban Air Mobility (UAM) is a transportation system that uses electric vertical takeoff and landing (eVTOL) aircraft to transport people or cargo within urban areas, helping to reduce congestion on the ground. Aircraft operate in a busy urban environment, and may be semi or fully autonomous. In this use case there is a need to automate communication between aircraft and ground to reduce...
During different consultancy services we had some tasks to achieve high level of stability for some Windows GStreamer applications. We had to resolve race conditions, memory leaks and D3D11-specific issues, such as crashes in the driver. This talk aims to summarize what we have learned from those experiences.
Spoiler: The tests were helpful here.
This talk is the usual "video editing stack update". I will present the work that has been done in and around GES and the new components that have been developed to make building video editing applications and web services more efficient and simpler.
HTTP Live Streaming (HLS), a widely adopted protocol for live video streaming, and has been supported by GStreamer for a long time. HLS enables streaming of multiple formats and bitrates, allowing players to dynamically adjust their streaming quality based on network conditions for ensuring optimal viewer experience.
The HLS specification introduced alternate renditions and variant streams,...
ICE underpins all WebRTC connections and provides peers the ability to discover the most efficient path for transferring data. We will take a brief look into the steps involved in the ICE process and a more in depth look into how a sans-IO implementation might be implemented using the example of librice.
Sometimes, when I'm not working on GStreamer, I work on open source VR projects, and sometimes when I work on open source VR projects - I use GStreamer!
This presentation will talk a bit about the state of open source VR support, how GStreamer relates to it, some interesting use cases for GStreamer in that space, and things that can be improved.
This talk will present recent progress on proper color handling in the GTK toolkit and surrounding topics, such as graphics offloading. Taken together these enable state-of-the-art Video playback in GTK apps (including HDR support) using gstreamer.
The talk will outline what is available in the imminent GTK 4.16 release, as well as our plans for the near future.
Since Direct3D11 support was added in 1.18, GStreamer’s Windows graphics and hardware acceleration features have been greatly improved up to a level of GStreamer's Linux support, or even better than other OS supports in some aspect.
Then Direct3D12 based features were introduced in GStreamer 1.24 which gives another chance to improve performance and expand feature set.
In this...
