GStreamer Conference 2024

6 Oct 2024, 19:00 → 10 Oct 2024, 17:00 America/New_York

Concordia University Conference Centre

October 7-10 | Montréal, Canada

The schedule timezone of the conference is Eastern Daylight Time (EDT, UTC-04:00, Canada/Eastern timezone), unless you set "Use my timezone" setting in your user preferences along with your current timezone.

Venue

The in-person venue is the Concordia University Conference Centre, located within the John Molson School of Business, in Montréal, Canada.

Concordia University Conference Centre
1450 Guy St., Montreal, Quebec, Canada, H3H 0A1

 

Conference Website

For more information and the latest updates please visit the main GStreamer Conference 2024 website.

You can also follow us @GStreamer on Twitter or Mastodon for the latest updates and to stay connected.

Sunday, 6 October

19:00 → 23:00 Social Events: Welcome drinks/food at McKibbin’s (Bishop St)

Monday, 7 October

08:00 → 09:23 Registration: Registration and Continental Breakfast

09:25 → 09:28 Opening Session

09:30 → 13:10 Room 1

09:30 GStreamer State of the Union

This talk will take the usual bird's eye look at what's been happening in and around GStreamer in the last release cycle(s) and look forward at what's next in the pipeline.

Speaker: Tim-Philipp Müller (Centricular)

GStreamer Conference 2024 - State of the Union - Tim-Philipp Müller.pdf

GStreamer State of the Union 2024 (html)

10:25 TV channel in the cloud with GStreamer

Over the last 10 years we (veset.tv) have been developing a cloud-based video player for linear TV channels. Our player features multiple streaming protocols and encoders, live and file-based inputs, overlay graphics, different types of subtitles, SCTE signaling and many other features.

Core of the player is a GStreamer pipeline where new source elements and branches are dynamically added and destroyed to achieve 24/7 playout.

I'm going to talk about how we structure the large pipeline, how we add and remove new elements, how and why we separate it into multiple processes, how we integrate it into a C++ application.

Speaker: Jurijs Satcs (veset.tv)

veset-gstreamer-2.pptx

11:15 Coffee break

11:30 GStreamer RTP sessions in Rust

RTP is a well known staple of the networked multimedia ecosystem and underpins many real time technologies such as, WebRTC, RTSP, VoIP, SIP, XMPP, and IPTV. GStreamer's existing RTP implementation is extremely flexible in supporting the myriad of use cases involved and it can be overwhelming to understand all intricacies. What if we could do better?

Speaker: Matthew Waters (Centricular)

2024 GStreamer RTP Sessions in Rust - Matthew Waters.pdf

12:05 Streamlining Video Presentation and Recording with GStreamer-Based Processing

In this presentation, I will summarize a library I am developing for video presentation and recording applications that assists users in creating content for video presentations. A key component of this system is GStreamer, which plays an integral role in real-time video and audio processing.

Key Features:
The app streams media to Amazon ECS during recording for real-time GStreamer-based processing, offering the following features:

• Conversion: Transforms video from webm format to HLS.
• Thumbnails Generation: Automatically creates thumbnail images from the video stream.
• Waveform Generation: Produces waveform visualizations to enhance the user experience.
• Audio Transcription: Converts spoken audio into text for accessibility and indexing.
• Notably, all of this processing is done 100% in memory, without utilizing the local filesystem.

Advanced Editing Capabilities:
Users can edit and enhance multiple recordings by:

• Mixing recordings together,
• Trimming and adding images or other visual elements,
• Combining and editing them to form a final presentation.

This is accomplished using GStreamer Editing Services (GES), allowing users to create structured presentations composed of logical chapters, enhancing both functionality and narrative flow.

Technical Implementation:
GStreamer functionality is encapsulated in a C++ library, integrated with a thin C layer, and exposed to Go, providing users with an easy-to-use interface.

Challenges and Solutions:
During development, I encountered and reported several GStreamer-related issues, many of which have since been resolved. Additionally, the C++ library includes simple yet highly useful wrappers that streamline automatic resource management, further improving and simplifying the development.

Conclusion:
In this presentation, I will demonstrate GStreamer's powerful capabilities within a modern video presentation app, showcasing how automatic resource management can simplify development and enhance overall workflow efficiency.

Speaker: Tomasz Mikołajczyk

12:40 Perfect Harmony: How we synchronized Audio and Video from Multiple Sources using GStreamer for Seamless Real-Time Streaming

At IPS, we offer product development services, with one of our specialties being mulitmedia streaming services. We built a steaming service with video and 2 separately sourced audio inputs to be delivered realtime and synchronized to end users.
This talk will showcase some of the challenges we had with synchronizing multiple streams without having timestamps within the same time reference, and without access to an NTP server.
We will talk about the solution we created using Gstreamer, what helped us, and parts of system behavior we still don’t fully understand.

Speakers: Tokunbo Quaye, Michael Yackavage

GstreamerPresoPerfectHarmony.pdf

10:25 → 13:10 Room 2

10:25 AI-assisted video encoding with AMD Alveo MA35D

The AMD Alveo MA35D is a powerful media accelerator that also includes a dedicated AI processor. Upcoming release of MA35 GStreamer SDK will bring its AI features also to our favourite multimedia library. On a series of example pipelines, I will show how to use the new Alveo GStreamer elements to detect regions of interest, like faces or text, in video streams, leverage the object detection data to improve encoding quality on a frame-by-frame basis, or retrieve the data from the hardware for further inspection with the help of GstAnalytics.

Speaker: Jakub Adam (Collabora Ltd.)

GST2024_MA35_AI_Encoding.pdf

10:50 GStreamer NVIDIA memory in Jetson: looking for an efficient way to process video frames when handling raw CUDA kernels

Mixing NVIDIA Jetson memory and the raw CUDA one is not straightforward. In this talk, we present the steps we follow to develop GStreamer plugins based on CUDA frame process. In order to work with this kind of memory, several data conversions and mappings are required. These mappings are expensive, so we based our filters on a custom CUDA mapped memory pool to get a noticeable speed-up.

Speaker: Diego Nieto Munoz (Fluendo)

GstConf2024 - NVIDIA Jetson talk.pdf

11:15 Coffee break

11:30 Real time volumetric video streaming

Volumetric video streaming offers users to indulge in immersion with the subject which is streamed onto user's VR/AR headset, as users can watch the subject from all possible angles, that too in real time. Probably this takes us one step closer to the Metaverse?
This talk includes topics which will cover how we achieved streaming volumetric video content in real time between two sites. The talk begins with a brief introduction of Volumetric Video and V3C (Visual Volumetric Video-based Coding) standard then it touches upon the hardware setup we use to capture the content and the software architecture (mostly consists of GStreamer pipelines) we use to materialize the goal of streaming volumetric video content from a computer and eventually rendering it on VR/AR headset. Potentially there will be a demo also in the end of the talk, if time permits.

Speaker: Sudarshan Bisht

12:05 AMD's Video Processing Engine

Newer AMD APUs come with a fixed-function block called VPE (video processing engine) for processing video data. It is a memory-to-memory copy engine that can scale, color space convert, and perform complex color transformations using a 3D LUT, such as gamut and tone mapping of HDR content. The functionality is exposed via vaapi's video processing (vp) API. I will talk about what VPE does, how we can use it today, and the power savings we're seeing. I will also talk about opportunities to increase the usage of the vaapi-vp API to take advantage of VPE in more use-cases.

Speaker: Harry Wentland (AMD)

20240912 VPE v0.8 - final.pdf

12:40 Open Standards for Media including ST 2110 and IPMX

Proposed information session on the history of VSF TR-03 which became ST 2110; additional VSF TR-10 specifications with expansions for Pro AV, IT, and consumer electronics (also known as "IPMX"); and explanation of the context of various media over IP protocols. Discussion of 3 main categories of equipment: professional production equipment, AV signal routing and management, and PC/IT. Discussion of how it has been easy/cheap to covert baseband signals (HDMI, SDI, Display Port, etc.) between each other and how connecting those 3 main categories of equipment was easy, but how the migration to AV-over-IP is "different" for each of those 3 main categories of equipment and connecting them is going to require new knowledge, equipment, and standards. Context of IPMX, ST 2110, NDI, Dante A/V, SDVoE, Aspeed, and other protocols. Discussion will culminate in how new PC-based equipment to deal with these opportunities has arrived and how G-Streamer will be used extensively as a hook-up to popular PC-based applications.

Speaker: Dan Maloney

13:15 → 14:25 Lunch (at the venue)

14:30 → 16:30 Room 1

14:30 New developments with the WebRTC crate

At the 2023 conference a new rust webrtc crate was introduced. Since then the crate has continued to gain adoption and contributors, in this talk I will briefly present the new developments.

Speaker: Mathieu Duponchelle (Centricular)

14:55 LiveKit Support for GStreamer WebRTC Elements

The GStreamer Rust plugins include generic source and sink elements for unidirectional WebRTC communications with no application code necessary in many cases. However, each WebRTC server has its own signalling protocol and to interact with any specific WebRTC implementation, a client must be implemented to perform WebRTC signalling over that protocol. A signalling protocol client as well as both a source and sink element have been implemented that interoperate with the LiveKit WebRTC server.
This presentation will discuss the implementation of both the source and sink elements and the complexities associated with them. Furthermore, it will introduce some extra features specific to LiveKit that have been implemented in addition to basic WebRTC communications.
The first feature added was video simulcasts which allows the sink to send multiple versions of the same video track to the LiveKit server at multiple qualities while the LiveKit server will choose what version to send to each receiver in the same room based on each receiver's network connection quality and preferences. The presentation will discuss how it was implemented as well as the internal design changes needed to support this feature.
Also, support for LiveKit's end-to-end encryption scheme was implemented for use cases that do not trust the key exchange and point-to-point encryption associated with a WebRTC Selective Forwarding Unit. The presentation will discuss the cryptographic filter elements, how they are integrated with the webrtcsrc and webrtcsink elements, and the codec-specific challenges encountered during development of this feature.

Speaker: Jordan Yelloz

jyelloz-gst-2024-livekit-webrtc.pdf

15:20 What's going on in my pipelines?

Debugging pipelines and GStreamer applications can be a challenging task. For example if you run hundreds of pipelines daily, deployed on dozens of servers. Or when a rare issue seems to happen only once in a while in production.

This talk will present various tools to improve the observability of your pipelines:

• tracing-gstreamer to generate logs with more context.
• Grafana loki to gather and search through GStreamer logs.
• Sentry to automatically report errors in your pipelines.

Speaker: Guillaume Desmottes

15:45 Gst-Analytics: Advancements in Machine Learning and Analytics Pipeline

With the release of version 1.24, GStreamer introduced native support for advance analytics and complex machine learning pipeline. We haven't stopped there; we've been expanding on this powerful analytics framework we unveiled last year. In this presentation we'll take you on a journey through the latest advancements showcasing improvements in analytics meta and new analytics elements. Finally, we will dive into the challenges of auto-plugging elements that exchange tensors such as analytics elements. These changes will facilitate analytics pipeline building to a new level.

Speaker: Daniel Morin (Collabora)

gst-conf-2024-analytics_113711.pdf

16:10 Which plugin should I use on Windows?

In GStreamer, there may be multiple Windows plugins with the same purpose which can confuse developers, and there might be Windows-specific plugins that developers were not aware of.

In this presentation, recommended plugins on Windows will be discussed for each case.

Speaker: Seungha Yang (Centricular)

14:30 → 16:30 Room 2

14:30 Experiences during the development of point cloud encoding for real-time 3D video streaming using GStreamer

Streaming from a PC a decent quality 3D video resulting from capturing a simple scene in a small room using a couple of cameras, though challenging, is getting possible. Using V3C standard, V-PCC extension for point cloud coding and GStreamer is compelling enough for adaptation. Because the standard’s reference implementation is not practically usable for real-time streaming, a more adequate solution is required. This solution applies certain simplifications and complexity reductions, without compromising quality.
This talk will include topics related to the development of the encoder using a single GStreamer pipeline run on a PC. It will begin by shortly introducing the V-PCC encoding. Next it will cover how a V-PCC encoder developed separately in C++ can be used in GStreamer through a plugin. The talk will then cover the pipeline in general: pre-recorded file-based sources (from live cameras), CPU-based point cloud reconstruction and CPU based V-PCC encoding, GPU accelerated video encoding, and then GPU accelerated video decoding, GPU based V-PCC decoding, point cloud reconstruction, and finally rendering all using OpenGL.
There could be some demo at the end of the talk if time permits.

Speaker: Mr Jozsef Szabo (Nokia Technologies Oy)

14:55 GStreamer Meets GPAC: Enhancing GStreamer's Capabilities through GPAC Integration

GPAC is an open-source multimedia framework focused on modularity and standards compliance. GPAC provides tools to process, inspect, package, stream, playback, and interact with media content. GPAC is best known for its wide MP4/ISOBMFF/CMAF/DASH/HLS/CENC capabilities and is popular among video enthusiasts, academic researchers, standardization bodies, and professional broadcasters.

In this presentation I’ll be discussing our ongoing efforts to bring interoperability between GStreamer and GPAC. We believe a seamless integration of GPAC within GStreamer would enhance GStreamer by adding the powerful muxing capabilities of GPAC. We have already seen some early successes and are currently working on making our integration robust and reliable.

I’ll begin my talk by explaining how GPAC filter-based architecture works, focusing on how filters connect, how negotiations are handled, and how these processes are similar to or differ from those in GStreamer. Next, I’ll showcase some example files we’ve created through this integration and discuss the challenges we’ve encountered during the process. I’ll wrap up by addressing the current hurdles we are working to overcome and providing examples of how this integration could benefit those who want to leverage the strengths of both software.

At Motion Spell, we believe this integration will pave the way for further collaboration between the GStreamer and GPAC communities. The enhanced capabilities of our multiplexer have the potential to fuel a wide range of applications that already use GStreamer.

Speaker: Deniz Ugur (Motion Spell)

15:20 GStreamer Nervous System for AI Brain : Introducing Python Analytics

With the growing success of machine learning (ML) language and speech models over the past four years, ML systems are behaving increasingly like human brains. These brains must be be fed with data, and GStreamer is the perfect framework to do it. But how do we remove obstacles to rapid adoption ?

ML research and commercial development takes place almost exclusively in the Python world, and the current dominant ML toolkit is PyTorch. PyTorch has succeeded for a number of reasons, including it's simplicity, strong community, rapid innovation, broad hardware support and ease of integration with the vast Python world. Over the past year, Pytorch has introduced a compile feature, a Just In Time (JIT) compiler that dynamically optimizes code for the current target hardware. Performance improvements are astonishing - in some cases compiled PyTorch is faster than TensorRT on Nvidia hardware. But compile is not limited to just one hardware platform.

Collabora was the first to upstream neural network support into GStreamer via ONNX analytics elements. ONNX
is a cross-platform inference engine whose C++ API has been integrated to enable new object detection and segmentation elements. We have also introduced the analytics meta-data framework, a framework for flexibly storing meta data generated from AI models, and the relationships between different meta data..

We now introduce a suite of GStreamer custom elements and classes written in Python that allow users to easily and rapidly support all the latest AI models, using native PyTorch support. We provide a package with base classes supporting models for audio, video and Large Language Models (LLMs). The package works with the latest GStreamer version and inter-operates with the new meta data framework. Performance enhancements such as batching and managing device memory buffers are available out of the box.

In addition to the base classes, we also provide elements that perform object detection, tracking, speech to text, text to speech and LLM chat-bot features. There is a Kafka element that can send meta data to a Kafka server.

The list of elements continues to grow rapidly, supporting any of the many Hugging Face models with ease. Our goal is no less than to provide a complete upstream solution for GStreamer Analytics via PyTorch, making upstream GStreamer the number one multimedia framework for machine learning.

Speaker: Aaron Boxer (Collabora Inc.)

gstreamer_nervous_system_ai_brain.pdf

15:45 Learnings of building a scaleable webrtc media pipeline in rust using gstreamer (cancelled)

I work at a company called Dyte. We are an audio/video communication sdk company. I mainly work at the recordings team. At our company we have been experimenting with gstreamer for almost a year and making a new media pipeline that powers not only powers our new recording infrastructure but also acts as a source for other services that wanna consume media of the room eg. some of our AI stuff. I want to talk about my experience building that, things that helped making it more scalable and robust.

Speaker: Mr ramyak mehra

16:10 A New RTSP Source Element Written in Rust

rtspsrc2 was merged into gst-plugins-rs early this year. In this presentation I'll talk about its architecture, its purpose, what it can do, and what it is supposed to do that the existing rtspsrc can never do.

You'll also hear about my experience writing a network source element in Rust.

Speaker: Nirbheek Chauhan (Centricular Ltd)

A New RTSP Source Element Written in Rust

A New RTSP Source Element Written in Rust.pdf

16:35 → 16:55 Coffee break

17:00 → 19:00 Lightning Talks

17:00 GQuark in GStreamer structures — What nonsense!

GStreamer's GstStructure is using GQuarks for the string name and field names. GQuark is an API for uniquely representing a string by an integer, and while this property has some theoretical performance advantages it also comes with various disadvantages that cause problems in practice.

In this lightning talk I will explain these problems, and present a solution for this, which doesn't require API changes and doesn't have any relevant performance impact.

Speaker: Sebastian Dröge (Centricular Ltd)

presentation.pdf

17:05 Integration testing video with Test Containers by example

“Test Containers” is a framework agnostic library, implemented in several languages, allowing the tester to run most containers as part of their unit or integration testing, lessening the need of mock code.
The Gstreamer testing framework is focussed on testing the pipeline itself, but frequently video workflows comprise of executables communicating over a network. Using test containers we are able to test against real world scenarios for live streaming.
In this lightning talk we will focus on scenarios you are likely to encounter in the real work, working through examples, in order to test your video pipeline in an end to end fashion.

Speaker: Johannes Nel

testcontainers.pptx

17:10 Silhouette, the ML/PipeWire-powered virtual camera device app

Do you have a messy room in the background and you are about to jump on a video call? No worries, use Silhouette!

Silhouette is a desktop application that connects to your camera via PipeWire and exposes a PipeWire virtual camera device, using a GStreamer pipeline in the middle to filter the camera stream. In its initial implementation, it uses an ONNX model to remove the background and replace it with a single color, making the camera stream be a robust cut-out of your silhouette.

Speaker: George Kiagiadakis (Collabora)

GstConf2024_Silhouette.pdf

17:15 Last year's updates in Fluster.

Fluster is a testing framework written in Python for decoder conformance. I would like to share the great work and progress made since last year and the project roadmap. I will list some improvements, including more decoders and test suites, new installation methods, and more.

Speaker: Rubén Gonzalez

FluendoFlusterGstConf2024LT.pdf

17:20 A new GStreamer plugin to leverage the skia 2D drawing library

This will present some new elements written in rust that are being developed to leverage skia to draw inside GStreamer.

Speaker: Thibault Saunier (Igalia)

GStreamer skia plugin

17:25 New RTP payloaders & depayloaders in Rust

As part of a project sponsored by the Sovereign Tech Fund (STF), new GStreamer RTP payloaders and depayloaders for all relevant formats were developed in Rust, together with new base classes and RTP packet parsing / writing implementations. These can act as drop-in replacement for the old RTP payloaders and depayloaders.

In this lightning talk I will present the results of this work, together with some details about the implementation, current status and future plans.

The parts of the STF project included a new RTP manager (rtpbin / rtpjitterbuffer) implementation and a new RTSP source, which are introduced in other talks.

Speaker: Sebastian Dröge (Centricular Ltd)

presentation.pdf

17:30 Using cefsrc for real time graphics production

We have been exploring using web frameworks (like d3 and three.js) to build out compelling graphics that we can stream using cefsrc.

Speaker: Sid Sethupathi Sid Sethupathi

gst-2024-cefsrc-graphics-lightning-talk.pdf

17:35 Closed Captions: Additional Things GStreamer Can Do

An update to last year's talk 'Closed Captions: What GStreamer Can Do' outlining some of the new additions in GStreamer's closed caption story.

Speaker: Matthew Waters (Centricular)

2024-closedcaptions Additional things that GStreamer can do - Matthew Waters.pdf

17:40 Is an H264 encoder base clase for hardware accelerated API possible?

Describe the ongoing work to develop a simple H.264 encoder base class for VA and Vulkan

Speaker: Victor Manuel Jáquez Leal (Igalia)

H264_encode_base_class.pdf

17:45 Effects of a GStreamer version upgrade on Yocto-based application.

This presentation discusses effects of a Gstreamer version upgrade on Yocto-based application. Discussion items include :
system behavior changes observed
defaults that changed in gstreamer
adjustments we made for optimal outcomes.
elements/components that we expected and disappeared

Speaker: Michael Yackavage

GstreamerUpgradeWithYoctoLightningTalk.pdf

17:50 GstVA updates

Past, present and future of GstVA elements

Speaker: Victor Manuel Jáquez Leal (Igalia)

GstVA_Updates_2024.pdf

17:55 GStreamer plugin updates on webOS as a platform

LGE has a software platform called webOS, which can be experienced mostly in television made by LGE.
Last year, a GStreamer plugin was introduced to contain various SoC vendor's plugins such as decoder and sink.
This time, in order to expand webOS to various devices, as a platform,
we would like to introduce a concept stucture that is under developing and is able to apply platform specific properties or signals into various SoCs.
It has been developed based on V4L2 based plugins, but we have considered to be applicable for non-V4L2 based plugins.

Speaker: Seungwook Cha

Seungwook Cha - GStreamerConference2024_LightningTalk_20241006_1.pdf

18:00 An update on GStreamer validate

This lightning talk will be a very short introduction to the GStreamer validation framework and will expose the new features that have been developed in the last few years to make testing and debugging GStreamer a more pleasant task.

Speaker: Thibault Saunier (Igalia)

GStreamer validate update

18:05 Fluendo and Cerbero, testing challenges

We will describe how we use Cerbero at Fluendo, the problems we have faced, and how we plan to overcome them by upstreaming Cerbero's testing capabilities.

Speaker: Jorge Zapata

Fluendo and Cerbero.pdf

18:10 Yet another V4L2 update

Once again, I would like to share the great work and huge progress made in the V4L2 GStreamer plugin and highlight the contributors working hard behind the scene.

Speaker: Nicolas Dufresne (Collabora)

Nicolas - V4L2 Update.pdf

18:15 A new Windows web browser source plugin - webview2

In this lightning talk, I will showcase a new web browser source plugin for Windows

Speaker: Seungha Yang (Centricular)

WebView2.pdf

18:20 Seeking Perfection: Challenges and Solution attempts in Implementing HLS Stream Seeking with RTP Synchronization Using GStreamer.

At IPS, we offer product development services, with one of our specialties being mulitmedia streaming services. We are attempting to add a “seek” feature for an existing non-realtime HLS streaming service, with RTP Synchronization of 2 stream listeners
We will discuss the challenge of having the “seek” operation succeed in the Sender Pipeline but receiver pipelines stall due to RTP Jitter buffer errors. We will discuss the work-around solution that we created with Gstreamer and some of the challenges that we’re still encountering.

Challenge:
Failed utilization of RTSP - causes QOS errors in video decoder - thus receiver playback gets stalled until QOS dropped.

Attempted Solution :
Introduced a 3rd pipeline that precedes a sender pipeline that does 2 seeks. Seek is successful, except if within 10 secs of previous playback location

Suggestions for Improvement :
Make Seek feature in RTSP send a flush message to the receiver pipeline. Currently in the 1.18 version, the reset message gets dropped out of the udpsink of the sender pipeline and never processed in the RTP receiver pipeline.

Speaker: Tokunbo Quaye

GstreamerSeekLightningTalk.pdf

18:25 On using Intersink/intersrc for live pipeline manipulation

Here we show how to use the intersink/intersrc elements for creating independent GStreamer pipelines that can be connected and manipulated live.

Speaker: Juan David Adarve Bermudez (Genius Sports)

Encoder hot-swapping

GStreamer2024_presentation.pdf

18:30 Inserting ID3 tags in Transport stream

Here we show how to use a gstreamer plugin element for inserting id3 tags inside transport stream

Speaker: Sebastian Benitez

Gstreamer_conf_ID3.pdf

18:35 Is it easy to write a video sink plugin 😋?

Some years ago I had hacked together a mechanism to record video clips from a live stream on-demand without having to re-encode them. While there are closed source offerings by Nvidia, etc for the same; I thought it shouldn't be so difficult to build an open source version of it. I'd like to share some of my learnings from the process of developing it in this talk.

Speaker: Nikhil Krishna Renganathan Suresh

VideoSink.pptx

18:40 An alternative to tee for simple cases

When you split a pipeline using a tee, it means resynchronization two threads afterwards. In the specific case where one branch just produces new Meta, we created an alternative. Introducing the original buffer elements.

Speaker: Olivier Crête (Collabora)

Olivier Crête - Original Buffer .pdf

19:30 → 23:30 Social Events: Welcome Party (food and drinks)

Tuesday, 8 October

09:00 → 09:35 Continental Breakfast

09:40 → 13:00 Room 1

09:40 Scenic and GStreamer on stage : collaborative theater adventures at the Society for Arts and Technology

This talk will present the use of GStreamer at the Society for Arts and Technology (SAT). It will begin with a brief overview of the SAT's history and its position in the technological ecosystem before delving into the various ways GStreamer is utilized within the organization. Specific applications include Scenic, Switcher, Shmdata, and C++ development, as well as the use of Cerbero as a build system for our software platform.

The talk aims to provide insights into the diverse applications of GStreamer at the SAT, highlighting both the benefits and challenges encountered. It will offer valuable information for developers and users interested in leveraging GStreamer for multimedia projects.

The presentation will then explore use cases such as SRT streaming experimentation with OSM, explorations with VDO.ninja, Raspberry Ninja, WHIP/WHEP tests, and webrtcdsp based AEC. It will conclude by looking towards the future with ongoing and planned projects, including a Python WebRTC prototype using latest rswebrtc and gstwebrtc-api development, Scenic 5 development, and investigations into porting our tools to embedded platforms with support for hardware encoders and decoders.

Speakers: Olivier Gauthier (Society for Arts and Technology) , Jean-Michaël Celerier (Society for Arts and Technology)

Scenic and GStreamer on stage _ collaborative theater adventures at the Society for Arts and Technology_no_video.pdf

10:15 Playback support, 2024 update

Since last year's presentation on the new playback element, more changes and improvements have happened.

This talk will show what has happened since then, including peripheral new elements and libraries.

Speaker: Edward Hervey (Centricular Ltd)

10:50 From GL to your Encoder, Zero-Copy Made Possible

GStreamer is a constantly evolving project. The advancements we continually make in the framework can certainly improve support for your embedded Linux hardware, even if it's from a decade ago. In this presentation, Nicolas will walk you through his journey streaming a Qt6 application over the network using a H.264 hardware encoder on an i.MX6 Linux platform in a fully zero-copy fashion. This is made possible with the new DMABuf caps negotiation and recent progress in GStreamer GL layer.

Speaker: Nicolas Dufresne (Collabora)

Nicolas Dufresne - GStreamer Conference 2024 - GL Encoding.pdf

11:25 Coffee break

11:40 GstWebRTC / WebKit state of the union

The WebKit WPE and GTK ports are aiming to leverage GstWebRTC as their WebRTC backend. Over the years we made progress towards this goal both in WebKit and in GStreamer. During this talk we will present the current integration status of GstWebRTC in WebKit and the achievements accomplished since GStreamer Conference 2023. We will focus on new features such as simulcast, SFrames, and provide an overview on spec compliance, tests coverage and dedicated use-cases beyond pure video-conferencing scenarios.

Speakers: Philippe Normand (Igalia) , Carlos Bentzen (Igalia)

gstwebrtc_2024.pdf

12:15 It’s time for some clock rate matching

Matching different clocks is a common problem in real-time audio processing — audio may originate from one device, and may need to be rendered to a different device driven by a different clock, possibly running at a slightly different rate. Managing these clock rate differences is an important factor in avoiding underruns and overruns in real-time audio applications.

This problem is partially addressed in GStreamer via GstClock's synchronisation and GstAudioBaseSink's clock slaving mechanisms. As the new de-facto Linux audio subsystem, PipeWire also implements clock rate matching and adaptive resampling, building on work done in the JACK community over the last decade and more.

In this talk we will lay out the basics of the problem space, and then review the state of the art in both PipeWire and GStreamer. Contrasting the approaches taken should be interesting in itself, and maybe we'll find some new ideas to be inspired by.

While the subject matter is a little bit in the weeds, there will be a gentle introduction for audience members with varying levels of familiarity with the topic.

Speaker: Arun Raghavan (Asymptotic Inc.)

time-for-rate-matching-print.pdf

09:40 → 13:00 Room 2

09:40 Gst.WASM Launched

This is an update after a year from the GstWASM: GStreamer for the web introductory talk. We will present the advances made, the challenges we faced, the ported elements, and finally, samples! - something we missed last time.

Speaker: Jorge Zapata

10:15 State of QUIC in GStreamer

QUIC is a general purpose and connection-oriented transport protocol like TCP, however, unlike TCP it’s an application level protocol that’s built on top of UDP. It is built with the aim to improve latency, bandwidth estimation, congestion and solve the head-of-line blocking that has been typical of TCP-based connections.

Being quite new, there are some interesting challenges in realizing QUIC concepts and application layer protocols in GStreamer. In this talk, we cover:

• Some basics of QUIC
• A review of the current state and challenges of the QUIC elements
• An overview of media-related application layer protocols (RTP over QUIC, Media over QUIC, etc.)
• Potential future work in this space

Speaker: Mr Sanchayan Maity (asymptotic.io)

State of QUIC in GStreamer

10:50 Raven AI Engine: a framework to develop AI computer vision GStreamer elements

Raven AI Engine is a new cross-platform framework for developing AI-powered computer vision projects with full hardware acceleration across a wide range of edge and desktop devices. Raven provides ready-to-use solutions with pre-trained models for common vision tasks like tracking, background removal, or face anonymization, enabling developers to integrate these solutions into their applications quickly.

In this talk, we will present an overview of the engine and the existing vision tasks provided, describing the models and pre and post-processing modules in use. We will also present the integration layers with GStreamer and the internal framework we developed to create GStreamer elements.

We will finish the presentation by showcasing some of the projects into which we have integrated Raven with GStreamer, including a Sports Analysis SDK for LongoMatch and virtual avatar software for gaming.

Speaker: Andoni Morales Alastruey (Fluendo)

11:25 Coffee break

11:40 macOS and iOS Support: History and Future Goals

Over the past 2-3 releases, a number of improvements have been done to plugins that use Apple-specific APIs, such as AVFoundation, CoreAudio, VideoToolbox, etc. There have also been build-system improvements, including continuous integration.

The out-of-the-box experience on macOS is much improved now, although there is still work to be done.

Attend to get an overview of the improvements that have been done and are planned in the near future.

Speaker: Mr Nirbheek Chauhan (Centricular Ltd)

macOS and iOS Support - History and Future Goals

macOS and iOS Support - History and Future Goals.pdf

12:15 How we rewrote GStreamer in Elixir - 8 years later

A long time ago I joined a software agency working on a bunch of multimedia projects. Working for startups, we needed to iterate fast and deliver POCs validating their ideas. That's when the concept was born: a tool that would be powerful and customizable, yet written in a high-level language and possibly forgiving errors. That's how the Membrane Framework came to life, with API inspired by the best-designed tool we knew - GStreamer, and based on the most suitable foundation we found - the rock-solid Erlang VM and the new, modern language operating on it - Elixir. The Erlang VM has been specifically designed and used for telecom solutions since the '80s, focusing on high availability thanks to built-in concurrency and fault tolerance mechanisms. Elixir gives decent developer experience and welcoming community on top of that.

In the talk, I'll explain why exactly we chose this approach, what challenges we've been facing and where we are today. I'll share insights on writing media streaming software and adopting GStreamer concepts in a high-level, functional language.

Speaker: Mateusz Front (Software Mansion)

12:40 Update on Encrypted Media Extensions in GStreamer

In order to support decryption of protected media in GStreamer, the W3C EME specification was implemented as a GStreamer library as a basis for integrating existing content decryption modules. The initial implementation contains an integration with the Sparkle CDM library to support decryption with any OpenCDM-compatible module.
This presentation will discuss recent progress on this project. In order to simplify usage of this feature, an element was implemented which can hide the details of the EME API and communications pattern for some simple use cases such as fixed user-supplied keys and more complex interactions like the DASH specification's interoperable license request model to support automatic decryption of content based on data in a DASH manifest.
Finally, the presentation will discuss the future plans for playback of protected content with GStreamer such as support for trusted execution environments on Android devices.

Speaker: Jordan Yelloz

jyelloz-gst-2024-eme.pdf

13:05 → 14:15 Lunch (at the venue)

14:20 → 15:50 Room 1

14:20 GStreamer & Rust: What has happened over the last 5 years

It has been 5 years now since the last update at the conference about the status of Rust usage in GStreamer. Since then a lot has happened: a lot of new plugins and applications were written in Rust, some Rust code has been added to GStreamer core, the Rust plugins are shipped with the GStreamer binaries for all platforms, and even about 25% of all commits during the 1.24 release cycle were in Rust.

In this talk I will give an overview of everything that has happened, and where we are at with writing GStreamer applications, plugins and libraries in Rust.

Speaker: Sebastian Dröge (Centricular Ltd)

presentation.pdf

14:55 unixfd: Zero-copy multi-process pipeline

In this presentation we will dive into new unixfdsrc/unixfdsink elements and GstMeta serialization, how they are used to build zero-copy multi-process pipelines for Netflix's testing tools.

Speaker: Xavier Claessens (Netflix)

GStreamer conf 2024 - Unixfd.pdf

15:30 GStreamer Vulkan Video: 2024 edition

State of the art of the present and coming Vulkan video elements. We'll talk about architecture, codecs and challenges to achieve vulkan support ...

Speakers: Stéphane Cerveau (Igalia) , Victor Manuel Jáquez Leal (Igalia)

GST Vulkan Video_ 2024 edition.pdf

14:20 → 15:50 Room 2

14:20 Embedded audio policies made easy with WirePlumber

Building robust audio policies on embedded systems that integrate well with upstream APIs and components such as PipeWire and PulseAudio has been a challenging task for a long time. In many implementations, custom hardware DSP management components are built, bypassing existing upstream functionality, making integration and maintenance harder than they need to be. In this talk, George will explain how many of these difficulties can be overcome by integrating management directly into WirePlumber, the default session manager of PipeWire, and how existing functionality can be factored in to build complex solutions with ease. Emphasis will be given also on how recent developments in WirePlumber have improved this particular use case.

Speaker: George Kiagiadakis (Collabora)

GstConf2024_Embedded-Audio-Policies-with-WirePlumber.pdf

14:55 Real-Time Network Audio with GStreamer on Windows

GStreamer's cross-platform support and robust RTP stack make it an excellent choice for building audio streaming applications. We leveraged these strengths to develop an AES67-compliant streaming application on Windows, designed to interoperate seamlessly with Dante devices.

We came across a number of interesting challenges in the process, some specific to being able to send packets every millisecond on Windows, some related to scaling the number of channels we could transmit with limited CPU resources.

In this talk, I will walk through our journey of building this application, the challenges we faced, and how we overcame them. I will then cover some work that remains to be done to enable low-latency applications with GStreamer on Windows.

Speaker: Mr Taruntej Kanakamalla (asymptotic.io)

Real-Time_Network_Audio_with_GStreamer_on_Windows.pdf

15:30 Tea. Earl Grey. Hot. Automated Air-Ground Communication with GStreamer

Urban Air Mobility (UAM) is a transportation system that uses electric vertical takeoff and landing (eVTOL) aircraft to transport people or cargo within urban areas, helping to reduce congestion on the ground. Aircraft operate in a busy urban environment, and may be semi or fully autonomous. In this use case there is a need to automate communication between aircraft and ground to reduce communication errors and enhance situational awareness for both pilots and ground crews.

South Korea plans to commercialize UAM by next year with the goal of reducing congestion in the capital of Seoul. We worked with Korea's second largest corporation, SK Telecom, to combine GStreamer's excellent WebRTC support with commodity ML speech to text (STT), text to speech (TTS) and large language model (LLM) packages to create an open source automated communication workflow for UAM.

I will discuss how we implemented and tuned this system, and also touch on one particular problem we faced: how to apply STT to a mixture of Korean and English speech.

Speaker: Aaron Boxer (Collabora Inc.)

tea_earl_grey_hot.pdf

15:55 → 16:25 Coffee break

16:30 → 18:25 Room 1

16:30 Video Editing with GStreamer: an update

This talk is the usual "video editing stack update". I will present the work that has been done in and around GES and the new components that have been developed to make building video editing applications and web services more efficient and simpler.

Speaker: Thibault Saunier (Igalia)

16:55 HLS master playlist management made easy

HTTP Live Streaming (HLS), a widely adopted protocol for live video streaming, and has been supported by GStreamer for a long time. HLS enables streaming of multiple formats and bitrates, allowing players to dynamically adjust their streaming quality based on network conditions for ensuring optimal viewer experience.

The HLS specification introduced alternate renditions and variant streams, features that allow the inclusion of multiple language options and video qualities within a single stream. This functionality is implemented through a hierarchical playlist structure: a top-level 'master playlist' that outlines the available streams and renditions, complemented by individual 'media playlists' for each variant.

While client-side support for consuming master playlists has long been available, the existing GStreamer HLS stream creation elements were limited. They only supported the production of media playlists for individual variants. Application developers had to manage individual media playlist pipelines and generate the master playlist themselves.

This talk will explore the design and implementation of a new GStreamer convenience bin that generates master playlists with variant streams and alternate renditions. Additionally, we'll discuss planned enhancements and potential feature additions to further improve this functionality.

Speaker: Mr Sanchayan Maity (asymptotic.io)

HLS master playlist management made easy

17:20 HDR in GTK

This talk will present recent progress on proper color handling in the GTK toolkit and surrounding topics, such as graphics offloading. Taken together these enable state-of-the-art Video playback in GTK apps (including HDR support) using gstreamer.
The talk will outline what is available in the imminent GTK 4.16 release, as well as our plans for the near future.

Speaker: Matthias Clasen (Red Hat)

17:55 GStreamer and display clock synchronization

Compositors require video frames to be submitted in a specific time window to be rendered at the correct vsync signal. This introduces requirements for the video sink element to adjust to the pipeline clock and to the display clock which may drift unrelated to the pipeline clock.

Furthermore, some existing video sinks are targeting UI frameworks and are not directly related to a specific compositor and thus may have unrelated refresh cycles.

This talk summarizes the general issue and highlights the problems with some existing video sinks.

Speaker: Jochen Henneberg

16:30 → 18:25 Room 2

16:30 Developing and debugging on Windows, Fluendo experience.

During different consultancy services we had some tasks to achieve high level of stability for some Windows GStreamer applications. We had to resolve race conditions, memory leaks and D3D11-specific issues, such as crashes in the driver. This talk aims to summarize what we have learned from those experiences.

Spoiler: The tests were helpful here.

Speakers: Rubén Gonzalez, Diego Nieto Munoz (Fluendo)

16:55 librice: a sans-IO ICE networking library

ICE underpins all WebRTC connections and provides peers the ability to discover the most efficient path for transferring data. We will take a brief look into the steps involved in the ICE process and a more in depth look into how a sans-IO implementation might be implemented using the example of librice.

Speaker: Matthew Waters (Centricular)

2024 librice - a sans-IO ICE networking library - Matthew Waters.pdf

17:20 GStreamer and VR

Sometimes, when I'm not working on GStreamer, I work on open source VR projects, and sometimes when I work on open source VR projects - I use GStreamer!

This presentation will talk a bit about the state of open source VR support, how GStreamer relates to it, some interesting use cases for GStreamer in that space, and things that can be improved.

Speaker: Jan Schmidt (Centricular Ltd)

GStreamer & VR-final.pdf

17:55 Windows Graphics Stack Improvements – Direct3D12

Since Direct3D11 support was added in 1.18, GStreamer’s Windows graphics and hardware acceleration features have been greatly improved up to a level of GStreamer's Linux support, or even better than other OS supports in some aspect.

Then Direct3D12 based features were introduced in GStreamer 1.24 which gives another chance to improve performance and expand feature set.

In this presentation, the design of GStreamer Direct3D12 integration layer will be discussed briefly. And the GStreamer Direct3D12 plugin features will be introduced with focus on the differences between Direct3D12 and Direct3D11 plugins.

Speaker: Seungha Yang (Centricular)

18:30 → 18:35 Closing Session

18:45 → 21:00 Social Events: Networking reception with the Montreal Digital Arts community