Write once, run anywhere inference with GStreamer

GStreamer Conference 2026

10 October 2026

Sebastian Dröge <sebastian@centricular.com>

Agenda

1. The problem: fragmentation
2. The solution: GStreamer!
3. Building the infrastructure
4. Write once, run anywhere: are we there yet?
5. Beyond video: audio and LLMs
6. The future

The problem

Fragmentation across hardware, APIs and platforms

Many accelerators, no common API

• Every modern device ships with 1-5 different inference components
  • CPU, NPU, iGPU, dGPU, specialized accelerators

• But: Different APIs

  • per component type, vendor (NVIDIA, AMD, Intel, Qualcomm, ...)
    and platform (Linux, macOS, Windows, ...)

• ONNX RT provides a low-level inference API

  • The media handling around it (pre/post-processing, result handling,
    tracking, overlays, ...) you have to build yourself

Vendor-specific GStreamer ecosystems

• Multiple vendor-specific GStreamer extension libraries
  • NVIDIA DeepStream, Intel DLStreamer, Qualcomm QAIRT/imsdk, ...
  • NNStreamer: vendor-agnostic, but developed independently of the GStreamer project

• Effect: different APIs, different elements

  • Vendor lock-in: app changes per platform

• Either build custom abstraction layers or lock into one solution

  • Maintenance cost, limited deployment, risks

We already have experience with this situation: Codecs

• Dozens of vendor codec plugins, but all behave the same
  • Generic infrastructure: parsers, decodebin, ...

• How about we solve it the same way?

The foundation: Analytics API in 1.24

• Model/hardware-agnostic metadata (classification, object detection, relational, ...)

• Model info files: generic description for model inputs / outputs

• Video inference element based on ONNX RT

• Overlay and tracking elements operating on metadata

• One infrastructure for all hardware, all vendors, all platforms

• Started by Collabora, now a team effort across the whole community

And it keeps growing

• More metadata types: keypoints, segmentation, grouping, text, ...
  • Plus corresponding overlays

• More tensor decoders for different models

  • Recently mostly in Rust

• Basic batching support

• Simple IoU-based object tracker

• And more!

What does this look like in practice?

Generic tensors and caps signaling

• Use caps for signaling model tensor requirements
  • Inputs and outputs
  • Extension fields to existing caps or plain tensor caps
  • Model IDs, tensor layout, normalization, ...
  • Also for memory negotiation

video/x-raw
  tensors: tensorgroups
    yolo-26-end2end-out:
      tensor/strided
        tensor-id: yolo-26-end2end-out
        dims: <1, 1..MAX, 6>
        dims-order: row-major
        type: float32

Independent components

• Split into independent components with different tasks

  • Model-specific or -agnostic pre/post processing
  • Inference-engine specific elements
  • New components can be provided externally

• Generic analytics metadata for common use-cases

  • Object detection, keypoints, ...

Inference elements

• Generic ONNX RT, TFLite/LiteRT and ExecuTorch elements

  • Different backends, model-agnostic
  • Model information via shared model info files

• Specialized inference elements (e.g. burn-based YoloX)

• Negotiate requirements via caps (including zero-copy)

• Switching from one backend or engine to another with the same model:
  just change one element, or a property

Model-specific tensor decoders

• Convert model-specific output tensors

  • Generic metadata: object detection, keypoints, ...
  • New streams: independent audio/video/text streams

• Independent of inference element

• tensordecodebin automatically selects the right decoder based on caps

Multi-model cascades

• Chaining different models

  • Pre/post-processing and batching between models

• Example: text detection → crop → OCR → re-associate with original positions

• Different inference engines or hardware per model in the same process

• Design still under discussion

Multi-model cascade in practice

Model coverage

• Available now: (MobileNet) SSD and classification, Yolo v8-v12 & 26 (detection,
  OBB, segmentation), Ultra light face detection, CTC text recognition (OCR),
  PIPnet (facial landmarks), hand detection / landmarks

• In MRs: CRAFT text detection, ZeroDCE (low-light image enhancement),
  FaceDet light, ...

What you get out of it

Run anywhere: are we there yet?

• Selection at runtime based on what exists
  • Plug & play: every other part of the pipeline stays the same

• onnxinference: CUDA, ROCm, DirectML (Windows), Vitis AI/VSI (NPUs), ...

  • In MRs: CoreML (Apple), OpenVINO (Intel), NNAPI (Android)
  • More easy to add

• tfliteinference: CPU, XNNPACK, EdgeTPU, VSI NPU, arbitrary external ones

• executorchinference: all of the above plus more, statically linked

• Backend options for every platform: Linux, macOS, Windows, Android, iOS

The same pipeline, every platform

Beyond Video

• Generic infrastructure currently only covers video input and output,
  text and metadata output

• In the future this will change but:

Audio and LLMs

• whisper.cpp: speech to text, see Mathieu's lightning talk!
  • audio.cpp in the future? diarization, audio generation, ...

• llama.cpp: generic LLM support, see my lightning talk

  • text to text or image/video to text with a system prompt

• Both support many backends: CPU, Vulkan, ROCm, CUDA, Metal, ...

• Not yet integrated with the generic infrastructure

  • Needs design work!

Future directions

Different media types and new model classes

• Audio in/out, text input in addition to video

• Multi-modal models

  • CLIP-style: image + text query to a similarity score (text-based filtering)
  • Multi-modal LLMs

• LLMs with structured output (e.g. JSON-based output for further processing)

• Decision models like Jev

  • Text or multi-modal inputs + question to a typed, probabilistic decision

Making it better

• Zero-copy everywhere
  • Currently only HIP/ROCm & D3D12 in onnxinference for video
  • Also for intermediate tensors

• More batching strategies

• More tracking element choices, also hardware-accelerated

Making it more complete

• Generic video pre/post-processing (initial MR)

• More infrastructure for chaining models

• More tensor decoders, more generic analytics metas

• All as reusable, composable components

And getting more people involved!

• Start using it in your products

• Report what is missing, what the papercuts are

• Iterate on the API and improve / complete it

• Involve more contributors and vendors

The ultimate goal

• Right now, DeepStream is the most advanced GStreamer-based AI SDK

• We want GStreamer to do everything DeepStream can do, directly in
  GStreamer, vendor- and platform-agnostic

  • Ideally with vendor support: see Oskar's and Daniel's DLStreamer talk
  • We're actively working on AMD GPU/NPU support

• Ultimately: run the same AI pipeline on your phone and laptop, your
  server, smart cameras, cars, robots, and wherever GStreamer runs

• We are getting there!

Questions? Comments?

Code is all available in the GStreamer repositories or merge requests.

Feel free to also talk to me later!