nocl: OpenCL on CUDA

What is this?

• Compute only gallium driver
• Useable by Rusticl
• Uses the CUDA Driver API
  • Shipped as part of the driver, not CUDA!
• Uses PTX
• Upstream MR

But why?

• Open-Source OpenCL implementation on the prop. driver
• Support more extensions
• Could be used for perf comparisons
• We want open standards, not closed ones
• To learn more about how CUDA works
• I’ve never written a new gallium driver
• Learn about PTX
  • We need PTX to nir for NVK (DLSS support)

Status

• Implements OpenCL 3.1
• 99%+ CTS pass rate
• Requires CUDA 6.5 or newer
  • Supports all CUDA capable GPUs (in theory)

Is it merged yet?

• No
• Want to figure out images first

General

• libcuda.so is dlopened
  • No hard dependency
  • Needs CUDA SDK at build time
• Uses versioned CUDA entrypointers
  • cuGetProcAddress_v2
  • Allows easy feature detection at runtime

CUDA

• CUDA runtime has global state 😩
  • Global default “CUStream” to act as a pipe_context
  • Most APIs pick the global stream to operate on
• Luckily it’s legacy
  • Should move away from it, but I ran into issues
  • New model uses a thread local stream 😩
  • Not all APIs have a replacement with an explicit stream 😩
• Causes weird synchronization bugs

API acceleration

• Most gallium APIs have counterparts in CUDA
  • resource_copy_region => cuMemcpyAsync/cuMemcpy3DAsync
  • texture_subdata => cuMemcpy3D
  • clear_buffer => cuMemsetD{8,16,32}Async
• Can just consume host pointers in many places
  • Makes operation on host memory very trivial

PTX

• String based, no binary format 😩
• Unstructured CFG
  • Branching
  • Function calls
• High-level assembly language
  • Instructions
  • Typed variables 😩
  • Predicated execution

PTX

• Instructions have many modifiers
  • Src/dst types
  • Source swizzles
  • Other random things
  • Dsts on the left side
• selp.b16 %r5, 0x3c00, 0x0000, %r3.x;

What’s the compiler doing?

1. Runs some NIR passes
2. Translates NIR to PTX
3. Uses the CUDA JIT

• Uses Rust for its string format API
• Surprisingly small: 2000 loc
• Uses NAKs lower_cf pass
  • Should we generalize it?

Example

nir_op_b2f16 => {
   let src = src_string(alu.get_src(0), 0);
   let dst = def_string(&alu.def, 0);
   self.block.push(format!("selp.b16 {dst}, 0x3c00, 0x0000, {src};"));
}

=>

selp.b16 %r5, 0x3c00, 0x0000, %r3.x;

Compiler performance

Images in OpenCL

• There are image read ops with sampler args
• Samplers are passed as kernel arguments
• They are independent from the image

kernel void test(image2d_t img, sampler_t samp,
                 int4 global* out, float2 coords)
{
    *out = read_imagei(img, samp, coords);
}

image handle formwat

• Image handles are 32 bits
  • Unified
    • 31:0 image and sampler
  • Independent
    • 19:0 image
    • 31:20 sampler
  • Used to index into texture/sampler tables

The problem

• CUDA only supports unified
• OpenCL/Vulkan need independent
  • Nivida’s OpenCL also uses independent
  • It’s layered on top of CUDA
  • Statically linked libcuda
    • Probably some secret flag?
    • Didn’t find anything

The problem - shader

• Would need to map sampler+image pairs to a linear index
• Maybe just have a 2D table?
  • And rely on constant-folding/copy-prop?

kernel void test(image2d_t img, sampler_t samp_a, sampler_t samp_b,
                 global int4 *out, float2 coords)
{
    sampler_t samp = coords.x >= 2.0 ? samp_a : samp_b;
    *out = read_imagei(img, samp, coords);
}

Backend compiler todos

• Casting
  • PTX has implicit casting support
  • Currently it’s all emulated
• Image support
• More optimizations
  • It seems the PTX JIT doesn’t optimize very well
  • Relies on upper layers (e.g. nvcc)
• A bit much duplicated code

SVM - curent

• Rusticl manages VM reservations
  • Reserves VM space for each device
  • Creates pipe_resource
  • Assigns VM address

SVM - CUDA

• CUDA has managed alllocations
  • GPU == host address
  • CUDA picks the address
• There is an explicit virtual memory management API
  • Allows reserving virtual memory
  • Assign addresses to allocations!
  • Exactly what we need

gl sharing

• CUDA has APIs for this specifically
• Uses GL API handles:

cuGraphicsGLRegisterImage(
   CUgraphicsResource *pCudaResource, GLuint image,
   GLenum target, unsigned int Flags)

• Nvidia’s impl supports this

Use NAK as a compiler?

• Could help with more targeted perf comparisons
• Would need to support CUDAs kernel arguments
• Need to create an ELF cubin
• (I hope somebody NAKs the idea)

Others frontends?

• Teflon support might be a fun idea
• Maybe va?
  • Not sure it really makes sense though

Current situation

• CUDA is kinda the most important compute API
• What can we do to improve Gallium or Vulkan?

Problems

• Some of the current APIs are suboptimal for compute workloads
• subdata/map/unmap/etc.. might do host side copies
  • Some drivers hw accelerate them
• Some drivers compile at launch_grid time
• No API for fetching GPU binaries
• set_global_binding + launch_grid is kinda bad
  • set_global_binding only guarantees valid VM for one dispatch

API hw acceleration

• CUDA provides optimized hw acceleration for most copy APIs transparently
  • Gallium drivers should report what operations are accelerated

New launch_grid API

• CUDA: cuLaunchKernel
  • Grid params as arguments
  • Kernel arguments passed as argument
  • No state!
• Gallium: launch_grid
  • Grid params as a struct
  • Memory objects bound to the context
  • Kernel arguments are in managed cb0
  • Can we do better?

Shiva?

• Maybe just wait for shiva?