Reducing locking contention in drm gpu drivers to reduce stutters

This is about something we had to solve for Steam Frame

Let’s talk about the Frame

We do this thing called async reprojection

The problem

Sometimes we run low on memory, that might lead to swapping (zram)

The problem

• Swap-in can’t happen on fault on GPUs
• Resources made resident before submitting to HW
• Objects won’t be swapped out while they have an active fence

Shrinker

• bridges the gap for dma devices
• when low on memory a scan is triggered
• happens synchronously on allocations
• goes through gem objects and marks pages as evictable/purgeable

The easy part

• We want vrcompositor to always have resources resident
  • just add a way of flagging some VMs as unevictable
  • skip objects that are mapped to an evictable VM (including shared objects)

Not enough …

We still had stutters

The problem

Imagine we had:

make_reources_resident();
drm_sched_push_job();

The problem

Won’t work:

make_reources_resident();
/* resources aren't active yet -> race with shrinker */
drm_sched_push_job();

The problem

What it actually looks like

lock_all_objects(); // not quite with vmbind
/* shrinker must take locks too, guaranteed not to race */
make_reources_resident();
drm_sched_push_job();
/* objects are now active */
release_all_objects();
/* shrinker can now run, but objects are active */

The problem

lock_all_objects(); // this includes shared objects
make_reources_resident();
drm_sched_push_job();
release_all_objects();

vrcompositor and game inevitably share buffers

The problem

lock_all_objects();
make_reources_resident(); // this can take a while in the game's thread
drm_sched_push_job();
release_all_objects();

The problem

lock_all_objects(); // vrcompositor will be stuck here
make_reources_resident(); 
drm_sched_push_job();
release_all_objects();

The problem

• vrcompositor will miss a frame
• even if all of it’s resources are resident
• very noticeable

Our downstream solution

lock_all_objects();
atomic_inc(&vm->unevictable);
collect_evicted_objects(&evicted_objects);
release_all_objects();
/* won't race because of the unevictable flag */
make_ojects_resident(evicted_objects);

lock_all_objects();
atomic_dec(&vm->unevictable);
drm_sched_push_job();
release_all_objects();

Takeaways

• We need the shrinker to wait on submissions
• We don’t want submissions to wait on other submissions

A lock is a too big of an hammer

Previous attemps

drm_sched_push_job(jop);
/* objects are active now so won't race */
make_reources_resident(); 

But execution might start before resources are resident -> fault

Previous attemps

add_dependency(job, validate_fence);
drm_sched_push_job(jop);

/* objects are active now so won't race */
make_reources_resident(); 
signal(validate_fence);

Previous attempts

• Validation can fail, annoying to handle
• Allocations must never happen before code that signals a fence

Why?

• fences can keep respures alive
• allocations block until memory is avilable
• fences signaled after allocations -> deadlock

How do we solve it upstream?

• We don’t like piling downstream hacks
• Others can benefit
• I’ll post an RFC
  • look for an email from anna.maniscalco2000@gmail.com