| Age | Commit message (Collapse) | Author |
|---|
|
Add gpu_test_buddy_dirty_tracker_performance to demonstrate the key
advantage of the decoupled dirty-tracker design over the old dual-tree
/ force_merge approach.
The test runs two scenarios on a 4 GiB pool after alternating
clear/dirty fragmentation at 4 KiB granularity:
1. Contiguous 2 GiB alloc: smaller than the 4 GiB span so it routes
through the contiguous allocator rather than the exact-range fast
path. The old design requires __force_merge() to rebuild the
block; the new design coalesces during free() so the alloc is
O(log N).
old (force_merge) - 45 ms
dirty tracker design - 11 ms
2. Repeated 256 KiB alloc throughput: the old design pays
__force_merge() on every alloc; the new design does not.
old (force_merge) - 86 ms
dirty tracker design - 20 ms
v2:
- Force a contiguous 256 KiB allocation in the repeated-alloc loop so
the baseline actually exercises __force_merge(). (sashiko)
v3:
- Make the first test allocate 2 GiB instead of the full 4 GiB pool,
so it really uses the contiguous path; update the numbers. (sashiko)
Assisted-by: Claude:claude-opus-4-8
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260825172432.80355-2-Arunpravin.PaneerSelvam@amd.com
|
|
The current buddy allocator maintains separate clear_tree[] and
dirty_tree[] rbtrees per order, preventing coalescing between cleared
and dirty buddies. Under mixed workloads, this creates a merge barrier:
adjacent buddies frequently end up split across trees, forcing reliance
on __force_merge() during allocation.
__force_merge() performs an O(N x max_order) scan under the VRAM manager
lock, leading to allocation stalls and failures for large contiguous
requests even when sufficient total free memory is available.
Solution
Replace the dual-tree design with:
- A single free_tree[order] rbtree for dirty and mixed free blocks
(fully cleared free blocks float outside this tree)
- A lightweight out-of-band dirty tracker (gpu_dirty_tracker)
Fully cleared free blocks are tracked outside the buddy trees using an
augmented interval rbtree, enabling O(log E) lookup of the largest
cleared extents.
Buddy coalescing is now unconditional in __gpu_buddy_free(), regardless
of clear/dirty state. This removes the merge barrier and eliminates the
need for __force_merge().
Benefits
- Correct high-order allocations after mixed clear/dirty workloads
- Elimination of O(N x max_order) merge cost from the allocation path
- O(log E) cleared-extent lookup replacing O(N) scans
- Predictable allocation latency under fragmentation
- Reduced complexity with a single tree per order
Test:
dEQP-VK.memory.allocation.basic.size_8KiB.reverse.count_4000
Below data is from /sys/kernel/debug/dri/1/amdgpu_vram_mm:
Base (dual-tree), before VKCTS test:
order- 6 free: 6 MiB, blocks: 26
order- 5 free: 1 MiB, blocks: 15
order- 4 free: 960 KiB, blocks: 15
order- 3 free: 5 MiB, blocks: 171
order- 2 free: 2 MiB, blocks: 176
order- 1 free: 1 MiB, blocks: 165
order- 0 free: 16 KiB, blocks: 4
Base (dual-tree), after VKCTS test:
order- 6 free: 768 KiB, blocks: 3
order- 5 free: 499 MiB, blocks: 3999
order- 4 free: 250 MiB, blocks: 4001
order- 3 free: 129 MiB, blocks: 4157
order- 2 free: 65 MiB, blocks: 4161
order- 1 free: 63 MiB, blocks: 8138
order- 0 free: 20 KiB, blocks: 5
Dirty tracker, before VKCTS test:
order- 6 free: 4 MiB, blocks: 19
order- 5 free: 2 MiB, blocks: 18
order- 4 free: 704 KiB, blocks: 11
order- 3 free: 5 MiB, blocks: 168
order- 2 free: 2 MiB, blocks: 174
order- 1 free: 1 MiB, blocks: 167
order- 0 free: 32 KiB, blocks: 8
Dirty tracker, after VKCTS test:
order- 6 free: 4 MiB, blocks: 19
order- 5 free: 2 MiB, blocks: 18
order- 4 free: 704 KiB, blocks: 11
order- 3 free: 5 MiB, blocks: 168
order- 2 free: 2 MiB, blocks: 174
order- 1 free: 1 MiB, blocks: 167
order- 0 free: 28 KiB, blocks: 7
v2:
- Code-style cleanup and minor refactoring
- Renamed locals for clarity
v3:
- Keep cleared blocks inside free_tree[] instead of floating them.
- Add subtree_has_dirty rbtree augment for O(log N) dirty-first walk.
v4:
- Fixed checkpatch warnings.
- Optimized gpu_buddy_reset_clear() to a single post-order walk that
flips block headers and recomputes the rbtree augment in one pass.
- Propagate subtree_max_size top-down in insert_extent() so ancestors
are not left with stale values on no-rotation inserts. (sashiko)
- Drop the whole extent in gpu_dirty_tracker_mark_dirty() when the
inside-split allocation fails, avoiding a stale clear claim. (sashiko)
- Make gpu_dirty_tracker_find() alignment-aware and fall back to the
dirty tree on steered failure to avoid spurious -ENOSPC. (sashiko)
v5:
- Track dirty extents instead of cleared ones: steer dirty allocs onto
tracked dirty windows and pick clear allocs via a free-tree augment,
avoiding clear-memory wastage by keeping cleared free blocks untouched
during dirty allocation.
v6:
- Make __alloc_range_bias() return the highest/right-most address by
default, establishing top-down as the intended placement for
range-biased allocations.
- Honour GPU_BUDDY_CLEAR_ALLOCATION in __alloc_range_bias() by steering
the descent towards clear subtrees for non-top-down clear
requests. (sashiko)
- Skip dirty-tracker steering for offset-aligned requests so they keep
their min_block_size alignment. (sashiko)
- sashiko reported that the __GFP_NOFAIL dirty-extent allocations on
the free path could deadlock during memory reclaim, since that is a
GFP_KERNEL allocation on the free path; move to a per-tracker
mempool so extent nodes are guaranteed without __GFP_NOFAIL.
(sashiko)
- Derive each free block's clear/dirty class from the blocks already
in hand on split, free, alloc, trim and init instead of querying the
dirty tracker, removing the tracker lookups from the hot paths.
v7:
- Preserve mixed-block clear state in __gpu_buddy_free() when a mixed
split child is re-merged after an undone split. (sashiko)
- Prefer a fully-clear block over a mixed one of the same order via a
single ordered clear-state max augment on free_tree[].
v8:
- Coalesce contiguous dirty blocks in __gpu_buddy_free_list() into one
dirty extent update instead of one mark_dirty() per block. (Matthew)
v9:
- Reset has_clear on allocation so a mixed block taken whole and later
freed fully dirty is not re-tracked as mixed. (sashiko)
v10:
- Use a plain slab allocation for dirty extents; skip and log once on
failure. (Matthew)
- Assert a non-zero size in the dirty-tracker range helpers. (Matthew)
- Drop the cached clear_avail member; derive it on demand. (Matthew)
- Collapse the two dirty branches of gpu_buddy_reset_clear(). (Matthew)
- Move the gpu_block_state enum above the gpu_buddy_block kernel-doc
so the doc directly precedes its struct. (Matthew)
- Mark the gpu_dirty_tracker struct private. (Matthew)
- Preserve a block's clear state on non-clear allocation instead of
force-dirtying it. (Matthew)
- Drop the redundant header clear in gpu_buddy_block_trim(). (Matthew)
v11:
- Assert the manager lock is held in gpu_buddy_clear_avail(). (Matthew)
Assisted-by: Claude:claude-opus-4-8
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260825172432.80355-1-Arunpravin.PaneerSelvam@amd.com
|
|
The resume clearance test skipped every other allocation, expecting an
interleaved clear/dirty layout. But the buddy allocator hands out blocks
contiguously, so this just allocated half the pages in one chunk and never
exercised gpu_buddy_reset_clear()'s force-merge of opposite-state buddies.
Allocate all pages into two lists instead and free one cleared, one dirty,
to build a truly interleaved pattern.
v2: Use for loops instead of do-while for the allocation loops (Jani Nikula)
Fixes: e3335ccbf4da ("drm/tests/gpu_buddy: add a new test case for buffer clearance during resume")
Reported-by: Sashiko-bot <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260721114236.507578-1-Arunpravin.PaneerSelvam@amd.com?part=1
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Christian König <christian.koenig@amd.com>
Assisted-by: GitHub_Copilot:claude-opus-4.8
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260803065656.2960810-1-Arunpravin.PaneerSelvam@amd.com
|
|
Add a new KUnit test gpu_test_buddy_addr_to_block() that validates the
gpu_buddy_allocated_addr_to_block() helper which traces a address back
to its allocated buddy block.
The test covers:
- Exact address matching returns the correct allocated block
- An unallocated address inside the manager should return NULL
- An address outside the manager should return -ENXIO
v4(MattA):
- Add test for unaligned address
v3(Sashiko):
- remove unused target_addr variable
v2(Sashiko):
- Drop the mutex and lockdep annotation; standalone KUnit tests do
not register a driver lock.
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patch.msgid.link/20260806053624.3215216-6-tejas.upadhyay@intel.com
|
|
Add a dedicated .kunitconfig for running the GPU buddy allocator tests.
Signed-off-by: Krzysztof Niemiec <krzysztof.niemiec@intel.com>
Reviewed-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Reviewed-by: Andi Shyti <andi.shyti@linux.intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Link: https://patch.msgid.link/20260729091816.27860-4-krzysztof.niemiec@intel.com
Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
|
|
Add a new unit test case for buffer clearance issue during
resume.
Using a non-power-of-two mm size, allocate alternating blocks of
4KiB in an even sequence and free them as cleared. All alternate
blocks should be marked as dirty and the split blocks should be
merged back to their original size when the blocks clear reset
function is called.
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260721114236.507578-1-Arunpravin.PaneerSelvam@amd.com
|
|
Add a new kunit test gpu_test_buddy_alloc_range() that exercises the
__gpu_buddy_alloc_range() exact-range allocation path, triggered when
start + size == end with flags=0.
The test covers:
- Basic exact-range allocation of the full mm
- Exact-range allocation of equal sub-ranges (quarters)
- Minimum chunk-size exact ranges at start, middle, and end offsets
- Non power-of-two mm size with multiple roots, including cross-root
exact-range allocation
- Randomized exact-range allocations of N contiguous page-aligned
slices in random order
- Negative: partially allocated range must reject overlapping exact
alloc
- Negative: checkerboard allocation pattern rejects exact range over
partially occupied pairs
- Negative: misaligned start, unaligned size, and out-of-bounds end
- Free and re-allocate the same exact range across multiple iterations
- Various power-of-two exact ranges at natural alignment
Cc: Christian König <christian.koenig@amd.com>
Cc: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Suggested-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patch.msgid.link/20260302150947.47535-2-sanjay.kumar.yadav@intel.com
|
|
Add KUnit test to validate offset-aligned allocations in the DRM buddy
allocator.
Validate offset-aligned allocation:
The test covers allocations with sizes smaller than the alignment constraint
and verifies correct size preservation, offset alignment, and behavior across
multiple allocation sizes. It also exercises fragmentation by freeing
alternating blocks and confirms that allocation fails once all aligned offsets
are consumed.
Stress subtree_max_alignment propagation:
Exercise subtree_max_alignment tracking by allocating blocks with descending
alignment constraints and freeing them in reverse order. This verifies that
free-tree augmentation correctly propagates the maximum offset alignment
present in each subtree at every stage.
v2:
- Move the patch to gpu/tests/gpu_buddy_test.c file.
v3:
- Fixed build warnings reported by kernel test robot <lkp@intel.com>
v4:(Matthew)
- Use IS_ALIGNED() instead of manual alignment checks
- Simplify order iteration loop for readability
- Remove extra newline
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260306060155.2114-2-Arunpravin.PaneerSelvam@amd.com
|
|
Some DRM tests cross the 1s execution time threshold that defines a test
as slow. Let's flag them as such.
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de>
Link: https://patch.msgid.link/20260224110310.1854608-1-mripard@kernel.org
Signed-off-by: Maxime Ripard <mripard@kernel.org>
|
|
Move the DRM buddy allocator one level up so that it can be used by GPU
drivers (example, nova-core) that have usecases other than DRM (such as
VFIO vGPU support). Modify the API, structures and Kconfigs to use
"gpu_buddy" terminology. Adapt the drivers and tests to use the new API.
The commit cannot be split due to bisectability, however no functional
change is intended. Verified by running K-UNIT tests and build tested
various configurations.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
[airlied: I've split this into two so git can find copies easier.
I've also just nuked drm_random library, that stuff needs to be done
elsewhere and only the buddy tests seem to be using it].
Signed-off-by: Dave Airlie <airlied@redhat.com>
|
|
Move the DRM buddy allocator one level up so that it can be used by GPU
drivers (example, nova-core) that have usecases other than DRM (such as
VFIO vGPU support). Modify the API, structures and Kconfigs to use
"gpu_buddy" terminology. Adapt the drivers and tests to use the new API.
The commit cannot be split due to bisectability, however no functional
change is intended. Verified by running K-UNIT tests and build tested
various configurations.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
[airlied: I've split this into two so git can find copies easier.
I've also just nuked drm_random library, that stuff needs to be done
elsewhere and only the buddy tests seem to be using it].
Signed-off-by: Dave Airlie <airlied@redhat.com>
|
