| Age | Commit message (Collapse) | Author |
|---|
|
Backmerging to get drm-misc-next up to v7.3-rc2. Requested for
commit 3a2c4d55e32a ("treewide: refresh kmalloc_obj() conversions").
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
|
|
This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
|
|
Getting drm-misc-next up to v7.3-rc1.
In exynos, there was a conflict in exynos_dbi_bind(). The merge resolves
it to the state of commit 3cc8eee9f346 ("drm/exynos: remove dependency
on DRM simple helpers").
Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
|
|
The current buddy allocator maintains separate clear_tree[] and
dirty_tree[] rbtrees per order, preventing coalescing between cleared
and dirty buddies. Under mixed workloads, this creates a merge barrier:
adjacent buddies frequently end up split across trees, forcing reliance
on __force_merge() during allocation.
__force_merge() performs an O(N x max_order) scan under the VRAM manager
lock, leading to allocation stalls and failures for large contiguous
requests even when sufficient total free memory is available.
Solution
Replace the dual-tree design with:
- A single free_tree[order] rbtree for dirty and mixed free blocks
(fully cleared free blocks float outside this tree)
- A lightweight out-of-band dirty tracker (gpu_dirty_tracker)
Fully cleared free blocks are tracked outside the buddy trees using an
augmented interval rbtree, enabling O(log E) lookup of the largest
cleared extents.
Buddy coalescing is now unconditional in __gpu_buddy_free(), regardless
of clear/dirty state. This removes the merge barrier and eliminates the
need for __force_merge().
Benefits
- Correct high-order allocations after mixed clear/dirty workloads
- Elimination of O(N x max_order) merge cost from the allocation path
- O(log E) cleared-extent lookup replacing O(N) scans
- Predictable allocation latency under fragmentation
- Reduced complexity with a single tree per order
Test:
dEQP-VK.memory.allocation.basic.size_8KiB.reverse.count_4000
Below data is from /sys/kernel/debug/dri/1/amdgpu_vram_mm:
Base (dual-tree), before VKCTS test:
order- 6 free: 6 MiB, blocks: 26
order- 5 free: 1 MiB, blocks: 15
order- 4 free: 960 KiB, blocks: 15
order- 3 free: 5 MiB, blocks: 171
order- 2 free: 2 MiB, blocks: 176
order- 1 free: 1 MiB, blocks: 165
order- 0 free: 16 KiB, blocks: 4
Base (dual-tree), after VKCTS test:
order- 6 free: 768 KiB, blocks: 3
order- 5 free: 499 MiB, blocks: 3999
order- 4 free: 250 MiB, blocks: 4001
order- 3 free: 129 MiB, blocks: 4157
order- 2 free: 65 MiB, blocks: 4161
order- 1 free: 63 MiB, blocks: 8138
order- 0 free: 20 KiB, blocks: 5
Dirty tracker, before VKCTS test:
order- 6 free: 4 MiB, blocks: 19
order- 5 free: 2 MiB, blocks: 18
order- 4 free: 704 KiB, blocks: 11
order- 3 free: 5 MiB, blocks: 168
order- 2 free: 2 MiB, blocks: 174
order- 1 free: 1 MiB, blocks: 167
order- 0 free: 32 KiB, blocks: 8
Dirty tracker, after VKCTS test:
order- 6 free: 4 MiB, blocks: 19
order- 5 free: 2 MiB, blocks: 18
order- 4 free: 704 KiB, blocks: 11
order- 3 free: 5 MiB, blocks: 168
order- 2 free: 2 MiB, blocks: 174
order- 1 free: 1 MiB, blocks: 167
order- 0 free: 28 KiB, blocks: 7
v2:
- Code-style cleanup and minor refactoring
- Renamed locals for clarity
v3:
- Keep cleared blocks inside free_tree[] instead of floating them.
- Add subtree_has_dirty rbtree augment for O(log N) dirty-first walk.
v4:
- Fixed checkpatch warnings.
- Optimized gpu_buddy_reset_clear() to a single post-order walk that
flips block headers and recomputes the rbtree augment in one pass.
- Propagate subtree_max_size top-down in insert_extent() so ancestors
are not left with stale values on no-rotation inserts. (sashiko)
- Drop the whole extent in gpu_dirty_tracker_mark_dirty() when the
inside-split allocation fails, avoiding a stale clear claim. (sashiko)
- Make gpu_dirty_tracker_find() alignment-aware and fall back to the
dirty tree on steered failure to avoid spurious -ENOSPC. (sashiko)
v5:
- Track dirty extents instead of cleared ones: steer dirty allocs onto
tracked dirty windows and pick clear allocs via a free-tree augment,
avoiding clear-memory wastage by keeping cleared free blocks untouched
during dirty allocation.
v6:
- Make __alloc_range_bias() return the highest/right-most address by
default, establishing top-down as the intended placement for
range-biased allocations.
- Honour GPU_BUDDY_CLEAR_ALLOCATION in __alloc_range_bias() by steering
the descent towards clear subtrees for non-top-down clear
requests. (sashiko)
- Skip dirty-tracker steering for offset-aligned requests so they keep
their min_block_size alignment. (sashiko)
- sashiko reported that the __GFP_NOFAIL dirty-extent allocations on
the free path could deadlock during memory reclaim, since that is a
GFP_KERNEL allocation on the free path; move to a per-tracker
mempool so extent nodes are guaranteed without __GFP_NOFAIL.
(sashiko)
- Derive each free block's clear/dirty class from the blocks already
in hand on split, free, alloc, trim and init instead of querying the
dirty tracker, removing the tracker lookups from the hot paths.
v7:
- Preserve mixed-block clear state in __gpu_buddy_free() when a mixed
split child is re-merged after an undone split. (sashiko)
- Prefer a fully-clear block over a mixed one of the same order via a
single ordered clear-state max augment on free_tree[].
v8:
- Coalesce contiguous dirty blocks in __gpu_buddy_free_list() into one
dirty extent update instead of one mark_dirty() per block. (Matthew)
v9:
- Reset has_clear on allocation so a mixed block taken whole and later
freed fully dirty is not re-tracked as mixed. (sashiko)
v10:
- Use a plain slab allocation for dirty extents; skip and log once on
failure. (Matthew)
- Assert a non-zero size in the dirty-tracker range helpers. (Matthew)
- Drop the cached clear_avail member; derive it on demand. (Matthew)
- Collapse the two dirty branches of gpu_buddy_reset_clear(). (Matthew)
- Move the gpu_block_state enum above the gpu_buddy_block kernel-doc
so the doc directly precedes its struct. (Matthew)
- Mark the gpu_dirty_tracker struct private. (Matthew)
- Preserve a block's clear state on non-clear allocation instead of
force-dirtying it. (Matthew)
- Drop the redundant header clear in gpu_buddy_block_trim(). (Matthew)
v11:
- Assert the manager lock is held in gpu_buddy_clear_avail(). (Matthew)
Assisted-by: Claude:claude-opus-4-8
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260825172432.80355-1-Arunpravin.PaneerSelvam@amd.com
|
|
Linux 7.2
There was a lot of conflicts this round between fixes and next,
and I'd like to get the merge resolutions that we have in drm-tip.
Signed-off-by: Dave Airlie <airlied@redhat.com>
|
|
Add helper with primary purpose is to efficiently trace a specific
physical memory address back to its corresponding TTM buffer object.
v3:
- use mm->chunk_size minimum allocation granularity (Arun)
v2:
- %s/gpu_buddy_addr_to_block/gpu_buddy_allocated_addr_to_block(MattA)
- remove clear->avail and split nodes check(MattA)
- Adapt lockdep(MattB)
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
Cc: Arunpravin Paneer Selvam <arunpravin.paneerselvam@amd.com>
Cc: dri-devel@lists.freedesktop.org
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patch.msgid.link/20260806053624.3215216-5-tejas.upadhyay@intel.com
|
|
The try_harder contiguous fallback could return a range whose start
offset did not match the caller's min_block_size. When a candidate's
start is misaligned, realign it: free the misaligned run and reallocate
exactly @size at the next lower min_block_size boundary. This keeps the
returned size unchanged with no surplus to trim, and rejects the request
only when no aligned candidate fits.
v2: align misaligned candidates down to min_block_size instead of
bailing out, for both the RHS and LHS paths (Matthew).
Fixes: 0a1844bf0b53 ("drm/buddy: Improve contiguous memory allocation")
Suggested-by: Christian König <christian.koenig@amd.com>
Cc: Matthew Auld <matthew.auld@intel.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Timur Kristóf <timur.kristof@gmail.com>
Cc: stable@vger.kernel.org
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Tested-by: John Olender <john.olender@gmail.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patch.msgid.link/20260709131050.1022759-1-Arunpravin.PaneerSelvam@amd.com
|
|
Extend the scoreboard approach from the previous commit to used blocks,
so drm_buddy_print() can report per-order allocation pressure in O(1).
Unlike free blocks, an allocated block can leave the allocated state
through mark_free() (normal free and gpu_buddy_block_trim()) or be
consumed directly by gpu_block_free() during coalescing. Both sites are
guarded by gpu_buddy_block_is_allocated() and paired with the increment
in mark_allocated().
v3:
- Assert scoreboard is empty at fini(), as sanity check (Matthew Auld)
v2:
- Update after fix for use-after-free in split_block() call sites
- Change goto label to out_free_used_scoreboard for clarity
- Make drm_buddy_print() and gpu_buddy_print() symmetric for used and
free
Assisted-by: GitHub Copilot:claude-sonnet-4.6
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://lore.kernel.org/r/20260522092600.32818-6-francois.dugast@intel.com
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
|
|
Reporting per-order free block counts in drm_buddy_print() currently
requires walking all rbtrees, which is O(n) over the total number of
free blocks and holds the allocator lock for the duration. This becomes
expensive on large VRAM heaps with many small free fragments.
Maintain a free_scoreboard[] array indexed by order instead, so that
the count for any order is always available in O(1). The scoreboard is
kept accurate by hooking into the four places where a block's free state
changes: mark_free(), mark_allocated(), mark_split(), and the sites in
__gpu_buddy_free(), __force_merge(), and the four err_undo paths that
call rbtree_remove() directly on free blocks without going through
mark_*().
The print functions are simplified as a result: the rbtree traversal
is replaced by a direct array lookup.
v3: Update after introducing __gpu_buddy_undo_splits() helper
v2: Update after fix for use-after-free in split_block() call sites
Assisted-by: GitHub Copilot:claude-sonnet-4.6
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://lore.kernel.org/r/20260522092600.32818-5-francois.dugast@intel.com
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
|
|
The pattern of merging a block back with its buddy on error paths is
duplicated across multiple locations. Extract it into a
__gpu_buddy_undo_splits() helper to avoid repetition and prepare for
future changes.
Suggested-by: Matthew Auld <matthew.auld@intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://lore.kernel.org/r/20260522092600.32818-4-francois.dugast@intel.com
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
|
|
The err_undo label in alloc_from_freetree() is only reachable via a
goto from inside the `while (tmp != order)` loop, which means tmp is
guaranteed to differ from order at that point. The surrounding
`if (tmp != order)` guard was therefore always true and can be dropped
without any behavioral change.
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://lore.kernel.org/r/20260522092600.32818-3-francois.dugast@intel.com
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
|
|
When split_block() fails it returns before calling mark_split(), leaving
the block in the FREE state and still linked in the rbtree. The four
err_undo paths then call __gpu_buddy_free() without first removing the
block from the tree, which leads to two distinct bugs:
- If the buddy is also free, __gpu_buddy_free() merges the two siblings
by calling gpu_block_free(mm, block) while block->rb is still linked
in the tree. Any subsequent rbtree traversal will follow the now-
dangling pointer, causing a use-after-free.
- In alloc_from_freetree(), where there is no buddy guard,
__gpu_buddy_free() always reaches mark_free() -> rbtree_insert() with
block still in the tree, corrupting the rbtree.
The same pattern is already used correctly in __force_merge(): call
rbtree_remove() to unlink the block before handing it to
__gpu_buddy_free(). Apply the same fix to all four err_undo sites.
Reported-by: Sashiko <sashiko-bot@kernel.org>
Assisted-by: GitHub Copilot:claude-sonnet-4.6
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://lore.kernel.org/r/20260522092600.32818-2-francois.dugast@intel.com
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
|
|
gpu_buddy APIs are expected to be called with the driver-provided lock
held, but there is no runtime enforcement of this contract. Add lockdep
annotations to catch locking violations early.
Introduce gpu_buddy_driver_set_lock() for the driver to register the
lock that protects the buddy manager. Add gpu_buddy_driver_lock_held()
assertions to all exported gpu_buddy and drm_buddy APIs that
access/modify the manager state. The lock_dep_map field is only compiled
in when CONFIG_LOCKDEP is enabled, adding zero overhead to production
builds.
Wire up xe_ttm_vram_mgr to register its mutex with the buddy manager
after initialization.
Assisted-by: Copilot:claude-opus-4.6
Suggested-by: Matthew Brost <matthew.brost@intel.com>
Signed-off-by: Tejas Upadhyay <tejas.upadhyay@intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patch.msgid.link/20260508065544.4049240-2-tejas.upadhyay@intel.com
|
|
Large alignment requests previously forced the buddy allocator to search by
alignment order, which often caused higher-order free blocks to be split even
when a suitably aligned smaller region already existed within them. This led
to excessive fragmentation, especially for workloads requesting small sizes
with large alignment constraints.
This change prioritizes the requested allocation size during the search and
uses an augmented RB-tree field (subtree_max_alignment) to efficiently locate
free blocks that satisfy both size and offset-alignment requirements. As a
result, the allocator can directly select an aligned sub-region without
splitting larger blocks unnecessarily.
A practical example is the VKCTS test
dEQP-VK.memory.allocation.basic.size_8KiB.reverse.count_4000, which repeatedly
allocates 8 KiB buffers with a 256 KiB alignment. Previously, such allocations
caused large blocks to be split aggressively, despite smaller aligned regions
being sufficient. With this change, those aligned regions are reused directly,
significantly reducing fragmentation.
This improvement is visible in the amdgpu VRAM buddy allocator state
(/sys/kernel/debug/dri/1/amdgpu_vram_mm). After the change, higher-order blocks
are preserved and the number of low-order fragments is substantially reduced.
Before:
order- 5 free: 1936 MiB, blocks: 15490
order- 4 free: 967 MiB, blocks: 15486
order- 3 free: 483 MiB, blocks: 15485
order- 2 free: 241 MiB, blocks: 15486
order- 1 free: 241 MiB, blocks: 30948
After:
order- 5 free: 493 MiB, blocks: 3941
order- 4 free: 246 MiB, blocks: 3943
order- 3 free: 123 MiB, blocks: 4101
order- 2 free: 61 MiB, blocks: 4101
order- 1 free: 61 MiB, blocks: 8018
By avoiding unnecessary splits, this change improves allocator efficiency and
helps maintain larger contiguous free regions under heavy offset-aligned
allocation workloads.
v2:(Matthew)
- Update augmented information along the path to the inserted node.
v3:
- Move the patch to gpu/buddy.c file.
v4:(Matthew)
- Use the helper instead of calling _ffs directly
- Remove gpu_buddy_block_order(block) >= order check and drop order
- Drop !node check as all callers handle this already
- Return larger than any other possible alignment for __ffs64(0)
- Replace __ffs with __ffs64
v5:(Matthew)
- Drop subtree_max_alignment initialization at gpu_block_alloc()
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Suggested-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Link: https://patch.msgid.link/20260306060155.2114-1-Arunpravin.PaneerSelvam@amd.com
|
|
Introduce gpu_buddy_assert(), a small helper that wraps WARN_ON() and,
when CONFIG_KUNIT is enabled, also calls kunit_fail_current_test() so
that any active KUnit test is marked as failed. In non-KUnit builds
the macro reduces to WARN_ON(), preserving existing behaviour.
Stringify the asserted condition in the failure message to make it easy
to identify which assertion fired. Leave the WARN_ON() in
gpu_buddy_block_trim() unchanged, as it returns -EINVAL and the caller
already observes the failure via the return code.
Cc: Christian König <christian.koenig@amd.com>
Cc: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Suggested-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patch.msgid.link/20260227130037.53615-2-sanjay.kumar.yadav@intel.com
|
|
Move gpu_buddy_block_state(), gpu_buddy_block_is_allocated(),
and gpu_buddy_block_is_split() from gpu_buddy.h to gpu_buddy.c
as static functions since they have no external callers.
Remove gpu_get_buddy() as it was an unused exported wrapper
around the internal __get_buddy().
No functional changes.
v2:
- Rebased after DRM buddy allocator moved to drivers/gpu/
- Keep gpu_buddy_block_is_free() in header since it's now
used by drm_buddy.c
- Updated commit message
Cc: Christian König <christian.koenig@amd.com>
Cc: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Suggested-by: Matthew Auld <matthew.auld@intel.com>
Signed-off-by: Sanjay Yadav <sanjay.kumar.yadav@intel.com>
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Signed-off-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Link: https://patch.msgid.link/20260212092527.718455-6-sanjay.kumar.yadav@intel.com
|
|
Move the DRM buddy allocator one level up so that it can be used by GPU
drivers (example, nova-core) that have usecases other than DRM (such as
VFIO vGPU support). Modify the API, structures and Kconfigs to use
"gpu_buddy" terminology. Adapt the drivers and tests to use the new API.
The commit cannot be split due to bisectability, however no functional
change is intended. Verified by running K-UNIT tests and build tested
various configurations.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
[airlied: I've split this into two so git can find copies easier.
I've also just nuked drm_random library, that stuff needs to be done
elsewhere and only the buddy tests seem to be using it].
Signed-off-by: Dave Airlie <airlied@redhat.com>
|
|
Move the DRM buddy allocator one level up so that it can be used by GPU
drivers (example, nova-core) that have usecases other than DRM (such as
VFIO vGPU support). Modify the API, structures and Kconfigs to use
"gpu_buddy" terminology. Adapt the drivers and tests to use the new API.
The commit cannot be split due to bisectability, however no functional
change is intended. Verified by running K-UNIT tests and build tested
various configurations.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Reviewed-by: Dave Airlie <airlied@redhat.com>
[airlied: I've split this into two so git can find copies easier.
I've also just nuked drm_random library, that stuff needs to be done
elsewhere and only the buddy tests seem to be using it].
Signed-off-by: Dave Airlie <airlied@redhat.com>
|
