Currently the rust allocator simply passes the size of the type Layout
to krealloc(), and in theory the alignment requirement from the type
Layout may be larger than the guarantee provided by SLAB, which means
the allocated object is mis-aligned.
Fix this by adjusting the allocation size to the nearest power of two,
which SLAB always guarantees a size-aligned allocation. And because Rust
guarantees that the original size must be a multiple of alignment and
the alignment must be a power of two, then the alignment requirement is
satisfied.
Suggested-by: Vlastimil Babka <[email protected]>
Co-developed-by: Andreas Hindborg (Samsung) <[email protected]>
Signed-off-by: Andreas Hindborg (Samsung) <[email protected]>
Signed-off-by: Boqun Feng <[email protected]>
Cc: [email protected] # v6.1+
---
rust/bindings/bindings_helper.h | 1 +
rust/kernel/allocator.rs | 74 ++++++++++++++++++++++++++-------
2 files changed, 60 insertions(+), 15 deletions(-)
diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index 3e601ce2548d..058954961bfc 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -13,5 +13,6 @@
#include <linux/sched.h>
/* `bindgen` gets confused at certain things. */
+const size_t BINDINGS_ARCH_SLAB_MINALIGN = ARCH_SLAB_MINALIGN;
const gfp_t BINDINGS_GFP_KERNEL = GFP_KERNEL;
const gfp_t BINDINGS___GFP_ZERO = __GFP_ZERO;
diff --git a/rust/kernel/allocator.rs b/rust/kernel/allocator.rs
index 397a3dd57a9b..fae11d1fdba7 100644
--- a/rust/kernel/allocator.rs
+++ b/rust/kernel/allocator.rs
@@ -9,6 +9,36 @@
struct KernelAllocator;
+/// Calls `krealloc` with a proper size to alloc a new object aligned to `new_layout`'s alignment.
+///
+/// # Safety
+///
+/// - `ptr` can be either null or a pointer which has been allocated by this allocator.
+/// - `new_layout` must have a non-zero size.
+unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: bindings::gfp_t) -> *mut u8 {
+ // Customized layouts from `Layout::from_size_align()` can have size < align, so pad first.
+ let layout = new_layout.pad_to_align();
+
+ let mut size = layout.size();
+
+ if layout.align() > bindings::BINDINGS_ARCH_SLAB_MINALIGN {
+ // The alignment requirement exceeds the slab guarantee, thus try to enlarge the size
+ // to use the "power-of-two" size/alignment guarantee (see comments in `kmalloc()` for
+ // more information).
+ //
+ // Note that `layout.size()` (after padding) is guaranteed to be a multiple of
+ // `layout.align()`, so `next_power_of_two` gives enough alignment guarantee.
+ size = size.next_power_of_two();
+ }
+
+ // SAFETY:
+ // - `ptr` is either null or a pointer returned from a previous `k{re}alloc()` by the
+ // function safety requirement.
+ // - `size` is greater than 0 since it's either a `layout.size()` (which cannot be zero
+ // according to the function safety requirement) or a result from `next_power_of_two()`.
+ unsafe { bindings::krealloc(ptr as *const core::ffi::c_void, size, flags) as *mut u8 }
+}
+
unsafe impl GlobalAlloc for KernelAllocator {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
// `krealloc()` is used instead of `kmalloc()` because the latter is
@@ -30,10 +60,20 @@ unsafe fn dealloc(&self, ptr: *mut u8, _layout: Layout) {
// to extract the object file that has them from the archive. For the moment,
// let's generate them ourselves instead.
//
+// Note: Although these are *safe* functions, but they are only generated at
+// `GlobalAlloc` callsites, hence we assume the parameters obey the same
+// `GlobalAlloc` function safety requirements: size and align should form a
+// valid layout, and size is greater than 0.
+//
// Note that `#[no_mangle]` implies exported too, nowadays.
#[no_mangle]
-fn __rust_alloc(size: usize, _align: usize) -> *mut u8 {
- unsafe { bindings::krealloc(core::ptr::null(), size, bindings::GFP_KERNEL) as *mut u8 }
+fn __rust_alloc(size: usize, align: usize) -> *mut u8 {
+ // SAFETY: See assumption above.
+ let layout = unsafe { Layout::from_size_align_unchecked(size, align) };
+
+ // SAFETY: `ptr::null_mut()` is null, per assumption above the size of `layout` is greater
+ // than 0.
+ unsafe { krealloc_aligned(ptr::null_mut(), layout, bindings::GFP_KERNEL) }
}
#[no_mangle]
@@ -42,23 +82,27 @@ fn __rust_dealloc(ptr: *mut u8, _size: usize, _align: usize) {
}
#[no_mangle]
-fn __rust_realloc(ptr: *mut u8, _old_size: usize, _align: usize, new_size: usize) -> *mut u8 {
- unsafe {
- bindings::krealloc(
- ptr as *const core::ffi::c_void,
- new_size,
- bindings::GFP_KERNEL,
- ) as *mut u8
- }
+fn __rust_realloc(ptr: *mut u8, _old_size: usize, align: usize, new_size: usize) -> *mut u8 {
+ // SAFETY: See assumption above.
+ let new_layout = unsafe { Layout::from_size_align_unchecked(new_size, align) };
+
+ // SAFETY: Per assumption above, `ptr` is allocated by `__rust_*` before, and the size of
+ // `new_layout` is greater than 0.
+ unsafe { krealloc_aligned(ptr, new_layout, bindings::GFP_KERNEL) }
}
#[no_mangle]
-fn __rust_alloc_zeroed(size: usize, _align: usize) -> *mut u8 {
+fn __rust_alloc_zeroed(size: usize, align: usize) -> *mut u8 {
+ // SAFETY: See assumption above.
+ let layout = unsafe { Layout::from_size_align_unchecked(size, align) };
+
+ // SAFETY: `ptr::null_mut()` is null, per assumption above the size of `layout` is greater
+ // than 0.
unsafe {
- bindings::krealloc(
- core::ptr::null(),
- size,
+ krealloc_aligned(
+ ptr::null_mut(),
+ layout,
bindings::GFP_KERNEL | bindings::__GFP_ZERO,
- ) as *mut u8
+ )
}
}
--
2.41.0
On Sun, Jul 30, 2023 at 3:29 AM Boqun Feng <[email protected]> wrote:
>
> +// Note: Although these are *safe* functions, but they are only generated at
> +// `GlobalAlloc` callsites, hence we assume the parameters obey the same
> +// `GlobalAlloc` function safety requirements: size and align should form a
> +// valid layout, and size is greater than 0.
Thanks for adding all the `// SAFETY` comments here Boqun!
Björn, do they look good to you? (since you fixed the issue in the compiler)
On this comment in particular, "generated at `GlobalAlloc` callsites"
sounds a bit confusing to me. Would "... called by the compiler with
parameters that obey ..." make sense? Or does the sentence refer to
the normal case (i.e. when the functions are generated)? Anyway, it is
not a big deal.
Cheers,
Miguel
On Sun, Jul 30, 2023 at 10:41:54PM +0000, Björn Roy Baron wrote:
> On Jul 30, 2023, 10:43 PM, Miguel Ojeda < [email protected]> wrote:
> > On Sun, Jul 30, 2023 at 3:29 AM Boqun > Feng <[email protected]> wrote:
> > >
> > > +// Note: Although these are *safe* functions, but they are only generated at
> > > +// `GlobalAlloc` callsites, hence we assume the parameters obey the same
> > > +// `GlobalAlloc` function safety requirements: size and align should form a
> > > +// valid layout, and size is greater than 0.
> >
> > Thanks for adding all the `// SAFETY` comments here Boqun!
> >
> > Björn, do they look good to you? (since you fixed the issue in the compiler)
>
> Based on a quick look, yes. The __rust_* methods that are normally generated by the compiled directly jump to the respective global allocator method, so they have the same safety requirements.
>
Good to know, thanks!
> >
> > On this comment in particular, "generated at `GlobalAlloc` callsites"
> sounds a bit confusing to me. Would "... called by the compiler with
> parameters that obey ..." make sense? Or does the sentence refer to
Agreed. It's better. So reword as below:
// Note: Although these are *safe* functions, but they are called by the
// compiler with the parameters that obey the same `GlobalAlloc`
// function safety requirements: size and align should form a valid
// layout, and size is greater than 0.
Regards,
Boqun
> the normal case (i.e. when the functions are generated)? Anyway, it is
> not a big deal.
> >
> > Cheers,
> > Miguel
>
> Cheers,
> Björn
>
> (resent as I accidentally sent html instead of plain text)
On Jul 30, 2023, 10:43 PM, Miguel Ojeda < [email protected]> wrote:
> On Sun, Jul 30, 2023 at 3:29 AM Boqun > Feng <[email protected]> wrote:
> >
> > +// Note: Although these are *safe* functions, but they are only generated at
> > +// `GlobalAlloc` callsites, hence we assume the parameters obey the same
> > +// `GlobalAlloc` function safety requirements: size and align should form a
> > +// valid layout, and size is greater than 0.
>
> Thanks for adding all the `// SAFETY` comments here Boqun!
>
> Björn, do they look good to you? (since you fixed the issue in the compiler)
Based on a quick look, yes. The __rust_* methods that are normally generated by the compiled directly jump to the respective global allocator method, so they have the same safety requirements.
>
> On this comment in particular, "generated at `GlobalAlloc` callsites"
sounds a bit confusing to me. Would "... called by the compiler with
parameters that obey ..." make sense? Or does the sentence refer to
the normal case (i.e. when the functions are generated)? Anyway, it is
not a big deal.
>
> Cheers,
> Miguel
Cheers,
Björn
(resent as I accidentally sent html instead of plain text)
On Mon, Jul 31, 2023 at 12:54 AM Boqun Feng <[email protected]> wrote:
>
> Good to know, thanks!
Yeah, thanks Björn!
> Agreed. It's better. So reword as below:
>
> // Note: Although these are *safe* functions, but they are called by the
> // compiler with the parameters that obey the same `GlobalAlloc`
> // function safety requirements: size and align should form a valid
> // layout, and size is greater than 0.
+1, thanks!
Applied to `rust-fixes`, but please feel free to send `Reviewed-by`s.
Cheers,
Miguel
On 7/30/23 03:29, Boqun Feng wrote:
> Currently the rust allocator simply passes the size of the type Layout
> to krealloc(), and in theory the alignment requirement from the type
> Layout may be larger than the guarantee provided by SLAB, which means
> the allocated object is mis-aligned.
>
> Fix this by adjusting the allocation size to the nearest power of two,
> which SLAB always guarantees a size-aligned allocation. And because Rust
> guarantees that the original size must be a multiple of alignment and
> the alignment must be a power of two, then the alignment requirement is
> satisfied.
>
> Suggested-by: Vlastimil Babka <[email protected]>
> Co-developed-by: Andreas Hindborg (Samsung) <[email protected]>
> Signed-off-by: Andreas Hindborg (Samsung) <[email protected]>
> Signed-off-by: Boqun Feng <[email protected]>
> Cc: [email protected] # v6.1+
Acked-by: Vlastimil Babka <[email protected]>