From: Benno Lossin <[email protected]>
This is the fourth version of the pin-init API. See [1] for v3.
The tree at [2] contains these patches applied on top of 6.3-rc1.
The Rust-doc documentation of the pin-init API can be found at [3].
These patches are a long way coming, since I held a presentation on
safe pinned initialization at Kangrejos [4]. And my discovery of this
problem was almost a year ago [5].
- #1 enables the `pin_macro` feature which is already stable in Rust
version 1.68.
- #2 adds a utility macro `quote!` for proc-macros. This macro converts
the typed characters directly into Rust tokens that are the output of
proc-macros. It is used by the pin-init API proc-macros.
- #3 changes the `Err` types of the constructor functions of `Arc` and
`UniqueArc` to `AllocError`.
- #4 adds the `assume_init` function to `UniqueArc<MaybeUninit<T>>` that
unsafely assumes the pointee to be initialized and returns a
`UniqueArc<T>`. `UniqueArc::write` is modified to use this new function.
Later patches use it as well.
- #5 adds `Opaque::raw_get` to access the value inside of an `Opaque` from
a raw pointer only.
- #6-11 introduce the pin-init API. The commit message of #4 details the
problem it solves and lays out the overall architecture. The
implementation details are fairly complex; however, this is required to
provide a safe API for users -- reducing the amount of `unsafe` code is a
key goal of the Rust support in the kernel. An example of the
before/after difference from the point of view of users is provided in
the commit message. It is a goal to at some point have pin-init as a
language feature of Rust. A first step in this direction is the Field
Projection RFC [6].
- #12 adds important traits and macros from pin-init to the prelude.
- #13 adds functions for easier initialization of `Opaque<T>` via
FFI and raw pointer initializer functions. This is necessary when writing
Rust wrappers and will be used by Wedson's `sync` module patches.
- #14 improves the `UniqueArc::try_new_uninit` function by using the
pin-init API. The old version first allocated uninitialized memory on the
stack and then moved it into the location in the heap. The new version
directly allocates this on the heap.
- #15 adds functions for initializing already allocated `UniqueArc`s, this
will be used by the Android Binder driver.
--
Changelog:
v3 -> v4:
- Improve documentation.
- Fixing doc-tests imports, comments and full paths in macro examples.
- Implement `Zeroable` for many more types.
- Fix unsoundness: `Zeroable` allowed to create fat pointers with a null
VTABLE pointer.
- Split fallible components from `stack_pin_init!` into
`stack_try_pin_init!`.
- Move `Invariant` type alias and `InitClosure` into `__internal`.
- Change the error type of the constructor functions of `Arc` and
`UniqueArc` to `AllocError`.
- Add `try` variants to `InPlaceInit` to allow custom error types.
- Make `StackInit::init` a safe function.
- Rename `OnlyCallFromDrop::create` to `OnlyCallFromDrop::new`.
- Enable the `pin_macro` feature for use inside of `stack_pin_init!`.
- Make `quote!` and `quote_spanned!` use absolute paths.
v2 -> v3:
- Split the big original commit into six smaller commits.
- Use `PinnedDrop` in the `RawFoo` code example.
- Move the `init::common::ffi_init` functions to `Opaque::ffi_init`.
- Move all `#[doc(hidden)]`, internal types into the new `__internal`
module.
- Specify safety guarantees and requirements of the initializer macros.
- Add a detailed example of what the expanded code of the various macros
looks like.
- Require generics in the initializer macros to use turbofish syntax
(`::<>`).
- Refactor the internals of the initializer macros, this way they have
better type inference and generic arguments can be omitted more often.
- Replace `init::from_value` with a blanket impl of the initializer traits
for ordinary objects.
- Add initializing functions for already allocated `UniqueArc`s.
- Add `Opaque::manual_init` functions akin to `ffi_init`, but they take an
`extern "Rust" fn` instead.
- Documentation and inline comment improvements.
v1 -> v2:
- Split the common module and `UniqueArc::assume_init` into their own
commits.
- Change the generics syntax of `pin_init!` to reflect normal struct
generic syntax.
- Replace `PinnedDrop::__ensure_no_unsafe_op_in_drop` with an only unsafely
creatable token.
- Hide `StackInit<T>` in the docs, because it is internal API.
- Improve macro internals of `pin_init!` according to Gary's review.
- Add check for `PhantomPinned` fields without a `#[pin]` attribute in
`#[pin_data]`, as those fields will not have the intended effect.
- Add docs to `quote.rs`.
[1] https://lore.kernel.org/rust-for-linux/[email protected]/
[2] https://github.com/y86-dev/linux.git patch/pinned-init-v4
[3] https://rust-for-linux.github.io/docs/pinned-init/kernel/init
[4] https://kangrejos.com
[5] https://github.com/Rust-for-Linux/linux/issues/772
[6] https://github.com/rust-lang/rfcs/pull/3318
Benno Lossin (14):
rust: enable the `pin_macro` feature
rust: sync: change error type of constructor functions
rust: sync: add `assume_init` to `UniqueArc`
rust: types: add `Opaque::raw_get`
rust: add pin-init API core
rust: init: add initialization macros
rust: init/sync: add `InPlaceInit` trait to pin-initialize smart
pointers
rust: init: add `PinnedDrop` trait and macros
rust: init: add `stack_pin_init!` macro
rust: init: add `Zeroable` trait and `init::zeroed` function
rust: prelude: add `pin-init` API items to prelude
rust: types: add common init-helper functions for `Opaque`
rust: sync: reduce stack usage of `UniqueArc::try_new_uninit`
rust: sync: add functions for initializing `UniqueArc<MaybeUninit<T>>`
Gary Guo (1):
rust: macros: add `quote!` macro
rust/kernel/init.rs | 1429 ++++++++++++++++++++++++++++++++
rust/kernel/init/__internal.rs | 234 ++++++
rust/kernel/init/macros.rs | 971 ++++++++++++++++++++++
rust/kernel/lib.rs | 7 +
rust/kernel/prelude.rs | 6 +-
rust/kernel/sync/arc.rs | 80 +-
rust/kernel/types.rs | 55 ++
rust/macros/lib.rs | 80 ++
rust/macros/pin_data.rs | 79 ++
rust/macros/pinned_drop.rs | 49 ++
rust/macros/quote.rs | 143 ++++
scripts/Makefile.build | 2 +-
12 files changed, 3127 insertions(+), 8 deletions(-)
create mode 100644 rust/kernel/init.rs
create mode 100644 rust/kernel/init/__internal.rs
create mode 100644 rust/kernel/init/macros.rs
create mode 100644 rust/macros/pin_data.rs
create mode 100644 rust/macros/pinned_drop.rs
create mode 100644 rust/macros/quote.rs
base-commit: fe15c26ee26efa11741a7b632e9f23b01aca4cc6
--
2.39.2
From: Benno Lossin <[email protected]>
Change the error type of the constructors of `Arc` and `UniqueArc` to be
`AllocError` instead of `Error`. This makes the API more clear as to
what can go wrong when calling `try_new` or its variants.
Signed-off-by: Benno Lossin <[email protected]>
Cc: Gary Guo <[email protected]>
Cc: Andreas Hindborg <[email protected]>
Cc: Alice Ryhl <[email protected]>
---
rust/kernel/sync/arc.rs | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/rust/kernel/sync/arc.rs b/rust/kernel/sync/arc.rs
index f2f1c83d72ba..aa7135f0f238 100644
--- a/rust/kernel/sync/arc.rs
+++ b/rust/kernel/sync/arc.rs
@@ -17,11 +17,11 @@
use crate::{
bindings,
- error::Result,
types::{ForeignOwnable, Opaque},
};
use alloc::boxed::Box;
use core::{
+ alloc::AllocError,
marker::{PhantomData, Unsize},
mem::{ManuallyDrop, MaybeUninit},
ops::{Deref, DerefMut},
@@ -149,7 +149,7 @@ unsafe impl<T: ?Sized + Sync + Send> Sync for Arc<T> {}
impl<T> Arc<T> {
/// Constructs a new reference counted instance of `T`.
- pub fn try_new(contents: T) -> Result<Self> {
+ pub fn try_new(contents: T) -> Result<Self, AllocError> {
// INVARIANT: The refcount is initialised to a non-zero value.
let value = ArcInner {
// SAFETY: There are no safety requirements for this FFI call.
@@ -469,7 +469,7 @@ pub struct UniqueArc<T: ?Sized> {
impl<T> UniqueArc<T> {
/// Tries to allocate a new [`UniqueArc`] instance.
- pub fn try_new(value: T) -> Result<Self> {
+ pub fn try_new(value: T) -> Result<Self, AllocError> {
Ok(Self {
// INVARIANT: The newly-created object has a ref-count of 1.
inner: Arc::try_new(value)?,
@@ -477,7 +477,7 @@ impl<T> UniqueArc<T> {
}
/// Tries to allocate a new [`UniqueArc`] instance whose contents are not initialised yet.
- pub fn try_new_uninit() -> Result<UniqueArc<MaybeUninit<T>>> {
+ pub fn try_new_uninit() -> Result<UniqueArc<MaybeUninit<T>>, AllocError> {
Ok(UniqueArc::<MaybeUninit<T>> {
// INVARIANT: The newly-created object has a ref-count of 1.
inner: Arc::try_new(MaybeUninit::uninit())?,
--
2.39.2
From: Benno Lossin <[email protected]>
Adds the `assume_init` function to `UniqueArc<MaybeUninit<T>>` that
unsafely assumes the value to be initialized and yields a value of type
`UniqueArc<T>`. This function is used when manually initializing the
pointee of an `UniqueArc`.
Signed-off-by: Benno Lossin <[email protected]>
Reviewed-by: Wedson Almeida Filho <[email protected]>
Reviewed-by: Andreas Hindborg <[email protected]>
Reviewed-by: Alice Ryhl <[email protected]>
---
rust/kernel/sync/arc.rs | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/rust/kernel/sync/arc.rs b/rust/kernel/sync/arc.rs
index aa7135f0f238..eee7008e5e3e 100644
--- a/rust/kernel/sync/arc.rs
+++ b/rust/kernel/sync/arc.rs
@@ -489,6 +489,17 @@ impl<T> UniqueArc<MaybeUninit<T>> {
/// Converts a `UniqueArc<MaybeUninit<T>>` into a `UniqueArc<T>` by writing a value into it.
pub fn write(mut self, value: T) -> UniqueArc<T> {
self.deref_mut().write(value);
+ // SAFETY: We just wrote the value to be initialized.
+ unsafe { self.assume_init() }
+ }
+
+ /// Unsafely assume that `self` is initialized.
+ ///
+ /// # Safety
+ ///
+ /// The caller guarantees that the value behind this pointer has been initialized. It is
+ /// *immediate* UB to call this when the value is not initialized.
+ pub unsafe fn assume_init(self) -> UniqueArc<T> {
let inner = ManuallyDrop::new(self).inner.ptr;
UniqueArc {
// SAFETY: The new `Arc` is taking over `ptr` from `self.inner` (which won't be
--
2.39.2
From: Benno Lossin <[email protected]>
This function mirrors `UnsafeCell::raw_get`. It avoids creating a
reference and allows solely using raw pointers.
The `pin-init` API will be using this, since uninitialized memory
requires raw pointers.
Signed-off-by: Benno Lossin <[email protected]>
Reviewed-by: Gary Guo <[email protected]>
Reviewed-by: Andreas Hindborg <[email protected]>
Reviewed-by: Alice Ryhl <[email protected]>
---
rust/kernel/types.rs | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/rust/kernel/types.rs b/rust/kernel/types.rs
index 9d0fdbc55843..ff2b2fac951d 100644
--- a/rust/kernel/types.rs
+++ b/rust/kernel/types.rs
@@ -238,6 +238,14 @@ impl<T> Opaque<T> {
pub fn get(&self) -> *mut T {
UnsafeCell::raw_get(self.0.as_ptr())
}
+
+ /// Gets the value behind `this`.
+ ///
+ /// This function is useful to get access to the value without creating intermediate
+ /// references.
+ pub const fn raw_get(this: *const Self) -> *mut T {
+ UnsafeCell::raw_get(this.cast::<UnsafeCell<T>>())
+ }
}
/// A sum type that always holds either a value of type `L` or `R`.
--
2.39.2
From: Benno Lossin <[email protected]>
This API is used to facilitate safe pinned initialization of structs. It
replaces cumbersome `unsafe` manual initialization with elegant safe macro
invocations.
Due to the size of this change it has been split into six commits:
1. This commit introducing the basic public interface: traits and
functions to represent and create initializers.
2. Adds the `#[pin_data]`, `pin_init!`, `try_pin_init!`, `init!` and
`try_init!` macros along with their internal types.
3. Adds the `InPlaceInit` trait that allows using an initializer to create
an object inside of a `Box<T>` and other smart pointers.
4. Adds the `PinnedDrop` trait and adds macro support for it in
the `#[pin_data]` macro.
5. Adds the `stack_pin_init!` macro allowing to pin-initialize a struct on
the stack.
6. Adds the `Zeroable` trait and `init::zeroed` function to initialize
types that have `0x00` in all bytes as a valid bit pattern.
--
In this section the problem that the new pin-init API solves is outlined.
This message describes the entirety of the API, not just the parts
introduced in this commit. For a more granular explanation and additional
information on pinning and this issue, view [1].
Pinning is Rust's way of enforcing the address stability of a value. When a
value gets pinned it will be impossible for safe code to move it to another
location. This is done by wrapping pointers to said object with `Pin<P>`.
This wrapper prevents safe code from creating mutable references to the
object, preventing mutable access, which is needed to move the value.
`Pin<P>` provides `unsafe` functions to circumvent this and allow
modifications regardless. It is then the programmer's responsibility to
uphold the pinning guarantee.
Many kernel data structures require a stable address, because there are
foreign pointers to them which would get invalidated by moving the
structure. Since these data structures are usually embedded in structs to
use them, this pinning property propagates to the container struct.
Resulting in most structs in both Rust and C code needing to be pinned.
So if we want to have a `mutex` field in a Rust struct, this struct also
needs to be pinned, because a `mutex` contains a `list_head`. Additionally
initializing a `list_head` requires already having the final memory
location available, because it is initialized by pointing it to itself. But
this presents another challenge in Rust: values have to be initialized at
all times. There is the `MaybeUninit<T>` wrapper type, which allows
handling uninitialized memory, but this requires using the `unsafe` raw
pointers and a casting the type to the initialized variant.
This problem gets exacerbated when considering encapsulation and the normal
safety requirements of Rust code. The fields of the Rust `Mutex<T>` should
not be accessible to normal driver code. After all if anyone can modify
the fields, there is no way to ensure the invariants of the `Mutex<T>` are
upheld. But if the fields are inaccessible, then initialization of a
`Mutex<T>` needs to be somehow achieved via a function or a macro. Because
the `Mutex<T>` must be pinned in memory, the function cannot return it by
value. It also cannot allocate a `Box` to put the `Mutex<T>` into, because
that is an unnecessary allocation and indirection which would hurt
performance.
The current solution was to split this function into two parts:
1. A `new` function that returns a partially initialized `Mutex<T>`,
2. An `init` function that requires the `Mutex<T>` to be pinned and that
fully initializes the `Mutex<T>`.
Both of these functions have to be marked `unsafe`, since a call to `new`
needs to be accompanied with a call to `init`, otherwise using the
`Mutex<T>` could result in UB. And because calling `init` twice also is not
safe. While `Mutex<T>` initialization cannot fail, other structs might
also have to allocate memory, which would result in conditional successful
initialization requiring even more manual accommodation work.
Combine this with the problem of pin-projections -- the way of accessing
fields of a pinned struct -- which also have an `unsafe` API, pinned
initialization is riddled with `unsafe` resulting in very poor ergonomics.
Not only that, but also having to call two functions possibly multiple
lines apart makes it very easy to forget it outright or during refactoring.
Here is an example of the current way of initializing a struct with two
synchronization primitives (see [2] for the full example):
struct SharedState {
state_changed: CondVar,
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn try_new() -> Result<Arc<Self>> {
let mut state = Pin::from(UniqueArc::try_new(Self {
// SAFETY: `condvar_init!` is called below.
state_changed: unsafe { CondVar::new() },
// SAFETY: `mutex_init!` is called below.
inner: unsafe {
Mutex::new(SharedStateInner { token_count: 0 })
},
})?);
// SAFETY: `state_changed` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.state_changed)
};
kernel::condvar_init!(pinned, "SharedState::state_changed");
// SAFETY: `inner` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.inner)
};
kernel::mutex_init!(pinned, "SharedState::inner");
Ok(state.into())
}
}
The pin-init API of this patch solves this issue by providing a
comprehensive solution comprised of macros and traits. Here is the example
from above using the pin-init API:
#[pin_data]
struct SharedState {
#[pin]
state_changed: CondVar,
#[pin]
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn new() -> impl PinInit<Self> {
pin_init!(Self {
state_changed <- new_condvar!("SharedState::state_changed"),
inner <- new_mutex!(
SharedStateInner { token_count: 0 },
"SharedState::inner",
),
})
}
}
Notably the way the macro is used here requires no `unsafe` and thus comes
with the usual Rust promise of safe code not introducing any memory
violations. Additionally it is now up to the caller of `new()` to decide
the memory location of the `SharedState`. They can choose at the moment
`Arc<T>`, `Box<T>` or the stack.
--
The API has the following architecture:
1. Initializer traits `PinInit<T, E>` and `Init<T, E>` that act like
closures.
2. Macros to create these initializer traits safely.
3. Functions to allow manually writing initializers.
The initializers (an `impl PinInit<T, E>`) receive a raw pointer pointing
to uninitialized memory and their job is to fully initialize a `T` at that
location. If initialization fails, they return an error (`E`) by value.
This way of initializing cannot be safely exposed to the user, since it
relies upon these properties outside of the control of the trait:
- the memory location (slot) needs to be valid memory,
- if initialization fails, the slot should not be read from,
- the value in the slot should be pinned, so it cannot move and the memory
cannot be deallocated until the value is dropped.
This is why using an initializer is facilitated by another trait that
ensures these requirements.
These initializers can be created manually by just supplying a closure that
fulfills the same safety requirements as `PinInit<T, E>`. But this is an
`unsafe` operation. To allow safe initializer creation, the `pin_init!` is
provided along with three other variants: `try_pin_init!`, `try_init!` and
`init!`. These take a modified struct initializer as a parameter and
generate a closure that initializes the fields in sequence.
The macros take great care in upholding the safety requirements:
- A shadowed struct type is used as the return type of the closure instead
of `()`. This is to prevent early returns, as these would prevent full
initialization.
- To ensure every field is only initialized once, a normal struct
initializer is placed in unreachable code. The type checker will emit
errors if a field is missing or specified multiple times.
- When initializing a field fails, the whole initializer will fail and
automatically drop fields that have been initialized earlier.
- Only the correct initializer type is allowed for unpinned fields. You
cannot use a `impl PinInit<T, E>` to initialize a structurally not pinned
field.
To ensure the last point, an additional macro `#[pin_data]` is needed. This
macro annotates the struct itself and the user specifies structurally
pinned and not pinned fields.
Because dropping a pinned struct is also not allowed to break the pinning
invariants, another macro attribute `#[pinned_drop]` is needed. This
macro is introduced in a following commit.
These two macros also have mechanisms to ensure the overall safety of the
API. Additionally, they utilize a combined proc-macro, declarative macro
design: first a proc-macro enables the outer attribute syntax `#[...]` and
does some important pre-parsing. Notably this prepares the generics such
that the declarative macro can handle them using token trees. Then the
actual parsing of the structure and the emission of code is handled by a
declarative macro.
For pin-projections the crates `pin-project` [3] and `pin-project-lite` [4]
had been considered, but were ultimately rejected:
- `pin-project` depends on `syn` [5] which is a very big dependency, around
50k lines of code.
- `pin-project-lite` is a more reasonable 5k lines of code, but contains a
very complex declarative macro to parse generics. On top of that it
would require modification that would need to be maintained
independently.
Link: https://rust-for-linux.com/the-safe-pinned-initialization-problem [1]
Link: https://github.com/Rust-for-Linux/linux/blob/f509ede33fc10a07eba3da14aa00302bd4b5dddd/samples/rust/rust_miscdev.rs [2]
Link: https://crates.io/crates/pin-project [3]
Link: https://crates.io/crates/pin-project-lite [4]
Link: https://crates.io/crates/syn [5]
Co-developed-by: Gary Guo <[email protected]>
Signed-off-by: Gary Guo <[email protected]>
Signed-off-by: Benno Lossin <[email protected]>
Cc: Andreas Hindborg <[email protected]>
Cc: Alice Ryhl <[email protected]>
---
rust/kernel/init.rs | 183 +++++++++++++++++++++++++++++++++
rust/kernel/init/__internal.rs | 45 ++++++++
rust/kernel/lib.rs | 7 ++
scripts/Makefile.build | 2 +-
4 files changed, 236 insertions(+), 1 deletion(-)
create mode 100644 rust/kernel/init.rs
create mode 100644 rust/kernel/init/__internal.rs
diff --git a/rust/kernel/init.rs b/rust/kernel/init.rs
new file mode 100644
index 000000000000..f8d147e47fe7
--- /dev/null
+++ b/rust/kernel/init.rs
@@ -0,0 +1,183 @@
+// SPDX-License-Identifier: Apache-2.0 OR MIT
+
+//! API to safely and fallibly initialize pinned `struct`s using in-place constructors.
+//!
+//! It also allows in-place initialization of big `struct`s that would otherwise produce a stack
+//! overflow.
+//!
+//! Most `struct`s from the [`sync`] module need to be pinned, because they contain self-referential
+//! `struct`s from C. [Pinning][pinning] is Rust's way of ensuring data does not move.
+//!
+//! # Overview
+//!
+//! To initialize a `struct` with an in-place constructor you will need two things:
+//! - an in-place constructor,
+//! - a memory location that can hold your `struct`.
+//!
+//! To get an in-place constructor there are generally two options:
+//! - a custom function/macro returning an in-place constructor provided by someone else,
+//! - using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
+//!
+//! Aside from pinned initialization, this API also supports in-place construction without pinning,
+//! the macros/types/functions are generally named like the pinned variants without the `pin`
+//! prefix.
+//!
+//! [`sync`]: kernel::sync
+//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
+//! [structurally pinned fields]:
+//! https://doc.rust-lang.org/std/pin/index.html#pinning-is-structural-for-field
+//! [`Arc<T>`]: crate::sync::Arc
+//! [`impl PinInit<Foo>`]: PinInit
+//! [`impl PinInit<T, E>`]: PinInit
+//! [`impl Init<T, E>`]: Init
+//! [`Opaque`]: kernel::types::Opaque
+//! [`pin_data`]: ::macros::pin_data
+//! [`UniqueArc<T>`]: kernel::sync::UniqueArc
+//! [`Box<T>`]: alloc::boxed::Box
+
+use core::{convert::Infallible, marker::PhantomData, mem::MaybeUninit};
+
+#[doc(hidden)]
+pub mod __internal;
+
+/// A pin-initializer for the type `T`.
+///
+/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
+/// be [`Box<T>`], [`Arc<T>`], [`UniqueArc<T>`].
+///
+/// Also see the [module description](self).
+///
+/// # Safety
+///
+/// When implementing this type you will need to take great care. Also there are probably very few
+/// cases where a manual implementation is necessary. Use [`pin_init_from_closure`] where possible.
+///
+/// The [`PinInit::__pinned_init`] function
+/// - returns `Ok(())` if it initialized every field of `slot`,
+/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
+/// - `slot` can be deallocated without UB occurring,
+/// - `slot` does not need to be dropped,
+/// - `slot` is not partially initialized.
+/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
+///
+/// [`Arc<T>`]: crate::sync::Arc
+/// [`Arc::pin_init`]: crate::sync::Arc::pin_init
+/// [`UniqueArc<T>`]: kernel::sync::UniqueArc
+/// [`Box<T>`]: alloc::boxed::Box
+#[must_use = "An initializer must be used in order to create its value."]
+pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
+ /// Initializes `slot`.
+ ///
+ /// # Safety
+ ///
+ /// - `slot` is a valid pointer to uninitialized memory.
+ /// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
+ /// deallocate.
+ /// - `slot` will not move until it is dropped, i.e. it will be pinned.
+ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E>;
+}
+
+/// An initializer for `T`.
+///
+/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
+/// be [`Box<T>`], [`Arc<T>`], [`UniqueArc<T>`]. Because [`PinInit<T, E>`] is a super trait, you can
+/// use every function that takes it as well.
+///
+/// Also see the [module description](self).
+///
+/// # Safety
+///
+/// When implementing this type you will need to take great care. Also there are probably very few
+/// cases where a manual implementation is necessary. Use [`init_from_closure`] where possible.
+///
+/// The [`Init::__init`] function
+/// - returns `Ok(())` if it initialized every field of `slot`,
+/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
+/// - `slot` can be deallocated without UB occurring,
+/// - `slot` does not need to be dropped,
+/// - `slot` is not partially initialized.
+/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
+///
+/// The `__pinned_init` function from the supertrait [`PinInit`] needs to execute the exact same
+/// code as `__init`.
+///
+/// Contrary to its supertype [`PinInit<T, E>`] the caller is allowed to
+/// move the pointee after initialization.
+///
+/// [`Arc<T>`]: crate::sync::Arc
+/// [`UniqueArc<T>`]: kernel::sync::UniqueArc
+/// [`Box<T>`]: alloc::boxed::Box
+#[must_use = "An initializer must be used in order to create its value."]
+pub unsafe trait Init<T: ?Sized, E = Infallible>: PinInit<T, E> {
+ /// Initializes `slot`.
+ ///
+ /// # Safety
+ ///
+ /// - `slot` is a valid pointer to uninitialized memory.
+ /// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
+ /// deallocate.
+ unsafe fn __init(self, slot: *mut T) -> Result<(), E>;
+}
+
+/// Creates a new [`PinInit<T, E>`] from the given closure.
+///
+/// # Safety
+///
+/// The closure:
+/// - returns `Ok(())` if it initialized every field of `slot`,
+/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
+/// - `slot` can be deallocated without UB occurring,
+/// - `slot` does not need to be dropped,
+/// - `slot` is not partially initialized.
+/// - may assume that the `slot` does not move if `T: !Unpin`,
+/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
+#[inline]
+pub const unsafe fn pin_init_from_closure<T: ?Sized, E>(
+ f: impl FnOnce(*mut T) -> Result<(), E>,
+) -> impl PinInit<T, E> {
+ __internal::InitClosure(f, PhantomData)
+}
+
+/// Creates a new [`Init<T, E>`] from the given closure.
+///
+/// # Safety
+///
+/// The closure:
+/// - returns `Ok(())` if it initialized every field of `slot`,
+/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
+/// - `slot` can be deallocated without UB occurring,
+/// - `slot` does not need to be dropped,
+/// - `slot` is not partially initialized.
+/// - the `slot` may move after initialization.
+/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
+#[inline]
+pub const unsafe fn init_from_closure<T: ?Sized, E>(
+ f: impl FnOnce(*mut T) -> Result<(), E>,
+) -> impl Init<T, E> {
+ __internal::InitClosure(f, PhantomData)
+}
+
+/// An initializer that leaves the memory uninitialized.
+///
+/// The initializer is a no-op. The `slot` memory is not changed.
+#[inline]
+pub fn uninit<T, E>() -> impl Init<MaybeUninit<T>, E> {
+ // SAFETY: The memory is allowed to be uninitialized.
+ unsafe { init_from_closure(|_| Ok(())) }
+}
+
+// SAFETY: Every type can be initialized by-value.
+unsafe impl<T> PinInit<T> for T {
+ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), Infallible> {
+ unsafe { slot.write(self) };
+ Ok(())
+ }
+}
+
+// SAFETY: Every type can be initialized by-value.
+unsafe impl<T> Init<T> for T {
+ unsafe fn __init(self, slot: *mut T) -> Result<(), Infallible> {
+ unsafe { slot.write(self) };
+ Ok(())
+ }
+}
diff --git a/rust/kernel/init/__internal.rs b/rust/kernel/init/__internal.rs
new file mode 100644
index 000000000000..ddc03c3f0730
--- /dev/null
+++ b/rust/kernel/init/__internal.rs
@@ -0,0 +1,45 @@
+// SPDX-License-Identifier: Apache-2.0 OR MIT
+
+//! This module contains API-internal items for pin-init.
+//!
+//! These items must not be used outside of
+//! - `kernel/init.rs`
+//! - `macros/pin_data.rs`
+//! - `macros/pinned_drop.rs`
+
+use super::*;
+
+/// See the [nomicon] for what subtyping is. See also [this table].
+///
+/// [nomicon]: https://doc.rust-lang.org/nomicon/subtyping.html
+/// [this table]: https://doc.rust-lang.org/nomicon/phantom-data.html#table-of-phantomdata-patterns
+type Invariant<T> = PhantomData<fn(*mut T) -> *mut T>;
+
+/// This is the module-internal type implementing `PinInit` and `Init`. It is unsafe to create this
+/// type, since the closure needs to fulfill the same safety requirement as the
+/// `__pinned_init`/`__init` functions.
+pub(crate) struct InitClosure<F, T: ?Sized, E>(pub(crate) F, pub(crate) Invariant<(E, T)>);
+
+// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
+// `__pinned_init` invariants.
+unsafe impl<T: ?Sized, F, E> PinInit<T, E> for InitClosure<F, T, E>
+where
+ F: FnOnce(*mut T) -> Result<(), E>,
+{
+ #[inline]
+ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
+ (self.0)(slot)
+ }
+}
+
+// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
+// `__init` invariants.
+unsafe impl<T: ?Sized, F, E> Init<T, E> for InitClosure<F, T, E>
+where
+ F: FnOnce(*mut T) -> Result<(), E>,
+{
+ #[inline]
+ unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
+ (self.0)(slot)
+ }
+}
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index 4317b6d5f50b..821bd067151c 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -16,7 +16,9 @@
#![feature(coerce_unsized)]
#![feature(core_ffi_c)]
#![feature(dispatch_from_dyn)]
+#![feature(explicit_generic_args_with_impl_trait)]
#![feature(generic_associated_types)]
+#![feature(new_uninit)]
#![feature(pin_macro)]
#![feature(receiver_trait)]
#![feature(unsize)]
@@ -26,11 +28,16 @@
#[cfg(not(CONFIG_RUST))]
compile_error!("Missing kernel configuration for conditional compilation");
+#[allow(unused_extern_crates)]
+// Allow proc-macros to refer to `::kernel` inside the `kernel` crate (this crate).
+extern crate self as kernel;
+
#[cfg(not(test))]
#[cfg(not(testlib))]
mod allocator;
mod build_assert;
pub mod error;
+pub mod init;
pub mod prelude;
pub mod print;
mod static_assert;
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 76323201232a..f9bdc01c8191 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -277,7 +277,7 @@ $(obj)/%.lst: $(src)/%.c FORCE
# Compile Rust sources (.rs)
# ---------------------------------------------------------------------------
-rust_allowed_features := core_ffi_c
+rust_allowed_features := core_ffi_c,explicit_generic_args_with_impl_trait
rust_common_cmd = \
RUST_MODFILE=$(modfile) $(RUSTC_OR_CLIPPY) $(rust_flags) \
--
2.39.2
From: Benno Lossin <[email protected]>
Add the following initializer macros:
- `#[pin_data]` to annotate structurally pinned fields of structs,
needed for `pin_init!` and `try_pin_init!` to select the correct
initializer of fields.
- `pin_init!` create a pin-initializer for a struct with the
`Infallible` error type.
- `try_pin_init!` create a pin-initializer for a struct with a custom
error type (`kernel::error::Error` is the default).
- `init!` create an in-place-initializer for a struct with the
`Infallible` error type.
- `try_init!` create an in-place-initializer for a struct with a custom
error type (`kernel::error::Error` is the default).
Also add their needed internal helper traits and structs.
Co-developed-by: Gary Guo <[email protected]>
Signed-off-by: Gary Guo <[email protected]>
Signed-off-by: Benno Lossin <[email protected]>
Cc: Alice Ryhl <[email protected]>
Cc: Andreas Hindborg <[email protected]>
---
rust/kernel/init.rs | 810 ++++++++++++++++++++++++++++++++-
rust/kernel/init/__internal.rs | 124 +++++
rust/kernel/init/macros.rs | 707 ++++++++++++++++++++++++++++
rust/macros/lib.rs | 29 ++
rust/macros/pin_data.rs | 79 ++++
rust/macros/quote.rs | 2 -
6 files changed, 1744 insertions(+), 7 deletions(-)
create mode 100644 rust/kernel/init/macros.rs
create mode 100644 rust/macros/pin_data.rs
diff --git a/rust/kernel/init.rs b/rust/kernel/init.rs
index f8d147e47fe7..14cc5d2dafef 100644
--- a/rust/kernel/init.rs
+++ b/rust/kernel/init.rs
@@ -14,7 +14,8 @@
//! - an in-place constructor,
//! - a memory location that can hold your `struct`.
//!
-//! To get an in-place constructor there are generally two options:
+//! To get an in-place constructor there are generally three options:
+//! - directly creating an in-place constructor using the [`pin_init!`] macro,
//! - a custom function/macro returning an in-place constructor provided by someone else,
//! - using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
//!
@@ -22,6 +23,87 @@
//! the macros/types/functions are generally named like the pinned variants without the `pin`
//! prefix.
//!
+//! # Examples
+//!
+//! ## Using the [`pin_init!`] macro
+//!
+//! If you want to use [`PinInit`], then you will have to annotate your `struct` with
+//! `#[`[`pin_data`]`]`. It is a macro that uses `#[pin]` as a marker for
+//! [structurally pinned fields]. After doing this, you can then create an in-place constructor via
+//! [`pin_init!`]. The syntax is almost the same as normal `struct` initializers. The difference is
+//! that you need to write `<-` instead of `:` for fields that you want to initialize in-place.
+//!
+//! ```rust
+//! # #![allow(clippy::disallowed_names, clippy::new_ret_no_self)]
+//! use kernel::{prelude::*, sync::Mutex, new_mutex};
+//! # use core::pin::Pin;
+//! #[pin_data]
+//! struct Foo {
+//! #[pin]
+//! a: Mutex<usize>,
+//! b: u32,
+//! }
+//!
+//! let foo = pin_init!(Foo {
+//! a <- new_mutex!(42, "Foo::a"),
+//! b: 24,
+//! });
+//! ```
+//!
+//! `foo` now is of the type [`impl PinInit<Foo>`]. We can now use any smart pointer that we like
+//! (or just the stack) to actually initialize a `Foo`:
+//!
+//! ```rust
+//! # #![allow(clippy::disallowed_names, clippy::new_ret_no_self)]
+//! # use kernel::{prelude::*, sync::Mutex, new_mutex};
+//! # use core::pin::Pin;
+//! # #[pin_data]
+//! # struct Foo {
+//! # #[pin]
+//! # a: Mutex<usize>,
+//! # b: u32,
+//! # }
+//! # let foo = pin_init!(Foo {
+//! # a <- new_mutex!(42, "Foo::a"),
+//! # b: 24,
+//! # });
+//! let foo: Result<Pin<Box<Foo>>> = Box::pin_init(foo);
+//! ```
+//!
+//! For more information see the [`pin_init!`] macro.
+//!
+//! ## Using a custom function/macro that returns an initializer
+//!
+//! Many types from the kernel supply a function/macro that returns an initializer, because the
+//! above method only works for types where you can access the fields.
+//!
+//! ```rust
+//! # use kernel::{new_mutex, sync::{Arc, Mutex}};
+//! let mtx: Result<Arc<Mutex<usize>>> = Arc::pin_init(new_mutex!(42, "example::mtx"));
+//! ```
+//!
+//! To declare an init macro/function you just return an [`impl PinInit<T, E>`]:
+//!
+//! ```rust
+//! # #![allow(clippy::disallowed_names, clippy::new_ret_no_self)]
+//! # use kernel::{sync::Mutex, prelude::*, new_mutex, init::PinInit, try_pin_init};
+//! #[pin_data]
+//! struct DriverData {
+//! #[pin]
+//! status: Mutex<i32>,
+//! buffer: Box<[u8; 1_000_000]>,
+//! }
+//!
+//! impl DriverData {
+//! fn new() -> impl PinInit<Self, Error> {
+//! try_pin_init!(Self {
+//! status <- new_mutex!(0, "DriverData::status"),
+//! buffer: Box::init(kernel::init::zeroed())?,
+//! })
+//! }
+//! }
+//! ```
+//!
//! [`sync`]: kernel::sync
//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
//! [structurally pinned fields]:
@@ -33,12 +115,732 @@
//! [`Opaque`]: kernel::types::Opaque
//! [`pin_data`]: ::macros::pin_data
//! [`UniqueArc<T>`]: kernel::sync::UniqueArc
-//! [`Box<T>`]: alloc::boxed::Box
-use core::{convert::Infallible, marker::PhantomData, mem::MaybeUninit};
+use alloc::boxed::Box;
+use core::{cell::Cell, convert::Infallible, marker::PhantomData, mem::MaybeUninit, ptr};
#[doc(hidden)]
pub mod __internal;
+#[doc(hidden)]
+pub mod macros;
+
+/// Construct an in-place, pinned initializer for `struct`s.
+///
+/// This macro defaults the error to [`Infallible`]. If you need [`Error`], then use
+/// [`try_pin_init!`].
+///
+/// The syntax is almost identical to that of a normal `struct` initializer:
+///
+/// ```rust
+/// # #![allow(clippy::disallowed_names, clippy::new_ret_no_self)]
+/// # use kernel::{init, pin_init, macros::pin_data, init::*};
+/// # use core::pin::Pin;
+/// #[pin_data]
+/// struct Foo {
+/// a: usize,
+/// b: Bar,
+/// }
+///
+/// #[pin_data]
+/// struct Bar {
+/// x: u32,
+/// }
+///
+/// # fn demo() -> impl PinInit<Foo> {
+/// let a = 42;
+///
+/// let initializer = pin_init!(Foo {
+/// a,
+/// b: Bar {
+/// x: 64,
+/// },
+/// });
+/// # initializer }
+/// # Box::pin_init(demo()).unwrap();
+/// ```
+///
+/// Arbitrary Rust expressions can be used to set the value of a variable.
+///
+/// The fields are initialized in the order that they appear in the initializer. So it is possible
+/// to read already initialized fields using raw pointers.
+///
+/// IMPORTANT: You are not allowed to create references to fields of the struct inside of the
+/// initializer.
+///
+/// # Init-functions
+///
+/// When working with this API it is often desired to let others construct your types without
+/// giving access to all fields. This is where you would normally write a plain function `new`
+/// that would return a new instance of your type. With this API that is also possible.
+/// However, there are a few extra things to keep in mind.
+///
+/// To create an initializer function, simply declare it like this:
+///
+/// ```rust
+/// # #![allow(clippy::disallowed_names, clippy::new_ret_no_self)]
+/// # use kernel::{init, pin_init, prelude::*, init::*};
+/// # use core::pin::Pin;
+/// # #[pin_data]
+/// # struct Foo {
+/// # a: usize,
+/// # b: Bar,
+/// # }
+/// # #[pin_data]
+/// # struct Bar {
+/// # x: u32,
+/// # }
+/// impl Foo {
+/// fn new() -> impl PinInit<Self> {
+/// pin_init!(Self {
+/// a: 42,
+/// b: Bar {
+/// x: 64,
+/// },
+/// })
+/// }
+/// }
+/// ```
+///
+/// Users of `Foo` can now create it like this:
+///
+/// ```rust
+/// # #![allow(clippy::disallowed_names, clippy::new_ret_no_self)]
+/// # use kernel::{init, pin_init, macros::pin_data, init::*};
+/// # use core::pin::Pin;
+/// # #[pin_data]
+/// # struct Foo {
+/// # a: usize,
+/// # b: Bar,
+/// # }
+/// # #[pin_data]
+/// # struct Bar {
+/// # x: u32,
+/// # }
+/// # impl Foo {
+/// # fn new() -> impl PinInit<Self> {
+/// # pin_init!(Self {
+/// # a: 42,
+/// # b: Bar {
+/// # x: 64,
+/// # },
+/// # })
+/// # }
+/// # }
+/// let foo = Box::pin_init(Foo::new());
+/// ```
+///
+/// They can also easily embed it into their own `struct`s:
+///
+/// ```rust
+/// # #![allow(clippy::disallowed_names, clippy::new_ret_no_self)]
+/// # use kernel::{init, pin_init, macros::pin_data, init::*};
+/// # use core::pin::Pin;
+/// # #[pin_data]
+/// # struct Foo {
+/// # a: usize,
+/// # b: Bar,
+/// # }
+/// # #[pin_data]
+/// # struct Bar {
+/// # x: u32,
+/// # }
+/// # impl Foo {
+/// # fn new() -> impl PinInit<Self> {
+/// # pin_init!(Self {
+/// # a: 42,
+/// # b: Bar {
+/// # x: 64,
+/// # },
+/// # })
+/// # }
+/// # }
+/// #[pin_data]
+/// struct FooContainer {
+/// #[pin]
+/// foo1: Foo,
+/// #[pin]
+/// foo2: Foo,
+/// other: u32,
+/// }
+///
+/// impl FooContainer {
+/// fn new(other: u32) -> impl PinInit<Self> {
+/// pin_init!(Self {
+/// foo1 <- Foo::new(),
+/// foo2 <- Foo::new(),
+/// other,
+/// })
+/// }
+/// }
+/// ```
+///
+/// Here we see that when using `pin_init!` with `PinInit`, one needs to write `<-` instead of `:`.
+/// This signifies that the given field is initialized in-place. As with `struct` initializers, just
+/// writing the field (in this case `other`) without `:` or `<-` means `other: other,`.
+///
+/// # Syntax
+///
+/// As already mentioned in the examples above, inside of `pin_init!` a `struct` initializer with
+/// the following modifications is expected:
+/// - Fields that you want to initialize in-place have to use `<-` instead of `:`.
+/// - In front of the initializer you can write `&this in` to have access to a [`NonNull<Self>`]
+/// pointer named `this` inside of the initializer.
+///
+/// For instance:
+///
+/// ```rust
+/// # use kernel::pin_init;
+/// # use macros::pin_data;
+/// # use core::{ptr::addr_of_mut, marker::PhantomPinned};
+/// #[pin_data]
+/// struct Buf {
+/// // `ptr` points into `buf`.
+/// ptr: *mut u8,
+/// buf: [u8; 64],
+/// #[pin]
+/// pin: PhantomPinned,
+/// }
+/// pin_init!(&this in Buf {
+/// buf: [0; 64],
+/// ptr: unsafe { addr_of_mut!((*this.as_ptr()).buf).cast() },
+/// pin: PhantomPinned,
+/// });
+/// ```
+///
+/// [`try_pin_init!`]: kernel::try_pin_init
+/// [`NonNull<Self>`]: core::ptr::NonNull
+/// [`Error`]: kernel::error::Error
+// For a detailed example of how this macro works, see the module documentation of the hidden
+// module `__internal` inside of `init/__internal.rs`.
+#[macro_export]
+macro_rules! pin_init {
+ ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
+ $($fields:tt)*
+ }) => {
+ $crate::try_pin_init!(
+ @this($($this)?),
+ @typ($t $(::<$($generics),*>)?),
+ @fields($($fields)*),
+ @error(::core::convert::Infallible),
+ )
+ };
+}
+
+/// Construct an in-place, fallible pinned initializer for `struct`s.
+///
+/// If the initialization can complete without error (or [`Infallible`]), then use [`pin_init!`].
+///
+/// You can use the `?` operator or use `return Err(err)` inside the initializer to stop
+/// initialization and return the error.
+///
+/// IMPORTANT: if you have `unsafe` code inside of the initializer you have to ensure that when
+/// initialization fails, the memory can be safely deallocated without any further modifications.
+///
+/// This macro defaults the error to [`Error`].
+///
+/// The syntax is identical to [`pin_init!`] with the following exception: you can append `? $type`
+/// after the `struct` initializer to specify the error type you want to use.
+///
+/// # Examples
+///
+/// ```rust
+/// # #![feature(new_uninit)]
+/// use kernel::{init::{self, PinInit}, error::Error};
+/// #[pin_data]
+/// struct BigBuf {
+/// big: Box<[u8; 1024 * 1024 * 1024]>,
+/// small: [u8; 1024 * 1024],
+/// ptr: *mut u8,
+/// }
+///
+/// impl BigBuf {
+/// fn new() -> impl PinInit<Self, Error> {
+/// try_pin_init!(Self {
+/// big: Box::init(init::zeroed())?,
+/// small: [0; 1024 * 1024],
+/// ptr: core::ptr::null_mut(),
+/// }? Error)
+/// }
+/// }
+/// ```
+///
+/// [`Error`]: kernel::error::Error
+// For a detailed example of how this macro works, see the module documentation of the hidden
+// module `__internal` inside of `init/__internal.rs`.
+#[macro_export]
+macro_rules! try_pin_init {
+ ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
+ $($fields:tt)*
+ }) => {
+ $crate::try_pin_init!(
+ @this($($this)?),
+ @typ($t $(::<$($generics),*>)? ),
+ @fields($($fields)*),
+ @error($crate::error::Error),
+ )
+ };
+ ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
+ $($fields:tt)*
+ }? $err:ty) => {
+ $crate::try_pin_init!(
+ @this($($this)?),
+ @typ($t $(::<$($generics),*>)? ),
+ @fields($($fields)*),
+ @error($err),
+ )
+ };
+ (
+ @this($($this:ident)?),
+ @typ($t:ident $(::<$($generics:ty),*>)?),
+ @fields($($fields:tt)*),
+ @error($err:ty),
+ ) => {{
+ // We do not want to allow arbitrary returns, so we declare this type as the `Ok` return
+ // type and shadow it later when we insert the arbitrary user code. That way there will be
+ // no possibility of returning without `unsafe`.
+ struct __InitOk;
+ // Get the pin data from the supplied type.
+ let data = unsafe {
+ use $crate::init::__internal::HasPinData;
+ $t$(::<$($generics),*>)?::__pin_data()
+ };
+ // Ensure that `data` really is of type `PinData` and help with type inference:
+ let init = $crate::init::__internal::PinData::make_closure::<_, __InitOk, $err>(
+ data,
+ move |slot| {
+ {
+ // Shadow the structure so it cannot be used to return early.
+ struct __InitOk;
+ // Create the `this` so it can be referenced by the user inside of the
+ // expressions creating the individual fields.
+ $(let $this = unsafe { ::core::ptr::NonNull::new_unchecked(slot) };)?
+ // Initialize every field.
+ $crate::try_pin_init!(init_slot:
+ @data(data),
+ @slot(slot),
+ @munch_fields($($fields)*,),
+ );
+ // We use unreachable code to ensure that all fields have been mentioned exactly
+ // once, this struct initializer will still be type-checked and complain with a
+ // very natural error message if a field is forgotten/mentioned more than once.
+ #[allow(unreachable_code, clippy::diverging_sub_expression)]
+ if false {
+ $crate::try_pin_init!(make_initializer:
+ @slot(slot),
+ @type_name($t),
+ @munch_fields($($fields)*,),
+ @acc(),
+ );
+ }
+ // Forget all guards, since initialization was a success.
+ $crate::try_pin_init!(forget_guards:
+ @munch_fields($($fields)*,),
+ );
+ }
+ Ok(__InitOk)
+ }
+ );
+ let init = move |slot| -> ::core::result::Result<(), $err> {
+ init(slot).map(|__InitOk| ())
+ };
+ let init = unsafe { $crate::init::pin_init_from_closure::<_, $err>(init) };
+ init
+ }};
+ (init_slot:
+ @data($data:ident),
+ @slot($slot:ident),
+ @munch_fields($(,)?),
+ ) => {
+ // Endpoint of munching, no fields are left.
+ };
+ (init_slot:
+ @data($data:ident),
+ @slot($slot:ident),
+ // In-place initialization syntax.
+ @munch_fields($field:ident <- $val:expr, $($rest:tt)*),
+ ) => {
+ let $field = $val;
+ // Call the initializer.
+ //
+ // SAFETY: `slot` is valid, because we are inside of an initializer closure, we
+ // return when an error/panic occurs.
+ // We also use the `data` to require the correct trait (`Init` or `PinInit`) for `$field`.
+ unsafe { $data.$field(::core::ptr::addr_of_mut!((*$slot).$field), $field)? };
+ // Create the drop guard.
+ //
+ // We only give access to `&DropGuard`, so it cannot be forgotten via safe code.
+ //
+ // SAFETY: We forget the guard later when initialization has succeeded.
+ let $field = &unsafe {
+ $crate::init::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
+ };
+
+ $crate::try_pin_init!(init_slot:
+ @data($data),
+ @slot($slot),
+ @munch_fields($($rest)*),
+ );
+ };
+ (init_slot:
+ @data($data:ident),
+ @slot($slot:ident),
+ // Direct value init, this is safe for every field.
+ @munch_fields($field:ident $(: $val:expr)?, $($rest:tt)*),
+ ) => {
+ $(let $field = $val;)?
+ // Initialize the field.
+ //
+ // SAFETY: The memory at `slot` is uninitialized.
+ unsafe { ::core::ptr::write(::core::ptr::addr_of_mut!((*$slot).$field), $field) };
+ // Create the drop guard:
+ //
+ // We only give access to `&DropGuard`, so it cannot be accidentally forgotten.
+ //
+ // SAFETY: We forget the guard later when initialization has succeeded.
+ let $field = &unsafe {
+ $crate::init::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
+ };
+
+ $crate::try_pin_init!(init_slot:
+ @data($data),
+ @slot($slot),
+ @munch_fields($($rest)*),
+ );
+ };
+ (make_initializer:
+ @slot($slot:ident),
+ @type_name($t:ident),
+ @munch_fields($(,)?),
+ @acc($($acc:tt)*),
+ ) => {
+ // Endpoint, nothing more to munch, create the initializer.
+ // Since we are in the `if false` branch, this will never get executed. We abuse `slot` to
+ // get the correct type inference here:
+ unsafe {
+ ::core::ptr::write($slot, $t {
+ $($acc)*
+ });
+ }
+ };
+ (make_initializer:
+ @slot($slot:ident),
+ @type_name($t:ident),
+ @munch_fields($field:ident <- $val:expr, $($rest:tt)*),
+ @acc($($acc:tt)*),
+ ) => {
+ $crate::try_pin_init!(make_initializer:
+ @slot($slot),
+ @type_name($t),
+ @munch_fields($($rest)*),
+ @acc($($acc)* $field: ::core::panic!(),),
+ );
+ };
+ (make_initializer:
+ @slot($slot:ident),
+ @type_name($t:ident),
+ @munch_fields($field:ident $(: $val:expr)?, $($rest:tt)*),
+ @acc($($acc:tt)*),
+ ) => {
+ $crate::try_pin_init!(make_initializer:
+ @slot($slot),
+ @type_name($t),
+ @munch_fields($($rest)*),
+ @acc($($acc)* $field: ::core::panic!(),),
+ );
+ };
+ (forget_guards:
+ @munch_fields($(,)?),
+ ) => {
+ // Munching finished.
+ };
+ (forget_guards:
+ @munch_fields($field:ident <- $val:expr, $($rest:tt)*),
+ ) => {
+ unsafe { $crate::init::__internal::DropGuard::forget($field) };
+
+ $crate::try_pin_init!(forget_guards:
+ @munch_fields($($rest)*),
+ );
+ };
+ (forget_guards:
+ @munch_fields($field:ident $(: $val:expr)?, $($rest:tt)*),
+ ) => {
+ unsafe { $crate::init::__internal::DropGuard::forget($field) };
+
+ $crate::try_pin_init!(forget_guards:
+ @munch_fields($($rest)*),
+ );
+ };
+}
+
+/// Construct an in-place initializer for `struct`s.
+///
+/// This macro defaults the error to [`Infallible`]. If you need [`Error`], then use
+/// [`try_init!`].
+///
+/// The syntax is identical to [`pin_init!`] and its safety caveats also apply:
+/// - `unsafe` code must guarantee either full initialization or return an error and allow
+/// deallocation of the memory.
+/// - the fields are initialized in the order given in the initializer.
+/// - no references to fields are allowed to be created inside of the initializer.
+///
+/// This initializer is for initializing data in-place that might later be moved. If you want to
+/// pin-initialize, use [`pin_init!`].
+///
+/// [`Error`]: kernel::error::Error
+// For a detailed example of how this macro works, see the module documentation of the hidden
+// module `__internal` inside of `init/__internal.rs`.
+#[macro_export]
+macro_rules! init {
+ ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
+ $($fields:tt)*
+ }) => {
+ $crate::try_init!(
+ @this($($this)?),
+ @typ($t $(::<$($generics),*>)?),
+ @fields($($fields)*),
+ @error(::core::convert::Infallible),
+ )
+ }
+}
+
+/// Construct an in-place fallible initializer for `struct`s.
+///
+/// This macro defaults the error to [`Error`]. If you need [`Infallible`], then use
+/// [`init!`].
+///
+/// The syntax is identical to [`try_pin_init!`]. If you want to specify a custom error,
+/// append `? $type` after the `struct` initializer.
+/// The safety caveats from [`try_pin_init!`] also apply:
+/// - `unsafe` code must guarantee either full initialization or return an error and allow
+/// deallocation of the memory.
+/// - the fields are initialized in the order given in the initializer.
+/// - no references to fields are allowed to be created inside of the initializer.
+///
+/// # Examples
+///
+/// ```rust
+/// use kernel::{init::PinInit, error::Error};
+/// struct BigBuf {
+/// big: Box<[u8; 1024 * 1024 * 1024]>,
+/// small: [u8; 1024 * 1024],
+/// }
+///
+/// impl BigBuf {
+/// fn new() -> impl Init<Self, Error> {
+/// try_init!(Self {
+/// big: {
+/// let zero = Box::try_new_zeroed()?;
+/// unsafe { zero.assume_init() }
+/// },
+/// small: [0; 1024 * 1024],
+/// }? Error)
+/// }
+/// }
+/// ```
+///
+/// [`Error`]: kernel::error::Error
+// For a detailed example of how this macro works, see the module documentation of the hidden
+// module `__internal` inside of `init/__internal.rs`.
+#[macro_export]
+macro_rules! try_init {
+ ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
+ $($fields:tt)*
+ }) => {
+ $crate::try_init!(
+ @this($($this)?),
+ @typ($t $(::<$($generics),*>)?),
+ @fields($($fields)*),
+ @error($crate::error::Error),
+ )
+ };
+ ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
+ $($fields:tt)*
+ }? $err:ty) => {
+ $crate::try_init!(
+ @this($($this)?),
+ @typ($t $(::<$($generics),*>)?),
+ @fields($($fields)*),
+ @error($err),
+ )
+ };
+ (
+ @this($($this:ident)?),
+ @typ($t:ident $(::<$($generics:ty),*>)?),
+ @fields($($fields:tt)*),
+ @error($err:ty),
+ ) => {{
+ // We do not want to allow arbitrary returns, so we declare this type as the `Ok` return
+ // type and shadow it later when we insert the arbitrary user code. That way there will be
+ // no possibility of returning without `unsafe`.
+ struct __InitOk;
+ // Get the init data from the supplied type.
+ let data = unsafe {
+ use $crate::init::__internal::HasInitData;
+ $t$(::<$($generics),*>)?::__init_data()
+ };
+ // Ensure that `data` really is of type `InitData` and help with type inference:
+ let init = $crate::init::__internal::InitData::make_closure::<_, __InitOk, $err>(
+ data,
+ move |slot| {
+ {
+ // Shadow the structure so it cannot be used to return early.
+ struct __InitOk;
+ // Create the `this` so it can be referenced by the user inside of the
+ // expressions creating the individual fields.
+ $(let $this = unsafe { ::core::ptr::NonNull::new_unchecked(slot) };)?
+ // Initialize every field.
+ $crate::try_init!(init_slot:
+ @slot(slot),
+ @munch_fields($($fields)*,),
+ );
+ // We use unreachable code to ensure that all fields have been mentioned exactly
+ // once, this struct initializer will still be type-checked and complain with a
+ // very natural error message if a field is forgotten/mentioned more than once.
+ #[allow(unreachable_code, clippy::diverging_sub_expression)]
+ if false {
+ $crate::try_init!(make_initializer:
+ @slot(slot),
+ @type_name($t),
+ @munch_fields($($fields)*,),
+ @acc(),
+ );
+ }
+ // Forget all guards, since initialization was a success.
+ $crate::try_init!(forget_guards:
+ @munch_fields($($fields)*,),
+ );
+ }
+ Ok(__InitOk)
+ }
+ );
+ let init = move |slot| -> ::core::result::Result<(), $err> {
+ init(slot).map(|__InitOk| ())
+ };
+ let init = unsafe { $crate::init::init_from_closure::<_, $err>(init) };
+ init
+ }};
+ (init_slot:
+ @slot($slot:ident),
+ @munch_fields( $(,)?),
+ ) => {
+ // Endpoint of munching, no fields are left.
+ };
+ (init_slot:
+ @slot($slot:ident),
+ @munch_fields($field:ident <- $val:expr, $($rest:tt)*),
+ ) => {
+ let $field = $val;
+ // Call the initializer.
+ //
+ // SAFETY: `slot` is valid, because we are inside of an initializer closure, we
+ // return when an error/panic occurs.
+ unsafe {
+ $crate::init::Init::__init($field, ::core::ptr::addr_of_mut!((*$slot).$field))?;
+ }
+ // Create the drop guard.
+ //
+ // We only give access to `&DropGuard`, so it cannot be accidentally forgotten.
+ //
+ // SAFETY: We forget the guard later when initialization has succeeded.
+ let $field = &unsafe {
+ $crate::init::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
+ };
+
+ $crate::try_init!(init_slot:
+ @slot($slot),
+ @munch_fields($($rest)*),
+ );
+ };
+ (init_slot:
+ @slot($slot:ident),
+ // Direct value init.
+ @munch_fields($field:ident $(: $val:expr)?, $($rest:tt)*),
+ ) => {
+ $(let $field = $val;)?
+ // Call the initializer.
+ //
+ // SAFETY: The memory at `slot` is uninitialized.
+ unsafe { ::core::ptr::write(::core::ptr::addr_of_mut!((*$slot).$field), $field) };
+ // Create the drop guard.
+ //
+ // We only give access to `&DropGuard`, so it cannot be accidentally forgotten.
+ //
+ // SAFETY: We forget the guard later when initialization has succeeded.
+ let $field = &unsafe {
+ $crate::init::__internal::DropGuard::new(::core::ptr::addr_of_mut!((*$slot).$field))
+ };
+
+ $crate::try_init!(init_slot:
+ @slot($slot),
+ @munch_fields($($rest)*),
+ );
+ };
+ (make_initializer:
+ @slot($slot:ident),
+ @type_name($t:ident),
+ @munch_fields( $(,)?),
+ @acc($($acc:tt)*),
+ ) => {
+ // Endpoint, nothing more to munch, create the initializer.
+ // Since we are in the `if false` branch, this will never get executed. We abuse `slot` to
+ // get the correct type inference here:
+ unsafe {
+ ::core::ptr::write($slot, $t {
+ $($acc)*
+ });
+ }
+ };
+ (make_initializer:
+ @slot($slot:ident),
+ @type_name($t:ident),
+ @munch_fields($field:ident <- $val:expr, $($rest:tt)*),
+ @acc($($acc:tt)*),
+ ) => {
+ $crate::try_init!(make_initializer:
+ @slot($slot),
+ @type_name($t),
+ @munch_fields($($rest)*),
+ @acc($($acc)*$field: ::core::panic!(),),
+ );
+ };
+ (make_initializer:
+ @slot($slot:ident),
+ @type_name($t:ident),
+ @munch_fields($field:ident $(: $val:expr)?, $($rest:tt)*),
+ @acc($($acc:tt)*),
+ ) => {
+ $crate::try_init!(make_initializer:
+ @slot($slot),
+ @type_name($t),
+ @munch_fields($($rest)*),
+ @acc($($acc)*$field: ::core::panic!(),),
+ );
+ };
+ (forget_guards:
+ @munch_fields($(,)?),
+ ) => {
+ // Munching finished.
+ };
+ (forget_guards:
+ @munch_fields($field:ident <- $val:expr, $($rest:tt)*),
+ ) => {
+ unsafe { $crate::init::__internal::DropGuard::forget($field) };
+
+ $crate::try_init!(forget_guards:
+ @munch_fields($($rest)*),
+ );
+ };
+ (forget_guards:
+ @munch_fields($field:ident $(: $val:expr)?, $($rest:tt)*),
+ ) => {
+ unsafe { $crate::init::__internal::DropGuard::forget($field) };
+
+ $crate::try_init!(forget_guards:
+ @munch_fields($($rest)*),
+ );
+ };
+}
/// A pin-initializer for the type `T`.
///
@@ -63,7 +865,6 @@ pub mod __internal;
/// [`Arc<T>`]: crate::sync::Arc
/// [`Arc::pin_init`]: crate::sync::Arc::pin_init
/// [`UniqueArc<T>`]: kernel::sync::UniqueArc
-/// [`Box<T>`]: alloc::boxed::Box
#[must_use = "An initializer must be used in order to create its value."]
pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
/// Initializes `slot`.
@@ -106,7 +907,6 @@ pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
///
/// [`Arc<T>`]: crate::sync::Arc
/// [`UniqueArc<T>`]: kernel::sync::UniqueArc
-/// [`Box<T>`]: alloc::boxed::Box
#[must_use = "An initializer must be used in order to create its value."]
pub unsafe trait Init<T: ?Sized, E = Infallible>: PinInit<T, E> {
/// Initializes `slot`.
diff --git a/rust/kernel/init/__internal.rs b/rust/kernel/init/__internal.rs
index ddc03c3f0730..0105290250f8 100644
--- a/rust/kernel/init/__internal.rs
+++ b/rust/kernel/init/__internal.rs
@@ -43,3 +43,127 @@ where
(self.0)(slot)
}
}
+
+/// This trait is only implemented via the `#[pin_data]` proc-macro. It is used to facilitate
+/// the pin projections within the initializers.
+///
+/// # Safety
+///
+/// Only the `init` module is allowed to use this trait.
+pub unsafe trait HasPinData {
+ type PinData: PinData;
+
+ unsafe fn __pin_data() -> Self::PinData;
+}
+
+/// Marker trait for pinning data of structs.
+///
+/// # Safety
+///
+/// Only the `init` module is allowed to use this trait.
+pub unsafe trait PinData: Copy {
+ type Datee: ?Sized + HasPinData;
+
+ /// Type inference helper function.
+ fn make_closure<F, O, E>(self, f: F) -> F
+ where
+ F: FnOnce(*mut Self::Datee) -> Result<O, E>,
+ {
+ f
+ }
+}
+
+/// This trait is automatically implemented for every type. It aims to provide the same type
+/// inference help as `HasPinData`.
+///
+/// # Safety
+///
+/// Only the `init` module is allowed to use this trait.
+pub unsafe trait HasInitData {
+ type InitData: InitData;
+
+ unsafe fn __init_data() -> Self::InitData;
+}
+
+/// Same function as `PinData`, but for arbitrary data.
+///
+/// # Safety
+///
+/// Only the `init` module is allowed to use this trait.
+pub unsafe trait InitData: Copy {
+ type Datee: ?Sized + HasInitData;
+
+ /// Type inference helper function.
+ fn make_closure<F, O, E>(self, f: F) -> F
+ where
+ F: FnOnce(*mut Self::Datee) -> Result<O, E>,
+ {
+ f
+ }
+}
+
+pub struct AllData<T: ?Sized>(PhantomData<fn(Box<T>) -> Box<T>>);
+
+impl<T: ?Sized> Clone for AllData<T> {
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<T: ?Sized> Copy for AllData<T> {}
+
+unsafe impl<T: ?Sized> InitData for AllData<T> {
+ type Datee = T;
+}
+
+unsafe impl<T: ?Sized> HasInitData for T {
+ type InitData = AllData<T>;
+
+ unsafe fn __init_data() -> Self::InitData {
+ AllData(PhantomData)
+ }
+}
+
+/// When a value of this type is dropped, it drops a `T`.
+///
+/// Can be forgotton to prevent the drop.
+pub struct DropGuard<T: ?Sized>(*mut T, Cell<bool>);
+
+impl<T: ?Sized> DropGuard<T> {
+ /// Creates a new [`DropGuard<T>`]. It will [`ptr::drop_in_place`] `ptr` when it gets dropped.
+ ///
+ /// # Safety
+ ///
+ /// `ptr` must be a valid pointer.
+ ///
+ /// It is the callers responsibility that `self` will only get dropped if the pointee of `ptr`:
+ /// - has not been dropped,
+ /// - is not accessible by any other means,
+ /// - will not be dropped by any other means.
+ #[inline]
+ pub unsafe fn new(ptr: *mut T) -> Self {
+ Self(ptr, Cell::new(true))
+ }
+
+ /// Prevents this guard from dropping the supplied pointer.
+ ///
+ /// # Safety
+ ///
+ /// This function is unsafe in order to prevent safe code from forgetting this guard. It should
+ /// only be called by the macros in this module.
+ #[inline]
+ pub unsafe fn forget(&self) {
+ self.1.set(false);
+ }
+}
+
+impl<T: ?Sized> Drop for DropGuard<T> {
+ #[inline]
+ fn drop(&mut self) {
+ if self.1.get() {
+ // SAFETY: A `DropGuard` can only be constructed using the unsafe `new` function
+ // ensuring that this operation is safe.
+ unsafe { ptr::drop_in_place(self.0) }
+ }
+ }
+}
diff --git a/rust/kernel/init/macros.rs b/rust/kernel/init/macros.rs
new file mode 100644
index 000000000000..e27c309c7ffd
--- /dev/null
+++ b/rust/kernel/init/macros.rs
@@ -0,0 +1,707 @@
+// SPDX-License-Identifier: Apache-2.0 OR MIT
+
+//! This module provides the macros that actually implement the proc-macros `pin_data` and
+//! `pinned_drop`.
+//!
+//! These macros should never be called directly, since they expect their input to be
+//! in a certain format which is internal. Use the proc-macros instead.
+//!
+//! This architecture has been chosen because the kernel does not yet have access to `syn` which
+//! would make matters a lot easier for implementing these as proc-macros.
+//!
+//! # Macro expansion example
+//!
+//! This section is intended for readers trying to understand the macros in this module and the
+//! `pin_init!` macros from `init.rs`.
+//!
+//! We will look at the following example:
+//!
+//! ```rust
+//! # use kernel::init::*;
+//! #[pin_data]
+//! #[repr(C)]
+//! struct Bar<T> {
+//! #[pin]
+//! t: T,
+//! pub x: usize,
+//! }
+//!
+//! impl<T> Bar<T> {
+//! fn new(t: T) -> impl PinInit<Self> {
+//! pin_init!(Self { t, x: 0 })
+//! }
+//! }
+//! ```
+//!
+//! This example includes the most common and important features of the pin-init API.
+//!
+//! Below you can find individual section about the different macro invocations. Here are some
+//! general things we need to take into account when designing macros:
+//! - use global paths, similarly to file paths, these start with the separator: `::core::panic!()`
+//! this ensures that the correct item is used, since users could define their own `mod core {}`
+//! and then their own `panic!` inside to execute arbitrary code inside of our macro.
+//! - macro `unsafe` hygene: we need to ensure that we do not expand arbitrary, user-supplied
+//! expressions inside of an `unsafe` block in the macro, because this would allow users to do
+//! `unsafe` operations without an associated `unsafe` block.
+//!
+//! ## `#[pin_data]` on `Bar`
+//!
+//! This macro is used to specify which fields are structurally pinned and which fields are not. It
+//! is placed on the struct definition and allows `#[pin]` to be placed on the fields.
+//!
+//! Here is the definition of `Bar` from our example:
+//!
+//! ```rust
+//! # use kernel::init::*;
+//! #[pin_data]
+//! #[repr(C)]
+//! struct Bar<T> {
+//! t: T,
+//! pub x: usize,
+//! }
+//! ```
+//!
+//! This expands to the following code:
+//!
+//! ```rust
+//! // Firstly the normal definition of the struct, attributes are preserved:
+//! #[repr(C)]
+//! struct Bar<T> {
+//! t: T,
+//! pub x: usize,
+//! }
+//! // Then an anonymous constant is defined, this is because we do not want any code to access the
+//! // types that we define inside:
+//! const _: () = {
+//! // We define the pin-data carrying struct, it is a ZST and needs to have the same generics,
+//! // since we need to implement access functions for each field and thus need to know its
+//! // type.
+//! struct __ThePinData<T> {
+//! __phantom: ::core::marker::PhantomData<fn(Bar<T>) -> Bar<T>>,
+//! }
+//! // We implement `Copy` for the pin-data struct, since all functions it defines will take
+//! // `self` by value.
+//! impl<T> ::core::clone::Clone for __ThePinData<T> {
+//! fn clone(&self) -> Self {
+//! *self
+//! }
+//! }
+//! impl<T> ::core::marker::Copy for __ThePinData<T> {}
+//! // For every field of `Bar`, the pin-data struct will define a function with the same name
+//! // and accessor (`pub` or `pub(crate)` etc.). This function will take a pointer to the
+//! // field (`slot`) and a `PinInit` or `Init` depending on the projection kind of the field
+//! // (if pinning is structural for the field, then `PinInit` otherwise `Init`).
+//! #[allow(dead_code)]
+//! impl<T> __ThePinData<T> {
+//! unsafe fn t<E>(
+//! self,
+//! slot: *mut T,
+//! init: impl ::kernel::init::Init<T, E>,
+//! ) -> ::core::result::Result<(), E> {
+//! unsafe { ::kernel::init::Init::__init(init, slot) }
+//! }
+//! pub unsafe fn x<E>(
+//! self,
+//! slot: *mut usize,
+//! init: impl ::kernel::init::Init<usize, E>,
+//! ) -> ::core::result::Result<(), E> {
+//! unsafe { ::kernel::init::Init::__init(init, slot) }
+//! }
+//! }
+//! // Implement the internal `HasPinData` trait that associates `Bar` with the pin-data struct
+//! // that we constructed beforehand.
+//! unsafe impl<T> ::kernel::init::__internal::HasPinData for Bar<T> {
+//! type PinData = __ThePinData<T>;
+//! unsafe fn __pin_data() -> Self::PinData {
+//! __ThePinData {
+//! __phantom: ::core::marker::PhantomData,
+//! }
+//! }
+//! }
+//! // Implement the internal `PinData` trait that marks the pin-data struct as a pin-data
+//! // struct. This is important to ensure that no user can implement a rouge `__pin_data`
+//! // function without using `unsafe`.
+//! unsafe impl<T> ::kernel::init::__internal::PinData for __ThePinData<T> {
+//! type Datee = Bar<T>;
+//! }
+//! // Now we only want to implement `Unpin` for `Bar` when every structurally pinned field is
+//! // `Unpin`. In other words, whether `Bar` is `Unpin` only depends on structurally pinned
+//! // fields (those marked with `#[pin]`). These fields will be listed in this struct, in our
+//! // case no such fields exist, hence this is almost empty. The two phantomdata fields exist
+//! // for two reasons:
+//! // - `__phantom`: every generic must be used, since we cannot really know which generics
+//! // are used, we declere all and then use everything here once.
+//! // - `__phantom_pin`: uses the `'__pin` lifetime and ensures that this struct is invariant
+//! // over it. The lifetime is needed to work around the limitation that trait bounds must
+//! // not be trivial, e.g. the user has a `#[pin] PhantomPinned` field -- this is
+//! // unconditionally `!Unpin` and results in an error. The lifetime tricks the compiler
+//! // into accepting these bounds regardless.
+//! #[allow(dead_code)]
+//! struct __Unpin<'__pin, T> {
+//! __phantom_pin: ::core::marker::PhantomData<fn(&'__pin ()) -> &'__pin ()>,
+//! __phantom: ::core::marker::PhantomData<fn(Bar<T>) -> Bar<T>>,
+//! }
+//! #[doc(hidden)]
+//! impl<'__pin, T>
+//! ::core::marker::Unpin for Bar<T> where __Unpin<'__pin, T>: ::core::marker::Unpin {}
+//! // Now we need to ensure that `Bar` does not implement `Drop`, since that would give users
+//! // access to `&mut self` inside of `drop` even if the struct was pinned. This could lead to
+//! // UB with only safe code, so we disallow this by giving a trait implementation error using
+//! // a direct impl and a blanket implementation.
+//! trait MustNotImplDrop {}
+//! // Normally `Drop` bounds do not have the correct semantics, but for this purpose they do
+//! // (normally people want to know if a type has any kind of drop glue at all, here we want
+//! // to know if it has any kind of custom drop glue, which is exactly what this bound does).
+//! #[allow(drop_bounds)]
+//! impl<T: ::core::ops::Drop> MustNotImplDrop for T {}
+//! impl<T> MustNotImplDrop for Bar<T> {}
+//! };
+//! ```
+//!
+//! ## `pin_init!` in `impl Bar`
+//!
+//! This macro creates an pin-initializer for the given struct. It requires that the struct is
+//! annotated by `#[pin_data]`.
+//!
+//! Here is the impl on `Bar` defining the new function:
+//!
+//! ```rust
+//! impl<T> Bar<T> {
+//! fn new(t: T) -> impl PinInit<Self> {
+//! pin_init!(Self { t, x: 0 })
+//! }
+//! }
+//! ```
+//!
+//! This expands to the following code:
+//!
+//! ```rust
+//! impl<T> Bar<T> {
+//! fn new(t: T) -> impl PinInit<Self> {
+//! {
+//! // We do not want to allow arbitrary returns, so we declare this type as the `Ok`
+//! // return type and shadow it later when we insert the arbitrary user code. That way
+//! // there will be no possibility of returning without `unsafe`.
+//! struct __InitOk;
+//! // Get the pin-data type from the initialized type.
+//! // - the function is unsafe, hence the unsafe block
+//! // - we `use` the `HasPinData` trait in the block, it is only available in that
+//! // scope.
+//! let data = unsafe {
+//! use ::kernel::init::__internal::HasPinData;
+//! Self::__pin_data()
+//! };
+//! // Use `data` to help with type inference, the closure supplied will have the type
+//! // `FnOnce(*mut Self) -> Result<__InitOk, Infallible>`.
+//! let init = ::kernel::init::__internal::PinData::make_closure::<
+//! _,
+//! __InitOk,
+//! ::core::convert::Infallible,
+//! >(data, move |slot| {
+//! {
+//! // Shadow the structure so it cannot be used to return early. If a user
+//! // tries to write `return Ok(__InitOk)`, then they get a type error, since
+//! // that will refer to this struct instead of the one defined above.
+//! struct __InitOk;
+//! // This is the expansion of `t,`, which is syntactic sugar for `t: t,`.
+//! unsafe { ::core::ptr::write(&raw mut (*slot).t, t) };
+//! // Since initialization could fail later (not in this case, since the error
+//! // type is `Infallible`) we will need to drop this field if it fails. This
+//! // `DropGuard` will drop the field when it gets dropped and has not yet
+//! // been forgotten. We make a reference to it, so users cannot `mem::forget`
+//! // it from the initializer, since the name is the same as the field.
+//! let t = &unsafe {
+//! ::kernel::init::__internal::DropGuard::new(&raw mut (*slot).t)
+//! };
+//! // Expansion of `x: 0,`:
+//! // Since this can be an arbitrary expression we cannot place it inside of
+//! // the `unsafe` block, so we bind it here.
+//! let x = 0;
+//! unsafe { ::core::ptr::write(&raw mut (*slot).x, x) };
+//! let x = &unsafe {
+//! ::kernel::init::__internal::DropGuard::new(&raw mut (*slot).x)
+//! };
+//!
+//! // Here we use the type checker to ensuer that every field has been
+//! // initialized exactly once, since this is `if false` it will never get
+//! // executed, but still type-checked.
+//! // Additionally we abuse `slot` to automatically infer the correct type for
+//! // the struct. This is also another check that every field is accessible
+//! // from this scope.
+//! #[allow(unreachable_code, clippy::diverging_sub_expression)]
+//! if false {
+//! unsafe {
+//! ::core::ptr::write(
+//! slot,
+//! Self {
+//! // We only care about typecheck finding every field here,
+//! // the expression does not matter, just conjure one using
+//! // `panic!()`:
+//! t: ::core::panic!(),
+//! x: ::core::panic!(),
+//! },
+//! );
+//! };
+//! }
+//! // Since initialization has successfully completed, we can now forget the
+//! // guards.
+//! unsafe { ::kernel::init::__internal::DropGuard::forget(t) };
+//! unsafe { ::kernel::init::__internal::DropGuard::forget(x) };
+//! }
+//! // We leave the scope above and gain access to the previously shadowed
+//! // `__InitOk` that we need to return.
+//! Ok(__InitOk)
+//! });
+//! // Change the return type of the closure.
+//! let init = move |slot| -> ::core::result::Result<(), ::core::convert::Infallible> {
+//! init(slot).map(|__InitOk| ())
+//! };
+//! // Construct the initializer.
+//! let init = unsafe {
+//! ::kernel::init::pin_init_from_closure::<_, ::core::convert::Infallible>(init)
+//! };
+//! init
+//! }
+//! }
+//! }
+//! ```
+
+/// This macro first parses the struct definition such that it separates pinned and not pinned
+/// fields. Afterwards it declares the struct and implement the `PinData` trait safely.
+#[doc(hidden)]
+#[macro_export]
+macro_rules! __pin_data {
+ // Proc-macro entry point, this is supplied by the proc-macro pre-parsing.
+ (parse_input:
+ @args($($pinned_drop:ident)?),
+ @sig(
+ $(#[$($struct_attr:tt)*])*
+ $vis:vis struct $name:ident
+ $(where $($whr:tt)*)?
+ ),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @body({ $($fields:tt)* }),
+ ) => {
+ // We now use token munching to iterate through all of the fields. While doing this we
+ // identify fields marked with `#[pin]`, these fields are the 'pinned fields'. The user
+ // wants these to be structurally pinned. The rest of the fields are the
+ // 'not pinned fields'. Additionally we collect all fields, since we need them in the right
+ // order to declare the struct.
+ //
+ // In this call we also put some explaining comments for the parameters.
+ $crate::__pin_data!(find_pinned_fields:
+ // Attributes on the struct itself, these will just be propagated to be put onto the
+ // struct definition.
+ @struct_attrs($(#[$($struct_attr)*])*),
+ // The visibility of the struct.
+ @vis($vis),
+ // The name of the struct.
+ @name($name),
+ // The 'impl generics', the generics that will need to be specified on the struct inside
+ // of an `impl<$ty_generics>` block.
+ @impl_generics($($impl_generics)*),
+ // The 'ty generics', the generics that will need to be specified on the impl blocks.
+ @ty_generics($($ty_generics)*),
+ // The where clause of any impl block and the declaration.
+ @where($($($whr)*)?),
+ // The remaining fields tokens that need to be processed.
+ // We add a `,` at the end to ensure correct parsing.
+ @fields_munch($($fields)* ,),
+ // The pinned fields.
+ @pinned(),
+ // The not pinned fields.
+ @not_pinned(),
+ // All fields.
+ @fields(),
+ // The accumulator containing all attributes already parsed.
+ @accum(),
+ // Contains `yes` or `` to indicate if `#[pin]` was found on the current field.
+ @is_pinned(),
+ // The proc-macro argument, this should be `PinnedDrop` or ``.
+ @pinned_drop($($pinned_drop)?),
+ );
+ };
+ (find_pinned_fields:
+ @struct_attrs($($struct_attrs:tt)*),
+ @vis($vis:vis),
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ // We found a PhantomPinned field, this should generally be pinned!
+ @fields_munch($field:ident : $($($(::)?core::)?marker::)?PhantomPinned, $($rest:tt)*),
+ @pinned($($pinned:tt)*),
+ @not_pinned($($not_pinned:tt)*),
+ @fields($($fields:tt)*),
+ @accum($($accum:tt)*),
+ // This field is not pinned.
+ @is_pinned(),
+ @pinned_drop($($pinned_drop:ident)?),
+ ) => {
+ ::core::compile_error!(concat!(
+ "The field `",
+ stringify!($field),
+ "` of type `PhantomPinned` only has an effect, if it has the `#[pin]` attribute.",
+ ));
+ $crate::__pin_data!(find_pinned_fields:
+ @struct_attrs($($struct_attrs)*),
+ @vis($vis),
+ @name($name),
+ @impl_generics($($impl_generics)*),
+ @ty_generics($($ty_generics)*),
+ @where($($whr)*),
+ @fields_munch($($rest)*),
+ @pinned($($pinned)* $($accum)* $field: ::core::marker::PhantomPinned,),
+ @not_pinned($($not_pinned)*),
+ @fields($($fields)* $($accum)* $field: ::core::marker::PhantomPinned,),
+ @accum(),
+ @is_pinned(),
+ @pinned_drop($($pinned_drop)?),
+ );
+ };
+ (find_pinned_fields:
+ @struct_attrs($($struct_attrs:tt)*),
+ @vis($vis:vis),
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ // We reached the field declaration.
+ @fields_munch($field:ident : $type:ty, $($rest:tt)*),
+ @pinned($($pinned:tt)*),
+ @not_pinned($($not_pinned:tt)*),
+ @fields($($fields:tt)*),
+ @accum($($accum:tt)*),
+ // This field is pinned.
+ @is_pinned(yes),
+ @pinned_drop($($pinned_drop:ident)?),
+ ) => {
+ $crate::__pin_data!(find_pinned_fields:
+ @struct_attrs($($struct_attrs)*),
+ @vis($vis),
+ @name($name),
+ @impl_generics($($impl_generics)*),
+ @ty_generics($($ty_generics)*),
+ @where($($whr)*),
+ @fields_munch($($rest)*),
+ @pinned($($pinned)* $($accum)* $field: $type,),
+ @not_pinned($($not_pinned)*),
+ @fields($($fields)* $($accum)* $field: $type,),
+ @accum(),
+ @is_pinned(),
+ @pinned_drop($($pinned_drop)?),
+ );
+ };
+ (find_pinned_fields:
+ @struct_attrs($($struct_attrs:tt)*),
+ @vis($vis:vis),
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ // We reached the field declaration.
+ @fields_munch($field:ident : $type:ty, $($rest:tt)*),
+ @pinned($($pinned:tt)*),
+ @not_pinned($($not_pinned:tt)*),
+ @fields($($fields:tt)*),
+ @accum($($accum:tt)*),
+ // This field is not pinned.
+ @is_pinned(),
+ @pinned_drop($($pinned_drop:ident)?),
+ ) => {
+ $crate::__pin_data!(find_pinned_fields:
+ @struct_attrs($($struct_attrs)*),
+ @vis($vis),
+ @name($name),
+ @impl_generics($($impl_generics)*),
+ @ty_generics($($ty_generics)*),
+ @where($($whr)*),
+ @fields_munch($($rest)*),
+ @pinned($($pinned)*),
+ @not_pinned($($not_pinned)* $($accum)* $field: $type,),
+ @fields($($fields)* $($accum)* $field: $type,),
+ @accum(),
+ @is_pinned(),
+ @pinned_drop($($pinned_drop)?),
+ );
+ };
+ (find_pinned_fields:
+ @struct_attrs($($struct_attrs:tt)*),
+ @vis($vis:vis),
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ // We found the `#[pin]` attr.
+ @fields_munch(#[pin] $($rest:tt)*),
+ @pinned($($pinned:tt)*),
+ @not_pinned($($not_pinned:tt)*),
+ @fields($($fields:tt)*),
+ @accum($($accum:tt)*),
+ @is_pinned($($is_pinned:ident)?),
+ @pinned_drop($($pinned_drop:ident)?),
+ ) => {
+ $crate::__pin_data!(find_pinned_fields:
+ @struct_attrs($($struct_attrs)*),
+ @vis($vis),
+ @name($name),
+ @impl_generics($($impl_generics)*),
+ @ty_generics($($ty_generics)*),
+ @where($($whr)*),
+ @fields_munch($($rest)*),
+ // We do not include `#[pin]` in the list of attributes, since it is not actually an
+ // attribute that is defined somewhere.
+ @pinned($($pinned)*),
+ @not_pinned($($not_pinned)*),
+ @fields($($fields)*),
+ @accum($($accum)*),
+ // Set this to `yes`.
+ @is_pinned(yes),
+ @pinned_drop($($pinned_drop)?),
+ );
+ };
+ (find_pinned_fields:
+ @struct_attrs($($struct_attrs:tt)*),
+ @vis($vis:vis),
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ // We reached the field declaration with visibility, for simplicity we only munch the
+ // visibility and put it into `$accum`.
+ @fields_munch($fvis:vis $field:ident $($rest:tt)*),
+ @pinned($($pinned:tt)*),
+ @not_pinned($($not_pinned:tt)*),
+ @fields($($fields:tt)*),
+ @accum($($accum:tt)*),
+ @is_pinned($($is_pinned:ident)?),
+ @pinned_drop($($pinned_drop:ident)?),
+ ) => {
+ $crate::__pin_data!(find_pinned_fields:
+ @struct_attrs($($struct_attrs)*),
+ @vis($vis),
+ @name($name),
+ @impl_generics($($impl_generics)*),
+ @ty_generics($($ty_generics)*),
+ @where($($whr)*),
+ @fields_munch($field $($rest)*),
+ @pinned($($pinned)*),
+ @not_pinned($($not_pinned)*),
+ @fields($($fields)*),
+ @accum($($accum)* $fvis),
+ @is_pinned($($is_pinned)?),
+ @pinned_drop($($pinned_drop)?),
+ );
+ };
+ (find_pinned_fields:
+ @struct_attrs($($struct_attrs:tt)*),
+ @vis($vis:vis),
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ // Some other attribute, just put it into `$accum`.
+ @fields_munch(#[$($attr:tt)*] $($rest:tt)*),
+ @pinned($($pinned:tt)*),
+ @not_pinned($($not_pinned:tt)*),
+ @fields($($fields:tt)*),
+ @accum($($accum:tt)*),
+ @is_pinned($($is_pinned:ident)?),
+ @pinned_drop($($pinned_drop:ident)?),
+ ) => {
+ $crate::__pin_data!(find_pinned_fields:
+ @struct_attrs($($struct_attrs)*),
+ @vis($vis),
+ @name($name),
+ @impl_generics($($impl_generics)*),
+ @ty_generics($($ty_generics)*),
+ @where($($whr)*),
+ @fields_munch($($rest)*),
+ @pinned($($pinned)*),
+ @not_pinned($($not_pinned)*),
+ @fields($($fields)*),
+ @accum($($accum)* #[$($attr)*]),
+ @is_pinned($($is_pinned)?),
+ @pinned_drop($($pinned_drop)?),
+ );
+ };
+ (find_pinned_fields:
+ @struct_attrs($($struct_attrs:tt)*),
+ @vis($vis:vis),
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ // We reached the end of the fields, plus an optional additional comma, since we added one
+ // before and the user is also allowed to put a trailing comma.
+ @fields_munch($(,)?),
+ @pinned($($pinned:tt)*),
+ @not_pinned($($not_pinned:tt)*),
+ @fields($($fields:tt)*),
+ @accum(),
+ @is_pinned(),
+ @pinned_drop($($pinned_drop:ident)?),
+ ) => {
+ // Declare the struct with all fields in the correct order.
+ $($struct_attrs)*
+ $vis struct $name <$($impl_generics)*>
+ where $($whr)*
+ {
+ $($fields)*
+ }
+
+ // We put the rest into this const item, because it then will not be accessible to anything
+ // outside.
+ const _: () = {
+ // We declare this struct which will host all of the projection function for our type.
+ // it will be invariant over all generic parameters which are inherited from the
+ // struct.
+ $vis struct __ThePinData<$($impl_generics)*>
+ where $($whr)*
+ {
+ __phantom: ::core::marker::PhantomData<
+ fn($name<$($ty_generics)*>) -> $name<$($ty_generics)*>
+ >,
+ }
+
+ impl<$($impl_generics)*> ::core::clone::Clone for __ThePinData<$($ty_generics)*>
+ where $($whr)*
+ {
+ fn clone(&self) -> Self { *self }
+ }
+
+ impl<$($impl_generics)*> ::core::marker::Copy for __ThePinData<$($ty_generics)*>
+ where $($whr)*
+ {}
+
+ // Make all projection functions.
+ $crate::__pin_data!(make_pin_data:
+ @pin_data(__ThePinData),
+ @impl_generics($($impl_generics)*),
+ @ty_generics($($ty_generics)*),
+ @where($($whr)*),
+ @pinned($($pinned)*),
+ @not_pinned($($not_pinned)*),
+ );
+
+ // SAFETY: We have added the correct projection functions above to `__ThePinData` and
+ // we also use the least restrictive generics possible.
+ unsafe impl<$($impl_generics)*>
+ $crate::init::__internal::HasPinData for $name<$($ty_generics)*>
+ where $($whr)*
+ {
+ type PinData = __ThePinData<$($ty_generics)*>;
+
+ unsafe fn __pin_data() -> Self::PinData {
+ __ThePinData { __phantom: ::core::marker::PhantomData }
+ }
+ }
+
+ unsafe impl<$($impl_generics)*>
+ $crate::init::__internal::PinData for __ThePinData<$($ty_generics)*>
+ where $($whr)*
+ {
+ type Datee = $name<$($ty_generics)*>;
+ }
+
+ // This struct will be used for the unpin analysis. Since only structurally pinned
+ // fields are relevant whether the struct should implement `Unpin`.
+ #[allow(dead_code)]
+ struct __Unpin <'__pin, $($impl_generics)*>
+ where $($whr)*
+ {
+ __phantom_pin: ::core::marker::PhantomData<fn(&'__pin ()) -> &'__pin ()>,
+ __phantom: ::core::marker::PhantomData<
+ fn($name<$($ty_generics)*>) -> $name<$($ty_generics)*>
+ >,
+ // Only the pinned fields.
+ $($pinned)*
+ }
+
+ #[doc(hidden)]
+ impl<'__pin, $($impl_generics)*> ::core::marker::Unpin for $name<$($ty_generics)*>
+ where
+ __Unpin<'__pin, $($ty_generics)*>: ::core::marker::Unpin,
+ $($whr)*
+ {}
+
+ // We need to disallow normal `Drop` implementation, the exact behavior depends on
+ // whether `PinnedDrop` was specified as the parameter.
+ $crate::__pin_data!(drop_prevention:
+ @name($name),
+ @impl_generics($($impl_generics)*),
+ @ty_generics($($ty_generics)*),
+ @where($($whr)*),
+ @pinned_drop($($pinned_drop)?),
+ );
+ };
+ };
+ // When no `PinnedDrop` was specified, then we have to prevent implementing drop.
+ (drop_prevention:
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ @pinned_drop(),
+ ) => {
+ // We prevent this by creating a trait that will be implemented for all types implementing
+ // `Drop`. Additionally we will implement this trait for the struct leading to a conflict,
+ // if it also implements `Drop`
+ trait MustNotImplDrop {}
+ #[allow(drop_bounds)]
+ impl<T: ::core::ops::Drop> MustNotImplDrop for T {}
+ impl<$($impl_generics)*> MustNotImplDrop for $name<$($ty_generics)*>
+ where $($whr)* {}
+ };
+ // If some other parameter was specified, we emit a readable error.
+ (drop_prevention:
+ @name($name:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ @pinned_drop($($rest:tt)*),
+ ) => {
+ compile_error!(
+ "Wrong parameters to `#[pin_data]`, expected nothing or `PinnedDrop`, got '{}'.",
+ stringify!($($rest)*),
+ );
+ };
+ (make_pin_data:
+ @pin_data($pin_data:ident),
+ @impl_generics($($impl_generics:tt)*),
+ @ty_generics($($ty_generics:tt)*),
+ @where($($whr:tt)*),
+ @pinned($($(#[$($p_attr:tt)*])* $pvis:vis $p_field:ident : $p_type:ty),* $(,)?),
+ @not_pinned($($(#[$($attr:tt)*])* $fvis:vis $field:ident : $type:ty),* $(,)?),
+ ) => {
+ // For every field, we create a projection function according to its projection type. If a
+ // field is structurally pinned, then it must be initialized via `PinInit`, if it is not
+ // structurally pinned, then it can be initialized via `Init`.
+ //
+ // The functions are `unsafe` to prevent accidentally calling them.
+ #[allow(dead_code)]
+ impl<$($impl_generics)*> $pin_data<$($ty_generics)*>
+ where $($whr)*
+ {
+ $(
+ $pvis unsafe fn $p_field<E>(
+ self,
+ slot: *mut $p_type,
+ init: impl $crate::init::PinInit<$p_type, E>,
+ ) -> ::core::result::Result<(), E> {
+ unsafe { $crate::init::PinInit::__pinned_init(init, slot) }
+ }
+ )*
+ $(
+ $fvis unsafe fn $field<E>(
+ self,
+ slot: *mut $type,
+ init: impl $crate::init::Init<$type, E>,
+ ) -> ::core::result::Result<(), E> {
+ unsafe { $crate::init::Init::__init(init, slot) }
+ }
+ )*
+ }
+ };
+}
diff --git a/rust/macros/lib.rs b/rust/macros/lib.rs
index 82b520f024dd..4def038fe71a 100644
--- a/rust/macros/lib.rs
+++ b/rust/macros/lib.rs
@@ -7,6 +7,7 @@ mod quote;
mod concat_idents;
mod helpers;
mod module;
+mod pin_data;
mod vtable;
use proc_macro::TokenStream;
@@ -168,3 +169,31 @@ pub fn vtable(attr: TokenStream, ts: TokenStream) -> TokenStream {
pub fn concat_idents(ts: TokenStream) -> TokenStream {
concat_idents::concat_idents(ts)
}
+
+/// Used to specify the pinning information of the fields of a struct.
+///
+/// This is somewhat similar in purpose as
+/// [pin-project-lite](https://crates.io/crates/pin-project-lite).
+/// Place this macro on a struct definition and then `#[pin]` in front of the attributes of each
+/// field you want to structurally pin.
+///
+/// This macro enables the use of the [`pin_init!`] macro. When pin-initializing a `struct`,
+/// then `#[pin]` directs the type of intializer that is required.
+///
+/// # Examples
+///
+/// ```rust,ignore
+/// #[pin_data]
+/// struct DriverData {
+/// #[pin]
+/// queue: Mutex<Vec<Command>>,
+/// buf: Box<[u8; 1024 * 1024]>,
+/// }
+/// ```
+///
+/// [`pin_init!`]: ../kernel/macro.pin_init.html
+// ^ cannot use direct link, since `kernel` is not a dependency of `macros`.
+#[proc_macro_attribute]
+pub fn pin_data(inner: TokenStream, item: TokenStream) -> TokenStream {
+ pin_data::pin_data(inner, item)
+}
diff --git a/rust/macros/pin_data.rs b/rust/macros/pin_data.rs
new file mode 100644
index 000000000000..954149d77181
--- /dev/null
+++ b/rust/macros/pin_data.rs
@@ -0,0 +1,79 @@
+// SPDX-License-Identifier: Apache-2.0 OR MIT
+
+use proc_macro::{Punct, Spacing, TokenStream, TokenTree};
+
+pub(crate) fn pin_data(args: TokenStream, input: TokenStream) -> TokenStream {
+ // This proc-macro only does some pre-parsing and then delegates the actual parsing to
+ // `kernel::__pin_data!`.
+ //
+ // In here we only collect the generics, since parsing them in declarative macros is very
+ // elaborate. We also do not need to analyse their structure, we only need to collect them.
+
+ // `impl_generics`, the declared generics with their bounds.
+ let mut impl_generics = vec![];
+ // Only the names of the generics, without any bounds.
+ let mut ty_generics = vec![];
+ // Tokens not related to the generics e.g. the `impl` token.
+ let mut rest = vec![];
+ // The current level of `<`.
+ let mut nesting = 0;
+ let mut toks = input.into_iter();
+ // If we are at the beginning of a generic parameter.
+ let mut at_start = true;
+ for tt in &mut toks {
+ match tt.clone() {
+ TokenTree::Punct(p) if p.as_char() == '<' => {
+ if nesting >= 1 {
+ impl_generics.push(tt);
+ }
+ nesting += 1;
+ }
+ TokenTree::Punct(p) if p.as_char() == '>' => {
+ if nesting == 0 {
+ break;
+ } else {
+ nesting -= 1;
+ if nesting >= 1 {
+ impl_generics.push(tt);
+ }
+ if nesting == 0 {
+ break;
+ }
+ }
+ }
+ tt => {
+ if nesting == 1 {
+ match &tt {
+ TokenTree::Ident(i) if i.to_string() == "const" => {}
+ TokenTree::Ident(_) if at_start => {
+ ty_generics.push(tt.clone());
+ ty_generics.push(TokenTree::Punct(Punct::new(',', Spacing::Alone)));
+ at_start = false;
+ }
+ TokenTree::Punct(p) if p.as_char() == ',' => at_start = true,
+ TokenTree::Punct(p) if p.as_char() == '\'' && at_start => {
+ ty_generics.push(tt.clone());
+ }
+ _ => {}
+ }
+ }
+ if nesting >= 1 {
+ impl_generics.push(tt);
+ } else if nesting == 0 {
+ rest.push(tt);
+ }
+ }
+ }
+ }
+ rest.extend(toks);
+ // This should be the body of the struct `{...}`.
+ let last = rest.pop();
+ quote!(::kernel::__pin_data! {
+ parse_input:
+ @args(#args),
+ @sig(#(#rest)*),
+ @impl_generics(#(#impl_generics)*),
+ @ty_generics(#(#ty_generics)*),
+ @body(#last),
+ })
+}
diff --git a/rust/macros/quote.rs b/rust/macros/quote.rs
index 94a6277182ee..c8e08b3c1e4c 100644
--- a/rust/macros/quote.rs
+++ b/rust/macros/quote.rs
@@ -38,7 +38,6 @@ impl ToTokens for TokenStream {
/// This is a similar to the
/// [`quote_spanned!`](https://docs.rs/quote/latest/quote/macro.quote_spanned.html) macro from the
/// `quote` crate but provides only just enough functionality needed by the current `macros` crate.
-#[allow(unused_macros)]
macro_rules! quote_spanned {
($span:expr => $($tt:tt)*) => {
#[allow(clippy::vec_init_then_push)]
@@ -137,7 +136,6 @@ macro_rules! quote_spanned {
/// `macros` crate.
///
/// [`Span::mixed_site()`]: https://doc.rust-lang.org/proc_macro/struct.Span.html#method.mixed_site
-#[allow(unused_macros)]
macro_rules! quote {
($($tt:tt)*) => {
quote_spanned!(::proc_macro::Span::mixed_site() => $($tt)*)
--
2.39.2
On Fri, Mar 31, 2023 at 09:52:59PM +0000, [email protected] wrote:
> From: Benno Lossin <[email protected]>
>
> This API is used to facilitate safe pinned initialization of structs. It
> replaces cumbersome `unsafe` manual initialization with elegant safe macro
> invocations.
>
> Due to the size of this change it has been split into six commits:
> 1. This commit introducing the basic public interface: traits and
> functions to represent and create initializers.
> 2. Adds the `#[pin_data]`, `pin_init!`, `try_pin_init!`, `init!` and
> `try_init!` macros along with their internal types.
> 3. Adds the `InPlaceInit` trait that allows using an initializer to create
> an object inside of a `Box<T>` and other smart pointers.
> 4. Adds the `PinnedDrop` trait and adds macro support for it in
> the `#[pin_data]` macro.
> 5. Adds the `stack_pin_init!` macro allowing to pin-initialize a struct on
> the stack.
> 6. Adds the `Zeroable` trait and `init::zeroed` function to initialize
> types that have `0x00` in all bytes as a valid bit pattern.
>
> --
>
> In this section the problem that the new pin-init API solves is outlined.
> This message describes the entirety of the API, not just the parts
> introduced in this commit. For a more granular explanation and additional
> information on pinning and this issue, view [1].
>
> Pinning is Rust's way of enforcing the address stability of a value. When a
> value gets pinned it will be impossible for safe code to move it to another
> location. This is done by wrapping pointers to said object with `Pin<P>`.
> This wrapper prevents safe code from creating mutable references to the
> object, preventing mutable access, which is needed to move the value.
> `Pin<P>` provides `unsafe` functions to circumvent this and allow
> modifications regardless. It is then the programmer's responsibility to
> uphold the pinning guarantee.
>
> Many kernel data structures require a stable address, because there are
> foreign pointers to them which would get invalidated by moving the
> structure. Since these data structures are usually embedded in structs to
> use them, this pinning property propagates to the container struct.
> Resulting in most structs in both Rust and C code needing to be pinned.
>
> So if we want to have a `mutex` field in a Rust struct, this struct also
> needs to be pinned, because a `mutex` contains a `list_head`. Additionally
> initializing a `list_head` requires already having the final memory
> location available, because it is initialized by pointing it to itself. But
> this presents another challenge in Rust: values have to be initialized at
> all times. There is the `MaybeUninit<T>` wrapper type, which allows
> handling uninitialized memory, but this requires using the `unsafe` raw
> pointers and a casting the type to the initialized variant.
>
> This problem gets exacerbated when considering encapsulation and the normal
> safety requirements of Rust code. The fields of the Rust `Mutex<T>` should
> not be accessible to normal driver code. After all if anyone can modify
> the fields, there is no way to ensure the invariants of the `Mutex<T>` are
> upheld. But if the fields are inaccessible, then initialization of a
> `Mutex<T>` needs to be somehow achieved via a function or a macro. Because
> the `Mutex<T>` must be pinned in memory, the function cannot return it by
> value. It also cannot allocate a `Box` to put the `Mutex<T>` into, because
> that is an unnecessary allocation and indirection which would hurt
> performance.
>
> The current solution was to split this function into two parts:
nit: since this is the only solution that is going upstream, perhaps here we
should say, the "current solution in the the `rust` tree" or something to that
effect so that a reader with less context would know what we're talking about.
>
> 1. A `new` function that returns a partially initialized `Mutex<T>`,
> 2. An `init` function that requires the `Mutex<T>` to be pinned and that
> fully initializes the `Mutex<T>`.
>
> Both of these functions have to be marked `unsafe`, since a call to `new`
> needs to be accompanied with a call to `init`, otherwise using the
> `Mutex<T>` could result in UB. And because calling `init` twice also is not
> safe. While `Mutex<T>` initialization cannot fail, other structs might
> also have to allocate memory, which would result in conditional successful
> initialization requiring even more manual accommodation work.
>
> Combine this with the problem of pin-projections -- the way of accessing
> fields of a pinned struct -- which also have an `unsafe` API, pinned
> initialization is riddled with `unsafe` resulting in very poor ergonomics.
> Not only that, but also having to call two functions possibly multiple
> lines apart makes it very easy to forget it outright or during refactoring.
>
> Here is an example of the current way of initializing a struct with two
> synchronization primitives (see [2] for the full example):
>
> struct SharedState {
> state_changed: CondVar,
> inner: Mutex<SharedStateInner>,
> }
>
> impl SharedState {
> fn try_new() -> Result<Arc<Self>> {
> let mut state = Pin::from(UniqueArc::try_new(Self {
> // SAFETY: `condvar_init!` is called below.
> state_changed: unsafe { CondVar::new() },
> // SAFETY: `mutex_init!` is called below.
> inner: unsafe {
> Mutex::new(SharedStateInner { token_count: 0 })
> },
> })?);
>
> // SAFETY: `state_changed` is pinned when `state` is.
> let pinned = unsafe {
> state.as_mut().map_unchecked_mut(|s| &mut s.state_changed)
> };
> kernel::condvar_init!(pinned, "SharedState::state_changed");
>
> // SAFETY: `inner` is pinned when `state` is.
> let pinned = unsafe {
> state.as_mut().map_unchecked_mut(|s| &mut s.inner)
> };
> kernel::mutex_init!(pinned, "SharedState::inner");
>
> Ok(state.into())
> }
> }
>
> The pin-init API of this patch solves this issue by providing a
> comprehensive solution comprised of macros and traits. Here is the example
> from above using the pin-init API:
>
> #[pin_data]
> struct SharedState {
> #[pin]
> state_changed: CondVar,
> #[pin]
> inner: Mutex<SharedStateInner>,
> }
>
> impl SharedState {
> fn new() -> impl PinInit<Self> {
> pin_init!(Self {
> state_changed <- new_condvar!("SharedState::state_changed"),
> inner <- new_mutex!(
> SharedStateInner { token_count: 0 },
> "SharedState::inner",
> ),
> })
> }
> }
>
> Notably the way the macro is used here requires no `unsafe` and thus comes
> with the usual Rust promise of safe code not introducing any memory
> violations. Additionally it is now up to the caller of `new()` to decide
> the memory location of the `SharedState`. They can choose at the moment
> `Arc<T>`, `Box<T>` or the stack.
>
> --
>
> The API has the following architecture:
> 1. Initializer traits `PinInit<T, E>` and `Init<T, E>` that act like
> closures.
> 2. Macros to create these initializer traits safely.
> 3. Functions to allow manually writing initializers.
>
> The initializers (an `impl PinInit<T, E>`) receive a raw pointer pointing
> to uninitialized memory and their job is to fully initialize a `T` at that
> location. If initialization fails, they return an error (`E`) by value.
>
> This way of initializing cannot be safely exposed to the user, since it
> relies upon these properties outside of the control of the trait:
> - the memory location (slot) needs to be valid memory,
> - if initialization fails, the slot should not be read from,
> - the value in the slot should be pinned, so it cannot move and the memory
> cannot be deallocated until the value is dropped.
>
> This is why using an initializer is facilitated by another trait that
> ensures these requirements.
>
> These initializers can be created manually by just supplying a closure that
> fulfills the same safety requirements as `PinInit<T, E>`. But this is an
> `unsafe` operation. To allow safe initializer creation, the `pin_init!` is
> provided along with three other variants: `try_pin_init!`, `try_init!` and
> `init!`. These take a modified struct initializer as a parameter and
> generate a closure that initializes the fields in sequence.
> The macros take great care in upholding the safety requirements:
> - A shadowed struct type is used as the return type of the closure instead
> of `()`. This is to prevent early returns, as these would prevent full
> initialization.
> - To ensure every field is only initialized once, a normal struct
> initializer is placed in unreachable code. The type checker will emit
> errors if a field is missing or specified multiple times.
> - When initializing a field fails, the whole initializer will fail and
> automatically drop fields that have been initialized earlier.
> - Only the correct initializer type is allowed for unpinned fields. You
> cannot use a `impl PinInit<T, E>` to initialize a structurally not pinned
> field.
>
> To ensure the last point, an additional macro `#[pin_data]` is needed. This
> macro annotates the struct itself and the user specifies structurally
> pinned and not pinned fields.
>
> Because dropping a pinned struct is also not allowed to break the pinning
> invariants, another macro attribute `#[pinned_drop]` is needed. This
> macro is introduced in a following commit.
>
> These two macros also have mechanisms to ensure the overall safety of the
> API. Additionally, they utilize a combined proc-macro, declarative macro
> design: first a proc-macro enables the outer attribute syntax `#[...]` and
> does some important pre-parsing. Notably this prepares the generics such
> that the declarative macro can handle them using token trees. Then the
> actual parsing of the structure and the emission of code is handled by a
> declarative macro.
>
> For pin-projections the crates `pin-project` [3] and `pin-project-lite` [4]
> had been considered, but were ultimately rejected:
> - `pin-project` depends on `syn` [5] which is a very big dependency, around
> 50k lines of code.
> - `pin-project-lite` is a more reasonable 5k lines of code, but contains a
> very complex declarative macro to parse generics. On top of that it
> would require modification that would need to be maintained
> independently.
>
> Link: https://rust-for-linux.com/the-safe-pinned-initialization-problem [1]
> Link: https://github.com/Rust-for-Linux/linux/blob/f509ede33fc10a07eba3da14aa00302bd4b5dddd/samples/rust/rust_miscdev.rs [2]
> Link: https://crates.io/crates/pin-project [3]
> Link: https://crates.io/crates/pin-project-lite [4]
> Link: https://crates.io/crates/syn [5]
> Co-developed-by: Gary Guo <[email protected]>
> Signed-off-by: Gary Guo <[email protected]>
> Signed-off-by: Benno Lossin <[email protected]>
> Cc: Andreas Hindborg <[email protected]>
> Cc: Alice Ryhl <[email protected]>
Reviewed-by: Wedson Almeida Filho <[email protected]>
> ---
> rust/kernel/init.rs | 183 +++++++++++++++++++++++++++++++++
> rust/kernel/init/__internal.rs | 45 ++++++++
> rust/kernel/lib.rs | 7 ++
> scripts/Makefile.build | 2 +-
> 4 files changed, 236 insertions(+), 1 deletion(-)
> create mode 100644 rust/kernel/init.rs
> create mode 100644 rust/kernel/init/__internal.rs
>
> diff --git a/rust/kernel/init.rs b/rust/kernel/init.rs
> new file mode 100644
> index 000000000000..f8d147e47fe7
> --- /dev/null
> +++ b/rust/kernel/init.rs
> @@ -0,0 +1,183 @@
> +// SPDX-License-Identifier: Apache-2.0 OR MIT
> +
> +//! API to safely and fallibly initialize pinned `struct`s using in-place constructors.
> +//!
> +//! It also allows in-place initialization of big `struct`s that would otherwise produce a stack
> +//! overflow.
> +//!
> +//! Most `struct`s from the [`sync`] module need to be pinned, because they contain self-referential
> +//! `struct`s from C. [Pinning][pinning] is Rust's way of ensuring data does not move.
> +//!
> +//! # Overview
> +//!
> +//! To initialize a `struct` with an in-place constructor you will need two things:
> +//! - an in-place constructor,
> +//! - a memory location that can hold your `struct`.
> +//!
> +//! To get an in-place constructor there are generally two options:
> +//! - a custom function/macro returning an in-place constructor provided by someone else,
> +//! - using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
> +//!
> +//! Aside from pinned initialization, this API also supports in-place construction without pinning,
> +//! the macros/types/functions are generally named like the pinned variants without the `pin`
> +//! prefix.
> +//!
> +//! [`sync`]: kernel::sync
> +//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
> +//! [structurally pinned fields]:
> +//! https://doc.rust-lang.org/std/pin/index.html#pinning-is-structural-for-field
> +//! [`Arc<T>`]: crate::sync::Arc
> +//! [`impl PinInit<Foo>`]: PinInit
> +//! [`impl PinInit<T, E>`]: PinInit
> +//! [`impl Init<T, E>`]: Init
> +//! [`Opaque`]: kernel::types::Opaque
> +//! [`pin_data`]: ::macros::pin_data
> +//! [`UniqueArc<T>`]: kernel::sync::UniqueArc
> +//! [`Box<T>`]: alloc::boxed::Box
> +
> +use core::{convert::Infallible, marker::PhantomData, mem::MaybeUninit};
> +
> +#[doc(hidden)]
> +pub mod __internal;
> +
> +/// A pin-initializer for the type `T`.
> +///
> +/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
> +/// be [`Box<T>`], [`Arc<T>`], [`UniqueArc<T>`].
> +///
> +/// Also see the [module description](self).
> +///
> +/// # Safety
> +///
> +/// When implementing this type you will need to take great care. Also there are probably very few
> +/// cases where a manual implementation is necessary. Use [`pin_init_from_closure`] where possible.
> +///
> +/// The [`PinInit::__pinned_init`] function
> +/// - returns `Ok(())` if it initialized every field of `slot`,
> +/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
> +/// - `slot` can be deallocated without UB occurring,
> +/// - `slot` does not need to be dropped,
> +/// - `slot` is not partially initialized.
> +/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
> +///
> +/// [`Arc<T>`]: crate::sync::Arc
> +/// [`Arc::pin_init`]: crate::sync::Arc::pin_init
> +/// [`UniqueArc<T>`]: kernel::sync::UniqueArc
> +/// [`Box<T>`]: alloc::boxed::Box
> +#[must_use = "An initializer must be used in order to create its value."]
> +pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
> + /// Initializes `slot`.
> + ///
> + /// # Safety
> + ///
> + /// - `slot` is a valid pointer to uninitialized memory.
> + /// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
> + /// deallocate.
> + /// - `slot` will not move until it is dropped, i.e. it will be pinned.
> + unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E>;
> +}
> +
> +/// An initializer for `T`.
> +///
> +/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
> +/// be [`Box<T>`], [`Arc<T>`], [`UniqueArc<T>`]. Because [`PinInit<T, E>`] is a super trait, you can
> +/// use every function that takes it as well.
> +///
> +/// Also see the [module description](self).
> +///
> +/// # Safety
> +///
> +/// When implementing this type you will need to take great care. Also there are probably very few
> +/// cases where a manual implementation is necessary. Use [`init_from_closure`] where possible.
> +///
> +/// The [`Init::__init`] function
> +/// - returns `Ok(())` if it initialized every field of `slot`,
> +/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
> +/// - `slot` can be deallocated without UB occurring,
> +/// - `slot` does not need to be dropped,
> +/// - `slot` is not partially initialized.
> +/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
> +///
> +/// The `__pinned_init` function from the supertrait [`PinInit`] needs to execute the exact same
> +/// code as `__init`.
> +///
> +/// Contrary to its supertype [`PinInit<T, E>`] the caller is allowed to
> +/// move the pointee after initialization.
> +///
> +/// [`Arc<T>`]: crate::sync::Arc
> +/// [`UniqueArc<T>`]: kernel::sync::UniqueArc
> +/// [`Box<T>`]: alloc::boxed::Box
> +#[must_use = "An initializer must be used in order to create its value."]
> +pub unsafe trait Init<T: ?Sized, E = Infallible>: PinInit<T, E> {
> + /// Initializes `slot`.
> + ///
> + /// # Safety
> + ///
> + /// - `slot` is a valid pointer to uninitialized memory.
> + /// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
> + /// deallocate.
> + unsafe fn __init(self, slot: *mut T) -> Result<(), E>;
> +}
Rather than making `PinInit<T, E>` a supertrait here, wouldn't it make sense to
just implement `PinInit<T, E>` for every type that implements `Init<T, E>`? This
way we could relax the unsafe requirement that "`__pinned_init` needs to mimick
`__init`" by simply making it so. Something like (plus a proper safety comment):
unsafe impl<T, U: ?Sized, E> PinInit<U, E> for T where T: Init<U, E> {
unsafe fn __pinned_init(self, slot: *mut U) -> Result<(), E> {
unsafe { T::__init(self, slot) }
}
}
> +
> +/// Creates a new [`PinInit<T, E>`] from the given closure.
> +///
> +/// # Safety
> +///
> +/// The closure:
> +/// - returns `Ok(())` if it initialized every field of `slot`,
> +/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
> +/// - `slot` can be deallocated without UB occurring,
> +/// - `slot` does not need to be dropped,
> +/// - `slot` is not partially initialized.
> +/// - may assume that the `slot` does not move if `T: !Unpin`,
> +/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
> +#[inline]
> +pub const unsafe fn pin_init_from_closure<T: ?Sized, E>(
> + f: impl FnOnce(*mut T) -> Result<(), E>,
> +) -> impl PinInit<T, E> {
> + __internal::InitClosure(f, PhantomData)
> +}
> +
> +/// Creates a new [`Init<T, E>`] from the given closure.
> +///
> +/// # Safety
> +///
> +/// The closure:
> +/// - returns `Ok(())` if it initialized every field of `slot`,
> +/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
> +/// - `slot` can be deallocated without UB occurring,
> +/// - `slot` does not need to be dropped,
> +/// - `slot` is not partially initialized.
> +/// - the `slot` may move after initialization.
> +/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
> +#[inline]
> +pub const unsafe fn init_from_closure<T: ?Sized, E>(
> + f: impl FnOnce(*mut T) -> Result<(), E>,
> +) -> impl Init<T, E> {
> + __internal::InitClosure(f, PhantomData)
> +}
> +
> +/// An initializer that leaves the memory uninitialized.
> +///
> +/// The initializer is a no-op. The `slot` memory is not changed.
> +#[inline]
> +pub fn uninit<T, E>() -> impl Init<MaybeUninit<T>, E> {
> + // SAFETY: The memory is allowed to be uninitialized.
> + unsafe { init_from_closure(|_| Ok(())) }
> +}
> +
> +// SAFETY: Every type can be initialized by-value.
> +unsafe impl<T> PinInit<T> for T {
> + unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), Infallible> {
> + unsafe { slot.write(self) };
> + Ok(())
> + }
> +}
> +
> +// SAFETY: Every type can be initialized by-value.
> +unsafe impl<T> Init<T> for T {
> + unsafe fn __init(self, slot: *mut T) -> Result<(), Infallible> {
> + unsafe { slot.write(self) };
> + Ok(())
> + }
> +}
> diff --git a/rust/kernel/init/__internal.rs b/rust/kernel/init/__internal.rs
> new file mode 100644
> index 000000000000..ddc03c3f0730
> --- /dev/null
> +++ b/rust/kernel/init/__internal.rs
> @@ -0,0 +1,45 @@
> +// SPDX-License-Identifier: Apache-2.0 OR MIT
> +
> +//! This module contains API-internal items for pin-init.
> +//!
> +//! These items must not be used outside of
> +//! - `kernel/init.rs`
> +//! - `macros/pin_data.rs`
> +//! - `macros/pinned_drop.rs`
> +
> +use super::*;
> +
> +/// See the [nomicon] for what subtyping is. See also [this table].
> +///
> +/// [nomicon]: https://doc.rust-lang.org/nomicon/subtyping.html
> +/// [this table]: https://doc.rust-lang.org/nomicon/phantom-data.html#table-of-phantomdata-patterns
> +type Invariant<T> = PhantomData<fn(*mut T) -> *mut T>;
> +
> +/// This is the module-internal type implementing `PinInit` and `Init`. It is unsafe to create this
> +/// type, since the closure needs to fulfill the same safety requirement as the
> +/// `__pinned_init`/`__init` functions.
> +pub(crate) struct InitClosure<F, T: ?Sized, E>(pub(crate) F, pub(crate) Invariant<(E, T)>);
> +
> +// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
> +// `__pinned_init` invariants.
> +unsafe impl<T: ?Sized, F, E> PinInit<T, E> for InitClosure<F, T, E>
> +where
> + F: FnOnce(*mut T) -> Result<(), E>,
> +{
> + #[inline]
> + unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
> + (self.0)(slot)
> + }
> +}
> +
> +// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
> +// `__init` invariants.
> +unsafe impl<T: ?Sized, F, E> Init<T, E> for InitClosure<F, T, E>
> +where
> + F: FnOnce(*mut T) -> Result<(), E>,
> +{
> + #[inline]
> + unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
> + (self.0)(slot)
> + }
> +}
Since these are only used to implement `init_from_closure` and
`pin_init_from_closure`, would it make sense to move them into
`init_from_closure`, and have `pin_init_from_closure` just call
`init_from_closure` (given that if a type implements Init<T, E>, it also
implements PinInit<T, E>)?
> diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
> index 4317b6d5f50b..821bd067151c 100644
> --- a/rust/kernel/lib.rs
> +++ b/rust/kernel/lib.rs
> @@ -16,7 +16,9 @@
> #![feature(coerce_unsized)]
> #![feature(core_ffi_c)]
> #![feature(dispatch_from_dyn)]
> +#![feature(explicit_generic_args_with_impl_trait)]
> #![feature(generic_associated_types)]
> +#![feature(new_uninit)]
> #![feature(pin_macro)]
> #![feature(receiver_trait)]
> #![feature(unsize)]
> @@ -26,11 +28,16 @@
> #[cfg(not(CONFIG_RUST))]
> compile_error!("Missing kernel configuration for conditional compilation");
>
> +#[allow(unused_extern_crates)]
> +// Allow proc-macros to refer to `::kernel` inside the `kernel` crate (this crate).
> +extern crate self as kernel;
> +
> #[cfg(not(test))]
> #[cfg(not(testlib))]
> mod allocator;
> mod build_assert;
> pub mod error;
> +pub mod init;
> pub mod prelude;
> pub mod print;
> mod static_assert;
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index 76323201232a..f9bdc01c8191 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -277,7 +277,7 @@ $(obj)/%.lst: $(src)/%.c FORCE
> # Compile Rust sources (.rs)
> # ---------------------------------------------------------------------------
>
> -rust_allowed_features := core_ffi_c
> +rust_allowed_features := core_ffi_c,explicit_generic_args_with_impl_trait
>
> rust_common_cmd = \
> RUST_MODFILE=$(modfile) $(RUSTC_OR_CLIPPY) $(rust_flags) \
> --
> 2.39.2
>
>
On 01.04.23 07:24, Wedson Almeida Filho wrote:
> On Fri, Mar 31, 2023 at 09:52:59PM +0000, [email protected] wrote:
>> From: Benno Lossin <[email protected]>
>>
>> This API is used to facilitate safe pinned initialization of structs. It
>> replaces cumbersome `unsafe` manual initialization with elegant safe macro
>> invocations.
>>
>> Due to the size of this change it has been split into six commits:
>> 1. This commit introducing the basic public interface: traits and
>> functions to represent and create initializers.
>> 2. Adds the `#[pin_data]`, `pin_init!`, `try_pin_init!`, `init!` and
>> `try_init!` macros along with their internal types.
>> 3. Adds the `InPlaceInit` trait that allows using an initializer to create
>> an object inside of a `Box<T>` and other smart pointers.
>> 4. Adds the `PinnedDrop` trait and adds macro support for it in
>> the `#[pin_data]` macro.
>> 5. Adds the `stack_pin_init!` macro allowing to pin-initialize a struct on
>> the stack.
>> 6. Adds the `Zeroable` trait and `init::zeroed` function to initialize
>> types that have `0x00` in all bytes as a valid bit pattern.
>>
>> --
>>
>> In this section the problem that the new pin-init API solves is outlined.
>> This message describes the entirety of the API, not just the parts
>> introduced in this commit. For a more granular explanation and additional
>> information on pinning and this issue, view [1].
>>
>> Pinning is Rust's way of enforcing the address stability of a value. When a
>> value gets pinned it will be impossible for safe code to move it to another
>> location. This is done by wrapping pointers to said object with `Pin<P>`.
>> This wrapper prevents safe code from creating mutable references to the
>> object, preventing mutable access, which is needed to move the value.
>> `Pin<P>` provides `unsafe` functions to circumvent this and allow
>> modifications regardless. It is then the programmer's responsibility to
>> uphold the pinning guarantee.
>>
>> Many kernel data structures require a stable address, because there are
>> foreign pointers to them which would get invalidated by moving the
>> structure. Since these data structures are usually embedded in structs to
>> use them, this pinning property propagates to the container struct.
>> Resulting in most structs in both Rust and C code needing to be pinned.
>>
>> So if we want to have a `mutex` field in a Rust struct, this struct also
>> needs to be pinned, because a `mutex` contains a `list_head`. Additionally
>> initializing a `list_head` requires already having the final memory
>> location available, because it is initialized by pointing it to itself. But
>> this presents another challenge in Rust: values have to be initialized at
>> all times. There is the `MaybeUninit<T>` wrapper type, which allows
>> handling uninitialized memory, but this requires using the `unsafe` raw
>> pointers and a casting the type to the initialized variant.
>>
>> This problem gets exacerbated when considering encapsulation and the normal
>> safety requirements of Rust code. The fields of the Rust `Mutex<T>` should
>> not be accessible to normal driver code. After all if anyone can modify
>> the fields, there is no way to ensure the invariants of the `Mutex<T>` are
>> upheld. But if the fields are inaccessible, then initialization of a
>> `Mutex<T>` needs to be somehow achieved via a function or a macro. Because
>> the `Mutex<T>` must be pinned in memory, the function cannot return it by
>> value. It also cannot allocate a `Box` to put the `Mutex<T>` into, because
>> that is an unnecessary allocation and indirection which would hurt
>> performance.
>>
>> The current solution was to split this function into two parts:
>
> nit: since this is the only solution that is going upstream, perhaps here we
> should say, the "current solution in the the `rust` tree" or something to that
> effect so that a reader with less context would know what we're talking abo
Sure.
>>
>> 1. A `new` function that returns a partially initialized `Mutex<T>`,
>> 2. An `init` function that requires the `Mutex<T>` to be pinned and that
>> fully initializes the `Mutex<T>`.
>>
>> Both of these functions have to be marked `unsafe`, since a call to `new`
>> needs to be accompanied with a call to `init`, otherwise using the
>> `Mutex<T>` could result in UB. And because calling `init` twice also is not
>> safe. While `Mutex<T>` initialization cannot fail, other structs might
>> also have to allocate memory, which would result in conditional successful
>> initialization requiring even more manual accommodation work.
>>
>> Combine this with the problem of pin-projections -- the way of accessing
>> fields of a pinned struct -- which also have an `unsafe` API, pinned
>> initialization is riddled with `unsafe` resulting in very poor ergonomics.
>> Not only that, but also having to call two functions possibly multiple
>> lines apart makes it very easy to forget it outright or during refactoring.
>>
>> Here is an example of the current way of initializing a struct with two
>> synchronization primitives (see [2] for the full example):
>>
>> struct SharedState {
>> state_changed: CondVar,
>> inner: Mutex<SharedStateInner>,
>> }
>>
>> impl SharedState {
>> fn try_new() -> Result<Arc<Self>> {
>> let mut state = Pin::from(UniqueArc::try_new(Self {
>> // SAFETY: `condvar_init!` is called below.
>> state_changed: unsafe { CondVar::new() },
>> // SAFETY: `mutex_init!` is called below.
>> inner: unsafe {
>> Mutex::new(SharedStateInner { token_count: 0 })
>> },
>> })?);
>>
>> // SAFETY: `state_changed` is pinned when `state` is.
>> let pinned = unsafe {
>> state.as_mut().map_unchecked_mut(|s| &mut s.state_changed)
>> };
>> kernel::condvar_init!(pinned, "SharedState::state_changed");
>>
>> // SAFETY: `inner` is pinned when `state` is.
>> let pinned = unsafe {
>> state.as_mut().map_unchecked_mut(|s| &mut s.inner)
>> };
>> kernel::mutex_init!(pinned, "SharedState::inner");
>>
>> Ok(state.into())
>> }
>> }
>>
>> The pin-init API of this patch solves this issue by providing a
>> comprehensive solution comprised of macros and traits. Here is the example
>> from above using the pin-init API:
>>
>> #[pin_data]
>> struct SharedState {
>> #[pin]
>> state_changed: CondVar,
>> #[pin]
>> inner: Mutex<SharedStateInner>,
>> }
>>
>> impl SharedState {
>> fn new() -> impl PinInit<Self> {
>> pin_init!(Self {
>> state_changed <- new_condvar!("SharedState::state_changed"),
>> inner <- new_mutex!(
>> SharedStateInner { token_count: 0 },
>> "SharedState::inner",
>> ),
>> })
>> }
>> }
>>
>> Notably the way the macro is used here requires no `unsafe` and thus comes
>> with the usual Rust promise of safe code not introducing any memory
>> violations. Additionally it is now up to the caller of `new()` to decide
>> the memory location of the `SharedState`. They can choose at the moment
>> `Arc<T>`, `Box<T>` or the stack.
>>
>> --
>>
>> The API has the following architecture:
>> 1. Initializer traits `PinInit<T, E>` and `Init<T, E>` that act like
>> closures.
>> 2. Macros to create these initializer traits safely.
>> 3. Functions to allow manually writing initializers.
>>
>> The initializers (an `impl PinInit<T, E>`) receive a raw pointer pointing
>> to uninitialized memory and their job is to fully initialize a `T` at that
>> location. If initialization fails, they return an error (`E`) by value.
>>
>> This way of initializing cannot be safely exposed to the user, since it
>> relies upon these properties outside of the control of the trait:
>> - the memory location (slot) needs to be valid memory,
>> - if initialization fails, the slot should not be read from,
>> - the value in the slot should be pinned, so it cannot move and the memory
>> cannot be deallocated until the value is dropped.
>>
>> This is why using an initializer is facilitated by another trait that
>> ensures these requirements.
>>
>> These initializers can be created manually by just supplying a closure that
>> fulfills the same safety requirements as `PinInit<T, E>`. But this is an
>> `unsafe` operation. To allow safe initializer creation, the `pin_init!` is
>> provided along with three other variants: `try_pin_init!`, `try_init!` and
>> `init!`. These take a modified struct initializer as a parameter and
>> generate a closure that initializes the fields in sequence.
>> The macros take great care in upholding the safety requirements:
>> - A shadowed struct type is used as the return type of the closure instead
>> of `()`. This is to prevent early returns, as these would prevent full
>> initialization.
>> - To ensure every field is only initialized once, a normal struct
>> initializer is placed in unreachable code. The type checker will emit
>> errors if a field is missing or specified multiple times.
>> - When initializing a field fails, the whole initializer will fail and
>> automatically drop fields that have been initialized earlier.
>> - Only the correct initializer type is allowed for unpinned fields. You
>> cannot use a `impl PinInit<T, E>` to initialize a structurally not pinned
>> field.
>>
>> To ensure the last point, an additional macro `#[pin_data]` is needed. This
>> macro annotates the struct itself and the user specifies structurally
>> pinned and not pinned fields.
>>
>> Because dropping a pinned struct is also not allowed to break the pinning
>> invariants, another macro attribute `#[pinned_drop]` is needed. This
>> macro is introduced in a following commit.
>>
>> These two macros also have mechanisms to ensure the overall safety of the
>> API. Additionally, they utilize a combined proc-macro, declarative macro
>> design: first a proc-macro enables the outer attribute syntax `#[...]` and
>> does some important pre-parsing. Notably this prepares the generics such
>> that the declarative macro can handle them using token trees. Then the
>> actual parsing of the structure and the emission of code is handled by a
>> declarative macro.
>>
>> For pin-projections the crates `pin-project` [3] and `pin-project-lite` [4]
>> had been considered, but were ultimately rejected:
>> - `pin-project` depends on `syn` [5] which is a very big dependency, around
>> 50k lines of code.
>> - `pin-project-lite` is a more reasonable 5k lines of code, but contains a
>> very complex declarative macro to parse generics. On top of that it
>> would require modification that would need to be maintained
>> independently.
>>
>> Link: https://rust-for-linux.com/the-safe-pinned-initialization-problem [1]
>> Link: https://github.com/Rust-for-Linux/linux/blob/f509ede33fc10a07eba3da14aa00302bd4b5dddd/samples/rust/rust_miscdev.rs [2]
>> Link: https://crates.io/crates/pin-project [3]
>> Link: https://crates.io/crates/pin-project-lite [4]
>> Link: https://crates.io/crates/syn [5]
>> Co-developed-by: Gary Guo <[email protected]>
>> Signed-off-by: Gary Guo <[email protected]>
>> Signed-off-by: Benno Lossin <[email protected]>
>> Cc: Andreas Hindborg <[email protected]>
>> Cc: Alice Ryhl <[email protected]>
>
> Reviewed-by: Wedson Almeida Filho <[email protected]>
>
>> ---
>> rust/kernel/init.rs | 183 +++++++++++++++++++++++++++++++++
>> rust/kernel/init/__internal.rs | 45 ++++++++
>> rust/kernel/lib.rs | 7 ++
>> scripts/Makefile.build | 2 +-
>> 4 files changed, 236 insertions(+), 1 deletion(-)
>> create mode 100644 rust/kernel/init.rs
>> create mode 100644 rust/kernel/init/__internal.rs
>>
>> diff --git a/rust/kernel/init.rs b/rust/kernel/init.rs
>> new file mode 100644
>> index 000000000000..f8d147e47fe7
>> --- /dev/null
>> +++ b/rust/kernel/init.rs
>> @@ -0,0 +1,183 @@
>> +// SPDX-License-Identifier: Apache-2.0 OR MIT
>> +
>> +//! API to safely and fallibly initialize pinned `struct`s using in-place constructors.
>> +//!
>> +//! It also allows in-place initialization of big `struct`s that would otherwise produce a stack
>> +//! overflow.
>> +//!
>> +//! Most `struct`s from the [`sync`] module need to be pinned, because they contain self-referential
>> +//! `struct`s from C. [Pinning][pinning] is Rust's way of ensuring data does not move.
>> +//!
>> +//! # Overview
>> +//!
>> +//! To initialize a `struct` with an in-place constructor you will need two things:
>> +//! - an in-place constructor,
>> +//! - a memory location that can hold your `struct`.
>> +//!
>> +//! To get an in-place constructor there are generally two options:
>> +//! - a custom function/macro returning an in-place constructor provided by someone else,
>> +//! - using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
>> +//!
>> +//! Aside from pinned initialization, this API also supports in-place construction without pinning,
>> +//! the macros/types/functions are generally named like the pinned variants without the `pin`
>> +//! prefix.
>> +//!
>> +//! [`sync`]: kernel::sync
>> +//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
>> +//! [structurally pinned fields]:
>> +//! https://doc.rust-lang.org/std/pin/index.html#pinning-is-structural-for-field
>> +//! [`Arc<T>`]: crate::sync::Arc
>> +//! [`impl PinInit<Foo>`]: PinInit
>> +//! [`impl PinInit<T, E>`]: PinInit
>> +//! [`impl Init<T, E>`]: Init
>> +//! [`Opaque`]: kernel::types::Opaque
>> +//! [`pin_data`]: ::macros::pin_data
>> +//! [`UniqueArc<T>`]: kernel::sync::UniqueArc
>> +//! [`Box<T>`]: alloc::boxed::Box
>> +
>> +use core::{convert::Infallible, marker::PhantomData, mem::MaybeUninit};
>> +
>> +#[doc(hidden)]
>> +pub mod __internal;
>> +
>> +/// A pin-initializer for the type `T`.
>> +///
>> +/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
>> +/// be [`Box<T>`], [`Arc<T>`], [`UniqueArc<T>`].
>> +///
>> +/// Also see the [module description](self).
>> +///
>> +/// # Safety
>> +///
>> +/// When implementing this type you will need to take great care. Also there are probably very few
>> +/// cases where a manual implementation is necessary. Use [`pin_init_from_closure`] where possible.
>> +///
>> +/// The [`PinInit::__pinned_init`] function
>> +/// - returns `Ok(())` if it initialized every field of `slot`,
>> +/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
>> +/// - `slot` can be deallocated without UB occurring,
>> +/// - `slot` does not need to be dropped,
>> +/// - `slot` is not partially initialized.
>> +/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
>> +///
>> +/// [`Arc<T>`]: crate::sync::Arc
>> +/// [`Arc::pin_init`]: crate::sync::Arc::pin_init
>> +/// [`UniqueArc<T>`]: kernel::sync::UniqueArc
>> +/// [`Box<T>`]: alloc::boxed::Box
>> +#[must_use = "An initializer must be used in order to create its value."]
>> +pub unsafe trait PinInit<T: ?Sized, E = Infallible>: Sized {
>> + /// Initializes `slot`.
>> + ///
>> + /// # Safety
>> + ///
>> + /// - `slot` is a valid pointer to uninitialized memory.
>> + /// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
>> + /// deallocate.
>> + /// - `slot` will not move until it is dropped, i.e. it will be pinned.
>> + unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E>;
>> +}
>> +
>> +/// An initializer for `T`.
>> +///
>> +/// To use this initializer, you will need a suitable memory location that can hold a `T`. This can
>> +/// be [`Box<T>`], [`Arc<T>`], [`UniqueArc<T>`]. Because [`PinInit<T, E>`] is a super trait, you can
>> +/// use every function that takes it as well.
>> +///
>> +/// Also see the [module description](self).
>> +///
>> +/// # Safety
>> +///
>> +/// When implementing this type you will need to take great care. Also there are probably very few
>> +/// cases where a manual implementation is necessary. Use [`init_from_closure`] where possible.
>> +///
>> +/// The [`Init::__init`] function
>> +/// - returns `Ok(())` if it initialized every field of `slot`,
>> +/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
>> +/// - `slot` can be deallocated without UB occurring,
>> +/// - `slot` does not need to be dropped,
>> +/// - `slot` is not partially initialized.
>> +/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
>> +///
>> +/// The `__pinned_init` function from the supertrait [`PinInit`] needs to execute the exact same
>> +/// code as `__init`.
>> +///
>> +/// Contrary to its supertype [`PinInit<T, E>`] the caller is allowed to
>> +/// move the pointee after initialization.
>> +///
>> +/// [`Arc<T>`]: crate::sync::Arc
>> +/// [`UniqueArc<T>`]: kernel::sync::UniqueArc
>> +/// [`Box<T>`]: alloc::boxed::Box
>> +#[must_use = "An initializer must be used in order to create its value."]
>> +pub unsafe trait Init<T: ?Sized, E = Infallible>: PinInit<T, E> {
>> + /// Initializes `slot`.
>> + ///
>> + /// # Safety
>> + ///
>> + /// - `slot` is a valid pointer to uninitialized memory.
>> + /// - the caller does not touch `slot` when `Err` is returned, they are only permitted to
>> + /// deallocate.
>> + unsafe fn __init(self, slot: *mut T) -> Result<(), E>;
>> +}
>
> Rather than making `PinInit<T, E>` a supertrait here, wouldn't it make sense to
> just implement `PinInit<T, E>` for every type that implements `Init<T, E>`? This
> way we could relax the unsafe requirement that "`__pinned_init` needs to mimick
> `__init`" by simply making it so. Something like (plus a proper safety comment):
>
> unsafe impl<T, U: ?Sized, E> PinInit<U, E> for T where T: Init<U, E> {
> unsafe fn __pinned_init(self, slot: *mut U) -> Result<(), E> {
> unsafe { T::__init(self, slot) }
> }
> }
This makes sense.
>> +
>> +/// Creates a new [`PinInit<T, E>`] from the given closure.
>> +///
>> +/// # Safety
>> +///
>> +/// The closure:
>> +/// - returns `Ok(())` if it initialized every field of `slot`,
>> +/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
>> +/// - `slot` can be deallocated without UB occurring,
>> +/// - `slot` does not need to be dropped,
>> +/// - `slot` is not partially initialized.
>> +/// - may assume that the `slot` does not move if `T: !Unpin`,
>> +/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
>> +#[inline]
>> +pub const unsafe fn pin_init_from_closure<T: ?Sized, E>(
>> + f: impl FnOnce(*mut T) -> Result<(), E>,
>> +) -> impl PinInit<T, E> {
>> + __internal::InitClosure(f, PhantomData)
>> +}
>> +
>> +/// Creates a new [`Init<T, E>`] from the given closure.
>> +///
>> +/// # Safety
>> +///
>> +/// The closure:
>> +/// - returns `Ok(())` if it initialized every field of `slot`,
>> +/// - returns `Err(err)` if it encountered an error and then cleaned `slot`, this means:
>> +/// - `slot` can be deallocated without UB occurring,
>> +/// - `slot` does not need to be dropped,
>> +/// - `slot` is not partially initialized.
>> +/// - the `slot` may move after initialization.
>> +/// - while constructing the `T` at `slot` it upholds the pinning invariants of `T`.
>> +#[inline]
>> +pub const unsafe fn init_from_closure<T: ?Sized, E>(
>> + f: impl FnOnce(*mut T) -> Result<(), E>,
>> +) -> impl Init<T, E> {
>> + __internal::InitClosure(f, PhantomData)
>> +}
>> +
>> +/// An initializer that leaves the memory uninitialized.
>> +///
>> +/// The initializer is a no-op. The `slot` memory is not changed.
>> +#[inline]
>> +pub fn uninit<T, E>() -> impl Init<MaybeUninit<T>, E> {
>> + // SAFETY: The memory is allowed to be uninitialized.
>> + unsafe { init_from_closure(|_| Ok(())) }
>> +}
>> +
>> +// SAFETY: Every type can be initialized by-value.
>> +unsafe impl<T> PinInit<T> for T {
>> + unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), Infallible> {
>> + unsafe { slot.write(self) };
>> + Ok(())
>> + }
>> +}
>> +
>> +// SAFETY: Every type can be initialized by-value.
>> +unsafe impl<T> Init<T> for T {
>> + unsafe fn __init(self, slot: *mut T) -> Result<(), Infallible> {
>> + unsafe { slot.write(self) };
>> + Ok(())
>> + }
>> +}
>> diff --git a/rust/kernel/init/__internal.rs b/rust/kernel/init/__internal.rs
>> new file mode 100644
>> index 000000000000..ddc03c3f0730
>> --- /dev/null
>> +++ b/rust/kernel/init/__internal.rs
>> @@ -0,0 +1,45 @@
>> +// SPDX-License-Identifier: Apache-2.0 OR MIT
>> +
>> +//! This module contains API-internal items for pin-init.
>> +//!
>> +//! These items must not be used outside of
>> +//! - `kernel/init.rs`
>> +//! - `macros/pin_data.rs`
>> +//! - `macros/pinned_drop.rs`
>> +
>> +use super::*;
>> +
>> +/// See the [nomicon] for what subtyping is. See also [this table].
>> +///
>> +/// [nomicon]: https://doc.rust-lang.org/nomicon/subtyping.html
>> +/// [this table]: https://doc.rust-lang.org/nomicon/phantom-data.html#table-of-phantomdata-patterns
>> +type Invariant<T> = PhantomData<fn(*mut T) -> *mut T>;
>> +
>> +/// This is the module-internal type implementing `PinInit` and `Init`. It is unsafe to create this
>> +/// type, since the closure needs to fulfill the same safety requirement as the
>> +/// `__pinned_init`/`__init` functions.
>> +pub(crate) struct InitClosure<F, T: ?Sized, E>(pub(crate) F, pub(crate) Invariant<(E, T)>);
>> +
>> +// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
>> +// `__pinned_init` invariants.
>> +unsafe impl<T: ?Sized, F, E> PinInit<T, E> for InitClosure<F, T, E>
>> +where
>> + F: FnOnce(*mut T) -> Result<(), E>,
>> +{
>> + #[inline]
>> + unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
>> + (self.0)(slot)
>> + }
>> +}
>> +
>> +// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
>> +// `__init` invariants.
>> +unsafe impl<T: ?Sized, F, E> Init<T, E> for InitClosure<F, T, E>
>> +where
>> + F: FnOnce(*mut T) -> Result<(), E>,
>> +{
>> + #[inline]
>> + unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
>> + (self.0)(slot)
>> + }
>> +}
>
> Since these are only used to implement `init_from_closure` and
> `pin_init_from_closure`, would it make sense to move them into
> `init_from_closure`, and have `pin_init_from_closure` just call
> `init_from_closure` (given that if a type implements Init<T, E>, it also
> implements PinInit<T, E>)?
Calling `init_from_closure` from `pin_init_from_closure` results in a strange
compile error about a cycle when computing the return type of
`pin_init_from_closure`.
>> diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
>> index 4317b6d5f50b..821bd067151c 100644
>> --- a/rust/kernel/lib.rs
>> +++ b/rust/kernel/lib.rs
>> @@ -16,7 +16,9 @@
>> #![feature(coerce_unsized)]
>> #![feature(core_ffi_c)]
>> #![feature(dispatch_from_dyn)]
>> +#![feature(explicit_generic_args_with_impl_trait)]
>> #![feature(generic_associated_types)]
>> +#![feature(new_uninit)]
>> #![feature(pin_macro)]
>> #![feature(receiver_trait)]
>> #![feature(unsize)]
>> @@ -26,11 +28,16 @@
>> #[cfg(not(CONFIG_RUST))]
>> compile_error!("Missing kernel configuration for conditional compilation");
>>
>> +#[allow(unused_extern_crates)]
>> +// Allow proc-macros to refer to `::kernel` inside the `kernel` crate (this crate).
>> +extern crate self as kernel;
>> +
>> #[cfg(not(test))]
>> #[cfg(not(testlib))]
>> mod allocator;
>> mod build_assert;
>> pub mod error;
>> +pub mod init;
>> pub mod prelude;
>> pub mod print;
>> mod static_assert;
>> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
>> index 76323201232a..f9bdc01c8191 100644
>> --- a/scripts/Makefile.build
>> +++ b/scripts/Makefile.build
>> @@ -277,7 +277,7 @@ $(obj)/%.lst: $(src)/%.c FORCE
>> # Compile Rust sources (.rs)
>> # ---------------------------------------------------------------------------
>>
>> -rust_allowed_features := core_ffi_c
>> +rust_allowed_features := core_ffi_c,explicit_generic_args_with_impl_trait
>>
>> rust_common_cmd = \
>> RUST_MODFILE=$(modfile) $(RUSTC_OR_CLIPPY) $(rust_flags) \
>> --
>> 2.39.2
>>
>>
--
Cheers,
Benno
[email protected] writes:
> From: Benno Lossin <[email protected]>
>
> Change the error type of the constructors of `Arc` and `UniqueArc` to be
> `AllocError` instead of `Error`. This makes the API more clear as to
> what can go wrong when calling `try_new` or its variants.
>
> Signed-off-by: Benno Lossin <[email protected]>
> Cc: Gary Guo <[email protected]>
> Cc: Andreas Hindborg <[email protected]>
> Cc: Alice Ryhl <[email protected]>
> ---
Reviewed-by: Andreas Hindborg <[email protected]>
> rust/kernel/sync/arc.rs | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/rust/kernel/sync/arc.rs b/rust/kernel/sync/arc.rs
> index f2f1c83d72ba..aa7135f0f238 100644
> --- a/rust/kernel/sync/arc.rs
> +++ b/rust/kernel/sync/arc.rs
> @@ -17,11 +17,11 @@
>
> use crate::{
> bindings,
> - error::Result,
> types::{ForeignOwnable, Opaque},
> };
> use alloc::boxed::Box;
> use core::{
> + alloc::AllocError,
> marker::{PhantomData, Unsize},
> mem::{ManuallyDrop, MaybeUninit},
> ops::{Deref, DerefMut},
> @@ -149,7 +149,7 @@ unsafe impl<T: ?Sized + Sync + Send> Sync for Arc<T> {}
>
> impl<T> Arc<T> {
> /// Constructs a new reference counted instance of `T`.
> - pub fn try_new(contents: T) -> Result<Self> {
> + pub fn try_new(contents: T) -> Result<Self, AllocError> {
> // INVARIANT: The refcount is initialised to a non-zero value.
> let value = ArcInner {
> // SAFETY: There are no safety requirements for this FFI call.
> @@ -469,7 +469,7 @@ pub struct UniqueArc<T: ?Sized> {
>
> impl<T> UniqueArc<T> {
> /// Tries to allocate a new [`UniqueArc`] instance.
> - pub fn try_new(value: T) -> Result<Self> {
> + pub fn try_new(value: T) -> Result<Self, AllocError> {
> Ok(Self {
> // INVARIANT: The newly-created object has a ref-count of 1.
> inner: Arc::try_new(value)?,
> @@ -477,7 +477,7 @@ impl<T> UniqueArc<T> {
> }
>
> /// Tries to allocate a new [`UniqueArc`] instance whose contents are not initialised yet.
> - pub fn try_new_uninit() -> Result<UniqueArc<MaybeUninit<T>>> {
> + pub fn try_new_uninit() -> Result<UniqueArc<MaybeUninit<T>>, AllocError> {
> Ok(UniqueArc::<MaybeUninit<T>> {
> // INVARIANT: The newly-created object has a ref-count of 1.
> inner: Arc::try_new(MaybeUninit::uninit())?,