Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start of Kobj support: Semaphores and Threads #12

Merged
merged 39 commits into from
Oct 17, 2024
Merged

Start of Kobj support: Semaphores and Threads #12

merged 39 commits into from
Oct 17, 2024

Conversation

d3zd3z
Copy link
Collaborator

@d3zd3z d3zd3z commented Oct 11, 2024

This PR pulls out some of what is implemented in #5, to try to reduce how much code needs to be reviewed.

In addition, this also incorporates @teburd suggestions around the safety of Thread and related.

The thread initialization is now safe in the sense that a given thread can only be initialized once, and a given stack can also only be used once.

The sys::Semaphore is safe to get and use, based on the documentation of the zephyr API.

A disadvantage here is that the initialization is more complicated, and currently stacks cannot be declared in kobj_define! as an array. A possible solution to this is with either a proc-macro, or to find a supporting proc macro package to help with this (fixed).

d3zd3z added 26 commits October 11, 2024 10:30
Move the module wrapper into `lib.rs` so that we can add documentation
to the module itself.

Signed-off-by: David Brown <[email protected]>
Document a handful of functions so that we can enforce documentation.

Signed-off-by: David Brown <[email protected]>
Add a directive to consider missing documentation on public symbols to
be a compliation error.

Signed-off-by: David Brown <[email protected]>
Create a simple error type to handle Zephyr errors.  This can be
expanded in the future to better be able to distinguish the errors, but
this allows us to properly return values from wrapped functions.

Signed-off-by: David Brown <[email protected]>
Provide the atomic types  from portable-atomic in
`zephyr::sync::atomic`, and, if allocation is enabled, provide
`zephyr::sync::Arc`.  The `alloc::arc` will only be available on Zephyr
targets that have atomic intrinsics, and the portable one can be
supported on any Zephyr target.  There are some limitations described in
the documentation.

Signed-off-by: David Brown <[email protected]>
Create wrappers for static kernel objects.  This is done through a few
wrapper types.  Generally, a StaticKernelObject which contains a 'value'
which is an unsafe cell containing the actual zephyr-typed kernel
object, and then some traits to use these, specifically `KobjGet` to
fetch the underlying pointer, and `KobjInit` which supports
initialization and getting a higher-level wrapper around the underlying
object pointer.

Signed-off-by: David Brown <[email protected]>
The atomic operations in Rust's core crate only work on platforms that
have atomic intrinsics.  Zephyr supports atomics on other platforms as
well, generally implementing them with spinlocks.

Use the portable-atomics library, and configure it so use the spinlock
provided by the critical section crate.
The `sys::Semaphore` is a wrapper for Zephyr's `k_sem` type.  This adds
rules to `kobj_define!` to statically declare these, which will have a
`.take()` method, giving an instance that can be used to coordinate
between threads.

Signed-off-by: David Brown <[email protected]>
Add a dependent type to the Wrapped trait to indicate the arguments that
'take' and hence init needs.  These will typically be either `()`, or a
tuple with the Zephyr object's parameters.

Signed-off-by: David Brown <[email protected]>
Implement a general critical section handler, using spinlocks, for Rust
users to be able to use the `critical-section` crate.  This crate is
used by other crates as well.

Signed-off-by: David Brown <[email protected]>
The Rust attribute that indicates structure alignment only supports
alignment values that are numeric constants.  This makes it difficult to
set alignment based on a value coming from the build environment, which
is commonly done.

Implement a bit of a workaround with an `AlignAs` type that is
parameterized with the alignment.  Parameters can be compile-time
constants not just numeric constants.  This works by having instances
for each of the desired alignments that each have the numeric constant
as the alignment.  The instances can be expanded if necessary.

Signed-off-by: David Brown <[email protected]>
Create a config `CONFIG_RUST_ALLOC` that will hook Rust's allocation system
into the `malloc`/`free` allocator provided on Zephyr.  This will allow the
`alloc` crate in rust to be used.

Signed-off-by: David Brown <[email protected]>
Bindgen only exports defined values that are clearly numeric constants,
which is thrown off when the defined value has a cast.  Work around this
by defining wrapper constants that bindgen is able to expose.

We use the `ZR_` prefix (Zephyr Rust) to avoid conflicts with any other
symbols.

Signed-off-by: David Brown <[email protected]>
These are wrappers that alloc Zephyr threads to be statically declared
as well as started, from safe code.

Signed-off-by: David Brown <[email protected]>
The dining philosophers example, in Rust.  This is generalized, and
provides multiple implementations, exercising various different
synchronization primivites in Zephyr

Signed-off-by: David Brown <[email protected]>
Upgrade the versions of all crates in this module to match the current
Zephyr version of 3.7.0.  It isn't completely clear if this is the right
thing to do, as Zephyr doesn't maintain semantic versioning.  But having
the numbers match will help make it clear which version of Zephyr goes
with which version of this crate.

Signed-off-by: David Brown <[email protected]>
This function is safe, so add a simple wrapper around the unsafe
version.

Signed-off-by: David Brown <[email protected]>
It just prints the type, but allows other structs containing Semaphores
to be printed.

Signed-off-by: David Brown <[email protected]>
This is required to allow it to be a static.

Signed-off-by: David Brown <[email protected]>
The semaphore implementation is safe to use in multiple places, so mut
is not necessary and prevents their use in most cases.

Signed-off-by: David Brown <[email protected]>
This is not intended to be visible, but needed for macros.

Signed-off-by: David Brown <[email protected]>
These need to be initialized by macros, so expose them publicly.  It
would probably be better to export a constructor.

Signed-off-by: David Brown <[email protected]>
This is needed to initialize this value into a static.

Signed-off-by: David Brown <[email protected]>
Depend upon the 3.7.0 version of Zephyr.

Signed-off-by: David Brown <[email protected]>
Use of Box depends on Alloc being enabled, so also conditionalize the
extern and use declarations.

Signed-off-by: David Brown <[email protected]>
Update the yaml to remove not-yet-implemented variants, and make sure
the regexp matches things the test actually prints.

Signed-off-by: David Brown <[email protected]>
@d3zd3z d3zd3z requested review from teburd, cfriedt and ithinuel October 11, 2024 22:44
Fix the version dependency to match the Zephyr release version.

Signed-off-by: David Brown <[email protected]>
Comment on lines 90 to 91
/// Construct an empty of these objects, with the zephyr data zero-filled. This is safe in the
/// sense that Zephyr we track the initialization, and they start in the uninitialized state.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate the attention to good docs.

Not sure if these suggestions are the same as your original intent.

Suggested change
/// Construct an empty of these objects, with the zephyr data zero-filled. This is safe in the
/// sense that Zephyr we track the initialization, and they start in the uninitialized state.
/// Construct an empty instance of these objects, with the zephyr data zero-filled. This is safe in the
/// sense that in Zephyr we track the initialization, and they start in the uninitialized state.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this should be addressed, namely by rewriting the whole doc section.

/// permitted count.
pub fn give(&self) {
unsafe {
k_sem_give(self.item)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I needed to remind myself that omitting the semicolon with the last expression of a function returns that value 🦀

Does the documentation engine publish return type information for these functions?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would have to be declared in the code, it is never inferred. In this case, give returns nothing, and the zephyr function is void.

Copy link
Member

@cfriedt cfriedt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me - just a few comments.

@d3zd3z
Copy link
Collaborator Author

d3zd3z commented Oct 12, 2024

Looks reasonable to me - just a few comments.

A couple of things I’d like to do are to add some more comments justifying the safety of code that uses unsafe, see if I can make a few pub fields not pub with constructors, and try to figure out how to allow declaring an array of stacks.

d3zd3z added 11 commits October 12, 2024 07:30
The bindgen code copies the doc strings directly from the C headers.
These have a different syntax than the Rust doc headers.  Until we can
find or create a tool to convert these, disable these warnings.

Signed-off-by: David Brown <[email protected]>
Add comments to the object module describing the safety goals, and how
we achieve them.

Signed-off-by: David Brown <[email protected]>
The docs reference `alloc::format`.  Include the extern for this crate
so that this will work.

Signed-off-by: David Brown <[email protected]>
Fix a few broken doc links, by filling in full references.

Signed-off-by: David Brown <[email protected]>
This had an `extern crate alloc` for a link in the docs, but there is no
actual dependency on this.  Replace this with a resolved link to the
standard documentation for formatting.

Signed-off-by: David Brown <[email protected]>
Although the user is not intended to actually create these static thread
stack objects, make an alias for the type in `_export` so that the macro
is a little easier to read.  This will become more important as this
type picks up some constructors.

Signed-off-by: David Brown <[email protected]>
Instead of fully expanding the thread stack initializer in the
kobj_define macro, create a hidden constructor as a const fn.

Signed-off-by: David Brown <[email protected]>
Create a const constructor function for building the stack arrays.  This
requires unsafe, and can't use MaybeUninit, because the array version of
this is still unsable.  Just use zero initialization, and make sure the
loop initializes everything.

Use this function in the kobj_define macro in order to allow apps to
define arrays of thread stacks, in addition to arrays of threads.

Signed-off-by: David Brown <[email protected]>
Now that kboj_define supports arrays of thread stacks, change the
expanded definitions into array definitions.  The demo now properly
supports changing `NUM_PHIL` to use a different number of philosophers.

Signed-off-by: David Brown <[email protected]>
Add a small docgen crate.  To help facilitate checking the docs, this
project can be built, and then `cargo doc` run there.  This includes the
symlink for the config file, because that is needed in order for this to
be able to build.

Signed-off-by: David Brown <[email protected]>
Ensure that the links actually resolve correctly, eliminating all of the
warnings from a doc build.

Signed-off-by: David Brown <[email protected]>
@d3zd3z
Copy link
Collaborator Author

d3zd3z commented Oct 15, 2024

I have implemented kobj_define! support for arrays of thread stacks, reflecting this in the philosopher docs.

I've also added a docgen, which can be built, and then allows cargo doc to generate the documentation. This will reveal any warnings about broken links, and also have a local rendered copy. I've fixed the few warnings that crept in because the docs weren't being generated.

@d3zd3z d3zd3z requested a review from cfriedt October 15, 2024 16:08
Don't just run hello world, but all samples that are under samples.

Signed-off-by: David Brown <[email protected]>
Duration::millis_at_least(((delay + 1) * period) as Tick)
}

kobj_define! {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good - certainly an improvement

Copy link
Collaborator

@Ablu Ablu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool stuff! 🎉

I did not manage to get through all commits today. But here are some comments already :).
There is a decent chance that some of my confusion is due to my limited knowledge about Zephyr internals!

It is mostly the different types for "StaticFoo" and "Foo" that confuse me a bit... My naive feeling would be that there should be ways to do a 1:1 mapping of C type to Rust type without needing the two variants? But again, chances are I miss things! 😇

Comment on lines -57 to -76
writeln!(&mut f, "pub mod kconfig {{").unwrap();

let file = File::open(&dotconfig).expect("Unable to open dotconfig");
for line in BufReader::new(file).lines() {
let line = line.expect("reading line from dotconfig");
if let Some(caps) = config_hex.captures(&line) {
writeln!(&mut f, " #[allow(dead_code)]").unwrap();
writeln!(&mut f, " pub const {}: usize = {};",
writeln!(&mut f, "#[allow(dead_code)]").unwrap();
writeln!(&mut f, "pub const {}: usize = {};",
&caps[1], &caps[2]).unwrap();
} else if let Some(caps) = config_int.captures(&line) {
writeln!(&mut f, " #[allow(dead_code)]").unwrap();
writeln!(&mut f, " pub const {}: isize = {};",
writeln!(&mut f, "#[allow(dead_code)]").unwrap();
writeln!(&mut f, "pub const {}: isize = {};",
&caps[1], &caps[2]).unwrap();
} else if let Some(caps) = config_str.captures(&line) {
writeln!(&mut f, " #[allow(dead_code)]").unwrap();
writeln!(&mut f, " pub const {}: &'static str = {};",
writeln!(&mut f, "#[allow(dead_code)]").unwrap();
writeln!(&mut f, "pub const {}: &'static str = {};",
&caps[1], &caps[2]).unwrap();
}
}
writeln!(&mut f, "}}").unwrap();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads like it snuck into an unrelated commit? It does not seem to have anything to do with documentation?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't sneak, it is very intentional. I needed to move the pub mod kconfig { up into lib.rs so that I could add both doc comments and directives to not fail on missing docs on the entries. To do that, I had to remove that from the generated code, and I unindented it as well.

Comment on lines +42 to +56
/// Map a return result from Zephyr into an Result.
///
/// Negative return results being considered errors.
pub fn to_result(code: c_int) -> Result<c_int> {
if code < 0 {
Err(Error(-code as u32))
} else {
Ok(code)
}
}

/// Map a return result, with a void result.
pub fn to_result_void(code: c_int) -> Result<()> {
to_result(code).map(|_| ())
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder: Do these have to be pub? They seem to be mostly internal helpers? I find it to be slightly awkward API to be exposed to the outside world. So should this maybe be pub(crate)?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely a consideration. I still think they are useful for an app user that wants to use a binding that we haven't wrapped yet.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reducing the API may make it easier for people to use. This extra API seems to be a convenience API, but, at least for me, it just adds extra API for people who will use the crate to learn.

One possibility here would be to add a ZephyrCode type that could implement those methods. I think using the type Error is misleading because we are using it for error and also for normal returning code.

/// Map a return result from Zephyr into an Result.
///
/// Negative return results being considered errors.
pub fn to_result(code: c_int) -> Result<c_int> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the c_int in the return needed for compatibility here? As this function already checks for negative values, I would find a uint more intuitive from an API usability perspective.

I would have also expected a convert to non FFI types, given that we are using the Rust Result, too?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is to just keep the type as much like the underlying type, and any conversions can happen by the wrapper that is using this.

Comment on lines 57 to 78
/// Each can be wrapped appropriately. The wrapped type is the instance that holds the raw pointer.
pub trait Wrapped {
/// The wrapped type. This is what `take()` on the StaticKernelObject will return after
/// initialization.
type T;

/// Initialize this kernel object, and return the wrapped pointer.
fn get_wrapped(&self) -> Self::T;
}

/// A state indicating an uninitialized kernel object.
///
/// This must be zero, as kernel objects will
/// be represetned as zero-initialized memory.
pub const KOBJ_UNINITIALIZED: usize = 0;

/// A state indicating a kernel object that is being initialized.
pub const KOBJ_INITING: usize = 1;

/// A state indicating a kernel object that has completed initialization. This also means that the
/// take has been called. And shouldn't be allowed additional times.
pub const KOBJ_INITIALIZED: usize = 2;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I typically see people keeping the struct definition + impl's together, so I would consider moving this block before the struct definition?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll look through it, it gets messy when there are multiple structs and traits that interact with each other.

/// sense that Zephyr we track the initialization, and they start in the uninitialized state.
pub const fn new() -> StaticKernelObject<T> {
StaticKernelObject {
value: unsafe { mem::zeroed() },
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may lack context here, but I like how most projects put // SAFETY comments before unsafe blocks. That makes reviewing a lot easier IMHO. Here one could probably add a comment about how later code ensures that this is fully initialized at some point.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, I think this whole method should probably just be unsafe.

Ordering::Acquire)
{
return None;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you already have a Result<> type, you can also do: .ok()? on it [1]

[1] https://doc.rust-lang.org/std/result/enum.Result.html#method.ok

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll look at it.

type T;

/// Initialize this kernel object, and return the wrapped pointer.
fn get_wrapped(&self) -> Self::T;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit confused about the "pointer" documentation here while the trait really takes any type.

Overall, I am a bit confused about this being abstracted with pointers. Couldn't this also be abstracted with references on the Rust side?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They don't follow the semantics of Rust references, they are never dereferenced from Rust, and are really just part of the C api. Also, this has been remove anyway (but it is still a wrapped pointer).

pub struct StaticKernelObject<T> {
#[allow(dead_code)]
/// The underlying zephyr kernel object.
value: UnsafeCell<T>,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may not have grasped the full context yet, but to me this mostly seems to be used as a MaybeUninit? 🤔 Could that be used instead here? It reads like it will do the same job, but provide some more explicit meaning.

Also: Nothing prevents somebody from moving such an object, right? Are those C counterparts fine with being moved? I would assume synchronization primitives keep all kind of self referential pointers internally. So I am a bit surprised that this can safely be wrapped without some sort of Pinning / modeling things are references to ensure things cannot be moved around.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The UnsafeCell is absolutely critical to this working. It is treated as magical by the Rust compiler and basically tells it that even though the container isn't mut, this T inside of it might change. Since the C code is likely to change these, we need to declare this like this.

}
let result = self.get_wrapped();
self.init.store(KOBJ_INITIALIZED, Ordering::Release);
Some(result)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an example that uses this (maybe my grep skills fail me :P)?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look in samples. It is used by any user code using statically declared kernel objects.

The reason to have these two types is that most kernel objects in Zephyr need to be declared statically. But static globals aren't very useful in Rust (without unsafe), so we have this init_once method that is called on the global static to return a single instance that wraps the pointer to the static, and can be used more freely in Rust.

Comment on lines +48 to +53
/// A zephyr `k_sem` usable from safe Rust code.
#[derive(Clone)]
pub struct Semaphore {
/// The raw Zephyr `k_sem`.
item: *mut k_sem,
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to work fine as other code carefully avoids constructing this for non-static semahpores. Does zephyr also have dynamic ones? I found these internal pointers made it harder to reason about safety eventually.

I wonder: Have you considered something like:

[repr(transparent)]
pub struct Semaphore { 
    value: UnsafeCell<MaybeUninit<k_sem>>,
    _pin: PhantomPinned,
}

?
(see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/rust/kernel/types.rs#n261 / rust-lang/rust#43467 for context)

Then of course most code would work with a &Semaphore, but that seems like a more direct mapping of C -> Rust to me?

If all you ever deal with are static objects, then I guess this is fine too. But I think the direct wrapping is going to be a pattern elsewhere eventually anyway? It seems to me like that may also allow to unify this together with the "static object" that currently needs to be defined separately?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So Semaphore is not a static semaphore, it is a singleton reference to some kind of semaphore. The StaticSemaphore is one way of declaring the semaphores (and currently the only way).

I intend to add dynamic variants of many of these, but as the Zephyr dynamic object support is rather poor at the moment, these will likely just have a pool of the objects that they take from. The Semaphore type will be the same, but there will lookly be a Semaphore::new that returns one from the pool. When it is dropped, it will be returned to the pool.

Copy link
Collaborator

@teburd teburd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems ok to me, gotta start somewhere with this stuff and I think my major concerns are mostly covered. Time will tell what ends up working/not working more definitively.

@d3zd3z
Copy link
Collaborator Author

d3zd3z commented Oct 17, 2024

@Ablu I think I'm going to move forward with merging this, and address some of your concerns in the next PR. The ones about code organization and safety comments don't really affect the code itself. Hopefully, the concerns about the static stuff will make more sense in the next PR as I add sys::Mutex and the more Rust-friendly sync::Mutex that makes use of it. I chose Semaphore for this PR because k_sem is about the only kernel object in Zephyr that is perfectly usable directly from safe Rust code. The rest will have sys wrappers, but will have other mechanisms to make them more usable from safe code.

@d3zd3z d3zd3z merged commit 5eb7283 into main Oct 17, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

5 participants