Hello folks,
I'm working on putting together an LLVM "Micro Conference" for the
upcoming Linux Plumbers Conf
(https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
solidified yet, but I would really like to run a session on support
for Rust "in tree." I suspect we could cover technical aspects of
what that might look like (I have a prototype of that, was trivial to
wire up KBuild support), but also a larger question of "should we do
this?" or "how might we place limits on where this can be used?"
Question to folks explicitly in To:, are you planning on attending plumbers?
If so, would this be an interesting topic that you'd participate in?
--
Thanks,
~Nick Desaulniers
Hi Nick,
On Thu, Jul 9, 2020 at 8:42 PM Nick Desaulniers <[email protected]> wrote:
>
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree." I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
Agreed, I wrote one ~half a year ago that compiled TUs through rustc
mimicking what we do with cc to some degree (i.e. no cargo etc.), and
it was quite pleasant.
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
Indeed, I would say that is going to be the major topic, not so much
the technical bits.
> Question to folks explicitly in To:, are you planning on attending plumbers?
I wasn't, but it definitely is an interesting topic that I wouldn't
want to miss!
Cheers,
Miguel
On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree." I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?
I hadn't planned to attend the virtual event, but this sounds like a
topic I absolutely have to attend. Please follow up if this proposal
gets accepted.
I'd love to see a path to incorporating Rust into the kernel, as long as
we can ensure that:
- There are appropriate Rustic interfaces that are natural and safe to
use (not just C FFI, and not *just* trivial transformations like
slices instead of buffer+len pairs).
- Those Rustic interfaces are easy to maintain and evolve with the kernel.
- We provide compelling use cases that go beyond just basic safety, such
as concurrency checking, or lifetimes for object ownership.
- We make Rust fit naturally into the kernel's norms and standards,
while also introducing some of Rust's norms and standards where they
make sense. (We want to fit into the kernel, and at the same time, we
don't want to hastily saw off all the corners that don't immediately
fit, because some of those corners provide value. Let's take our
time.)
- We move slowly and carefully, making sure it's a gradual introduction,
and give people time to incorporate the Rust toolchain into their
kernel workflows.
Also, with my "Rust language team lead" hat on, I'd be happy to have the
Linux kernel feeding into Rust language development priorities. If
building Rustic interfaces within the kernel requires some additional
language features, we should see what enhancements to the language would
best serve those requirements. I've often seen the sentiment that
co-evolving Linux and a C compiler would be beneficial for both; I think
the same would be true of Linux and the Rust compiler.
On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree." I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?
Yes, I'll be there.
thanks,
greg k-h
On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote:
> On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > Hello folks,
> > I'm working on putting together an LLVM "Micro Conference" for the
> > upcoming Linux Plumbers Conf
> > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> > solidified yet, but I would really like to run a session on support
> > for Rust "in tree." I suspect we could cover technical aspects of
> > what that might look like (I have a prototype of that, was trivial to
> > wire up KBuild support), but also a larger question of "should we do
> > this?" or "how might we place limits on where this can be used?"
> >
> > Question to folks explicitly in To:, are you planning on attending plumbers?
> >
> > If so, would this be an interesting topic that you'd participate in?
>
> Yes, I'll be there.
We actually had this dicussion a while back and there were some more
people interested in this. I'd be interested to attend this and I've
spoken with Kees and a few others about this topic at last Plumbers (I
think Greg might have been around for this informal discussion as well.
But I might be imagining things.).
Christian
On Fri, Jul 10, 2020 at 02:50:22PM +0200, Christian Brauner wrote:
> On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote:
> > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > > Hello folks,
> > > I'm working on putting together an LLVM "Micro Conference" for the
> > > upcoming Linux Plumbers Conf
> > > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> > > solidified yet, but I would really like to run a session on support
> > > for Rust "in tree." I suspect we could cover technical aspects of
> > > what that might look like (I have a prototype of that, was trivial to
> > > wire up KBuild support), but also a larger question of "should we do
> > > this?" or "how might we place limits on where this can be used?"
> > >
> > > Question to folks explicitly in To:, are you planning on attending plumbers?
> > >
> > > If so, would this be an interesting topic that you'd participate in?
> >
> > Yes, I'll be there.
>
> We actually had this dicussion a while back and there were some more
> people interested in this. I'd be interested to attend this and I've
> spoken with Kees and a few others about this topic at last Plumbers (I
> think Greg might have been around for this informal discussion as well.
> But I might be imagining things.).
I'm quite interested in this topic still, yes. :) (And will be attending
Plumbers.)
We had interesting discussions with the LSS 2019 "Rust in the kernel"[1]
speakers, and I know Nick has reviewed their work[2] too. Perhaps we should
reach out to them as well?
-Kees
[1] https://ldpreload.com/p/kernel-modules-in-rust-lssna2019.pdf
[2] https://github.com/fishinabarrel/linux-kernel-module-rust
--
Kees Cook
On Fri, Jul 10, 2020 at 02:50:22PM +0200, Christian Brauner wrote:
> On Fri, Jul 10, 2020 at 08:28:03AM +0200, Greg KH wrote:
> > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > > Hello folks,
> > > I'm working on putting together an LLVM "Micro Conference" for the
> > > upcoming Linux Plumbers Conf
> > > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> > > solidified yet, but I would really like to run a session on support
> > > for Rust "in tree." I suspect we could cover technical aspects of
> > > what that might look like (I have a prototype of that, was trivial to
> > > wire up KBuild support), but also a larger question of "should we do
> > > this?" or "how might we place limits on where this can be used?"
> > >
> > > Question to folks explicitly in To:, are you planning on attending plumbers?
> > >
> > > If so, would this be an interesting topic that you'd participate in?
> >
> > Yes, I'll be there.
>
> We actually had this dicussion a while back and there were some more
> people interested in this. I'd be interested to attend this and I've
> spoken with Kees and a few others about this topic at last Plumbers (I
> think Greg might have been around for this informal discussion as well.
> But I might be imagining things.).
I was around for one of the informal conversations with Greg and Kees
and others.
As I recall, Greg's biggest condition for initial introduction of this
was to do the same kind of "turn this Kconfig option on and turn an
option under it off" trick that LTO uses, so that neither "make
allnoconfig" nor "make allyesconfig" would require Rust until we've had
plenty of time to experiment with it. And that seems entirely
reasonable to me too.
- Josh
On Fri, Jul 10, 2020 at 12:28:30PM -0400, Alex Gaynor wrote:
> :wave:,
>
> Hey Kees, that's Geoffrey Thomas and I, we're both on this thread :-)
*face palm* Hello! I swear I can read. Though perhaps not well enough
before lunch. :)
--
Kees Cook
On Fri, Jul 10, 2020 at 3:59 PM Josh Triplett <[email protected]> wrote:
>
> As I recall, Greg's biggest condition for initial introduction of this
> was to do the same kind of "turn this Kconfig option on and turn an
> option under it off" trick that LTO uses, so that neither "make
> allnoconfig" nor "make allyesconfig" would require Rust until we've had
> plenty of time to experiment with it.
No, please make it a "is rust available" automatic config option. The
exact same way we already do the compiler versions and check for
various availability of compiler flags at config time.
See init/Kconfig for things like
config LD_IS_LLD
def_bool $(success,$(LD) -v | head -n 1 | grep -q LLD)
and the rust support should be similar. Something like
config RUST_IS_AVAILABLE
def_bool $(success,$(RUST) ..sometest..)
because I _don't_ want us to be in the situation where any new rust
support isn't even build-tested by default.
Quite the reverse. I'd want the first rust driver (or whatever) to be
introduced in such a simple format that failures will be obvious and
simple.
The _worst_ situation to be in is that s (small) group of people start
testing their very special situation, and do bad and crazy things
because "nobody else cares, it's hidden".
No, thank you.
Linus
On Thu, 9 Jul 2020, Nick Desaulniers wrote:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree." I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?
Like Alex - I hadn't decided yet but I'll definitely attend for this!
--
Geoffrey Thomas
https://ldpreload.com
[email protected]
On Fri, Jul 10, 2020 at 04:54:11PM -0700, Linus Torvalds wrote:
> On Fri, Jul 10, 2020 at 3:59 PM Josh Triplett <[email protected]> wrote:
> > As I recall, Greg's biggest condition for initial introduction of this
> > was to do the same kind of "turn this Kconfig option on and turn an
> > option under it off" trick that LTO uses, so that neither "make
> > allnoconfig" nor "make allyesconfig" would require Rust until we've had
> > plenty of time to experiment with it.
>
> No, please make it a "is rust available" automatic config option. The
> exact same way we already do the compiler versions and check for
> various availability of compiler flags at config time.
That sounds even better, and will definitely allow for more testing.
We just need to make sure that any kernel CI infrastructure tests that
right away, then, so that failures don't get introduced by a patch from
someone without a Rust toolchain and not noticed until someone with a
Rust toolchain tests it.
- Josh
On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
>...
> but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>...
I won't attend, but I do have a topic that should be covered:
Firefox always depends on recent Rust, which forces distributions to
update Rust in stable releases.
As an example:
Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
It would not sound good to me if security updates of distribution
kernels might additionally end up using a different version of the
Rust compiler - the toolchain for the kernel should be stable.
Would Rust usage in the kernel require distributions to ship
a "Rust for Firefox" and a "Rust for the kernel"?
> Thanks,
> ~Nick Desaulniers
cu
Adrian
On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> >...
> > but also a larger question of "should we do
> > this?" or "how might we place limits on where this can be used?"
> >...
>
> I won't attend, but I do have a topic that should be covered:
>
> Firefox always depends on recent Rust, which forces distributions to
> update Rust in stable releases.
>
> As an example:
> Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
>
> It would not sound good to me if security updates of distribution
> kernels might additionally end up using a different version of the
> Rust compiler - the toolchain for the kernel should be stable.
>
> Would Rust usage in the kernel require distributions to ship
> a "Rust for Firefox" and a "Rust for the kernel"?
Rust has hard stability guarantees when upgrading from one stable
version to the next. If code compiles with a given stable version of
Rust, it'll compile with a newer stable version of Rust. Given that, a
stable distribution will just need a single sufficiently up-to-date Rust
that meets the minimum version requirements of both Firefox and Linux.
(That would not apply if the kernel used nightly Rust, since
nightly-only features are allowed to change before becoming stable;
that's one reason why we should use stable Rust, and try to get Firefox
to stick to stable Rust.)
On Sun, Jul 12, 2020 at 12:39 PM Josh Triplett <[email protected]> wrote:
>
> Rust has hard stability guarantees when upgrading from one stable
> version to the next.
I think the worry is more about actual compiler bugs, not the set of
exposed features.
That's always been the biggest pain point. Compiler bugs are very
rare, but they are so incredibly hard to debug when they happen that
they end up being extra special.
Random "we need this compiler for this feature" is actually fairly
rare. Yes, the most recent case of me just saying "let's use 4.9
rather than 4.8" was due to that, but honestly, that's the exception
rather than the rule, and is to occasionally simplify the code (and
the test coverage).
The most common case of compiler version checks are due to "compiler
XYZ is known to mis-compile ABC on target IDK".
Linus
On Sun, Jul 12, 2020 at 12:39:44PM -0700, Josh Triplett wrote:
> On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> > On Thu, Jul 09, 2020 at 11:41:47AM -0700, Nick Desaulniers wrote:
> > >...
> > > but also a larger question of "should we do
> > > this?" or "how might we place limits on where this can be used?"
> > >...
> >
> > I won't attend, but I do have a topic that should be covered:
> >
> > Firefox always depends on recent Rust, which forces distributions to
> > update Rust in stable releases.
> >
> > As an example:
> > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> >
> > It would not sound good to me if security updates of distribution
> > kernels might additionally end up using a different version of the
> > Rust compiler - the toolchain for the kernel should be stable.
> >
> > Would Rust usage in the kernel require distributions to ship
> > a "Rust for Firefox" and a "Rust for the kernel"?
>
> Rust has hard stability guarantees when upgrading from one stable
> version to the next. If code compiles with a given stable version of
> Rust, it'll compile with a newer stable version of Rust. Given that, a
> stable distribution will just need a single sufficiently up-to-date Rust
> that meets the minimum version requirements of both Firefox and Linux.
>...
API stability avoids problems that are visible early as build errors.
Rust cannot offer a hard stability guarantee that there will never be
a code generation regression on any platform.
Rust gets updated frequently.
Sometimes this also changes the LLVM version used by Rust.
Debian stable supports targets like ARMv5 and 32bit MIPS.
Distribution kernel updates are often automatically installed
on user hardware.
cu
Adrian
Nick Desaulniers <[email protected]> writes:
> Hello folks,
> I'm working on putting together an LLVM "Micro Conference" for the
> upcoming Linux Plumbers Conf
> (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> solidified yet, but I would really like to run a session on support
> for Rust "in tree." I suspect we could cover technical aspects of
> what that might look like (I have a prototype of that, was trivial to
> wire up KBuild support), but also a larger question of "should we do
> this?" or "how might we place limits on where this can be used?"
>
> Question to folks explicitly in To:, are you planning on attending plumbers?
>
> If so, would this be an interesting topic that you'd participate in?
I have two big concerns about actually using rust.
1) How large is the rust language support, and will each rust module
need to duplicate it. I seem to remember someone mentioning it is
noticable in size.
2) What is rust usable for? The rust type system will not admit
doubly linked lists (or anything where two pointers point at the
same memory) unless you are using an unsafe block.
Now maybe all of that can be wrapped up in libraries written in
C that Rust can just call, so rust might be useful for building
drivers.
What I am certain of is that in the core kernel where I tend to spend
my time not being able to use doubly linked lists looks like a
non-starter.
Eric
On Mon, Jul 13, 2020 at 01:11:13PM -0500, Eric W. Biederman wrote:
> Nick Desaulniers <[email protected]> writes:
>
> > Hello folks,
> > I'm working on putting together an LLVM "Micro Conference" for the
> > upcoming Linux Plumbers Conf
> > (https://www.linuxplumbersconf.org/event/7/page/47-attend). It's not
> > solidified yet, but I would really like to run a session on support
> > for Rust "in tree." I suspect we could cover technical aspects of
> > what that might look like (I have a prototype of that, was trivial to
> > wire up KBuild support), but also a larger question of "should we do
> > this?" or "how might we place limits on where this can be used?"
> >
> > Question to folks explicitly in To:, are you planning on attending plumbers?
> >
> > If so, would this be an interesting topic that you'd participate in?
>
> I have two big concerns about actually using rust.
>
> 1) How large is the rust language support, and will each rust module
> need to duplicate it. I seem to remember someone mentioning it is
> noticable in size.
There should only be a single copy, which could either be in the kernel
(if there's Rust code compiled into the kernel) or be in a "rust.ko"
support module. As long as you use the same version of Rust for the
kernel and all modules, you can supply symbols dynamically.
There are a few other steps we can take to control code size, as well.
In particular, there are tradeoffs between performance and size, such as
the amount of code in generics vs non-generics. (We also need to get
some further optimizations into Rust on that front, such as tail
merging.)
As for the size overall, given that we'll just be providing the portions
of "core" and "alloc" that the built code actually uses, and likely not
providing "std" at all, I would expect the size to remain relatively
small.
I very much care about overall kernel size, and I'm happy to help make
sure it remains reasonable.
> 2) What is rust usable for? The rust type system will not admit
> doubly linked lists (or anything where two pointers point at the
> same memory) unless you are using an unsafe block.
There are libraries like intrusive-collections which implement
kernel-style structures with potentially multiple nodes in the structure
that put them into multiple lists at once. I would expect us to use
those (or in some cases implement our own). They don't need to be
written in C, just unsafe Rust that's wrapped up in a safe interface.
Just as the kernel has a variety of higher-level interfaces and data
structures that make working in the kernel sometimes *easier* than the
average C program, I'd expect that we'll end up with common Rust
structures that do what people need, rather than people reimplementing
their own.
- Josh Triplett
From: Adrian Bunk
> Sent: 12 July 2020 21:45
....
> Rust gets updated frequently.
> Sometimes this also changes the LLVM version used by Rust.
> Debian stable supports targets like ARMv5 and 32bit MIPS.
> Distribution kernel updates are often automatically installed
> on user hardware.
This reminds me of why I never want to use an online compiler
service - never mind how hard companies push them.
If I need to do a bug-fix build of something that was released
2 (or more) years ago I want to use exactly the same toolchain
(warts and all) that was used for the original build.
If the compiler has changed I need to do a full test - just
in case it compiles some 'dodgy' code differently.
With the same compiler I only need to test the fix.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
On Sun, Jul 12, 2020 at 9:39 PM Josh Triplett <[email protected]> wrote:
> On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> >
> > As an example:
> > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> >
> > It would not sound good to me if security updates of distribution
> > kernels might additionally end up using a different version of the
> > Rust compiler - the toolchain for the kernel should be stable.
> >
> > Would Rust usage in the kernel require distributions to ship
> > a "Rust for Firefox" and a "Rust for the kernel"?
>
> Rust has hard stability guarantees when upgrading from one stable
> version to the next. If code compiles with a given stable version of
> Rust, it'll compile with a newer stable version of Rust. Given that, a
> stable distribution will just need a single sufficiently up-to-date Rust
> that meets the minimum version requirements of both Firefox and Linux.
>
> (That would not apply if the kernel used nightly Rust, since
> nightly-only features are allowed to change before becoming stable;
> that's one reason why we should use stable Rust, and try to get Firefox
> to stick to stable Rust.)
I would expect we'd want a fairly tight coupling between kernel
releases and minimum rust releases at first. Whatever is the latest
stable rust version during the kernel's merge window might be
assumed to be the minimum version for the life of that kernel, but
an LTS release would not suddenly start relying on features
from a newer compiler (thought it might warn about known bugs).
This might mean that linux-next requires a beta version of rust, if
the release is expected before the merge window and it contains
an important change. Staying with fairly recent versions of clang
certainly helped in getting clang and the kernel to co-evolve and
get to the point we are now in using it as an alternative to gcc.
While Linux used to build with 12 year old compilers (4.1 until
2018), we now require a 6 year old gcc (4.9) or 1 year old
clang/llvm. I don't know whether these will fully converge over
time but it seems sensible that the minimum rust frontend version
we require for a new kernel release would eventually also fall
in that range, requiring a compiler that is no more than a few
years old, but not requiring the latest stable release.
Arnd
On Thu, Jul 16, 2020 at 03:06:01PM +0200, Arnd Bergmann wrote:
> On Sun, Jul 12, 2020 at 9:39 PM Josh Triplett <[email protected]> wrote:
> > On Sun, Jul 12, 2020 at 03:31:51PM +0300, Adrian Bunk wrote:
> > >
> > > As an example:
> > > Ubuntu LTS releases upgrade to a new Rust version every 1-2 months.
> > > Ubuntu 16.04 started with Rust 1.7.0 and is now at Rust 1.41.0.
> > >
> > > It would not sound good to me if security updates of distribution
> > > kernels might additionally end up using a different version of the
> > > Rust compiler - the toolchain for the kernel should be stable.
> > >
> > > Would Rust usage in the kernel require distributions to ship
> > > a "Rust for Firefox" and a "Rust for the kernel"?
> >
> > Rust has hard stability guarantees when upgrading from one stable
> > version to the next. If code compiles with a given stable version of
> > Rust, it'll compile with a newer stable version of Rust. Given that, a
> > stable distribution will just need a single sufficiently up-to-date Rust
> > that meets the minimum version requirements of both Firefox and Linux.
> >
> > (That would not apply if the kernel used nightly Rust, since
> > nightly-only features are allowed to change before becoming stable;
> > that's one reason why we should use stable Rust, and try to get Firefox
> > to stick to stable Rust.)
>
> I would expect we'd want a fairly tight coupling between kernel
> releases and minimum rust releases at first. Whatever is the latest
> stable rust version during the kernel's merge window might be
> assumed to be the minimum version for the life of that kernel, but
> an LTS release would not suddenly start relying on features
> from a newer compiler (thought it might warn about known bugs).
Agreed; we should be careful that any backported fix to an LTS kernel
does not increase the required Rust toolchain.
> While Linux used to build with 12 year old compilers (4.1 until
> 2018), we now require a 6 year old gcc (4.9) or 1 year old
> clang/llvm. I don't know whether these will fully converge over
> time but it seems sensible that the minimum rust frontend version
> we require for a new kernel release would eventually also fall
> in that range, requiring a compiler that is no more than a few
> years old, but not requiring the latest stable release.
I expect in the short term that we will likely have a need for features
from recent stable releases, especially when those features were added
specifically to support the kernel or similar, but the rate at which we
need new features will slow over time, and eventually we'll go from
"latest stable" to "within a year or so".
On Thu, Jul 16, 2020 at 03:06:01PM +0200, Arnd Bergmann wrote:
>
> I would expect we'd want a fairly tight coupling between kernel
> releases and minimum rust releases at first. Whatever is the latest
> stable rust version during the kernel's merge window might be
> assumed to be the minimum version for the life of that kernel, but
> an LTS release would not suddenly start relying on features
> from a newer compiler (thought it might warn about known bugs).
>
> This might mean that linux-next requires a beta version of rust, if
> the release is expected before the merge window and it contains
> an important change.
I would expect this is absolutely not wanted,
it would make testing recent kernels very hard.
If you want to keep a tool that tightly to the kernel,
please bundle it with the kernel and build it as part
of the kernel build.
I would suggest to start with a proper design/specification what the
kernel wants to use, so that you are confident that a compiler
implementing this will be sufficient for the next 5 years.
As a secondary benefit, starting with a proper design often brings
a better result than adding permanent features piece by piece.
As a tertiary benefit, it would avoid tying the kernel to one specific
compiler implementation. A compiler like mrustc or a hypothetical Rust
frontend for gcc could then implement a superset of what the kernel
needs.
> Staying with fairly recent versions of clang
> certainly helped in getting clang and the kernel to co-evolve and
> get to the point we are now in using it as an alternative to gcc.
The main difference is between an alternative to an existing tool,
and a mandatory new tool.
> While Linux used to build with 12 year old compilers (4.1 until
> 2018), we now require a 6 year old gcc (4.9) or 1 year old
> clang/llvm. I don't know whether these will fully converge over
> time but it seems sensible that the minimum rust frontend version
> we require for a new kernel release would eventually also fall
> in that range, requiring a compiler that is no more than a few
> years old, but not requiring the latest stable release.
The correct range for a mandatory tool are the 6 to 12 years for gcc.
Debian stable and Ubuntu LTS are providing (different) mechanisms
for installing the kernel from the next stable/LTS release 2 years
later[1] for supporting new hardware.
If kernel 5.12 LTS cannot be compiled on Ubuntu 20.04 LTS with
the 2019 gcc 9 there would be pain downstream.
In the embedded world spreads far wider than these 3 years are common.
I would currently have a real-life usecase for compiling a recent
kernel with a gcc 4.0 (sic) toolchain.
Properly supporting 15 year old toolchains would be painful upstream,
supporting 6 year old toolchains is a reasonable compromise between
not being too painful upstream while rarely causing pain downstream.
What applies to gcc does also apply to other external tools used
during the kernel build.
> Arnd
cu
Adrian
[1] following a new upstream kernel stable branch every 6 months (Ubuntu)
or the latest upstream stable kernels (Debian) until this is reached
From: Adrian Bunk
> Sent: 19 July 2020 19:19
...
> The correct range for a mandatory tool are the 6 to 12 years for gcc.
>
> Debian stable and Ubuntu LTS are providing (different) mechanisms
> for installing the kernel from the next stable/LTS release 2 years
> later[1] for supporting new hardware.
> If kernel 5.12 LTS cannot be compiled on Ubuntu 20.04 LTS with
> the 2019 gcc 9 there would be pain downstream.
We have customers that are (still) using RHEL6 (2.6.32 era).
Since we have to build C++ programs that will run on those
systems we have systems with very old toolchains.
(Yes, you can build in a chroot or an a VM but that needs
setting up quite carefully - easier to keep the old machine.)
It is rather a PITA when kernel builds deprecate the installed gcc.
Now I'm not sure a 5.x kernel will run with the userspace from RHEL6.
But I know it runs with that from Ubuntu 13.04.
In a few years I'll be able to use the 13.04 system for release builds
but it is still far too new for that.
However I can no longer build 'current' custom kernels on it.
I've other (newer) systems but they aren't any (maybe much) faster.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
Hi!
> > No, please make it a "is rust available" automatic config option. The
> > exact same way we already do the compiler versions and check for
> > various availability of compiler flags at config time.
>
> That sounds even better, and will definitely allow for more testing.
>
> We just need to make sure that any kernel CI infrastructure tests that
> right away, then, so that failures don't get introduced by a patch from
> someone without a Rust toolchain and not noticed until someone with a
> Rust toolchain tests it.
So... I'm fan of Rust, but while trying to use it one thing was obvious: it
takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM.
Kernel is quite big project, can CI infrastructure handle additional load?
Will developers see significantly longer compile times when Rust is widespread?
Best regards,
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
On Tue, Jul 28, 2020 at 10:40:38PM +0200, Pavel Machek wrote:
> > We just need to make sure that any kernel CI infrastructure tests that
> > right away, then, so that failures don't get introduced by a patch from
> > someone without a Rust toolchain and not noticed until someone with a
> > Rust toolchain tests it.
>
> So... I'm fan of Rust, but while trying to use it one thing was obvious: it
> takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM.
>
> Kernel is quite big project, can CI infrastructure handle additional load?
>
> Will developers see significantly longer compile times when Rust is widespread?
I wouldn't expect the addition of Rust to the kernel to substantially
impact overall build time; on balance, I'd expect the major bottleneck
in kernel builds to continue to be linking and other serialized steps,
not compiling and other highly parallel steps.
There are also *many* things that can be done to improve Rust build time
in a project. And I don't expect that in-kernel Rust will have many
dependencies on third-party crates (since they'd need to be checked into
the tree).
On Sun, Jul 12, 2020 at 12:39:44PM -0700, Josh Triplett wrote:
>...
> Rust has hard stability guarantees when upgrading from one stable
> version to the next. If code compiles with a given stable version of
> Rust, it'll compile with a newer stable version of Rust.
>...
In librsvg, breakages with more recent Rust versions in the past year
required updates of two vendored crates:
https://gitlab.gnome.org/GNOME/librsvg/-/commit/de26c4d8b192ed0224e6d38f54e429838608b902
https://gitlab.gnome.org/GNOME/librsvg/-/commit/696e4a6be2aeb00ea27945f94da066757431684d
For updating Rust in Debian stable for the next Firefox ESR update it
would actually be useful if these violations of the "hard stability
guarantee" in Rust get fixed, so that the old librsvg 2.44.10 builds
again with the latest Rust.
It also makes me wonder how such regressions slip into Rust releases.
cu
Adrian
On Mon, 24 Aug 2020, Adrian Bunk wrote:
> In librsvg, breakages with more recent Rust versions in the past year
> required updates of two vendored crates:
Interesting!
> https://gitlab.gnome.org/GNOME/librsvg/-/commit/de26c4d8b192ed0224e6d38f54e429838608b902
Looks like this was, for a while, a warning and not an error:
https://github.com/servo/rust-cssparser/commit/3c98d22c5de3b696bf1fde2b6c90069812312aa6
= warning: this error has been downgraded to a warning for backwards compatibility with previous releases
= warning: this represents potential undefined behavior in your code and this warning will become a hard error in the future
> https://gitlab.gnome.org/GNOME/librsvg/-/commit/696e4a6be2aeb00ea27945f94da066757431684d
Same here, and it looks like the same warning/error, too:
https://github.com/dimforge/nalgebra/issues/561
= warning: this error has been downgraded to a warning for backwards compatibility with previous releases
= warning: this represents potential undefined behavior in your code and this warning will become a hard error in the future
Following some links in that second issue, I see that this seems to be a
summary of what's going on:
https://github.com/rust-lang/rust/issues/59159
in particular this paragraph: "at the time that NLL [non-lexical
lifetimes] was stabilized, the compiler's acceptance of this borrowing
pattern was categorized by the NLL team as a 'known bug'. The NLL team
assumed that, as a bug fix, the compiler would be allowed to start
rejecting the pattern in the future."
That is, both of these cases were code that should never have been
accepted by the compiler because it is unsound, but the initial
implementation of NLL was not clever enough to detect it. They later
turned it into a warning, and later turned that into an error.
There is a link to this policy document which explains their process for
breaking existing code in the case where it's necessary to fix a compiler
bug or similar:
https://github.com/rust-lang/rfcs/blob/master/text/1589-rustc-bug-fix-procedure.md
There is also a link to this comment about why they decided to take this
approach:
https://github.com/rust-lang/rust/pull/58739#issuecomment-476387184
which includes the followup task, "We do a retrospective and look to ways
to alter our processes to better prevent this in the future." That is, it
seems to me that the Rust team sees it as a mistake that they ended up in
this situation.
Josh, do you know if that retrospective has happened? (I see some mumbling
about NLL -> Polonius, can we be confident something similar won't happen
with that? :) )
> For updating Rust in Debian stable for the next Firefox ESR update it
> would actually be useful if these violations of the "hard stability
> guarantee" in Rust get fixed, so that the old librsvg 2.44.10 builds
> again with the latest Rust.
>
> It also makes me wonder how such regressions slip into Rust releases.
I do conceptually agree with this, even though it is not technically a
"regression" (because it was really the old compiler that was buggy, not
the new one). If I understand it right, neither of these lines of code
were valid with the pre-NLL implementation either. They were only accepted
by the initial NLL implementation. While the purpose of NLL was to enable
writing (valid/sound) code that wouldn't have been accepted by the
previous, simpler implementation of lifetimes, it seems like it should
have been possible to opt out of it while there were "known bugs"
precisely to prevent these sort of effective-regressions. (And I suspect
that it was in fact possible to do so, but perhaps not documented clearly
/ perhaps there wasn't an awareness that this was a thing that users who
deeply value stability would want to do.)
And yeah, I can see the value of a --accept-previously-accepted-buggy-code
flag from the distros' point of view (even if I can also see why Rust
upstream would not be super excited about it :) ). After all, if the
choice is between the distro not taking _any_ bug fixes in rustc and the
distro taking all but one, the latter seems like a better option.
The other takeaway, I think, is that you should regularly compile with
warnings turned into hard errors. The policy document above (Rust RFC
1589) says that all such changes need to be a warning and not a hard error
for at least one compiler reversion. That happened in this case, and both
fixes were applied in the vendored crates while they were still warnings.
For any kernel Rust use, I'd strongly advocate for running CI on every
stable branch after each compiler release, preferably after each compiler
beta release, and shaking out any warnings - even if they are not warnings
the compiler will turn into errors on its own, they may still point to
logic bugs in the code.
(The Rust folks have some automation named "crater" for running these
sorts of tests across the Rust ecosystem, which the document mentions. I'd
expect that anything shipped in major distros, including librsvg + its
dependencies as well as any Rust in the kernel, should be included in
crater.)
--
Geoffrey Thomas
https://ldpreload.com
[email protected]
On Tue 2020-07-28 23:34:17, Josh Triplett wrote:
> On Tue, Jul 28, 2020 at 10:40:38PM +0200, Pavel Machek wrote:
> > > We just need to make sure that any kernel CI infrastructure tests that
> > > right away, then, so that failures don't get introduced by a patch from
> > > someone without a Rust toolchain and not noticed until someone with a
> > > Rust toolchain tests it.
> >
> > So... I'm fan of Rust, but while trying to use it one thing was obvious: it
> > takes _significantly_ longer than C to compile and needs gigabyte a lot of RAM.
> >
> > Kernel is quite big project, can CI infrastructure handle additional load?
> >
> > Will developers see significantly longer compile times when Rust is widespread?
>
> I wouldn't expect the addition of Rust to the kernel to substantially
> impact overall build time; on balance, I'd expect the major bottleneck
> in kernel builds to continue to be linking and other serialized steps,
> not compiling and other highly parallel steps.
Well.. not everyone has 32 cores in their notebook.
> There are also *many* things that can be done to improve Rust build time
> in a project. And I don't expect that in-kernel Rust will have many
> dependencies on third-party crates (since they'd need to be checked into
Okay. I did some refactoring recently and I really wished kernel was
in Rust (and not in C)... so lets see what happens.
Best regards,
Pavel
--
http://www.livejournal.com/~pavelmachek
On Thu, Dec 17, 2020 at 10:45 PM Pavel Machek <[email protected]> wrote:
>
> Well.. not everyone has 32 cores in their notebook.
It is true that complex Rust, like complex C++, does have high
compilation times. But it all depends on how much one relies on
certain language features. Straightforward Rust code is quick to
compile.
For reference, some quick stats :-)
On a given machine, building v5.10 with a minimal config under -j3
takes 3 minutes. With Rust support enabled and 4 trivial Rust modules,
it takes 50 seconds more. "A big increase!", you may indeed claim, but
those 50 seconds are basically all spent on the shared Rust support.
The actual Rust modules compile very quickly afterwards. For example,
touching either `drivers/char/mem.c` or one of the trivial Rust
modules takes the same time in that machine: 10 seconds.
This is for a minimal config -- when you start dealing with
`allmodconfig` builds, or when you start having a hundred Rust modules
instead of 4, the upfront cost becomes very small per Rust module.
> Okay. I did some refactoring recently and I really wished kernel was
> in Rust (and not in C)... so lets see what happens.
Indeed, I think it is definitely worth it.
Cheers,
Miguel