Hi All,
The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
Yet, various unofficial reports indicate it will be supported for 6 years. And AOSP has already declared the use
of 5.10 kernel in their Android S and T releases.
Is there some way we could make the LTS support more clear.
A 2 year declaration is not LTS any more.
If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
https://www.kernel.org/category/releases.html
Regards,
Scott
On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
>
> Yet, various unofficial reports indicate it will be supported for 6
> years. And AOSP has already declared the use of 5.10 kernel in their
> Android S and T releases.
5.10 will also be used for Debian Bullseye.
--
⢀⣴⠾⠻⢶⣦⠀ .--[ Makefile ]
⣾⠁⢠⠒⠀⣿⡁ # beware of races
⢿⡄⠘⠷⠚⠋⠀ all: pillage burn
⠈⠳⣄⠀⠀⠀⠀ `----
On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> Hi All,
>
> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
Because they want to use all of the latest stuff that 5.10 provides
them. Don't you want faster and more secure kernels for your devices?
> Yet, various unofficial reports indicate it will be supported for 6 years.
Rumors are nice, aren't they :)
>? And AOSP has already declared the use
> of 5.10 kernel in their Android S and T releases.
Publically? Where? And is that really the name of the new Android
releases, I thought they switched to numbers now (hence the naming of
the current android-common kernel branches, marketing is fun...)
> Is there some way we could make the LTS support more clear.
> A 2 year declaration is not LTS any more.
Not true at all, a "normal" stable kernel is dropped after the next
release happens, making their lifespan about 4 months long. 2 years is
much longer than 4 months, so it still is a "long term supported" kernel
in contrast, correct?
> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
> https://www.kernel.org/category/releases.html
Why? What would that change?
Ok, seriously, this happens every year, and every year we go through the
same thing, it's not like this is somehow new, right?
I want to see companies _using_ the kernel, and most importantly,
_updating_ their devices with it, to know if it is worth to keep around
for longer than 2 years. I also, hopefully, want to see how those
companies will help me out in the testing and maintenance of that kernel
version in order to make supporting it for 6 years actually possible.
So, are you planning on using 5.10? Will you will be willing to help
out in testing the -rc releases I make to let me know if there are any
problems, and to help in pointing out and backporting any specific
patches that your platforms need for that kernel release?
When I get this kind of promises and support from companies, then I am
glad to bump up the length of the kernel support from 2 to 6 years, and
I mark it on the web site. Traditionally this happens in Febuary/March
once I hear from enough companies. Can I count on your support in this
endeavor?
Also, a meta-comment. Please reconsider using a single kernel version
for longer than 2 years on systems that you actively support and
maintain. It's generally a bad idea unless you are stuck with millions
of out-of-tree code that something like a customer-unfriendly SoC vendor
provides. If you are stuck in that type of situation, well they have
decided to spend extra money to keep their out-of-tree code alive, so
why are they forcing you to also spend extra money and energy?
I can go on about this topic at length if you want me to, I have lots of
examples of how to, and not to, maintain a kernel for a device for a
long period of time...
thanks,
greg k-h
On 1/25/2021 11:29 PM, Greg Kroah-Hartman wrote:
> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
>> Hi All,
>>
>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
>
> Because they want to use all of the latest stuff that 5.10 provides
> them. Don't you want faster and more secure kernels for your devices?
>
>> Yet, various unofficial reports indicate it will be supported for 6 years.
>
> Rumors are nice, aren't they :)
>
>> And AOSP has already declared the use
>> of 5.10 kernel in their Android S and T releases.
>
> Publically? Where? And is that really the name of the new Android
> releases, I thought they switched to numbers now (hence the naming of
> the current android-common kernel branches, marketing is fun...)
>
>> Is there some way we could make the LTS support more clear.
>> A 2 year declaration is not LTS any more.
>
> Not true at all, a "normal" stable kernel is dropped after the next
> release happens, making their lifespan about 4 months long. 2 years is
> much longer than 4 months, so it still is a "long term supported" kernel
> in contrast, correct?
>
>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
>> https://www.kernel.org/category/releases.html
>
> Why? What would that change?
I believe what Scott is looking for is a form of official statement that
indicates that 5.10 is indeed a 6 years Long Term Stable kernel, as
opposed to an informal piece of information like this:
https://twitter.com/kernellogger/status/1320731458311491587
(personally that is how I got to hear about 5.10 being selected, taking
this with a grain of salt).
>
> Ok, seriously, this happens every year, and every year we go through the
> same thing, it's not like this is somehow new, right?
It is not new indeed but there is a recurring pattern, and I have been
told by some of our customers who use the kernel.org's releases.html
page that this is how they know which kernel is going to be LTS and plan
ahead accordingly.
If you feel making such a statement actually hurts the kernel project as
a whole, or sort of steers people into cramming all of their desired
features/bug fixes towards particular versions of the kernel that may,
or may not be 6 years LTS, that may be an argument against announcing
before, but maybe not after the kernel is released?
>
> I want to see companies _using_ the kernel, and most importantly,
> _updating_ their devices with it, to know if it is worth to keep around
> for longer than 2 years. I also, hopefully, want to see how those
> companies will help me out in the testing and maintenance of that kernel
> version in order to make supporting it for 6 years actually possible.
This really depends on which market you are in we have (Set-top-box and
cable modem) customers that do *want* to use newer kernels but are so
slow in adopting new ones that by the time they start the kernel has
already been phased out 2 years ago.
We have worked (and are still working) with a customer who wanted so
badly 4.1 they rattled our whole organization and had my team on call to
get there, only to start working on it seriously in 2019. The problem
was not necessarily the SoC vendor but the various OEMs and their bigger
pile of out of tree patches and the fact that the platforms were so old
that the essential knowledge of all of their little details was gone
(fired, retired, moved on) already.
6 years provides a great window of time for people to get to a quality
production with the kernel while offering enough time for slow users to
transition over and yet still have a comfortable window of updates. 2
years is a bit too short and will result in the SoC vendor or whoever
gets paid for to either continue support and maybe endorse the
responsibility of doing critical back ports themselves (meltdown,
spectre, etc.) with all the implied liability, or completely drop
support for the unmaintained kernel. In both situations, the device's
security is not as great as if a 6 years LTS kernel had been chosen.
People who can jump on the 2 year stable kernel will, and those who
cannot will not, so having 6 years is a good way to avoid skipping too
much yet try to live with what is/was current at the time.
>
> So, are you planning on using 5.10? Will you will be willing to help
> out in testing the -rc releases I make to let me know if there are any
> problems, and to help in pointing out and backporting any specific
> patches that your platforms need for that kernel release?
>
> When I get this kind of promises and support from companies, then I am
> glad to bump up the length of the kernel support from 2 to 6 years, and
> I mark it on the web site. Traditionally this happens in Febuary/March
> once I hear from enough companies. Can I count on your support in this
> endeavor?
I cannot comment on Scott's plan since we don't work in the same
business group, however we have active customers on Set-top-box and
cable modem devices that actively use the 4.9, start adopting 5.4 now
and by the end of this year the 5.10 kernel and I believe you do see me
requesting specific patches to be included once in a while ;). These
customers are a mix of RDK video, Android and just custom made Linux,
they all want a 6 years LTS kernel though because it puts them on a
support timeline they can work with given their pace and kernel expertise.
>
> Also, a meta-comment. Please reconsider using a single kernel version
> for longer than 2 years on systems that you actively support and
> maintain. It's generally a bad idea unless you are stuck with millions
> of out-of-tree code that something like a customer-unfriendly SoC vendor
> provides. If you are stuck in that type of situation, well they have
> decided to spend extra money to keep their out-of-tree code alive, so
> why are they forcing you to also spend extra money and energy?
We may get paid extra money by said customers to keep maintaining
whatever old kernel they have because it still is cheaper for them than
doing new shipments and deployments of devices in the field. This means
we sell fewer devices to them for a while, but by the same token we have
exclusivity for a while and do not yet open up a window for a new
project to be with a different vendor. Is not that great conflict of
self interest?
The other thing that we have seen, especially with Android is that you
start a project on one of the 3 kernels maintained at the time, get to
market, get maintenance updates on a branch (so many branches) and
that's it, no more updates for you, which is where the 6 year LTS is
sort of handy.
PS: the usual stuff: I am not expressing ideas from my employer here,
just my own.
--
Florian
Hi Greg,
On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
>> Hi All,
>>
>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
> Because they want to use all of the latest stuff that 5.10 provides
> them. Don't you want faster and more secure kernels for your devices?
Yes, 5.10 is a more secure and less buggy kernel than 5.4.
You don't need to convince me which is the better kernel to start using.
The kernel improves every version and we are at a point where regressions rarely happen.
But, it is not our choice what version a customer chooses as they have many valid reasons to stabilize their code.
They do have extensive QA cycles to go through and cannot simply pick up a newer kernel version to get a security fix or bug fix.
As everyone, we also have testing and cannot test every kernel version extensively so must choose wisely what to test.
We also track the mainline a test as well to ensure functionality - but stress testing is not done at the same level as a full QA cycle.
Unfortunately some people think that 5.4 LTS is more stable than 5.10. And it also has a longer lifetime.
So, people starting a new project today decide to use 5.4 (or even earlier) kernels as they deem these stable.
For these reasons we would be forced to backport (upstreamed code) to an old kernel for new features and processors, etc not supported in 5.4.
You can see how such a kernel would not be as stable as 5.10 if a single subtle fix is not backported. But, try convincing people to go with 5.10 with only a 2 year LTS lifespan vs. 5.4 with many more years.
Hopefully you understand the kernel is not the product - the end solution is what is tested and QA'd.
And, picking up small, safe LTS versions requires a much smaller QA cycle than a kernel upgrade with thousands of possible changes in the code every kernel version.
>
>> Yet, various unofficial reports indicate it will be supported for 6 years.
> Rumors are nice, aren't they :)
>
>> And AOSP has already declared the use
>> of 5.10 kernel in their Android S and T releases.
> Publically? Where? And is that really the name of the new Android
> releases, I thought they switched to numbers now (hence the naming of
> the current android-common kernel branches, marketing is fun...)
https://source.android.com/devices/architecture/kernel/android-common
Feature and launch kernels provides kernels supported per version.
>
>> Is there some way we could make the LTS support more clear.
>> A 2 year declaration is not LTS any more.
> Not true at all, a "normal" stable kernel is dropped after the next
> release happens, making their lifespan about 4 months long. 2 years is
> much longer than 4 months, so it still is a "long term supported" kernel
> in contrast, correct?
Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
The timeframes are very different.
>
>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
>> https://www.kernel.org/category/releases.html
> Why? What would that change?
>
> Ok, seriously, this happens every year, and every year we go through the
> same thing, it's not like this is somehow new, right?
No, but why do we need to keep playing the same game every year now.
>
> I want to see companies _using_ the kernel, and most importantly,
> _updating_ their devices with it, to know if it is worth to keep around
> for longer than 2 years. I also, hopefully, want to see how those
> companies will help me out in the testing and maintenance of that kernel
> version in order to make supporting it for 6 years actually possible.
>
> So, are you planning on using 5.10? Will you will be willing to help
> out in testing the -rc releases I make to let me know if there are any
> problems, and to help in pointing out and backporting any specific
> patches that your platforms need for that kernel release?
> When I get this kind of promises and support from companies, then I am
> glad to bump up the length of the kernel support from 2 to 6 years, and
> I mark it on the web site. Traditionally this happens in Febuary/March
> once I hear from enough companies. Can I count on your support in this
> endeavor?
It is a chicken and egg problem and 2 year LTS declaration is not helping customers commit to 5.10.
We are using 5.10 right now and testing it. But, we don't make the end products.
If customers decide to use 5.10 we will continue to test, support, and backport any specific patches to it.
If they choose 5.4 that would be the one we would have to backport and try and support.
The backport is not just for simple drivers.
VFIO, IOMMU, and other unidentified architecture support would need to be backported.
So 5.10 is the much better decision to pickup any LTS fixes in the future.
LTS 6 year support isn't the only decision factor though but makes it a showstopper
in convincing some management decisions for those not savvy with kernel LTS support.
Yes, customers have their own code they add to the kernel which we never see.
Unfortunately "they have other products on old LTS version".
So we need to convince them to move to new kernel version AND port their code.
>
> Also, a meta-comment. Please reconsider using a single kernel version
> for longer than 2 years on systems that you actively support and
> maintain. It's generally a bad idea unless you are stuck with millions
> of out-of-tree code that something like a customer-unfriendly SoC vendor
> provides. If you are stuck in that type of situation, well they have
> decided to spend extra money to keep their out-of-tree code alive, so
> why are they forcing you to also spend extra money and energy?
Obviously a management decision out of supplier's complete control.
>
> I can go on about this topic at length if you want me to, I have lots of
> examples of how to, and not to, maintain a kernel for a device for a
> long period of time...
You don't need to convince me.
>
> thanks,
>
> greg k-h
On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
> Hi Greg,
>
>
> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
> > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> >> Hi All,
> >>
> >> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
> >> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
> > Because they want to use all of the latest stuff that 5.10 provides
> > them. Don't you want faster and more secure kernels for your devices?
> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
Great, use it, ship it to your customers and we are all happy. What do
you need me for any of this? :)
> >> ? And AOSP has already declared the use
> >> of 5.10 kernel in their Android S and T releases.
> > Publically? Where? And is that really the name of the new Android
> > releases, I thought they switched to numbers now (hence the naming of
> > the current android-common kernel branches, marketing is fun...)
> https://source.android.com/devices/architecture/kernel/android-common
> Feature and launch kernels provides kernels supported per version.
Oh nice, didn't know that.
But note, Android kernels do not reflect the lifespan of LTS kernels.
If that were the case, I would still be supporting 3.18 as they are
doing that at the moment for their devices and customers, and will be
doing so for I think another full year.
So while Android is nice to see here, remember that is what Google is
promising to support for their users. You can do the same thing for
your users, what do you need me here for this? You can do the same
thing that Google is doing for 3.18 right now, pick the stable fixes
from upstream, backport them, test them, and push them out to their
users.
While Google is a great help to me in the LTS effort, providing huge
amounts of resources to enable my life easier with this (i.e. funding
Linaro's testing efforts), their promise to their customers/users does
not depend on me keeping LTS kernels alive, if I stopped tomorrow their
contracts are still in place and they know how to do this work
themselves (as is proof with 3.18).
So you can provide the same kind of guarantee to support any kernel
version for any amount of time to any customer you like, it shouldn't
require me to do that work for you, right?
> >> Is there some way we could make the LTS support more clear.
> >> A 2 year declaration is not LTS any more.
> > Not true at all, a "normal" stable kernel is dropped after the next
> > release happens, making their lifespan about 4 months long. 2 years is
> > much longer than 4 months, so it still is a "long term supported" kernel
> > in contrast, correct?
> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
> The timeframes are very different.
At this point in time, anyone wanting a kernel longer than 2 years
should know how this all works.
If not, please do some basic research, I have written whitepapers on
this and given numerous talks. The information is out there...
> >> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
> >> https://www.kernel.org/category/releases.html
> > Why? What would that change?
> >
> > Ok, seriously, this happens every year, and every year we go through the
> > same thing, it's not like this is somehow new, right?
> No, but why do we need to keep playing the same game every year now.
Because, 5.4 almost did not become "6 years" of support from me. That
was because in the beginning, no one said they were going to use it in
their devices and offer me help in testing and backporting. Only when I
knew for sure that we had people helping this out did I change the date
on kernel.org.
So far the jury is still out for 5.10, are you willing to help with
this? If not, why are you willing to hope that others are going to do
your work for you? I am talking to some companies, but am not willing
to commit to anything in public just yet, because no one has committed
to me yet.
What would you do if you were in my situation?
thanks,
greg k-h
Hi Greg,
On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
> > > Ok, seriously, this happens every year, and every year we go through the
> > > same thing, it's not like this is somehow new, right?
> > No, but why do we need to keep playing the same game every year now.
>
> Because, 5.4 almost did not become "6 years" of support from me. That
> was because in the beginning, no one said they were going to use it in
> their devices and offer me help in testing and backporting. Only when I
> knew for sure that we had people helping this out did I change the date
> on kernel.org.
>
> So far the jury is still out for 5.10, are you willing to help with
> this? If not, why are you willing to hope that others are going to do
> your work for you? I am talking to some companies, but am not willing
> to commit to anything in public just yet, because no one has committed
> to me yet.
This is interesting, because I used to extend LTS kernels after you dropped
them exactly for the reason I needed them. With 6 months of porting+testing,
~3 years of major version life in field, and roughly 6 extra months covering
late starts or minor extensions to help match a deadline, I figured that 4
years was the minimum we needed for our products. As such, 6 years add some
comfort, but as you probably remember I once got scared not knowing if 4.19
was going to be extended or not as we were having plenty in field and I did
not plan to get back to that maintenance myself. These days, the 6 years
duration allows us to skip one upgrade if we're too late, but we still
prefer to upgrade every year. It's also true that before engaging in that
direction before the official statement was on the site, each time I
preferred to ask you how you felt about it, and I'm sure most major players
decide after checking with you as well. For sure knowing better early would
be much better but I understand your concerns about doing that job for free
and for no reason.
I initially got the impression that the extra resources you got were enough
to help you and to keep me away from annoying you again with pushing my late
branches, but it's certain that free time cannot extend forever. And You,
Ben, Sasha and me definitely know how painful it is to backport past 2.5-3
years, especially with security stuff that everyone expects but often is
the riskiest. While most of my time is spent in userland these days, I'm
still willing to help as I can if that saves from from having to takeover
an old kernel again on my week-ends.
One of my impression, which might or might not be shared, is that the
duration is more important than the frequency, which means that having
one extended kernel every two years would bring much more value than
maintaining all of them only 3 years, and would equally result in cutting
the effort in half: with 6 years, you still have 5 years if you upgrade
every two years.
> What would you do if you were in my situation?
I'd continue to ask for a shared effort for extended support, which
everyone benefits from.
Maybe one possibility would be to start gauging upfront around september
whether or not the end-of-year's LTS should be an extended LTS or not,
and if so, what's needed and who's willing to particpate. I suspect that
numerous companies have available resources to help but are not even
aware that they could help, and they're seeing something which works
extremely well so there's nothing to try to improve. And if nobody
provides resources, it means there's not enough interest to create yet
another extra LTS kernel so better save the effort.
Just my two cents,
Willy
On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
> > Hi Greg,
> >
> >
> > On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
> > > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> > >> Hi All,
> > >>
> > >> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
> > >> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
> > > Because they want to use all of the latest stuff that 5.10 provides
> > > them. Don't you want faster and more secure kernels for your devices?
> > Yes, 5.10 is a more secure and less buggy kernel than 5.4.
>
> Great, use it, ship it to your customers and we are all happy. What do
> you need me for any of this? :)
>
> > >> ? And AOSP has already declared the use
> > >> of 5.10 kernel in their Android S and T releases.
> > > Publically? Where? And is that really the name of the new Android
> > > releases, I thought they switched to numbers now (hence the naming of
> > > the current android-common kernel branches, marketing is fun...)
> > https://source.android.com/devices/architecture/kernel/android-common
> > Feature and launch kernels provides kernels supported per version.
>
> Oh nice, didn't know that.
>
> But note, Android kernels do not reflect the lifespan of LTS kernels.
> If that were the case, I would still be supporting 3.18 as they are
> doing that at the moment for their devices and customers, and will be
> doing so for I think another full year.
>
> So while Android is nice to see here, remember that is what Google is
> promising to support for their users. You can do the same thing for
> your users, what do you need me here for this? You can do the same
> thing that Google is doing for 3.18 right now, pick the stable fixes
> from upstream, backport them, test them, and push them out to their
> users.
>
> While Google is a great help to me in the LTS effort, providing huge
> amounts of resources to enable my life easier with this (i.e. funding
> Linaro's testing efforts), their promise to their customers/users does
> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
> contracts are still in place and they know how to do this work
> themselves (as is proof with 3.18).
>
> So you can provide the same kind of guarantee to support any kernel
> version for any amount of time to any customer you like, it shouldn't
> require me to do that work for you, right?
>
> > >> Is there some way we could make the LTS support more clear.
> > >> A 2 year declaration is not LTS any more.
> > > Not true at all, a "normal" stable kernel is dropped after the next
> > > release happens, making their lifespan about 4 months long. 2 years is
> > > much longer than 4 months, so it still is a "long term supported" kernel
> > > in contrast, correct?
> > Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
> > The timeframes are very different.
>
> At this point in time, anyone wanting a kernel longer than 2 years
> should know how this all works.
>
> If not, please do some basic research, I have written whitepapers on
> this and given numerous talks. The information is out there...
>
> > >> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
> > >> https://www.kernel.org/category/releases.html
> > > Why? What would that change?
> > >
> > > Ok, seriously, this happens every year, and every year we go through the
> > > same thing, it's not like this is somehow new, right?
> > No, but why do we need to keep playing the same game every year now.
>
> Because, 5.4 almost did not become "6 years" of support from me. That
> was because in the beginning, no one said they were going to use it in
> their devices and offer me help in testing and backporting. Only when I
> knew for sure that we had people helping this out did I change the date
> on kernel.org.
>
> So far the jury is still out for 5.10, are you willing to help with
> this? If not, why are you willing to hope that others are going to do
> your work for you? I am talking to some companies, but am not willing
> to commit to anything in public just yet, because no one has committed
> to me yet.
Following up on this as I did not hear back from you. Are you and/or
your company willing to help out with the testing of 5.10 to ensure that
it is a LTS kernel? So far I have not had any companies agree to help
out with this effort, which is sad to see as it seems that companies
want 6 years of stable kernels, yet do not seem to be able to at the
least, do a test-build/run of those kernels, which is quite odd...
If you want to point people at your company this link that explains it
all in a single location instead of an email thread:
http://www.kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/
that would be great.
thanks,
greg k-h
Hi Greg,
On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
> On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
>> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
>>> Hi Greg,
>>>
>>>
>>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
>>>> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
>>>>> Hi All,
>>>>>
>>>>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
>>>>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
>>>> Because they want to use all of the latest stuff that 5.10 provides
>>>> them. Don't you want faster and more secure kernels for your devices?
>>> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
>>
>> Great, use it, ship it to your customers and we are all happy. What do
>> you need me for any of this? :)
>>
>>>>> And AOSP has already declared the use
>>>>> of 5.10 kernel in their Android S and T releases.
>>>> Publically? Where? And is that really the name of the new Android
>>>> releases, I thought they switched to numbers now (hence the naming of
>>>> the current android-common kernel branches, marketing is fun...)
>>> https://source.android.com/devices/architecture/kernel/android-common
>>> Feature and launch kernels provides kernels supported per version.
>>
>> Oh nice, didn't know that.
>>
>> But note, Android kernels do not reflect the lifespan of LTS kernels.
>> If that were the case, I would still be supporting 3.18 as they are
>> doing that at the moment for their devices and customers, and will be
>> doing so for I think another full year.
>>
>> So while Android is nice to see here, remember that is what Google is
>> promising to support for their users. You can do the same thing for
>> your users, what do you need me here for this? You can do the same
>> thing that Google is doing for 3.18 right now, pick the stable fixes
>> from upstream, backport them, test them, and push them out to their
>> users.
>>
>> While Google is a great help to me in the LTS effort, providing huge
>> amounts of resources to enable my life easier with this (i.e. funding
>> Linaro's testing efforts), their promise to their customers/users does
>> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
>> contracts are still in place and they know how to do this work
>> themselves (as is proof with 3.18).
>>
>> So you can provide the same kind of guarantee to support any kernel
>> version for any amount of time to any customer you like, it shouldn't
>> require me to do that work for you, right?
>>
>>>>> Is there some way we could make the LTS support more clear.
>>>>> A 2 year declaration is not LTS any more.
>>>> Not true at all, a "normal" stable kernel is dropped after the next
>>>> release happens, making their lifespan about 4 months long. 2 years is
>>>> much longer than 4 months, so it still is a "long term supported" kernel
>>>> in contrast, correct?
>>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
>>> The timeframes are very different.
>>
>> At this point in time, anyone wanting a kernel longer than 2 years
>> should know how this all works.
>>
>> If not, please do some basic research, I have written whitepapers on
>> this and given numerous talks. The information is out there...
>>
>>>>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
>>>>> https://www.kernel.org/category/releases.html
>>>> Why? What would that change?
>>>>
>>>> Ok, seriously, this happens every year, and every year we go through the
>>>> same thing, it's not like this is somehow new, right?
>>> No, but why do we need to keep playing the same game every year now.
>>
>> Because, 5.4 almost did not become "6 years" of support from me. That
>> was because in the beginning, no one said they were going to use it in
>> their devices and offer me help in testing and backporting. Only when I
>> knew for sure that we had people helping this out did I change the date
>> on kernel.org.
>>
>> So far the jury is still out for 5.10, are you willing to help with
>> this? If not, why are you willing to hope that others are going to do
>> your work for you? I am talking to some companies, but am not willing
>> to commit to anything in public just yet, because no one has committed
>> to me yet.
>
> Following up on this as I did not hear back from you. Are you and/or
> your company willing to help out with the testing of 5.10 to ensure that
> it is a LTS kernel? So far I have not had any companies agree to help
> out with this effort, which is sad to see as it seems that companies
> want 6 years of stable kernels, yet do not seem to be able to at the
> least, do a test-build/run of those kernels, which is quite odd...
I personally cannot commit to supporting this kernel for 6 years
(and personally do not want to backport new features to a 6 year old kernel).
And customers are finicky and ask for one thing and then change their mind later.
We'll have to see what decisions are made at a company level for this as there
are added costs to run tests on LTS kernel branches. We already run extensive QA on
whatever active development branches are in use and a subset on the mainline
branch as well. QA resources are finite and committing those for 6 years is
not something that makes sense if customers drop that kernel version.
Testing of the LTS kernel changes really moves out of our hands and into the
customer's testing after our major releases to them.
Other difficulty with the LTS version is the frequency it is updated. We would not
pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
is identified would be when we update and perform any meaningful testing when in maintainence.
>
> If you want to point people at your company this link that explains it
> all in a single location instead of an email thread:
> http://www.kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/
> that would be great.
Thanks for the link. Will forward it to see if it helps make any commitment from customers
or company.
The link Pavel sent looks interesting.
"CIP project intends to maintain 5.10 for a long time, so you can expect testing/help from them:
https://www.cip-project.org/blog/2020/12/02/cip-to-embark-on-kernel-5-10-development-for-slts
"
If the CIP project has committed to 10 years you would think they would be in contact
with you to add their support to the 5.10 LTS effort.
>
> thanks,
>
> greg k-h
>
Thanks,
Scott
On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
> Hi Greg,
>
> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
> > On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
> >> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
> >>> Hi Greg,
> >>>
> >>>
> >>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
> >>>> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> >>>>> Hi All,
> >>>>>
> >>>>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
> >>>>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
> >>>> Because they want to use all of the latest stuff that 5.10 provides
> >>>> them. Don't you want faster and more secure kernels for your devices?
> >>> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
> >>
> >> Great, use it, ship it to your customers and we are all happy. What do
> >> you need me for any of this? :)
> >>
> >>>>> ? And AOSP has already declared the use
> >>>>> of 5.10 kernel in their Android S and T releases.
> >>>> Publically? Where? And is that really the name of the new Android
> >>>> releases, I thought they switched to numbers now (hence the naming of
> >>>> the current android-common kernel branches, marketing is fun...)
> >>> https://source.android.com/devices/architecture/kernel/android-common
> >>> Feature and launch kernels provides kernels supported per version.
> >>
> >> Oh nice, didn't know that.
> >>
> >> But note, Android kernels do not reflect the lifespan of LTS kernels.
> >> If that were the case, I would still be supporting 3.18 as they are
> >> doing that at the moment for their devices and customers, and will be
> >> doing so for I think another full year.
> >>
> >> So while Android is nice to see here, remember that is what Google is
> >> promising to support for their users. You can do the same thing for
> >> your users, what do you need me here for this? You can do the same
> >> thing that Google is doing for 3.18 right now, pick the stable fixes
> >> from upstream, backport them, test them, and push them out to their
> >> users.
> >>
> >> While Google is a great help to me in the LTS effort, providing huge
> >> amounts of resources to enable my life easier with this (i.e. funding
> >> Linaro's testing efforts), their promise to their customers/users does
> >> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
> >> contracts are still in place and they know how to do this work
> >> themselves (as is proof with 3.18).
> >>
> >> So you can provide the same kind of guarantee to support any kernel
> >> version for any amount of time to any customer you like, it shouldn't
> >> require me to do that work for you, right?
> >>
> >>>>> Is there some way we could make the LTS support more clear.
> >>>>> A 2 year declaration is not LTS any more.
> >>>> Not true at all, a "normal" stable kernel is dropped after the next
> >>>> release happens, making their lifespan about 4 months long. 2 years is
> >>>> much longer than 4 months, so it still is a "long term supported" kernel
> >>>> in contrast, correct?
> >>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
> >>> The timeframes are very different.
> >>
> >> At this point in time, anyone wanting a kernel longer than 2 years
> >> should know how this all works.
> >>
> >> If not, please do some basic research, I have written whitepapers on
> >> this and given numerous talks. The information is out there...
> >>
> >>>>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
> >>>>> https://www.kernel.org/category/releases.html
> >>>> Why? What would that change?
> >>>>
> >>>> Ok, seriously, this happens every year, and every year we go through the
> >>>> same thing, it's not like this is somehow new, right?
> >>> No, but why do we need to keep playing the same game every year now.
> >>
> >> Because, 5.4 almost did not become "6 years" of support from me. That
> >> was because in the beginning, no one said they were going to use it in
> >> their devices and offer me help in testing and backporting. Only when I
> >> knew for sure that we had people helping this out did I change the date
> >> on kernel.org.
> >>
> >> So far the jury is still out for 5.10, are you willing to help with
> >> this? If not, why are you willing to hope that others are going to do
> >> your work for you? I am talking to some companies, but am not willing
> >> to commit to anything in public just yet, because no one has committed
> >> to me yet.
> >
> > Following up on this as I did not hear back from you. Are you and/or
> > your company willing to help out with the testing of 5.10 to ensure that
> > it is a LTS kernel? So far I have not had any companies agree to help
> > out with this effort, which is sad to see as it seems that companies
> > want 6 years of stable kernels, yet do not seem to be able to at the
> > least, do a test-build/run of those kernels, which is quite odd...
> I personally cannot commit to supporting this kernel for 6 years
> (and personally do not want to backport new features to a 6 year old kernel).
Why would you backport new features to an old kernel? That's not what
they are there for.
> And customers are finicky and ask for one thing and then change their mind later.
Sure, but it's up to you to define what you will support, for your
customers, which is what I am asking here.
> We'll have to see what decisions are made at a company level for this as there
> are added costs to run tests on LTS kernel branches. We already run extensive QA on
> whatever active development branches are in use and a subset on the mainline
> branch as well. QA resources are finite and committing those for 6 years is
> not something that makes sense if customers drop that kernel version.
> Testing of the LTS kernel changes really moves out of our hands and into the
> customer's testing after our major releases to them.
That feels wrong, but hey, I'm not your customer. If I was, I would be
demanding all LTS updates for the specific kernel branch you gave me as
I am paying for that support :)
> Other difficulty with the LTS version is the frequency it is updated. We would not
> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> is identified would be when we update and perform any meaningful testing when in maintainence.
How are you "identifying" these "critical fixes"? We fix at least one
known security issue a week, and probably multitudes of
unknown-at-this-moment ones. How are you determining when you need to
send a new base kernel update off to your customers? At such long
intervals it feels like anyone using your kernel releases is woefully
insecure.
Although if you do get people to update on a quarterly basis no matter
what, that's better than nothing, I'll take that. What's the odds that
this really happens?
> > If you want to point people at your company this link that explains it
> > all in a single location instead of an email thread:
> > http://www.kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/
> > that would be great.
> Thanks for the link. Will forward it to see if it helps make any commitment from customers
> or company.
Thanks. When can I expect to hear something back, even if it is a "no,
we don't care about supporting 5.10"?
> The link Pavel sent looks interesting.
> "CIP project intends to maintain 5.10 for a long time, so you can expect testing/help from them:
> https://www.cip-project.org/blog/2020/12/02/cip-to-embark-on-kernel-5-10-development-for-slts
> "
>
> If the CIP project has committed to 10 years you would think they would be in contact
> with you to add their support to the 5.10 LTS effort.
They are doing testing right now, see the announcements where they test
each stable -rc release. But they have not talked to me about 5.10
yet, Their model is that they will, somehow in a way that is yet to be
determined, take over maintaining these releases _after_ I drop them.
But they are only going to do so for a very specific hardware platform
or two, so anyone using anything other than their specific boards, is
going to be out of luck.
Which makes sense, scoping support like this to those that actually care
about a specific hardware platform is much easier than the work that I
and Sasha do in support it for all platforms. So if you are interested
in 10 years, please work with CIP to get your platform into that list.
thanks,
greg k-h
Hi!
(Second attempt to reply, as first one is not in the archives?!)
> > So far the jury is still out for 5.10, are you willing to help with
> > this? If not, why are you willing to hope that others are going to do
> > your work for you? I am talking to some companies, but am not willing
> > to commit to anything in public just yet, because no one has committed
> > to me yet.
>
> Following up on this as I did not hear back from you. Are you and/or
> your company willing to help out with the testing of 5.10 to ensure that
> it is a LTS kernel? So far I have not had any companies agree to help
> out with this effort, which is sad to see as it seems that companies
> want 6 years of stable kernels, yet do not seem to be able to at the
> least, do a test-build/run of those kernels, which is quite odd...
We expect to maintain 5.10.X kernel for 10 years, which means you can
expect similar testing we do for 4.4 and 4.19 to happen for 5.10:
https://www.cip-project.org/blog/2020/12/02/cip-to-embark-on-kernel-5-10-development-for-slts
If Broadcom (or anyone else) needs long-term support for 5.10 kernel,
they are welcome to explore/join the CIP project. That should happen
even if 5.10-stable is only mainained for 2 years.
More information can be found here:
https://wiki.linuxfoundation.org/civilinfrastructureplatform/start
https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptesting/cipreferencehardware
Best regards,
Pavel
PS: Speaking for myself, but not saying anything not publicly known, I
believe :-). I believe I can get more official statements if someone
needs them.
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
On Thu, Feb 18, 2021 at 08:43:48AM +0100, Greg Kroah-Hartman wrote:
> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
> > Other difficulty with the LTS version is the frequency it is updated.
What a stange statement! So basically if fixes come in quickly so that
customers are not exposed too long to well-known issues, it's a difficulty ?
I guess by now every serious OS vendor provides at least weekly fixes, and
at an era where devices are all interconnected, it's really necessary
(unless of course you don't care about your customer's security).
> > We would not
> > pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> > is identified would be when we update and perform any meaningful testing when in maintainence.
>
> How are you "identifying" these "critical fixes"? We fix at least one
> known security issue a week, and probably multitudes of
> unknown-at-this-moment ones. How are you determining when you need to
> send a new base kernel update off to your customers? At such long
> intervals it feels like anyone using your kernel releases is woefully
> insecure.
+1! It seems like this dangerous practice will never end :-(
Let me explain a personal experience. When I took over 2.6.32 many years
ago, Greg asked me to adapt to the new maintenance process involving the
patch reviews. At first I feared that it would increase my amount of work.
And it did. But I also discovered how important these reviews were, because
I started to get lots of "don't take this one in this version" and more
importantly "if you merge this you'll need these ones as well". And very
quickly I discovered how bogus the branches I used to maintain before
had been, given the high feedback ratio!
So based on this experience, I can assure anyone doing cherry-picks in
their garage from LTS kernels that they're doing crap and that they must
not distribute these kernels to anyone because THESE KERNELS ARE DANGEROUS.
It's even very easy to introduce vulnerabilities by doing this!
The only set of fixes that can be trusted are the "official" stable
kernels, because they are the only ones that are approved by the patches
authors themselves. Adding more stuff on top of stable kernels is fine
(and done at your own risk), but randomly dropping stuff from stable
kernels just because you don't think you need that is totally non-sense
and must not be done anymore!
Willy
Hi!
> Why would you backport new features to an old kernel? That's not what
> they are there for.
For CIP project, this is one of advantages for "supported" boards, as
we'll backport patches improving their support as long as those
patches are mainline.
> > If the CIP project has committed to 10 years you would think they would be in contact
> > with you to add their support to the 5.10 LTS effort.
>
> They are doing testing right now, see the announcements where they test
> each stable -rc release. But they have not talked to me about 5.10
> yet, Their model is that they will, somehow in a way that is yet to be
> determined, take over maintaining these releases _after_ I drop them.
> But they are only going to do so for a very specific hardware platform
> or two, so anyone using anything other than their specific boards, is
> going to be out of luck.
I'd say this is a bit inaccurate :-). We care about different boards
at armv7, armv8 and x86-64 architectures, which actually means we care
about quite big part of the kernel already. Rough estimate is that 50%
of patches in stable touch our configurations, I could collect better
statistics.
As you said, I'm not sure everything is fully determined at this
point, but... we'll certainly try to support community effort for 4.4,
4.19 and 5.10 maintainance for as long as possible.
It is well possible that we'll continue to maintain even
configurations we can't test (we don't have s390 to test on) on
best-effort basis (to help community and because applying a patch may
be easier than determining if someone in CIP community depends on it).
> Which makes sense, scoping support like this to those that actually care
> about a specific hardware platform is much easier than the work that I
> and Sasha do in support it for all platforms. So if you are interested
> in 10 years, please work with CIP to get your platform into that list.
And yes, this is very good suggestion.
Best regards,
Pavel
PS: Again, I'm speaking for myself.
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
On Thu, Feb 18, 2021 at 04:15:11PM +0200, Jari Ruusu wrote:
> Willy Tarreau wrote:
> > The only set of fixes that can be trusted are the "official" stable
> > kernels, because they are the only ones that are approved by the patches
> > authors themselves. Adding more stuff on top of stable kernels is fine
> > (and done at your own risk), but randomly dropping stuff from stable
> > kernels just because you don't think you need that is totally non-sense
> > and must not be done anymore!
>
> This may be little bit off-topic... but stable kernel.org kernels
> can also bit-rot badly because of "selective" backporting... as in
> anything that does not apply cleanly gets dropped regardless of
> how critical they are.
>
> I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
> kernel.org stable kernels is currently missing many critical
> locking fixes.
Why has no one asked for the specific upstream commits to be backported
if this is the case?
> As a result, that in-tree iwlwifi driver causes
> erratic behavior to random unrelated processes, and has been doing
> so for many months now. My not-so-politically correct opinion is
> that in-tree iwlwifi is completely FUBAR unless someone steps up
> to do professional quality backport of those locking fixes from
> upstream out-of-tree Intel version [1] [2] of the driver.
Why does any out-of-tree driver come into play here? What is wrong with
the in-kernel code?
> For me
> only way to get properly working WiFi on my laptop computer is to
> compile that Intel out-of-tree version. Sad, but true.
Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
is not the recommended thing to use the LTS kernels for.
thanks,
greg k-h
Willy Tarreau wrote:
> The only set of fixes that can be trusted are the "official" stable
> kernels, because they are the only ones that are approved by the patches
> authors themselves. Adding more stuff on top of stable kernels is fine
> (and done at your own risk), but randomly dropping stuff from stable
> kernels just because you don't think you need that is totally non-sense
> and must not be done anymore!
This may be little bit off-topic... but stable kernel.org kernels
can also bit-rot badly because of "selective" backporting... as in
anything that does not apply cleanly gets dropped regardless of
how critical they are.
I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
kernel.org stable kernels is currently missing many critical
locking fixes. As a result, that in-tree iwlwifi driver causes
erratic behavior to random unrelated processes, and has been doing
so for many months now. My not-so-politically correct opinion is
that in-tree iwlwifi is completely FUBAR unless someone steps up
to do professional quality backport of those locking fixes from
upstream out-of-tree Intel version [1] [2] of the driver. For me
only way to get properly working WiFi on my laptop computer is to
compile that Intel out-of-tree version. Sad, but true.
[1] https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi/core_release
[2] https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
On Thu, Feb 18, 2021 at 04:15:11PM +0200, Jari Ruusu wrote:
> Willy Tarreau wrote:
> > The only set of fixes that can be trusted are the "official" stable
> > kernels, because they are the only ones that are approved by the patches
> > authors themselves. Adding more stuff on top of stable kernels is fine
> > (and done at your own risk), but randomly dropping stuff from stable
> > kernels just because you don't think you need that is totally non-sense
> > and must not be done anymore!
>
> This may be little bit off-topic... but stable kernel.org kernels
> can also bit-rot badly because of "selective" backporting... as in
> anything that does not apply cleanly gets dropped regardless of
> how critical they are.
Sure it will. And the huge difference is that it usually gets quickly
spotted and fixed. For sensitive servers I tend to apply the principle
of not necessarily updating to the latest stable kernel but one or two
versions before it which nobody complained about. And I'm pretty fine
with skipping a significant number of updates (we all do that anyway).
> I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
> kernel.org stable kernels is currently missing many critical
> locking fixes. As a result, that in-tree iwlwifi driver causes
> erratic behavior to random unrelated processes, and has been doing
> so for many months now. My not-so-politically correct opinion is
> that in-tree iwlwifi is completely FUBAR unless someone steps up
> to do professional quality backport of those locking fixes from
> upstream out-of-tree Intel version [1] [2] of the driver.
I see, and it happens with plenty of other drivers or subsystems. Is
it in any way the stable branch's or stable maintainer's fault if
someone doesn't correctly do the backporting job on their driver ? No.
Is it expected that a driver works perfectly from its inclusion ? No.
Is it expected that a driver can always be fixed without a significant
rework that risks more breakage than fixes ? No. Some design limitations
or errors can require so many changes that they're unfixable in place.
I even had to *document* security issues in 2.4 because fixing them was
riskier than keeping them. This happens in any piece of software.
It's always been the case that some older kernels work less well than
some newer ones due to limited features, partially wrong drivers etc,
and getting better drivers is a valid reason for upgrading to a more
recent one. However the older driver ought to continue to be maintained
in a working state for those for whom it works fine.
> For me
> only way to get properly working WiFi on my laptop computer is to
> compile that Intel out-of-tree version. Sad, but true.
That's perfectly fine from my point of view. I've been doing the same
for certain driver (e.g. e100 vs eepro100 15 years ago) and have been
pleased to be able to stop using those out-of-tree versions. This is
also in order to make this possible for those who need to do it that
LTS kernels provide a lot of value: such out-of-tree drivers tend to
take some time to resynchronize with latest updates, and once they're
updated, you can use your machine for quite some time with them.
Obviously if somemone is able to figure the required fixes for the locking
bugs you mentioned above and to submit patches for stable branches, I'm
sure Greg will appreciate! But maybe that's not fixable there and you need
to upgrade. Usually you pick an LTS kernel for a specific hardware. If it
works that's great. But you cannot expect hardware to suddenly start to
work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
that's basically all and that's not their purpose.
Willy
On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>> Following up on this as I did not hear back from you. Are you and/or
>> your company willing to help out with the testing of 5.10 to ensure that
>> it is a LTS kernel? So far I have not had any companies agree to help
>> out with this effort, which is sad to see as it seems that companies
>> want 6 years of stable kernels, yet do not seem to be able to at the
>> least, do a test-build/run of those kernels, which is quite odd...
>I personally cannot commit to supporting this kernel for 6 years
>(and personally do not want to backport new features to a 6 year old kernel).
>And customers are finicky and ask for one thing and then change their mind later.
Why would we commit to maintining an upstream LTS for 6 years then? If
no one ends up using it (and we don't want anyone using older LTS
kernels) we're still stuck maintaining it.
>We'll have to see what decisions are made at a company level for this as there
>are added costs to run tests on LTS kernel branches. We already run extensive QA on
This sounds very wrong: it's ok to get volunteers to commit to 6 years
while the company that is asking for it won't do the same?
Shouldn't Broadcom commit to the work involved here first?
>whatever active development branches are in use and a subset on the mainline
>branch as well. QA resources are finite and committing those for 6 years is
>not something that makes sense if customers drop that kernel version.
>Testing of the LTS kernel changes really moves out of our hands and into the
>customer's testing after our major releases to them.
Keep in mind that QA resources are generally more abundant than
engineering resources that need to actually backport stuff to old
kernels.
--
Thanks,
Sasha
On 2/18/2021 3:31 AM, Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 08:43:48AM +0100, Greg Kroah-Hartman wrote:
>> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>>> Other difficulty with the LTS version is the frequency it is updated.
>
> What a stange statement! So basically if fixes come in quickly so that
> customers are not exposed too long to well-known issues, it's a difficulty ?
> I guess by now every serious OS vendor provides at least weekly fixes, and
> at an era where devices are all interconnected, it's really necessary
> (unless of course you don't care about your customer's security).
>
>>> We would not
>>> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
>>> is identified would be when we update and perform any meaningful testing when in maintainence.
>>
>> How are you "identifying" these "critical fixes"? We fix at least one
>> known security issue a week, and probably multitudes of
>> unknown-at-this-moment ones. How are you determining when you need to
>> send a new base kernel update off to your customers? At such long
>> intervals it feels like anyone using your kernel releases is woefully
>> insecure.
>
> +1! It seems like this dangerous practice will never end :-(
>
> Let me explain a personal experience. When I took over 2.6.32 many years
> ago, Greg asked me to adapt to the new maintenance process involving the
> patch reviews. At first I feared that it would increase my amount of work.
> And it did. But I also discovered how important these reviews were, because
> I started to get lots of "don't take this one in this version" and more
> importantly "if you merge this you'll need these ones as well". And very
> quickly I discovered how bogus the branches I used to maintain before
> had been, given the high feedback ratio!
>
> So based on this experience, I can assure anyone doing cherry-picks in
> their garage from LTS kernels that they're doing crap and that they must
> not distribute these kernels to anyone because THESE KERNELS ARE DANGEROUS.
> It's even very easy to introduce vulnerabilities by doing this!
Yes absolutely.
>
> The only set of fixes that can be trusted are the "official" stable
> kernels, because they are the only ones that are approved by the patches
> authors themselves.
Well, let us say that the authors had a chance to review the backports
being applied but given the volume maybe they did and silence means
agreement, or maybe they did not get a chance to review those changes.
Let us say that the trust level of the offical stable kernels is just
the highest of all kernels that are out there?
> Adding more stuff on top of stable kernels is fine
> (and done at your own risk), but randomly dropping stuff from stable
> kernels just because you don't think you need that is totally non-sense
> and must not be done anymore!
Yes, definitively not setting up for success.
--
Florian
On Thursday, February 18, 2021 4:33 PM, Willy Tarreau <[email protected]> wrote:
> Usually you pick an LTS kernel for a specific hardware. If it
> works that's great. But you cannot expect hardware to suddenly start to
> work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
> that's basically all and that's not their purpose.
It was the other way around. Fine working in-tree driver got
broken by backported "fixes". I did mention bit-rot.
In-tree iwlwifi worked half-ok on early 4.9.y stable. If
connection somehow de-autheticated (out of radio range or
whatever) it crashed the kernel spectacularly. Eventually that was
fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
kernels. On second half of year 2020 (don't remember exactly when)
iwlwifi started causing erratic behavior when some random process
terminated, as if some exit processing left some resources
un-freed or something weird like that. Upgraded to 4.19.y kernels
in hope to fix the issue. Nope, same problems continued there as
well. Replacing in-tree iwlwifi with out-of-tree upstream Intel
version solved the problem for me.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
On Thu, Feb 18, 2021 at 05:19:54PM +0000, Jari Ruusu wrote:
> In-tree iwlwifi worked half-ok on early 4.9.y stable. If
> connection somehow de-autheticated (out of radio range or
> whatever) it crashed the kernel spectacularly. Eventually that was
> fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
> kernels. On second half of year 2020 (don't remember exactly when)
> iwlwifi started causing erratic behavior when some random process
> terminated, as if some exit processing left some resources
> un-freed or something weird like that.
So you bisected the stable kernel, and reported the problem so the
problem commit could be dealt with?
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
On 2/18/2021 8:51 AM, Sasha Levin wrote:
> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>>> Following up on this as I did not hear back from you. Are you and/or
>>> your company willing to help out with the testing of 5.10 to ensure that
>>> it is a LTS kernel? So far I have not had any companies agree to help
>>> out with this effort, which is sad to see as it seems that companies
>>> want 6 years of stable kernels, yet do not seem to be able to at the
>>> least, do a test-build/run of those kernels, which is quite odd...
>> I personally cannot commit to supporting this kernel for 6 years
>> (and personally do not want to backport new features to a 6 year old
>> kernel).
>> And customers are finicky and ask for one thing and then change their
>> mind later.
>
> Why would we commit to maintining an upstream LTS for 6 years then? If
> no one ends up using it (and we don't want anyone using older LTS
> kernels) we're still stuck maintaining it.
>
>> We'll have to see what decisions are made at a company level for this
>> as there
>> are added costs to run tests on LTS kernel branches. We already run
>> extensive QA on
>
> This sounds very wrong: it's ok to get volunteers to commit to 6 years
> while the company that is asking for it won't do the same?
>
> Shouldn't Broadcom commit to the work involved here first?
There are different groups within Broadcom, and Scott and I belong to
different groups, so I can only speak for mine. We can talk about why I
do not use the same email domain as Scott if you would like, and no, I
cannot help you fix the Broadcom Wi-Fi drivers :)
My group is committed to using the 5.10 kernel for the next 6 years
because of Android TV and we will do our best to test the 5.10 stable RC
as they come. We are a small team of 7 people in the grand scheme of our
larger business activities, and we get pulled into way too many things,
so we may skip testing a few stable releases once in a while. I do not
know how to make it more official than that.
As a company, we are most likely shooting ourselves in the foot by not
having a point of coordination with the Linux Foundation and key people
like you, Greg and other participants in the stable kernel. The usual
left hand and right hand not having yet discovered each other, hey hi
there left hand! I can see about remedying that at least for the
interest of the group I work in.
--
Florian
On 2/17/2021 11:48 AM, Scott Branden wrote:
> Hi Greg,
>
> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>> On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
>>> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
>>>> Hi Greg,
>>>>
>>>>
>>>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
>>>>> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
>>>>>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
>>>>> Because they want to use all of the latest stuff that 5.10 provides
>>>>> them. Don't you want faster and more secure kernels for your devices?
>>>> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
>>>
>>> Great, use it, ship it to your customers and we are all happy. What do
>>> you need me for any of this? :)
>>>
>>>>>> And AOSP has already declared the use
>>>>>> of 5.10 kernel in their Android S and T releases.
>>>>> Publically? Where? And is that really the name of the new Android
>>>>> releases, I thought they switched to numbers now (hence the naming of
>>>>> the current android-common kernel branches, marketing is fun...)
>>>> https://source.android.com/devices/architecture/kernel/android-common
>>>> Feature and launch kernels provides kernels supported per version.
>>>
>>> Oh nice, didn't know that.
>>>
>>> But note, Android kernels do not reflect the lifespan of LTS kernels.
>>> If that were the case, I would still be supporting 3.18 as they are
>>> doing that at the moment for their devices and customers, and will be
>>> doing so for I think another full year.
>>>
>>> So while Android is nice to see here, remember that is what Google is
>>> promising to support for their users. You can do the same thing for
>>> your users, what do you need me here for this? You can do the same
>>> thing that Google is doing for 3.18 right now, pick the stable fixes
>>> from upstream, backport them, test them, and push them out to their
>>> users.
>>>
>>> While Google is a great help to me in the LTS effort, providing huge
>>> amounts of resources to enable my life easier with this (i.e. funding
>>> Linaro's testing efforts), their promise to their customers/users does
>>> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
>>> contracts are still in place and they know how to do this work
>>> themselves (as is proof with 3.18).
>>>
>>> So you can provide the same kind of guarantee to support any kernel
>>> version for any amount of time to any customer you like, it shouldn't
>>> require me to do that work for you, right?
>>>
>>>>>> Is there some way we could make the LTS support more clear.
>>>>>> A 2 year declaration is not LTS any more.
>>>>> Not true at all, a "normal" stable kernel is dropped after the next
>>>>> release happens, making their lifespan about 4 months long. 2 years is
>>>>> much longer than 4 months, so it still is a "long term supported" kernel
>>>>> in contrast, correct?
>>>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
>>>> The timeframes are very different.
>>>
>>> At this point in time, anyone wanting a kernel longer than 2 years
>>> should know how this all works.
>>>
>>> If not, please do some basic research, I have written whitepapers on
>>> this and given numerous talks. The information is out there...
>>>
>>>>>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
>>>>>> https://www.kernel.org/category/releases.html
>>>>> Why? What would that change?
>>>>>
>>>>> Ok, seriously, this happens every year, and every year we go through the
>>>>> same thing, it's not like this is somehow new, right?
>>>> No, but why do we need to keep playing the same game every year now.
>>>
>>> Because, 5.4 almost did not become "6 years" of support from me. That
>>> was because in the beginning, no one said they were going to use it in
>>> their devices and offer me help in testing and backporting. Only when I
>>> knew for sure that we had people helping this out did I change the date
>>> on kernel.org.
>>>
>>> So far the jury is still out for 5.10, are you willing to help with
>>> this? If not, why are you willing to hope that others are going to do
>>> your work for you? I am talking to some companies, but am not willing
>>> to commit to anything in public just yet, because no one has committed
>>> to me yet.
>>
>> Following up on this as I did not hear back from you. Are you and/or
>> your company willing to help out with the testing of 5.10 to ensure that
>> it is a LTS kernel? So far I have not had any companies agree to help
>> out with this effort, which is sad to see as it seems that companies
>> want 6 years of stable kernels, yet do not seem to be able to at the
>> least, do a test-build/run of those kernels, which is quite odd...
> I personally cannot commit to supporting this kernel for 6 years
> (and personally do not want to backport new features to a 6 year old kernel).
> And customers are finicky and ask for one thing and then change their mind later.
> We'll have to see what decisions are made at a company level for this as there
> are added costs to run tests on LTS kernel branches. We already run extensive QA on
> whatever active development branches are in use and a subset on the mainline
> branch as well. QA resources are finite and committing those for 6 years is
> not something that makes sense if customers drop that kernel version.
> Testing of the LTS kernel changes really moves out of our hands and into the
> customer's testing after our major releases to them.
There are many different things, and I am not sure I follow you on all
counts.
If your customers are asking you for a 6 year LTS kernel, does not that
mean they themselves plan to commit to that time frame?
Assuming you are delivering a downstream kernel with all that is needed
to run on the hardware you support plus all the extra fancy stuff your
customers pay you for, what more do the customers do on top? Do they
happen to merge stable updates with your downstream tree, and why do
they do that, should not you be their upstream?
>
> Other difficulty with the LTS version is the frequency it is updated. We would not
> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> is identified would be when we update and perform any meaningful testing when in maintainence.
Well you really have the best of both worlds here, you are not forced to
update your downstream fork of the stable kernel every week, though
bonus points if you do.
For instance I try to test all the stable release candidates for 4.9,
5.4 and 5.10, and merge the stable tags every week when they show up. We
do release to our customers every 6 weeks though, so usually they will
jump several stable releases, and they are fine with that.
Yes it takes time, but I would rather do that than have to continuously
respond to questions about is this CVE fixed in kernel X.Y.Z like it
used to be before we did that. It also helps catch problem faster before
customers do, the usual benefits of continuous integration/delivery.
--
Florian
On Thu, Feb 18, 2021 at 05:19:54PM +0000, Jari Ruusu wrote:
> On Thursday, February 18, 2021 4:33 PM, Willy Tarreau <[email protected]> wrote:
> > Usually you pick an LTS kernel for a specific hardware. If it
> > works that's great. But you cannot expect hardware to suddenly start to
> > work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
> > that's basically all and that's not their purpose.
>
> It was the other way around. Fine working in-tree driver got
> broken by backported "fixes". I did mention bit-rot.
It did? Please let us stable maintainers know about, we will always
gladly revert problems patches. What commits caused the problem?
> In-tree iwlwifi worked half-ok on early 4.9.y stable. If
> connection somehow de-autheticated (out of radio range or
> whatever) it crashed the kernel spectacularly. Eventually that was
> fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
> kernels. On second half of year 2020 (don't remember exactly when)
> iwlwifi started causing erratic behavior when some random process
> terminated, as if some exit processing left some resources
> un-freed or something weird like that. Upgraded to 4.19.y kernels
> in hope to fix the issue. Nope, same problems continued there as
> well. Replacing in-tree iwlwifi with out-of-tree upstream Intel
> version solved the problem for me.
So something in the 4.9.y and 4.14.y stable kernels caused a regression,
can you please do 'git bisect' to let us know what broke?
And if 4.19.0 was always broken, why didn't you report that as well?
How about 5.11, have you tried that? If not, please do so and report it
to the developers, otherwise how can it ever get fixed?
thanks,
greg k-h
On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> As a company, we are most likely shooting ourselves in the foot by not
> having a point of coordination with the Linux Foundation and key people
> like you, Greg and other participants in the stable kernel.
What does the LF have to do with this?
We are here, on the mailing lists, working with everyone. Just test the
-rc releases we make and let us know if they work or not for you, it's
not a lot of "coordination" needed at all.
Otherwise, if no one is saying that they are going to need these for 6
years and are willing to use it in their project (i.e. and test it),
there's no need for us to maintain it for that long, right?
thanks,
greg k-h
On 2/18/2021 9:53 AM, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>> As a company, we are most likely shooting ourselves in the foot by not
>> having a point of coordination with the Linux Foundation and key people
>> like you, Greg and other participants in the stable kernel.
>
> What does the LF have to do with this?
I did not know whether the commitment/plan to use a given LTS kernel had
to be funneled through the Linux Foundation and then down to you, but I
am happy that this can be done publicly via a mailing list instead.
>
> We are here, on the mailing lists, working with everyone. Just test the
> -rc releases we make and let us know if they work or not for you, it's
> not a lot of "coordination" needed at all.
>
> Otherwise, if no one is saying that they are going to need these for 6
> years and are willing to use it in their project (i.e. and test it),
> there's no need for us to maintain it for that long, right?
Right, that is straight forward, works for me!
--
Florian
Hi Florian!
On Thu, Feb 18, 2021 at 09:42:00AM -0800, Florian Fainelli wrote:
> > Other difficulty with the LTS version is the frequency it is updated. We would not
> > pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> > is identified would be when we update and perform any meaningful testing when in maintainence.
>
> Well you really have the best of both worlds here, you are not forced to
> update your downstream fork of the stable kernel every week, though
> bonus points if you do.
>
> For instance I try to test all the stable release candidates for 4.9,
> 5.4 and 5.10, and merge the stable tags every week when they show up. We
> do release to our customers every 6 weeks though, so usually they will
> jump several stable releases, and they are fine with that.
>
> Yes it takes time, but I would rather do that than have to continuously
> respond to questions about is this CVE fixed in kernel X.Y.Z like it
> used to be before we did that. It also helps catch problem faster before
> customers do, the usual benefits of continuous integration/delivery.
Totally agreed! In our use case at haproxy, it's slightly different but
not that much. We're much less sensitive to kernel bugs except for the
network parts and any long-term stability issue. However we're extremely
sensitive to openssl bugs and haproxy bugs. Thus we use them as triggers
to emit a new release and we systematically update the kernel to the
latest one. Our local patches usually apply pretty well on top of that.
We face maybe 1-3 rejects a year which take half an hour of extra manual
work, and roughly one regression every 3 years, essentially caused by
one of our local patches applying to the wrong place due to changes and
not caught by QA tests before being put online. I think that in ~15 years
(we started with 2.4), only a single customer was ever affected by a
regression caused by the kernel, it is so low we could almost laugh about
it. Quite frankly this is unrivaled, and it illustrates the huge benefit
in almost blindly following LTS this way! More quality, less work, and
faster response to critical bugs! For sure there's no "kernel hero" in
our dev team, but who really wants that anyway ?
Cheers,
Willy
On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> > As a company, we are most likely shooting ourselves in the foot by not
> > having a point of coordination with the Linux Foundation and key people
> > like you, Greg and other participants in the stable kernel.
>
> What does the LF have to do with this?
>
> We are here, on the mailing lists, working with everyone. Just test the
> -rc releases we make and let us know if they work or not for you, it's
> not a lot of "coordination" needed at all.
>
> Otherwise, if no one is saying that they are going to need these for 6
> years and are willing to use it in their project (i.e. and test it),
> there's no need for us to maintain it for that long, right?
Greg, please remember I expressed I really need them for slightly more than
3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
this saves me from having to take over these kernels after you, like in the
past, but I cannot engage on the regularity of my availability.
Overall I think that a lot of people completely underestimate the amount
of work it requires to maintain stable kernels, and how much it could be
distributed. By having a bunch of users participate a little bit more
(e.g. by sometimes backporting the patches that are essential to them,
by testing what's relevant to their use case), it already offloads a lot
of work. I don't think the extra work requires to be much organized if
there are enough participants to share the efforts.
Regards,
Willy
On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> > > As a company, we are most likely shooting ourselves in the foot by not
> > > having a point of coordination with the Linux Foundation and key people
> > > like you, Greg and other participants in the stable kernel.
> >
> > What does the LF have to do with this?
> >
> > We are here, on the mailing lists, working with everyone. Just test the
> > -rc releases we make and let us know if they work or not for you, it's
> > not a lot of "coordination" needed at all.
> >
> > Otherwise, if no one is saying that they are going to need these for 6
> > years and are willing to use it in their project (i.e. and test it),
> > there's no need for us to maintain it for that long, right?
>
> Greg, please remember I expressed I really need them for slightly more than
> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> this saves me from having to take over these kernels after you, like in the
> past, but I cannot engage on the regularity of my availability.
Ok, great!
That's one person/company saying they can help out (along with what CIP
has been stating.)
What about others? Broadcom started this conversation, odd that they
don't seem to want to help out :)
thanks,
greg k-h
On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>> having a point of coordination with the Linux Foundation and key people
>>>> like you, Greg and other participants in the stable kernel.
>>>
>>> What does the LF have to do with this?
>>>
>>> We are here, on the mailing lists, working with everyone. Just test the
>>> -rc releases we make and let us know if they work or not for you, it's
>>> not a lot of "coordination" needed at all.
>>>
>>> Otherwise, if no one is saying that they are going to need these for 6
>>> years and are willing to use it in their project (i.e. and test it),
>>> there's no need for us to maintain it for that long, right?
>>
>> Greg, please remember I expressed I really need them for slightly more than
>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>> this saves me from having to take over these kernels after you, like in the
>> past, but I cannot engage on the regularity of my availability.
>
> Ok, great!
>
> That's one person/company saying they can help out (along with what CIP
> has been stating.)
>
> What about others? Broadcom started this conversation, odd that they
> don't seem to want to help out :)
Greg, I'm sorry but I'm not in a position to provide such a commitment.
My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
>
> thanks,
>
> greg k-h
>
Hi!
> > For me
> > only way to get properly working WiFi on my laptop computer is to
> > compile that Intel out-of-tree version. Sad, but true.
>
> Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> is not the recommended thing to use the LTS kernels for.
Well, that's actually what distributions are doing, for example Debian
10.8 is on 4.19...
I expect -stable is what most users are running on their notebooks.
Best regards,
Pavel
--
http://www.livejournal.com/~pavelmachek
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>> having a point of coordination with the Linux Foundation and key people
> >>>> like you, Greg and other participants in the stable kernel.
> >>>
> >>> What does the LF have to do with this?
> >>>
> >>> We are here, on the mailing lists, working with everyone. Just test the
> >>> -rc releases we make and let us know if they work or not for you, it's
> >>> not a lot of "coordination" needed at all.
> >>>
> >>> Otherwise, if no one is saying that they are going to need these for 6
> >>> years and are willing to use it in their project (i.e. and test it),
> >>> there's no need for us to maintain it for that long, right?
> >>
> >> Greg, please remember I expressed I really need them for slightly more than
> >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >> this saves me from having to take over these kernels after you, like in the
> >> past, but I cannot engage on the regularity of my availability.
> >
> > Ok, great!
> >
> > That's one person/company saying they can help out (along with what CIP
> > has been stating.)
> >
> > What about others? Broadcom started this conversation, odd that they
> > don't seem to want to help out :)
> Greg, I'm sorry but I'm not in a position to provide such a commitment.
Are you at least in a position to defend that ? There are necessarily
some people in your company who understand the benefits of using open
source provdided for free by others and who understand that devoting
a few people's time to this task is extremely cheap compared to the
amount of work required by having to do it entirely yourself for a
lower quality.
> My original question arose because the 5.10 kernel is declared as 2 years LTS
> while older LTS kernels are now 6 years.
(...)
> If all LTS kernels were declared as 3.5-4 years as Willy commented this would
> solve a few issues. 6 year LTS kernels would only have a maximum 1 year
> lifespan over the latest declared LTS kernel. Also, many products take a year
> or more to develop, there isn't any life left in an LTS kernel if it is only
> 2 years.
We all have the same problem regarding this but how do you want Greg to
engage into such a task by himself if he's not certain he can count on
others to help ? The few of us having worked on extended kernels know
that there's a limit around 2.5 years beyond which backports become much
harder to perform and to test. Doing it every year would result in 6 LTS
kernels to maintain in addition to the last 1-2 stable ones. That becomes
a huge amount of work! I even think that having one LTS kernel every 2
years but maintained one extra year (e.g. 5 vs 4 in my case) would reduce
the effort.
> After 1-3 years of kernel age the relevant parties that want to invest and
> care about supporting specific kernel versions longer should become apparent
> and could commit to longer support.
But that's exactly what's currently being done. Greg initially commits
to 2 years hoping to get some help to pursue this longer, and this causes
trouble to some of us not being certain upfront whether or not we're choosing
the right kernel. So only the solution I'm seeing is for Greg to know
early who jumps in so that those of us without the power or skill to
entirely maintain a kernel by themselves know early which version to
choose. Quite frankly if we ship an LTS kernel in a product, the least
we can do is to give back a little bit to make sure the situation remains
durable.
As such even if you are not in a position to provide such a commitment,
I'd appreciate it if you would bring these arguments to those who are in
such a position, so that I don't end up as one of the too few ones having
to share a significant part of that task to make sure this valuable kernel
continues to exist.
Thanks,
Willy
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>> having a point of coordination with the Linux Foundation and key people
>>>>> like you, Greg and other participants in the stable kernel.
>>>>
>>>> What does the LF have to do with this?
>>>>
>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>> -rc releases we make and let us know if they work or not for you, it's
>>>> not a lot of "coordination" needed at all.
>>>>
>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>> years and are willing to use it in their project (i.e. and test it),
>>>> there's no need for us to maintain it for that long, right?
>>>
>>> Greg, please remember I expressed I really need them for slightly more than
>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>> this saves me from having to take over these kernels after you, like in the
>>> past, but I cannot engage on the regularity of my availability.
>>
>> Ok, great!
>>
>> That's one person/company saying they can help out (along with what CIP
>> has been stating.)
>>
>> What about others? Broadcom started this conversation, odd that they
>> don't seem to want to help out :)
>Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
>My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
>One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
So this sounds like you have boatloads of out-of-tree code and need a
stable kernel to avoid having to rebase that code. This is not why the
LTS trees are around.
For new projects, the easiest route is to upstream your stuff and ship
the latest kernel.
>If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
>6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
>Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
Products are supposed to upgrade their kernel. If you released something
with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
5.4 kernel for that product.
>After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
But this is exactly what's happening now: we support LTS kernels for two
years, and after that interested parties can figure it out on their own
if it's worth it for them to keep going.
--
Thanks,
Sasha
On 2/18/2021 1:39 PM, Sasha Levin wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by
>>>>>> not
>>>>>> having a point of coordination with the Linux Foundation and key
>>>>>> people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just
>>>>> test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly
>>>> more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time
>>>> permits if
>>>> this saves me from having to take over these kernels after you, like
>>>> in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>
>> My original question arose because the 5.10 kernel is declared as 2
>> years LTS while older LTS kernels are now 6 years.
>> One problem this has created is requests to provide silicon support in
>> an older kernel version (for a new project) rather than starting from
>> a newer kernel version that more properly supports the (silicon and
>> non-silicon) features.
>
> So this sounds like you have boatloads of out-of-tree code and need a
> stable kernel to avoid having to rebase that code. This is not why the
> LTS trees are around.
>
> For new projects, the easiest route is to upstream your stuff and ship
> the latest kernel.
Agreed, however no matter how good you are at upstreaming your changes,
there is always a delta that you keep accumulating, it is just the
reality of things be it because a specific piece of code was just harder
to upstream you missed multiple kernel versions, or it was not suitable,
or it changed, or anything. Not everything goes upstream even if that
gets closer and closer to zero if you do a good job at it.
>
>> If all LTS kernels were declared as 3.5-4 years as Willy commented
>> this would solve a few issues.
>> 6 year LTS kernels would only have a maximum 1 year lifespan over the
>> latest declared LTS kernel.
>> Also, many products take a year or more to develop, there isn't any
>> life left in an LTS kernel if it is only 2 years.
>
> Products are supposed to upgrade their kernel. If you released something
> with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
> 5.4 kernel for that product.
The are lots of Android devices manufacturers that launch using of the 3
kernel versions supported by the given Android release they targeted and
they do not update the kernel version ever after that.
>
>> After 1-3 years of kernel age the relevant parties that want to invest
>> and care about supporting specific kernel versions longer should
>> become apparent and could commit to longer support. Perhaps you move
>> the burden of 6 years LTS elsewhere to longer term projects. But, I'm
>> sure many are happy because you continue doing such a great job in a
>> central location, especially those whose product lifespan is around 6
>> years.
>
> But this is exactly what's happening now: we support LTS kernels for two
> years, and after that interested parties can figure it out on their own
> if it's worth it for them to keep going.
>
--
Florian
On 2021-02-18 1:39 p.m., Sasha Levin wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>
>> My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
>> One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
>
> So this sounds like you have boatloads of out-of-tree code and need a
> stable kernel to avoid having to rebase that code. This is not why the
> LTS trees are around.
My question centered around the inconsistency of a 2 year LTS kernel and some of the problems
of having older LTS kernels declared as 6 years.
The discussion has gone sideways from there with many other assumptions.
There is not always control over what kernel version a customer uses,
how many hacks they have in it, when they upgrade it, what other vendor changes are in it, etc.
But, will keep this email chain to forward to others. It may help influence some decisions.
>
> For new projects, the easiest route is to upstream your stuff and ship
> the latest kernel.
>
>> If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
>> 6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
>> Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
>
> Products are supposed to upgrade their kernel. If you released something
> with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
> 5.4 kernel for that product.
>
>> After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
>
> But this is exactly what's happening now: we support LTS kernels for two
> years, and after that interested parties can figure it out on their own
> if it's worth it for them to keep going.
>
Hi Willy,
On 2021-02-18 1:00 p.m., Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
> Are you at least in a position to defend that ? There are necessarily
> some people in your company who understand the benefits of using open
> source provdided for free by others and who understand that devoting
> a few people's time to this task is extremely cheap compared to the
> amount of work required by having to do it entirely yourself for a
> lower quality.
>
>> My original question arose because the 5.10 kernel is declared as 2 years LTS
>> while older LTS kernels are now 6 years.
> (...)
>> If all LTS kernels were declared as 3.5-4 years as Willy commented this would
>> solve a few issues. 6 year LTS kernels would only have a maximum 1 year
>> lifespan over the latest declared LTS kernel. Also, many products take a year
>> or more to develop, there isn't any life left in an LTS kernel if it is only
>> 2 years.
>
> We all have the same problem regarding this but how do you want Greg to
> engage into such a task by himself if he's not certain he can count on
> others to help ? The few of us having worked on extended kernels know
> that there's a limit around 2.5 years beyond which backports become much
> harder to perform and to test. Doing it every year would result in 6 LTS
> kernels to maintain in addition to the last 1-2 stable ones. That becomes
> a huge amount of work! I even think that having one LTS kernel every 2
> years but maintained one extra year (e.g. 5 vs 4 in my case) would reduce
> the effort.
>
>> After 1-3 years of kernel age the relevant parties that want to invest and
>> care about supporting specific kernel versions longer should become apparent
>> and could commit to longer support.
>
> But that's exactly what's currently being done. Greg initially commits
> to 2 years hoping to get some help to pursue this longer, and this causes
> trouble to some of us not being certain upfront whether or not we're choosing
> the right kernel. So only the solution I'm seeing is for Greg to know
> early who jumps in so that those of us without the power or skill to
> entirely maintain a kernel by themselves know early which version to
> choose. Quite frankly if we ship an LTS kernel in a product, the least
> we can do is to give back a little bit to make sure the situation remains
> durable.
>
> As such even if you are not in a position to provide such a commitment,
> I'd appreciate it if you would bring these arguments to those who are in
> such a position, so that I don't end up as one of the too few ones having
> to share a significant part of that task to make sure this valuable kernel
> continues to exist.
Thanks - will forward such info as necessary.
>
> Thanks,
> Willy
>
On Thursday 18 February 2021 21:55:34 Pavel Machek wrote:
> Hi!
>
> > > For me
> > > only way to get properly working WiFi on my laptop computer is to
> > > compile that Intel out-of-tree version. Sad, but true.
> >
> > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > is not the recommended thing to use the LTS kernels for.
>
> Well, that's actually what distributions are doing, for example Debian
> 10.8 is on 4.19...
There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
> I expect -stable is what most users are running on their notebooks.
>
> Best regards,
> Pavel
--
Ondrej Zary
On Thursday, February 18, 2021 7:44 PM, Greg Kroah-Hartman <[email protected]> wrote:
> > It was the other way around. Fine working in-tree driver got
> > broken by backported "fixes". I did mention bit-rot.
>
> It did? Please let us stable maintainers know about, we will always
> gladly revert problems patches. What commits caused the problem?
I don't have a list of commits for you. It took me long time to
figure out that it was iwlwifi that was causing those problems.
In-tree iwlwifi on 4.19.y kernels needs professional quality
locking audit and backporting of necessary fixes from upstream
Intel out-of-tree version.
> So something in the 4.9.y and 4.14.y stable kernels caused a regression,
> can you please do 'git bisect' to let us know what broke?
My ability to do WiFi tests on that laptop computer are limited
for now. Earlier that laptop's connectivity to world was through
mobile WiFi router. That mobile WiFi router no longer has a
SIM-card, so no connectivity through that anymore. That laptop's
connectivity to world is now through wired ethernet to fiber.
It was actually this switch to ethernet/fiber that made me realize
the brokenness on in-tree iwlwifi on 4.19.y kernels. When in-tree
WiFi was not used, those problems never triggered. Switched back
to in-tree WiFi, and problems came back. Switched to out-of-tree
Intel version of iwlwifi, problems went away again. Then I looked
at the fixes in out-of-tree Intel version of iwlwifi that were
missing in in-tree version, and I understood why that was so.
Those stability issues on in-tree iwlwifi on 4.19.y kernels are
difficult to trigger. Sometimes it may take days to trigger it
once. Sometimes I was unlucky enough to trigger them more than
once a day. I say that two weeks of operation without issues are
needed to pronounce those issues gone.
Currently, special arrangements are needed for me to test WiFi at
all on that laptop computer, and those arrangements are something
I am not willing to commit for multi-week testing run. Sorry.
> And if 4.19.0 was always broken, why didn't you report that as well?
I didn't test early 4.19.y kernels.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Hi!
> > > > For me
> > > > only way to get properly working WiFi on my laptop computer is to
> > > > compile that Intel out-of-tree version. Sad, but true.
> > >
> > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > > is not the recommended thing to use the LTS kernels for.
> >
> > Well, that's actually what distributions are doing, for example Debian
> > 10.8 is on 4.19...
>
> There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
>
I can compile my own kernel, too. But if you go up the thread, it is
about iwlwifi becoming broken in 4.19, and Greg saying it is wrong
to put -stable on laptop. And -stable on laptop is norm, not the
exception.
Pavel
--
http://www.livejournal.com/~pavelmachek
On Fri, Feb 19, 2021 at 07:10:35AM +0000, Jari Ruusu wrote:
> On Thursday, February 18, 2021 7:44 PM, Greg Kroah-Hartman <[email protected]> wrote:
> > > It was the other way around. Fine working in-tree driver got
> > > broken by backported "fixes". I did mention bit-rot.
> >
> > It did? Please let us stable maintainers know about, we will always
> > gladly revert problems patches. What commits caused the problem?
>
> I don't have a list of commits for you. It took me long time to
> figure out that it was iwlwifi that was causing those problems.
>
> In-tree iwlwifi on 4.19.y kernels needs professional quality
> locking audit and backporting of necessary fixes from upstream
> Intel out-of-tree version.
That's not the goal of stable kernel releases/trees. If the driver
version that is in 4.19.y does not work for you on release 4.19.0, odds
of that "changing" in later stable releases is slim to none.
Especially without any specific bug reports or emails to the developers
to tell them what is broken and not working for you.
If however, you wish to stick with an out-of-tree driver, wonderful,
that's your choice. But don't claim that somehow the stable kernel
process is broken because of that, as it has nothing to do with this
type of thing.
Best of luck!
greg k-h
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>> having a point of coordination with the Linux Foundation and key people
> >>>> like you, Greg and other participants in the stable kernel.
> >>>
> >>> What does the LF have to do with this?
> >>>
> >>> We are here, on the mailing lists, working with everyone. Just test the
> >>> -rc releases we make and let us know if they work or not for you, it's
> >>> not a lot of "coordination" needed at all.
> >>>
> >>> Otherwise, if no one is saying that they are going to need these for 6
> >>> years and are willing to use it in their project (i.e. and test it),
> >>> there's no need for us to maintain it for that long, right?
> >>
> >> Greg, please remember I expressed I really need them for slightly more than
> >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >> this saves me from having to take over these kernels after you, like in the
> >> past, but I cannot engage on the regularity of my availability.
> >
> > Ok, great!
> >
> > That's one person/company saying they can help out (along with what CIP
> > has been stating.)
> >
> > What about others? Broadcom started this conversation, odd that they
> > don't seem to want to help out :)
> Greg, I'm sorry but I'm not in a position to provide such a commitment.
Ok, who at Broadcom do I need to talk to to get that type of commitment?
> My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
> One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
Sounds like your development model is broken, again, who do I need to
talk to in order to help you all fix this?
thanks,
greg k-h
On Fri, Feb 19, 2021 at 09:00:27AM +0100, Pavel Machek wrote:
> Hi!
>
> > > > > For me
> > > > > only way to get properly working WiFi on my laptop computer is to
> > > > > compile that Intel out-of-tree version. Sad, but true.
> > > >
> > > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > > > is not the recommended thing to use the LTS kernels for.
> > >
> > > Well, that's actually what distributions are doing, for example Debian
> > > 10.8 is on 4.19...
> >
> > There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
> >
>
> I can compile my own kernel, too. But if you go up the thread, it is
> about iwlwifi becoming broken in 4.19, and Greg saying it is wrong
> to put -stable on laptop. And -stable on laptop is norm, not the
> exception.
If distros want to "camp out" on an old kernel version, that's fine, as
I describe in:
http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/
many years ago.
But for someone using a device where they want to use "new" hardware,
that was made _after_ the kernel version was released, that's just
someone who needs to pick a better distro :)
thanks,
greg k-h
Hi Greg,
On 2021/1/26 15:29, Greg Kroah-Hartman wrote:
[...]
>
> I want to see companies _using_ the kernel, and most importantly,
> _updating_ their devices with it, to know if it is worth to keep around
> for longer than 2 years. I also, hopefully, want to see how those
> companies will help me out in the testing and maintenance of that kernel
> version in order to make supporting it for 6 years actually possible.
>
> So, are you planning on using 5.10? Will you will be willing to help
> out in testing the -rc releases I make to let me know if there are any
> problems, and to help in pointing out and backporting any specific
> patches that your platforms need for that kernel release?
We(Huawei) are willing to commit resources to help out in testing the
stable -rc releases, and to help to backport patches for stable kernels.
5.10 stable kernel will be used for openEuler [1] kernel and also inside
Huawei. From customer's feedback, it's very important to see the stable
kernel we used to be maintained for 6 years in the community, and we
will use 5.10 kernel for at least 6 years, so we are willing to help
you and help ourselves :)
In specific, we will start from the testing work, using HULK robot
(reports lots of bugs to mainline kernel) testing framework to test
compile, reboot, functional testing, and will extend to basic
performance regression testing in the future.
And we will start from ARM64 and X86 architecture first, and then extend
to other platforms.
For patch backporting, will send the bugfix patches (from mainline)
we spotted, but I think this work may not doing in regular but will
be triggered as needed.
Does this sound good to you?
Thanks
Hanjun
[1]: https://openeuler.org/en/
On Fri, Feb 19, 2021 at 04:54:24PM +0800, Hanjun Guo wrote:
> Hi Greg,
>
> On 2021/1/26 15:29, Greg Kroah-Hartman wrote:
> [...]
> >
> > I want to see companies _using_ the kernel, and most importantly,
> > _updating_ their devices with it, to know if it is worth to keep around
> > for longer than 2 years. I also, hopefully, want to see how those
> > companies will help me out in the testing and maintenance of that kernel
> > version in order to make supporting it for 6 years actually possible.
> >
> > So, are you planning on using 5.10? Will you will be willing to help
> > out in testing the -rc releases I make to let me know if there are any
> > problems, and to help in pointing out and backporting any specific
> > patches that your platforms need for that kernel release?
>
> We(Huawei) are willing to commit resources to help out in testing the
> stable -rc releases, and to help to backport patches for stable kernels.
Wonderful!
> 5.10 stable kernel will be used for openEuler [1] kernel and also inside
> Huawei. From customer's feedback, it's very important to see the stable
> kernel we used to be maintained for 6 years in the community, and we
> will use 5.10 kernel for at least 6 years, so we are willing to help
> you and help ourselves :)
>
> In specific, we will start from the testing work, using HULK robot
> (reports lots of bugs to mainline kernel) testing framework to test
> compile, reboot, functional testing, and will extend to basic
> performance regression testing in the future.
Great! Do you all need an email notification when the -rc releases come
out for the stable trees, or can you trigger off of the -rc stable git
tree? Different CI systems work in different ways :)
And if you can reply to the -rc release emails with a "Tested-by:" tag,
I will be glad to add that to the release commit when that happens to
show that you all have tested the release.
> And we will start from ARM64 and X86 architecture first, and then extend
> to other platforms.
That's a good start, the useful ones :)
> For patch backporting, will send the bugfix patches (from mainline)
> we spotted, but I think this work may not doing in regular but will
> be triggered as needed.
That's fine, it is not something that happens at a regular interval.
> Does this sound good to you?
Yes it does, thank you so much.
greg k-h
On Friday, February 19, 2021 10:22 AM, Greg Kroah-Hartman <[email protected]> wrote:
> That's not the goal of stable kernel releases/trees. If the driver
> version that is in 4.19.y does not work for you on release 4.19.0, odds
> of that "changing" in later stable releases is slim to none.
But in-tree iwlwifi worked fine on 4.9.y and 4.14.y , and then got broken
in later 4.14.y versions. I tried later versions of 4.19.y in an attempt to
fix the breakage. Basically you are are saying that I should have NOT
attempted to upgrage 4.9.y -> 4.14.y -> 4.19.y
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
On Fri, Feb 19, 2021 at 10:31:49AM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 10:22 AM, Greg Kroah-Hartman <[email protected]> wrote:
> > That's not the goal of stable kernel releases/trees. If the driver
> > version that is in 4.19.y does not work for you on release 4.19.0, odds
> > of that "changing" in later stable releases is slim to none.
>
> But in-tree iwlwifi worked fine on 4.9.y and 4.14.y , and then got broken
> in later 4.14.y versions.
Did you report that breakage to us and the developers of the driver?
Sounds like a regression that people would love to hear about and get
fixed.
> I tried later versions of 4.19.y in an attempt to
> fix the breakage. Basically you are are saying that I should have NOT
> attempted to upgrage 4.9.y -> 4.14.y -> 4.19.y
Without anyone reporting a problem, how can anyone know to fix it?
thanks,
greg k-h
On Friday, February 19, 2021 12:37 PM, Greg Kroah-Hartman <[email protected]> wrote:
> Did you report that breakage to us and the developers of the driver?
> Sounds like a regression that people would love to hear about and get
> fixed.
At that time I didn't know where the problem was, so I did not report it.
I only recently connected-the-dots that problem is in-tree iwlwifi, so I
am reporting it now.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
On Fri, Feb 19, 2021 at 10:57:30AM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 12:37 PM, Greg Kroah-Hartman <[email protected]> wrote:
> > Did you report that breakage to us and the developers of the driver?
> > Sounds like a regression that people would love to hear about and get
> > fixed.
>
> At that time I didn't know where the problem was, so I did not report it.
> I only recently connected-the-dots that problem is in-tree iwlwifi, so I
> am reporting it now.
Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
offending change?
thanks,
greg k-h
Hi Hanjun,
On 2/19/21 10:54 AM, Hanjun Guo wrote:
> In specific, we will start from the testing work, using HULK robot
> (reports lots of bugs to mainline kernel) testing framework to test
> compile, reboot, functional testing, and will extend to basic
> performance regression testing in the future.
I heard about Huawei ramping up kernel testing from someone at FOSDEM
2019. I wonder if it was you :) Nice to see your progress and the company
stepping up to help with testing!
Would you be interested in working with the Linux Foundation KernelCI project
on submitting your build and test results to the common database - KCIDB?
We are working on aggregating results from various testing systems so we can
provide a dashboard, and a single, aggregated e-mail report to subscribed
maintainers and developers.
We have a prototype dashboard at https://staging.kernelci.org:3000/ and are
working hard on making the e-mail reports good enough to start reaching out to
maintainers.
We already have ARM, Google Syzbot, Gentoo GKernelCI, Red Hat CKI, and, of
course, KernelCI native tests sending data to the database. Linaro Tuxsuite is
starting sending today. We could use your data, and of course any development
help you could spare :)
I wish I could show you my today's KCIDB presentation at DevConf.cz, but the
recording is not out yet. Meanwhile you can take a look at our presentation at
last year's Linux Plumbers: https://youtu.be/y9Glc90WUN0?t=10739
Or see our intro in an older blog post:
https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
Anyone wishing to contribute to KCIDB gets credentials and permissions to
submit to our "playground" setup where they can send their data, see it in a
dashboard, experiment without worrying about breaking anything, and decide if
they like it or not.
If you're interested, take a look at our Submission HOWTO:
https://github.com/kernelci/kcidb/blob/v8/SUBMISSION_HOWTO.md
and send an email to [email protected] (CC'd), or come over to the #kernelci
channel on freenode.net!
Nick
On 2/19/21 10:54 AM, Hanjun Guo wrote:
> Hi Greg,
>
> On 2021/1/26 15:29, Greg Kroah-Hartman wrote:
> [...]
>>
>> I want to see companies _using_ the kernel, and most importantly,
>> _updating_ their devices with it, to know if it is worth to keep around
>> for longer than 2 years. I also, hopefully, want to see how those
>> companies will help me out in the testing and maintenance of that kernel
>> version in order to make supporting it for 6 years actually possible.
>>
>> So, are you planning on using 5.10? Will you will be willing to help
>> out in testing the -rc releases I make to let me know if there are any
>> problems, and to help in pointing out and backporting any specific
>> patches that your platforms need for that kernel release?
>
> We(Huawei) are willing to commit resources to help out in testing the
> stable -rc releases, and to help to backport patches for stable kernels.
>
> 5.10 stable kernel will be used for openEuler [1] kernel and also inside
> Huawei. From customer's feedback, it's very important to see the stable
> kernel we used to be maintained for 6 years in the community, and we
> will use 5.10 kernel for at least 6 years, so we are willing to help
> you and help ourselves :)
>
> In specific, we will start from the testing work, using HULK robot
> (reports lots of bugs to mainline kernel) testing framework to test
> compile, reboot, functional testing, and will extend to basic
> performance regression testing in the future.
>
> And we will start from ARM64 and X86 architecture first, and then extend
> to other platforms.
>
> For patch backporting, will send the bugfix patches (from mainline)
> we spotted, but I think this work may not doing in regular but will
> be triggered as needed.
>
> Does this sound good to you?
>
> Thanks
> Hanjun
>
> [1]: https://openeuler.org/en/
>
> _______________________________________________
> linux-arm-kernel mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
> Ok, who at Broadcom do I need to talk to to get that type of commitment?
I am not sure if I was too subtle before, we (Broadcom) cannot give you
an unified voice to speak with because we are divided in silos/business
units that make their independent decisions.
The group I work in (STB/CM, different from Scott's) is committed to
using the 5.10 kernel for 6 years and that is a decision that has been
taken.
I could give you names of other decision makers in other business units
I know who also deliver Linux for their respective business units
however some of them may not make public appearances on mailing lists,
let alone care about upstreaming their changes so I do not know whether
a 6 years 5.10 kernel is even something they remotely entertain.
Hope this helps.
--
Florian
On Friday, February 19, 2021 1:16 PM, Greg Kroah-Hartman <[email protected]> wrote:
> Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
> offending change?
As I tried to explain earlier, that laptop's connection to world is no
longer iwlwifi based but ethernet connection to fibre. I can't do much
bisecting when iwlwifi is not being used at all. Sorry. My contribution
here is trying to point you guys to right direction.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
>
>
> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> >> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>>>> having a point of coordination with the Linux Foundation and key people
> >>>>>> like you, Greg and other participants in the stable kernel.
> >>>>>
> >>>>> What does the LF have to do with this?
> >>>>>
> >>>>> We are here, on the mailing lists, working with everyone. Just test the
> >>>>> -rc releases we make and let us know if they work or not for you, it's
> >>>>> not a lot of "coordination" needed at all.
> >>>>>
> >>>>> Otherwise, if no one is saying that they are going to need these for 6
> >>>>> years and are willing to use it in their project (i.e. and test it),
> >>>>> there's no need for us to maintain it for that long, right?
> >>>>
> >>>> Greg, please remember I expressed I really need them for slightly more than
> >>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >>>> this saves me from having to take over these kernels after you, like in the
> >>>> past, but I cannot engage on the regularity of my availability.
> >>>
> >>> Ok, great!
> >>>
> >>> That's one person/company saying they can help out (along with what CIP
> >>> has been stating.)
> >>>
> >>> What about others? Broadcom started this conversation, odd that they
> >>> don't seem to want to help out :)
> >> Greg, I'm sorry but I'm not in a position to provide such a commitment.
> >
> > Ok, who at Broadcom do I need to talk to to get that type of commitment?
>
> I am not sure if I was too subtle before, we (Broadcom) cannot give you
> an unified voice to speak with because we are divided in silos/business
> units that make their independent decisions.
That's fine, I'm totally used to that, large (and even small) companies
always have different groups with different roadmaps and policies.
> The group I work in (STB/CM, different from Scott's) is committed to
> using the 5.10 kernel for 6 years and that is a decision that has been
> taken.
Great! Will you all be testing the -rc releases and letting me know how
they work for your systems?
> I could give you names of other decision makers in other business units
> I know who also deliver Linux for their respective business units
> however some of them may not make public appearances on mailing lists,
> let alone care about upstreaming their changes so I do not know whether
> a 6 years 5.10 kernel is even something they remotely entertain.
That's fine, I'm not expecting emails from the list, we can take this
off-list if you like as it sounds like I need to talk to some different
managers there, right? :)
thanks,
greg k-h
On Fri, Feb 19, 2021 at 12:16:12PM +0100, Greg Kroah-Hartman wrote:
>
> Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
> offending change?
To be fair, especially with WiFi bugs, you may need to run for hours
or days before you are absolutely sure that a particular bisection
point will result in the the locking/kernel crash bug to manifest
itself. Worse, you have to be actively using the Wifi in order to see
the problem, and in some cases, it only triggers when you switch
between AP's, so you have to be actively using it in the work office,
and taking it between conference rooms, only to see your machine crash
taking your unsaved work, email drafts, etc. with it.
That being said, users should at least report the bug. (That's what I
did, when I ran into this a bunch of years ago, with an explanation of
"I'm trying to do a bisect, but it may take a few weeks for me to
figure out what the !@#!? is going on. In my case, I was trying to
use upstream -rcX kernels to dogfood on my work laptop, but the
principle is the same.)
Ultiumately, I solved the problem, by switching laptops to one that
didn't use an NVidia GPU (which sometimes forced me to stay 1-2
upstream versions behind, making life even more difficult when
debugging these issues, until the out-of-tree video driver got updated
to work with newer upstream), and which also had WiFi hardware which
was less subject to these issues.
It's unfortunate, but not all hardware is as well supported on Linux.
And in my case, because $WORK was using Enterprise WiFi systems with
AP's that don't get as much testing by developers, very few other
people could repro the bugs. That's life, and sometimes the only
solution is to switch hardware. And/or you just use a Chromebook in
those sorts of situations, separating your work/enterprise and
upstream development hardware, and be done with it. :-)
Cheers,
- Ted
On 2/19/2021 7:53 AM, Greg Kroah-Hartman wrote:
> On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
>>
>>
>> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>>>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>>>
>>>>>>> What does the LF have to do with this?
>>>>>>>
>>>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>>>> not a lot of "coordination" needed at all.
>>>>>>>
>>>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>>>> there's no need for us to maintain it for that long, right?
>>>>>>
>>>>>> Greg, please remember I expressed I really need them for slightly more than
>>>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>>>> this saves me from having to take over these kernels after you, like in the
>>>>>> past, but I cannot engage on the regularity of my availability.
>>>>>
>>>>> Ok, great!
>>>>>
>>>>> That's one person/company saying they can help out (along with what CIP
>>>>> has been stating.)
>>>>>
>>>>> What about others? Broadcom started this conversation, odd that they
>>>>> don't seem to want to help out :)
>>>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>>
>>> Ok, who at Broadcom do I need to talk to to get that type of commitment?
>>
>> I am not sure if I was too subtle before, we (Broadcom) cannot give you
>> an unified voice to speak with because we are divided in silos/business
>> units that make their independent decisions.
>
> That's fine, I'm totally used to that, large (and even small) companies
> always have different groups with different roadmaps and policies.
>
>> The group I work in (STB/CM, different from Scott's) is committed to
>> using the 5.10 kernel for 6 years and that is a decision that has been
>> taken.
>
> Great! Will you all be testing the -rc releases and letting me know how
> they work for your systems?
Yes I will, can you add me to your CC list for the stable candidates?
That helps me not having to dig for those announcements specifically.
>
>> I could give you names of other decision makers in other business units
>> I know who also deliver Linux for their respective business units
>> however some of them may not make public appearances on mailing lists,
>> let alone care about upstreaming their changes so I do not know whether
>> a 6 years 5.10 kernel is even something they remotely entertain.
>
> That's fine, I'm not expecting emails from the list, we can take this
> off-list if you like as it sounds like I need to talk to some different
> managers there, right? :)
I will put together a list and send you an email off list.
--
Florian
On 2021/2/19 17:08, Greg Kroah-Hartman wrote:
> On Fri, Feb 19, 2021 at 04:54:24PM +0800, Hanjun Guo wrote:
>> Hi Greg,
>>
>> On 2021/1/26 15:29, Greg Kroah-Hartman wrote:
>> [...]
>>>
>>> I want to see companies _using_ the kernel, and most importantly,
>>> _updating_ their devices with it, to know if it is worth to keep around
>>> for longer than 2 years. I also, hopefully, want to see how those
>>> companies will help me out in the testing and maintenance of that kernel
>>> version in order to make supporting it for 6 years actually possible.
>>>
>>> So, are you planning on using 5.10? Will you will be willing to help
>>> out in testing the -rc releases I make to let me know if there are any
>>> problems, and to help in pointing out and backporting any specific
>>> patches that your platforms need for that kernel release?
>>
>> We(Huawei) are willing to commit resources to help out in testing the
>> stable -rc releases, and to help to backport patches for stable kernels.
>
> Wonderful!
>
>> 5.10 stable kernel will be used for openEuler [1] kernel and also inside
>> Huawei. From customer's feedback, it's very important to see the stable
>> kernel we used to be maintained for 6 years in the community, and we
>> will use 5.10 kernel for at least 6 years, so we are willing to help
>> you and help ourselves :)
>>
>> In specific, we will start from the testing work, using HULK robot
>> (reports lots of bugs to mainline kernel) testing framework to test
>> compile, reboot, functional testing, and will extend to basic
>> performance regression testing in the future.
>
> Great! Do you all need an email notification when the -rc releases come
> out for the stable trees, or can you trigger off of the -rc stable git
> tree? Different CI systems work in different ways :)
We can trigger the test when you updated the -rc stable git tree,
by monitoring new commits for the stable branches. So if you push
all the commits at once for -rc stable branches, then our CI system
can work well.
>
> And if you can reply to the -rc release emails with a "Tested-by:" tag,
> I will be glad to add that to the release commit when that happens to
> show that you all have tested the release.
Thanks, will reply "Tested-by:" with -rc releases. We are working on
setting up the test farm and will report the test results in a week.
>
>> And we will start from ARM64 and X86 architecture first, and then extend
>> to other platforms.
>
> That's a good start, the useful ones :)
>
>> For patch backporting, will send the bugfix patches (from mainline)
>> we spotted, but I think this work may not doing in regular but will
>> be triggered as needed.
>
> That's fine, it is not something that happens at a regular interval.
>
>> Does this sound good to you?
>
> Yes it does, thank you so much.
>
> greg k-h
Thanks
Hanjun
On Sat, Feb 20, 2021 at 03:02:54PM +0800, Hanjun Guo wrote:
> On 2021/2/19 17:08, Greg Kroah-Hartman wrote:
> > On Fri, Feb 19, 2021 at 04:54:24PM +0800, Hanjun Guo wrote:
> > > Hi Greg,
> > >
> > > On 2021/1/26 15:29, Greg Kroah-Hartman wrote:
> > > [...]
> > > >
> > > > I want to see companies _using_ the kernel, and most importantly,
> > > > _updating_ their devices with it, to know if it is worth to keep around
> > > > for longer than 2 years. I also, hopefully, want to see how those
> > > > companies will help me out in the testing and maintenance of that kernel
> > > > version in order to make supporting it for 6 years actually possible.
> > > >
> > > > So, are you planning on using 5.10? Will you will be willing to help
> > > > out in testing the -rc releases I make to let me know if there are any
> > > > problems, and to help in pointing out and backporting any specific
> > > > patches that your platforms need for that kernel release?
> > >
> > > We(Huawei) are willing to commit resources to help out in testing the
> > > stable -rc releases, and to help to backport patches for stable kernels.
> >
> > Wonderful!
> >
> > > 5.10 stable kernel will be used for openEuler [1] kernel and also inside
> > > Huawei. From customer's feedback, it's very important to see the stable
> > > kernel we used to be maintained for 6 years in the community, and we
> > > will use 5.10 kernel for at least 6 years, so we are willing to help
> > > you and help ourselves :)
> > >
> > > In specific, we will start from the testing work, using HULK robot
> > > (reports lots of bugs to mainline kernel) testing framework to test
> > > compile, reboot, functional testing, and will extend to basic
> > > performance regression testing in the future.
> >
> > Great! Do you all need an email notification when the -rc releases come
> > out for the stable trees, or can you trigger off of the -rc stable git
> > tree? Different CI systems work in different ways :)
>
> We can trigger the test when you updated the -rc stable git tree,
> by monitoring new commits for the stable branches. So if you push
> all the commits at once for -rc stable branches, then our CI system
> can work well.
I do push to the -rc branches, but those branches are rebased, and I do
"intermediate" pushes as well. Meaning I push to have CI systems run on
the existing patch queue at times that are not only the "main" -rc
release periods.
Watch the branches for a few weeks to get an idea of how they work if
you are curious.
> > And if you can reply to the -rc release emails with a "Tested-by:" tag,
> > I will be glad to add that to the release commit when that happens to
> > show that you all have tested the release.
>
> Thanks, will reply "Tested-by:" with -rc releases. We are working on
> setting up the test farm and will report the test results in a week.
Great, thanks!
greg k-h
On Friday, February 19, 2021 5:23 PM, Jari Ruusu <[email protected]> wrote:
> My contribution here is trying to point you guys to right direction.
I have been able to narrow the beginning of the problem to these kernels:
4.14.188 ... 4.14.202
Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 5:23 PM, Jari Ruusu <[email protected]> wrote:
> > My contribution here is trying to point you guys to right direction.
>
> I have been able to narrow the beginning of the problem to these kernels:
> 4.14.188 ... 4.14.202
>
> Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
Great, any chance you can narrow this down to the commit itself?
thanks,
greg k-h
On Sat, Feb 20, 2021 at 05:05:14PM +0100, Greg Kroah-Hartman wrote:
> On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> > On Friday, February 19, 2021 5:23 PM, Jari Ruusu <[email protected]> wrote:
> > > My contribution here is trying to point you guys to right direction.
> >
> > I have been able to narrow the beginning of the problem to these kernels:
> > 4.14.188 ... 4.14.202
> >
> > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
>
> Great, any chance you can narrow this down to the commit itself?
What is strnage is that there was no iwlwifi driver change in this
interval. Only iwlegacy (don't know if this one was used instead).
Note, my laptop uses iwlwifi as well. I've met random issues with it
a year ago when I started to use wifi, such as wifi randomly freezing
during audio meetings, and automatically fixing itself after 1 minute.
I also found that a down-up cycle on the interface would fix it. It
happened less than once a day so it was not easy to diagnose, and given
how crappy all wifi chips are, I naturally attributed this to the
hardware. But since I upgraded to 5.4 months ago, I don't remember having
met that issue anymore, so it was likely related to the driver. All I can
say is that 4.19.68 was exhibiting this issue. I can't say for older ones
because I didn't use wifi before.
Just my two cents,
Willy
On Saturday, February 20, 2021 6:05 PM, Greg Kroah-Hartman <[email protected]> wrote:
> On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> > I have been able to narrow the beginning of the problem to these kernels:
> > 4.14.188 ... 4.14.202
> > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
>
> Great, any chance you can narrow this down to the commit itself?
I am not able to test WiFi on that laptop computer anymore,
because that laptop now connects to world using wired connection.
It was that WiFI->wired connection change that led me to realize
the buggyness of in-tree iwlwifi in 4.19.y kernels.
I did that narrowing to specific kernel versions by digging my
archived backup files and their notes. No tests were run.
Problems started triggering in those kernels that I mentioned in
earlier email. That does not mean that the bugs were not already
there in kernels older that those. Maybe some change just widened
"window of opportunity" enough for me to see the issues.
In-tree iwlwifi in 4.19.y is missing locking fixes. No amount of
smooth-talking is going to change that.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Hi Greg,
On 26.01.2021 [08:29:25 +0100], Greg Kroah-Hartman wrote:
> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> > Hi All,
> >
> > The 5.10 LTS kernel being officially LTS supported for 2 years
> > presents a problem: why would anyone select a 5.10 kernel with 2
> > year LTS when 5.4 kernel has a 6 year LTS.
<snip>
> > If 5.10 is "actually" going to be supported for 6 years it would be
> > quite valuable to make such a declaration.
> > https://www.kernel.org/category/releases.html
>
> Why? What would that change?
>
> Ok, seriously, this happens every year, and every year we go through
> the same thing, it's not like this is somehow new, right?
>
> I want to see companies _using_ the kernel, and most importantly,
> _updating_ their devices with it, to know if it is worth to keep
> around for longer than 2 years. I also, hopefully, want to see how
> those companies will help me out in the testing and maintenance of
> that kernel version in order to make supporting it for 6 years
> actually possible.
>
> So, are you planning on using 5.10? Will you will be willing to help
> out in testing the -rc releases I make to let me know if there are any
> problems, and to help in pointing out and backporting any specific
> patches that your platforms need for that kernel release?
>
> When I get this kind of promises and support from companies, then I am
> glad to bump up the length of the kernel support from 2 to 6 years,
> and I mark it on the web site. Traditionally this happens in
> Febuary/March once I hear from enough companies. Can I count on your
> support in this endeavor?
I am very sorry for the long delay on my end (I had privately e-mailed
Greg on January 28) -- DigitalOcean also intends to provide feedback and
testing on the 5.10 series. We intend to use it as the basis for our
next-next kernel and are very invested in ensuring the stability and
performance of the kernel.
Thanks,
Nish
On Fri, Feb 19, 2021 at 09:44:23AM -0800, Florian Fainelli wrote:
>
>
> On 2/19/2021 7:53 AM, Greg Kroah-Hartman wrote:
> > On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
> >>
> >>
> >> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> >>>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> >>>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >>>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>>>>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>>>>>> having a point of coordination with the Linux Foundation and key people
> >>>>>>>> like you, Greg and other participants in the stable kernel.
> >>>>>>>
> >>>>>>> What does the LF have to do with this?
> >>>>>>>
> >>>>>>> We are here, on the mailing lists, working with everyone. Just test the
> >>>>>>> -rc releases we make and let us know if they work or not for you, it's
> >>>>>>> not a lot of "coordination" needed at all.
> >>>>>>>
> >>>>>>> Otherwise, if no one is saying that they are going to need these for 6
> >>>>>>> years and are willing to use it in their project (i.e. and test it),
> >>>>>>> there's no need for us to maintain it for that long, right?
> >>>>>>
> >>>>>> Greg, please remember I expressed I really need them for slightly more than
> >>>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >>>>>> this saves me from having to take over these kernels after you, like in the
> >>>>>> past, but I cannot engage on the regularity of my availability.
> >>>>>
> >>>>> Ok, great!
> >>>>>
> >>>>> That's one person/company saying they can help out (along with what CIP
> >>>>> has been stating.)
> >>>>>
> >>>>> What about others? Broadcom started this conversation, odd that they
> >>>>> don't seem to want to help out :)
> >>>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
> >>>
> >>> Ok, who at Broadcom do I need to talk to to get that type of commitment?
> >>
> >> I am not sure if I was too subtle before, we (Broadcom) cannot give you
> >> an unified voice to speak with because we are divided in silos/business
> >> units that make their independent decisions.
> >
> > That's fine, I'm totally used to that, large (and even small) companies
> > always have different groups with different roadmaps and policies.
> >
> >> The group I work in (STB/CM, different from Scott's) is committed to
> >> using the 5.10 kernel for 6 years and that is a decision that has been
> >> taken.
> >
> > Great! Will you all be testing the -rc releases and letting me know how
> > they work for your systems?
>
> Yes I will, can you add me to your CC list for the stable candidates?
> That helps me not having to dig for those announcements specifically.
Now added. Of course _after_ I just sent out a round of stable -rc
releases, sorry about that :(
greg k-h
On Mon, Feb 22, 2021 at 08:00:52AM -0600, Nishanth Aravamudan wrote:
> Hi Greg,
>
> On 26.01.2021 [08:29:25 +0100], Greg Kroah-Hartman wrote:
> > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> > > Hi All,
> > >
> > > The 5.10 LTS kernel being officially LTS supported for 2 years
> > > presents a problem: why would anyone select a 5.10 kernel with 2
> > > year LTS when 5.4 kernel has a 6 year LTS.
>
> <snip>
>
> > > If 5.10 is "actually" going to be supported for 6 years it would be
> > > quite valuable to make such a declaration.
> > > https://www.kernel.org/category/releases.html
> >
> > Why? What would that change?
> >
> > Ok, seriously, this happens every year, and every year we go through
> > the same thing, it's not like this is somehow new, right?
> >
> > I want to see companies _using_ the kernel, and most importantly,
> > _updating_ their devices with it, to know if it is worth to keep
> > around for longer than 2 years. I also, hopefully, want to see how
> > those companies will help me out in the testing and maintenance of
> > that kernel version in order to make supporting it for 6 years
> > actually possible.
> >
> > So, are you planning on using 5.10? Will you will be willing to help
> > out in testing the -rc releases I make to let me know if there are any
> > problems, and to help in pointing out and backporting any specific
> > patches that your platforms need for that kernel release?
> >
> > When I get this kind of promises and support from companies, then I am
> > glad to bump up the length of the kernel support from 2 to 6 years,
> > and I mark it on the web site. Traditionally this happens in
> > Febuary/March once I hear from enough companies. Can I count on your
> > support in this endeavor?
>
> I am very sorry for the long delay on my end (I had privately e-mailed
> Greg on January 28) -- DigitalOcean also intends to provide feedback and
> testing on the 5.10 series. We intend to use it as the basis for our
> next-next kernel and are very invested in ensuring the stability and
> performance of the kernel.
Great! If you need me to add your cc: to the -rc release announcements
so that you have an easy email to respond to, just let me know and I can
do so.
Any ideas when the testing will start happening? There's a 5.10-rc
release out there right now if you want to start today :)
thanks,
greg k-h
On 2021/2/20 17:53, Greg Kroah-Hartman wrote:
> On Sat, Feb 20, 2021 at 03:02:54PM +0800, Hanjun Guo wrote:
>> On 2021/2/19 17:08, Greg Kroah-Hartman wrote:
>>> On Fri, Feb 19, 2021 at 04:54:24PM +0800, Hanjun Guo wrote:
>>>> Hi Greg,
>>>>
>>>> On 2021/1/26 15:29, Greg Kroah-Hartman wrote:
>>>> [...]
>>>>>
>>>>> I want to see companies _using_ the kernel, and most importantly,
>>>>> _updating_ their devices with it, to know if it is worth to keep around
>>>>> for longer than 2 years. I also, hopefully, want to see how those
>>>>> companies will help me out in the testing and maintenance of that kernel
>>>>> version in order to make supporting it for 6 years actually possible.
>>>>>
>>>>> So, are you planning on using 5.10? Will you will be willing to help
>>>>> out in testing the -rc releases I make to let me know if there are any
>>>>> problems, and to help in pointing out and backporting any specific
>>>>> patches that your platforms need for that kernel release?
>>>>
>>>> We(Huawei) are willing to commit resources to help out in testing the
>>>> stable -rc releases, and to help to backport patches for stable kernels.
>>>
>>> Wonderful!
>>>
>>>> 5.10 stable kernel will be used for openEuler [1] kernel and also inside
>>>> Huawei. From customer's feedback, it's very important to see the stable
>>>> kernel we used to be maintained for 6 years in the community, and we
>>>> will use 5.10 kernel for at least 6 years, so we are willing to help
>>>> you and help ourselves :)
>>>>
>>>> In specific, we will start from the testing work, using HULK robot
>>>> (reports lots of bugs to mainline kernel) testing framework to test
>>>> compile, reboot, functional testing, and will extend to basic
>>>> performance regression testing in the future.
>>>
>>> Great! Do you all need an email notification when the -rc releases come
>>> out for the stable trees, or can you trigger off of the -rc stable git
>>> tree? Different CI systems work in different ways :)
>>
>> We can trigger the test when you updated the -rc stable git tree,
>> by monitoring new commits for the stable branches. So if you push
>> all the commits at once for -rc stable branches, then our CI system
>> can work well.
>
> I do push to the -rc branches, but those branches are rebased, and I do
> "intermediate" pushes as well. Meaning I push to have CI systems run on
> the existing patch queue at times that are not only the "main" -rc
> release periods.
>
> Watch the branches for a few weeks to get an idea of how they work if
> you are curious.
Will run our CI system based on your -rc branch to see if anything
doesn't work, then update our system as needed.
Thanks
Hanjun
Hi Nick,
Sorry for taking so long to reply you, we had discussions on how to
corporate with KCIDB, please see my comments inline.
On 2021/2/19 22:45, Nikolai Kondrashov wrote:
> Hi Hanjun,
>
> On 2/19/21 10:54 AM, Hanjun Guo wrote:
> > In specific, we will start from the testing work, using HULK robot
> > (reports lots of bugs to mainline kernel) testing framework to test
> > compile, reboot, functional testing, and will extend to basic
> > performance regression testing in the future.
>
> I heard about Huawei ramping up kernel testing from someone at FOSDEM
> 2019. I wonder if it was you :) Nice to see your progress and the company
> stepping up to help with testing!
I Cced Yongjun and Jinyue, they are the key persons :)
>
> Would you be interested in working with the Linux Foundation KernelCI
> project
> on submitting your build and test results to the common database - KCIDB?
Yes, we are willing to sent the test results to KCIDB.
For now, all the tests are inside the company, which blocks us to
directly sent the test results out of the test machine due to the
security policy, it takes us sometime to discuss how to send out
the test results, and we may need do the test in a public cloud.
>
> We are working on aggregating results from various testing systems so we
> can
> provide a dashboard, and a single, aggregated e-mail report to subscribed
> maintainers and developers.
>
> We have a prototype dashboard at https://staging.kernelci.org:3000/ and are
> working hard on making the e-mail reports good enough to start reaching
> out to
> maintainers.
>
> We already have ARM, Google Syzbot, Gentoo GKernelCI, Red Hat CKI, and, of
> course, KernelCI native tests sending data to the database. Linaro
> Tuxsuite is
> starting sending today. We could use your data, and of course any
> development
> help you could spare :)
How can we connect to the KCIDB? Can we send the test data out by email
first?
>
> I wish I could show you my today's KCIDB presentation at DevConf.cz, but
> the
> recording is not out yet. Meanwhile you can take a look at our
> presentation at
> last year's Linux Plumbers: https://youtu.be/y9Glc90WUN0?t=10739
>
> Or see our intro in an older blog post:
> https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>
>
> Anyone wishing to contribute to KCIDB gets credentials and permissions to
> submit to our "playground" setup where they can send their data, see it
> in a
> dashboard, experiment without worrying about breaking anything, and
> decide if
> they like it or n >
> If you're interested, take a look at our Submission HOWTO:
> https://github.com/kernelci/kcidb/blob/v8/SUBMISSION_HOWTO.md
> and send an email to [email protected] (CC'd), or come over to the
> #kernelci
> channel on freenode.net!
Thanks!
Hanjun
Hi Hanjun,
On 2/26/21 10:03 AM, Hanjun Guo wrote:
> On 2021/2/19 22:45, Nikolai Kondrashov wrote:
>> Would you be interested in working with the Linux Foundation KernelCI project
>> on submitting your build and test results to the common database - KCIDB?
>
> Yes, we are willing to sent the test results to KCIDB.
Wonderful!
> For now, all the tests are inside the company, which blocks us to
> directly sent the test results out of the test machine due to the
> security policy, it takes us sometime to discuss how to send out
> the test results, and we may need do the test in a public cloud.
I understand. We faced the same dilemma at Red Hat with the CKI project, where
I'm employed. In the end we just decided to publish the logs from the test
lab (most sensitive part, I suppose) as they are, but then Red Hat is already
a very open company.
Here's an example of all logs we publish for a revision:
https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/22/624489
And here's that revision in KCIDB:
https://staging.kernelci.org:3000/d/revision/revision?orgId=1&var-dataset=kernelci04&var-id=093ae67c90a629599a0cb2300c5686ec7d8bbdd0
Take your time, there is a range of solutions to choose from, starting from
isolating your test machines/boards, to running the test in the public cloud
as you mention. Although the latter lacks the benefit of testing your own
hardware, of course.
Please also note that we require the tested kernel's code to be publicly
available to accept results.
>> We are working on aggregating results from various testing systems so we can
>> provide a dashboard, and a single, aggregated e-mail report to subscribed
>> maintainers and developers.
>>
>> We have a prototype dashboard at https://staging.kernelci.org:3000/ and are
>> working hard on making the e-mail reports good enough to start reaching out to
>> maintainers.
>>
>> We already have ARM, Google Syzbot, Gentoo GKernelCI, Red Hat CKI, and, of
>> course, KernelCI native tests sending data to the database. Linaro Tuxsuite is
>> starting sending today. We could use your data, and of course any development
>> help you could spare :)
>
> How can we connect to the KCIDB? Can we send the test data out by email
> first?
The simplest way would be using our "kcidb-submit" command-line tool.
Here's a minimal example of sending an empty report:
echo '{"version":{"major":3,"minor":0}}' |
kcidb-submit -p kernelci-production -t kernelci_new
Another way could be using our Python 3 library, which is a little more
involved. Here's an example sending the same empty report in Python:
import kcidb
client = kcidb.Client(project_id="kernelci-production",
topic_name="kernelci_new")
client.submit({"version":{"major":3,"minor":0}})
Finally, you can send (validated) data using the Google Cloud Pub/Sub service
directly, using the library in one of the supported languages:
https://cloud.google.com/pubsub/docs/tutorials
The latter would be the least stable option (change-wise), but something e.g.
Google Syzbot is already using.
See more details in our Submission HOWTO:
https://github.com/kernelci/kcidb/blob/main/SUBMISSION_HOWTO.md
Although email submissions are theoretically possible (with due
authentication, setup, etc.), they're not currently supported, and we would
not put them at the top of our (rather large) priority list. Topmost item
being getting the data we already have to developers.
Would be excited to have you on board!
Nick
On 2/26/21 10:03 AM, Hanjun Guo wrote:
> Hi Nick,
>
> Sorry for taking so long to reply you, we had discussions on how to
> corporate with KCIDB, please see my comments inline.
>
> On 2021/2/19 22:45, Nikolai Kondrashov wrote:
>> Hi Hanjun,
>>
>> On 2/19/21 10:54 AM, Hanjun Guo wrote:
>> > In specific, we will start from the testing work, using HULK robot
>> > (reports lots of bugs to mainline kernel) testing framework to test
>> > compile, reboot, functional testing, and will extend to basic
>> > performance regression testing in the future.
>>
>> I heard about Huawei ramping up kernel testing from someone at FOSDEM
>> 2019. I wonder if it was you :) Nice to see your progress and the company
>> stepping up to help with testing!
>
> I Cced Yongjun and Jinyue, they are the key persons :)
>
>>
>> Would you be interested in working with the Linux Foundation KernelCI project
>> on submitting your build and test results to the common database - KCIDB?
>
> Yes, we are willing to sent the test results to KCIDB.
>
> For now, all the tests are inside the company, which blocks us to
> directly sent the test results out of the test machine due to the
> security policy, it takes us sometime to discuss how to send out
> the test results, and we may need do the test in a public cloud.
>
>>
>> We are working on aggregating results from various testing systems so we can
>> provide a dashboard, and a single, aggregated e-mail report to subscribed
>> maintainers and developers.
>>
>> We have a prototype dashboard at https://staging.kernelci.org:3000/ and are
>> working hard on making the e-mail reports good enough to start reaching out to
>> maintainers.
>>
>> We already have ARM, Google Syzbot, Gentoo GKernelCI, Red Hat CKI, and, of
>> course, KernelCI native tests sending data to the database. Linaro Tuxsuite is
>> starting sending today. We could use your data, and of course any development
>> help you could spare :)
>
> How can we connect to the KCIDB? Can we send the test data out by email
> first?
>
>>
>> I wish I could show you my today's KCIDB presentation at DevConf.cz, but the
>> recording is not out yet. Meanwhile you can take a look at our presentation at
>> last year's Linux Plumbers: https://youtu.be/y9Glc90WUN0?t=10739
>>
>> Or see our intro in an older blog post:
>> https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/
>>
>> Anyone wishing to contribute to KCIDB gets credentials and permissions to
>> submit to our "playground" setup where they can send their data, see it in a
>> dashboard, experiment without worrying about breaking anything, and decide if
>> they like it or n >
>> If you're interested, take a look at our Submission HOWTO:
>> https://github.com/kernelci/kcidb/blob/v8/SUBMISSION_HOWTO.md
>> and send an email to [email protected] (CC'd), or come over to the #kernelci
>> channel on freenode.net!
>
> Thanks!
> Hanjun