Hi all,
I'd like to announce the linux-stable security tree project. The purpose
is to create a derivative tree from the regular stable tree that would
contain only commits that fix security vulnerabilities.
Quite a few users of the stable trees pointed out that on complex deployments,
where validation is non-trivial, there is little incentive to follow the
stable tree after the product has been deployed to production. There is no
interest in "random" kernel fixes and the only requirements are to keep up
with security vulnerabilities.
Given this, a few projects preferred to delay important kernel updates, and
a few even stopped updating the tree altogether, exposing them to critical
vulnerabilities.
This project provides an easy way to receive only important security commits,
which are usually only a few in each release, and makes it easy to incorporate
them into existing projects.
The tree is available at:
https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/
Support is provided for all active -stable trees (https://www.kernel.org/category/releases.html).
Branches/tags for unsupported versions of >=3.0 kernels were also generated for reference.
Thanks,
Sasha
On 4/11/16, Sasha Levin <[email protected]> wrote:
> Hi all,
>
>
> I'd like to announce the linux-stable security tree project. The purpose
> is to create a derivative tree from the regular stable tree that would
> contain only commits that fix security vulnerabilities.
>
> Quite a few users of the stable trees pointed out that on complex
> deployments,
> where validation is non-trivial, there is little incentive to follow the
> stable tree after the product has been deployed to production. There is no
> interest in "random" kernel fixes and the only requirements are to keep up
> with security vulnerabilities.
>
> Given this, a few projects preferred to delay important kernel updates, and
> a few even stopped updating the tree altogether, exposing them to critical
> vulnerabilities.
>
> This project provides an easy way to receive only important security
> commits,
> which are usually only a few in each release, and makes it easy to
> incorporate
> them into existing projects.
>
> The tree is available at:
>
> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/
>
> Support is provided for all active -stable trees
> (https://www.kernel.org/category/releases.html).
> Branches/tags for unsupported versions of >=3.0 kernels were also generated
> for reference.
>
>
> Thanks,
> Sasha
>
>
>
Hi Sasha,
Can you push a copy of this tree to github like Linus has done with
torvalds/linux so folks who want to develop on it can fork it from
there.
Jeff
On 4/11/16, Sasha Levin <[email protected]> wrote:
> Hi all,
>
>
> I'd like to announce the linux-stable security tree project. The purpose
> is to create a derivative tree from the regular stable tree that would
> contain only commits that fix security vulnerabilities.
>
> Quite a few users of the stable trees pointed out that on complex
> deployments,
> where validation is non-trivial, there is little incentive to follow the
> stable tree after the product has been deployed to production. There is no
> interest in "random" kernel fixes and the only requirements are to keep up
> with security vulnerabilities.
>
> Given this, a few projects preferred to delay important kernel updates, and
> a few even stopped updating the tree altogether, exposing them to critical
> vulnerabilities.
>
> This project provides an easy way to receive only important security
> commits,
> which are usually only a few in each release, and makes it easy to
> incorporate
> them into existing projects.
>
> The tree is available at:
>
> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/
>
> Support is provided for all active -stable trees
> (https://www.kernel.org/category/releases.html).
> Branches/tags for unsupported versions of >=3.0 kernels were also generated
> for reference.
>
>
> Thanks,
> Sasha
>
>
>
Hi Sasha,
[root@166-70-213-246 kernel]#
[root@166-70-213-246 kernel]#
[root@166-70-213-246 kernel]# git clone
https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git
Cloning into 'linux-stable-security'...
fatal: repository
'https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/'
not found
[root@166-70-213-246 kernel]# git clone
https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git
Cloning into 'linux-stable-security'...
fatal: repository
'https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/'
not found
[root@166-70-213-246 kernel]#
Git says the tree is not clonable.
Jeff
On Mon, Apr 11, 2016 at 01:53:41PM -0400, Sasha Levin wrote:
> Hi all,
>
>
> I'd like to announce the linux-stable security tree project. The purpose
> is to create a derivative tree from the regular stable tree that would
> contain only commits that fix security vulnerabilities.
And how are you going to define "security vulnerabilities"?
Also, what kernel branches are you going to do this for, and for how
long?
> Quite a few users of the stable trees pointed out that on complex deployments,
> where validation is non-trivial, there is little incentive to follow the
> stable tree after the product has been deployed to production. There is no
> interest in "random" kernel fixes and the only requirements are to keep up
> with security vulnerabilities.
"random" is in the eye of the beholder :)
Everyone's definition of that is different, I think you will find that
it's best to just do the full tree, or pick and choose what you want
from the tree based on your own needs, instead of trying to judge what
those needs are.
> Given this, a few projects preferred to delay important kernel updates, and
> a few even stopped updating the tree altogether, exposing them to critical
> vulnerabilities.
What projects are those? Is it really that hard to take the current
stable trees for these projects? Why is this tree going to somehow be
easier?
> This project provides an easy way to receive only important security commits,
> which are usually only a few in each release, and makes it easy to incorporate
> them into existing projects.
Define "important". Now go and look at the tty bug we fixed that people
only realized was "important" 1 1/2 years later and explain if you
would, or would not have, taken that patch in this tree.
Good luck with this, personally, I think projects should just take the
stable tree as really, it's not that much churn at all, given the % of
the mainline tree.
thanks,
greg k-h
On 04/11/2016 02:41 PM, Greg KH wrote:
> On Mon, Apr 11, 2016 at 01:53:41PM -0400, Sasha Levin wrote:
>> Hi all,
>>
>>
>> I'd like to announce the linux-stable security tree project. The purpose
>> is to create a derivative tree from the regular stable tree that would
>> contain only commits that fix security vulnerabilities.
>
> And how are you going to define "security vulnerabilities"?
In general, anything that qualified for a CVE plus anything exploitable
by a local unprivileged user.
> Also, what kernel branches are you going to do this for, and for how
> long?
All active -stable kernels, for as long as those corresponding kernels
are supported. This isn't a brand new tree, but is a subset of it's
corresponding -stable tree.
>> Quite a few users of the stable trees pointed out that on complex deployments,
>> where validation is non-trivial, there is little incentive to follow the
>> stable tree after the product has been deployed to production. There is no
>> interest in "random" kernel fixes and the only requirements are to keep up
>> with security vulnerabilities.
>
> "random" is in the eye of the beholder :)
It's not "random" for us building the tree, but it's very much "random"
for the users for see fixes for drivers they've never even heard of before.
> Everyone's definition of that is different, I think you will find that
> it's best to just do the full tree, or pick and choose what you want
> from the tree based on your own needs, instead of trying to judge what
> those needs are.
It would probably be ideal if everyone could cherry pick correctly only and
all the commits they require, but reality is quite far from this ideal, which
is why we have things like the stable trees - to simplify the process.
>> Given this, a few projects preferred to delay important kernel updates, and
>> a few even stopped updating the tree altogether, exposing them to critical
>> vulnerabilities.
>
> What projects are those? Is it really that hard to take the current
> stable trees for these projects? Why is this tree going to somehow be
> easier?
I can give you an example offlist (mostly because I don't want to start listing
projects in production with outdated kernels who could use security updates).
>> This project provides an easy way to receive only important security commits,
>> which are usually only a few in each release, and makes it easy to incorporate
>> them into existing projects.
>
> Define "important". Now go and look at the tty bug we fixed that people
> only realized was "important" 1 1/2 years later and explain if you
> would, or would not have, taken that patch in this tree.
Probably not, but I would have taken it after it received a CVE number.
Same applies to quite a few commits that end up in stable - no one thinks
they're stable material at first until someone points out it's crashing
his production boxes for the past few months.
> Good luck with this, personally, I think projects should just take the
> stable tree as really, it's not that much churn at all, given the % of
> the mainline tree.
Thanks,
Sasha
On 04/11/2016 02:17 PM, Jeff Merkey wrote:
> On 4/11/16, Sasha Levin <[email protected]> wrote:
>> Hi all,
>>
>>
>> I'd like to announce the linux-stable security tree project. The purpose
>> is to create a derivative tree from the regular stable tree that would
>> contain only commits that fix security vulnerabilities.
>>
>> Quite a few users of the stable trees pointed out that on complex
>> deployments,
>> where validation is non-trivial, there is little incentive to follow the
>> stable tree after the product has been deployed to production. There is no
>> interest in "random" kernel fixes and the only requirements are to keep up
>> with security vulnerabilities.
>>
>> Given this, a few projects preferred to delay important kernel updates, and
>> a few even stopped updating the tree altogether, exposing them to critical
>> vulnerabilities.
>>
>> This project provides an easy way to receive only important security
>> commits,
>> which are usually only a few in each release, and makes it easy to
>> incorporate
>> them into existing projects.
>>
>> The tree is available at:
>>
>> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/
>>
>> Support is provided for all active -stable trees
>> (https://www.kernel.org/category/releases.html).
>> Branches/tags for unsupported versions of >=3.0 kernels were also generated
>> for reference.
>>
>>
>> Thanks,
>> Sasha
>>
>>
>>
>
> Hi Sasha,
>
> Can you push a copy of this tree to github like Linus has done with
> torvalds/linux so folks who want to develop on it can fork it from
> there.
Sure! Now pushed to https://github.com/sashalevin/linux-stable-security
Thanks,
Sasha
On Mon, Apr 11, 2016 at 8:34 PM, Jeff Merkey <[email protected]> wrote:
> On 4/11/16, Sasha Levin <[email protected]> wrote:
>> Hi all,
>>
>>
>> I'd like to announce the linux-stable security tree project. The purpose
>> is to create a derivative tree from the regular stable tree that would
>> contain only commits that fix security vulnerabilities.
>>
>> Quite a few users of the stable trees pointed out that on complex
>> deployments,
>> where validation is non-trivial, there is little incentive to follow the
>> stable tree after the product has been deployed to production. There is no
>> interest in "random" kernel fixes and the only requirements are to keep up
>> with security vulnerabilities.
>>
>> Given this, a few projects preferred to delay important kernel updates, and
>> a few even stopped updating the tree altogether, exposing them to critical
>> vulnerabilities.
>>
>> This project provides an easy way to receive only important security
>> commits,
>> which are usually only a few in each release, and makes it easy to
>> incorporate
>> them into existing projects.
>>
>> The tree is available at:
>>
>> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/
>>
>> Support is provided for all active -stable trees
>> (https://www.kernel.org/category/releases.html).
>> Branches/tags for unsupported versions of >=3.0 kernels were also generated
>> for reference.
>>
>>
>> Thanks,
>> Sasha
>>
>>
>>
>
> Hi Sasha,
>
> [root@166-70-213-246 kernel]#
> [root@166-70-213-246 kernel]#
> [root@166-70-213-246 kernel]# git clone
> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git
> Cloning into 'linux-stable-security'...
> fatal: repository
> 'https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/'
> not found
> [root@166-70-213-246 kernel]# git clone
> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git
> Cloning into 'linux-stable-security'...
> fatal: repository
> 'https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/'
> not found
> [root@166-70-213-246 kernel]#
>
>
> Git says the tree is not clonable.
Because you're trying to clone the cgit webinterface. :-)
--
Thanks,
//richard
On 04/11/2016 02:34 PM, Jeff Merkey wrote:
> On 4/11/16, Sasha Levin <[email protected]> wrote:
>> Hi all,
>>
>>
>> I'd like to announce the linux-stable security tree project. The purpose
>> is to create a derivative tree from the regular stable tree that would
>> contain only commits that fix security vulnerabilities.
>>
>> Quite a few users of the stable trees pointed out that on complex
>> deployments,
>> where validation is non-trivial, there is little incentive to follow the
>> stable tree after the product has been deployed to production. There is no
>> interest in "random" kernel fixes and the only requirements are to keep up
>> with security vulnerabilities.
>>
>> Given this, a few projects preferred to delay important kernel updates, and
>> a few even stopped updating the tree altogether, exposing them to critical
>> vulnerabilities.
>>
>> This project provides an easy way to receive only important security
>> commits,
>> which are usually only a few in each release, and makes it easy to
>> incorporate
>> them into existing projects.
>>
>> The tree is available at:
>>
>> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/
>>
>> Support is provided for all active -stable trees
>> (https://www.kernel.org/category/releases.html).
>> Branches/tags for unsupported versions of >=3.0 kernels were also generated
>> for reference.
>>
>>
>> Thanks,
>> Sasha
>>
>>
>>
>
> Hi Sasha,
>
> [root@166-70-213-246 kernel]#
> [root@166-70-213-246 kernel]#
> [root@166-70-213-246 kernel]# git clone
> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git
> Cloning into 'linux-stable-security'...
> fatal: repository
> 'https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/'
> not found
> [root@166-70-213-246 kernel]# git clone
> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git
> Cloning into 'linux-stable-security'...
> fatal: repository
> 'https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/'
> not found
> [root@166-70-213-246 kernel]#
Either of the following will let you clone it:
git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux-stable-security.git
https://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux-stable-security.git
https://kernel.googlesource.com/pub/scm/linux/kernel/git/sashal/linux-stable-security.git
Thanks,
Sasha
On 4/11/16, Sasha Levin <[email protected]> wrote:
> On 04/11/2016 02:34 PM, Jeff Merkey wrote:
>> On 4/11/16, Sasha Levin <[email protected]> wrote:
>>> Hi all,
>>>
>>>
>>> I'd like to announce the linux-stable security tree project. The purpose
>>> is to create a derivative tree from the regular stable tree that would
>>> contain only commits that fix security vulnerabilities.
>>>
>>> Quite a few users of the stable trees pointed out that on complex
>>> deployments,
>>> where validation is non-trivial, there is little incentive to follow the
>>> stable tree after the product has been deployed to production. There is
>>> no
>>> interest in "random" kernel fixes and the only requirements are to keep
>>> up
>>> with security vulnerabilities.
>>>
>>> Given this, a few projects preferred to delay important kernel updates,
>>> and
>>> a few even stopped updating the tree altogether, exposing them to
>>> critical
>>> vulnerabilities.
>>>
>>> This project provides an easy way to receive only important security
>>> commits,
>>> which are usually only a few in each release, and makes it easy to
>>> incorporate
>>> them into existing projects.
>>>
>>> The tree is available at:
>>>
>>> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/
>>>
>>> Support is provided for all active -stable trees
>>> (https://www.kernel.org/category/releases.html).
>>> Branches/tags for unsupported versions of >=3.0 kernels were also
>>> generated
>>> for reference.
>>>
>>>
>>> Thanks,
>>> Sasha
>>>
>>>
>>>
>>
>> Hi Sasha,
>>
>> [root@166-70-213-246 kernel]#
>> [root@166-70-213-246 kernel]#
>> [root@166-70-213-246 kernel]# git clone
>> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git
>> Cloning into 'linux-stable-security'...
>> fatal: repository
>> 'https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/'
>> not found
>> [root@166-70-213-246 kernel]# git clone
>> https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git
>> Cloning into 'linux-stable-security'...
>> fatal: repository
>> 'https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/'
>> not found
>> [root@166-70-213-246 kernel]#
>
> Either of the following will let you clone it:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux-stable-security.git
> https://git.kernel.org/pub/scm/linux/kernel/git/sashal/linux-stable-security.git
> https://kernel.googlesource.com/pub/scm/linux/kernel/git/sashal/linux-stable-security.git
>
>
> Thanks,
> Sasha
>
Thanks Guys,
:-)
Jeff
On Mon, Apr 11, 2016 at 02:58:37PM -0400, Sasha Levin wrote:
> On 04/11/2016 02:41 PM, Greg KH wrote:
> > On Mon, Apr 11, 2016 at 01:53:41PM -0400, Sasha Levin wrote:
> >> Hi all,
> >>
> >>
> >> I'd like to announce the linux-stable security tree project. The purpose
> >> is to create a derivative tree from the regular stable tree that would
> >> contain only commits that fix security vulnerabilities.
> >
> > And how are you going to define "security vulnerabilities"?
>
> In general, anything that qualified for a CVE plus anything exploitable
> by a local unprivileged user.
>
> > Also, what kernel branches are you going to do this for, and for how
> > long?
>
> All active -stable kernels, for as long as those corresponding kernels
> are supported. This isn't a brand new tree, but is a subset of it's
> corresponding -stable tree.
>
> >> Quite a few users of the stable trees pointed out that on complex deployments,
> >> where validation is non-trivial, there is little incentive to follow the
> >> stable tree after the product has been deployed to production. There is no
> >> interest in "random" kernel fixes and the only requirements are to keep up
> >> with security vulnerabilities.
> >
> > "random" is in the eye of the beholder :)
>
> It's not "random" for us building the tree, but it's very much "random"
> for the users for see fixes for drivers they've never even heard of before.
Then why would it matter if they pulled the tree or not? How are you
going to judge which driver fixes to take and which not to? Why not
take them all if they fix bugs?
> > Everyone's definition of that is different, I think you will find that
> > it's best to just do the full tree, or pick and choose what you want
> > from the tree based on your own needs, instead of trying to judge what
> > those needs are.
>
> It would probably be ideal if everyone could cherry pick correctly only and
> all the commits they require, but reality is quite far from this ideal, which
> is why we have things like the stable trees - to simplify the process.
>
> >> Given this, a few projects preferred to delay important kernel updates, and
> >> a few even stopped updating the tree altogether, exposing them to critical
> >> vulnerabilities.
> >
> > What projects are those? Is it really that hard to take the current
> > stable trees for these projects? Why is this tree going to somehow be
> > easier?
>
> I can give you an example offlist (mostly because I don't want to start listing
> projects in production with outdated kernels who could use security updates).
>
> >> This project provides an easy way to receive only important security commits,
> >> which are usually only a few in each release, and makes it easy to incorporate
> >> them into existing projects.
> >
> > Define "important". Now go and look at the tty bug we fixed that people
> > only realized was "important" 1 1/2 years later and explain if you
> > would, or would not have, taken that patch in this tree.
>
> Probably not, but I would have taken it after it received a CVE number.
>
> Same applies to quite a few commits that end up in stable - no one thinks
> they're stable material at first until someone points out it's crashing
> his production boxes for the past few months.
Yes, but those are rare, what you are doing here is suddenly having to
judge if a bug is a "security" issue or not. You are now in the
position of trying to determine "can this be exploited or not", for
every commit, and that's a very hard call, as is seen by this specific
issue.
If you only take things you "know" are issues, well, you miss lots of
fixes that were really security things but only found out later, so
people have to scamble to fix their kernels much faster than they needed
to.
Putting up a tree like this isn't going to cause people to update their
trees any quicker. If they haven't been doing it now, they aren't going
to do it with this tree, as their workloads don't allow them to take
updated kernel trees.
In short, it's not the fact that we have stable trees that are "too big"
for them to take, it's the fact that they just can't take and test
updates properly. They need to fix their processes, it's not a
deficiency on our side from everyone I have talked to, including the
example you gave me off-list :)
good luck!
greg k-h
On 04/11/2016 04:09 PM, Greg KH wrote:
> On Mon, Apr 11, 2016 at 02:58:37PM -0400, Sasha Levin wrote:
>> On 04/11/2016 02:41 PM, Greg KH wrote:
>>> On Mon, Apr 11, 2016 at 01:53:41PM -0400, Sasha Levin wrote:
>>>> Quite a few users of the stable trees pointed out that on complex deployments,
>>>> where validation is non-trivial, there is little incentive to follow the
>>>> stable tree after the product has been deployed to production. There is no
>>>> interest in "random" kernel fixes and the only requirements are to keep up
>>>> with security vulnerabilities.
>>>
>>> "random" is in the eye of the beholder :)
>>
>> It's not "random" for us building the tree, but it's very much "random"
>> for the users for see fixes for drivers they've never even heard of before.
>
> Then why would it matter if they pulled the tree or not? How are you
> going to judge which driver fixes to take and which not to? Why not
> take them all if they fix bugs?
Because some fixes introduce bug on their own? Take a look at how many
commits in the stable tree have a "Fixes:" tag that points to a commit
that's also in the stable tree.
Look at the opposite side of this question: why would anyone take a commit
that fixes a bug he doesn't care about? Are the benefits really worth it
considering the risks?
[snip]
>>> Define "important". Now go and look at the tty bug we fixed that people
>>> only realized was "important" 1 1/2 years later and explain if you
>>> would, or would not have, taken that patch in this tree.
>>
>> Probably not, but I would have taken it after it received a CVE number.
>>
>> Same applies to quite a few commits that end up in stable - no one thinks
>> they're stable material at first until someone points out it's crashing
>> his production boxes for the past few months.
>
> Yes, but those are rare, what you are doing here is suddenly having to
> judge if a bug is a "security" issue or not. You are now in the
> position of trying to determine "can this be exploited or not", for
> every commit, and that's a very hard call, as is seen by this specific
> issue.
The stable stuff isn't rare as you might think, even more: the amount of
actual CVE fixes that are not in the stable tree might surprise you.
This usually happens for the reason you described, we look at a commit
that has an innocent subject/description, not CC'ed to stable@ and we figure
that it's not stable material until someone shows how to exploit it and
requests a CVE.
This is not rare.
> If you only take things you "know" are issues, well, you miss lots of
> fixes that were really security things but only found out later, so
> people have to scamble to fix their kernels much faster than they needed
> to.
I don't only take known issues. I don't claim to have a 100% success rate,
but this is the same story as the stable tree and the patch selection there.
> Putting up a tree like this isn't going to cause people to update their
> trees any quicker. If they haven't been doing it now, they aren't going
> to do it with this tree, as their workloads don't allow them to take
> updated kernel trees.
>
> In short, it's not the fact that we have stable trees that are "too big"
> for them to take, it's the fact that they just can't take and test
> updates properly. They need to fix their processes, it's not a
> deficiency on our side from everyone I have talked to, including the
> example you gave me off-list :)
Pulling 100+ "random" (yes, I used it again) commits would require a very
different review process than cherry picking ~5 commits that you can more
easily review and validate.
This is actually what happens now; projects get to the point they don't
want to update their whole kernel tree anymore so that just freezes because
they don't want to re-validate the whole thing over and over, but they
still cherry pick upstream and out-of-tree commits that they care about.
If they added a handful of security commits to cherry pick and carefully
review their security will be much better than what happens now.
Thanks,
Sasha
Hi Sasha,
On Mon, Apr 11, 2016 at 04:38:17PM -0400, Sasha Levin wrote:
> > How are you
> > going to judge which driver fixes to take and which not to? Why not
> > take them all if they fix bugs?
>
> Because some fixes introduce bug on their own? Take a look at how many
> commits in the stable tree have a "Fixes:" tag that points to a commit
> that's also in the stable tree.
I'm using stable trees myself in the load balancing products we ship at
work. I've met a single bug during the whole 3.10 lifetime and it was
caused by one of our out-of-tree patch that applied at the wrong place
after an update. I'd generally say that -stable quality is very good,
if not excellent. Several people review the patches before they get
merged, several ones build and even boot them. It's not that random.
Look, one patch was just dropped from 3.14.64 because it failed a build
test in one environment. This one will never hit end users.
> Look at the opposite side of this question: why would anyone take a commit
> that fixes a bug he doesn't care about? Are the benefits really worth it
> considering the risks?
That's exactly what most people do. I don't update to each and every kernel.
When I see xen, lvm, drm and audio changes I don't need them in my products.
But when I'm seeing network fixes I study them and often decide that it's
worth upgrading. Sometimes I pick a single fix from the queue because I can't
wait for next release. Many of Greg's kernels more or less focus on certain
topics, probably due to the way he deals with his mailbox and patch storms,
so it's often easy to quickly decide if you're going to need to update or
not.
> [snip]
>
> >>> Define "important". Now go and look at the tty bug we fixed that people
> >>> only realized was "important" 1 1/2 years later and explain if you
> >>> would, or would not have, taken that patch in this tree.
> >>
> >> Probably not, but I would have taken it after it received a CVE number.
> >>
> >> Same applies to quite a few commits that end up in stable - no one thinks
> >> they're stable material at first until someone points out it's crashing
> >> his production boxes for the past few months.
> >
> > Yes, but those are rare, what you are doing here is suddenly having to
> > judge if a bug is a "security" issue or not. You are now in the
> > position of trying to determine "can this be exploited or not", for
> > every commit, and that's a very hard call, as is seen by this specific
> > issue.
Especially for networking stuff or things related to local resource usage
where some people consider it represents a local DoS risk and others
consider that it's just irrelevant to their servers since they have no
local users.
> The stable stuff isn't rare as you might think, even more: the amount of
> actual CVE fixes that are not in the stable tree might surprise you.
I would personally not be surprized since Ben used to feed me with a lot
of fixes I had never seen previously. What is unclear to me is if your
tree will contain only a selection of patches that are already in the
respective branches, or a backport of security fixes that we can pick
from to feed our stable branches and limit the risk of missing them.
*This* actually could be useful to everyone, starting from our users.
(...)
> This is actually what happens now; projects get to the point they don't
> want to update their whole kernel tree anymore so that just freezes because
> they don't want to re-validate the whole thing over and over, but they
> still cherry pick upstream and out-of-tree commits that they care about.
>
> If they added a handful of security commits to cherry pick and carefully
> review their security will be much better than what happens now.
Actually I do think that for end users it's a regression. People will
start reusing outdated kernels which only contain the most critical fixes
known, but will still suffer from memory leaks, deadlocks, kernel panics,
data corruption etc. Every single bug that doesn't have a CVE attached to
it in fact, which means 99% of the bugs that bring a system down in
production. It makes me think about people who only pick security fixes
from openssl and not the regular batch of missing null checks, and who
complain all the time that their systems are unstable while they simply
don't apply fixes.
You see, when I started with the "hotfix" tree 11 years ago for kernel
2.4, I intented to only pick the most critical fixes, they would fit
in just a README and were counted on one hand. One year later there
were 150 just because everything becomes critical for *some* workloads.
I *do* think that having a central reference for fixes that come with
a reproducer (hence many security fixes) can be useful as it would
offer an opportunity for better testing backports when they become
tricky : it often takes much more time to try to set up a test with
a reproducer than it takes to backport and adjust the fix (not always
true). But when it comes to security issues often the reporter cares
about the quality of the backport and helps there.
Just my two cents,
Willy
On 04/11/2016 05:17 PM, Willy Tarreau wrote:
> Hi Sasha,
>
> On Mon, Apr 11, 2016 at 04:38:17PM -0400, Sasha Levin wrote:
>>> How are you
>>> going to judge which driver fixes to take and which not to? Why not
>>> take them all if they fix bugs?
>>
>> Because some fixes introduce bug on their own? Take a look at how many
>> commits in the stable tree have a "Fixes:" tag that points to a commit
>> that's also in the stable tree.
>
> I'm using stable trees myself in the load balancing products we ship at
> work. I've met a single bug during the whole 3.10 lifetime and it was
> caused by one of our out-of-tree patch that applied at the wrong place
> after an update. I'd generally say that -stable quality is very good,
> if not excellent. Several people review the patches before they get
> merged, several ones build and even boot them. It's not that random.
> Look, one patch was just dropped from 3.14.64 because it failed a build
> test in one environment. This one will never hit end users.
Bugs in the stable tree are created not only because of maintainer errors or
lack of reviews, but also because fixes themselves have bugs.
While maintaining 3.18, I've noticed that I ended up missing one commit because
it's "Fixes:" field had a commit hash that wasn't in the tree, only to find later
that that commit was pulled in in a previous stable tree and had a different hash
in the local tree.
I've since automated that lookup, and that case gets hit quite often.
As this suggests, we don't only fix bugs in stable trees, we also introduce
a fair amount of new ones.
>> Look at the opposite side of this question: why would anyone take a commit
>> that fixes a bug he doesn't care about? Are the benefits really worth it
>> considering the risks?
>
> That's exactly what most people do. I don't update to each and every kernel.
> When I see xen, lvm, drm and audio changes I don't need them in my products.
> But when I'm seeing network fixes I study them and often decide that it's
> worth upgrading. Sometimes I pick a single fix from the queue because I can't
> wait for next release. Many of Greg's kernels more or less focus on certain
> topics, probably due to the way he deals with his mailbox and patch storms,
> so it's often easy to quickly decide if you're going to need to update or
> not.
I fully agree with you here, if you can hand pick the commits you want and
study every commit that goes in I can't really argue with that.
But the target audience here is different. I'm thinking about projects where
the kernel isn't even close to being center stage, and it doesn't interest
anyone as long as things work. There is no chance in hell that they would
start picking commits unless you hold a gun to their hand.
My thinking here is that it's easier to convince them to take a handful of
patches in rather than hundreds.
Also, I'm not trying to replace the stable tree. I'm just trying to present
a stopgap solution that projects can use in between complete stable kernel
updates (or changes in kernel version). This is one of the reasons I'm not
adding any new commits in here that aren't in the stable tree already.
>> [snip]
>>
>>>>> Define "important". Now go and look at the tty bug we fixed that people
>>>>> only realized was "important" 1 1/2 years later and explain if you
>>>>> would, or would not have, taken that patch in this tree.
>>>>
>>>> Probably not, but I would have taken it after it received a CVE number.
>>>>
>>>> Same applies to quite a few commits that end up in stable - no one thinks
>>>> they're stable material at first until someone points out it's crashing
>>>> his production boxes for the past few months.
>>>
>>> Yes, but those are rare, what you are doing here is suddenly having to
>>> judge if a bug is a "security" issue or not. You are now in the
>>> position of trying to determine "can this be exploited or not", for
>>> every commit, and that's a very hard call, as is seen by this specific
>>> issue.
>
> Especially for networking stuff or things related to local resource usage
> where some people consider it represents a local DoS risk and others
> consider that it's just irrelevant to their servers since they have no
> local users.
Right. I try to pick those, but agree that the whole "security commit" term
is (very) vaguely defined.
>> The stable stuff isn't rare as you might think, even more: the amount of
>> actual CVE fixes that are not in the stable tree might surprise you.
>
> I would personally not be surprized since Ben used to feed me with a lot
> of fixes I had never seen previously. What is unclear to me is if your
> tree will contain only a selection of patches that are already in the
> respective branches, or a backport of security fixes that we can pick
> from to feed our stable branches and limit the risk of missing them.
> *This* actually could be useful to everyone, starting from our users.
The plan is to have only a subset of the stable tree, you won't see any
new commits here.
I agree it would be useful to have a better way of identifying security
commits and making sure that they all get applied, but unfortunately I'm
in the same boat as the rest of the maintainers and don't have a surefire
way of getting it done.
Maybe we can spin off a lts-security@ mailing list just for patches, and
make sure that every patch we need to apply gets sent there by someone
on that list?
> (...)
>> This is actually what happens now; projects get to the point they don't
>> want to update their whole kernel tree anymore so that just freezes because
>> they don't want to re-validate the whole thing over and over, but they
>> still cherry pick upstream and out-of-tree commits that they care about.
>>
>> If they added a handful of security commits to cherry pick and carefully
>> review their security will be much better than what happens now.
>
> Actually I do think that for end users it's a regression. People will
> start reusing outdated kernels which only contain the most critical fixes
> known, but will still suffer from memory leaks, deadlocks, kernel panics,
> data corruption etc. Every single bug that doesn't have a CVE attached to
> it in fact, which means 99% of the bugs that bring a system down in
> production. It makes me think about people who only pick security fixes
> from openssl and not the regular batch of missing null checks, and who
> complain all the time that their systems are unstable while they simply
> don't apply fixes.
Right, but once the system is up and running it's hard to convince someone
to take in ~100 commits that may or may not fix or introduce a bug.
If they are capable of cherry picking exactly the commits they want, then
it's perfect. Otherwise, the kernel doesn't go anywhere.
> You see, when I started with the "hotfix" tree 11 years ago for kernel
> 2.4, I intented to only pick the most critical fixes, they would fit
> in just a README and were counted on one hand. One year later there
> were 150 just because everything becomes critical for *some* workloads.
>
> I *do* think that having a central reference for fixes that come with
> a reproducer (hence many security fixes) can be useful as it would
> offer an opportunity for better testing backports when they become
> tricky : it often takes much more time to try to set up a test with
> a reproducer than it takes to backport and adjust the fix (not always
> true). But when it comes to security issues often the reporter cares
> about the quality of the backport and helps there.
Agreed, but I'm not sure how we can pull it off.
Thanks,
Sasha
On Mon, Apr 11, 2016 at 06:48:32PM -0400, Sasha Levin wrote:
> As this suggests, we don't only fix bugs in stable trees, we also introduce
> a fair amount of new ones.
Sure, and the older the kernel, the higher the number of bugs reintroduced
due to a faulty backport (often relying on a missing patch). I tell every
possible user that the quality improves over time until a point where more
bugs get added than fixed, which is where it's time to stop. I almost reached
that point in 2.4 and I started to replace fixes by documentation for existing
bugs that would not be fixed :-/
> >> Look at the opposite side of this question: why would anyone take a commit
> >> that fixes a bug he doesn't care about? Are the benefits really worth it
> >> considering the risks?
> >
> > That's exactly what most people do. I don't update to each and every kernel.
> > When I see xen, lvm, drm and audio changes I don't need them in my products.
> > But when I'm seeing network fixes I study them and often decide that it's
> > worth upgrading. Sometimes I pick a single fix from the queue because I can't
> > wait for next release. Many of Greg's kernels more or less focus on certain
> > topics, probably due to the way he deals with his mailbox and patch storms,
> > so it's often easy to quickly decide if you're going to need to update or
> > not.
>
> I fully agree with you here, if you can hand pick the commits you want and
> study every commit that goes in I can't really argue with that.
I really don't study them, I look at the list and say "OK nothing interesting
here, let's skip it". I only pick a fix when I have recently been notified of
a bug that I want fixed in our products.
> But the target audience here is different. I'm thinking about projects where
> the kernel isn't even close to being center stage, and it doesn't interest
> anyone as long as things work. There is no chance in hell that they would
> start picking commits unless you hold a gun to their hand.
Definitely, I agree. But the problem I'm having is that often they don't
care at all about bugs, they *believe* that security bugs are the only
ones to care about. You can force them to upgrade by reminding them that
the kernel they're running is full of CVE that need to be fixed. Once they
have the option to fix CVEs only and ignore real bugs, you won't be able
to force them to upgrade to get their bugs fixed.
> My thinking here is that it's easier to convince them to take a handful of
> patches in rather than hundreds.
I tend to think that the term "security" in the tree's name is misleading
the users into thinking that what they find there is enough for them. Most
incompetent admins think they need only "security" fixes. Maybe you should
use the term "minimal" or something like this to indicate that this is the
smallest set of fixes that should absolutely be running on any system.
Because quite frankly a filesystem corruption has much more impact on an
NFS server than a local info leak (which is often tagged as security).
> Right. I try to pick those, but agree that the whole "security commit" term
> is (very) vaguely defined.
Absolutely. For me, a bug becomes a security bug if it impacts the system's
stability, integrity or confidentiality in a way that can be triggered by a
non-authorized user more often than what would otherwise happen. Thus it
solely depends on the surrounding environment. For example in our products
we don't use DNS at all but regardless we fixed the recent glibc getaddrinfo
bug because it took less energy to apply the fix than to explain to customers
why they weren't at risk.
> The plan is to have only a subset of the stable tree, you won't see any
> new commits here.
OK.
> I agree it would be useful to have a better way of identifying security
> commits and making sure that they all get applied, but unfortunately I'm
> in the same boat as the rest of the maintainers and don't have a surefire
> way of getting it done.
>
> Maybe we can spin off a lts-security@ mailing list just for patches, and
> make sure that every patch we need to apply gets sent there by someone
> on that list?
I'm not sure it would be that useful. I guess that if we had a central
list somewhere listing the upstream commits that should be tagged as
such in an easily searchable way (ie by scripts), it would be more useful.
We could even amend the commit messages to mention the CVE id once it's
assigned, and sometimes "relies on this preliminary patch". That's a
tough work though, I don't know who would be interested in working on
this.
> > (...)
> >> This is actually what happens now; projects get to the point they don't
> >> want to update their whole kernel tree anymore so that just freezes because
> >> they don't want to re-validate the whole thing over and over, but they
> >> still cherry pick upstream and out-of-tree commits that they care about.
> >>
> >> If they added a handful of security commits to cherry pick and carefully
> >> review their security will be much better than what happens now.
> >
> > Actually I do think that for end users it's a regression. People will
> > start reusing outdated kernels which only contain the most critical fixes
> > known, but will still suffer from memory leaks, deadlocks, kernel panics,
> > data corruption etc. Every single bug that doesn't have a CVE attached to
> > it in fact, which means 99% of the bugs that bring a system down in
> > production. It makes me think about people who only pick security fixes
> > from openssl and not the regular batch of missing null checks, and who
> > complain all the time that their systems are unstable while they simply
> > don't apply fixes.
>
> Right, but once the system is up and running it's hard to convince someone
> to take in ~100 commits that may or may not fix or introduce a bug.
The problem works two ways :
- if a subsystem is not relevant to some users, they won't be interested
by a fix for this subsystem, just like they won't be affected by a risk
of regression ;
- if this subsystem is relevant to them, sure they risk a regression but
they also need to apply fixes otherwise they *know* they're running
bogus code.
I think we may have more of an issue educating our users than an issue
with the code we distribute.
Willy
On Tue, Apr 12, 2016 at 08:22:37AM +0200, Willy Tarreau wrote:
> I think we may have more of an issue educating our users than an issue
> with the code we distribute.
I think that's the issue here, combined with users that just don't want
to upgrade as it's "hard" for them. There's nothing we can do about
making it easier than we already do, and really, taking 200 patches is
just the same work for them to take 100 patches in a release, so I
don't buy the "I only want a small subset of fixes" argument, as it
doesn't make any sense.
So I worry that this tree will give people a _false_ sense of security,
thinking that they have properly covered the needed fixes when really,
there are lots of fixes they didn't even know they needed that got
applied to the "normal" stable tree for issues they will hit.
Also, people need to stop being afraid of the large numbers of stable
patches, and compare it to the overall % of changes, and realize that
what we take in stable releases is really a trivially small number of
the overall changes made to the kernel. So much so that you might argue
that it's safer to run the real kernel releases and not rely on stable
releases :)
thanks,
greg k-h
On Mon, Apr 11, 2016 at 11:35:08PM -0700, Greg KH wrote:
> On Tue, Apr 12, 2016 at 08:22:37AM +0200, Willy Tarreau wrote:
> > I think we may have more of an issue educating our users than an issue
> > with the code we distribute.
>
> I think that's the issue here, combined with users that just don't want
> to upgrade as it's "hard" for them. There's nothing we can do about
> making it easier than we already do, and really, taking 200 patches is
> just the same work for them to take 100 patches in a release, so I
> don't buy the "I only want a small subset of fixes" argument, as it
> doesn't make any sense.
I've already met people who used to estimate the need for each patch
individually. In the end they face many more problems in production
than any other user just because they drop the important ones or those
that enable certain "security" fixes to work reliably. Most often it's
a way for them to secure their job because nobody else can do this work
around them. When they leave the team, their boss generally decides to
go to regular distros with their simple update process in the end.
> So I worry that this tree will give people a _false_ sense of security,
> thinking that they have properly covered the needed fixes when really,
> there are lots of fixes they didn't even know they needed that got
> applied to the "normal" stable tree for issues they will hit.
Also the stable tree gets fixed thanks to feedback from users. When we
do something wrong it doesn't last long.
> Also, people need to stop being afraid of the large numbers of stable
> patches, and compare it to the overall % of changes, and realize that
> what we take in stable releases is really a trivially small number of
> the overall changes made to the kernel.
Yep. As a rule of thumb, I'd rather recommend people who fear updates to
always stay late by 2-3 versions and monitor the more recent changelogs
for possible fixes for previous versions. That might possibly make them
more comfortable. It's not a good practice but it's already better than
not applying the required fixes at all.
And let me restate it, most of us are our own users. We *are* careful.
My company ships load balancers which achieve multi-year uptimes (when
people don't reboot to apply fixes). We don't want to mess up with a
faulty update there. Ben gets his work distributed to all debian users
and certainly doesn't want to take risks either.
Willy
I'd like to add my 2c here as a mere user of many of the stable trees
for many years in my projects.
The stable trees are excellent. The quality of the people selecting
patches and the selection process is excellent. I've rarely been on the
bad end of a regression. I've always used stable kernels in virtually
all my own projects, as well as many clients' projects.
I agree with a lot of what Greg and Willy have said here. There are
plenty of examples of quite serious, non-security related (yes I know
defining security/non-security is problematic) bugs being fixed in
stable releases. IMO you deserve everything you get if you only applied
the fixes in Sasha's new tree and ignored the stable releases completely.
None-the-less, I applaud and thank Sasha for this new effort, and I
personally will find it very useful. Yes, the lines between bug fix and
security fix are very blurred, and so this tree won't have every
"security" fix. But I believe and trust it *will* at least contain fixes
for bugs that have the most severe security impact.
Where I will find this very useful is in having a "place" where I can
see what are probably the most important security fixes applicable to
the stable trees I am interested in. Because if I may offer one
criticism of the kernel stable trees in general, it is that it is very
hard to find and identify fixes for known security vulnerabilities.
Whenever I want to update the kernel in one of my projects, I find
myself having to hunt around a lot for information, stringing together
bits from bug reports, mailing lists, git commits, to track down whether
or not a particular vulnerability is fixed in a stable tree. Not
always, sometimes it is very clear that a particular fix in a particular
stable release fixes a known vulnerability, especially with commits e.g.
referencing CVEs in the header or commit message. At other times there
might be absolutely nothing in the fix to indicate this fixes a known
vulnerability.
So anything which improves visibility, which this certainly does, is a
good thing in my opinion.
Eddie
Hi Eddie,
On Tue, Apr 12, 2016 at 01:31:21PM +0100, Eddie Chapman wrote:
> None-the-less, I applaud and thank Sasha for this new effort, and I
> personally will find it very useful. Yes, the lines between bug fix and
> security fix are very blurred, and so this tree won't have every "security"
> fix. But I believe and trust it *will* at least contain fixes for bugs that
> have the most severe security impact.
It will only contain them if they are already in the respective stable trees,
which means that when I miss a fix (common), it won't appear there either.
At first I thought "oh cool, a repository of known things that must absolutely
be fixed, that will help me do my backports" and in the end I fear it will be
blindly used by end users who don't understand what they're missing but who
still believe they limit the risk of upgrades. Just this morning I saw a
report of a user saying that haproxy crashes is 2.6.24 kernel which is
"otherwise perfectly stable and achieves multi-years uptime"... Imagine
what such users will do when backporting fixes into they multi-thousands-bugs
kernel!
> Where I will find this very useful is in having a "place" where I can see
> what are probably the most important security fixes applicable to the stable
> trees I am interested in. Because if I may offer one criticism of the
> kernel stable trees in general, it is that it is very hard to find and
> identify fixes for known security vulnerabilities. Whenever I want to update
> the kernel in one of my projects, I find myself having to hunt around a lot
> for information, stringing together bits from bug reports, mailing lists,
> git commits, to track down whether or not a particular vulnerability is
> fixed in a stable tree. Not always, sometimes it is very clear that a
> particular fix in a particular stable release fixes a known vulnerability,
> especially with commits e.g. referencing CVEs in the header or commit
> message. At other times there might be absolutely nothing in the fix to
> indicate this fixes a known vulnerability.
I agree with this and it's not inherent to the stable trees but to the fact
that vulnerability IDs are assigned via a parallel process and often after
a fix gets merged, so the link between the commit and the CVE ID is easily
lost. Yes a central repo of known bugs by kernel branch and the commits
which fix them would be immensely useful including to the maintainers, but
it's a huge work and I hardly see who would work on such a thing. Note that
it doesn't necessarily need to focus on security only. Any painful bug should
be referenced as present first, then fixed with the relevant commit IDs once
the backports are merged. The "fixes" tag in upstream commits is already a
step forward, but in any case we'd need post-documentation since it regularly
happens that a bug happens to be accidently fixed by some later changes.
Regards,
Willy
On 12/04/16 13:52, Willy Tarreau wrote:
> On Tue, Apr 12, 2016 at 01:31:21PM +0100, Eddie Chapman wrote:
>> None-the-less, I applaud and thank Sasha for this new effort, and I
>> personally will find it very useful. Yes, the lines between bug fix and
>> security fix are very blurred, and so this tree won't have every "security"
>> fix. But I believe and trust it *will* at least contain fixes for bugs that
>> have the most severe security impact.
>
> It will only contain them if they are already in the respective stable trees,
> which means that when I miss a fix (common), it won't appear there either.
> At first I thought "oh cool, a repository of known things that must absolutely
> be fixed, that will help me do my backports" and in the end I fear it will be
> blindly used by end users who don't understand what they're missing but who
> still believe they limit the risk of upgrades. Just this morning I saw a
> report of a user saying that haproxy crashes is 2.6.24 kernel which is
> "otherwise perfectly stable and achieves multi-years uptime"... Imagine
> what such users will do when backporting fixes into they multi-thousands-bugs
> kernel!
Yes, agreed. I'm sure it's not going to be an exhaustive, complete
repository. But I think it is better than no repository. And I think
you are right there is a risk some will use it blindly. However, it
seems to me if they make this mistake then by definition they were
probably already screwed with regards the issues we're discussing here,
so things can't really effectively get any worse. So by blindly applying
a small tree of what have already been regarded as important fixes their
situation might then be upgraded from 100% screwed to maybe only 70%
screwed.