Subject: Re: [ANNOUNCE] linux-stable security tree
To: Greg KH <gregkh@linuxfoundation.org>
References: <570BE4A5.20200@oracle.com> <20160411184148.GA23140@kroah.com>
 <570BF3DD.2060900@oracle.com> <20160411200904.GB24106@kroah.com>
Cc: LKML <linux-kernel@vger.kernel.org>, stable <stable@vger.kernel.org>,
        lwn@lwn.net
From: Sasha Levin <sasha.levin@oracle.com>
Message-ID: <570C0B39.1090408@oracle.com>
Date: Mon, 11 Apr 2016 16:38:17 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
In-Reply-To: <20160411200904.GB24106@kroah.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 4060
Lines: 89

On 04/11/2016 04:09 PM, Greg KH wrote:
> On Mon, Apr 11, 2016 at 02:58:37PM -0400, Sasha Levin wrote:
>> On 04/11/2016 02:41 PM, Greg KH wrote:
>>> On Mon, Apr 11, 2016 at 01:53:41PM -0400, Sasha Levin wrote:
>>>> Quite a few users of the stable trees pointed out that on complex deployments,
>>>> where validation is non-trivial, there is little incentive to follow the
>>>> stable tree after the product has been deployed to production. There is no
>>>> interest in "random" kernel fixes and the only requirements are to keep up
>>>> with security vulnerabilities.
>>>
>>> "random" is in the eye of the beholder :)
>>
>> It's not "random" for us building the tree, but it's very much "random"
>> for the users for see fixes for drivers they've never even heard of before.
> 
> Then why would it matter if they pulled the tree or not?  How are you
> going to judge which driver fixes to take and which not to?  Why not
> take them all if they fix bugs?

Because some fixes introduce bug on their own? Take a look at how many
commits in the stable tree have a "Fixes:" tag that points to a commit
that's also in the stable tree.

Look at the opposite side of this question: why would anyone take a commit
that fixes a bug he doesn't care about? Are the benefits really worth it
considering the risks?

[snip]

>>> Define "important".  Now go and look at the tty bug we fixed that people
>>> only realized was "important" 1 1/2 years later and explain if you
>>> would, or would not have, taken that patch in this tree.
>>
>> Probably not, but I would have taken it after it received a CVE number.
>>
>> Same applies to quite a few commits that end up in stable - no one thinks
>> they're stable material at first until someone points out it's crashing
>> his production boxes for the past few months.
> 
> Yes, but those are rare, what you are doing here is suddenly having to
> judge if a bug is a "security" issue or not.  You are now in the
> position of trying to determine "can this be exploited or not", for
> every commit, and that's a very hard call, as is seen by this specific
> issue.

The stable stuff isn't rare as you might think, even more: the amount of
actual CVE fixes that are not in the stable tree might surprise you.

This usually happens for the reason you described, we look at a commit
that has an innocent subject/description, not CC'ed to stable@ and we figure
that it's not stable material until someone shows how to exploit it and
requests a CVE.

This is not rare.

> If you only take things you "know" are issues, well, you miss lots of
> fixes that were really security things but only found out later, so
> people have to scamble to fix their kernels much faster than they needed
> to.

I don't only take known issues. I don't claim to have a 100% success rate,
but this is the same story as the stable tree and the patch selection there.

> Putting up a tree like this isn't going to cause people to update their
> trees any quicker.  If they haven't been doing it now, they aren't going
> to do it with this tree, as their workloads don't allow them to take
> updated kernel trees.
> 
> In short, it's not the fact that we have stable trees that are "too big"
> for them to take, it's the fact that they just can't take and test
> updates properly.  They need to fix their processes, it's not a
> deficiency on our side from everyone I have talked to, including the
> example you gave me off-list :)

Pulling 100+ "random" (yes, I used it again) commits would require a very
different review process than cherry picking ~5 commits that you can more
easily review and validate.

This is actually what happens now; projects get to the point they don't
want to update their whole kernel tree anymore so that just freezes because
they don't want to re-validate the whole thing over and over, but they
still cherry pick upstream and out-of-tree commits that they care about.

If they added a handful of security commits to cherry pick and carefully
review their security will be much better than what happens now.


Thanks,
Sasha