2005-03-09 07:29:05

by Greg KH

[permalink] [raw]
Subject: [RFC] -stable, how it's going to work.

So here's a first cut at how this 2.6 -stable release process is going
to work that Chris and I have come up with. Does anyone have any
problems/issues/questions with this?

thanks,

greg k-h

-------------------

Everything you ever wanted to know about Linux 2.6 -stable releases.


Rules on what kind of patches are accepted, and what ones are not, into
the "-stable" tree:
- It must be obviously correct and tested.
- It can not bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a
problem..." type thing.)
- It must fix a problem that causes a build error (but not for things
marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
security issue, or some "oh, that's not good" issue. In short,
something critical.
- No "theoretical race condition" issues, unless an explanation of how
the race can be exploited.
- It can not contain any "trivial" fixes in it (spelling changes,
whitespace cleanups, etc.)
- It must be accepted by the relevant subsystem maintainer.
- It must follow Documentation/SubmittingPatches rules.

Procedure for submitting patches to the -stable tree:
- Send the patch, after verifying that it follows the above rules, to
[email protected].
- The sender will receive an ack when the patch has been accepted into
the queue, or a nak if the patch is rejected. This response might
take a few days, according to the developer's schedules.
- If accepted, the patch will be added to the -stable queue, for review
by other developers.
- Security patches should not be sent to this alias, but instead to the
documented [email protected].

Review cycle:
- When the -stable maintainers decide for a review cycle, the patches
will be sent to the review committee, and the maintainer of the
affected area of the patch (unless the submitter is the maintainer of
the area) and CC: to the linux-kernel mailing list.
- The review committee has 48 hours in which to ack or nak the patch.
- If the patch is rejected by a member of the committee, or linux-kernel
members object to the patch by bringing up issues that the maintainer
and members did not realize, the patch will be dropped from the
queue.
- At the end of the review cycle, the acked patches will be added to
the latest -stable release, and a new -stable release will happen.
- Security patches will be accepted into the -stable tree directly from
the security kernel team, and not go through the normal review cycle.
Contact the kernel security team for more details on this procedure.

Review committe:
- This will be made up of a number of kernel developers who have
volunteered for this task, and a few that haven't.


2005-03-09 09:57:33

by Andi Kleen

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

Greg KH <[email protected]> writes:
>
> Rules on what kind of patches are accepted, and what ones are not, into
> the "-stable" tree:
> - It must be obviously correct and tested.
> - It can not bigger than 100 lines, with context.

This rule seems silly. What happens when a security fix needs 150 lines?

Better maybe a rule like "The patch should be the minimal and safest
change to fix an issue". But see below for an exception.

> - It must fix only one thing.
> - It must fix a real bug that bothers people (not a, "This could be a
> problem..." type thing.)
> - It must fix a problem that causes a build error (but not for things
> marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
> security issue, or some "oh, that's not good" issue. In short,
> something critical.
> - No "theoretical race condition" issues, unless an explanation of how
> the race can be exploited.
> - It can not contain any "trivial" fixes in it (spelling changes,
> whitespace cleanups, etc.)
> - It must be accepted by the relevant subsystem maintainer.



> - It must follow Documentation/SubmittingPatches rules.

One rule I'm missing:

- It must be accepted to mainline.

That is what big enterprise distributions often require and I think
it's a good rule. Otherwise you risk code and feature set drift
and we don't want to repeat the 2.4 mistakes again where some
subsystems had more fixes in 2.4 than 2.6.

Also your rules encourage to do different patches for -stable
(e.g. with less comment changes etc.) than for mainline. I don't
think that's a very good thing. Sometimes it is unavoidable
and sometimes the mainline patches are just too big and intrusive,
but in general it's imho best to apply the same patches
to mainline and backport trees. This has also the advantage
that the patch is best tested as possible; slimmed down patches
usually have a risk of malfunction.

If a mainline patch violates too many of your other rules
("Fixes one thing; doesn't do cosmetic changes etc.") perhaps
the mainline patch just needs to be improved.

So in general there should be a preference to apply the same
patch as mainline, unless it is very big.

> - Security patches will be accepted into the -stable tree directly from
> the security kernel team, and not go through the normal review cycle.
> Contact the kernel security team for more details on this procedure.

This also sounds like a bad rule. How come the security team has more
competence to review patches than the subsystem maintainers? I can
see the point of overruling maintainers on security issues when they
are not responsive, but if they are I think the should be still the
main point of contact.

-Andi

2005-03-09 10:11:20

by Arjan van de Ven

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, 2005-03-09 at 10:56 +0100, Andi Kleen wrote:
> One rule I'm missing:
>
> - It must be accepted to mainline.
>

I absolutely agree with Andi on this one.


> If a mainline patch violates too many of your other rules
> ("Fixes one thing; doesn't do cosmetic changes etc.") perhaps
> the mainline patch just needs to be improved.
>

I can see this as getting an exception occasionally, but it should be a
well thought out exception and not a general rule

> > - Security patches will be accepted into the -stable tree directly from
> > the security kernel team, and not go through the normal review cycle.
> > Contact the kernel security team for more details on this procedure.
>
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers? I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

yeah; the security patch is public anyway, so why not have the regular
review on it as well? Why would such a patch be special?


2005-03-09 10:18:06

by Russell King

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 11:10:59AM +0100, Arjan van de Ven wrote:
> On Wed, 2005-03-09 at 10:56 +0100, Andi Kleen wrote:
> > One rule I'm missing:
> >
> > - It must be accepted to mainline.
> >
>
> I absolutely agree with Andi on this one.

What about the case (as highlighted in previous discussions) that the
stable tree needs a simple "dirty" fix, whereas mainline takes the
complex "clean" fix?

>From what's been said above, this means that the stable tree doesn't
get the simple "dirty" fix, because that isn't what's been accepted
into mainline. However, it could take the complex "clean" fix, but
that may have too high an impact.

--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core

2005-03-09 10:25:32

by Arjan van de Ven

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, 2005-03-09 at 10:17 +0000, Russell King wrote:
> On Wed, Mar 09, 2005 at 11:10:59AM +0100, Arjan van de Ven wrote:
> > On Wed, 2005-03-09 at 10:56 +0100, Andi Kleen wrote:
> > > One rule I'm missing:
> > >
> > > - It must be accepted to mainline.
> > >
> >
> > I absolutely agree with Andi on this one.
>
> What about the case (as highlighted in previous discussions) that the
> stable tree needs a simple "dirty" fix, whereas mainline takes the
> complex "clean" fix?

that's the exception I talked about a bit later in my mail. It should be
rare and very deliberate. And in fact once the mainline change ripples
out into maturity I rather replace the -stable one with that later on,
even if it's a bit more invasive.

2005-03-09 10:29:00

by Andi Kleen

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 10:17:28AM +0000, Russell King wrote:
> On Wed, Mar 09, 2005 at 11:10:59AM +0100, Arjan van de Ven wrote:
> > On Wed, 2005-03-09 at 10:56 +0100, Andi Kleen wrote:
> > > One rule I'm missing:
> > >
> > > - It must be accepted to mainline.
> > >
> >
> > I absolutely agree with Andi on this one.
>
> What about the case (as highlighted in previous discussions) that the
> stable tree needs a simple "dirty" fix, whereas mainline takes the
> complex "clean" fix?

That's ok, as long as the maintainers agree it's an equivalent fix

What should just be ruled out is something getting fixed in stable
and not getting fixed in mainline. And the mainline patch should
always go in first and preferably tested for a short time there.

But in general it is a judgement call: if the "big fix" is not too big
or risky I would prefer the big fix just to avoid code drift.
A custom fix for stable should be more the exception than the rule.

-Andi

2005-03-09 10:36:03

by Russell King

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 11:24:58AM +0100, Arjan van de Ven wrote:
> On Wed, 2005-03-09 at 10:17 +0000, Russell King wrote:
> > What about the case (as highlighted in previous discussions) that the
> > stable tree needs a simple "dirty" fix, whereas mainline takes the
> > complex "clean" fix?
>
> that's the exception I talked about a bit later in my mail. It should be
> rare and very deliberate. And in fact once the mainline change ripples
> out into maturity I rather replace the -stable one with that later on,
> even if it's a bit more invasive.

Is that really necessary with a stable tree which may only be around
for about 2 months before the next stable tree is forked (which would
have the mature mainline fix in) ?

There is another point here though, which I think is much more important.
Remember that the original issue which caused the -stable tree to be
created was a concern over the testing that Linus' kernels were getting.
Also, realise that by creating a -stable tree, we haven't increased the
number of testers which Linus' kernels are seeing.

Given that, how can we decide that a complex fix has matured enough
in Linus' kernel to warrant replacing a (proven) fix which users are
perfectly happy with in the corresponding -stable tree?

I thought the -stable tree is targeted towards stability, not towards
"lets replace this change with some other because we as developers think
it's better".

--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core

2005-03-09 18:05:48

by Alan

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Mer, 2005-03-09 at 09:56, Andi Kleen wrote:
> - It must be accepted to mainline.

Strongly disagree. What if the mainline fix is a rewrite of the core API
involved. Some times you need to put in the short term fix. What must
never happen is people accepting that fix as long term.

How about

- It must be accepted to mainline, or the accepted mainline patch be
deemed too complex or risky to backport and thus a simple obvious
alternative fix applied to stable ONLY.

2005-03-09 18:32:38

by Chris Wright

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

* Andi Kleen ([email protected]) wrote:
> Greg KH <[email protected]> writes:
> >
> > Rules on what kind of patches are accepted, and what ones are not, into
> > the "-stable" tree:
> > - It must be obviously correct and tested.
> > - It can not bigger than 100 lines, with context.
>
> This rule seems silly. What happens when a security fix needs 150 lines?
>
> Better maybe a rule like "The patch should be the minimal and safest
> change to fix an issue". But see below for an exception.

It's just a guideline to scope the work. But a fixed size is probably
less meaningful than your wording.

> > - It must fix only one thing.
> > - It must fix a real bug that bothers people (not a, "This could be a
> > problem..." type thing.)
> > - It must fix a problem that causes a build error (but not for things
> > marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
> > security issue, or some "oh, that's not good" issue. In short,
> > something critical.
> > - No "theoretical race condition" issues, unless an explanation of how
> > the race can be exploited.
> > - It can not contain any "trivial" fixes in it (spelling changes,
> > whitespace cleanups, etc.)
> > - It must be accepted by the relevant subsystem maintainer.
>
> > - It must follow Documentation/SubmittingPatches rules.
>
> One rule I'm missing:
>
> - It must be accepted to mainline.

This can violate the principle of keeping fixes simple for -stable tree.
And Linus/Andrew don't want to litter mainline with patch series that
do simple fix followed by complete fix meant for developement branch.

> That is what big enterprise distributions often require and I think
> it's a good rule. Otherwise you risk code and feature set drift
> and we don't want to repeat the 2.4 mistakes again where some
> subsystems had more fixes in 2.4 than 2.6.

I agree, it's a good rule, but these should be small, temporal diffs
from mainline. For example, -ac tree will sometimes do the simpler fix,
whereas mainline does proper complete fix.

> Also your rules encourage to do different patches for -stable
> (e.g. with less comment changes etc.) than for mainline. I don't
> think that's a very good thing. Sometimes it is unavoidable
> and sometimes the mainline patches are just too big and intrusive,
> but in general it's imho best to apply the same patches
> to mainline and backport trees. This has also the advantage
> that the patch is best tested as possible; slimmed down patches
> usually have a risk of malfunction.
>
> If a mainline patch violates too many of your other rules
> ("Fixes one thing; doesn't do cosmetic changes etc.") perhaps
> the mainline patch just needs to be improved.

Good point.

> So in general there should be a preference to apply the same
> patch as mainline, unless it is very big.

Agreed.

> > - Security patches will be accepted into the -stable tree directly from
> > the security kernel team, and not go through the normal review cycle.
> > Contact the kernel security team for more details on this procedure.
>
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers? I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

They don't, the security patches should still be reviewed by subsystem
maintainer. Point here is, sometimes there's disclosure coordination
happening as well.

thanks,
-chris
--
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net

2005-03-09 18:36:23

by Chris Wright

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

* Alan Cox ([email protected]) wrote:
> On Mer, 2005-03-09 at 09:56, Andi Kleen wrote:
> > - It must be accepted to mainline.
>
> Strongly disagree. What if the mainline fix is a rewrite of the core API
> involved. Some times you need to put in the short term fix. What must
> never happen is people accepting that fix as long term.
>
> How about
>
> - It must be accepted to mainline, or the accepted mainline patch be
> deemed too complex or risky to backport and thus a simple obvious
> alternative fix applied to stable ONLY.

Yes.

2005-03-09 18:41:15

by Greg KH

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 10:56:33AM +0100, Andi Kleen wrote:
> Greg KH <[email protected]> writes:
> >
> > Rules on what kind of patches are accepted, and what ones are not, into
> > the "-stable" tree:
> > - It must be obviously correct and tested.
> > - It can not bigger than 100 lines, with context.
>
> This rule seems silly. What happens when a security fix needs 150 lines?

Then we bend the rules and accept it :)

We'll take these as a case-by-case basis...

> > - Security patches will be accepted into the -stable tree directly from
> > the security kernel team, and not go through the normal review cycle.
> > Contact the kernel security team for more details on this procedure.
>
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers? I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

Security fixes go from the security team to Linus's tree directly, and
usually the subsystem maintainer has already been notified and has
reviewedit. At that point in time, they are public and accepted into
mainline, and need to be made availble to the -stable users as soon as
possible.

That is why the "fast track" is going to happen, the patch really was
reviewed properly, just not in public :)

thanks,

greg k-h

2005-03-09 18:41:59

by Greg KH

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 06:00:45PM +0000, Alan Cox wrote:
> On Mer, 2005-03-09 at 09:56, Andi Kleen wrote:
> > - It must be accepted to mainline.
>
> Strongly disagree. What if the mainline fix is a rewrite of the core API
> involved. Some times you need to put in the short term fix. What must
> never happen is people accepting that fix as long term.
>
> How about
>
> - It must be accepted to mainline, or the accepted mainline patch be
> deemed too complex or risky to backport and thus a simple obvious
> alternative fix applied to stable ONLY.

Yes, that's acceptable, I agree that Andi's statment is not ok (see the
.1 release for 2 patches that were accepted in a "simple" way because
the real fix was too complex.)

I'll go update the document with this.

thanks,

greg k-h

2005-03-09 18:51:28

by Marcelo Tosatti

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.


> > - Security patches will be accepted into the -stable tree directly from
> > the security kernel team, and not go through the normal review cycle.
> > Contact the kernel security team for more details on this procedure.
>
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers? I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

The security team is going to work with the subsystem maintainers,
not overrule them.

That would be indeed insane.

2005-03-09 19:31:30

by Andi Kleen

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 06:00:45PM +0000, Alan Cox wrote:
> On Mer, 2005-03-09 at 09:56, Andi Kleen wrote:
> > - It must be accepted to mainline.
>
> Strongly disagree. What if the mainline fix is a rewrite of the core API
> involved. Some times you need to put in the short term fix. What must
> never happen is people accepting that fix as long term.
>
> How about
>
> - It must be accepted to mainline, or the accepted mainline patch be
> deemed too complex or risky to backport and thus a simple obvious
> alternative fix applied to stable ONLY.

That is what I wrote later in my mail anyways (did you really read it completely?:) See also the followup discussion with Russel and Arjan.

In general stable specific fixes should be the exception, not the rule though.

-Andi

2005-03-09 19:49:26

by Andi Kleen

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 10:28:22AM -0800, Chris Wright wrote:
> * Andi Kleen ([email protected]) wrote:
> > Greg KH <[email protected]> writes:
> > >
> > > Rules on what kind of patches are accepted, and what ones are not, into
> > > the "-stable" tree:
> > > - It must be obviously correct and tested.
> > > - It can not bigger than 100 lines, with context.
> >
> > This rule seems silly. What happens when a security fix needs 150 lines?
> >
> > Better maybe a rule like "The patch should be the minimal and safest
> > change to fix an issue". But see below for an exception.
>
> It's just a guideline to scope the work. But a fixed size is probably
> less meaningful than your wording.
>
> > > - It must fix only one thing.
> > > - It must fix a real bug that bothers people (not a, "This could be a
> > > problem..." type thing.)
> > > - It must fix a problem that causes a build error (but not for things
> > > marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
> > > security issue, or some "oh, that's not good" issue. In short,
> > > something critical.
> > > - No "theoretical race condition" issues, unless an explanation of how
> > > the race can be exploited.
> > > - It can not contain any "trivial" fixes in it (spelling changes,
> > > whitespace cleanups, etc.)
> > > - It must be accepted by the relevant subsystem maintainer.
> >
> > > - It must follow Documentation/SubmittingPatches rules.
> >
> > One rule I'm missing:
> >
> > - It must be accepted to mainline.
>
> This can violate the principle of keeping fixes simple for -stable tree.
> And Linus/Andrew don't want to litter mainline with patch series that
> do simple fix followed by complete fix meant for developement branch.

But it risks code drift like we had in 2.4 with older kernels
having more fixes than the newer kernel. And that way lies madness.

I think it is very very important to avoid this.

If you prefer you can rewrite the rule like

"Fix must in mainline first. In exceptional cases when the fix
in mainline is too intrusive or risky a simpler version of the patch
can be applied to stable. In this case the mainline fix must be already
accepted. For most cases the full fix should be applied to avoid code drift"


> I agree, it's a good rule, but these should be small, temporal diffs
> from mainline. For example, -ac tree will sometimes do the simpler fix,
> whereas mainline does proper complete fix.

You make it sound like all patches are super complicated and
not suitable for backporting.

>From my experiences maintaining distribution kernel most mainline changes
can be just completely backported.

>
> > > the security kernel team, and not go through the normal review cycle.
> > > Contact the kernel security team for more details on this procedure.
> >
> > This also sounds like a bad rule. How come the security team has more
> > competence to review patches than the subsystem maintainers? I can
> > see the point of overruling maintainers on security issues when they
> > are not responsive, but if they are I think the should be still the
> > main point of contact.
>
> They don't, the security patches should still be reviewed by subsystem
> maintainer. Point here is, sometimes there's disclosure coordination
> happening as well.

Ok, how does it coordinate with the vendor-sec process?
And at what point is the subsystem maintainer notified.

The security thing seems to be still quite half backed to me...

-Andi

2005-03-09 20:10:01

by Greg KH

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 08:39:36PM +0100, Andi Kleen wrote:
> On Wed, Mar 09, 2005 at 10:34:08AM -0800, Greg KH wrote:
> > On Wed, Mar 09, 2005 at 10:56:33AM +0100, Andi Kleen wrote:
> > > Greg KH <[email protected]> writes:
> > > >
> > > > Rules on what kind of patches are accepted, and what ones are not, into
> > > > the "-stable" tree:
> > > > - It must be obviously correct and tested.
> > > > - It can not bigger than 100 lines, with context.
> > >
> > > This rule seems silly. What happens when a security fix needs 150 lines?
> >
> > Then we bend the rules and accept it :)
> >
> > We'll take these as a case-by-case basis...
> >
> > > > - Security patches will be accepted into the -stable tree directly from
> > > > the security kernel team, and not go through the normal review cycle.
> > > > Contact the kernel security team for more details on this procedure.
> > >
> > > This also sounds like a bad rule. How come the security team has more
> > > competence to review patches than the subsystem maintainers? I can
> > > see the point of overruling maintainers on security issues when they
> > > are not responsive, but if they are I think the should be still the
> > > main point of contact.
> >
> > Security fixes go from the security team to Linus's tree directly, and
> > usually the subsystem maintainer has already been notified and has
> > reviewedit. At that point in time, they are public and accepted into
>
> What guarantees that?

The kernel security team's proceedures.

> Basically what I would like to avoid is that the security team
> merges something through the backdoor that the maintainer considers crap.
>
> If anything you should have a rule like
>
> "Send to maintainer. If he doesn't ACK in 24h send it directly"
>
>
> > mainline, and need to be made availble to the -stable users as soon as
> > possible.
> >
> > That is why the "fast track" is going to happen, the patch really was
> > reviewed properly, just not in public :)
>
> Well, when you really want to have such formal rules (which is a novelty in
> Linux space BTW, for many years we did fine with unwritten rules) then you
> should spell it out completely. Or alternatively drop all the formal
> rules and do it informally like it was always done.

I'd love to do it informally, but the rules are going to be used to make
our lives easier, by having something to point to when we want to reject
something, and having something that everyone can refer to when trying
to understand what it is we are attempting to do here.

If they get too complex, or large, we will have to revisit them.

So, let's stop arguing about the semantics of the rules, and see if what
we have proposed actually works in real-life. If that doesn't work out,
we can revisit it then.

thanks,

greg k-h

2005-03-09 21:23:52

by Chris Wright

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

* Andi Kleen ([email protected]) wrote:
> On Wed, Mar 09, 2005 at 10:28:22AM -0800, Chris Wright wrote:
> > * Andi Kleen ([email protected]) wrote:
> > > Greg KH <[email protected]> writes:

> > > One rule I'm missing:
> > >
> > > - It must be accepted to mainline.
> >
> > This can violate the principle of keeping fixes simple for -stable tree.
> > And Linus/Andrew don't want to litter mainline with patch series that
> > do simple fix followed by complete fix meant for developement branch.
>
> But it risks code drift like we had in 2.4 with older kernels
> having more fixes than the newer kernel. And that way lies madness.
>
> I think it is very very important to avoid this.
>
> If you prefer you can rewrite the rule like
>
> "Fix must in mainline first. In exceptional cases when the fix
> in mainline is too intrusive or risky a simpler version of the patch
> can be applied to stable. In this case the mainline fix must be already
> accepted. For most cases the full fix should be applied to avoid code drift"

I think we've all agreed that's the intention.

> > I agree, it's a good rule, but these should be small, temporal diffs
> > from mainline. For example, -ac tree will sometimes do the simpler fix,
> > whereas mainline does proper complete fix.
>
> You make it sound like all patches are super complicated and
> not suitable for backporting.

I didn't think I did, that's why I said 'sometimes'. Just acknowledging
what does really happen.

> > They don't, the security patches should still be reviewed by subsystem
> > maintainer. Point here is, sometimes there's disclosure coordination
> > happening as well.
>
> Ok, how does it coordinate with the vendor-sec process?
> And at what point is the subsystem maintainer notified.

That's part of the vendor coordination mentioned in the policy. And
subsystem maintainer is notified as part of vetting the issue/solution,
as stated in the policy.

> The security thing seems to be still quite half backed to me...

Take a look at the policy I posted last night and give me suggestions
for improvements.

thanks,
-chris
--
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net

2005-03-09 23:14:01

by Russell King

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 08:44:01PM +0100, Andi Kleen wrote:
> But it risks code drift like we had in 2.4 with older kernels
> having more fixes than the newer kernel. And that way lies madness.

I believe it's going to work like this:

* simple fixes are submitted to Linus and -stable, and are appropriately
merged.
* complex "correct" fixes too large for -stable are submitted to Linus,
with a simplified version for -stable.

When the next Linus kernel is released, anything in the previous -stable
series is effectively discarded, and the next -stable series is produced
from the new release point.

Obviously, the -stable sucker can continue with his existing -stable
series if he so wishes, but I would expect that any fixes in, eg,
2.6.11.x would not be propagated by the -stable sucker to 2.6.12.x.

This may be a good thing - it encourages people to get the fixes into
the Linus tree, thereby keeping the code drift to a minimum. Any
drift would only exist for one of these -stable branches, which may
only survive for maybe one Linus kernel release cycle.

--
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 Serial core

2005-03-09 21:23:53

by Bill Davidsen

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

http://localhost/blogAndi Kleen wrote:
> Greg KH <[email protected]> writes:
>
>>Rules on what kind of patches are accepted, and what ones are not, into
>>the "-stable" tree:
>> - It must be obviously correct and tested.
>> - It can not bigger than 100 lines, with context.
>
>
> This rule seems silly. What happens when a security fix needs 150 lines?
>
> Better maybe a rule like "The patch should be the minimal and safest
> change to fix an issue". But see below for an exception.

I was willing to accept that there might be an exploitable security
issue taking more than 100 lines to plug the hole (remember, check your
elegance at the door), and I like your suggestion regardless of length.
But I don't think any more exceptions are desirable. As you said, "see
below."
>
>> - It must fix only one thing.
>> - It must fix a real bug that bothers people (not a, "This could be a
>> problem..." type thing.)
>> - It must fix a problem that causes a build error (but not for things
>> marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
>> security issue, or some "oh, that's not good" issue. In short,
>> something critical.
>> - No "theoretical race condition" issues, unless an explanation of how
>> the race can be exploited.
>> - It can not contain any "trivial" fixes in it (spelling changes,
>> whitespace cleanups, etc.)
>> - It must be accepted by the relevant subsystem maintainer.

I have the feeling that drivers will need some special consideration,
like allowing additions to an existing blacklist to prevent "almost
working" with similar but not identical chips and the like.

> One rule I'm missing:
>
> - It must be accepted to mainline.
>
> That is what big enterprise distributions often require and I think
> it's a good rule. Otherwise you risk code and feature set drift
> and we don't want to repeat the 2.4 mistakes again where some
> subsystems had more fixes in 2.4 than 2.6.

Right problem, wrong solution. What you is that SOME solution go in
mainline. My example above of adding a chipset to a blacklist is a
perfect case, where you want the chipset to become supported, not long
term "officially unsupported." And band-aids like using the BKL or long
linear searches just to prevent something which causes a hard hang or
oops should not be blessed and included in mainline.

I know you understand this, I just think it's worth saying more clearly.
>
> Also your rules encourage to do different patches for -stable
> (e.g. with less comment changes etc.) than for mainline. I don't
> think that's a very good thing. Sometimes it is unavoidable
> and sometimes the mainline patches are just too big and intrusive,
> but in general it's imho best to apply the same patches
> to mainline and backport trees. This has also the advantage
> that the patch is best tested as possible; slimmed down patches
> usually have a risk of malfunction.

Different in function as well as style. Testing complex patches can be
done in -mm, reliable is often easier than optimal.

> So in general there should be a preference to apply the same
> patch as mainline, unless it is very big.

I can't imagine anyone not using a good patch in mainline, I can imagine
some really brute force patches in stable, because they are easier to
vat than a really correct solution.
>
>
>> - Security patches will be accepted into the -stable tree directly from
>> the security kernel team, and not go through the normal review cycle.
>> Contact the kernel security team for more details on this procedure.
>
>
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers? I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

I was actually thinking that the normal patch might want a fast path in
practice. Some things are clearly wrong, like off by one errors in array
handling and the like, using pointers to freed data or which could be
null, that kind of thing is pretty unsubtle, and I would think would
need fewer eyes-on. But until we see how this works in practice, let's
not fix it. The stable kernel should have a stable process, which should
not be changed unless there's a demonstrated problem.

--
-bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
last possible moment - but no longer" -me

2005-03-10 06:19:21

by Andi Kleen

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Wed, Mar 09, 2005 at 10:34:08AM -0800, Greg KH wrote:
> On Wed, Mar 09, 2005 at 10:56:33AM +0100, Andi Kleen wrote:
> > Greg KH <[email protected]> writes:
> > >
> > > Rules on what kind of patches are accepted, and what ones are not, into
> > > the "-stable" tree:
> > > - It must be obviously correct and tested.
> > > - It can not bigger than 100 lines, with context.
> >
> > This rule seems silly. What happens when a security fix needs 150 lines?
>
> Then we bend the rules and accept it :)
>
> We'll take these as a case-by-case basis...
>
> > > - Security patches will be accepted into the -stable tree directly from
> > > the security kernel team, and not go through the normal review cycle.
> > > Contact the kernel security team for more details on this procedure.
> >
> > This also sounds like a bad rule. How come the security team has more
> > competence to review patches than the subsystem maintainers? I can
> > see the point of overruling maintainers on security issues when they
> > are not responsive, but if they are I think the should be still the
> > main point of contact.
>
> Security fixes go from the security team to Linus's tree directly, and
> usually the subsystem maintainer has already been notified and has
> reviewedit. At that point in time, they are public and accepted into

What guarantees that?

Basically what I would like to avoid is that the security team
merges something through the backdoor that the maintainer considers crap.

If anything you should have a rule like

"Send to maintainer. If he doesn't ACK in 24h send it directly"


> mainline, and need to be made availble to the -stable users as soon as
> possible.
>
> That is why the "fast track" is going to happen, the patch really was
> reviewed properly, just not in public :)

Well, when you really want to have such formal rules (which is a novelty in
Linux space BTW, for many years we did fine with unwritten rules) then you
should spell it out completely. Or alternatively drop all the formal
rules and do it informally like it was always done.


-Andi

2005-03-10 10:01:10

by NeilBrown

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Tuesday March 8, [email protected] wrote:
> So here's a first cut at how this 2.6 -stable release process is going
> to work that Chris and I have come up with. Does anyone have any
> problems/issues/questions with this?

One rule that I thought would make sense, but that I don't see listed
is:

- must fix a regression

If some problem was in 2.6.X and is still there in 2.6.X+1, then there
is no great rush to fix it for 2.6.(X+1).1, it can wait for 2.6.(X+2).

Security issues could be an exception to this, or it could be seen
that the publication of a security vulnerability causes a regression
which needs to be fixed...

NeilBrown

2005-03-10 10:18:53

by Arjan van de Ven

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Thu, 2005-03-10 at 21:00 +1100, Neil Brown wrote:
> On Tuesday March 8, [email protected] wrote:
> > So here's a first cut at how this 2.6 -stable release process is going
> > to work that Chris and I have come up with. Does anyone have any
> > problems/issues/questions with this?
>
> One rule that I thought would make sense, but that I don't see listed
> is:
>
> - must fix a regression
>
> If some problem was in 2.6.X and is still there in 2.6.X+1, then there
> is no great rush to fix it for 2.6.(X+1).1, it can wait for 2.6.(X+2).

this is tricky. I mean, what if it's a datacorruption thing? Sure older
kernels may have it too... yet at the same time it'd be nice to get it
fixed. And once you go down this slipperly slope you end up with the
original ruleset again, eg it must be "somehow relatively important".


2005-03-10 16:50:40

by Greg KH

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Thu, Mar 10, 2005 at 09:00:51PM +1100, Neil Brown wrote:
> On Tuesday March 8, [email protected] wrote:
> > So here's a first cut at how this 2.6 -stable release process is going
> > to work that Chris and I have come up with. Does anyone have any
> > problems/issues/questions with this?
>
> One rule that I thought would make sense, but that I don't see listed
> is:
>
> - must fix a regression

That, and a zillion other specific wordings that people suggested fall
under the:
or some "oh, that's not good" issue
rule.

I didn't feel like being all lawyer-like and explicitly spelling out all
of the different kinds of bugs that we would be accepting patches for :)

So yes, I don't have a problem with patches to fix regressions.

thanks,

greg k-h

2005-03-10 17:36:03

by Lee Revell

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> That, and a zillion other specific wordings that people suggested fall
> under the:
> or some "oh, that's not good" issue
> rule.

So just to be 100% clear, no sound with 2.6.N where the sound worked
with 2.6.N-1 absolutely does qualify. Right?

Lee

2005-03-10 17:56:13

by Linus Torvalds

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.



On Thu, 10 Mar 2005, Lee Revell wrote:
>
> So just to be 100% clear, no sound with 2.6.N where the sound worked
> with 2.6.N-1 absolutely does qualify. Right?

If you can send in a patch that fixes it in an obvious way and in less
than 100 lines of context diff, hell yes.

Remember: all the other constraints still hold. Don't fall into the trap
of believing that "if it fixes a regression, it's for -stable". It needs
to be _obvious_, and it needs to be small enough that bugs are unlikely.

And that "small enough" is really important. Bugs do happen. Even in
"obvious" patches. The whole _point_ of -stable is to try to make them
less likely, and the strict constraints are very much a part of that.

Linus

2005-03-10 17:50:54

by Greg KH

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Thu, Mar 10, 2005 at 12:27:23PM -0500, Lee Revell wrote:
> On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > That, and a zillion other specific wordings that people suggested fall
> > under the:
> > or some "oh, that's not good" issue
> > rule.
>
> So just to be 100% clear, no sound with 2.6.N where the sound worked
> with 2.6.N-1 absolutely does qualify. Right?

Hm, do you think that is a "good" thing to have happen?...

greg k-h

2005-03-10 17:56:08

by Chris Wright

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

* Lee Revell ([email protected]) wrote:
> On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > That, and a zillion other specific wordings that people suggested fall
> > under the:
> > or some "oh, that's not good" issue
> > rule.
>
> So just to be 100% clear, no sound with 2.6.N where the sound worked
> with 2.6.N-1 absolutely does qualify. Right?

Depends, is listening to music while you work critical...? j/k ;-)
Yeah, that's a driver regression...used to work, now it's broken.
If fix is back out all changes, that's not so nice, if it's a
'one-liner' then definitely. Have a concrete example and patch?

thanks
-chris
--
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net

2005-03-10 18:32:18

by Lee Revell

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Thu, 2005-03-10 at 09:31 -0800, Greg KH wrote:
> On Thu, Mar 10, 2005 at 12:27:23PM -0500, Lee Revell wrote:
> > On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > > That, and a zillion other specific wordings that people suggested fall
> > > under the:
> > > or some "oh, that's not good" issue
> > > rule.
> >
> > So just to be 100% clear, no sound with 2.6.N where the sound worked
> > with 2.6.N-1 absolutely does qualify. Right?
>
> Hm, do you think that is a "good" thing to have happen?...

OK, so it sounds like scheduling latency regressions also qualify. This
could make a system that worked on 2.6.N-1 unusable on 2.6.N, and the
fixes here (usually restoring a lockbreak) are almost always small and
obvious. And users do report this, usualy in the form of "JACK was
usable under foo kernel but I get xruns with the same config under bar
kernel".

Lee

2005-03-10 18:15:44

by Lee Revell

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Thu, 2005-03-10 at 09:43 -0800, Chris Wright wrote:
> * Lee Revell ([email protected]) wrote:
> > On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > > That, and a zillion other specific wordings that people suggested fall
> > > under the:
> > > or some "oh, that's not good" issue
> > > rule.
> >
> > So just to be 100% clear, no sound with 2.6.N where the sound worked
> > with 2.6.N-1 absolutely does qualify. Right?
>
> Depends, is listening to music while you work critical...? j/k ;-)
> Yeah, that's a driver regression...used to work, now it's broken.
> If fix is back out all changes, that's not so nice, if it's a
> 'one-liner' then definitely. Have a concrete example and patch?

Not yet. We are still trying to figure out whether 2.6.11 introduced an
ALSA regression or not. See the "intel 8x0 went silent" thread.

Lee

2005-03-11 00:32:57

by NeilBrown

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Thursday March 10, [email protected] wrote:
> On Thu, Mar 10, 2005 at 09:00:51PM +1100, Neil Brown wrote:
> > On Tuesday March 8, [email protected] wrote:
> > > So here's a first cut at how this 2.6 -stable release process is going
> > > to work that Chris and I have come up with. Does anyone have any
> > > problems/issues/questions with this?
> >
> > One rule that I thought would make sense, but that I don't see listed
> > is:
> >
> > - must fix a regression
>
> That, and a zillion other specific wordings that people suggested fall
> under the:
> or some "oh, that's not good" issue
> rule.
>
> I didn't feel like being all lawyer-like and explicitly spelling out all
> of the different kinds of bugs that we would be accepting patches for :)
>
> So yes, I don't have a problem with patches to fix regressions.

I think you did not get the meaning that I was trying to convey...

I didn't mean "If it fixes a regression, it should be accepted."
I meant "If it does not fix a regression, it should not be accepted."

I have the impression that the 2.6.x.y series were suggested because
of regressions in 2.6.11 over 2.6.10 and we needed a way to respond to
that. I think it should purely be a response to that.


The issue that made me think through this is the locking bug in nfs
(there is a significant delay in getting a contended lock after the
holder drops it). It has been suggested at least twice for 2.6.11.y.
While it would be nice to have it fixed, I really don't think it
should be a candidate for 2.6.11.y. It should go into 2.6.12
(which it will) and that should be released 6-10 weeks post 2.6.11
which, given the bug as been around for a lot longer than that without
being widely noticed, should be soon enough.

I fear that if too many bug fixes go into 2.6.11.y, it might seem to
take the pressure of 2.6.12 coming out in a timely fashion, and I
wouldn't like to see that.

NeilBrown

2005-03-11 01:50:21

by NeilBrown

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Thursday March 10, [email protected] wrote:
> On Thu, 2005-03-10 at 21:00 +1100, Neil Brown wrote:
> > One rule that I thought would make sense, but that I don't see listed
> > is:
> >
> > - must fix a regression
> >
> > If some problem was in 2.6.X and is still there in 2.6.X+1, then there
> > is no great rush to fix it for 2.6.(X+1).1, it can wait for 2.6.(X+2).
>
> this is tricky. I mean, what if it's a datacorruption thing? Sure older
> kernels may have it too... yet at the same time it'd be nice to get it
> fixed. And once you go down this slipperly slope you end up with the
> original ruleset again, eg it must be "somehow relatively important".
>

My position is that
- there are lots of things that "it'd be nice to get it fixed", and
that is what 2.6.X is for
- It is a very slippery slope
- No one expect Linux to be perfect. But people do expect it to get
better without getting worse. It is precisely to fix this
expectation that (I understand) 2.6.x.y was created.

If a data corruption bug has been there for 10 weeks without being
noticed, then the real risk is not that great. We are calling it
"-release", not "-hardened".

Just my shilling-and-six worth.

NeilBrown

2005-03-11 02:47:02

by J. Bruce Fields

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Fri, Mar 11, 2005 at 11:10:37AM +1100, Neil Brown wrote:
> I didn't mean "If it fixes a regression, it should be accepted."
> I meant "If it does not fix a regression, it should not be accepted."

... Presumably with the obvious exception for security fixes.--b.

2005-03-11 04:59:57

by Chris Friesen

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

Neil Brown wrote:

> If a data corruption bug has been there for 10 weeks without being
> noticed, then the real risk is not that great. We are calling it
> "-release", not "-hardened".

I disagree. If there's a simple, obvious, small fix that passes all the
other criteria, it should go into -stable ASAP after passing review.
Then the -stable maintainers will push the fix to Andrew/Linux, and it
will go into the next 2.6.x.

Let's keep -stable as good as possible, while still abiding by all the
other rules.

Chris

2005-03-11 07:08:47

by Andi Kleen

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

Chris Friesen <[email protected]> writes:

> Neil Brown wrote:
>
>> If a data corruption bug has been there for 10 weeks without being
>> noticed, then the real risk is not that great. We are calling it
>> "-release", not "-hardened".
>
> I disagree. If there's a simple, obvious, small fix that passes all
> the other criteria, it should go into -stable ASAP after passing
> review. Then the -stable maintainers will push the fix to
> Andrew/Linux, and it will go into the next 2.6.x.

No way, it needs to go into mainline first and then maybe later
into stable. Doing stable first would lead to code drift because
a lot of people would only care about stable and we would be back
in the bad old days when older kernels had more fixes than newer
ones.

-Andi

2005-03-11 10:19:45

by Pavel Machek

[permalink] [raw]
Subject: Re: [RFC] -stable, how it's going to work.

On Čt 10-03-05 13:25:19, Lee Revell wrote:
> On Thu, 2005-03-10 at 09:31 -0800, Greg KH wrote:
> > On Thu, Mar 10, 2005 at 12:27:23PM -0500, Lee Revell wrote:
> > > On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > > > That, and a zillion other specific wordings that people suggested fall
> > > > under the:
> > > > or some "oh, that's not good" issue
> > > > rule.
> > >
> > > So just to be 100% clear, no sound with 2.6.N where the sound worked
> > > with 2.6.N-1 absolutely does qualify. Right?
> >
> > Hm, do you think that is a "good" thing to have happen?...
>
> OK, so it sounds like scheduling latency regressions also qualify. This
> could make a system that worked on 2.6.N-1 unusable on 2.6.N, and the
> fixes here (usually restoring a lockbreak) are almost always small and
> obvious. And users do report this, usualy in the form of "JACK was
> usable under foo kernel but I get xruns with the same config under bar
> kernel".

No, I do not think we want to extend it that far. Latency regression
is more of "oh, who cares" issue ;-).
Pavel
--
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!