FYI, today i committed a scheduler performance fix that has a number of
commit prerequisites for -stable integration. Those commits are not
marked -stable.
Previously, in similar situations, i solved it by email-forwarding the
prereq commits to [email protected]. (or by waiting for your 'it does
not apply to -stable' email and figuring out the prereqs then.)
But we can move this into the Git commit space too, and minimize the
work for the -stable team, via a new -stable tag variant. So with this
new commit i started using a new tagging scheme in the commit itself:
Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Cc: <[email protected]> # .32.x
LKML-Reference: <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
The tag sequence has the meaning of:
git cherry-pick a1f84a3
git cherry-pick 1b9508f
git cherry-pick fd21073
git cherry-pick <this commit>
and i'm wondering whether this tagging scheme is fine with your -stable
scripting, etc.
A further question is, i can see using this tagging scheme in the future
in merge commits log messages too - will your scripts notice that too?
For example if there's a few commits left in tip:*/urgent branches
(tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
v2.6.32 is released, i will then merge them into tip:sched/core,
tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
a notification area to 'activate' them for -stable backporting via a
merge commit.
This is how such merge commits would look like:
Merge branch 'core/urgent' into core/rcu
Merge reason: Pick up urgent fixes that did not make it into .32.0
Cc: <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
Signed-off-by: Ingo Molnar <[email protected]>
This is not so rare of a situation as it might seem - for the trees i
maintain it happens in almost every release cycle - i typically skip
urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
fixes - but they'd still be eligible for -stable.
I've attached the full commit below. The prereq commits are not uptream
yet, and they dont carry a -stable backporting tag as the -stable
relevance was not anticipated at that point yet. They will all be
upstream in the next merge window when Linus merges the relevant tree -
and then all these tags become visible to the -stable team's scripts.
What do you think about this new -stable tagging variant? To me it looks
quite intuitive, less error-prone and it is more informative as well.
Furthermore, it gives us some freedom to mark commits as backport
candidates later on. I kept them oneliners for the purpose of making
them all self-sufficient tags.
( Sidenote: i wouldnt go as far as to generate null Git commits to mark
backports after the fact - this scheme is for a series of commits that
get 'completed' - there's usually a final followup commit that can
embedd this information. )
Ingo
---------------------------->
>From eae0c9dfb534cb3449888b9601228efa6480fdb5 Mon Sep 17 00:00:00 2001
From: Mike Galbraith <[email protected]>
Date: Tue, 10 Nov 2009 03:50:02 +0100
Subject: [PATCH] sched: Fix and clean up rate-limit newidle code
Commit 1b9508f, "Rate-limit newidle" has been confirmed to fix
the netperf UDP loopback regression reported by Alex Shi.
This is a cleanup and a fix:
- moved to a more out of the way spot
- fix to ensure that balancing doesn't try to balance
runqueues which haven't gone online yet, which can
mess up CPU enumeration during boot.
Reported-by: Alex Shi <[email protected]>
Reported-by: Zhang, Yanmin <[email protected]>
Signed-off-by: Mike Galbraith <[email protected]>
Acked-by: Peter Zijlstra <[email protected]>
Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Cc: <[email protected]> # .32.x
LKML-Reference: <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
---
kernel/sched.c | 28 +++++++++++++++-------------
1 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/kernel/sched.c b/kernel/sched.c
index 23e3535..ad37776 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -2354,17 +2354,6 @@ static int try_to_wake_up(struct task_struct *p, unsigned int state,
if (rq != orig_rq)
update_rq_clock(rq);
- if (rq->idle_stamp) {
- u64 delta = rq->clock - rq->idle_stamp;
- u64 max = 2*sysctl_sched_migration_cost;
-
- if (delta > max)
- rq->avg_idle = max;
- else
- update_avg(&rq->avg_idle, delta);
- rq->idle_stamp = 0;
- }
-
WARN_ON(p->state != TASK_WAKING);
cpu = task_cpu(p);
@@ -2421,6 +2410,17 @@ out_running:
#ifdef CONFIG_SMP
if (p->sched_class->task_wake_up)
p->sched_class->task_wake_up(rq, p);
+
+ if (unlikely(rq->idle_stamp)) {
+ u64 delta = rq->clock - rq->idle_stamp;
+ u64 max = 2*sysctl_sched_migration_cost;
+
+ if (delta > max)
+ rq->avg_idle = max;
+ else
+ update_avg(&rq->avg_idle, delta);
+ rq->idle_stamp = 0;
+ }
#endif
out:
task_rq_unlock(rq, &flags);
@@ -4098,7 +4098,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,
unsigned long flags;
struct cpumask *cpus = __get_cpu_var(load_balance_tmpmask);
- cpumask_setall(cpus);
+ cpumask_copy(cpus, cpu_online_mask);
/*
* When power savings policy is enabled for the parent domain, idle
@@ -4261,7 +4261,7 @@ load_balance_newidle(int this_cpu, struct rq *this_rq, struct sched_domain *sd)
int all_pinned = 0;
struct cpumask *cpus = __get_cpu_var(load_balance_tmpmask);
- cpumask_setall(cpus);
+ cpumask_copy(cpus, cpu_online_mask);
/*
* When power savings policy is enabled for the parent domain, idle
@@ -9522,6 +9522,8 @@ void __init sched_init(void)
rq->cpu = i;
rq->online = 0;
rq->migration_thread = NULL;
+ rq->idle_stamp = 0;
+ rq->avg_idle = 2*sysctl_sched_migration_cost;
INIT_LIST_HEAD(&rq->migration_queue);
rq_attach_root(rq, &def_root_domain);
#endif
On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
> FYI, today i committed a scheduler performance fix that has a number of
> commit prerequisites for -stable integration. Those commits are not
> marked -stable.
>
> Previously, in similar situations, i solved it by email-forwarding the
> prereq commits to [email protected]. (or by waiting for your 'it does
> not apply to -stable' email and figuring out the prereqs then.)
>
> But we can move this into the Git commit space too, and minimize the
> work for the -stable team, via a new -stable tag variant. So with this
> new commit i started using a new tagging scheme in the commit itself:
>
> Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
> Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
> Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
> Cc: <[email protected]> # .32.x
> LKML-Reference: <[email protected]>
> Signed-off-by: Ingo Molnar <[email protected]>
>
> The tag sequence has the meaning of:
>
> git cherry-pick a1f84a3
> git cherry-pick 1b9508f
> git cherry-pick fd21073
> git cherry-pick <this commit>
>
> and i'm wondering whether this tagging scheme is fine with your -stable
> scripting, etc.
It would work just fine.
I only rely on one main script right now, one that runs from James's
directory on kernel.org that picks out the "Cc: <[email protected]>"
lines and forwards the full commit message to [email protected].
Then I have a simple script that I just pass the git commit id and
formats it properly for inclusion on the quilt tree for the stable
queue. If you list the other git commit ids that are needed as a
prerequesite as you did above, that's trivial to also pick out.
So I think this is good for me and my workflow.
> A further question is, i can see using this tagging scheme in the future
> in merge commits log messages too - will your scripts notice that too?
Hm, I don't think we look at merges as there's nothing there to actually
commit.
> For example if there's a few commits left in tip:*/urgent branches
> (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> v2.6.32 is released, i will then merge them into tip:sched/core,
> tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> a notification area to 'activate' them for -stable backporting via a
> merge commit.
>
> This is how such merge commits would look like:
>
> Merge branch 'core/urgent' into core/rcu
>
> Merge reason: Pick up urgent fixes that did not make it into .32.0
>
> Cc: <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> Signed-off-by: Ingo Molnar <[email protected]>
>
> This is not so rare of a situation as it might seem - for the trees i
> maintain it happens in almost every release cycle - i typically skip
> urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> fixes - but they'd still be eligible for -stable.
Ok, that would be good and fine with me.
James, would your script pick this up, or does it need to also pay
attention to merge commits?
> I've attached the full commit below. The prereq commits are not uptream
> yet, and they dont carry a -stable backporting tag as the -stable
> relevance was not anticipated at that point yet. They will all be
> upstream in the next merge window when Linus merges the relevant tree -
> and then all these tags become visible to the -stable team's scripts.
>
> What do you think about this new -stable tagging variant? To me it looks
> quite intuitive, less error-prone and it is more informative as well.
> Furthermore, it gives us some freedom to mark commits as backport
> candidates later on. I kept them oneliners for the purpose of making
> them all self-sufficient tags.
I agree.
> ( Sidenote: i wouldnt go as far as to generate null Git commits to mark
> backports after the fact - this scheme is for a series of commits that
> get 'completed' - there's usually a final followup commit that can
> embedd this information. )
That's fine, don't worry about this.
thanks,
greg k-h
* Greg KH <[email protected]> wrote:
> > I've attached the full commit below. The prereq commits are not
> > uptream yet, and they dont carry a -stable backporting tag as the
> > -stable relevance was not anticipated at that point yet. They will
> > all be upstream in the next merge window when Linus merges the
> > relevant tree - and then all these tags become visible to the
> > -stable team's scripts.
> >
> > What do you think about this new -stable tagging variant? To me it
> > looks quite intuitive, less error-prone and it is more informative
> > as well. Furthermore, it gives us some freedom to mark commits as
> > backport candidates later on. I kept them oneliners for the purpose
> > of making them all self-sufficient tags.
>
> I agree.
Ok - thanks for the confirmation - i've pushed out the first such
commit. (Let me know if there's any problem with it down the line - it
will be a few weeks, in the next merge window, until it truly
'activates' for -stable.)
Ingo
On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:
> On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
> > FYI, today i committed a scheduler performance fix that has a number of
> > commit prerequisites for -stable integration. Those commits are not
> > marked -stable.
> >
> > Previously, in similar situations, i solved it by email-forwarding the
> > prereq commits to [email protected]. (or by waiting for your 'it does
> > not apply to -stable' email and figuring out the prereqs then.)
> >
> > But we can move this into the Git commit space too, and minimize the
> > work for the -stable team, via a new -stable tag variant. So with this
> > new commit i started using a new tagging scheme in the commit itself:
> >
> > Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
> > Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
> > Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
> > Cc: <[email protected]> # .32.x
> > LKML-Reference: <[email protected]>
> > Signed-off-by: Ingo Molnar <[email protected]>
> >
> > The tag sequence has the meaning of:
> >
> > git cherry-pick a1f84a3
> > git cherry-pick 1b9508f
> > git cherry-pick fd21073
> > git cherry-pick <this commit>
> >
> > and i'm wondering whether this tagging scheme is fine with your -stable
> > scripting, etc.
>
> It would work just fine.
>
> I only rely on one main script right now, one that runs from James's
> directory on kernel.org that picks out the "Cc: <[email protected]>"
> lines and forwards the full commit message to [email protected].
>
> Then I have a simple script that I just pass the git commit id and
> formats it properly for inclusion on the quilt tree for the stable
> queue. If you list the other git commit ids that are needed as a
> prerequesite as you did above, that's trivial to also pick out.
>
> So I think this is good for me and my workflow.
So I take this to mean that I don't alter my script and you pick out the
precursors with yours ...
> > A further question is, i can see using this tagging scheme in the future
> > in merge commits log messages too - will your scripts notice that too?
>
> Hm, I don't think we look at merges as there's nothing there to actually
> commit.
>
> > For example if there's a few commits left in tip:*/urgent branches
> > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > v2.6.32 is released, i will then merge them into tip:sched/core,
> > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > a notification area to 'activate' them for -stable backporting via a
> > merge commit.
> >
> > This is how such merge commits would look like:
> >
> > Merge branch 'core/urgent' into core/rcu
> >
> > Merge reason: Pick up urgent fixes that did not make it into .32.0
> >
> > Cc: <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> > Signed-off-by: Ingo Molnar <[email protected]>
> >
> > This is not so rare of a situation as it might seem - for the trees i
> > maintain it happens in almost every release cycle - i typically skip
> > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > fixes - but they'd still be eligible for -stable.
>
> Ok, that would be good and fine with me.
>
> James, would your script pick this up, or does it need to also pay
> attention to merge commits?
No ... because merge commits should effectively be empty (and when
they're not you can't generate an applyable diff). If I understand the
workflow, the desire is to have the whole branch sent to stable by
tagging the merge commit. That's possible ... it's exactly the same
logic I use in the commit scripts for the SCSI tree, so it should be
possible to extract the logic.
By the looks of the above it's only a few commits, or is it the entire
branch?
> > I've attached the full commit below. The prereq commits are not uptream
> > yet, and they dont carry a -stable backporting tag as the -stable
> > relevance was not anticipated at that point yet. They will all be
> > upstream in the next merge window when Linus merges the relevant tree -
> > and then all these tags become visible to the -stable team's scripts.
> >
> > What do you think about this new -stable tagging variant? To me it looks
> > quite intuitive, less error-prone and it is more informative as well.
> > Furthermore, it gives us some freedom to mark commits as backport
> > candidates later on. I kept them oneliners for the purpose of making
> > them all self-sufficient tags.
>
> I agree.
>
> > ( Sidenote: i wouldnt go as far as to generate null Git commits to mark
> > backports after the fact - this scheme is for a series of commits that
> > get 'completed' - there's usually a final followup commit that can
> > embedd this information. )
>
> That's fine, don't worry about this.
The question is, how important is this?
One of the assumptions behind the current setup is that I assume
backports are independent (so the order of transmission doesn't matter
that much). This isn't always true, but the exceptions tend to get
handled manually. Part of what the above is requesting is an
implementation that starts to care about ordering.
James
On Tue, 2009-11-10 at 05:20 +0100, Ingo Molnar wrote:
> * Greg KH <[email protected]> wrote:
>
> > > I've attached the full commit below. The prereq commits are not
> > > uptream yet, and they dont carry a -stable backporting tag as the
> > > -stable relevance was not anticipated at that point yet. They will
> > > all be upstream in the next merge window when Linus merges the
> > > relevant tree - and then all these tags become visible to the
> > > -stable team's scripts.
> > >
> > > What do you think about this new -stable tagging variant? To me it
> > > looks quite intuitive, less error-prone and it is more informative
> > > as well. Furthermore, it gives us some freedom to mark commits as
> > > backport candidates later on. I kept them oneliners for the purpose
> > > of making them all self-sufficient tags.
> >
> > I agree.
>
> Ok - thanks for the confirmation - i've pushed out the first such
> commit. (Let me know if there's any problem with it down the line - it
> will be a few weeks, in the next merge window, until it truly
> 'activates' for -stable.)
Can you give me the commit id in your tree so I can run a few tests?
Thanks,
James
James Bottomley wrote:
[...]
>> On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
>> > FYI, today i committed a scheduler performance fix that has a number of
>> > commit prerequisites for -stable integration. Those commits are not
>> > marked -stable.
[...]
>> > we can move this into the Git commit space too, and minimize the
>> > work for the -stable team, via a new -stable tag variant.
[...]
> The question is, how important is this?
>
> One of the assumptions behind the current setup is that I assume
> backports are independent (so the order of transmission doesn't matter
> that much). This isn't always true, but the exceptions tend to get
> handled manually. Part of what the above is requesting is an
> implementation that starts to care about ordering.
More importantly, isn't this against the character of the -stable kernel
branches as _safe and simple_ hotfix branches?
If a fix has a number of prerequisites which ar not -stable fixes
themselves, then it is more than a hint that this fix is not really well
suited for -stable.
--
Stefan Richter
-=====-==--= =-== -=-=-
http://arcgraph.de/sr/
* James Bottomley <[email protected]> wrote:
> On Tue, 2009-11-10 at 05:20 +0100, Ingo Molnar wrote:
> > * Greg KH <[email protected]> wrote:
> >
> > > > I've attached the full commit below. The prereq commits are not
> > > > uptream yet, and they dont carry a -stable backporting tag as the
> > > > -stable relevance was not anticipated at that point yet. They will
> > > > all be upstream in the next merge window when Linus merges the
> > > > relevant tree - and then all these tags become visible to the
> > > > -stable team's scripts.
> > > >
> > > > What do you think about this new -stable tagging variant? To me it
> > > > looks quite intuitive, less error-prone and it is more informative
> > > > as well. Furthermore, it gives us some freedom to mark commits as
> > > > backport candidates later on. I kept them oneliners for the purpose
> > > > of making them all self-sufficient tags.
> > >
> > > I agree.
> >
> > Ok - thanks for the confirmation - i've pushed out the first such
> > commit. (Let me know if there's any problem with it down the line - it
> > will be a few weeks, in the next merge window, until it truly
> > 'activates' for -stable.)
>
> Can you give me the commit id in your tree so I can run a few tests?
Let me re-quote what you quoted above:
> > > > I've attached the full commit below. [...]
Ingo
On Tue, 10 Nov 2009, Ingo Molnar wrote:
>
> * James Bottomley <[email protected]> wrote:
> >
> > Can you give me the commit id in your tree so I can run a few tests?
>
> Let me re-quote what you quoted above:
>
> > > > > I've attached the full commit below. [...]
Let me re-quote James:
"Can you give me the commit id in your tree so I can run a few tests?"
He wanted the tree. So that he can run tests on his machinery. His
machinery that does _not_ take some emailed commit description, but works
on git trees.
So Ingo, if you want to be snarky about quoting people, at least be snarky
with a reason.
Linus
* Linus Torvalds <[email protected]> wrote:
>
>
> On Tue, 10 Nov 2009, Ingo Molnar wrote:
> >
> > * James Bottomley <[email protected]> wrote:
> > >
> > > Can you give me the commit id in your tree so I can run a few tests?
> >
> > Let me re-quote what you quoted above:
> >
> > > > > > I've attached the full commit below. [...]
>
> Let me re-quote James:
>
> "Can you give me the commit id in your tree so I can run a few tests?"
>
> He wanted the tree. So that he can run tests on his machinery. His
> machinery that does _not_ take some emailed commit description, but
> works on git trees.
>
> So Ingo, if you want to be snarky about quoting people, at least be
> snarky with a reason.
The commit i quoted had the commit ID on its first line (i did that so
because i anticipated someone wanting to check my tree), but here it is
again plus the (well-known to James) URI of the -tip tree:
> From eae0c9dfb534cb3449888b9601228efa6480fdb5 Mon Sep 17 00:00:00 2001
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git
Thanks,
Ingo
On 11/10/2009 09:52 AM, Stefan Richter wrote:
> More importantly, isn't this against the character of the -stable kernel
> branches as _safe and simple_ hotfix branches?
>
> If a fix has a number of prerequisites which ar not -stable fixes
> themselves, then it is more than a hint that this fix is not really well
> suited for -stable.
Alternately, it's conceivable that the prerequisites were not
in-and-of-themselves candidates for -stable (maybe they didn't do
anything by themselves) but when combined with the final commit the
overall change is suitable for inclusion in -stable.
Chris
On Tue, Nov 10, 2009 at 09:29:48AM -0500, James Bottomley wrote:
> On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:
> > On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
> > > FYI, today i committed a scheduler performance fix that has a number of
> > > commit prerequisites for -stable integration. Those commits are not
> > > marked -stable.
> > >
> > > Previously, in similar situations, i solved it by email-forwarding the
> > > prereq commits to [email protected]. (or by waiting for your 'it does
> > > not apply to -stable' email and figuring out the prereqs then.)
> > >
> > > But we can move this into the Git commit space too, and minimize the
> > > work for the -stable team, via a new -stable tag variant. So with this
> > > new commit i started using a new tagging scheme in the commit itself:
> > >
> > > Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
> > > Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
> > > Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
> > > Cc: <[email protected]> # .32.x
> > > LKML-Reference: <[email protected]>
> > > Signed-off-by: Ingo Molnar <[email protected]>
> > >
> > > The tag sequence has the meaning of:
> > >
> > > git cherry-pick a1f84a3
> > > git cherry-pick 1b9508f
> > > git cherry-pick fd21073
> > > git cherry-pick <this commit>
> > >
> > > and i'm wondering whether this tagging scheme is fine with your -stable
> > > scripting, etc.
> >
> > It would work just fine.
> >
> > I only rely on one main script right now, one that runs from James's
> > directory on kernel.org that picks out the "Cc: <[email protected]>"
> > lines and forwards the full commit message to [email protected].
> >
> > Then I have a simple script that I just pass the git commit id and
> > formats it properly for inclusion on the quilt tree for the stable
> > queue. If you list the other git commit ids that are needed as a
> > prerequesite as you did above, that's trivial to also pick out.
> >
> > So I think this is good for me and my workflow.
>
> So I take this to mean that I don't alter my script and you pick out the
> precursors with yours ...
Exactly, it's easier for me that way, as I know the dependancy of what
needs to go before what. And it's just so trivial to feed a git commit
id to my script :)
> > > A further question is, i can see using this tagging scheme in the future
> > > in merge commits log messages too - will your scripts notice that too?
> >
> > Hm, I don't think we look at merges as there's nothing there to actually
> > commit.
> >
> > > For example if there's a few commits left in tip:*/urgent branches
> > > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > > v2.6.32 is released, i will then merge them into tip:sched/core,
> > > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > > a notification area to 'activate' them for -stable backporting via a
> > > merge commit.
> > >
> > > This is how such merge commits would look like:
> > >
> > > Merge branch 'core/urgent' into core/rcu
> > >
> > > Merge reason: Pick up urgent fixes that did not make it into .32.0
> > >
> > > Cc: <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> > > Signed-off-by: Ingo Molnar <[email protected]>
> > >
> > > This is not so rare of a situation as it might seem - for the trees i
> > > maintain it happens in almost every release cycle - i typically skip
> > > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > > fixes - but they'd still be eligible for -stable.
> >
> > Ok, that would be good and fine with me.
> >
> > James, would your script pick this up, or does it need to also pay
> > attention to merge commits?
>
> No ... because merge commits should effectively be empty (and when
> they're not you can't generate an applyable diff). If I understand the
> workflow, the desire is to have the whole branch sent to stable by
> tagging the merge commit. That's possible ... it's exactly the same
> logic I use in the commit scripts for the SCSI tree, so it should be
> possible to extract the logic.
>
> By the looks of the above it's only a few commits, or is it the entire
> branch?
I'm thinking the commit would be the merge, right Ingo? So it would
just be a single commit that has the marker in it.
> > > I've attached the full commit below. The prereq commits are not uptream
> > > yet, and they dont carry a -stable backporting tag as the -stable
> > > relevance was not anticipated at that point yet. They will all be
> > > upstream in the next merge window when Linus merges the relevant tree -
> > > and then all these tags become visible to the -stable team's scripts.
> > >
> > > What do you think about this new -stable tagging variant? To me it looks
> > > quite intuitive, less error-prone and it is more informative as well.
> > > Furthermore, it gives us some freedom to mark commits as backport
> > > candidates later on. I kept them oneliners for the purpose of making
> > > them all self-sufficient tags.
> >
> > I agree.
> >
> > > ( Sidenote: i wouldnt go as far as to generate null Git commits to mark
> > > backports after the fact - this scheme is for a series of commits that
> > > get 'completed' - there's usually a final followup commit that can
> > > embedd this information. )
> >
> > That's fine, don't worry about this.
>
> The question is, how important is this?
>
> One of the assumptions behind the current setup is that I assume
> backports are independent (so the order of transmission doesn't matter
> that much). This isn't always true, but the exceptions tend to get
> handled manually. Part of what the above is requesting is an
> implementation that starts to care about ordering.
No, I'll take care of the ordering myself. Heck, I already have to do
that today with our current setup as we have dependant staging patches
right now. I just want to make sure the merge commits will get picked
up and sent to me so I can then pick the correct git commit ids out of
them.
thanks,
greg k-h
On Tue, Nov 10, 2009 at 04:52:22PM +0100, Stefan Richter wrote:
> James Bottomley wrote:
> [...]
> >> On Tue, Nov 10, 2009 at 04:48:31AM +0100, Ingo Molnar wrote:
> >> > FYI, today i committed a scheduler performance fix that has a number of
> >> > commit prerequisites for -stable integration. Those commits are not
> >> > marked -stable.
> [...]
> >> > we can move this into the Git commit space too, and minimize the
> >> > work for the -stable team, via a new -stable tag variant.
> [...]
> > The question is, how important is this?
> >
> > One of the assumptions behind the current setup is that I assume
> > backports are independent (so the order of transmission doesn't matter
> > that much). This isn't always true, but the exceptions tend to get
> > handled manually. Part of what the above is requesting is an
> > implementation that starts to care about ordering.
>
> More importantly, isn't this against the character of the -stable kernel
> branches as _safe and simple_ hotfix branches?
>
> If a fix has a number of prerequisites which ar not -stable fixes
> themselves, then it is more than a hint that this fix is not really well
> suited for -stable.
Not true, we have been doing this kind of thing for quite some time now.
Sometimes it's just a simple "this patch cleans up the code, and the
second one fixes it in an obvious manner" type thing. It is easier for
me and everyone else for us to apply 2 commits to the -stable tree,
instead of rewriting the second patch that actually does the fix and
hope I got it all correct in doing so.
It's also easier to review stuff, which is the most important thing.
thanks,
greg k-h
* Greg KH <[email protected]> wrote:
> > By the looks of the above it's only a few commits, or is it the
> > entire branch?
>
> I'm thinking the commit would be the merge, right Ingo? So it would
> just be a single commit that has the marker in it.
Correct.
This is really a special case, a small variation of the commit eae0c9d
-stable tagging scheme i outlined in the first mail.
When i merge */urgent branches into the for-linus branch in the merge
window, i cannot change the commits anymore (it would amount to a
rebase), but i have the opportunity to modify the merge commit message
itself. (which is typically a regular merge commit and does not carry
any -stable actionable change itself.)
I already annotate merge commits today - for example:
| commit 43315956509ca6913764861ac7dec128b91eb1ec
| Merge: 9bf4e7f 6beba7a
| Author: Ingo Molnar <[email protected]>
| Date: Fri Oct 23 08:23:20 2009 +0200
|
| Merge branch 'perf/core' into perf/probes
|
| Conflicts:
| tools/perf/Makefile
|
| Merge reason:
|
| - fix the conflict
| - pick up the pr_*() infrastructure to queue up dependent patch
|
| Signed-off-by: Ingo Molnar <[email protected]>
Note how i already put a SOB line into the merge commit - i treat every
merge as something that 'had to be done' so they are never arbitrary and
always carry real information.
So my idea was to potentially use the extended -stable notification
scheme in certain merge commits too. Here's a mockup merge commit log:
Merge branch 'sched/urgent' into sched/core
Conflicts:
tools/perf/Makefile
Merge reason:
- resolve the conflict
- queue up urgent fixes for the next merge window
Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Signed-off-by: Ingo Molnar <[email protected]>
Note that the merge commit itself carries no action for -stable: there's
no "Cc: <[email protected]>" line - only 'pointer' lines in the form
of:
Cc: <[email protected]> # .32.x: sha1: title
But ... if you or Linus dislikes this direction of tagging for some
reason i can still do the manual approach as well. It seemed useful to
me though and it would be a natural portion of my workflow.
Ingo
* Greg KH <[email protected]> wrote:
> > More importantly, isn't this against the character of the -stable
> > kernel branches as _safe and simple_ hotfix branches?
> >
> > If a fix has a number of prerequisites which ar not -stable fixes
> > themselves, then it is more than a hint that this fix is not really
> > well suited for -stable.
>
> Not true, we have been doing this kind of thing for quite some time
> now. Sometimes it's just a simple "this patch cleans up the code, and
> the second one fixes it in an obvious manner" type thing. It is
> easier for me and everyone else for us to apply 2 commits to the
> -stable tree, instead of rewriting the second patch that actually does
> the fix and hope I got it all correct in doing so.
>
> It's also easier to review stuff, which is the most important thing.
Yeah. This new tagging scheme doesnt really allow anything 'new' per se
- it just helps the existing practice some more. All these commits were
-stable candidates anyway, in exactly the same order - the only
difference the new tagging scheme adds here is a more organized,
in-upsream-Git way of communicating it to you.
This is also easier and less error prone for me than using email: i can
do all the -stable tagging when i create a commit - or if i see that a
commit has prereqs and those should be in -stable too. In those
situations i check out the last stable kernel version, and cherry-pick
the prereqs and the target commit, to see that it cherry-picks without
conflicts.
But i cannot send you an email to [email protected] just yet: as i
havent fully tested the last commit yet, and have not pushed it out yet.
The commit ID is not stable yet.
So without the in-commit tagging, i'd have to remember to send you an
email in an hour (or in a day - whenever testing is done) - and that is
error prone and easy to forget. The prereqs might be lost, etc. It's
better to do this all in one well-focused moment of time, gather the
information and mention it in the changelog.
Ingo
Ingo Molnar <[email protected]> writes:
> Yeah. This new tagging scheme doesnt really allow anything 'new' per se
> - it just helps the existing practice some more. All these commits were
> -stable candidates anyway, in exactly the same order - the only
> difference the new tagging scheme adds here is a more organized,
> in-upsream-Git way of communicating it to you.
I am just a bystander, but if it were truly in-upstream-git way, wouldn't
you be forking a branch from the tagged target release (the latest of
2.6.32.X), and queuing only the changes meant for -stable to it, and
giving the name of the branch to git people and sending out patches from
that branch for e-mailed review and application?
There won't be any special tagging required, only a dedicated branch.
Or am I missing something?
* Chris Friesen <[email protected]> wrote:
> On 11/10/2009 09:52 AM, Stefan Richter wrote:
>
> > More importantly, isn't this against the character of the -stable kernel
> > branches as _safe and simple_ hotfix branches?
> >
> > If a fix has a number of prerequisites which ar not -stable fixes
> > themselves, then it is more than a hint that this fix is not really well
> > suited for -stable.
>
> Alternately, it's conceivable that the prerequisites were not
> in-and-of-themselves candidates for -stable (maybe they didn't do
> anything by themselves) but when combined with the final commit the
> overall change is suitable for inclusion in -stable.
Yeah.
The way i do it as a maintainer is that when i add a new commit that i
realize as a -stable candidate, and i know that it has no semantic
prereqs (such as a new API, etc.), i git-cherry-pick it into stable in a
test branch. If that works fine i mark it -stable straight away.
If it conflicts, i figure out the prereqs, and look at those. If they
are too big, or too risky, i often decide not to mark a patch for
-stable backporting. (If it's not obvious to be in -stable then it
should not be in -stable, almost by definition - a _LOT_ of people are
using the stable kernels.)
If the prereqs look sane and are wanted for -stable, i end up with a
list of 2 or at most 3 commits that will cherry-pick cleanly. (rarely
more than that - the 4 commits here are really an exception - they are a
string of prereqs that are also fixes)
I send that list of commits to [email protected].
( Sidenote: rarely does it make sense to port a conflicting commit to
-stable. The risks of introducing some regression are just too high.
Cherry-picking of commits, (while not entirely risk-free of course),
is far more robust in practice. (conflicting backmerges do happen too
occasionally, for high-profile bug fixes that justify the cost.) )
So this 'send the list to [email protected]' step is simplified via
these tags:
Cc: <[email protected]> # .32.x: a1f84a3: sched: Check for an idle shared cache
Cc: <[email protected]> # .32.x: 1b9508f: sched: Rate-limit newidle
Cc: <[email protected]> # .32.x: fd21073: sched: Fix affinity logic
Cc: <[email protected]> # .32.x
I'd have done that via a plain email in any case - so this scheme does
not enable anything new - it just simplifies the process and makes it a
bit more robust.
Ingo
On Tue, Nov 10, 2009 at 01:11:17PM -0800, Junio C Hamano wrote:
> Ingo Molnar <[email protected]> writes:
>
> > Yeah. This new tagging scheme doesnt really allow anything 'new' per se
> > - it just helps the existing practice some more. All these commits were
> > -stable candidates anyway, in exactly the same order - the only
> > difference the new tagging scheme adds here is a more organized,
> > in-upsream-Git way of communicating it to you.
>
> I am just a bystander, but if it were truly in-upstream-git way, wouldn't
> you be forking a branch from the tagged target release (the latest of
> 2.6.32.X), and queuing only the changes meant for -stable to it, and
> giving the name of the branch to git people and sending out patches from
> that branch for e-mailed review and application?
>
> There won't be any special tagging required, only a dedicated branch.
>
> Or am I missing something?
Yes, these are patches going to Linus's tree, which would be 2.6.33 at
the time. I need to know the ids to add them back to the older .32.y
tree.
hope this helps,
greg k-h
* Junio C Hamano <[email protected]> wrote:
> Ingo Molnar <[email protected]> writes:
>
> > Yeah. This new tagging scheme doesnt really allow anything 'new' per se
> > - it just helps the existing practice some more. All these commits were
> > -stable candidates anyway, in exactly the same order - the only
> > difference the new tagging scheme adds here is a more organized,
> > in-upsream-Git way of communicating it to you.
>
> I am just a bystander, but if it were truly in-upstream-git way,
> wouldn't you be forking a branch from the tagged target release (the
> latest of 2.6.32.X), and queuing only the changes meant for -stable to
> it, and giving the name of the branch to git people and sending out
> patches from that branch for e-mailed review and application?
>
> There won't be any special tagging required, only a dedicated branch.
>
> Or am I missing something?
There's no Git flow towards -stable. It's either forwarded emails, or
tags in the upstream kernel. Also, _only_ commits that were pulled by
Linus are eligible for -stable.
So the pull requests all first go to Linus - then can any commit flow to
-stable.
But even if it was possible to send pull requests to Greg, marking
commits as -stable candidates is more natural in the commit log itself.
That informs people ('hey, that's a dangerous patch, dont mark it for
-stable!!' or 'hey, why isnt this commit tagged to stable??'), and it
also ensures it that only commits from Linus's tree flow towards
-stable.
Ingo
On Tue, 2009-11-10 at 11:37 -0800, Greg KH wrote:
> On Tue, Nov 10, 2009 at 09:29:48AM -0500, James Bottomley wrote:
> > On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:
> > > > A further question is, i can see using this tagging scheme in the future
> > > > in merge commits log messages too - will your scripts notice that too?
> > >
> > > Hm, I don't think we look at merges as there's nothing there to actually
> > > commit.
> > >
> > > > For example if there's a few commits left in tip:*/urgent branches
> > > > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > > > v2.6.32 is released, i will then merge them into tip:sched/core,
> > > > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > > > a notification area to 'activate' them for -stable backporting via a
> > > > merge commit.
> > > >
> > > > This is how such merge commits would look like:
> > > >
> > > > Merge branch 'core/urgent' into core/rcu
> > > >
> > > > Merge reason: Pick up urgent fixes that did not make it into .32.0
> > > >
> > > > Cc: <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> > > > Signed-off-by: Ingo Molnar <[email protected]>
> > > >
> > > > This is not so rare of a situation as it might seem - for the trees i
> > > > maintain it happens in almost every release cycle - i typically skip
> > > > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > > > fixes - but they'd still be eligible for -stable.
> > >
> > > Ok, that would be good and fine with me.
> > >
> > > James, would your script pick this up, or does it need to also pay
> > > attention to merge commits?
> >
> > No ... because merge commits should effectively be empty (and when
> > they're not you can't generate an applyable diff). If I understand the
> > workflow, the desire is to have the whole branch sent to stable by
> > tagging the merge commit. That's possible ... it's exactly the same
> > logic I use in the commit scripts for the SCSI tree, so it should be
> > possible to extract the logic.
> >
> > By the looks of the above it's only a few commits, or is it the entire
> > branch?
>
> I'm thinking the commit would be the merge, right Ingo? So it would
> just be a single commit that has the marker in it.
OK, so I can make it send you this just by removing the --no-merge flag
from the git rev-list the script uses to sift through what changed
(which I've already done).
The slight problem is that further down, to generate the patch the
script uses git format-patch -k --stdout commit^..commit. For a merge
commit, this will generate a patch equivalent to the entire branch that
was merged, even though the commit message will only pick out some of
these ... is this OK?
If not, I can look at using git show instead to generate the patches (it
will effectively generate null diffs for merge points with the stable
tag, which is closer to what you want).
Alternatively, if you pick up the commits from Linus' tree anyway, I
could just stop producing diffs, which will save email bandwidth and
then be automatically correct whether the commit is a merge or not.
James
On Wed, 11 Nov 2009, James Bottomley wrote:
>
> The slight problem is that further down, to generate the patch the
> script uses git format-patch -k --stdout commit^..commit. For a merge
> commit, this will generate a patch equivalent to the entire branch that
> was merged, even though the commit message will only pick out some of
> these ... is this OK?
Don't do that.
Since you want to show a single commit, not a commit range, don't use
format-patch, use something like
git show --pretty=email -M --cc --stat $commit
instead. That will not show the difference between a commit and its
first parent (which for a merge diff is more than one commit, and can be
absolutely _huge_), but will show just the named commit (which for a merge
commit will usually be an empty diff, but will show how conflicts were
resolved if they weren't just taken from one or the other side).
(Of course, maybe you'll want to change the exact flags in question. For
example, '--stat' for merge commits is often useful, but only if the merge
was done by an upper-level maintainer. Anybody who does reverse merges
will just get totally meaningless diffstats, since the diffstat is always
done against the first parent, so anybody who does back-merges will see
the stat of everthing _I_ have merged in the meantime, which is generally
not useful)
Linus
On Wed, 2009-11-11 at 12:00 -0800, Linus Torvalds wrote:
>
> On Wed, 11 Nov 2009, James Bottomley wrote:
> >
> > The slight problem is that further down, to generate the patch the
> > script uses git format-patch -k --stdout commit^..commit. For a merge
> > commit, this will generate a patch equivalent to the entire branch that
> > was merged, even though the commit message will only pick out some of
> > these ... is this OK?
>
> Don't do that.
>
> Since you want to show a single commit, not a commit range, don't use
> format-patch, use something like
>
> git show --pretty=email -M --cc --stat $commit
>
> instead. That will not show the difference between a commit and its
> first parent (which for a merge diff is more than one commit, and can be
> absolutely _huge_), but will show just the named commit (which for a merge
> commit will usually be an empty diff, but will show how conflicts were
> resolved if they weren't just taken from one or the other side).
>
> (Of course, maybe you'll want to change the exact flags in question. For
> example, '--stat' for merge commits is often useful, but only if the merge
> was done by an upper-level maintainer. Anybody who does reverse merges
> will just get totally meaningless diffstats, since the diffstat is always
> done against the first parent, so anybody who does back-merges will see
> the stat of everthing _I_ have merged in the meantime, which is generally
> not useful)
Right ... that's why I was wondering if Greg still needs the diff. If
not, I'll just send standard git log output with no diff. If he does,
I'll use something like the above.
So, now that you're showing an interest in all of this, how about I send
you the final update hook to add to your tree? It will produce less
latency than the cron job I run on kernel.org to pull your tree into one
I have here ever twenty minutes or so and run the hooks.
James
On Wed, 11 Nov 2009, James Bottomley wrote:
>
> So, now that you're showing an interest in all of this, how about I send
> you the final update hook to add to your tree? It will produce less
> latency than the cron job I run on kernel.org to pull your tree into one
> I have here ever twenty minutes or so and run the hooks.
Quite frankly, I'd much rather not run hooks on my tree directly. The less
that can possibly get screwed up with my kernel.org tree, the happier I
am. So in many ways, I really prefer the whole "done separately by others"
approach.
But hey, send the hook to me, and I'll think about it.
Linus
On 11/11/2009 12:14 PM, Linus Torvalds wrote:
>
>
> On Wed, 11 Nov 2009, James Bottomley wrote:
>>
>> So, now that you're showing an interest in all of this, how about I send
>> you the final update hook to add to your tree? It will produce less
>> latency than the cron job I run on kernel.org to pull your tree into one
>> I have here ever twenty minutes or so and run the hooks.
>
> Quite frankly, I'd much rather not run hooks on my tree directly. The less
> that can possibly get screwed up with my kernel.org tree, the happier I
> am. So in many ways, I really prefer the whole "done separately by others"
> approach.
>
> But hey, send the hook to me, and I'll think about it.
>
One thing we might be able to do is to create an "exploder hook" on your
tree - something that asynchronously broadcasts a notification that
anyone can subscribe to.
-hpa
On Wed, 11 Nov 2009, H. Peter Anvin wrote:
> >
> > Quite frankly, I'd much rather not run hooks on my tree directly. The less
> > that can possibly get screwed up with my kernel.org tree, the happier I
> > am. So in many ways, I really prefer the whole "done separately by others"
> > approach.
> >
> > But hey, send the hook to me, and I'll think about it.
>
> One thing we might be able to do is to create an "exploder hook" on your
> tree - something that asynchronously broadcasts a notification that
> anyone can subscribe to.
Yeah, some minimal hook just informing any interested party about "ok,
Linus pushed to the main tree" (and in fact, perhaps generalized so that
you can show that you are interested in some particular branch of _any_
tree that people can just add to their trees) certainly doesn't sound like
a bad idea.
That way we can get rid of "polling by cron", without having to have the
tree owners themselves care about who is interested in exactly what..
Linus
On Wed, Nov 11, 2009 at 12:50:24PM -0600, James Bottomley wrote:
> On Tue, 2009-11-10 at 11:37 -0800, Greg KH wrote:
> > On Tue, Nov 10, 2009 at 09:29:48AM -0500, James Bottomley wrote:
> > > On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:
> > > > > A further question is, i can see using this tagging scheme in the future
> > > > > in merge commits log messages too - will your scripts notice that too?
> > > >
> > > > Hm, I don't think we look at merges as there's nothing there to actually
> > > > commit.
> > > >
> > > > > For example if there's a few commits left in tip:*/urgent branches
> > > > > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > > > > v2.6.32 is released, i will then merge them into tip:sched/core,
> > > > > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > > > > a notification area to 'activate' them for -stable backporting via a
> > > > > merge commit.
> > > > >
> > > > > This is how such merge commits would look like:
> > > > >
> > > > > Merge branch 'core/urgent' into core/rcu
> > > > >
> > > > > Merge reason: Pick up urgent fixes that did not make it into .32.0
> > > > >
> > > > > Cc: <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> > > > > Signed-off-by: Ingo Molnar <[email protected]>
> > > > >
> > > > > This is not so rare of a situation as it might seem - for the trees i
> > > > > maintain it happens in almost every release cycle - i typically skip
> > > > > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > > > > fixes - but they'd still be eligible for -stable.
> > > >
> > > > Ok, that would be good and fine with me.
> > > >
> > > > James, would your script pick this up, or does it need to also pay
> > > > attention to merge commits?
> > >
> > > No ... because merge commits should effectively be empty (and when
> > > they're not you can't generate an applyable diff). If I understand the
> > > workflow, the desire is to have the whole branch sent to stable by
> > > tagging the merge commit. That's possible ... it's exactly the same
> > > logic I use in the commit scripts for the SCSI tree, so it should be
> > > possible to extract the logic.
> > >
> > > By the looks of the above it's only a few commits, or is it the entire
> > > branch?
> >
> > I'm thinking the commit would be the merge, right Ingo? So it would
> > just be a single commit that has the marker in it.
>
> OK, so I can make it send you this just by removing the --no-merge flag
> from the git rev-list the script uses to sift through what changed
> (which I've already done).
>
> The slight problem is that further down, to generate the patch the
> script uses git format-patch -k --stdout commit^..commit. For a merge
> commit, this will generate a patch equivalent to the entire branch that
> was merged, even though the commit message will only pick out some of
> these ... is this OK?
>
> If not, I can look at using git show instead to generate the patches (it
> will effectively generate null diffs for merge points with the stable
> tag, which is closer to what you want).
>
> Alternatively, if you pick up the commits from Linus' tree anyway, I
> could just stop producing diffs, which will save email bandwidth and
> then be automatically correct whether the commit is a merge or not.
No, I'd like to keep diffs, and use Linus's suggestion to get the
correct diff, I have had that same bug in some scripts I have written as
well.
thanks,
greg k-h
* Linus Torvalds <[email protected]> wrote:
>
>
> On Wed, 11 Nov 2009, H. Peter Anvin wrote:
> > >
> > > Quite frankly, I'd much rather not run hooks on my tree directly. The less
> > > that can possibly get screwed up with my kernel.org tree, the happier I
> > > am. So in many ways, I really prefer the whole "done separately by others"
> > > approach.
> > >
> > > But hey, send the hook to me, and I'll think about it.
> >
> > One thing we might be able to do is to create an "exploder hook" on your
> > tree - something that asynchronously broadcasts a notification that
> > anyone can subscribe to.
>
> Yeah, some minimal hook just informing any interested party about "ok,
> Linus pushed to the main tree" (and in fact, perhaps generalized so that
> you can show that you are interested in some particular branch of _any_
> tree that people can just add to their trees) certainly doesn't sound like
> a bad idea.
>
> That way we can get rid of "polling by cron", without having to have
> the tree owners themselves care about who is interested in exactly
> what..
That's already possible via the [email protected] email
list. Anyone can subscribe to it and can procmail process it. It is also
more secure (and less intrusive to the tree owner) than any post-push
hook script running under the pusher's UID. Polling via cron is really
not required i think.
Ingo
On Wed, 2009-11-11 at 13:52 -0800, Linus Torvalds wrote:
>
> On Wed, 11 Nov 2009, H. Peter Anvin wrote:
> > >
> > > Quite frankly, I'd much rather not run hooks on my tree directly. The less
> > > that can possibly get screwed up with my kernel.org tree, the happier I
> > > am. So in many ways, I really prefer the whole "done separately by others"
> > > approach.
> > >
> > > But hey, send the hook to me, and I'll think about it.
> >
> > One thing we might be able to do is to create an "exploder hook" on your
> > tree - something that asynchronously broadcasts a notification that
> > anyone can subscribe to.
>
> Yeah, some minimal hook just informing any interested party about "ok,
> Linus pushed to the main tree" (and in fact, perhaps generalized so that
> you can show that you are interested in some particular branch of _any_
> tree that people can just add to their trees) certainly doesn't sound like
> a bad idea.
>
> That way we can get rid of "polling by cron", without having to have the
> tree owners themselves care about who is interested in exactly what..
Sure, I'd like to hook into this rather than use a polling cron job.
James
On 11/11/2009 11:45 PM, Ingo Molnar wrote:
>
> That's already possible via the [email protected] email
> list. Anyone can subscribe to it and can procmail process it. It is also
> more secure (and less intrusive to the tree owner) than any post-push
> hook script running under the pusher's UID. Polling via cron is really
> not required i think.
>
There are a couple of disadvantages with this -- a hera -> vger -> hera
bounce, in particular (those two machines are nowhere close to each
other, and there can be huge latecy.) For real-time robots running on
hera, a lower-latency mechanism would be useful.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.
On Wed, 2009-11-11 at 20:41 -0800, Greg KH wrote:
> On Wed, Nov 11, 2009 at 12:50:24PM -0600, James Bottomley wrote:
> > On Tue, 2009-11-10 at 11:37 -0800, Greg KH wrote:
> > > On Tue, Nov 10, 2009 at 09:29:48AM -0500, James Bottomley wrote:
> > > > On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:
> > > > > > A further question is, i can see using this tagging scheme in the future
> > > > > > in merge commits log messages too - will your scripts notice that too?
> > > > >
> > > > > Hm, I don't think we look at merges as there's nothing there to actually
> > > > > commit.
> > > > >
> > > > > > For example if there's a few commits left in tip:*/urgent branches
> > > > > > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > > > > > v2.6.32 is released, i will then merge them into tip:sched/core,
> > > > > > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > > > > > a notification area to 'activate' them for -stable backporting via a
> > > > > > merge commit.
> > > > > >
> > > > > > This is how such merge commits would look like:
> > > > > >
> > > > > > Merge branch 'core/urgent' into core/rcu
> > > > > >
> > > > > > Merge reason: Pick up urgent fixes that did not make it into .32.0
> > > > > >
> > > > > > Cc: <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> > > > > > Signed-off-by: Ingo Molnar <[email protected]>
> > > > > >
> > > > > > This is not so rare of a situation as it might seem - for the trees i
> > > > > > maintain it happens in almost every release cycle - i typically skip
> > > > > > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > > > > > fixes - but they'd still be eligible for -stable.
> > > > >
> > > > > Ok, that would be good and fine with me.
> > > > >
> > > > > James, would your script pick this up, or does it need to also pay
> > > > > attention to merge commits?
> > > >
> > > > No ... because merge commits should effectively be empty (and when
> > > > they're not you can't generate an applyable diff). If I understand the
> > > > workflow, the desire is to have the whole branch sent to stable by
> > > > tagging the merge commit. That's possible ... it's exactly the same
> > > > logic I use in the commit scripts for the SCSI tree, so it should be
> > > > possible to extract the logic.
> > > >
> > > > By the looks of the above it's only a few commits, or is it the entire
> > > > branch?
> > >
> > > I'm thinking the commit would be the merge, right Ingo? So it would
> > > just be a single commit that has the marker in it.
> >
> > OK, so I can make it send you this just by removing the --no-merge flag
> > from the git rev-list the script uses to sift through what changed
> > (which I've already done).
> >
> > The slight problem is that further down, to generate the patch the
> > script uses git format-patch -k --stdout commit^..commit. For a merge
> > commit, this will generate a patch equivalent to the entire branch that
> > was merged, even though the commit message will only pick out some of
> > these ... is this OK?
> >
> > If not, I can look at using git show instead to generate the patches (it
> > will effectively generate null diffs for merge points with the stable
> > tag, which is closer to what you want).
> >
> > Alternatively, if you pick up the commits from Linus' tree anyway, I
> > could just stop producing diffs, which will save email bandwidth and
> > then be automatically correct whether the commit is a merge or not.
>
> No, I'd like to keep diffs, and use Linus's suggestion to get the
> correct diff, I have had that same bug in some scripts I have written as
> well.
OK, so git show --pretty=email -u --stat -M --cc <commit>
seems to be what works well (the options are -u to generate diff plus
stats, -M to detect renames and --cc to try to generate a diff for merge
points showing if there have been non trivial fix ups).
I've attached a sample below of what it will look like for a stable
tagged merge point. As you can see, because this is a simple merge, the
diff is empty.
James
---
From: James Bottomley <[email protected]>
To: [email protected]
Subject: Patch Upstream: Merge branch 'master'
of /pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6
Date: 12/11/09 10:07:58 (Thu, 12 Nov 2009 16:07:58 GMT)
commit: f2d5c3c1e34a7cc006de8e3cf94fe11f68dad894
From: James Bottomley <[email protected]>
Date: Thu, 12 Nov 2009 15:51:38 +0000
Subject: [PATCH] Merge branch 'master' of /pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6
Cc: Stable Tree <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
Signed-off-by: James Bottomley <[email protected]>
drivers/scsi/bfa/bfad_im.c | 2 +-
drivers/scsi/ipr.c | 42 ++++++++++++++++++++++++++++-------
drivers/scsi/ipr.h | 1 +
drivers/scsi/libsas/sas_expander.c | 1 -
drivers/scsi/pmcraid.c | 10 ++++----
drivers/scsi/scsi_transport_fc.c | 3 ++
include/scsi/scsi_host.h | 29 +++++++++---------------
7 files changed, 54 insertions(+), 34 deletions(-)
On Thu, 2009-11-12 at 07:15 -0800, H. Peter Anvin wrote:
> On 11/11/2009 11:45 PM, Ingo Molnar wrote:
> >
> > That's already possible via the [email protected] email
> > list. Anyone can subscribe to it and can procmail process it. It is also
>
> There are a couple of disadvantages with this -- a hera -> vger -> hera
> bounce, in particular (those two machines are nowhere close to each
> other, and there can be huge latecy.) For real-time robots running on
> hera, a lower-latency mechanism would be useful.
Change the hook on hera that sends the email to use a local alias that
includes git-commits-header@vger and a local list that processes on hera
can subscribe to? That could avoid the bounce...
On 11/12/2009 09:01 AM, David Dillow wrote:
> On Thu, 2009-11-12 at 07:15 -0800, H. Peter Anvin wrote:
>> On 11/11/2009 11:45 PM, Ingo Molnar wrote:
>>>
>>> That's already possible via the [email protected] email
>>> list. Anyone can subscribe to it and can procmail process it. It is also
>>
>> There are a couple of disadvantages with this -- a hera -> vger -> hera
>> bounce, in particular (those two machines are nowhere close to each
>> other, and there can be huge latecy.) For real-time robots running on
>> hera, a lower-latency mechanism would be useful.
>
> Change the hook on hera that sends the email to use a local alias that
> includes git-commits-header@vger and a local list that processes on hera
> can subscribe to? That could avoid the bounce...
Not really... a lot of the bounce delay is due to mail processing on
hera itself.
-hpa
On Thu, Nov 12, 2009 at 10:11:57AM -0600, James Bottomley wrote:
> On Wed, 2009-11-11 at 20:41 -0800, Greg KH wrote:
> > On Wed, Nov 11, 2009 at 12:50:24PM -0600, James Bottomley wrote:
> > > On Tue, 2009-11-10 at 11:37 -0800, Greg KH wrote:
> > > > On Tue, Nov 10, 2009 at 09:29:48AM -0500, James Bottomley wrote:
> > > > > On Mon, 2009-11-09 at 20:14 -0800, Greg KH wrote:
> > > > > > > A further question is, i can see using this tagging scheme in the future
> > > > > > > in merge commits log messages too - will your scripts notice that too?
> > > > > >
> > > > > > Hm, I don't think we look at merges as there's nothing there to actually
> > > > > > commit.
> > > > > >
> > > > > > > For example if there's a few commits left in tip:*/urgent branches
> > > > > > > (tip:sched/urgent, tip:core/urgent, tip:x86/urgent, etc.) by the time
> > > > > > > v2.6.32 is released, i will then merge them into tip:sched/core,
> > > > > > > tip:core/core, tip:x86/core, etc. - and i could use the merge commit as
> > > > > > > a notification area to 'activate' them for -stable backporting via a
> > > > > > > merge commit.
> > > > > > >
> > > > > > > This is how such merge commits would look like:
> > > > > > >
> > > > > > > Merge branch 'core/urgent' into core/rcu
> > > > > > >
> > > > > > > Merge reason: Pick up urgent fixes that did not make it into .32.0
> > > > > > >
> > > > > > > Cc: <[email protected]> # .32.x: 83f5b01: rcu: Fix long-grace-period race
> > > > > > > Signed-off-by: Ingo Molnar <[email protected]>
> > > > > > >
> > > > > > > This is not so rare of a situation as it might seem - for the trees i
> > > > > > > maintain it happens in almost every release cycle - i typically skip
> > > > > > > urgent branch merges after -rc8/-rc9, unless they are very-very-urgent
> > > > > > > fixes - but they'd still be eligible for -stable.
> > > > > >
> > > > > > Ok, that would be good and fine with me.
> > > > > >
> > > > > > James, would your script pick this up, or does it need to also pay
> > > > > > attention to merge commits?
> > > > >
> > > > > No ... because merge commits should effectively be empty (and when
> > > > > they're not you can't generate an applyable diff). If I understand the
> > > > > workflow, the desire is to have the whole branch sent to stable by
> > > > > tagging the merge commit. That's possible ... it's exactly the same
> > > > > logic I use in the commit scripts for the SCSI tree, so it should be
> > > > > possible to extract the logic.
> > > > >
> > > > > By the looks of the above it's only a few commits, or is it the entire
> > > > > branch?
> > > >
> > > > I'm thinking the commit would be the merge, right Ingo? So it would
> > > > just be a single commit that has the marker in it.
> > >
> > > OK, so I can make it send you this just by removing the --no-merge flag
> > > from the git rev-list the script uses to sift through what changed
> > > (which I've already done).
> > >
> > > The slight problem is that further down, to generate the patch the
> > > script uses git format-patch -k --stdout commit^..commit. For a merge
> > > commit, this will generate a patch equivalent to the entire branch that
> > > was merged, even though the commit message will only pick out some of
> > > these ... is this OK?
> > >
> > > If not, I can look at using git show instead to generate the patches (it
> > > will effectively generate null diffs for merge points with the stable
> > > tag, which is closer to what you want).
> > >
> > > Alternatively, if you pick up the commits from Linus' tree anyway, I
> > > could just stop producing diffs, which will save email bandwidth and
> > > then be automatically correct whether the commit is a merge or not.
> >
> > No, I'd like to keep diffs, and use Linus's suggestion to get the
> > correct diff, I have had that same bug in some scripts I have written as
> > well.
>
> OK, so git show --pretty=email -u --stat -M --cc <commit>
>
> seems to be what works well (the options are -u to generate diff plus
> stats, -M to detect renames and --cc to try to generate a diff for merge
> points showing if there have been non trivial fix ups).
>
> I've attached a sample below of what it will look like for a stable
> tagged merge point. As you can see, because this is a simple merge, the
> diff is empty.
Looks great to me, thanks for doing this.
greg k-h