> -----?ʼ?ԭ??-----
> ??????: Peter Zijlstra <[email protected]>
> ????ʱ??: 2021??7??29?? 18:20
> ?ռ???: Li,Rongqing <[email protected]>
> ????: [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]
> ????: Re: [PATCH][RFC] sched/cpuacct: Fix cpuacct charge
>
> On Tue, Jul 20, 2021 at 02:04:41PM +0800, Li RongQing wrote:
> > get_irq_regs only work for current running cpu, but the task, whose
> > cpuacct will be charged, maybe run different cpu, like Cpu 2 wake up a
> > kernel thread to CPU 3, cause CPU 3 task are charged with the
> > following stack
> >
> > cpuacct_charge+0xd8/0x100
> > update_curr+0xe1/0x1e0
> > enqueue_entity+0x144/0x6e0
> > enqueue_task_fair+0x93/0x900
> > ttwu_do_activate+0x4b/0x90
> > try_to_wake_up+0x20b/0x530
> > ? update_dl_rq_load_avg+0x10f/0x210
> > swake_up_locked.part.1+0x13/0x40
> > swake_up_one+0x27/0x40
> > rcu_process_callbacks+0x484/0x4f0
> > ? run_rebalance_domains_bt+0x5a/0x180
> > __do_softirq+0xe3/0x308
> > irq_exit+0xf0/0x100
> > smp_apic_timer_interrupt+0x74/0x160
> > apic_timer_interrupt+0xf/0x20
> > </IRQ>
> > RIP: 0033:0x456947
> >
> > so define a get_irq_regs_cpu which returns the required cpu irq
> > registers
> >
> > BUT it should be not safe, and do not know what it should be like in MIPS?
> >
> > Fixes: dbe9337109c2 "(sched/cpuacct: Fix charge cpuacct.usage_sys)"
> > Co-developed-by: Zhao Jie <[email protected]>
> > Signed-off-by: Zhao Jie <[email protected]>
> > Signed-off-by: Li RongQing <[email protected]>
> > ---
> > include/asm-generic/irq_regs.h | 5 +++++
> > kernel/sched/cpuacct.c | 3 ++-
> > 2 files changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/include/asm-generic/irq_regs.h
> > b/include/asm-generic/irq_regs.h index 2e7c6e8..93e2579 100644
> > --- a/include/asm-generic/irq_regs.h
> > +++ b/include/asm-generic/irq_regs.h
> > @@ -21,6 +21,11 @@ static inline struct pt_regs *get_irq_regs(void)
> > return __this_cpu_read(__irq_regs);
> > }
> >
> > +static inline struct pt_regs *get_irq_regs_cpu(int cpu) {
> > + return per_cpu(__irq_regs, cpu);
> > +}
>
> This primitive just cannot be right... it'll get you some random data.
True
Seem no easy to fix. How about a partial fix
diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c
index 893eece..48b117e 100644
--- a/kernel/sched/cpuacct.c
+++ b/kernel/sched/cpuacct.c
@@ -340,7 +340,12 @@ void cpuacct_charge(struct task_struct *tsk, u64 cputime)
{
struct cpuacct *ca;
int index = CPUACCT_STAT_SYSTEM;
- struct pt_regs *regs = get_irq_regs() ? : task_pt_regs(tsk);
+ struct pt_regs *regs;
+
+ if (task_cpu(tsk) == raw_smp_processor_id())
+ regs = get_irq_regs() ? : task_pt_regs(tsk);
+ else
+ regs = task_pt_regs(tsk);
if (regs && user_mode(regs))
index = CPUACCT_STAT_USER;
On Fri, Jul 30, 2021 at 08:16:54AM +0000, Li,Rongqing wrote:
> > On Tue, Jul 20, 2021 at 02:04:41PM +0800, Li RongQing wrote:
> > > get_irq_regs only work for current running cpu, but the task, whose
> > > cpuacct will be charged, maybe run different cpu, like Cpu 2 wake up a
> > > kernel thread to CPU 3, cause CPU 3 task are charged with the
> > > following stack
> > >
> > > cpuacct_charge+0xd8/0x100
> > > update_curr+0xe1/0x1e0
> > > enqueue_entity+0x144/0x6e0
> > > enqueue_task_fair+0x93/0x900
> > > ttwu_do_activate+0x4b/0x90
> > > try_to_wake_up+0x20b/0x530
> > > ? update_dl_rq_load_avg+0x10f/0x210
> > > swake_up_locked.part.1+0x13/0x40
> > > swake_up_one+0x27/0x40
> > > rcu_process_callbacks+0x484/0x4f0
> > > ? run_rebalance_domains_bt+0x5a/0x180
> > > __do_softirq+0xe3/0x308
> > > irq_exit+0xf0/0x100
> > > smp_apic_timer_interrupt+0x74/0x160
> > > apic_timer_interrupt+0xf/0x20
> > > </IRQ>
> > > RIP: 0033:0x456947
> > >
> > > so define a get_irq_regs_cpu which returns the required cpu irq
> > > registers
> > >
> > > BUT it should be not safe, and do not know what it should be like in MIPS?
> > >
> > > Fixes: dbe9337109c2 "(sched/cpuacct: Fix charge cpuacct.usage_sys)"
> > > Co-developed-by: Zhao Jie <[email protected]>
> > > Signed-off-by: Zhao Jie <[email protected]>
> > > Signed-off-by: Li RongQing <[email protected]>
> > > ---
> > > include/asm-generic/irq_regs.h | 5 +++++
> > > kernel/sched/cpuacct.c | 3 ++-
> > > 2 files changed, 7 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/include/asm-generic/irq_regs.h
> > > b/include/asm-generic/irq_regs.h index 2e7c6e8..93e2579 100644
> > > --- a/include/asm-generic/irq_regs.h
> > > +++ b/include/asm-generic/irq_regs.h
> > > @@ -21,6 +21,11 @@ static inline struct pt_regs *get_irq_regs(void)
> > > return __this_cpu_read(__irq_regs);
> > > }
> > >
> > > +static inline struct pt_regs *get_irq_regs_cpu(int cpu) {
> > > + return per_cpu(__irq_regs, cpu);
> > > +}
> >
> > This primitive just cannot be right... it'll get you some random data.
>
> True
>
> Seem no easy to fix. How about a partial fix
>
> diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c
> index 893eece..48b117e 100644
> --- a/kernel/sched/cpuacct.c
> +++ b/kernel/sched/cpuacct.c
> @@ -340,7 +340,12 @@ void cpuacct_charge(struct task_struct *tsk, u64 cputime)
> {
> struct cpuacct *ca;
> int index = CPUACCT_STAT_SYSTEM;
> - struct pt_regs *regs = get_irq_regs() ? : task_pt_regs(tsk);
> + struct pt_regs *regs;
> +
> + if (task_cpu(tsk) == raw_smp_processor_id())
> + regs = get_irq_regs() ? : task_pt_regs(tsk);
> + else
> + regs = task_pt_regs(tsk);
>
> if (regs && user_mode(regs))
> index = CPUACCT_STAT_USER;
It still suffers from task_pt_regs().
Why not make cpuacct use cgroup2's approach? Remember only delta_exec
here, then on reading cpuacct.usage_*, use cputime_adjust() to scale the
user/sys from cpuacct_account_field().
It's arguably more than just a fix for cgroup1, but there have been a
few complaints about this function lately.
> rcu_read_lock();
>
> for (ca = task_ca(tsk); ca; ca = parent_ca(ca))
> __this_cpu_add(ca->cpuusage->usages[index], cputime);
>
> rcu_read_unlock();
By the way, I think the __this_cpu_add() can be wrong in cases like you
originally describe. Seems like a bug in 73e6aafd9ea8 ("sched/cpuacct:
Simplify the cpuacct code").
> > diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c index
> > 893eece..48b117e 100644
> > --- a/kernel/sched/cpuacct.c
> > +++ b/kernel/sched/cpuacct.c
> > @@ -340,7 +340,12 @@ void cpuacct_charge(struct task_struct *tsk, u64
> > cputime) {
> > struct cpuacct *ca;
> > int index = CPUACCT_STAT_SYSTEM;
> > - struct pt_regs *regs = get_irq_regs() ? : task_pt_regs(tsk);
> > + struct pt_regs *regs;
> > +
> > + if (task_cpu(tsk) == raw_smp_processor_id())
> > + regs = get_irq_regs() ? : task_pt_regs(tsk);
> > + else
> > + regs = task_pt_regs(tsk);
> >
> > if (regs && user_mode(regs))
> > index = CPUACCT_STAT_USER;
>
> It still suffers from task_pt_regs().
>
> Why not make cpuacct use cgroup2's approach? Remember only delta_exec
> here, then on reading cpuacct.usage_*, use cputime_adjust() to scale the
> user/sys from cpuacct_account_field().
>
I think your suggestion is reasonable, Could you send a patch
> It's arguably more than just a fix for cgroup1, but there have been a few
> complaints about this function lately.
>
> > rcu_read_lock();
> >
> > for (ca = task_ca(tsk); ca; ca = parent_ca(ca))
> > __this_cpu_add(ca->cpuusage->usages[index], cputime);
> >
> > rcu_read_unlock();
>
> By the way, I think the __this_cpu_add() can be wrong in cases like you originally
> describe. Seems like a bug in 73e6aafd9ea8 ("sched/cpuacct:
> Simplify the cpuacct code").
We find this issue too.
-Li
On Tue, Aug 17, 2021 at 03:55:08AM +0000, Li,Rongqing wrote:
> > > diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c index
> > > 893eece..48b117e 100644
> > > --- a/kernel/sched/cpuacct.c
> > > +++ b/kernel/sched/cpuacct.c
> > > @@ -340,7 +340,12 @@ void cpuacct_charge(struct task_struct *tsk, u64
> > > cputime) {
> > > struct cpuacct *ca;
> > > int index = CPUACCT_STAT_SYSTEM;
> > > - struct pt_regs *regs = get_irq_regs() ? : task_pt_regs(tsk);
> > > + struct pt_regs *regs;
> > > +
> > > + if (task_cpu(tsk) == raw_smp_processor_id())
> > > + regs = get_irq_regs() ? : task_pt_regs(tsk);
> > > + else
> > > + regs = task_pt_regs(tsk);
> > >
> > > if (regs && user_mode(regs))
> > > index = CPUACCT_STAT_USER;
> >
> > It still suffers from task_pt_regs().
> >
> > Why not make cpuacct use cgroup2's approach? Remember only delta_exec
> > here, then on reading cpuacct.usage_*, use cputime_adjust() to scale the
> > user/sys from cpuacct_account_field().
> >
>
> I think your suggestion is reasonable, Could you send a patch
I'll leave that to someone else, got other things going on for now.