Hi,
Beginning from 3.7 including 3.10 kernel, I noticed that from time to
time all CPUs turn out to be busy 100% and at the moment websites of my
clients work slowly or don’t work at all.
top screen:
https://www.dropbox.com/s/psuk2cwn6exinyd/3.9.cpu_migration.jpg
All CPUs are busy with kernel threads migration for about 3-5 minutes.
Exactly how often it periodically happens I don’t know, but it happens
once in 1-3 hours.
Approximately 30 my servers work on kernel 2.6.35 and there’s no such
problem there. We use kernels 3.7 – 3.9 on 5 servers and I see such
effect everywhere. If you downgrade on 2.6.35, then migration doesn’t
eat CPUs.
These servers are used for usual LAMP shared hosting.
Can I get migration as if on the kernel 2.6.35 or can this feature be
disabled?
--
BRGDS. Alexey Vlasov.
jabber: [email protected]
icq: 5128664
(CCs)
On Thu, 2013-08-22 at 19:00 +0400, Alexey Vlasov wrote:
> Hi,
>
> Beginning from 3.7 including 3.10 kernel, I noticed that from time to
> time all CPUs turn out to be busy 100% and at the moment websites of my
> clients work slowly or don’t work at all.
>
> top screen:
> https://www.dropbox.com/s/psuk2cwn6exinyd/3.9.cpu_migration.jpg
>
> All CPUs are busy with kernel threads migration for about 3-5 minutes.
> Exactly how often it periodically happens I don’t know, but it happens
> once in 1-3 hours.
>
> Approximately 30 my servers work on kernel 2.6.35 and there’s no such
> problem there. We use kernels 3.7 – 3.9 on 5 servers and I see such
> effect everywhere. If you downgrade on 2.6.35, then migration doesn’t
> eat CPUs.
>
> These servers are used for usual LAMP shared hosting.
>
> Can I get migration as if on the kernel 2.6.35 or can this feature be
> disabled?
I've not seen anything like this since the below went into 3.6, nor
heard of same. Before that, migration threads could _appear_ to eat CPU
under certain circumstances. Perhaps perf top -p <pick-a-thread> while
the problem is happening will show something.
commit 8f6189684eb4e85e6c593cd710693f09c944450a
Author: Mike Galbraith <[email protected]>
Date: Sat Aug 4 05:44:14 2012 +0200
sched: Fix migration thread runtime bogosity
Make stop scheduler class do the same accounting as other classes,
Migration threads can be caught in the act while doing exec balancing,
leading to the below due to use of unmaintained ->se.exec_start. The
load that triggered this particular instance was an apparently out of
control heavily threaded application that does system monitoring in
what equated to an exec bomb, with one of the VERY frequently migrated
tasks being ps.
%CPU PID USER CMD
99.3 45 root [migration/10]
97.7 53 root [migration/12]
97.0 57 root [migration/13]
90.1 49 root [migration/11]
89.6 65 root [migration/15]
88.7 17 root [migration/3]
80.4 37 root [migration/8]
78.1 41 root [migration/9]
44.2 13 root [migration/2]
Hi Peter,
On Sun, Aug 25, 2013 at 04:28:37PM +0200, Peter Zijlstra wrote:
>
> Gargh.. I've never seen anything like that. Nor ever had a report like
> this. Is there anything in particular one can do to try and reproduce
> this?
I don't know how to reproduce it. This happens by itself and only on
high-loaded servers. For example this happens almost every hour on one
server with kernel 3.8.11 with 10k web-sites and 5k MySQL databases. On
another server with kernel 3.9.4 with same load this can take place 3-5
times per day. Sometimes this happens almost synchronously on both
servers.
I returned to kernel 2.6.35 on servers where this often took place. Or
they are not high-loaded enough that this effect doesn't appear.
For example here is server which earlier worked on kernel 3.9.4. It is
high-loaded, but migration stopped to eat CPUs after downgrade to
2.6.35.
# uname -r
2.6.35.7
# uptime
13:56:34 up 32 days, 10:31, 10 users, load average: 24.44, 23.44, 24.13
# ps -u root -o user,bsdtime,comm | grep -E 'COMMAND|migration'
USER TIME COMMAND
root 4:20 migration/0
root 6:07 migration/1
root 17:00 migration/2
root 5:23 migration/3
root 16:43 migration/4
root 3:48 migration/5
root 12:28 migration/6
root 3:44 migration/7
root 12:25 migration/8
root 3:49 migration/9
root 1:52 migration/10
root 2:51 migration/11
root 1:28 migration/12
root 2:43 migration/13
root 2:16 migration/14
root 4:53 migration/15
root 2:15 migration/16
root 4:13 migration/17
root 2:13 migration/18
root 4:21 migration/19
root 2:07 migration/20
root 4:13 migration/21
root 2:13 migration/22
root 3:26 migration/23
For comparison 3.9.4:
# uptime
13:55:49 up 11 days, 15:36, 11 users, load average: 24.62, 24.36, 23.63
USER TIME COMMAND
root 233:51 migration/0
root 233:38 migration/1
root 231:57 migration/2
root 233:26 migration/3
root 231:46 migration/4
root 233:26 migration/5
root 231:37 migration/6
root 232:56 migration/7
root 231:09 migration/8
root 232:34 migration/9
root 231:04 migration/10
root 232:22 migration/11
root 230:50 migration/12
root 232:16 migration/13
root 230:38 migration/14
root 231:51 migration/15
root 230:04 migration/16
root 230:16 migration/17
root 230:06 migration/18
root 230:22 migration/19
root 229:45 migration/20
root 229:43 migration/21
root 229:27 migration/22
root 229:24 migration/23
root 229:11 migration/24
root 229:25 migration/25
root 229:16 migration/26
root 228:58 migration/27
root 228:48 migration/28
root 229:06 migration/29
root 228:25 migration/30
root 228:25 migration/31
> Could you perhaps send your .config and a function (or function-graph)
> trace for when this happens?
My .config
https://www.dropbox.com/s/vuwvalj58cfgahu/.config_3.9.4-1gb-csmb-tr
I can't make trace because it isn't turned on on my kernels. I will be
able to reboot servers on weekend as there are many clients there and
will send you trace.
> Also, do you use weird things like cgroup/cpusets or other such fancy
> stuff? If so, could you outline your setup?
Grsec patch is used on all kernels. Also there is following patch only on
kernel 3.8.11:
--- kernel/cgroup.c.orig
+++ kernel/cgroup.c
@@ -1931,7 +1931,8 @@
ss->attach(cgrp, &tset);
}
- synchronize_rcu();
+ synchronize_rcu_expedited();
/*
* wake up rmdir() waiter. the rmdir should fail since the
Aslo I use https://github.com/facebook/flashcache/
Actually I really use cgroup namely controllers cpuacct, memory, blkio.
I create cgroup for every user on server, where all users processes are
running. To make it work there are needed patches in Apache/prefork, SSH
and other users staff. There can be about 10k-15k users and accordingly
same amount of cgroups.
The other day I disabled all cgroups, but controllers are still mounted.
# cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 2 1 1
cpuacct 3 1 1
memory 4 1 1
blkio 5 1 1
But migration still eats CPUs. However I also use cgroup on kernel
2.6.35.
--
BRGDS. Alexey Vlasov.
On Sun, Aug 25, 2013 at 04:28:37PM +0200, Peter Zijlstra wrote:
> and a function (or function-graph)
> trace for when this happens?
Unfortunately I could not make trace.
First when migration threads start to eat CPUs and when I turn on trace:
# echo function > current_tracer
# echo 1 > tracing_on
then server is down at once. LA comes up to several thousands, it stops
to receive commands. Sometimes I even have to reboot it.
And second when I manage to get trace then it shows little useful data.
https://www.dropbox.com/s/kxgwr9fuckddtgf/trace.txt.gz
Probably this junk appears from grsec patch? And I don't see migration
threads in it.
> Perhaps perf top -p <pick-a-thread> while the problem is happening will
> show something.
Also "perf top" doesn't show anything. It gives same values when
migration threads eat CPUs and when everything is OK on server.
=========================================
PerfTop: 68932 irqs/sec kernel:60.3% exact: 0.0% [4000Hz cycles], (all, 32 CPUs)
---------------------------------------------------------------------------------
64.11% [netconsole] [k] 0xffffffff81000de3
34.19% [unknown] [.] 0x0000000000585c45
0.40% libc-2.15.so [.] 0x0000000000124b10
0.23% perf [.] 0x000000000004df4f
0.14% ld-2.15.so [.] 0x00000000000183e7
0.08% libc-2.15.so [.] _nss_files_parse_grent
0.06% libc-2.15.so [.] _nss_files_parse_pwent
0.04% libc-2.15.so [.] memchr
0.04% php-cgi [.] match
0.03% libcrypt-2.15.so [.] 0x0000000000004eab
0.03% bash [.] 0x000000000001fe5a
0.02% libc-2.15.so [.] _IO_getline_info
0.02% php-cgi [.] php_pcre_exec
0.02% libc-2.15.so [.] _dl_addr
0.02% eaccelerator.so [.] eaccelerator_crc32
0.02% php-cgi [.] zend_hash_quick_find
0.02% libnss_compat-2.15.so [.] 0x00000000000055f1
0.01% php-cgi [.] zendparse
0.01% libc-2.15.so [.] fgets_unlocked
0.01% libnss_compat-2.15.so [.] _nss_compat_initgroups_dyn
=========================================
Now server is more loaded and eating CPUs by migration threads happens more often
and it lasts longer.
--
BRGDS. Alexey Vlasov.
* Alexey Vlasov <[email protected]> wrote:
> On Sun, Aug 25, 2013 at 04:28:37PM +0200, Peter Zijlstra wrote:
>
> > and a function (or function-graph)
> > trace for when this happens?
>
> Unfortunately I could not make trace.
> First when migration threads start to eat CPUs and when I turn on trace:
> # echo function > current_tracer
> # echo 1 > tracing_on
> then server is down at once. LA comes up to several thousands, it stops
> to receive commands. Sometimes I even have to reboot it.
Could you try the latest -tip tree at:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
In particular this fix:
5a8e01f8fa51 sched/cputime: Do not scale when utime == 0
Could perhaps fix the phantom CPU overhead you are seeing?
Thanks,
Ingo
On Thu, Sep 05, 2013 at 01:12:52PM +0200, Ingo Molnar wrote:
>
> Could you try the latest -tip tree at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
>
> In particular this fix:
>
> 5a8e01f8fa51 sched/cputime: Do not scale when utime == 0
>
> Could perhaps fix the phantom CPU overhead you are seeing?
This kernel solved problems.
Thank you.
--
BRGDS. Alexey Vlasov.
Hi Ingo,
On Wed, Sep 11, 2013 at 07:18:14PM +0400, Alexey Vlasov wrote:
> On Thu, Sep 05, 2013 at 01:12:52PM +0200, Ingo Molnar wrote:
> >
> > Could you try the latest -tip tree at:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
> >
> > In particular this fix:
> >
> > 5a8e01f8fa51 sched/cputime: Do not scale when utime == 0
> >
> > Could perhaps fix the phantom CPU overhead you are seeing?
>
> This kernel solved problems.
> Thank you.
One problem is solved, but there appeared another one. Now this kernel
began to crash.
Crashdump is in attachment.
--
BRGDS. Alexey Vlasov.
