Hi,
I'm fuzzing 3.4.0-rc5 (mostly with
http://code.google.com/p/iknowthis/), and got to the point that new
fuzzing threads/processes don't want to run any more. I have script
that periodically sends SIGCONT to all processes.
while [ 1 ]; do su test -c 'kill -CONT -1'; su test2 -c 'kill -CONT
-1'; su nobody -c 'kill -CONT -1'; sleep 300; done
It doesn't work:
root@ise-test:~/kern-fuz# ./cont.sh
su: Cannot fork user shell
su: Cannot fork user shell
su: Cannot fork user shell
root@ise-test:~/kern-fuz# strace -e mmap,clone su test -c 'kill -CONT
-1' 2>&1 | grep "= \-1"
clone(child_stack=0,
flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7fadf334f9f0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = -1 ENOMEM (Cannot allocate memory)
It happens with every user (even newly created one), so it doesn't
seem to be any per-id rlimit.
root@ise-test:~/kern-fuz# su - test3
su: Cannot fork user shell
root@ise-test:~/kern-fuz# su - test3
$ <-- success
root@ise-test:~/kern-fuz# su - test3
su: Cannot fork user shell
root@ise-test:~/kern-fuz# su - test3
su: Cannot fork user shell
root@ise-test:~/kern-fuz# su - test3
su: Cannot fork user shell
Not sure how to debug it yet. I can run kdb/kgdb on it, but before I
dive into mm structures, I though I attach some proc files from your
entertainment, maybe you can spot anything interesting there.
Also, whatever happened on this machine, i.e. any syscall during
fuzzing, was invoked from non-root user.
--
Robert Święcki
On Mon, 7 May 2012, Robert Święcki wrote:
> root@ise-test:~/kern-fuz# ./cont.sh
> su: Cannot fork user shell
> su: Cannot fork user shell
> su: Cannot fork user shell
>
> root@ise-test:~/kern-fuz# strace -e mmap,clone su test -c 'kill -CONT
> -1' 2>&1 | grep "= \-1"
> clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7fadf334f9f0) = -1 ENOMEM (Cannot allocate memory)
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = -1 ENOMEM (Cannot allocate memory)
Hmmm... That looks like some maximum virtual memory limit was violated.
Check ulimit and the overcommit settings (see /proc/meminfo's commitlimit
etc)
On Mon, May 7, 2012 at 10:15 PM, Christoph Lameter <[email protected]> wrote:
> On Mon, 7 May 2012, Robert Święcki wrote:
>
>> root@ise-test:~/kern-fuz# ./cont.sh
>> su: Cannot fork user shell
>> su: Cannot fork user shell
>> su: Cannot fork user shell
>>
>> root@ise-test:~/kern-fuz# strace -e mmap,clone su test -c 'kill -CONT
>> -1' 2>&1 | grep "= \-1"
>> clone(child_stack=0,
>> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
>> child_tidptr=0x7fadf334f9f0) = -1 ENOMEM (Cannot allocate memory)
>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
>> 0) = -1 ENOMEM (Cannot allocate memory)
>
> Hmmm... That looks like some maximum virtual memory limit was violated.
>
> Check ulimit and the overcommit settings (see /proc/meminfo's commitlimit
> etc)
Yup (btw: I attached dump of some proc files and some debug commands
in the original e-mail - can be found here
http://marc.info/?l=linux-kernel&m=133640623421007&w=2 in case some
MTA removed them)
CommitLimit: 1981528 kB
Committed_AS: 1916788 kB
just not sure if Committed_AS should present this kind of value. Did I
just hit a legitimate condition, or may it suggest a bug? I'm a bit
puzzled cause
root@ise-test:/proc# grep Mem /proc/meminfo
MemTotal: 3963060 kB
MemFree: 3098324 kB
Also, some sysctl values:
vm.overcommit_memory = 2
vm.overcommit_ratio = 50
--
Robert Święcki
On Mon, 7 May 2012, Robert Święcki wrote:
> Yup (btw: I attached dump of some proc files and some debug commands
> in the original e-mail - can be found here
> http://marc.info/?l=linux-kernel&m=133640623421007&w=2 in case some
> MTA removed them)
>
> CommitLimit: 1981528 kB
> Committed_AS: 1916788 kB
>
> just not sure if Committed_AS should present this kind of value. Did I
> just hit a legitimate condition, or may it suggest a bug? I'm a bit
> puzzled cause
This is a legitimate condition. No bug.
>
> root@ise-test:/proc# grep Mem /proc/meminfo
> MemTotal: 3963060 kB
> MemFree: 3098324 kB
Physical memory is free in quantity but virtual memory is exhausted.
> Also, some sysctl values:
> vm.overcommit_memory = 2
> vm.overcommit_ratio = 50
Setting overcommit memory to 2 means that the app is strictly policed
for staying within bounds on virtual memory. Dont do that.
See linux source linux/Documentation/vm/overcommit-accounting for more
details.
On Tue, May 8, 2012 at 4:02 PM, Christoph Lameter <[email protected]> wrote:
> On Mon, 7 May 2012, Robert Święcki wrote:
>
>> Yup (btw: I attached dump of some proc files and some debug commands
>> in the original e-mail - can be found here
>> http://marc.info/?l=linux-kernel&m=133640623421007&w=2 in case some
>> MTA removed them)
>>
>> CommitLimit: 1981528 kB
>> Committed_AS: 1916788 kB
>>
>> just not sure if Committed_AS should present this kind of value. Did I
>> just hit a legitimate condition, or may it suggest a bug? I'm a bit
>> puzzled cause
>
> This is a legitimate condition. No bug.
>>
>> root@ise-test:/proc# grep Mem /proc/meminfo
>> MemTotal: 3963060 kB
>> MemFree: 3098324 kB
>
> Physical memory is free in quantity but virtual memory is exhausted.
>
>> Also, some sysctl values:
>> vm.overcommit_memory = 2
>> vm.overcommit_ratio = 50
>
> Setting overcommit memory to 2 means that the app is strictly policed
> for staying within bounds on virtual memory. Dont do that.
>
> See linux source linux/Documentation/vm/overcommit-accounting for more
> details.
Thanks Christoph.
--
Robert Święcki
> Setting overcommit memory to 2 means that the app is strictly policed
> for staying within bounds on virtual memory. Dont do that.
For a fuzz test you probably do want it at 2 to avoid the box dying in a
swap storm.
Alan