From: Michael Neuling <mikey@neuling.org>
To: anton@samba.org, linux-kernel@vger.kernel.org, linuxppc-dev@ozlabs.org,
       Serge Hallyn <serue@us.ibm.com>, WANG Cong <xiyou.wangcong@gmail.com>,
       Paul Mackerras <paulus@samba.org>, benh@kernel.crashing.org
cc: miltonm@bga.com
MIME-Version: 1.0
Subject: Stack size protection broken on ppc64
Date: Sat, 06 Feb 2010 11:43:13 +1100
Message-ID: <3984.1265416993@neuling.org>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1433
Lines: 43

On recent ppc64 kernels, limiting the stack (using 'ulimit -s blah') is
now more restrictive than it was before.  On 2.6.31 with 4k pages I
could run 'ulimit -s 16; /usr/bin/test' without a problem.  Now with
mainline, even 'ulimit -s 64; /usr/bin/test' gets killed.

Using 64k pages is even worse.  I can't even run '/bin/ls' with a 1MB
stack (ulimit -s 1024; /bin/ls).  Hence, it seems new kernels are too
restrictive, rather than the old kernels being too liberal.

I've not tested with any other architectures.  

Bisecting, I found that this is the culprit (which is in 2.6.32)

  commit fc63cf237078c86214abcb2ee9926d8ad289da9b
  Author: Anton Blanchard <anton@samba.org>
  exec:   setup_arg_pages() fails to return errors

Looking at the patch, it's probably just unmasking a preexisting issue.
The error path for expand_stack() (and others) was modified to:
---
	ret = expand_stack(vma, stack_base);
	if (ret)
		ret = -EFAULT;

out_unlock:
	up_write(&mm->mmap_sem);
-       return 0;
+       return ret;
 }
 EXPORT_SYMBOL(setup_arg_pages);
---

So previously expand_stack errors were not returned correctly by
setup_arg_pages, but now they are.

Any clues how to fix this?

Mikey
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/