The function may_expand_vm() may return a false positive if the proposed
increment (npages) is sufficient large to overflow the expression:
cur + npages
Assuming that cur < lim is an invariant, the proposed patch re-arranges
the expression to avoid unsigned arithmetic overflow.
More robustly:
if (cur > lim || npages > lim - cur)
return 0;
might be preferred if it cannot be guaranteed that cur < lim.
--- linux-2.6.21_mvlcge500/mm/mmap.c.orig 2008-06-30 22:43:38.000000000
-0700
+++ linux-2.6.21_mvlcge500/mm/mmap.c 2009-10-16 07:42:23.000000000 -0700
@@ -2303,7 +2303,7 @@ int may_expand_vm(struct mm_struct *mm,
lim = current->signal->rlim[RLIMIT_AS].rlim_cur >> PAGE_SHIFT;
- if (cur + npages > lim)
+ if (npages > lim - cur)
return 0;
return 1;
}