Date: Mon, 19 Oct 2009 09:53:50 +0200
From: Johannes Weiner <hannes@cmpxchg.org>
To: Earl Chew <earl_chew@agilent.com>
Cc: linux-kernel@vger.kernel.org
Subject: Re: Arithmetic overflow in may_expand_vm()
Message-ID: <20091019075350.GA1769@cmpxchg.org>
References: <4AD75AE3.80803@agilent.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4AD75AE3.80803@agilent.com>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1220
Lines: 36

Hi,

On Thu, Oct 15, 2009 at 10:24:51AM -0700, Earl Chew wrote:
> This code currently reads:
> 
> >int may_expand_vm(struct mm_struct *mm, unsigned long npages)
> >{
> >        unsigned long cur = mm->total_vm;       /* pages */
> >        unsigned long lim;
> >
> >        lim = current->signal->rlim[RLIMIT_AS].rlim_cur >> PAGE_SHIFT;
> >
> >        if (cur + npages > lim)
> >                return 0;
> >        return 1;
> >}
> 
> If npages is stupendously large, the failure predicate may
> return a false negative due to (cur + npages) overflowing and
> wrapping.

Can this really happen?

npages always originates in a value of byte granularity, giving a
theoretical maximum of ~0UL >> PAGE_SHIFT (checking for more than the
number of addressable bytes just makes no sense).

And mm->total_vm is always PAGE_SIZE times smaller than total user
address space (which in turn is always less than ~0UL).

So I can not see this overflow being possible with PAGE_SHIFT > 0.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/