hi,
i'm trying to debug an ugly application with ElectricFence.
in fact, on x86-64 box with 8GB ram and 16GB swap i'm getting following error:
"ElectricFence Exiting: mprotect() failed: Cannot allocate memory"
the program has been compiled with gcc-4.5, glibc-2.11.1, kernel-2.6.32.
did you ever come across such (kernel/glibc) limitations?
here's a simple testcase which triggs -ENOMEM in mprotect().
#define N 100
#include<stdlib.h>
double **bm;
int main(){
int i;
long NN = 4*N*N;
int kmax=100;
bm = (double **)malloc((time_t)NN*sizeof(double *));
for(i=0; i<NN; ++i){
bm[i] = (double*)malloc((time_t)kmax*sizeof(double));
}
for(i=0; i<NN; ++i){
free(bm[i]);
}
free(bm);
return 0;
}
thanks for any hint,
BR,
Pawel.
On Tuesday 20 April 2010 19:05:20 Paweł Sikora wrote:
> i'm trying to debug an ugly application with ElectricFence.
electricfence does a lot of ugly memory tricks to do its thing, including, but
not limited to, overriding memory related symbols. best to seek help from the
electricfence authors.
unless of course, things still fail when you dont use electricfence.
-mike
On Wednesday 21 April 2010 01:17:22 Mike Frysinger wrote:
> On Tuesday 20 April 2010 19:05:20 Pawe? Sikora wrote:
> > i'm trying to debug an ugly application with ElectricFence.
>
> electricfence does a lot of ugly memory tricks to do its thing, including,
> but not limited to, overriding memory related symbols. best to seek help
> from the electricfence authors.
so, let's avoid EF and run following test:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
void* my_alloc( size_t n )
{
size_t ps = getpagesize();
printf( "request for %Zd bytes => ", n );
/* alloc PAGE_SIZE + n */
char* p = mmap( 0, ps + n, PROT_READ | PROT_WRITE, MAP_SHARED |
MAP_ANONYMOUS, -1, 0 );
if ( p == MAP_FAILED )
__builtin_abort();
/* block guard page */
int rc = mprotect( p, ps, PROT_NONE );
if ( rc != 0 )
__builtin_abort();
char* q = p + ps;
printf( "guard page @ %p, allocated region @ %p\n", p, q );
return q;
}
int main()
{
#define N 100
size_t NN = 4*100*100;
size_t kmax = 100;
int i;
double **bm = (double **)my_alloc( NN * sizeof( double* ) );
for( i = 0; i < NN; ++i )
{
bm[ i ] = (double*)my_alloc( kmax * sizeof( double ) );
}
// leak...
return 0;
}
and the result is...
(...)
mmap(NULL, 4896, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) =
0x7f5fd97df000
mprotect(0x7f5fd97df000, 4096, PROT_NONE) = -1 ENOMEM (Cannot allocate memory)
On Tuesday 20 April 2010 19:44:18 Pawe? Sikora wrote:
> so, let's avoid EF and run following test:
good, the info is useful now
> mmap(NULL, 4896, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) =
> 0x7f5fd97df000
> mprotect(0x7f5fd97df000, 4096, PROT_NONE) = -1 ENOMEM (Cannot allocate
> memory)
since you're using funcs that go straight to the kernel, this most likely isnt
related to glibc at all, so sticking to LKML would be best. the next step
would be to look at the linux internals and find out why this is failing.
-mike
Le mercredi 21 avril 2010 à 01:44 +0200, Paweł Sikora a écrit :
> On Wednesday 21 April 2010 01:17:22 Mike Frysinger wrote:
> > On Tuesday 20 April 2010 19:05:20 Paweł Sikora wrote:
> > > i'm trying to debug an ugly application with ElectricFence.
> >
> > electricfence does a lot of ugly memory tricks to do its thing, including,
> > but not limited to, overriding memory related symbols. best to seek help
> > from the electricfence authors.
>
> so, let's avoid EF and run following test:
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> #include <sys/mman.h>
>
> void* my_alloc( size_t n )
> {
> size_t ps = getpagesize();
> printf( "request for %Zd bytes => ", n );
> /* alloc PAGE_SIZE + n */
> char* p = mmap( 0, ps + n, PROT_READ | PROT_WRITE, MAP_SHARED |
> MAP_ANONYMOUS, -1, 0 );
> if ( p == MAP_FAILED )
> __builtin_abort();
> /* block guard page */
> int rc = mprotect( p, ps, PROT_NONE );
> if ( rc != 0 )
> __builtin_abort();
> char* q = p + ps;
> printf( "guard page @ %p, allocated region @ %p\n", p, q );
> return q;
> }
>
> int main()
> {
> #define N 100
> size_t NN = 4*100*100;
> size_t kmax = 100;
> int i;
>
> double **bm = (double **)my_alloc( NN * sizeof( double* ) );
> for( i = 0; i < NN; ++i )
> {
> bm[ i ] = (double*)my_alloc( kmax * sizeof( double ) );
> }
> // leak...
> return 0;
> }
>
> and the result is...
>
> (...)
> mmap(NULL, 4896, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) =
> 0x7f5fd97df000
> mprotect(0x7f5fd97df000, 4096, PROT_NONE) = -1 ENOMEM (Cannot allocate memory)
Have you checked available memory on your system ? Or user limit ?
You test program is going to allocate
79 + 1 pages for bm
1 + 1 for each double arrays (x 40000)
So in the end your program is allocating 80080 pages, so about
312MBytes.
It not that big for a 64bits system.
Check limits such as
-d the maximum size of a process's data segment
-l the maximum size a process may lock into memory
-m the maximum resident set size
Regards.
--
Yann Droneaud
Dnia 21-04-2010 o 11:22:19 Yann Droneaud <[email protected]> napisał(a):
> Le mercredi 21 avril 2010 à 01:44 +0200, Paweł Sikora a écrit :
>> On Wednesday 21 April 2010 01:17:22 Mike Frysinger wrote:
>> > On Tuesday 20 April 2010 19:05:20 Paweł Sikora wrote:
>> > > i'm trying to debug an ugly application with ElectricFence.
>> >
>> > electricfence does a lot of ugly memory tricks to do its thing,
>> including,
>> > but not limited to, overriding memory related symbols. best to seek
>> help
>> > from the electricfence authors.
>>
>> so, let's avoid EF and run following test:
>>
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <unistd.h>
>> #include <sys/mman.h>
>>
>> void* my_alloc( size_t n )
>> {
>> size_t ps = getpagesize();
>> printf( "request for %Zd bytes => ", n );
>> /* alloc PAGE_SIZE + n */
>> char* p = mmap( 0, ps + n, PROT_READ | PROT_WRITE, MAP_SHARED |
>> MAP_ANONYMOUS, -1, 0 );
>> if ( p == MAP_FAILED )
>> __builtin_abort();
>> /* block guard page */
>> int rc = mprotect( p, ps, PROT_NONE );
>> if ( rc != 0 )
>> __builtin_abort();
>> char* q = p + ps;
>> printf( "guard page @ %p, allocated region @ %p\n", p, q );
>> return q;
>> }
>>
>> int main()
>> {
>> #define N 100
>> size_t NN = 4*100*100;
>> size_t kmax = 100;
>> int i;
>>
>> double **bm = (double **)my_alloc( NN * sizeof( double* ) );
>> for( i = 0; i < NN; ++i )
>> {
>> bm[ i ] = (double*)my_alloc( kmax * sizeof( double ) );
>> }
>> // leak...
>> return 0;
>> }
>>
>> and the result is...
>>
>> (...)
>> mmap(NULL, 4896, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0)
>> =
>> 0x7f5fd97df000
>> mprotect(0x7f5fd97df000, 4096, PROT_NONE) = -1 ENOMEM (Cannot allocate
>> memory)
>
> Have you checked available memory on your system ? Or user limit ?
>
> You test program is going to allocate
> 79 + 1 pages for bm
> 1 + 1 for each double arrays (x 40000)
>
> So in the end your program is allocating 80080 pages, so about
> 312MBytes.
>
> It not that big for a 64bits system.
>
> Check limits such as
> -d the maximum size of a process's data segment
> -l the maximum size a process may lock into memory
> -m the maximum resident set size
$ ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes unlimited
-n: file descriptors 1024
-l: locked-in-memory size (kb) 64
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 64024
-q: bytes in POSIX msg queues 819200
-e: max nice 0
-r: max rt priority 0
$ free -m
total used free shared buffers cached
Mem: 8004 1779 6224 0 36 1228
-/+ buffers/cache: 514 7489
Swap: 15625 0 15625
imho the testcase has enough hardware resources.
Dnia 21-04-2010 o 11:22:19 Yann Droneaud <[email protected]> napisał(a):
>> (...)
>> mmap(NULL, 4896, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0)
>> =
>> 0x7f5fd97df000
>> mprotect(0x7f5fd97df000, 4096, PROT_NONE) = -1 ENOMEM (Cannot allocate
>> memory)
>
> Have you checked available memory on your system ? Or user limit ?
>
> You test program is going to allocate
> 79 + 1 pages for bm
> 1 + 1 for each double arrays (x 40000)
>
> So in the end your program is allocating 80080 pages, so about
> 312MBytes.
>
> It not that big for a 64bits system.
afaics in gdb, the mprotect fails at i=32756. it's near to 2^15.
maybe some kernel data structures are full?
On Wed, 2010-04-21 at 01:05 +0200, Paweł Sikora wrote:
> hi,
>
> i'm trying to debug an ugly application with ElectricFence.
> in fact, on x86-64 box with 8GB ram and 16GB swap i'm getting following error:
>
> "ElectricFence Exiting: mprotect() failed: Cannot allocate memory"
>
> the program has been compiled with gcc-4.5, glibc-2.11.1, kernel-2.6.32.
> did you ever come across such (kernel/glibc) limitations?
>
> here's a simple testcase which triggs -ENOMEM in mprotect().
You probably depleted the max map count, see:
/proc/sys/vm/max_map_count
We have a limit on the number of maps you can have, those mprotect()
calls split you maps like crazy, see also /proc/$pid/maps.
eg. change your second test program to include something like:
char buf[128];
snprintf(buf, sizeof(buf), "cat /proc/%d/maps", (int)getpid());
system(buf);
at the end after lowering your NN count to fit, and observe the result
of those mprotect() calls.
Dnia 21-04-2010 o 13:16:50 Peter Zijlstra <[email protected]>
napisał(a):
> On Wed, 2010-04-21 at 01:05 +0200, Paweł Sikora wrote:
>> hi,
>>
>> i'm trying to debug an ugly application with ElectricFence.
>> in fact, on x86-64 box with 8GB ram and 16GB swap i'm getting following
>> error:
>>
>> "ElectricFence Exiting: mprotect() failed: Cannot allocate memory"
>>
>> the program has been compiled with gcc-4.5, glibc-2.11.1, kernel-2.6.32.
>> did you ever come across such (kernel/glibc) limitations?
>>
>> here's a simple testcase which triggs -ENOMEM in mprotect().
>
> You probably depleted the max map count, see:
> /proc/sys/vm/max_map_count
yes, that is the clue :)
the limit in /proc/sys/vm/max_map_count was set to 65530.
with `echo 128000 > /proc/sys/vm/max_map_count` the testcase passes.
thanks for hint.