Hi,
I noticed that there are (quite large) entries in /proc/pid/maps that
have PROT_NONE, right after an existing mapping:
7fffe4000000-7fffe406a000 rw-p 7fffe4000000 00:00 0
7fffe406a000-7fffe8000000 ---p 7fffe406a000 00:00 0
7ffff76d1000-7ffff76e0000 r-xp 00000000 09:03 260750
/lib/libbz2.so.1.0.4
7ffff76e0000-7ffff78df000 ---p 0000f000 09:03 260750
/lib/libbz2.so.1.0.4
I don't mind that 2Mb map, but what is 7fffe406a000-7fffe8000000 ---p ?
(63M)
Is it coming from glibc mapping memory as PROT_NONE, and using
mprotect/madvise to make it writable, and then
caching the mappings for future use, rather than freeing them?
/proc/pid/smaps shows:
7fffe406a000-7fffe8000000 ---p 7fffe406a000 00:00 0
Size: 65112 kB
Rss: 0 kB
Pss: 0 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 0 kB
Referenced: 0 kB
Swap: 0 kB
I straced the program creating these, and I couldn't find anything with
7fffe406a000, but only before that address:
[pid 31928] mprotect(0x7fffe4000000, 135168, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4021000, 131072, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4041000, 32768, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4049000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe404a000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe404b000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe404c000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe404d000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe404e000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe404f000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4050000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4051000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4052000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4053000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4054000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4055000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4056000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4057000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4058000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe4059000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe405a000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe405b000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe405c000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 31928] mprotect(0x7fffe405d000, 53248, PROT_READ|PROT_WRITE) = 0
[pid 31928] madvise(0x7fffe4069000, 4096, 0x4 /* MADV_??? */) = 0
[pid 31928] madvise(0x7fffe4021000, 294912, 0x4 /* MADV_??? */) = 0
[pid 31928] madvise(0x7fffe4021000, 4096, 0x4 /* MADV_??? */) = 0
There is an mprotect and madvise that end at 0x7fffe406a000.
Those mprotects and madvise are coming from glibc. Its strange that I
don't see the mmap only the mprotect, but I used strace -f.
This happens on:
Linux debian 2.6.26-1-amd64 #1 SMP Thu Oct 9 14:16:53 UTC 2008 x86_64
GNU/Linux
strace is here:
http://edwintorok.googlepages.com/log2.bz2
Best regards,
--Edwin
On Wed, 5 Nov 2008, Török Edwin wrote:
>
> I noticed that there are (quite large) entries in /proc/pid/maps that
> have PROT_NONE, right after an existing mapping:
> 7fffe4000000-7fffe406a000 rw-p 7fffe4000000 00:00 0
> 7fffe406a000-7fffe8000000 ---p 7fffe406a000 00:00 0
> 7ffff76d1000-7ffff76e0000 r-xp 00000000 09:03 260750
> /lib/libbz2.so.1.0.4
> 7ffff76e0000-7ffff78df000 ---p 0000f000 09:03 260750
> /lib/libbz2.so.1.0.4
>
> I don't mind that 2Mb map, but what is 7fffe406a000-7fffe8000000 ---p ?
> (63M)
>
> Is it coming from glibc mapping memory as PROT_NONE, and using
> mprotect/madvise to make it writable, and then
> caching the mappings for future use, rather than freeing them?
mmap PROT_NONE to reserve an arena, munmap to trim off top and bottom,
mprotect to make areas read+writable, madvise 0x4 to say MADV_DONTNEED
on some parts. gcc? Or the application itself (clamd) and its libs?
> I straced the program creating these, and I couldn't find anything with
> 7fffe406a000, but only before that address:
>
> [pid 31928] mprotect(0x7fffe4000000, 135168, PROT_READ|PROT_WRITE) = 0
[snipped]
> [pid 31928] madvise(0x7fffe4021000, 4096, 0x4 /* MADV_??? */) = 0
>
> There is an mprotect and madvise that end at 0x7fffe406a000.
> Those mprotects and madvise are coming from glibc. Its strange that I
> don't see the mmap only the mprotect, but I used strace -f.
>
> This happens on:
> Linux debian 2.6.26-1-amd64 #1 SMP Thu Oct 9 14:16:53 UTC 2008 x86_64
> GNU/Linux
>
> strace is here:
> http://edwintorok.googlepages.com/log2.bz2
Just before your first mprotect above there's:
[pid 31928] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0 <unfinished ...>
[pid 31938] futex(0x7ffff6e489e0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 31928] <... mmap resumed> ) = 0x7fffe2bd3000
which maps 7fffe2bd3000-7fffeabd3000; then
[pid 31928] munmap(0x7fffe2bd3000, 21155840 <unfinished ...>
[pid 31938] <... futex resumed> ) = 0
[pid 31928] <... munmap resumed> ) = 0
which unmaps 7fffe2bd3000-7fffe4000000; and then
[pid 31928] munmap(0x7fffe8000000, 45953024) = 0
which unmaps 7fffe8000000-7fffeabd3000.
So it's trimming off the rough edges to leave 7fffe4000000-7fffe8000000
mapped PROT_NONE, then mprotecting what it needs of that.
Why does it mmap too much then trim it down? Perhaps it's trying to
minimize pagetable usage, perhaps it's internally convenient to base
on rounded addresses, I don't know.
But the mmap is there: just easily overlooked because of the way it
munmaps too (with strace showing hex addresses but decimal sizes).
Hugh
On 2008-11-05 18:12, Hugh Dickins wrote:
> On Wed, 5 Nov 2008, Török Edwin wrote:
>
>> I noticed that there are (quite large) entries in /proc/pid/maps that
>> have PROT_NONE, right after an existing mapping:
>> 7fffe4000000-7fffe406a000 rw-p 7fffe4000000 00:00 0
>> 7fffe406a000-7fffe8000000 ---p 7fffe406a000 00:00 0
>> 7ffff76d1000-7ffff76e0000 r-xp 00000000 09:03 260750
>> /lib/libbz2.so.1.0.4
>> 7ffff76e0000-7ffff78df000 ---p 0000f000 09:03 260750
>> /lib/libbz2.so.1.0.4
>>
>> I don't mind that 2Mb map, but what is 7fffe406a000-7fffe8000000 ---p ?
>> (63M)
>>
>> Is it coming from glibc mapping memory as PROT_NONE, and using
>> mprotect/madvise to make it writable, and then
>> caching the mappings for future use, rather than freeing them?
>>
>
> mmap PROT_NONE to reserve an arena, munmap to trim off top and bottom,
> mprotect to make areas read+writable, madvise 0x4 to say MADV_DONTNEED
> on some parts. gcc? Or the application itself (clamd) and its libs?
>
>
>> I straced the program creating these, and I couldn't find anything with
>> 7fffe406a000, but only before that address:
>>
>> [pid 31928] mprotect(0x7fffe4000000, 135168, PROT_READ|PROT_WRITE) = 0
>>
> [snipped]
>
>> [pid 31928] madvise(0x7fffe4021000, 4096, 0x4 /* MADV_??? */) = 0
>>
>> There is an mprotect and madvise that end at 0x7fffe406a000.
>> Those mprotects and madvise are coming from glibc. Its strange that I
>> don't see the mmap only the mprotect, but I used strace -f.
>>
>> This happens on:
>> Linux debian 2.6.26-1-amd64 #1 SMP Thu Oct 9 14:16:53 UTC 2008 x86_64
>> GNU/Linux
>>
>> strace is here:
>> http://edwintorok.googlepages.com/log2.bz2
>>
>
> Just before your first mprotect above there's:
>
> [pid 31928] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0 <unfinished ...>
> [pid 31938] futex(0x7ffff6e489e0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 31928] <... mmap resumed> ) = 0x7fffe2bd3000
>
> which maps 7fffe2bd3000-7fffeabd3000; then
>
> [pid 31928] munmap(0x7fffe2bd3000, 21155840 <unfinished ...>
> [pid 31938] <... futex resumed> ) = 0
> [pid 31928] <... munmap resumed> ) = 0
>
> which unmaps 7fffe2bd3000-7fffe4000000; and then
>
> [pid 31928] munmap(0x7fffe8000000, 45953024) = 0
>
> which unmaps 7fffe8000000-7fffeabd3000.
>
> So it's trimming off the rough edges to leave 7fffe4000000-7fffe8000000
> mapped PROT_NONE, then mprotecting what it needs of that.
>
>
Thanks I completely missed the mmap, because I grepped for 7fffe4000000.
> Why does it mmap too much then trim it down? Perhaps it's trying to
> minimize pagetable usage, perhaps it's internally convenient to base
> on rounded addresses, I don't know.
>
> But the mmap is there: just easily overlooked because of the way it
> munmaps too (with strace showing hex addresses but decimal sizes).
I will get some stacktraces and figure out, know that I know which mmap
to look for (the one with MAP_NORESERVE).
Best regards,
--Edwin
On 2008-11-05 19:56, T?r?k Edwin wrote:
> On 2008-11-05 18:12, Hugh Dickins wrote:
>
>>>
>>>
>> mmap PROT_NONE to reserve an arena, munmap to trim off top and bottom,
>> mprotect to make areas read+writable, madvise 0x4 to say MADV_DONTNEED
>> on some parts. gcc? Or the application itself (clamd) and its libs?
>>
>>
>>
>
>> Why does it mmap too much then trim it down? Perhaps it's trying to
>> minimize pagetable usage, perhaps it's internally convenient to base
>> on rounded addresses, I don't know.
>>
>> But the mmap is there: just easily overlooked because of the way it
>> munmaps too (with strace showing hex addresses but decimal sizes).
>>
>
> I will get some stacktraces and figure out, know that I know which mmap
> to look for (the one with MAP_NORESERVE).
>
I found it, glibc: arena.c:669 does it, there's a comment explaining why:
/* If consecutive mmap (0, HEAP_MAX_SIZE << 1, ...) calls return decreasing
addresses as opposed to increasing, new_heap would badly fragment the
address space. In that case remember the second HEAP_MAX_SIZE part
aligned to HEAP_MAX_SIZE from last mmap (0, HEAP_MAX_SIZE << 1, ...)
call (if it is already aligned) and try to reuse it next time. We need
no locking for it, as kernel ensures the atomicity for us - worst case
we'll call mmap (addr, HEAP_MAX_SIZE, ...) for some value of addr in
multiple threads, but only one will succeed. */
Anyway it is MAP_NORESERVE, and PROT_NONE so it doesn't waste physical
or swap memory.
Best regards,
--Edwin
> [pid 31928] mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0 <unfinished ...>
> [pid 31938] futex(0x7ffff6e489e0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 31928] <... mmap resumed> ) = 0x7fffe2bd3000
>
> which maps 7fffe2bd3000-7fffeabd3000; then
>
> [pid 31928] munmap(0x7fffe2bd3000, 21155840 <unfinished ...>
> [pid 31938] <... futex resumed> ) = 0
> [pid 31928] <... munmap resumed> ) = 0
>
> which unmaps 7fffe2bd3000-7fffe4000000; and then
>
> [pid 31928] munmap(0x7fffe8000000, 45953024) = 0
>
> which unmaps 7fffe8000000-7fffeabd3000.
>
> So it's trimming off the rough edges to leave 7fffe4000000-7fffe8000000
> mapped PROT_NONE, then mprotecting what it needs of that.
>
> Why does it mmap too much then trim it down? Perhaps it's trying to
> minimize pagetable usage, perhaps it's internally convenient to base
> on rounded addresses, I don't know.
Yup.
glibc malloc's arena need heap size alignment.
it cause that struct arena address can calcurate "arena = any_ptr & ~(HEAP_SIZE-1)"
as old linux's "current" macro calculation.
> But the mmap is there: just easily overlooked because of the way it
> munmaps too (with strace showing hex addresses but decimal sizes).