vmalloc() fails after about 890MB because the kernel memory map is only for
about 1GB. I know there are some hacks and work-arounds to get more than
that, but instead of reinventing the wheel, I was hoping some kind soul would
tell me how (a few hints would be nice!)
The reason we use vmalloc() is because we need to apply memory pressure during
the allocating: memory should be swapped out to make room for our allocation.
We're trying to allocate up to 3GB on a 4GB machine. Thanks in advance!
Followup to: <[email protected]>
By author: Timur Tabi <[email protected]>
In newsgroup: linux.dev.kernel
>
> > Isn't this solved by just recompiling the kernel with HIGHMEM support?
>
>
> I don't think so. The Red Hat 7.1 kernel is compiled with "4GB" support,
> which apparently is the same as HIGHMEM. We see the 890MB kernel vmalloc
> limit still.
>
That's because you're running out of address space, not memory.
HIGHMEM doesn't do anything for the latter -- it can't. You start
running into a lot of fundamental problems when your memory size gets
in the same (or higher) ballpark than your address space.
The best solution is go buy a 64-bit CPU. There isn't much else you
can do about it.
-hpa
--
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Isn't this solved by just recompiling the kernel with HIGHMEM support?
John
On Fri, 19 Oct 2001, Timur Tabi wrote:
> vmalloc() fails after about 890MB because the kernel memory map is only for
> about 1GB. I know there are some hacks and work-arounds to get more than
> that, but instead of reinventing the wheel, I was hoping some kind soul would
> tell me how (a few hints would be nice!)
>
> The reason we use vmalloc() is because we need to apply memory pressure during
> the allocating: memory should be swapped out to make room for our allocation.
>
> We're trying to allocate up to 3GB on a 4GB machine. Thanks in advance!
>
> -
> Kernelnewbies: Help each other learn about the Linux kernel.
> Archive: http://mail.nl.linux.org/kernelnewbies/
> IRC Channel: irc.openprojects.net / #kernelnewbies
> Web Page: http://www.kernelnewbies.org/
>
--
John Tyner
[email protected]
John Tyner wrote:
> Isn't this solved by just recompiling the kernel with HIGHMEM support?
I don't think so. The Red Hat 7.1 kernel is compiled with "4GB" support,
which apparently is the same as HIGHMEM. We see the 890MB kernel vmalloc
limit still.
H. Peter Anvin wrote:
> That's because you're running out of address space, not memory.
> HIGHMEM doesn't do anything for the latter -- it can't. You start
> running into a lot of fundamental problems when your memory size gets
> in the same (or higher) ballpark than your address space.
>
> The best solution is go buy a 64-bit CPU. There isn't much else you
> can do about it.
That's completely missing the point of my request (which, I admit, I didn't
make clear). I need to allocate about 3/4 of available memory in the kernel.
If I had 2GB of RAM, I'd need to allocate 1.5GB. If I had 8 GB of RAM, I'd
need to allocate 6GB. I just used 3GB/4GB because it's our current test platform.
Timur Tabi wrote:
>
> That's completely missing the point of my request (which, I admit, I
> didn't make clear). I need to allocate about 3/4 of available memory in
> the kernel. If I had 2GB of RAM, I'd need to allocate 1.5GB. If I had
> 8 GB of RAM, I'd need to allocate 6GB. I just used 3GB/4GB because it's
> our current test platform.
>
There is no way you can allocate 6 GB of address space when you have 4 GB
to dole out -- and that includes to userspace. Linux tends to allocate
most of the address space (usually 3 GB) to userspace, because it can be
re-used between processes and it matches the needs of more users.
This puts fundamental limits on how much space is addressible in the
kernel. What you can do if your application permits is allocate HIGHMEM
pages, and use kmap()/kmap_atomic()/kunmap() to selectively bring them
into the address space on an as-need basis. THIS IS EXPENSIVE, have no
illusions about it, and doesn't give you space that is contiguous in
either linear nor physical space.
Obviously, on a 64-bit CPU these limitations utterly vanish, since address
space is no longer limited.
-hpa
On Friday 19 October 2001 16:21, Timur Tabi wrote:
> H. Peter Anvin wrote:
> > That's because you're running out of address space, not memory.
> > HIGHMEM doesn't do anything for the latter -- it can't. You start
> > running into a lot of fundamental problems when your memory size gets
> > in the same (or higher) ballpark than your address space.
> >
> > The best solution is go buy a 64-bit CPU. There isn't much else you
> > can do about it.
>
> That's completely missing the point of my request (which, I admit, I didn't
> make clear). I need to allocate about 3/4 of available memory in the
> kernel. If I had 2GB of RAM, I'd need to allocate 1.5GB. If I had 8 GB of
> RAM, I'd need to allocate 6GB. I just used 3GB/4GB because it's our
> current test platform.
Each user process has 32 bit pointers for memory. This means they only have
4 gigabytes of virtual address space, regardless of how many physical pages
the machine has. The kernel doesn't use segment:offset addressing. It just
uses the offset. Flat memory model.
Now your page tables can shuffle memory around (using physical pages, swap,
etc), but that's irrelevant. You've still only got 4 gigs of virtual space
per process to work with on a 32 bit system. The register you're ultimately
dereferencing to access memory isn't big enough to hold a number larger than
that.
To simplify the page tables, the kernel is in shared memory. The first
gigabyte of every process's address space is the kernel. The kernel's page
table is sort of spliced into everybody's page table. And there was much
rejoicing.
The problem is, you're trying to allocate kernel memory, meaning you hit the
1 gigabyte limit. And that includes the kernel itself. You can recompile so
the kernel has more than 1 gig of everybody's virtual address range reserved
to it, but that reduces the amount of virtual memory user space processes
have (4 gigs - 1 gig) because the kernel space (shared memory) is mapped into
each and every process so the process can access kernel resources easily.
(You do NOT want to change page tables for every system call. We're not
going there.)
To allocate more memory than that, you need to allocate pages in user space.
To allocate more than 4 gigabytes of memory, you MUST use more than one
process (you only have 4 gigs of virtual address space per process, so two
processes must use overlapping virtual address ranges to point to different
pages. Perhaps they could communicate with each other through a shared mmap.
Congratulations, you've just reinvented expanded memory.)
Or, we could get back to the "buy a real computer" answer everybody's been
giving you: buy a 64 bit computer that has more than 4 gigabytes of virtual
address space per process. Then you have 64 bit pointers for your virtual
addresses, and can directly address virtual petabytes or exabytes or whatever
comes next...
Rob
Rob Landley wrote:
>
> Each user process has 32 bit pointers for memory. This means they only have
> 4 gigabytes of virtual address space, regardless of how many physical pages
> the machine has. The kernel doesn't use segment:offset addressing. It just
> uses the offset. Flat memory model.
>
And even if it did (on i386) it wouldn't help... the segment:offset is
folded into a single 32-bit space before paging.
-hpa
> I need to allocate about 3/4 of available memory in the kernel.
> If I had 2GB of RAM, I'd need to allocate 1.5GB. If I had 8 GB
> of RAM, I'd need to allocate 6GB. I just used 3GB/4GB because
> it's our current test platform.
The best you can do, IMHO:
1. reserve a 3 GB chunk of memory at boot
2. create a regular user process
3. have that process make a system call which will never return
4. in that system call, wipe out all memory mappings in the process
5. hand-craft a 3 GB memory mapping (0 GB virt --> 1 GB phys)
6. call your desired code, remembering to schedule by hand
Albert D. Cahalan wrote:
> 1. reserve a 3 GB chunk of memory at boot
> 2. create a regular user process
> 3. have that process make a system call which will never return
> 4. in that system call, wipe out all memory mappings in the process
> 5. hand-craft a 3 GB memory mapping (0 GB virt --> 1 GB phys)
> 6. call your desired code, remembering to schedule by hand
Thanks, that's the most useful idea I've gotten. It's crazy, but it just
might work!
Too bad no one else on this list can think outside of the box like you just did.