Hi,
I have a few questions with regards to alloc'ing and locking down memory.
An example
would be very useful. Please CC me on any responses.
1. Is there a mechanism to lock down large amounts of memory (>128M, upto
256M).
Can 256M be allocated using vmalloc, if so is it swappable?
2. Is it possible for a user process and kernel to access the same shared
memory?
3. Can a shared memory have visibility across processes, i.e. can process A
access
memory that was allocated by process B?
4. When a process exits will it cause a close to occur on the device?
Thanks,
Bhavana
On Fri, 2002-08-16 at 17:38, Bhavana Nagendra wrote:
> Hi,
>
> I have a few questions with regards to alloc'ing and locking down memory.
> An example
> would be very useful. Please CC me on any responses.
>
> 1. Is there a mechanism to lock down large amounts of memory (>128M, upto
> 256M).
>From user space? kernel space? The answer is yes to both but the
mechnism is different.
> Can 256M be allocated using vmalloc, if so is it swappable?
It can be alloacted via vmalloc and AFAIK it is not swappable by
default. This doesn't sound like a very good idea though.
> 2. Is it possible for a user process and kernel to access the same shared
> memory?
Yes. See /proc/kcore for a very obvious example. Also "Linux device
drivers second edition" has many good exmaple on the subject in the
chapter devoted to mmap.
> 3. Can a shared memory have visibility across processes, i.e. can process A
> access
> memory that was allocated by process B?
Of course. This is the definition of shared memeory...
Just one thing to keep in mind - 'allocating' memory really doesn't do
that much as you might think. Until the memory is *accessed* for the
first time, all you got for the most part are some entries in a table
somwehere...
> 4. When a process exits will it cause a close to occur on the device?
Depends how you got the shared memeory. With mmap() it's yes (for
regular files at least), with shmget/shmat it's no by default. For mmap
of non regulat files (e.g. device files) anything the device file writer
had in mind is the answer.
man shmget, shmat, shmat and finally mmap will help you a lot.
Gilad.
--
Gilad Ben-Yossef <[email protected]>
Code mangler, senior coffee drinker and VP SIGSEGV
Qlusters ltd.
"You got an EMP device in the server room? That is so cool."
-- from a hackers-il thread on paranoia
On Sun, 2002-08-18 at 13:09, Gilad Ben-Yossef wrote:
> > Can 256M be allocated using vmalloc, if so is it swappable?
>
> It can be alloacted via vmalloc and AFAIK it is not swappable by
> default. This doesn't sound like a very good idea though.
There isnt enough address space for vmalloc to grab 256Mb. If you want
that much then you need to handle the fact its in page arrays not
virtually linear yourself.
> Yes. See /proc/kcore for a very obvious example. Also "Linux device
> drivers second edition" has many good exmaple on the subject in the
> chapter devoted to mmap.
A better worked example for large volumes of memory might be
drivers/media/video/bttv*.c.
On Sun, 2002-08-18 at 15:09, Gilad Ben-Yossef wrote:
> > 4. When a process exits will it cause a close to occur on the device?
>
> Depends how you got the shared memeory. With mmap() it's yes (for
> regular files at least), with shmget/shmat it's no by default. For mmap
> of non regulat files (e.g. device files) anything the device file writer
> had in mind is the answer.
>
> man shmget, shmat, shmat and finally mmap will help you a lot.
When I first read the question I was still thinking in terms of shared
memeory and thought you meant: 'will it destroy the shared memory
segment?'.
If the question was not related to shared memeory and you meant 'will it
do the same as close() on the file' the answer is yes as well unless the
device driver specifically chose to fail that for some reason (see the
watchdog devices for an example and a reason).
Gilad.
--
Gilad Ben-Yossef <[email protected]>
Code mangler, senior coffee drinker and VP SIGSEGV
Qlusters ltd.
"You got an EMP device in the server room? That is so cool."
-- from a hackers-il thread on paranoia
On Sun, 2002-08-18 at 15:18, Alan Cox wrote:
> On Sun, 2002-08-18 at 13:09, Gilad Ben-Yossef wrote:
> > > Can 256M be allocated using vmalloc, if so is it swappable?
> >
> > It can be alloacted via vmalloc and AFAIK it is not swappable by
> > default. This doesn't sound like a very good idea though.
>
> There isnt enough address space for vmalloc to grab 256Mb. If you want
> that much then you need to handle the fact its in page arrays not
> virtually linear yourself.
Oopss... indeed. 256M is twice the entire vmalloc address space to be
exact.
Thanks for correcting my mistake ;-)
Gilad.
--
Gilad Ben-Yossef <[email protected]>
http://benyossef.com
"Money talks, bullshit walks and GNU awks."
-- Shachar "Sun" Shemesh, debt collector for the GNU/Yakuza
Gilad Ben-Yossef wrote:
>
> > 1. Is there a mechanism to lock down large amounts of memory (>128M, upto
> > 256M).
>
> >From user space? kernel space? The answer is yes to both but the
> mechnism is different.
In the kernel space. I've looked at the boot-time allocation examples and
Rubini's
allocator.c. I don't have the option of making the user compile the kernel to
obtain a chunk of the high memory by passing in the 'mem' option at boot time.
Here's the scenario: I need to be able to alloc virtual memory, create the
page table
and temporarily pin down (lock) the pages in memory in order to do the DMA.
This
memory should be paged in and out as needed. This memory should also have
visibility across processes.
Does the VM_RESERVED flag lock down the memory so that it doesn't get paged out
during DMA?
>
>
> > Can 256M be allocated using vmalloc, if so is it swappable?
>
> It can be alloacted via vmalloc and AFAIK it is not swappable by
> default. This doesn't sound like a very good idea though.
Is there a good way to allocate large sums of memory in Linux? Doesn't have to
be
vmalloc but I don't think kmalloc, get_free_pages will work for this purpose. I
looked
into get_free_pages, but the largest order is 9 which results in 512 pages.
Does the memory allocated by vmalloc has visibility across processes?
>
>
> > 2. Is it possible for a user process and kernel to access the same shared
> > memory?
>
> Yes. See /proc/kcore for a very obvious example. Also "Linux device
> drivers second edition" has many good exmaple on the subject in the
> chapter devoted to mmap.
Great. I see remap_page_range and nopage examples.
>
>
> > 3. Can a shared memory have visibility across processes, i.e. can process A
> > access
> > memory that was allocated by process B?
>
> Of course. This is the definition of shared memeory...
> Just one thing to keep in mind - 'allocating' memory really doesn't do
> that much as you might think. Until the memory is *accessed* for the
> first time, all you got for the most part are some entries in a table
> somwehere...
>
> > 4. When a process exits will it cause a close to occur on the device?
>
> Depends how you got the shared memeory. With mmap() it's yes (for
> regular files at least), with shmget/shmat it's no by default. For mmap
> of non regulat files (e.g. device files) anything the device file writer
> had in mind is the answer.
I didn't mean shared memory. If several processes open a given device,
under normal conditions the data structure stays till the last close at which
time a
release is done. This depends on the usage or minor number count. Can there
be a case where the device exits before the processes close? In which case
the processes will be left hanging. How is the close handled if the driver is
killed?
>
>
> man shmget, shmat, shmat and finally mmap will help you a lot.
Yep, I'll look at them. Thanks a lot for your reply.
Bhavana
--
Bhavana Nagendra
(256) 319-1267
[email protected]
On Mon, 2002-08-19 at 15:06, Bhavana Nagendra wrote:
> Here's the scenario: I need to be able to alloc virtual memory, create the
> page table
> and temporarily pin down (lock) the pages in memory in order to do the DMA.
> This
> memory should be paged in and out as needed. This memory should also have
> visibility across processes.
OK. *AFAIK* most drivers that use DMA request the memory allocation with
GFP_DMA so that the allocators (whatever they chose) will give them
DMAbale memory from the DMA memory zone. Memory from that zone is fixed
(not pagable). You can then map that memory into proccesses address
space according to demand (and the Linux device drivers second addition
has a good example of doing that).
Now, if I understood you correctly you want to allocate 256M of memroy
and perform DMA into it? if so, you *cannot* use memory allocated using
vmalloc because the memory it supplies is virtual, that is it not
contigous in physical memory but might be scattered all over the place
and I doubt that whatever device you're DMAing from can handle that.
Also, I think you don't want this memory to be swapable, because if it
is swapped out and then in and might very well end up on a completly
different address in physical memory from where it were and again, I
don't think the device that does DMA will be able to handle that - all
the ones I know require physical addresses (well actualy bus addresses
but for the sake of argument let's ignore that for a second).
In short, I don't think you need what you think want... :-)
>
> Does the VM_RESERVED flag lock down the memory so that it doesn't get paged out
> during DMA?
AFAIK the VM_RESERVED flag will cause kswapd to ignore the page
completly - no paging in or out at all.
>
> >
> >
> > > Can 256M be allocated using vmalloc, if so is it swappable?
> >
> > It can be alloacted via vmalloc and AFAIK it is not swappable by
> > default. This doesn't sound like a very good idea though.
>
> Is there a good way to allocate large sums of memory in Linux? Doesn't have to
> be
> vmalloc but I don't think kmalloc, get_free_pages will work for this purpose. I
> looked
> into get_free_pages, but the largest order is 9 which results in 512 pages.
> Does the memory allocated by vmalloc has visibility across processes?
See my previous answer regarding why you don't vmalloced memory at
visibility.
> I didn't mean shared memory. If several processes open a given device,
> under normal conditions the data structure stays till the last close at which
> time a
> release is done. This depends on the usage or minor number count. Can there
> be a case where the device exits before the processes close? In which case
> the processes will be left hanging. How is the close handled if the driver is
> killed?
Very simple - you can't unload the device until the ref count says all
the users (proccesses in thuis case) have closed it.
Gilad.
--
Gilad Ben-Yossef <[email protected]>
http://benyossef.com
"Money talks, bullshit walks and GNU awks."
-- Shachar "Sun" Shemesh, debt collector for the GNU/Yakuza
> OK. *AFAIK* most drivers that use DMA request the memory
> allocation with
> GFP_DMA so that the allocators (whatever they chose) will give them
> DMAbale memory from the DMA memory zone. Memory from that
> zone is fixed
> (not pagable). You can then map that memory into proccesses address
> space according to demand (and the Linux device drivers
> second addition
> has a good example of doing that).
>
OK, understood.
> Now, if I understood you correctly you want to allocate 256M of memroy
> and perform DMA into it? if so, you *cannot* use memory
> allocated using
> vmalloc because the memory it supplies is virtual, that is it not
> contigous in physical memory but might be scattered all over the place
> and I doubt that whatever device you're DMAing from can handle that.
I was thinking a scatter gather page table would convert the virtual address
into physical addresses that DMA can use.
> Also, I think you don't want this memory to be swapable, because if it
> is swapped out and then in and might very well end up on a completly
> different address in physical memory from where it were and again, I
> don't think the device that does DMA will be able to handle that - all
> the ones I know require physical addresses (well actualy bus addresses
> but for the sake of argument let's ignore that for a second).
>
> In short, I don't think you need what you think want... :-)
Gee, how did you guess? :-) I was mistaken about who does the big 128M
alloc.
The 128M will actually be malloced in user space and filled up with data in
user space. This memory will then have to be mapped into kernel space and
DMA performed. Before DMA is performed the memory obviously needs to be
locked
down. How does this play out with respect to DMA memory zone and GFP_DMA
flag,
specifically pinning down memory?
Thanks Gilad!
> >
> > Does the VM_RESERVED flag lock down the memory so that it
> doesn't get paged out
> > during DMA?
>
> AFAIK the VM_RESERVED flag will cause kswapd to ignore the page
> completly - no paging in or out at all.
>
> >
> > >
> > >
> > > > Can 256M be allocated using vmalloc, if so is it swappable?
> > >
> > > It can be alloacted via vmalloc and AFAIK it is not swappable by
> > > default. This doesn't sound like a very good idea though.
> >
> > Is there a good way to allocate large sums of memory in
> Linux? Doesn't have to
> > be
> > vmalloc but I don't think kmalloc, get_free_pages will work
> for this purpose. I
> > looked
> > into get_free_pages, but the largest order is 9 which
> results in 512 pages.
>
> > Does the memory allocated by vmalloc has visibility across
> processes?
>
> See my previous answer regarding why you don't vmalloced memory at
> visibility.
>
> > I didn't mean shared memory. If several processes open a
> given device,
> > under normal conditions the data structure stays till the
> last close at which
> > time a
> > release is done. This depends on the usage or minor
> number count. Can there
> > be a case where the device exits before the processes
> close? In which case
> > the processes will be left hanging. How is the close
> handled if the driver is
> > killed?
>
> Very simple - you can't unload the device until the ref count says all
> the users (proccesses in thuis case) have closed it.
>
> Gilad.
>
> --
> Gilad Ben-Yossef <[email protected]>
> http://benyossef.com
>
> "Money talks, bullshit walks and GNU awks."
> -- Shachar "Sun" Shemesh, debt collector for the GNU/Yakuza
>
>
> Curiosity: why do you want to do device DMA buffer
> allocation from userland?
I need 256M memory for a graphics operation. It's a requiremment,
can't change it. There will be other reasonably sized allocs in kernel
space, this is a special case that will be done from userland. As
discussed earlier in this thread, there's no good way of alloc()ing
and pinning that much in DMA memory space, is there?
Gilad, I looked at mm/memory.c and map_user_kiobuf() lets me
map user memory into kernel memory and pins it down. A scatter
gatter mapping (say, pci_map_sg()) will create a seemingly
contiguous buffer for DMA purposes. Does that sound right to you?
Bhavana
On Tue, 20 Aug 2002, Bhavana Nagendra wrote:
> >
> > Curiosity: why do you want to do device DMA buffer
> > allocation from userland?
>
> I need 256M memory for a graphics operation. It's a requiremment,
> can't change it. There will be other reasonably sized allocs in kernel
> space, this is a special case that will be done from userland. As
> discussed earlier in this thread, there's no good way of alloc()ing
> and pinning that much in DMA memory space, is there?
>
> Gilad, I looked at mm/memory.c and map_user_kiobuf() lets me
> map user memory into kernel memory and pins it down. A scatter
> gatter mapping (say, pci_map_sg()) will create a seemingly
> contiguous buffer for DMA purposes. Does that sound right to you?
>
> Bhavana
You have to cheat. You can tell the kernel that you only have, say
128 Meg of RAM. Then, your driver adds PTEs for all the other
RAM you really have, easy ioremap(). The kernel doesn't know
nor care that there is RAM at those addresses. You can even call
request_mem_region() so that /proc/iomem has your entries in it.
This gets you all the real RAM you need plus it's guaranteed to
be contiguous and never paged.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
The US military has given us many words, FUBAR, SNAFU, now ENRON.
Yes, top management were graduates of West Point and Annapolis.
At 03:08 PM 8/20/2002 -0500, Bhavana Nagendra wrote:
> >
> > Curiosity: why do you want to do device DMA buffer
> > allocation from userland?
>
>I need 256M memory for a graphics operation. It's a requiremment,
>can't change it. There will be other reasonably sized allocs in kernel
>space, this is a special case that will be done from userland. As
>discussed earlier in this thread, there's no good way of alloc()ing
>and pinning that much in DMA memory space, is there?
Not that I know of. It seems to me that any interface that tried
to provide this would have to know what kind of device is going
to DMA from/to that ram.
Usually, when someone needs a large gob of contiguous ram,
folks suggest doing the allocation in kernel, and early.
-Mike
At 04:47 PM 8/20/2002 -0400, Richard B. Johnson wrote:
>On Tue, 20 Aug 2002, Bhavana Nagendra wrote:
>
> > >
> > > Curiosity: why do you want to do device DMA buffer
> > > allocation from userland?
> >
> > I need 256M memory for a graphics operation. It's a requiremment,
> > can't change it. There will be other reasonably sized allocs in kernel
> > space, this is a special case that will be done from userland. As
> > discussed earlier in this thread, there's no good way of alloc()ing
> > and pinning that much in DMA memory space, is there?
> >
> > Gilad, I looked at mm/memory.c and map_user_kiobuf() lets me
> > map user memory into kernel memory and pins it down. A scatter
> > gatter mapping (say, pci_map_sg()) will create a seemingly
> > contiguous buffer for DMA purposes. Does that sound right to you?
> >
> > Bhavana
>
>You have to cheat. You can tell the kernel that you only have, say
>128 Meg of RAM.
Why not just use early allocation? (if he has eg a 16G box, chopping
it down enough to get at 256M of DMA ram just ain't gonna work:)
-Mike
On Tue, 2002-08-20 at 23:08, Bhavana Nagendra wrote:
> >
> > Curiosity: why do you want to do device DMA buffer
> > allocation from userland?
>
> Gilad, I looked at mm/memory.c and map_user_kiobuf() lets me
> map user memory into kernel memory and pins it down. A scatter
> gatter mapping (say, pci_map_sg()) will create a seemingly
> contiguous buffer for DMA purposes. Does that sound right to you?
Can your device actually use scatter gather (has 'virtual bus addresses'
or can get the buffer in several chunks?) Can the device actually DMA
all the memory in the system?
In short, I have no idea. :-)
Gilad.
--
Gilad Ben-Yossef <[email protected]>
http://benyossef.com
"Money talks, bullshit walks and GNU awks."
-- Shachar "Sun" Shemesh, debt collector for the GNU/Yakuza
On Tue, 2002-08-20 at 17:51, Bhavana Nagendra wrote:
I think a disclaimer is due before we begin - I'm not Linux drivers
guru not do i play one of TV. Take all of this with the apropriate grain
of salt. Actually, make that a spoon of salt :-)
> > Now, if I understood you correctly you want to allocate 256M of memroy
> > and perform DMA into it? if so, you *cannot* use memory
> > allocated using
> > vmalloc because the memory it supplies is virtual, that is it not
> > contigous in physical memory but might be scattered all over the place
> > and I doubt that whatever device you're DMAing from can handle that.
>
> I was thinking a scatter gather page table would convert the virtual address
> into physical addresses that DMA can use.
Scatter gather solves the need for contigous physical memory space
problem.
BUT AFAIK scatter gather works *if you're device supports it*. Either
support it (or the machine) has 'virtual bus addresses' or it can take
the buffers it needs to proccess in several seperate chunks. Does it?
According to my limited understanding you can set the page reserved one
page at a time, DMA it and then unset the reserved bit again, *but* - a.
it might mean very long latencies/jitter to transfer the buffer and your
device must DMA from various high regions of memory. AFAIK most
devices/boards have very specific limitations on what can be transfered
via DMA.
Reserving the entire 256M in one go sounds... dangerous to me :-) I'm no
VM guru but I doubt that it's a good idea unless you ave gobs and gobs
of free physical pages available.
>
> > Also, I think you don't want this memory to be swapable, because if it
> > is swapped out and then in and might very well end up on a completly
> > different address in physical memory from where it were and again, I
> > don't think the device that does DMA will be able to handle that - all
> > the ones I know require physical addresses (well actualy bus addresses
> > but for the sake of argument let's ignore that for a second).
> >
> > In short, I don't think you need what you think want... :-)
>
> Gee, how did you guess? :-) I was mistaken about who does the big 128M
> alloc.
> The 128M will actually be malloced in user space and filled up with data in
> user space. This memory will then have to be mapped into kernel space and
> DMA performed. Before DMA is performed the memory obviously needs to be
> locked
> down. How does this play out with respect to DMA memory zone and GFP_DMA
> flag,
> specifically pinning down memory?
Look, all memory allocations in user space are nothing more then doing
mmap on some file. malloc(), shat() or what have you - in the end they
all do the same - request the kernel to add some entries to the proccess
page table. The true allocation occurs in kernel space, whether when the
proccess request it or more usually in response to a page fault. But
regardless of which it's something in the kernel anyways that needs to
find and secure those physical pages for you.
You have some serious strict limit on *what* memory you can get. You
might need it be contigous if your device cannot do scatter gather, it
needs to come from a memory area that the device can actually DMA and I
think you might even need to map these pages unto kernel address space,
which in cae of HIGHMEM areas might not be so easy as kmap slots are
scarce.
As other people have noted, the best bet is early allocation. The *best*
way to do it is via kernel interfaces but as noted before sometimes the
simplest thing is to tell the kernel at boot time (or equivilient) that
you have less memory then the machine actually has and simply map that
later. It's crude, but it works and on some embedded device it might be
the simplest option when the client refuses to recieve ANY changes to
the kernel sources used.
Good luck,
Gilad.
--
Gilad Ben-Yossef <[email protected]>
http://benyossef.com
"Money talks, bullshit walks and GNU awks."
-- Shachar "Sun" Shemesh, debt collector for the GNU/Yakuza
On Wed, 21 Aug 2002, Mike Galbraith wrote:
> At 03:08 PM 8/20/2002 -0500, Bhavana Nagendra wrote:
> > >
> > > Curiosity: why do you want to do device DMA buffer
> > > allocation from userland?
> >
> >I need 256M memory for a graphics operation. It's a requiremment,
> >can't change it. There will be other reasonably sized allocs in kernel
> >space, this is a special case that will be done from userland. As
> >discussed earlier in this thread, there's no good way of alloc()ing
> >and pinning that much in DMA memory space, is there?
>
> Not that I know of. It seems to me that any interface that tried
> to provide this would have to know what kind of device is going
> to DMA from/to that ram.
>
> Usually, when someone needs a large gob of contiguous ram,
> folks suggest doing the allocation in kernel, and early.
>
BTW: What is the limit for pci_alloc_consistent and friends? Can it really
provide 256MB?
Ciao,
Roland
+---------------------------+-------------------------+
| TU Muenchen | |
| Physik-Department E18 | Raum 3558 |
| James-Franck-Str. | Telefon 089/289-12592 |
| 85747 Garching | |
+---------------------------+-------------------------+
Roland Kuhn wrote:
>On Wed, 21 Aug 2002, Mike Galbraith wrote:
>
>>At 03:08 PM 8/20/2002 -0500, Bhavana Nagendra wrote:
>>
>>>>Curiosity: why do you want to do device DMA buffer
>>>>allocation from userland?
>>>>
>>>I need 256M memory for a graphics operation. It's a requiremment,
>>>can't change it. There will be other reasonably sized allocs in kernel
>>>space, this is a special case that will be done from userland. As
>>>discussed earlier in this thread, there's no good way of alloc()ing
>>>and pinning that much in DMA memory space, is there?
>>>
>>Not that I know of. It seems to me that any interface that tried
>>to provide this would have to know what kind of device is going
>>to DMA from/to that ram.
>>
>>Usually, when someone needs a large gob of contiguous ram,
>>folks suggest doing the allocation in kernel, and early.
>>
>BTW: What is the limit for pci_alloc_consistent and friends? Can it really
>provide 256MB?
>
Dunno. The page allocator however (lowest) can deliver 1 << MAX_ORDER
contiguous
pages per request.. unless fragmentation gets you that is, so I doubt
it's remotely possible
to get 256MB of _physically_ contiguous ram without doing early
allocation of some sort.
(bootmem, bigphysarea patches or whatnot)
-Mike