LinuxLists.cc - Problem: Out of memory after 2days with 2GB RAM

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, 2008-06-12 at 09:38 -0400, Rik van Riel wrote:
> On Thu, 12 Jun 2008 12:07:34 +0200
> "Zdenek Kabelac" <[email protected]> wrote:
>
> > It looks like while there was a huge amount of buffers and caches -
> > system was unable to allocate few pages for kmalloc in iwl3945 driver
> > after resume.
>
> It looks like this is because it wants to allocate 2**5 contiguous
> pages, which is 128kB of contiguous kernel memory.

64-bit system I assume?
The allocation should be 256 * 20 * sizeof(struct sk_buff *).

Try the patch below. It should improve code generation too.

I discussed this with Tomas previously and he says the hw is capable of
doing 20 fragments per frame (wonder why, Broadcom can do an unlimited
number...) but he complained about the networking stack not being able
to. Well, the hardware needs to support IP checksumming for that, hence,
afaik, only two fragments can ever be used (one for hw header, one for
frame)

This cuts the allocation to 10%, or (under) a page in all cases.

johannes

--- everything.orig/drivers/net/wireless/iwlwifi/iwl-3945.h 2008-06-12 15:50:29.000000000 +0200
+++ everything/drivers/net/wireless/iwlwifi/iwl-3945.h 2008-06-12 15:50:31.000000000 +0200
@@ -120,7 +120,7 @@ struct iwl3945_queue {
int iwl3945_queue_space(const struct iwl3945_queue *q);
int iwl3945_x2_queue_used(const struct iwl3945_queue *q, int i);

-#define MAX_NUM_OF_TBS (20)
+#define MAX_NUM_OF_TBS (2)

/* One for each TFD */
struct iwl3945_tx_info {
--- everything.orig/drivers/net/wireless/iwlwifi/iwl-dev.h 2008-06-12 15:50:18.000000000 +0200
+++ everything/drivers/net/wireless/iwlwifi/iwl-dev.h 2008-06-12 15:50:24.000000000 +0200
@@ -115,7 +115,7 @@ struct iwl_queue {
* space less than this */
} __attribute__ ((packed));

-#define MAX_NUM_OF_TBS (20)
+#define MAX_NUM_OF_TBS (2)

/* One for each TFD */
struct iwl_tx_info {

2008-06-12 14:12:28

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

2008/6/12 Johannes Berg <[email protected]>:
> On Thu, 2008-06-12 at 09:38 -0400, Rik van Riel wrote:
>> On Thu, 12 Jun 2008 12:07:34 +0200
>> "Zdenek Kabelac" <[email protected]> wrote:
>>
>> > It looks like while there was a huge amount of buffers and caches -
>> > system was unable to allocate few pages for kmalloc in iwl3945 driver
>> > after resume.
>>
>> It looks like this is because it wants to allocate 2**5 contiguous
>> pages, which is 128kB of contiguous kernel memory.
>
> 64-bit system I assume?
> The allocation should be 256 * 20 * sizeof(struct sk_buff *).
>
> Try the patch below. It should improve code generation too.

I'll surely try you patch - but is the iwl the only driver which needs
128kB continuous memory chunk?

I think that if the 128kB memchunks are exhausted in 2 days while
there is over 1GB of free RAM in buffers & caches I think some
defragmentation is needed then ?

btw: Does it really means that within those buffers kernel could not
find any adjacent 32 pages which could be made free ?

Zdenek

2008-06-12 14:20:57

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

> >> It looks like this is because it wants to allocate 2**5 contiguous
> >> pages, which is 128kB of contiguous kernel memory.
> >
> > 64-bit system I assume?
> > The allocation should be 256 * 20 * sizeof(struct sk_buff *).
> >
> > Try the patch below. It should improve code generation too.
>
> I'll surely try you patch - but is the iwl the only driver which needs
> 128kB continuous memory chunk?

I don't know. But I think it'll probably fail right after, trying to
allocate a 32kb buffer with pci_alloc_something....

johannes

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part

2008-06-12 15:43:52

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 4:54 PM, Johannes Berg
<[email protected]> wrote:
> On Thu, 2008-06-12 at 09:38 -0400, Rik van Riel wrote:
>> On Thu, 12 Jun 2008 12:07:34 +0200
>> "Zdenek Kabelac" <[email protected]> wrote:
>>
>> > It looks like while there was a huge amount of buffers and caches -
>> > system was unable to allocate few pages for kmalloc in iwl3945 driver
>> > after resume.
>>
>> It looks like this is because it wants to allocate 2**5 contiguous
>> pages, which is 128kB of contiguous kernel memory.
>
> 64-bit system I assume?
> The allocation should be 256 * 20 * sizeof(struct sk_buff *).

> Try the patch below. It should improve code generation too.
>
> I discussed this with Tomas previously and he says the hw is capable of
> doing 20 fragments per frame (wonder why, Broadcom can do an unlimited
> number...) but he complained about the networking stack not being able
> to.

This is scatter gather buffers that can be kicked in one DMA transaction.

Well, the hardware needs to support IP checksumming for that, hence,
> afaik, only two fragments can ever be used (one for hw header, one for
> frame)
This I still don't understand why but everybody is already tired to
explaining me why.. :) Just need to find time to dig into it.

> This cuts the allocation to 10%, or (under) a page in all cases.

Probably. it would be safe to use vmalloc for allocating txb anyway.
I'll give it a try.

There was already discussion on LKML about memory allocation problems
on X86_64, which might explain this regression. This didn't happen
before.

Tomas

2008-06-12 16:35:52

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 6:43 PM, Tomas Winkler <[email protected]> wrote:
> On Thu, Jun 12, 2008 at 4:54 PM, Johannes Berg
> <[email protected]> wrote:
>> On Thu, 2008-06-12 at 09:38 -0400, Rik van Riel wrote:
>>> On Thu, 12 Jun 2008 12:07:34 +0200
>>> "Zdenek Kabelac" <[email protected]> wrote:
>>>
>>> > It looks like while there was a huge amount of buffers and caches -
>>> > system was unable to allocate few pages for kmalloc in iwl3945 driver
>>> > after resume.
>>>
>>> It looks like this is because it wants to allocate 2**5 contiguous
>>> pages, which is 128kB of contiguous kernel memory.
>>
>> 64-bit system I assume?
>> The allocation should be 256 * 20 * sizeof(struct sk_buff *).
>
>> Try the patch below. It should improve code generation too.
>>
>> I discussed this with Tomas previously and he says the hw is capable of
>> doing 20 fragments per frame (wonder why, Broadcom can do an unlimited
>> number...) but he complained about the networking stack not being able
>> to.
>
> This is scatter gather buffers that can be kicked in one DMA transaction.
>
> Well, the hardware needs to support IP checksumming for that, hence,
>> afaik, only two fragments can ever be used (one for hw header, one for
>> frame)
> This I still don't understand why but everybody is already tired to
> explaining me why.. :) Just need to find time to dig into it.
>
>> This cuts the allocation to 10%, or (under) a page in all cases.
>
> Probably. it would be safe to use vmalloc for allocating txb anyway.
> I'll give it a try.

So vmalloc didn't break anything on the 32bit machine I'm just about
to install 64 one so it will take hour or two.. I will issue some
official patch after that.

> There was already discussion on LKML about memory allocation problems
> on X86_64, which might explain this regression. This didn't happen
> before.

This is the thread title if you are interested.
'x86/kernel/pci_dma.c: gfp |= __GFP_NORETRY'

Tomas

2008-06-12 16:38:20

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 5:12 PM, Zdenek Kabelac
<[email protected]> wrote:
> 2008/6/12 Johannes Berg <[email protected]>:
>> On Thu, 2008-06-12 at 09:38 -0400, Rik van Riel wrote:
>>> On Thu, 12 Jun 2008 12:07:34 +0200
>>> "Zdenek Kabelac" <[email protected]> wrote:
>>>
>>> > It looks like while there was a huge amount of buffers and caches -
>>> > system was unable to allocate few pages for kmalloc in iwl3945 driver
>>> > after resume.
>>>
>>> It looks like this is because it wants to allocate 2**5 contiguous
>>> pages, which is 128kB of contiguous kernel memory.
>>
>> 64-bit system I assume?
>> The allocation should be 256 * 20 * sizeof(struct sk_buff *).
>>
>> Try the patch below. It should improve code generation too.
>
> I'll surely try you patch - but is the iwl the only driver which needs
> 128kB continuous memory chunk?

We do some stupid free-alloc sequence on restart this is where it
fails. I'm still polishing a patch that eliminates it.
Tomas

2008-06-12 17:04:29

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

> > Try the patch below. It should improve code generation too.
> >
> > I discussed this with Tomas previously and he says the hw is capable of
> > doing 20 fragments per frame (wonder why, Broadcom can do an unlimited
> > number...) but he complained about the networking stack not being able
> > to.
>
> This is scatter gather buffers that can be kicked in one DMA transaction.
>
> Well, the hardware needs to support IP checksumming for that, hence,
> > afaik, only two fragments can ever be used (one for hw header, one for
> > frame)
> This I still don't understand why but everybody is already tired to
> explaining me why.. :) Just need to find time to dig into it.

And you can safely decrease the allocation to 10% as I do in my patch
because once you understand you'll see that you cannot possibly use
more. Hence, you can ack this patch ;)

> > This cuts the allocation to 10%, or (under) a page in all cases.
>
> Probably. it would be safe to use vmalloc for allocating txb anyway.
> I'll give it a try.

Yeah, but why bother if we can just allocate 10% of the size, waste a
lot less memory etc. mac80211 isn't going to pass in a scatter/gather
frame anyway.

> There was already discussion on LKML about memory allocation problems
> on X86_64, which might explain this regression. This didn't happen
> before.

Doesn't really matter, iwlwifi is _wasting_ this allocation, it cannot
possibly use all those buffers anyway.

The more interesting thing is the pci_alloc_consistent allocation right
below that is also _huge_, but that's because of the stupid hardware
design, or can the hardware cope with having the descriptors non-linear
in memory?

johannes

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part

2008-06-12 17:06:43

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

> > Probably. it would be safe to use vmalloc for allocating txb anyway.
> > I'll give it a try.
>
> So vmalloc didn't break anything on the 32bit machine I'm just about
> to install 64 one so it will take hour or two.. I will issue some
> official patch after that.

Well, I disagree, and I'll push my patch as soon as somebody confirms
that it doesn't break anything.

> > There was already discussion on LKML about memory allocation problems
> > on X86_64, which might explain this regression. This didn't happen
> > before.
>
> This is the thread title if you are interested.
> 'x86/kernel/pci_dma.c: gfp |= __GFP_NORETRY'

Like I said, it doesn't matter, there's no need to _waste_
18*256*sizeof(void *) bytes memory.

johannes

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part

2008-06-12 17:10:52

by Rik van Riel

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, 12 Jun 2008 18:43:37 +0300
"Tomas Winkler" <[email protected]> wrote:

> This is scatter gather buffers that can be kicked in one DMA transaction.

> This I still don't understand why but everybody is already tired to
> explaining me why.. :) Just need to find time to dig into it.

The only thing that makes no sense to me is why your
driver "needs" to allocate 10x as much memory in that
buffer than it will ever use.

What is the problem with the simpler solution, which
just reduces the size of the buffer to an amount of
memory that might actually get used?

--
All Rights Reversed

2008-06-12 17:36:00

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 8:03 PM, Johannes Berg
<[email protected]> wrote:
>
>> > Try the patch below. It should improve code generation too.
>> >
>> > I discussed this with Tomas previously and he says the hw is capable of
>> > doing 20 fragments per frame (wonder why, Broadcom can do an unlimited
>> > number...) but he complained about the networking stack not being able
>> > to.
>>
>> This is scatter gather buffers that can be kicked in one DMA transaction.
>>
>> Well, the hardware needs to support IP checksumming for that, hence,
>> > afaik, only two fragments can ever be used (one for hw header, one for
>> > frame)
>> This I still don't understand why but everybody is already tired to
>> explaining me why.. :) Just need to find time to dig into it.
>
> And you can safely decrease the allocation to 10% as I do in my patch
> because once you understand you'll see that you cannot possibly use
> more. Hence, you can ack this patch ;)
>
>> > This cuts the allocation to 10%, or (under) a page in all cases.
>>
>> Probably. it would be safe to use vmalloc for allocating txb anyway.
>> I'll give it a try.
>
> Yeah, but why bother if we can just allocate 10% of the size, waste a
> lot less memory etc. mac80211 isn't going to pass in a scatter/gather
> frame anyway.

Hope never dies. I actually have seen this speed up the throughput so
I will dig into it anyway.

>> There was already discussion on LKML about memory allocation problems
>> on X86_64, which might explain this regression. This didn't happen
>> before.
>
> Doesn't really matter, iwlwifi is _wasting_ this allocation, it cannot
> possibly use all those buffers anyway.

This matter actually for consistent allocation.

> The more interesting thing is the pci_alloc_consistent allocation right
> below that is also _huge_, but that's because of the stupid hardware
> design, or can the hardware cope with having the descriptors non-linear
> in memory?

We talk after your next HW design. How will configure 265 * 16
descriptors separately.

Tomas

2008-06-12 17:39:47

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 8:05 PM, Johannes Berg
<[email protected]> wrote:
>
>> > Probably. it would be safe to use vmalloc for allocating txb anyway.
>> > I'll give it a try.
>>
>> So vmalloc didn't break anything on the 32bit machine I'm just about
>> to install 64 one so it will take hour or two.. I will issue some
>> official patch after that.
>
> Well, I disagree, and I'll push my patch as soon as somebody confirms
> that it doesn't break anything.

Remember you are not a maintainer of this driver and second we are
open to all suggestions you don't have to use this kind of
statements...

>
>> > There was already discussion on LKML about memory allocation problems
>> > on X86_64, which might explain this regression. This didn't happen
>> > before.
>>
>> This is the thread title if you are interested.
>> 'x86/kernel/pci_dma.c: gfp |= __GFP_NORETRY'
>
> Like I said, it doesn't matter, there's no need to _waste_
> 18*256*sizeof(void *) bytes memory.

It does matter this is not pci allocation we are saving in your patch.

> johannes
>

2008-06-12 17:40:20

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

> > Yeah, but why bother if we can just allocate 10% of the size, waste a
> > lot less memory etc. mac80211 isn't going to pass in a scatter/gather
> > frame anyway.
>
> Hope never dies. I actually have seen this speed up the throughput so
> I will dig into it anyway.

Well, you can always add it back later if you make the networking stack
and mac80211 support it. It's a two-line patch after all.

> > The more interesting thing is the pci_alloc_consistent allocation right
> > below that is also _huge_, but that's because of the stupid hardware
> > design, or can the hardware cope with having the descriptors non-linear
> > in memory?
>
> We talk after your next HW design. How will configure 265 * 16
> descriptors separately.

Well, considering that other hardware does manage to do things
differently (say Broadcom because I know their DMA engine), I don't know
why your hw designers went wild with this. All you need is an
"end-of-frame" flag. But that's not really interesting to discuss,
unless this is actually controlled by the microcode and you can change
it.

johannes

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part

2008-06-12 17:48:23

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

> > Well, I disagree, and I'll push my patch as soon as somebody confirms
> > that it doesn't break anything.
>
> Remember you are not a maintainer of this driver and second we are
> open to all suggestions you don't have to use this kind of
> statements...

Yeah, you're right, I can't really do that. But I can submit the patch
to akpm, and I'm sure he'll take it after you provide your counter
argument about hope never dying again ;)

Frankly, I don't see why you're so opposed to this patch even if it
doesn't solve anything it probably leads to better code generation and
using a lot less memory.

Also, I know you cannot actually need those descriptors since mac80211
will never ever pass such frames, and _that_ is an area I do have at
least some influence over, so I'll surely notice when that changes.

> >> > There was already discussion on LKML about memory allocation problems
> >> > on X86_64, which might explain this regression. This didn't happen
> >> > before.
> >>
> >> This is the thread title if you are interested.
> >> 'x86/kernel/pci_dma.c: gfp |= __GFP_NORETRY'
> >
> > Like I said, it doesn't matter, there's no need to _waste_
> > 18*256*sizeof(void *) bytes memory.
>
> It does matter this is not pci allocation we are saving in your patch.

Well, thing is, my patch saves 18 KiB memory on 32-bit and 36 on 64-bit,
so I think we should merge it regardless. Yes, the pci allocation is
icky, and yes, it would be good to just do it once instead of over and
over again, but even if you change it to do _all_ those allocations just
once we should not be wasting those 18/36 KiB memory for nothing.

johannes

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part

2008-06-12 17:51:20

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 8:39 PM, Johannes Berg
<[email protected]> wrote:
>
>> > Yeah, but why bother if we can just allocate 10% of the size, waste a
>> > lot less memory etc. mac80211 isn't going to pass in a scatter/gather
>> > frame anyway.
>>
>> Hope never dies. I actually have seen this speed up the throughput so
>> I will dig into it anyway.
>
> Well, you can always add it back later if you make the networking stack
> and mac80211 support it. It's a two-line patch after all.

I will handle this.

>> > The more interesting thing is the pci_alloc_consistent allocation right
>> > below that is also _huge_, but that's because of the stupid hardware
>> > design, or can the hardware cope with having the descriptors non-linear
>> > in memory?
>>
>> We talk after your next HW design. How will configure 265 * 16
>> descriptors separately.
>
> Well, considering that other hardware does manage to do things
> differently (say Broadcom because I know their DMA engine), I don't know
> why your hw designers went wild with this. All you need is an
> "end-of-frame" flag. But that's not really interesting to discuss,
> unless this is actually controlled by the microcode and you can change
> it.
>
This is have to do something with HW packet scheduling

> johannes
>

2008-06-12 18:04:09

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 8:46 PM, Johannes Berg
<[email protected]> wrote:
>
>> > Well, I disagree, and I'll push my patch as soon as somebody confirms
>> > that it doesn't break anything.
>>
>> Remember you are not a maintainer of this driver and second we are
>> open to all suggestions you don't have to use this kind of
>> statements...
>
> Yeah, you're right, I can't really do that. But I can submit the patch
> to akpm, and I'm sure he'll take it after you provide your counter
> argument about hope never dying again ;)

Here you go again

> Frankly, I don't see why you're so opposed to this patch even if it
> doesn't solve anything it probably leads to better code generation and
> using a lot less memory.

I'm not against it. You;v decided that I'm fighting you because I gave
another solution.
Frankly we probably don't need this allocation at all. maybe one skb
is just enough
even with my never dying hope all fragments are in skb fragment list.
This still probably won't save pci memory allocation problem

Tomas

> Also, I know you cannot actually need those descriptors since mac80211
> will never ever pass such frames, and _that_ is an area I do have at
> least some influence over, so I'll surely notice when that changes.
>
>> >> > There was already discussion on LKML about memory allocation problems
>> >> > on X86_64, which might explain this regression. This didn't happen
>> >> > before.
>> >>
>> >> This is the thread title if you are interested.
>> >> 'x86/kernel/pci_dma.c: gfp |= __GFP_NORETRY'
>> >
>> > Like I said, it doesn't matter, there's no need to _waste_
>> > 18*256*sizeof(void *) bytes memory.
>>
>> It does matter this is not pci allocation we are saving in your patch.
>
> Well, thing is, my patch saves 18 KiB memory on 32-bit and 36 on 64-bit,
> so I think we should merge it regardless. Yes, the pci allocation is
> icky, and yes, it would be good to just do it once instead of over and
> over again, but even if you change it to do _all_ those allocations just
> once we should not be wasting those 18/36 KiB memory for nothing.
>
> johannes
>

2008-06-12 18:16:41

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

> I'm not against it. You;v decided that I'm fighting you because I gave
> another solution.

Ok, no, I'm not saying you shouldn't rewrite all the code to get rid of
it, but I think you can use a patch like mine interim as such a rewrite
is unlikely to go into 2.6.26, is it?

> Frankly we probably don't need this allocation at all. maybe one skb
> is just enough

That would be nice, indeed.

> even with my never dying hope all fragments are in skb fragment list.

:)

> This still probably won't save pci memory allocation problem

Yeah, true, that one needs to be done, but it could probably be done
only once when hw is probed rather than every time it is brought up.
Most likely not something you'll get to fix in 2.6.26 either though.

johannes

Attachments:

signature.asc (836.00 B)
This is a digitally signed message part

2008-06-12 19:03:37

by John W. Linville

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 07:46:52PM +0200, Johannes Berg wrote:
>
> > > Well, I disagree, and I'll push my patch as soon as somebody confirms
> > > that it doesn't break anything.
> >
> > Remember you are not a maintainer of this driver and second we are
> > open to all suggestions you don't have to use this kind of
> > statements...
>
> Yeah, you're right, I can't really do that. But I can submit the patch
> to akpm, and I'm sure he'll take it after you provide your counter
> argument about hope never dying again ;)

Is there some reason you think I would need to be cut out of the loop??
akpm isn't the only one who likes solutions...

John
--
John W. Linville
[email protected]

2008-06-12 20:11:43

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

2008/6/12 Johannes Berg <[email protected]>:
>
>> I'm not against it. You;v decided that I'm fighting you because I gave
>> another solution.
>
> Ok, no, I'm not saying you shouldn't rewrite all the code to get rid of
> it, but I think you can use a patch like mine interim as such a rewrite
> is unlikely to go into 2.6.26, is it?
>
>> Frankly we probably don't need this allocation at all. maybe one skb
>> is just enough
>
> That would be nice, indeed.
>
>> even with my never dying hope all fragments are in skb fragment list.
>
> :)
>
>> This still probably won't save pci memory allocation problem
>
> Yeah, true, that one needs to be done, but it could probably be done
> only once when hw is probed rather than every time it is brought up.
> Most likely not something you'll get to fix in 2.6.26 either though.

Well - it's great that there will be saved few kB in allocation of
never used pointers in iwl driver - but does this really solve the
problem that kernel gets relatively quickly out of memory for
allocations of this size - I guess iwl isn't the only driver
requesting 32 sequential pages.

Is it possible to track how this memory gets fragment/lost - who owns
the block and why they are not back in the pool?

btw with 8hour uptime at this moment I can see this:

DMA: 26*4kB 37*8kB 72*16kB 65*32kB 3*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 1*4096kB = 7920kB
DMA32: 203*4kB 79*8kB 26*16kB 11*32kB 6*64kB 9*128kB 3*256kB 2*512kB
2*1024kB 0*2048kB 0*4096kB = 7588kB

so at this moment I can see quiet a lot of free DMA memory - but in my
trace at the thread beginig after several suspend/resumes this memory
was gone....

Zdenek

2008-06-12 21:31:58

by Jiri Slaby

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On 06/12/2008 05:43 PM, Tomas Winkler wrote:
> On Thu, Jun 12, 2008 at 4:54 PM, Johannes Berg
> <[email protected]> wrote:
>> This cuts the allocation to 10%, or (under) a page in all cases.
>
> Probably. it would be safe to use vmalloc for allocating txb anyway.
> I'll give it a try.

Why it wouldn't be "safe". I suggested it to you already, since allocating 64k
by kmalloc for descriptors accessed only in kernel is crud. Moreover you're
mixing the buffer with its descriptors here? Or what you're considering to vmalloc?

2008-06-12 22:17:25

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, Jun 12, 2008 at 11:11 PM, Zdenek Kabelac
<[email protected]> wrote:
> 2008/6/12 Johannes Berg <[email protected]>:
>>
>>> I'm not against it. You;v decided that I'm fighting you because I gave
>>> another solution.
>>
>> Ok, no, I'm not saying you shouldn't rewrite all the code to get rid of
>> it, but I think you can use a patch like mine interim as such a rewrite
>> is unlikely to go into 2.6.26, is it?
>>
>>> Frankly we probably don't need this allocation at all. maybe one skb
>>> is just enough
>>
>> That would be nice, indeed.
>>
>>> even with my never dying hope all fragments are in skb fragment list.
>>
>> :)
>>
>>> This still probably won't save pci memory allocation problem
>>
>> Yeah, true, that one needs to be done, but it could probably be done
>> only once when hw is probed rather than every time it is brought up.
>> Most likely not something you'll get to fix in 2.6.26 either though.
>
>
> Well - it's great that there will be saved few kB in allocation of
> never used pointers in iwl driver - but does this really solve the
> problem that kernel gets relatively quickly out of memory for
> allocations of this size - I guess iwl isn't the only driver
> requesting 32 sequential pages.
>
> Is it possible to track how this memory gets fragment/lost - who owns
> the block and why they are not back in the pool?
>
> btw with 8hour uptime at this moment I can see this:
>
> DMA: 26*4kB 37*8kB 72*16kB 65*32kB 3*64kB 0*128kB 0*256kB 0*512kB
> 0*1024kB 0*2048kB 1*4096kB = 7920kB
> DMA32: 203*4kB 79*8kB 26*16kB 11*32kB 6*64kB 9*128kB 3*256kB 2*512kB
> 2*1024kB 0*2048kB 0*4096kB = 7588kB
>
> so at this moment I can see quiet a lot of free DMA memory - but in my
> trace at the thread beginig after several suspend/resumes this memory
> was gone....

Currently the driver frees the memory on down and allocates it back on
up. This is done even in initial
reset for sake of flow simplicity.
I'm not sure yet why the memory is actually accumulating, whether the
bug is in the driver or memory system is not clear to me yet. I
haven't seen this on older kernels or driver.
As I wrote I'm also polishing a patch that doesn't do this free-alloc
loop hope this will remedy somehow this problem. It has a drawback as
it will hold on memory even if devices is down.

Tomas

2008-06-12 22:27:16

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Fri, Jun 13, 2008 at 12:30 AM, Jiri Slaby <[email protected]> wrote:
> On 06/12/2008 05:43 PM, Tomas Winkler wrote:
>>
>> On Thu, Jun 12, 2008 at 4:54 PM, Johannes Berg
>> <[email protected]> wrote:
>>>
>>> This cuts the allocation to 10%, or (under) a page in all cases.
>>
>> Probably. it would be safe to use vmalloc for allocating txb anyway.
>> I'll give it a try.
>
> Why it wouldn't be "safe". I suggested it to you already, since allocating
> 64k by kmalloc for descriptors accessed only in kernel is crud. Moreover
> you're mixing the buffer with its descriptors here? Or what you're
> considering to vmalloc?
>
Not that. I just wasn't sure when I dropped the line I'm not doing it
under some spinlock or something like that.
Tomas

2008-06-13 00:45:17

by Andrew Morton

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thu, 12 Jun 2008 22:11:32 +0200
"Zdenek Kabelac" <[email protected]> wrote:

> Well - it's great that there will be saved few kB in allocation of
> never used pointers in iwl driver - but does this really solve the
> problem that kernel gets relatively quickly out of memory for
> allocations of this size - I guess iwl isn't the only driver
> requesting 32 sequential pages.

I hope it is the only one. Doing a 128kb GFP_ATOMIC allocation is
hopelessly unreliable. Even a 128k GFP_KERNEL allocation will fail all
over the place.

Please convert the driver to allocate no more than 4k at a time.

2008-06-13 14:07:38

by Rafael J. Wysocki

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

On Thursday, 12 of June 2008, Zdenek Kabelac wrote:
> Hello
>
> I'm attaching a trace where my machine has got into big troubles after
> 2 day usage and several successful suspend/resumes (this seems to be
> finally getting better now :))
>
> It looks like while there was a huge amount of buffers and caches -
> system was unable to allocate few pages for kmalloc in iwl3945 driver
> after resume.
>
> I've even tried to 3 > drop_cache and reinsert iwl driver - but this
> had fatal results - machine died completely with blinking caps lock -
> and no oops in the log for this case:
>
> This is the commit aab2545fdd6641b76af0ae96456c4ca9d1e50dad for the
> 2.6.26-rc5 I've been in this case.

Is this a regression from 2.6.25, BTW?

Rafael

2008-06-13 14:15:29

[permalink] [raw]

Subject: Re: Problem: Out of memory after 2days with 2GB RAM

2008/6/13 Rafael J. Wysocki <[email protected]>:
> On Thursday, 12 of June 2008, Zdenek Kabelac wrote:
>> Hello
>>
>> I'm attaching a trace where my machine has got into big troubles after
>> 2 day usage and several successful suspend/resumes (this seems to be
>> finally getting better now :))
>>
>> It looks like while there was a huge amount of buffers and caches -
>> system was unable to allocate few pages for kmalloc in iwl3945 driver
>> after resume.
>>
>> I've even tried to 3 > drop_cache and reinsert iwl driver - but this
>> had fatal results - machine died completely with blinking caps lock -
>> and no oops in the log for this case:
>>
>> This is the commit aab2545fdd6641b76af0ae96456c4ca9d1e50dad for the
>> 2.6.26-rc5 I've been in this case.
>
> Is this a regression from 2.6.25, BTW?

Well I've never seen this with 2.6.25 kernel - on the other hand
usually I've not been running machine for a longer period of time,
because suspend was failing too often I guess. Now it's more stable so
this bug has shown up.

It might be related to this issue as well http://lkml.org/lkml/2008/5/22/308

Zdenek

2008-06-30 11:31:20