LinuxLists.cc - [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

Andi Kleen wrote:
>>> [ 20.880366] DMA: Out of SW-IOMMU space for 263200 bytes at device ?
>>> [ 20.886858] Kernel panic - not syncing: DMA: Memory would be corrupted
>>>
>> Gad, never seen that before. Andi, Tony: help?
>>
>
> Either the driver is leaking or more likely something went
> wrong with the swiotlb initialization. Earlier boot messages
> might tell
>
> -Andi
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Attached the boot log and config file.

Thanks & Regards,
Kamalesh Babulal.

Attachments:

boot_log (15.42 kB)
dotconfig (30.33 kB)
Download all attachments

2007-08-22 21:04:35

[permalink] [raw]

Subject: RE: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

> Attached the boot log and config file.

Kamelesh,

I don't see anything obvious in the boot_log.

I used your "dotconfig" file to build a 2.6.23-rc3-mm1 kernel
and booted it on my test system ... it worked just fine
(except that for some reason the network did not come up :-( )

I tried to compare my boot log with yours ... you have some
different devices (E.g. your ethernet links are Tigon3 while
mine are e1000) ... so the memory leak in a driver theory
is a possibility.

Try adding a dump_stack() call to lib/swiotlb.c where the
"DMA: Out of SW-IOMMU space ..." message is printed. That
would tell us who is making the call that fails (they
might be an innocent bystander after someone else has
used all the space ... but this is unlikely).

-Ton

2007-08-22 22:24:51

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

Luck, Tony wrote:
>> Attached the boot log and config file.
>>
>
> Kamelesh,
>
> I don't see anything obvious in the boot_log.
>
> I used your "dotconfig" file to build a 2.6.23-rc3-mm1 kernel
> and booted it on my test system ... it worked just fine
> (except that for some reason the network did not come up :-( )
>
> I tried to compare my boot log with yours ... you have some
> different devices (E.g. your ethernet links are Tigon3 while
> mine are e1000) ... so the memory leak in a driver theory
> is a possibility.
>
> Try adding a dump_stack() call to lib/swiotlb.c where the
> "DMA: Out of SW-IOMMU space ..." message is printed. That
> would tell us who is making the call that fails (they
> might be an innocent bystander after someone else has
> used all the space ... but this is unlikely).
>
> -Ton
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
stack dump after the message "DMA: Out of SW-IOMMU space ...."

[ 20.865382] DMA: Out of SW-IOMMU space for 263200 bytes at device ?
[ 20.871946]
[ 20.871947] Call Trace:
[ 20.876287] [<a0000001000144e0>] show_stack+0x80/0xa0
[ 20.876289] sp=e00000014322f8f0 bsp=e000000143229170
[ 20.889731] [<a000000100014530>] dump_stack+0x30/0x60
[ 20.889733] sp=e00000014322fac0 bsp=e000000143229158
[ 20.903201] [<a0000001003aaa50>] swiotlb_full+0x50/0x120
[ 20.903202] sp=e00000014322fac0 bsp=e000000143229120
[ 20.916902] [<a0000001003aac40>] swiotlb_map_single+0x120/0x1c0
[ 20.916904] sp=e00000014322fac0 bsp=e0000001432290d8
[ 20.931215] [<a0000001003ab630>] swiotlb_alloc_coherent+0x150/0x240
[ 20.931217] sp=e00000014322fac0 bsp=e000000143229090
[ 20.945923] [<a000000100550860>] PrimeIocFifos+0x4c0/0xb20
[ 20.945925] sp=e00000014322fac0 bsp=e000000143229010
[ 20.959812] [<a000000100556a80>] mpt_do_ioc_recovery+0xd60/0x28e0
[ 20.959814] sp=e00000014322faf0 bsp=e000000143228f30
[ 20.974310] [<a00000010055c8f0>] mpt_attach+0x830/0x20e0
[ 20.974311] sp=e00000014322fdc0 bsp=e000000143228eb0
[ 20.988021] [<a0000001005667b0>] mptspi_probe+0x30/0x720
[ 20.988023] sp=e00000014322fdd0 bsp=e000000143228e60
[ 21.001743] [<a0000001003bb4b0>] pci_device_probe+0x1f0/0x2c0
[ 21.001745] sp=e00000014322fdd0 bsp=e000000143228e18
[ 21.015911] [<a00000010048f9e0>] driver_probe_device+0x180/0x400
[ 21.015913] sp=e00000014322fdd0 bsp=e000000143228dc8
[ 21.030305] [<a00000010048ff20>] __driver_attach+0xc0/0x160
[ 21.030307] sp=e00000014322fdd0 bsp=e000000143228d90
[ 21.044268] [<a00000010048dd90>] bus_for_each_dev+0xb0/0x120
[ 21.044269] sp=e00000014322fdd0 bsp=e000000143228d58
[ 21.058316] [<a00000010048f640>] driver_attach+0x40/0x60
[ 21.058318] sp=e00000014322fdf0 bsp=e000000143228d38
[ 21.072024] [<a00000010048e600>] bus_add_driver+0x120/0x400
[ 21.072026] sp=e00000014322fdf0 bsp=e000000143228cf8
[ 21.085989] [<a0000001004903c0>] driver_register+0xc0/0x180
[ 21.085990] sp=e00000014322fdf0 bsp=e000000143228cd8
[ 21.099952] [<a0000001003bb980>] __pci_register_driver+0xc0/0x140
[ 21.099953] sp=e00000014322fdf0 bsp=e000000143228ca0
[ 21.114467] [<a000000100943920>] mptspi_init+0x180/0x1c0
[ 21.114469] sp=e00000014322fe00 bsp=e000000143228c70
[ 21.130036] [<a0000001008fc5d0>] kernel_init+0x350/0x840
[ 21.130038] sp=e00000014322fe00 bsp=e000000143228c10
[ 21.143737] [<a000000100012150>] kernel_thread_helper+0xd0/0x100
[ 21.143739] sp=e00000014322fe30 bsp=e000000143228be0
[ 21.158141] [<a000000100008dc0>] start_kernel_thread+0x20/0x40
[ 21.158143] sp=e00000014322fe30 bsp=e000000143228be0
[ 21.172360] Kernel panic - not syncing: DMA: Memory would be corrupted

Thanks & Regards,
Kamalesh Babulal

2007-08-22 22:57:19

[permalink] [raw]

Subject: RE: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

[ 20.903201] [<a0000001003aaa50>] swiotlb_full+0x50/0x120
[ 20.903202] sp=e00000014322fac0 bsp=e000000143229120
[ 20.916902] [<a0000001003aac40>] swiotlb_map_single+0x120/0x1c0
[ 20.916904] sp=e00000014322fac0 bsp=e0000001432290d8
[ 20.931215] [<a0000001003ab630>] swiotlb_alloc_coherent+0x150/0x240
[ 20.931217] sp=e00000014322fac0 bsp=e000000143229090
[ 20.945923] [<a000000100550860>] PrimeIocFifos+0x4c0/0xb20
[ 20.945925] sp=e00000014322fac0 bsp=e000000143229010
[ 20.959812] [<a000000100556a80>] mpt_do_ioc_recovery+0xd60/0x28e0
[ 20.959814] sp=e00000014322faf0 bsp=e000000143228f30
[ 20.974310] [<a00000010055c8f0>] mpt_attach+0x830/0x20e0

Hmmm! So you were in the mpt/fusion driver when you ran out
of SWIOTLB space. That's an area where we both have the same
hardware ... and since it booted for me, it means that the
driver isn't totally broken.

I'm totally ignorant of what goes on inside this driver though.
You have more "ioc's" than I do. I only see messages from mpt
bringing up ioc0 & ioc1. Your boot_log also has ioc2 (which is
where you crash). Here's the sdiff(1) output comparing the MPT
part of your boot log with my successful boot of the same kernel
and config (your log is the one on the left). Maybe some MPT/Fusion
expert can spot something important in this bit?

-Tony

Fusion MPT base driver 3.04.05 Fusion MPT base driver 3.04.05
Copyright (c) 1999-2007 LSI Corporation Copyright (c) 1999-2007 LSI Corporation
Fusion MPT SPI Host driver 3.04.05 Fusion MPT SPI Host driver 3.04.05
GSI 40 (level, low) -> CPU 3 (0x0600) vector 71 | GSI 28 (level, low) -> CPU 0 (0xc018) vector 48
ACPI: PCI Interrupt 0000:01:03.0[A] -> GSI 40 (level, low) -> | ACPI: PCI Interrupt 0000:06:02.0[A] -> GSI 28 (level, low) ->
mptbase: Initiating ioc0 bringup mptbase: Initiating ioc0 bringup
ioc0: LSI53C1030 C0: Capabilities={Initiator} | ioc0: LSI53C1030 B2: Capabilities={Initiator}
scsi0 : ioc0: LSI53C1030 C0, FwRev=01032821h, Ports=1, MaxQ=2 | scsi0 : ioc0: LSI53C1030 B2, FwRev=01030a00h, Ports=1, MaxQ=2
scsi 0:0:0:0: Direct-Access IBM-ESXS DTN018C1UCDY10F S25 | scsi 0:0:0:0: Direct-Access QUANTUM ATLAS IV 9 SCA 0B0
target0:0:0: Beginning Domain Validation target0:0:0: Beginning Domain Validation
> target0:0:0: Domain Validation skipping write tests
target0:0:0: Ending Domain Validation target0:0:0: Ending Domain Validation
target0:0:0: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offse | target0:0:0: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offse
sd 0:0:0:0: [sda] 35548320 512-byte hardware sectors (18201 M | sd 0:0:0:0: [sda] 17942584 512-byte hardware sectors (9187 MB
sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, | sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled,
sd 0:0:0:0: [sda] 35548320 512-byte hardware sectors (18201 M | sd 0:0:0:0: [sda] 17942584 512-byte hardware sectors (9187 MB
sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, | sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled,
sda: sda1 sda2 | sda: sda1 sda2 sda3
sd 0:0:0:0: [sda] Attached SCSI disk sd 0:0:0:0: [sda] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0 sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 0:0:8:0: Processor IBM 25P3495a S320 1 1 | scsi 0:0:1:0: Direct-Access SEAGATE ST318406LC 010
target0:0:8: Beginning Domain Validation | target0:0:1: Beginning Domain Validation
target0:0:8: Ending Domain Validation | target0:0:1: Ending Domain Validation
target0:0:8: asynchronous | target0:0:1: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offse
scsi 0:0:8:0: Attached scsi generic sg1 type 3 | sd 0:0:1:0: [sdb] 35843670 512-byte hardware sectors (18352 M
GSI 41 (level, low) -> CPU 4 (0x1000) vector 72 | sd 0:0:1:0: [sdb] Write Protect is off
ACPI: PCI Interrupt 0000:01:03.1[B] -> GSI 41 (level, low) -> | sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled,
> sd 0:0:1:0: [sdb] 35843670 512-byte hardware sectors (18352 M
> sd 0:0:1:0: [sdb] Write Protect is off
> sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled,
> sdb: sdb1 sdb2 sdb3
> sd 0:0:1:0: [sdb] Attached SCSI disk
> sd 0:0:1:0: Attached scsi generic sg1 type 0
> scsi 0:0:6:0: Processor ESG-SHV SCA HSBP M17 1.0
> target0:0:6: Beginning Domain Validation
> target0:0:6: Ending Domain Validation
> target0:0:6: asynchronous
> scsi 0:0:6:0: Attached scsi generic sg2 type 3
> GSI 29 (level, low) -> CPU 1 (0xc218) vector 49
> ACPI: PCI Interrupt 0000:06:02.1[B] -> GSI 29 (level, low) ->
mptbase: Initiating ioc1 bringup mptbase: Initiating ioc1 bringup
ioc1: LSI53C1030 C0: Capabilities={Initiator} | ioc1: LSI53C1030 B2: Capabilities={Initiator}
scsi1 : ioc1: LSI53C1030 C0, FwRev=01032821h, Ports=1, MaxQ=2 | scsi1 : ioc1: LSI53C1030 B2, FwRev=01030a00h, Ports=1, MaxQ=2
GSI 142 (level, low) -> CPU 5 (0x1200) vector 73 | Fusion MPT SAS Host driver 3.04.05
ACPI: PCI Interrupt 0000:41:03.0[A] -> GSI 142 (level, low) - | USB Universal Host Controller Interface driver v3.0
mptbase: Initiating ioc2 bringup | GSI 16 (level, low) -> CPU 2 (0xc418) vector 50
ioc2: LSI53C1030 C0: Capabilities={Initiator} | ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) ->
DMA: Out of SW-IOMMU space for 263200 bytes at device ? | uhci_hcd 0000:00:1d.0: UHCI Host Controller
Kernel panic - not syncing: DMA: Memory would be corrupted | uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus n

2007-08-22 23:11:31

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

On Wed, Aug 22, 2007 at 03:56:57PM -0700, Luck, Tony wrote:
> [ 20.903201] [<a0000001003aaa50>] swiotlb_full+0x50/0x120
> [ 20.903202] sp=e00000014322fac0 bsp=e000000143229120
> [ 20.916902] [<a0000001003aac40>] swiotlb_map_single+0x120/0x1c0
> [ 20.916904] sp=e00000014322fac0 bsp=e0000001432290d8
> [ 20.931215] [<a0000001003ab630>] swiotlb_alloc_coherent+0x150/0x240
> [ 20.931217] sp=e00000014322fac0 bsp=e000000143229090
> [ 20.945923] [<a000000100550860>] PrimeIocFifos+0x4c0/0xb20
> [ 20.945925] sp=e00000014322fac0 bsp=e000000143229010
> [ 20.959812] [<a000000100556a80>] mpt_do_ioc_recovery+0xd60/0x28e0
> [ 20.959814] sp=e00000014322faf0 bsp=e000000143228f30
> [ 20.974310] [<a00000010055c8f0>] mpt_attach+0x830/0x20e0
>
>
> Hmmm! So you were in the mpt/fusion driver when you ran out
> of SWIOTLB space. That's an area where we both have the same
> hardware ... and since it booted for me, it means that the
> driver isn't totally broken.
>
> I'm totally ignorant of what goes on inside this driver though.
> You have more "ioc's" than I do. I only see messages from mpt
> bringing up ioc0 & ioc1. Your boot_log also has ioc2 (which is
> where you crash). Here's the sdiff(1) output comparing the MPT
> part of your boot log with my successful boot of the same kernel
> and config (your log is the one on the left). Maybe some MPT/Fusion
> expert can spot something important in this bit?

The more ioc's you have, the more space you will use.

jeremy

2007-08-22 23:27:24

[permalink] [raw]

Subject: RE: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

> The more ioc's you have, the more space you will use.

Default SW IOTLB allocation is 64MB ... how much should we see
used per ioc?

Kamelesh: You could try increasing the amount of sw iotlb space
available by booting with a swiotlb=131072 argument (argument
value is the number of 2K slabs to allocate ... 131072 would
give you four times as much space as the default allocation).

-Tony

2007-08-22 23:54:59

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

On Wed, Aug 22, 2007 at 04:27:09PM -0700, Luck, Tony wrote:
> > The more ioc's you have, the more space you will use.
>
> Default SW IOTLB allocation is 64MB ... how much should we see
> used per ioc?

Hmm. Must be something else going on then. It should be less than 1MB
per ioc plus whatever is used for streaming I/O.

| mptbase: Initiating ioc2 bringup | GSI 16 (level, low) -> CPU 2 (0xc418) vector 50
| ioc2: LSI53C1030 C0: Capabilities={Initiator} | ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) ->
| DMA: Out of SW-IOMMU space for 263200 bytes at device ? | uhci_hcd 0000:00:1d.0: UHCI Host Controller
| Kernel panic - not syncing: DMA: Memory would be corrupted | uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus n

2007-08-23 00:06:33

[permalink] [raw]

Subject: RE: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

> Hmm. Must be something else going on then. It should be less than 1MB
> per ioc plus whatever is used for streaming I/O.
>
> | mptbase: Initiating ioc2 bringup | GSI 16 (level, low) -> CPU 2 (0xc418) vector 50
> | ioc2: LSI53C1030 C0: Capabilities={Initiator} | ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) ->
> | DMA: Out of SW-IOMMU space for 263200 bytes at device ? | uhci_hcd 0000:00:1d.0: UHCI Host Controller
> | Kernel panic - not syncing: DMA: Memory would be corrupted | uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus n

I traced the pci_alloc_consistent calls from PrimeIocFifos on my
system. There are two calls for each ioc. The first is for
266368 bytes, the second for 16320 bytes.

I wonder why Kamalesh's system wants the slightly different
amount (263200 bytes) from what my system asks for?

It also looks to be a little unfriendly to swiotlb to ask for
more than 256K at a time (see IO_TLB_SEGSIZE) in swiotlb.c

-Tony

2007-08-23 01:10:01

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

On Wed, Aug 22, 2007 at 05:05:54PM -0700, Luck, Tony wrote:
> > Hmm. Must be something else going on then. It should be less than 1MB
> > per ioc plus whatever is used for streaming I/O.
> >
> > | mptbase: Initiating ioc2 bringup | GSI 16 (level, low) -> CPU 2 (0xc418) vector 50
> > | ioc2: LSI53C1030 C0: Capabilities={Initiator} | ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) ->
> > | DMA: Out of SW-IOMMU space for 263200 bytes at device ? | uhci_hcd 0000:00:1d.0: UHCI Host Controller
> > | Kernel panic - not syncing: DMA: Memory would be corrupted | uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus n
>
> I traced the pci_alloc_consistent calls from PrimeIocFifos on my
> system. There are two calls for each ioc. The first is for
> 266368 bytes, the second for 16320 bytes.
>
> I wonder why Kamalesh's system wants the slightly different
> amount (263200 bytes) from what my system asks for?
>
> It also looks to be a little unfriendly to swiotlb to ask for
> more than 256K at a time (see IO_TLB_SEGSIZE) in swiotlb.c
>
> -Tony

I believe those would vary a bit based on the exact firmware
rev and perhaps nvram settings. Also driver settings, but
those are presumably the same.

jeremy

2007-08-23 01:16:21

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

On Wed, Aug 22, 2007 at 06:09:48PM -0700, Jeremy Higdon wrote:
> > I traced the pci_alloc_consistent calls from PrimeIocFifos on my
> > system. There are two calls for each ioc. The first is for
> > 266368 bytes, the second for 16320 bytes.
> >
> > I wonder why Kamalesh's system wants the slightly different
> > amount (263200 bytes) from what my system asks for?
> >
> > It also looks to be a little unfriendly to swiotlb to ask for
> > more than 256K at a time (see IO_TLB_SEGSIZE) in swiotlb.c
> >
> > -Tony
>
> I believe those would vary a bit based on the exact firmware
> rev and perhaps nvram settings. Also driver settings, but
> those are presumably the same.

> ioc0: LSI53C1030 C0: Capabilities={Initiator} | ioc0: LSI53C1030 B2: Capabilities={Initiator}
> scsi0 : ioc0: LSI53C1030 C0, FwRev=01032821h, Ports=1, MaxQ=2 | scsi0 : ioc0: LSI53C1030 B2, FwRev=01030a00h, Ports=1, MaxQ=2
> ioc1: LSI53C1030 C0: Capabilities={Initiator} | ioc1: LSI53C1030 B2: Capabilities={Initiator}
> scsi1 : ioc1: LSI53C1030 C0, FwRev=01032821h, Ports=1, MaxQ=2 | scsi1 : ioc1: LSI53C1030 B2, FwRev=01030a00h, Ports=1, MaxQ=2
> ioc2: LSI53C1030 C0: Capabilities={Initiator} | ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) ->

Actually, you can see that you have a different chip rev level and different
firmware revs, so that's probably why the requested sizes are a little
different.

Compare /proc/mpt/ioc0/info if you're curious. There's probably a small
difference.

jeremy

2007-08-23 09:16:15

by mel

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

On (22/08/07 16:27), Luck, Tony didst pronounce:
> > The more ioc's you have, the more space you will use.
>
> Default SW IOTLB allocation is 64MB ... how much should we see
> used per ioc?
>
> Kamelesh: You could try increasing the amount of sw iotlb space
> available by booting with a swiotlb=131072 argument (argument
> value is the number of 2K slabs to allocate ... 131072 would
> give you four times as much space as the default allocation).
>

I tried that value and just in case swiotlb=262144. An IA-64 machines I
have here fails with the same message anyway. i.e.

[ 19.834906] mptbase: Initiating ioc1 bringup
[ 20.317152] ioc1: LSI53C1030 C0: Capabilities={Initiator}
[ 15.474303] scsi1 : ioc1: LSI53C1030 C0, FwRev=01032821h, Ports=1, MaxQ=222, IRQ=72
[ 20.669730] GSI 142 (level, low) -> CPU 5 (0x1200) vector 73
[ 20.675602] ACPI: PCI Interrupt 0000:41:03.0[A] -> GSI 142 (level, low) -> IRQ 73
[ 20.683508] mptbase: Initiating ioc2 bringup
[ 21.166796] ioc2: LSI53C1030 C0: Capabilities={Initiator}
[ 21.180539] DMA: Out of SW-IOMMU space for 263200 bytes at device ?
[ 21.187018] Kernel panic - not syncing: DMA: Memory would be corrupted

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab

2007-08-23 09:24:15

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

Luck, Tony wrote:
>> The more ioc's you have, the more space you will use.
>>
>
> Default SW IOTLB allocation is 64MB ... how much should we see
> used per ioc?
>
> Kamelesh: You could try increasing the amount of sw iotlb space
> available by booting with a swiotlb=131072 argument (argument
> value is the number of 2K slabs to allocate ... 131072 would
> give you four times as much space as the default allocation).
>
> -Tony
>
boot log after passing boot parameter swiotlb=131072

[ 0.000000] 0: 32768 -> 131072
[ 0.000000] 0: 262144 -> 1703936
[ 0.000000] 1: 4194304 -> 4980736
[ 0.000000] Built 2 zonelists in Zone order, mobility grouping on.
Total pages: 1563903
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line:
BOOT_IMAGE=scsi0:/EFI/debian/boot/vmlinuz-autobench root=/dev/sda2
console=tty0 console=ttyS0,115200n8 ro autobench_args: root=/dev/sda2
ABAT:1187857488 profile=2 swiotlb=131072
[ 0.000000] kernel profiling enabled (shift: 2)
<<snip>>
[ 20.408360] mptbase: Initiating ioc2 bringup
[ 20.892659] ioc2: LSI53C1030 C0: Capabilities={Initiator}
[ 20.902432] DMA: Out of SW-IOMMU space for 263200 bytes at device ?
[ 20.908992]
[ 20.908993] Call Trace:
[ 20.913324] [<a0000001000144e0>] show_stack+0x80/0xa0
[ 20.913327] sp=e00000014322f8f0
bsp=e000000143229170
[ 20.926764] [<a000000100014530>] dump_stack+0x30/0x60
[ 20.926766] sp=e00000014322fac0
bsp=e000000143229158
[ 20.940225] [<a0000001003aaa50>] swiotlb_full+0x50/0x120
[ 20.940227] sp=e00000014322fac0
bsp=e000000143229120
[ 20.953915] [<a0000001003aac40>] swiotlb_map_single+0x120/0x1c0
[ 20.953916] sp=e00000014322fac0
bsp=e0000001432290d8
[ 20.968223] [<a0000001003ab630>] swiotlb_alloc_coherent+0x150/0x240
[ 20.968226] sp=e00000014322fac0
bsp=e000000143229090
[ 20.982919] [<a000000100550860>] PrimeIocFifos+0x4c0/0xb20
[ 20.982922] sp=e00000014322fac0
bsp=e000000143229010
[ 20.996801] [<a000000100556a80>] mpt_do_ioc_recovery+0xd60/0x28e0
[ 20.996803] sp=e00000014322faf0
bsp=e000000143228f30
[ 21.011292] [<a00000010055c8f0>] mpt_attach+0x830/0x20e0
[ 21.011293] sp=e00000014322fdc0
bsp=e000000143228eb0
[ 21.025009] [<a0000001005667b0>] mptspi_probe+0x30/0x720
[ 21.025011] sp=e00000014322fdd0
bsp=e000000143228e60
[ 21.038722] [<a0000001003bb4b0>] pci_device_probe+0x1f0/0x2c0
[ 21.038724] sp=e00000014322fdd0
bsp=e000000143228e18
[ 21.052887] [<a00000010048f9e0>] driver_probe_device+0x180/0x400
[ 21.052888] sp=e00000014322fdd0
bsp=e000000143228dc8
[ 21.067276] [<a00000010048ff20>] __driver_attach+0xc0/0x160
[ 21.067277] sp=e00000014322fdd0
bsp=e000000143228d90
[ 21.081234] [<a00000010048dd90>] bus_for_each_dev+0xb0/0x120
[ 21.081236] sp=e00000014322fdd0
bsp=e000000143228d58
[ 21.095283] [<a00000010048f640>] driver_attach+0x40/0x60
[ 21.095285] sp=e00000014322fdf0
bsp=e000000143228d38
[ 21.108983] [<a00000010048e600>] bus_add_driver+0x120/0x400
[ 21.108985] sp=e00000014322fdf0
bsp=e000000143228cf8
[ 21.122942] [<a0000001004903c0>] driver_register+0xc0/0x180
[ 21.122943] sp=e00000014322fdf0
bsp=e000000143228cd8
[ 21.136906] [<a0000001003bb980>] __pci_register_driver+0xc0/0x140
[ 21.136908] sp=e00000014322fdf0
bsp=e000000143228ca0
[ 21.151422] [<a000000100943920>] mptspi_init+0x180/0x1c0
[ 21.151425] sp=e00000014322fe00
bsp=e000000143228c70
[ 21.166973] [<a0000001008fc5d0>] kernel_init+0x350/0x840
[ 21.166974] sp=e00000014322fe00
bsp=e000000143228c10
[ 21.180677] [<a000000100012150>] kernel_thread_helper+0xd0/0x100
[ 21.180679] sp=e00000014322fe30
bsp=e000000143228be0
[ 21.195082] [<a000000100008dc0>] start_kernel_thread+0x20/0x40
[ 21.195084] sp=e00000014322fe30
bsp=e000000143228be0
[ 21.209297] Kernel panic - not syncing: DMA: Memory would be corrupted

-
Kamalesh Babulal.

2007-08-23 13:29:19

by Yasunori Goto

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

> On (22/08/07 16:27), Luck, Tony didst pronounce:
> > > The more ioc's you have, the more space you will use.
> >
> > Default SW IOTLB allocation is 64MB ... how much should we see
> > used per ioc?
> >
> > Kamelesh: You could try increasing the amount of sw iotlb space
> > available by booting with a swiotlb=131072 argument (argument
> > value is the number of 2K slabs to allocate ... 131072 would
> > give you four times as much space as the default allocation).
> >
>
> I tried that value and just in case swiotlb=262144. An IA-64 machines I
> have here fails with the same message anyway. i.e.
>
> [ 19.834906] mptbase: Initiating ioc1 bringup
> [ 20.317152] ioc1: LSI53C1030 C0: Capabilities={Initiator}
> [ 15.474303] scsi1 : ioc1: LSI53C1030 C0, FwRev=01032821h, Ports=1, MaxQ=222, IRQ=72
> [ 20.669730] GSI 142 (level, low) -> CPU 5 (0x1200) vector 73
> [ 20.675602] ACPI: PCI Interrupt 0000:41:03.0[A] -> GSI 142 (level, low) -> IRQ 73
> [ 20.683508] mptbase: Initiating ioc2 bringup
> [ 21.166796] ioc2: LSI53C1030 C0: Capabilities={Initiator}
> [ 21.180539] DMA: Out of SW-IOMMU space for 263200 bytes at device ?
> [ 21.187018] Kernel panic - not syncing: DMA: Memory would be corrupted

I saw same trouble on my box, and I chased what was wrong.
Here is today's progress of mine.

__get_free_pages() of swiotlb_alloc_coherent() fails in rc3-mm1.
(See following patch)
But, it doesn't fail on rc2-mm2, and kernel can boot up.

Hmmm....

(2.6.23-rc3-mm1)
---
swiotlb_alloc_coherent flags=21 order=3 ret=0000000000000000
DMA: Out of SW-IOMMU space for 266368 bytes at device ?
Kernel panic - not syncing: DMA: Memory would be corrupted
---

(2.6.23-rc2-mm2)
---
swiotlb_alloc_coherent flags=21 order=3 ret=e000000020080000
:
(boot up continue...)

---
lib/swiotlb.c | 2 ++
1 file changed, 2 insertions(+)

Index: current/lib/swiotlb.c
===================================================================
--- current.orig/lib/swiotlb.c 2007-08-23 22:27:01.000000000 +0900
+++ current/lib/swiotlb.c 2007-08-23 22:29:49.000000000 +0900
@@ -455,6 +455,8 @@ swiotlb_alloc_coherent(struct device *hw
flags |= GFP_DMA;

ret = (void *)__get_free_pages(flags, order);
+
+ printk("%s flags=%0x order=%d ret=%p\n",__func__, flags, order, ret);
if (ret && address_needs_mapping(hwdev, virt_to_bus(ret))) {
/*
* The allocated memory isn't reachable by the device.

--
Yasunori Goto

2007-08-23 17:22:39

[permalink] [raw]

Subject: RE: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

> __get_free_pages() of swiotlb_alloc_coherent() fails in rc3-mm1.
> But, it doesn't fail on rc2-mm2, and kernel can boot up.

That looks to be part of the problem here ... failing an order=3
allocation during boot on a system that just a few lines earlier
in the boot log reported "Memory: 37474000k/37680640k available"
looks bad ... but perhaps having *more* memory is part of your problem.
You may have run low on GFP_DMA memory because some allocation
scaled by memory size has chewed up a lot of your memory. To check
this try booting with a "mem=4G" parameter and see if that helps
you.

But it is also bad that the swiotlb() code failed to handle this.
Can you check whether the problem is related to the size of the
allocation being just over 256K (a magic number for swiotlb since
IO_TLB_SEGSIZE is 128 times a slab size of 2k). Try changing
lib/swiotlb.c to set IO_TLB_SEGSIZE to 256 instead.

-Tony

2007-08-23 21:23:42

by Andrew Morton

[permalink] [raw]

Subject: Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted

On Thu, 23 Aug 2007 10:22:26 -0700
"Luck, Tony" <[email protected]> wrote:

> > __get_free_pages() of swiotlb_alloc_coherent() fails in rc3-mm1.
> > But, it doesn't fail on rc2-mm2, and kernel can boot up.
>
> That looks to be part of the problem here ... failing an order=3
> allocation during boot on a system that just a few lines earlier
> in the boot log reported "Memory: 37474000k/37680640k available"
> looks bad ... but perhaps having *more* memory is part of your problem.
> You may have run low on GFP_DMA memory because some allocation
> scaled by memory size has chewed up a lot of your memory. To check
> this try booting with a "mem=4G" parameter and see if that helps
> you.
>
> But it is also bad that the swiotlb() code failed to handle this.
> Can you check whether the problem is related to the size of the
> allocation being just over 256K (a magic number for swiotlb since
> IO_TLB_SEGSIZE is 128 times a slab size of 2k). Try changing
> lib/swiotlb.c to set IO_TLB_SEGSIZE to 256 instead.
>

Others are reporting machines which fail int he memory allcoator much
earlier, and which claim to have four CPUs and 16 nodes. So something is
very wonky in the rc3-mm1 page allocator.

I guess suspicion has to be directed at the memoryless-nodes patches, but
until that's cleared up I don't think there's much to be gained from
chasing this iommu problem, now that you've worked out that it's a bogus
memory allocation failure (thanks).

2007-08-24 06:55:18

by Yasunori Goto

[permalink] [raw]

Subject: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

I found find_next_best_node() was wrong.
I confirmed boot up by the following patch.
Mel-san, Kamalesh-san, could you try this?

Bye.
---

Fix decision of memoryless node in find_next_best_node().
This can be cause of SW-IOMMU's allocation failure.

This patch is for 2.6.23-rc3-mm1.

Signed-off-by: Yasunori Goto <[email protected]>

---
mm/page_alloc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Index: current/mm/page_alloc.c
===================================================================
--- current.orig/mm/page_alloc.c 2007-08-24 16:03:17.000000000 +0900
+++ current/mm/page_alloc.c 2007-08-24 16:04:06.000000000 +0900
@@ -2136,7 +2136,7 @@ static int find_next_best_node(int node,
* Note: N_HIGH_MEMORY state not guaranteed to be
* populated yet.
*/
- if (pgdat->node_present_pages)
+ if (!pgdat->node_present_pages)
continue;

/* Don't want a node to appear more than once */

--
Yasunori Goto

2007-08-24 14:52:49

by mel

[permalink] [raw]

Subject: Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

On (24/08/07 15:53), Yasunori Goto didst pronounce:
>
> I found find_next_best_node() was wrong.
> I confirmed boot up by the following patch.
> Mel-san, Kamalesh-san, could you try this?
>

This boots the IA-64 successful and gets rid of that DMA corrupts
memory message. As a bonus, it fixes up the memoryless nodes (the bug
where Total pages == 0 and there is a BUG in page_alloc.c) by building
zonelists properly. The machine still fails to boot with the more familiar
net/core/skbuff.c:95 but that is a separate problem.

Well spotted Yasunori-san.

Andrew, this fixes a real problem and should be considered a fix to
memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch unless
Christoph Lameter objects.

> Bye.
> ---
>
> Fix decision of memoryless node in find_next_best_node().
> This can be cause of SW-IOMMU's allocation failure.
>
> This patch is for 2.6.23-rc3-mm1.
>
> Signed-off-by: Yasunori Goto <[email protected]>
>

Acked-by: Mel Gorman <[email protected]>

> ---
> mm/page_alloc.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> Index: current/mm/page_alloc.c
> ===================================================================
> --- current.orig/mm/page_alloc.c 2007-08-24 16:03:17.000000000 +0900
> +++ current/mm/page_alloc.c 2007-08-24 16:04:06.000000000 +0900
> @@ -2136,7 +2136,7 @@ static int find_next_best_node(int node,
> * Note: N_HIGH_MEMORY state not guaranteed to be
> * populated yet.
> */
> - if (pgdat->node_present_pages)
> + if (!pgdat->node_present_pages)
> continue;
>
> /* Don't want a node to appear more than once */
>

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab

2007-08-24 15:50:16

by Lee Schermerhorn

[permalink] [raw]

Subject: Re: [PATCH] Fix find_next_best_node (Re: [BUG] 2.6.23-rc3-mm1 Kernel panic - not syncing: DMA: Memory would be corrupted)

On Fri, 2007-08-24 at 15:52 +0100, Mel Gorman wrote:
> On (24/08/07 15:53), Yasunori Goto didst pronounce:
> >
> > I found find_next_best_node() was wrong.
> > I confirmed boot up by the following patch.
> > Mel-san, Kamalesh-san, could you try this?
> >
>
> This boots the IA-64 successful and gets rid of that DMA corrupts
> memory message. As a bonus, it fixes up the memoryless nodes (the bug
> where Total pages == 0 and there is a BUG in page_alloc.c) by building
> zonelists properly. The machine still fails to boot with the more familiar
> net/core/skbuff.c:95 but that is a separate problem.
>
> Well spotted Yasunori-san.
>
> Andrew, this fixes a real problem and should be considered a fix to
> memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch unless
> Christoph Lameter objects.

I reworked that patch and posted the update on 16aug which does not have
this problem:

http://marc.info/?l=linux-mm&m=118729871101418&w=4

This should replace
memoryless-nodes-fixup-uses-of-node_online_map-in-generic-code.patch
in -mm.

Lee

2007-08-24 16:47:06