Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755844Ab3HFAQI (ORCPT ); Mon, 5 Aug 2013 20:16:08 -0400 Received: from mail-pa0-f50.google.com ([209.85.220.50]:50181 "EHLO mail-pa0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755636Ab3HFAQH (ORCPT ); Mon, 5 Aug 2013 20:16:07 -0400 Message-ID: <5200403B.3000206@ozlabs.ru> Date: Tue, 06 Aug 2013 10:15:55 +1000 From: Alexey Kardashevskiy User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130625 Thunderbird/17.0.7 MIME-Version: 1.0 To: Jay Fenlason CC: Santosh Rastapur , Divy Le Ray , "David S. Miller" , netdev@vger.kernel.org, Linux Kernel Mailing List Subject: Re: BUG cxgb3: Check and handle the dma mapping errors References: <51FF14F8.7000204@ozlabs.ru> <20130805184107.GA2998@redhat.com> In-Reply-To: <20130805184107.GA2998@redhat.com> Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3427 Lines: 80 On 08/06/2013 04:41 AM, Jay Fenlason wrote: > On Mon, Aug 05, 2013 at 12:59:04PM +1000, Alexey Kardashevskiy wrote: >> Hi! >> >> Recently I started getting multiple errors like this: >> >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> cxgb3 0006:01:00.0: iommu_alloc failed, tbl c000000003067980 vaddr >> c000001fbdaaa882 npages 1 >> ... and so on >> >> This is all happening on a PPC64 "powernv" platform machine. To trigger the >> error state, it is enough to _flood_ ping CXGB3 card from another machine >> (which has Emulex 10Gb NIC + Cisco switch). Just do "ping -f 172.20.1.2" >> and wait 10-15 seconds. >> >> >> The messages are coming from arch/powerpc/kernel/iommu.c and basically >> mean that the driver requested more pages than the DMA window has which is >> normally 1GB (there could be another possible source of errors - >> ppc_md.tce_build callback - but on powernv platform it always succeeds). >> >> >> The patch after which it broke is: >> commit f83331bab149e29fa2c49cf102c0cd8c3f1ce9f9 >> Author: Santosh Rastapur >> Date: Tue May 21 04:21:29 2013 +0000 >> cxgb3: Check and handle the dma mapping errors >> >> Any quick ideas? Thanks! > > That patch adds error checking to detect failed dma mapping requests. > Before it, the code always assumed that dma mapping requests succeded, > whether they actually do or not, so the fact that the older kernel > does not log errors only means that the failures are being ignored, > and any appearance of working is through pure luck. The machine could > have just crashed at that point. >From what I see, the patch adds map_skb() function which is called in two new places, so the patch does not just mechanically replace skb_frag_dma_map() to map_skb() or something like that. > What is the observed behavior of the system by the machine initiating > the ping flood? Do the older and newer kernels differ in the > percentage of pings that do not receive replies? The other machine stops receiving replies. It is using different adapter, not Chelsio and the kernel version does not really matter. > O the newer kernel, > when the mapping errors are detected, the packet that it is trying to > transmit is dropped, but I'm not at all sure what happens on the older > kernel after the dma mapping fails. As I mentioned earlier, I'm > surprised it does not crash. Perhaps the folks from Chelsio have a > better idea what happens after a dma mapping error is ignored? Any kernel cannot avoid platform's iommu_alloc() on ppc64/powernv so if there was a problem, we would have seen messages (and yes, kernel would have crashed). -- Alexey -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/