Date: Mon, 6 Aug 2012 10:07:06 -0400
From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Lukas Hejtmanek <xhejtman@ics.muni.cz>
Cc: yinghai@kernel.org, linux-pci@vger.kernel.org, roland@kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: mellanox mlx4_core and SR-IOV
Message-ID: <20120806140706.GC3093@phenom.dumpdata.com>
References: <9f903175-4080-4016-b9f8-83e8615c110e@default>
 <20120805080500.GH25979@ics.muni.cz>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20120805080500.GH25979@ics.muni.cz>
User-Agent: Mutt/1.5.21 (2010-09-15)
Content-Transfer-Encoding: 8BIT
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1930
Lines: 41

On Sun, Aug 05, 2012 at 10:05:00AM +0200, Lukas Hejtmanek wrote:
> Hi,
> 
> On Fri, Aug 03, 2012 at 06:49:59AM -0700, Konrad Wilk wrote:
> > This looks like you are using PV PCI passthrough? If so, did you
> > remember to use 'iommu=soft' to enable the Xen-SWIOTLB in your guest?
> > And are you booting with more than 4GB? Or is less than 3GB (so that you have
> > a nice gap in E820).
> 
> good catch. I forgot to pass swiotl=force for DomU in Xen. So now, it seems
> that mlx4_core works, mlx4_en (ethernet part) works as well. Unfortunately,
> the IB part does not. IB layer complains that SR-IOV is currently unsupported
> (kernel 3.5.0). So no luck here so far. 

Don't use swiotlb=force. That is for the old style kernels. Use iommu=soft.
> 
> There is OFED stack directly from Mellanox, that seems to support SR-IOV even
> for IB layer, but they have buildable sources only for RHEL/SLES kernels
> (2.6.32) and even correcting the sources to get it compile with 3.5.0 does not
> make things work. The driver complains about interrupts not working in Dom0 or
> even without Xen hypervisor at all.

So there is a bug that .. well, I thought I had fixed it with the
IB layer but maybe not. It was about VM_IO having to be used on the vmaps
being setup. But I can't recall the details. Perhaps the InfiniBand mailing
list might have some ... ah here it is:
http://old-list-archives.xen.org/archives/html/xen-devel/2011-01/msg00246.html
> 
> The only good point is, that I managed to convice Supermicro (board
> manufacturer), that enabling SR-IOV in BIOS leads to BIOS lockup, they
> confirmed it and maybe they provide BIOS upgrade.
> 
> Thanks all.
> 
> -- 
> Lukáš Hejtmánek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/