Return-Path: linux-nfs-owner@vger.kernel.org Received: from userp1040.oracle.com ([156.151.31.81]:32263 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750810AbaEAFQh (ORCPT ); Thu, 1 May 2014 01:16:37 -0400 Message-ID: <5361D89F.4070503@oracle.com> Date: Wed, 30 Apr 2014 22:16:15 -0700 From: Shirley Ma MIME-Version: 1.0 To: Doug Ledford CC: yanb , Phil Cayton , Susan K Coulter , Anna Schumaker , "Devesh.Sharma" , Steve Wise , Allen Andrews , linux-rdma , Linux NFS Mailing List , Or Gerlitz , Chuck Lever Subject: Re: NFSoRDMA developers bi-weekly meeting announcement (4/30) References: <53614C03.80606@oracle.com> <536161D0.3000600@oracle.com> <30686627.1636.1398902288314.JavaMail."Doug Ledford"@Phenom> In-Reply-To: <30686627.1636.1398902288314.JavaMail."Doug Ledford"@Phenom> Content-Type: text/plain; charset=UTF-8; format=flowed Sender: linux-nfs-owner@vger.kernel.org List-ID: On 04/30/2014 04:58 PM, Doug Ledford wrote: > On 04/302014 Shirley Ma wrote: >> On 04/30/2014 01:00 PM, Or Gerlitz wrote: >>> On Wed, Apr 30, 2014 at 10:47 PM, Chuck Lever >>> >>> >>>> If I understood Yan, he is trying to use NFS/RDMA in guests >>>> (kvm?). We >>>> are pretty sure that is not working at the moment, >>> can you provide a short 1-2 liner why/what is broken there? the >>> only >>> thing which I can think of to be not-supported over mlx4 VFs is the >>> proprietary FMRs, but AFAIK, the nfs-rdma code doesn't even have a >>> mode which uses them, right? >> I've created Xen guest on DomU. Dom0 PF works which has no mtts been >> enabled, however DomU I hit this problem by just mounting the file >> system: >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 66 pages(order 7) >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order >> 12) >> mlx4_core 0000:00:04.0: Failed to allocate mtts for 4096 pages(order >> 12) >> >> RDMA microbenchmark perftest works ok. I enabled mtts scripts when >> booting the Xen guest. cat /proc/mtrr: > What OS/RDMA stack are you using? I'm not familiar with any mtts > scripts, however I know there is an mtrr fixup script I wrote for > the RDMA stack in Fedora/RHEL (and so I assume it's in Oracle Linux > too, but I haven't checked). In fact, I assume that's the script > you are referring to based on the fact that your next bit of your > email cats the /proc/mtrr file. But I don't believe whether there > is an mtrr setting mixup or not that is should have any impact on > the mtts allocations in the driver. Even if your mtrr registers > were set incorrectly, the problem then becomes either A) a serious > performance bottleneck (in the case of Intel hardware that needs > write combining in order to get more than about 50MByte/s of > throughput on their cards) or B) failed operation because MMIO > writes to the card are being cached/write combined when they should > not be. > > I suspect this is more likely Xen related than mtts/mtrr related. Yes. That's the script I used. I wonder whether it's possible to disable mtrr on DomU guest to debug this. I am new to Xen. >> [root@ca-nfsdev1vm1 log]# cat /proc/mtrr >> reg00: base=0x0f0000000 ( 3840MB), size= 128MB, count=1: uncachable >> reg01: base=0x0f8000000 ( 3968MB), size= 64MB, count=1: uncachable >> >> lspci -v >> 00:04.0 InfiniBand: Mellanox Technologies MT25400 Family [ConnectX-2 >> Virtual Function] (rev b0) >> Subsystem: Mellanox Technologies Device 61b0 >> Physical Slot: 4 >> Flags: bus master, fast devsel, latency 0 >> Memory at f0000000 (64-bit, prefetchable) [size=128M] >> Capabilities: [60] Express Endpoint, MSI 00 >> Capabilities: [9c] MSI-X: Enable+ Count=4 Masked- >> Kernel driver in use: mlx4_core >> Kernel modules: mlx4_core >> >> I will need to find another machine to try KVM guest. Yan might hit a >> different problem. >> >> I have ConnectX-2, FW level is 2.11.2012. Yan has ConnectX-3, he >> tried >> it on KVM guest. >>>> but that is a priority >>>> to get fixed. Shirley has a lab set up and has been looking into >>>> it. >> Shirley >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-rdma" >> in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >>