Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751789AbdG1Ct7 (ORCPT ); Thu, 27 Jul 2017 22:49:59 -0400 Received: from szxga02-in.huawei.com ([45.249.212.188]:10791 "EHLO szxga02-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751738AbdG1Ct5 (ORCPT ); Thu, 27 Jul 2017 22:49:57 -0400 Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported To: Casey Leedom , Alexander Duyck References: <1499955692-26556-1-git-send-email-dingtianhong@huawei.com> <1499955692-26556-3-git-send-email-dingtianhong@huawei.com> <0260e398-bd8e-6615-6d5c-1f7c07b6fc09@huawei.com> <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <2feff1a5-25b7-eb6c-7628-7c2257c56e95@huawei.com> CC: Netdev , Bjorn Helgaas , "linux-arm-kernel@lists.infradead.org" , "David.Laight@aculab.com" , "ashok.raj@intel.com" , "Alex Williamson" , "l.stach@pengutronix.de" , "Suravee.Suthikulpanit@amd.com" , "catalin.marinas@arm.com" , "linux-pci@vger.kernel.org" , "will.deacon@arm.com" , Sinan Kaya , "robin.murphy@arm.com" , "linux-kernel@vger.kernel.org" , "davem@davemloft.net" , Ganesh GR , "asit.k.mallick@intel.com" , "jeffrey.t.kirsher@intel.com" , "mark.rutland@arm.com" , "gabriele.paoloni@huawei.com" , Michael Werner , "bhelgaas@google.com" , "patrick.j.cramer@intel.com" , "linuxarm@huawei.com" , "amira@mellanox.com" , "Bob.Shaw@amd.com" From: Ding Tianhong Message-ID: <52e7c477-2f26-16d2-79f7-5f6baba60685@huawei.com> Date: Fri, 28 Jul 2017 10:48:59 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.177.23.32] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090204.597AA639.004C,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 2a842aec291d72bd811839de189ab591 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4835 Lines: 108 On 2017/7/28 1:44, Casey Leedom wrote: > | From: Ding Tianhong > | Sent: Wednesday, July 26, 2017 6:01 PM > | > | On 2017/7/27 3:05, Casey Leedom wrote: > | > > | > Ding, send me a note if you'd like me to work that [cxgb4vf patch] up > | > for you. > | > | Ok, you could send the change log and I could put it in the v8 version > | together, will you base on the patch 3/3 or build a independence patch? > > Which ever you'd prefer. It would basically mirror the same exact code that > you've got for cxgb4. I.e. testing the setting of the VF's PCIe Capability > Device Control[Relaxed Ordering Enable], setting a new flag in > adpater->flags, testing that flag in cxgb4vf/sge.c:t4vf_sge_alloc_rxq(). > But since the VF's PF will already have disabled the PF's Relaxed Ordering > Enable, the VF will also have it's Relaxed Ordering Enable disabled and any > effort by the internal chip to send TLPs with the Relaxed Ordering Attribute > will be gated by the PCIe logic. So it's not critical that this be in the > first patch. Your call. Let me know if you'd like me to send that to you. > Good， please Send it to me, I will put it together and send the v8 this week, I think Bjorn will be back next week .:) > > | From: Ding Tianhong > | Sent: Wednesday, July 26, 2017 6:08 PM > | > | On 2017/7/27 2:26, Casey Leedom wrote: > | > > | > 1. Did we ever get any acknowledgement from either Intel or AMD > | > on this patch? I know that we can't ensure that, but it sure would > | > be nice since the PCI Quirks that we're putting in affect their > | > products. > | > | Still no Intel and AMD guys has ack this, this is what I am worried about, > | should I ping some man again ? > > By amusing coincidence, Patrik Cramer (now Cc'ed) from Intel sent me a note > yesterday with a link to the official Intel performance tuning documentation > which covers this issue: > > https://software.intel.com/sites/default/files/managed/9e/bc/64-ia-32-architectures-optimization-manual.pdf > > In section 3.9.1 we have: > > 3.9.1 Optimizing PCIe Performance for Accesses Toward Coherent Memory > and Toward MMIO Regions (P2P) > > In order to maximize performance for PCIe devices in the processors > listed in Table 3-6 below, the soft- ware should determine whether the > accesses are toward coherent memory (system memory) or toward MMIO > regions (P2P access to other devices). If the access is toward MMIO > region, then software can command HW to set the RO bit in the TLP > header, as this would allow hardware to achieve maximum throughput for > these types of accesses. For accesses toward coherent memory, software > can command HW to clear the RO bit in the TLP header (no RO), as this > would allow hardware to achieve maximum throughput for these types of > accesses. > > Table 3-6. Intel Processor CPU RP Device IDs for Processors Optimizing > PCIe Performance > > Processor CPU RP Device IDs > > Intel Xeon processors based on 6F01H-6F0EH > Broadwell microarchitecture > > Intel Xeon processors based on 2F01H-2F0EH > Haswell microarchitecture > > Unfortunately that's a pretty thin section. But it does expand the set of > Intel Root Complexes for which our Linux PCI Quirk will need to cover. So > you should add those to the next (and hopefully final) spin of your patch. > And, it also verifies the need to handle the use of Relaxed Ordering more > subtlely than simply turning it off since the NVMe peer-to-peer example I > keep bringing up would fall into the "need to use Relaxed Ordering" case ... > > It would have been nice to know why this is happening and if any future > processor would fix this. After all, Relaxed Ordering, is just supposed to > be a hint. At worst, a receiving device could just ignore the attribute > entirely. Obviously someone made an effort to implement it but ... it > didn't go the way they wanted. > > And, it also would have been nice to know if there was any hidden register > in these Intel Root Complexes which can completely turn off the effort to > pay attention to the Relaxed Ordering Attribute. We've spend an enormous > amount of effort on this issue here on the Linux PCI email list struggling > mightily to come up with a way to determine when it's > safe/recommended/not-recommended/unsafe to use Relaxed Ordering when > directing TLPs towards the Root Complex. And some architectures require RO > for decent performance so we can't just "turn it off" unilatterally. > I am glad to hear that more person were focus on this problem, It would be great if they could enter our discussion and give us more suggestion. :) Thanks Ding > Casey > > . >