Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932149AbbDQIB6 (ORCPT ); Fri, 17 Apr 2015 04:01:58 -0400 Received: from mail-am1on0137.outbound.protection.outlook.com ([157.56.112.137]:29328 "EHLO emea01-am1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752157AbbDQIBs (ORCPT ); Fri, 17 Apr 2015 04:01:48 -0400 X-Greylist: delayed 99734 seconds by postgrey-1.27 at vger.kernel.org; Fri, 17 Apr 2015 04:01:47 EDT Authentication-Results: suse.com; dkim=none (message not signed) header.d=none; Date: Fri, 17 Apr 2015 17:00:46 +0900 From: Hyong-Youb Kim To: "Luis R. Rodriguez" CC: Andy Walls , Hyong-Youb Kim , , Andy Lutomirski , Toshi Kani , "H. Peter Anvin" , Ingo Molnar , , Hal Rosenstock , Sean Hefty , Suresh Siddha , Rickard Strandqvist , Mike Marciniszyn , Roland Dreier , "Juergen Gross" , Mauro Carvalho Chehab , Borislav Petkov , Mel Gorman , Vlastimil Babka , Davidlohr Bueso , Johannes Berg , Felix Fietkau , "Benjamin Poirier" , , , , Ville =?iso-8859-1?Q?Syrj=E4l=E4?= , , , Subject: Re: ioremap_uc() followed by set_memory_wc() - burrying MTRR Message-ID: <20150417080046.GA4620@hykim-PC> References: <20150410210538.GB5622@wotan.suse.de> <1428699490.21794.5.camel@misato.fc.hp.com> <20150411012938.GC5622@wotan.suse.de> <20150413174938.GE5622@wotan.suse.de> <1429137531.1899.28.camel@palomino.walls.org> <20150415235816.GG5622@wotan.suse.de> <20150416041837.GA5712@hykim-PC> <20150416185426.GH5622@wotan.suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20150416185426.GH5622@wotan.suse.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-Originating-IP: [125.132.7.245] X-ClientProxiedBy: SIXPR01CA0014.apcprd01.prod.exchangelabs.com (25.163.105.142) To AM3PR05MB1300.eurprd05.prod.outlook.com (25.163.7.153) X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM3PR05MB1300;UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM3PR05MB1378; X-Microsoft-Antispam-PRVS: X-Forefront-Antispam-Report: BMV:1;SFV:NSPM;SFS:(10019020)(6009001)(24454002)(51704005)(42186005)(97756001)(33656002)(87976001)(83506001)(4001350100001)(23726002)(50196002)(50466002)(66066001)(33716001)(47776003)(77156002)(46102003)(62966003)(46406003)(2950100001)(93886004)(122386002)(110136001)(40100003)(92566002)(54356999)(50986999)(76176999)(7059030)(107986001);DIR:OUT;SFP:1102;SCL:1;SRVR:AM3PR05MB1300;H:hykim-PC;FPR:;SPF:None;MLV:sfv;LANG:en; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(5005006)(5002010);SRVR:AM3PR05MB1300;BCL:0;PCL:0;RULEID:;SRVR:AM3PR05MB1300; X-Forefront-PRVS: 0549E6FD50 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Apr 2015 08:01:30.7016 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR05MB1300 X-OriginatorOrg: cspi.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3153 Lines: 63 On Thu, Apr 16, 2015 at 08:54:26PM +0200, Luis R. Rodriguez wrote: > > Without WC, descriptors would end up as multiple 4B or 8B MWr packets > > to the NIC, which has a pretty big performance impact on this > > particular NIC. > > How big are the descriptors? Some are 64B (a batch of eight 8B descriptors). Some are 16B. > > Most registers that do not want WC are actually in BAR2, which is not > > mapped as WC. For registers that are in BAR0, we do "write to the > > register; wmb". If we want to wait till the NIC has seen the write, > > we do "write; wmb; read". > > Interesting, thanks, yeah using this as a work around to the problem sounds > plausible however it still would require likely making just as many changes to > the ivtv and ipath driver as to just do a proper split. I do wonder however if > this sort of work around can be generalized somehow though so that others could > use, if this sort of thing is going to become prevalent. If so then this would > serve two purposes: work around for the corner cases of MTRR use on Linux and > also these sorts of device constraints. These Myricom devices are very non-standard in my opinion, at least in the Ethernet world. Few, if any, other devices depend so much on WC like these do. I think almost all devices now have rings in host memory. The NIC pulls them via DMA. No need for WC, and no need to special case registers... > In order to determine if this is likely to be generally useful could you elaborate > a bit more about the detals of the performance issues of not bursting writes > for the descriptor on this device. For this particular Myricom device, performance penalty stems from the use of slow path in the firmware. They are not about how effectively we use PCI Express or latency or bandwidth. Small MWr packets end up casuing slow path processing via the firmware in this device. There are HPC low latency NICs that use WC for different reasons. To reduce latency as much as possible, some of these copy small packets to the NIC memory via PIO (BAR0, and so on), instead of DMA. They want WC mapping to minimize PCI Express packets/transactions. I do not know about video adapters and their use for WC. > Even if that is done a conversion over to this work around seems it may require > device specific nitpicks. For instance I note in myri10ge_submit_req() for > small writes you just do a reverse write and do the first set last, then > finally the last 32 bits are rewritten, I guess to trigger something? Right. The device polls the last word to start sending, DMA, etc. > > This approach has worked for this device for many years. I cannot say > > whether it works for other devices, though. > > I think it should but the more interesting question would be exactly > *why* it was needed for this device, who determined that, and why? Hopefully, the above text answers some of your questions. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/