Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965436AbbDVPXk (ORCPT ); Wed, 22 Apr 2015 11:23:40 -0400 Received: from cantor2.suse.de ([195.135.220.15]:49547 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932888AbbDVPXe (ORCPT ); Wed, 22 Apr 2015 11:23:34 -0400 Date: Wed, 22 Apr 2015 17:23:28 +0200 From: "Luis R. Rodriguez" To: Jason Gunthorpe Cc: Andy Lutomirski , mike.marciniszyn@intel.com, infinipath@intel.com, linux-rdma@vger.kernel.org, awalls@md.metrocast.net, Toshi Kani , "H. Peter Anvin" , Ingo Molnar , "linux-kernel@vger.kernel.org" , Hal Rosenstock , Sean Hefty , Suresh Siddha , Rickard Strandqvist , Roland Dreier , Juergen Gross , Mauro Carvalho Chehab , Borislav Petkov , Mel Gorman , Vlastimil Babka , Davidlohr Bueso , Dave Hansen , Jean-Christophe Plagniol-Villard , Thomas Gleixner , Ville Syrj?l? , Linux Fbdev development list , linux-media@vger.kernel.org, X86 ML , mcgrof@do-not-panic.com Subject: Re: ioremap_uc() followed by set_memory_wc() - burrying MTRR Message-ID: <20150422152328.GB5622@wotan.suse.de> References: <20150421224601.GY5622@wotan.suse.de> <20150421225732.GA17356@obsidianresearch.com> <20150421233907.GA5622@wotan.suse.de> <20150422053939.GA29609@obsidianresearch.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150422053939.GA29609@obsidianresearch.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3604 Lines: 67 On Tue, Apr 21, 2015 at 11:39:39PM -0600, Jason Gunthorpe wrote: > On Wed, Apr 22, 2015 at 01:39:07AM +0200, Luis R. Rodriguez wrote: > > > Mike, do you think the time is right to just remove the iPath driver? > > > > With PAT now being default the driver effectively won't work > > with write-combining on modern kernels. Even if systems are old > > they likely had PAT support, when upgrading kernels PAT will work > > but write-combing won't on ipath. > > Sorry, do you mean the driver already doesn't get WC? Or do you mean > after some more pending patches are applied? No, you have to consider the system used and the effects of calls used on the driver in light of this table: ---------------------------------------------------------------------- MTRR Non-PAT PAT Linux ioremap value Effective memory type ---------------------------------------------------------------------- Non-PAT | PAT PAT |PCD ||PWT ||| WC 000 WB _PAGE_CACHE_MODE_WB WC | WC WC 001 WC _PAGE_CACHE_MODE_WC WC* | WC WC 010 UC- _PAGE_CACHE_MODE_UC_MINUS WC* | UC WC 011 UC _PAGE_CACHE_MODE_UC UC | UC ---------------------------------------------------------------------- (*) denotes implementation defined and is discouraged ioremap_nocache() will use _PAGE_CACHE_MODE_UC_MINUS by default today, in the future we want to flip the switch and make _PAGE_CACHE_MODE_UC the default. When that flip occurs it will mean ipath cannot get write-combining on both non-PAT and PAT systems. Now that is for the future, lets review the current situation for ipath. For PAT capable systems if mtrr_add() is used today on a Linux system on a region mapped with ioremap_nocache() that will mean you effectively nullify the mtrr_add() effect as the combinatorial effect above yields an effective memory type of UC. For PAT systems you want to use ioremap_wc() on the region in which you need write-combining followed by arch_phys_wc_add() which will *only* call mtrr_add() *iff* PAT was not enabled. This also means we need to split the ioremap'd areas so that the area that is using ioremap_nocache() can never get write-combining (_PAGE_CACHE_MODE_UC). The ipath driver needs the regions split just as was done for the qib driver. Now we could just say that leaving things as-is is a non-issue if you are OK with non-write-combining effects being the default behaviour left on the ipath driver for PAT systems. In that case we can just use arch_phys_wc_add() on the driver and while it won't trigger the mtrr_add() on PAT systems it sill won't have any effect. We just typically don't want to see use of ioremap_nocache() paired with arch_phys_wc_add(), grammatically the correct thing to do is pair ioremap_wc() areas with a arch_phys_wc_add() to make the write-combining effects on non-PAT systems. If the ipath driver is not going to get he work required to split the regions though perhaps we can live with a corner case driver that annotates PAT must be disabled on the systems that use it and convert it to arch_phys_wc_add() to just help with phasing out of direct use of mtrr_add(). With this strategy if and when ipath driver gets a split done it would gain WC on both PAT and non-PAT. Luis -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/