Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756922AbcCUQBI (ORCPT ); Mon, 21 Mar 2016 12:01:08 -0400 Received: from mail-pf0-f177.google.com ([209.85.192.177]:33536 "EHLO mail-pf0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756592AbcCUP6H (ORCPT ); Mon, 21 Mar 2016 11:58:07 -0400 Subject: Re: Delete flush cache all in arm64 platform. To: Mark Rutland , Chen Feng References: <56EFABD3.7060700@hisilicon.com> <20160321100818.GA17326@leverpostej> Cc: catalin.marinas@arm.com, akpm@linux-foundation.org, mhocko@suse.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, xuyiping@hisilicon.com, suzhuangluan@hisilicon.com, saberlily.xia@hisilicon.com, dan.zhao@hisilicon.com, linux-arm-kernel@lists.infradead.org From: Laura Abbott Message-ID: <56F01A0A.3030208@redhat.com> Date: Mon, 21 Mar 2016 08:58:02 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <20160321100818.GA17326@leverpostej> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3445 Lines: 90 On 03/21/2016 03:08 AM, Mark Rutland wrote: > [adding LAKML] > > On Mon, Mar 21, 2016 at 04:07:47PM +0800, Chen Feng wrote: >> Hi Mark, > > Hi, > >> With 68234df4ea7939f98431aa81113fbdce10c4a84b >> arm64: kill flush_cache_all() >> The documented semantics of flush_cache_all are not possible to provide >> for arm64 (short of flushing the entire physical address space by VA), >> and there are currently no users; KVM uses VA maintenance exclusively, >> cpu_reset is never called, and the only two users outside of arch code >> cannot be built for arm64. >> >> While cpu_soft_reset and related functions (which call flush_cache_all) >> were thought to be useful for kexec, their current implementations only >> serve to mask bugs. For correctness kexec will need to perform >> maintenance by VA anyway to account for system caches, line migration, >> and other subtleties of the cache architecture. As the extent of this >> cache maintenance will be kexec-specific, it should probably live in the >> kexec code. >> >> This patch removes flush_cache_all, and related unused components, >> preventing further abuse. >> >> >> This patch delete the flush_cache_all interface. > > As the patch states, it does so because the documented semantics are > impossible to provide, as there is no portable mechanism to "flush" all > caches in the system. > > Set/Way operations cannot guarantee that data has been cleaned to the > PoC (i.e. memory), or invalidated from all levels of cache. Reasons > include: > > * They may race against background behaviour of the CPU (e.g. > speculation), which may allocate/evict/migrate lines. Depending on the > cache topology, this may "hide" lines from subsequent Set/Way > operations. > > * They are not broadcast, and do not affect other CPUs. Depending on the > implemented cache coherency protocols, other CPUs may be able to > acquire dirty lines, or retain lines in shared states, and hence these > may not be operated on. > > * They do not affect system caches (which respect cache maintenance by > VA in ARMv8-A). > > The only portable mechanism to perform cache maintenance to all relevant > caches is by VA. > >> But if we use VA to flush cache to do cache-coherency with other >> master(eg:gpu) >> >> We must iterate over the sg-list to flush by va to pa. >> >> In this way, the iterate of sg-list may cost too much time(sg-table to >> sg-list) if the sglist is too long. Take a look at the >> ion_pages_sync_for_device in ion. >> >> The driver(eg: ION) need to use this interface(flush cache all) to >> *improve the efficiency*. > > As above, we cannot use Set/Way operations for this, and cannot provide > a flush_cache_all interface. > > I'm not sure what to suggest regarding improving efficiency. > > Is walking the sglist the expensive portion, or is the problem the cost > of multiple page-size operations (each with their own barriers)? > Last time I looked at this, it was mostly the multiple page-size operations. Ion buffers can be big and easily bigger than the cache as well so flushing 8MB of buffer for a 2MB cache is really a performance killer. The Set/Way operations are an improvement on systems where they can be used. The way Ion does cache maintenance is full of sadness and despair in general. Until everything gets a significant rework, the best option may be minimization of code paths where cache operations are called. > Thanks, > Mark. > Thanks, Laura