Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp267868imu; Mon, 26 Nov 2018 21:05:51 -0800 (PST) X-Google-Smtp-Source: AJdET5cWWypYf4c+NkD/hGKUo9C6waM5uzRDB/KWSW6apbkGe0AzcVF5e60GBq/YfKNQptIhEoqQ X-Received: by 2002:aa7:85d7:: with SMTP id z23mr32432593pfn.205.1543295151709; Mon, 26 Nov 2018 21:05:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543295151; cv=none; d=google.com; s=arc-20160816; b=zlvpDFiz3ZIrGGj1ho5so5rlYqHaNy+Fqa3Khz2KFHgOlyoPwd0rE1Rz/Bc2zFjjRi 1bbklSIfbdFNgS2VYtP6m5U61ZaKKYAnIYMyAE6ggjlQhLfS0KmJR0zsHa0SkVIIbKlP /1dGmoqTAvLovoJh4wK3gWdJ1sSn7nyJKA+LumqO9dIdwlVsGJKuOaT9htIIbJXTehsL yhAGyw4fzIKuSjZumEKX4ZtCq0GeXn4HPkrikYfUS7Hw/jEXsj8/jk+F3vD9vT4diTQg Bg2lhXwkfIJ3O/6AWcDbaA0XxxCLMq3+HjMEaAUXmctXdghXpytK8hHOFkzmt/BnK/fh 1WBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:user-agent:references :message-id:in-reply-to:subject:cc:to:from:date:dmarc-filter :dkim-signature:dkim-signature; bh=6RT0hDgCSaOsoxUKf4Yu1nk5hl4v/VjKA/wmxBJbCKY=; b=s5IUCK49Gi6SLFy4VGi4hVsX+B/kyYV4YLntu76+J1v1UmIg3SqCfQ/cdL0uGVFs7J XfGKV8gLeGIplKajiFAtv755QZcFxBFOnd1dd11Q0d+RPy5KV+DZ1Ncg+Jm/dXT0GaVv LibeZdzFoEJcj6u8y6p/ieS/O6wQjCB7IYRLfpMy/s3nRcG2+c9c5fg8o1drW0BUzIlW m3bmA9JwckEkzcMPgn4zf86EdPsnGKBeeQ9Obf4xm6NlRu/8ap4OdbTxyvk4mrffldfV 3KBai6uj3disLxQrPDfcc2BpozUw/q+kc5sKRHiUaiFBKUVTRuJQBRvQ7WYK3jas8pnt 5Oqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@codeaurora.org header.s=default header.b=NCigEd2F; dkim=pass header.i=@codeaurora.org header.s=default header.b=CLFOQx0Z; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 4si2761246pff.161.2018.11.26.21.05.35; Mon, 26 Nov 2018 21:05:51 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@codeaurora.org header.s=default header.b=NCigEd2F; dkim=pass header.i=@codeaurora.org header.s=default header.b=CLFOQx0Z; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728285AbeK0P4Z (ORCPT + 99 others); Tue, 27 Nov 2018 10:56:25 -0500 Received: from smtp.codeaurora.org ([198.145.29.96]:51744 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727974AbeK0P4Y (ORCPT ); Tue, 27 Nov 2018 10:56:24 -0500 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id 8D64660B11; Tue, 27 Nov 2018 04:59:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1543294786; bh=cT/rEbtk10Dp/3bm/aMhSLIrwvlPlOdQNuo0EV7OZuI=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=NCigEd2FdE+PqrQMZUUs1FjgKobO7NRRvLmE2A8HWyv8IqZps/Q/o+hRFkkAecrow WqmUB+FGFsQrvgd/6LddjKHRlCRl0rO7frWsqeGMkJ4/k1nOzGu6PCSYc2ntvJ7Hw9 Hnu8TxoLcUC8pjCBUupxYcfiTu3Dxa3mFOOtR1uE= X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on pdx-caf-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=ALL_TRUSTED,BAYES_00, DKIM_INVALID,DKIM_SIGNED autolearn=no autolearn_force=no version=3.4.0 Received: from lmark-linux.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: lmark@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id EE7E76060A; Tue, 27 Nov 2018 04:59:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1543294785; bh=cT/rEbtk10Dp/3bm/aMhSLIrwvlPlOdQNuo0EV7OZuI=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=CLFOQx0Z9kqrNPxixXlcGnhzrN3VOM3CAqXb+TtgjubeofFvv8udT7ZVg+F1ZCEiR Ga4k/beHG1AgLNn+k59b/7Q548QXk7SYDbFYrmNy2UhkQsB0GL6RLPjcBsDF6ivU7r 9uVw4rEP+sVFpSi4vO/YFo9FLZk0M/4w4L51YzZA= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org EE7E76060A Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=lmark@codeaurora.org Date: Mon, 26 Nov 2018 20:59:44 -0800 (PST) From: Liam Mark X-X-Sender: lmark@lmark-linux.qualcomm.com To: Brian Starkey cc: nd , Sumit Semwal , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, devel@driverdev.osuosl.org, Martijn Coenen , dri-devel , John Stultz , Todd Kjos , Arve Hjonnevag , linaro-mm-sig@lists.linaro.org, Laura Abbott Subject: Re: [RFC PATCH v2] android: ion: How to properly clean caches for uncached allocations In-Reply-To: <20181120164636.jcw7li2uaa3cmwc3@DESKTOP-E1NTVVP.localdomain> Message-ID: References: <20181120164636.jcw7li2uaa3cmwc3@DESKTOP-E1NTVVP.localdomain> User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 20 Nov 2018, Brian Starkey wrote: > Hi Liam, > > I'm missing a bit of context here, but I did read the v1 thread. > Please accept my apologies if I'm re-treading trodden ground. > > I do know we're chasing nebulous ion "problems" on our end, which > certainly seem to be related to what you're trying to fix here. > > On Thu, Nov 01, 2018 at 03:15:06PM -0700, Liam Mark wrote: > >Based on the suggestions from Laura I created a first draft for a change > >which will attempt to ensure that uncached mappings are only applied to > >ION memory who's cache lines have been cleaned. > >It does this by providing cached mappings (for uncached ION allocations) > >until the ION buffer is dma mapped and successfully cleaned, then it > drops > >the userspace mappings and when pages are accessed they are faulted back > >in and uncached mappings are created. > > If I understand right, there's no way to portably clean the cache of > the kernel mapping before we map the pages into userspace. Is that > right? > Yes, it isn't always possible to clean the caches for an uncached mapping because a device is required by the DMA APIs to do cache maintenance and there isn't necessarily a device available (dma_buf_attach may not yet have been called). > Alternatively, can we just make ion refuse to give userspace a > non-cached mapping for pages which are mapped in the kernel as cached? These pages will all be mapped as cached in the kernel for 64 bit (kernel logical addresses) so you would always be refusing to create a non-cached mapping. > Would userspace using the dma-buf sync ioctl around its accesses do > the "right thing" in that case? > I don't think so, the dma-buf sync ioctl require a device to peform cache maintenance, but as mentioned above a device may not be available. > Given that as you pointed out, the kernel does still have a cached > mapping to these pages, trying to give the CPU a non-cached mapping of > those same pages while preserving consistency seems fraught. Wouldn't > it be better to make sure all CPU mappings are cached, and have CPU > clients use the dma_buf_{begin,end}_cpu_access() hooks to get > consistency where needed? > It is fraught, but unfortunately you can't rely on dma_buf_{begin,end}_cpu_access() to do cache maintenance as these calls require a device, and a device is not always available. > > > >This change has the following potential disadvantages: > >- It assumes that userpace clients won't attempt to access the buffer > >while it is being mapped as we are removing the userpspace mappings at > >this point (though it is okay for them to have it mapped) > >- It assumes that kernel clients won't hold a kernel mapping to the > buffer > >(ie dma_buf_kmap) while it is being dma-mapped. What should we do if > there > >is a kernel mapping at the time of dma mapping, fail the mapping, warn? > >- There may be a performance penalty as a result of having to fault in > the > >pages after removing the userspace mappings. > > I wonder if the dma-buf sync ioctl might provide a way for userspace > to opt-in to when the zap/fault happens. Zap on (DMA_BUF_SYNC_WRITE | > DMA_BUF_SYNC_WRITE_END) and fault on (DMA_BUF_SYNC_READ | > DMA_BUF_SYNC_START) > Not sure I understand, can you elaborate. Are you also adding a requirment that ION pages can't be mmaped during a call to dma_buf_map_attachment? > > > >It passes basic testing involving reading writing and reading from > >uncached system heap allocations before and after dma mapping. > > > >Please let me know if this is heading in the right direction and if there > >are any concerns. > > > >Signed-off-by: Liam Mark > >--- > > drivers/staging/android/ion/ion.c | 146 > +++++++++++++++++++++++++++++++++++++- > > drivers/staging/android/ion/ion.h | 9 +++ > > 2 files changed, 152 insertions(+), 3 deletions(-) > > > >diff --git a/drivers/staging/android/ion/ion.c > b/drivers/staging/android/ion/ion.c > >index 99073325b0c0..3dc0f5a265bf 100644 > >--- a/drivers/staging/android/ion/ion.c > >+++ b/drivers/staging/android/ion/ion.c > >@@ -96,6 +96,7 @@ static struct ion_buffer *ion_buffer_create(struct > ion_heap *heap, > > } > > > > INIT_LIST_HEAD(&buffer->attachments); > >+ INIT_LIST_HEAD(&buffer->vmas); > > mutex_init(&buffer->lock); > > mutex_lock(&dev->buffer_lock); > > ion_buffer_add(dev, buffer); > >@@ -117,6 +118,7 @@ void ion_buffer_destroy(struct ion_buffer *buffer) > > buffer->heap->ops->unmap_kernel(buffer->heap, buffer); > > } > > buffer->heap->ops->free(buffer); > >+ vfree(buffer->pages); > > kfree(buffer); > > } > > > >@@ -245,11 +247,29 @@ static void ion_dma_buf_detatch(struct dma_buf > *dmabuf, > > kfree(a); > > } > > > >+static bool ion_buffer_uncached_clean(struct ion_buffer *buffer) > >+{ > >+ return buffer->uncached_clean; > >+} > > nit: The function name sounds like a verb to me - as in "calling this > will clean the buffer". I feel ion_buffer_is_uncached_clean() would > read better. > Yes, that would be cleaner. Liam Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project