Received: by 2002:a05:6a10:206:0:0:0:0 with SMTP id 6csp4806301pxj; Tue, 22 Jun 2021 08:24:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyKUnwMgRn5oSQmF2AzQQ9y7pcyH+uZpaPf/QqMb87/hpTmJ463O5g5kHT8YIUv1stidUsw X-Received: by 2002:a05:6602:1234:: with SMTP id z20mr3321756iot.167.1624375494529; Tue, 22 Jun 2021 08:24:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624375494; cv=none; d=google.com; s=arc-20160816; b=fvPF61GzJBONgxXvpyYIRr0G8irWCz4nht1p/LW2IIABmVioeoxrEaCqwuM2CwXhU3 Ga0ttQO8k/CBcHjZnlGR1t8V156R+YJUtokhdklFaq/OZRFPz1B+sFmRJC11482NiKJM v/s9kKxD6kssH1VGnbGcc+wj01HFvwLyvcGEq0ySHu+ez/Sae1CQMjuSPBW2vaO8jIWl aNlmHCXuhAaJqTmFDZUbrejqnwOnYuoa6nX/iBuXRLH4Sjljw0h5Man/JK1AYcyr7o4P 0itVbuWaRbH3CP4H4MLbKGWb6TXE9foeYNUJ3pNFTj4yGTsxI0weHEaCKNGYeg98tAy1 8CFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-transfer-encoding :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature; bh=r42ocVEmWIA//O7Vp0R5aVeKrTj4f+CjM19TgcLu7yI=; b=Hx7w+IqNuZsSEoMMFkv3nHztp6c63bZYkFM07UYOK4xO1SA7manNdkXUxthBDzjbgK wPl5nW1/FUJyG5a5L2xIN7181Gvspq3hwyo5cNdbniDVRLY4uamY3xWjWT4zaeyQj0Vu ZXnPmhRXW12l3LsFkgU3tKcBErLFb00HxiKcXBx2qcJSaM+nes4EO6pQ7IyuSc4fcwHN qYOq3GNj5flDs7rtba4Iq6+ifkHtZQOk94gvp7JH6v1EigsYC+JcWeTZzXBSd8q70pas ZgXca6s+NpUEXWqhVU2tMm43iZEa/0nqU36qLbUZPNtgiDkCoHHIjKEqnzpOvC7FN4PJ RTKw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=JbYlgSzY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id y7si6158259ily.44.2021.06.22.08.24.41; Tue, 22 Jun 2021 08:24:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b=JbYlgSzY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232182AbhFVP0D (ORCPT + 99 others); Tue, 22 Jun 2021 11:26:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231761AbhFVP0B (ORCPT ); Tue, 22 Jun 2021 11:26:01 -0400 Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7DC23C061756 for ; Tue, 22 Jun 2021 08:23:45 -0700 (PDT) Received: by mail-qv1-xf2d.google.com with SMTP id r19so9409521qvw.5 for ; Tue, 22 Jun 2021 08:23:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=r42ocVEmWIA//O7Vp0R5aVeKrTj4f+CjM19TgcLu7yI=; b=JbYlgSzYmG1Fc66FYJM2c7uKHMNG0BKhfYRhDl/t4HfZIXCTkXkNOXDsHPqa4n2peB ysI6mqNNkHNeAKm8LsjMNdQE2b2zkO/XX6peVaIj0AMNhYUfmW34UNv99J2ObeyW/O4M AlP7HDwJNBY8OgpUgjTTorTmyLUlsVUZnNJdeZZN3fBd9t94MplJeK/FtWbF5wQXcwR5 J/44W0pKR1bz4Y8dMUyaa9V35Pyl3Yzhscasz1N63ZBqjUGbU5HIxyMOZAlZZ1TDg33w laau+inbi3QaWeLqDwOYYzVVpGP9v4S02gOhgQLFCKQspXRAAzDLVW1nl9ri0zrdJ0n3 86Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=r42ocVEmWIA//O7Vp0R5aVeKrTj4f+CjM19TgcLu7yI=; b=XHhv3jB+thoGW6UuPcbmHqDENuyP0pnIzGW2almur/98R58sLqMOKXuKMxlFeD+Vxe yEXuWveFRisXCpT92cnn/+CN2PGcwlZrxCq2x8jJvMXdATUcKvVHav4XzvVf8lj9Jtk2 O9B3BaBhLyT1wXlYlZchwWcavGvDB2Xw3oWO0AAtLM8RUCgRkLbX4H/SWo1JjY0CWKUj x4tHMTbgbhDyDa2rVjdQgqv4NPqhaqDk5v+OAK5hnRPp+ZPjHIl3bEixjvfQACDxE8E9 lyWAQKz88Cp3liogchFHlE3j3c9JrfQQwo8cZ5V+fjbzLOs/CFguUvt/Vl6sH9CkCXkU 612g== X-Gm-Message-State: AOAM530vWyQQ0/4iJ8oE/RGXwNIFsitXP5Z3zgxc5tJzOsXCTpv9qY25 CbknIOfwDQg5gpAo0QZzlNM8HQ== X-Received: by 2002:ad4:56e5:: with SMTP id cr5mr25603185qvb.7.1624375424489; Tue, 22 Jun 2021 08:23:44 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-113-94.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.113.94]) by smtp.gmail.com with ESMTPSA id z3sm13252730qkj.40.2021.06.22.08.23.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 08:23:44 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lviFb-00ADEI-G0; Tue, 22 Jun 2021 12:23:43 -0300 Date: Tue, 22 Jun 2021 12:23:43 -0300 From: Jason Gunthorpe To: Christian =?utf-8?B?S8O2bmln?= Cc: Oded Gabbay , Christian =?utf-8?B?S8O2bmln?= , Gal Pressman , sleybo@amazon.com, linux-rdma , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Message-ID: <20210622152343.GO1096940@ziepe.ca> References: <20210621141217.GE1096940@ziepe.ca> <20210621175511.GI1096940@ziepe.ca> <20210621232912.GK1096940@ziepe.ca> <20210622120142.GL1096940@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jun 22, 2021 at 02:23:03PM +0200, Christian König wrote: > Am 22.06.21 um 14:01 schrieb Jason Gunthorpe: > > On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote: > > > On Tue, Jun 22, 2021 at 9:37 AM Christian König > > > wrote: > > > > Am 22.06.21 um 01:29 schrieb Jason Gunthorpe: > > > > > On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded Gabbay wrote: > > > > > > > > > > > Another thing I want to emphasize is that we are doing p2p only > > > > > > through the export/import of the FD. We do *not* allow the user to > > > > > > mmap the dma-buf as we do not support direct IO. So there is no access > > > > > > to these pages through the userspace. > > > > > Arguably mmaping the memory is a better choice, and is the direction > > > > > that Logan's series goes in. Here the use of DMABUF was specifically > > > > > designed to allow hitless revokation of the memory, which this isn't > > > > > even using. > > > > The major problem with this approach is that DMA-buf is also used for > > > > memory which isn't CPU accessible. > > That isn't an issue here because the memory is only intended to be > > used with P2P transfers so it must be CPU accessible. > > No, especially P2P is often done on memory resources which are not even > remotely CPU accessible. That is a special AMD thing, P2P here is PCI P2P and all PCI memory is CPU accessible. > > > > > So you are taking the hit of very limited hardware support and reduced > > > > > performance just to squeeze into DMABUF.. > > You still have the issue that this patch is doing all of this P2P > > stuff wrong - following the already NAK'd AMD approach. > > Well that stuff was NAKed because we still use sg_tables, not because we > don't want to allocate struct pages. sg lists in general. > The plan is to push this forward since DEVICE_PRIVATE clearly can't handle > all of our use cases and is not really a good fit to be honest. > > IOMMU is now working as well, so as far as I can see we are all good here. How? Is that more AMD special stuff? This patch series never calls to the iommu driver, AFAICT. > > > I'll go and read Logan's patch-set to see if that will work for us in > > > the future. Please remember, as Daniel said, we don't have struct page > > > backing our device memory, so if that is a requirement to connect to > > > Logan's work, then I don't think we will want to do it at this point. > > It is trivial to get the struct page for a PCI BAR. > > Yeah, but it doesn't make much sense. Why should we create a struct page for > something that isn't even memory in a lot of cases? Because the iommu and other places need this handle to setup their stuff. Nobody has yet been brave enough to try to change those flows to be able to use a physical CPU address. This is why we have a special struct page type just for PCI BAR memory. Jason