Received: by 2002:a05:6a10:1d13:0:0:0:0 with SMTP id pp19csp2998389pxb; Tue, 24 Aug 2021 12:32:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx+KoqLyEqMdP559LOxeLiUIXkyM6PNZWu9NnkJap8p/3NpqMa7NwvHm3clxSGByMBEF555 X-Received: by 2002:a5d:9693:: with SMTP id m19mr20492636ion.72.1629833551584; Tue, 24 Aug 2021 12:32:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629833551; cv=none; d=google.com; s=arc-20160816; b=wZAZPh3mWyNosfIWvNQmOLMK7xufuk47bcyQ333ewllOCVW8HFSC/zXsFJ7/M6/npx csBD7W3BHaL8OXjyZM3TKe6YDr7RTGQOskMagu1CxfXttPerpcYfZYcHu7CVVPv/Nv9M ZEFEqbbAeEc4a2eJB/As9mHXHbLm3bJGgG1f4a3q8o0I6YA6XBEtZRTXzCDBKbpeVKj8 El9TV2D+Z3PA4y521HaaQB0oyUgXgIV83iHPlW4UuyZRblL9l1fPLNRqzFZNBz80njqY fLuznpRbJnreDRUf3w9NpKCzwW9XFS0Q4Oa6dHvQosO6BHRzxHxjAcCV4QNwCMBx1ajH 9Djg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=nISK/DnSMLsPKL33eQfrjbTBgFhp/AOhrSTXl8tgvVY=; b=WQsvm16IOlQc4SLnZnC38QEd05teff3noMSjFLtmF/lgqHNZa9RpacjqDq662W3XKT EyI13FZpAqdWrPtUTGxJwCvr8fV7OTmyJAOeEKMgOU90XDQr+jbpueQ0lublvhFhud3h 9FRZfk4uIjiHJPH3f37V/ZkZqN3M2ck6kcarn1yDICFXLF1rndJmgH1bL0ws+m9A9I2i lWgJwmZD4N9TRGfHHbAh43l/y5Cjjc0q9eEbvxn3v+4ssUKrLoj4+0dFszM6psHnIOUS DARCAtkSSYWTUOosIYDNKhORKvLC4ahyVVQizSDarf7DTo92ZMt4RFkU3JFIvvuhqi9E 69GQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b="BBZ/DdRg"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id u4si16522350jak.106.2021.08.24.12.32.18; Tue, 24 Aug 2021 12:32:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@ziepe.ca header.s=google header.b="BBZ/DdRg"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234287AbhHXTbk (ORCPT + 99 others); Tue, 24 Aug 2021 15:31:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229913AbhHXTbj (ORCPT ); Tue, 24 Aug 2021 15:31:39 -0400 Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBF95C061757 for ; Tue, 24 Aug 2021 12:30:54 -0700 (PDT) Received: by mail-qt1-x835.google.com with SMTP id t9so9737819qtp.2 for ; Tue, 24 Aug 2021 12:30:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=nISK/DnSMLsPKL33eQfrjbTBgFhp/AOhrSTXl8tgvVY=; b=BBZ/DdRgt5H9lez5WzLUWLJI8CSm49bjqG/ijHPqdU//5r21+B/V9LDmau1ile99vj ihXEXykNbCWxBdSt1o+/OBIH3V6JByVsYZzFSIvZZA9pAuP1aGsZD1dx/JOQtpO2F070 xkmZBbOZYa2jF9TmJTfo46ih/34THHx3HNpZ200z1wfuD2NUWKDUtHpXhSoZiVJ79CK4 WIqvRnCwVSMZ5qqvSTkaScBbjOjAUluNvFDFs3ao3V2i0DYaiCk5S344s1rEeQWfzGoH 0kL2BPZBmWJzrZdjhRM7ZGEbilknKYgR2dlZRPL8wNh4i3BfXwj7Z6qyQkxhunknYZvn 2hdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=nISK/DnSMLsPKL33eQfrjbTBgFhp/AOhrSTXl8tgvVY=; b=WnpJq1/pqa0rvROM7OCJ4MIk7DFn5b2/rcS171corI/4TV1v3ogAVTGn+JCEmbGSzz vNfDCDw9PUf8mVwCfsDlEhH1j5gNL1ZfJm1N+wzcDRdVSFV2IMefgeO7fUe4dTUj3R1i GKO9t93kddoRuaZWgLvq7ISPuXSkF5o/c8YzhY4QjWy9tAzhj65SS+ywhSlAe13KNRz9 2ueyKMe82fLugqVleutRusfZ7tLkrOsBBt+GOZQx30V1RjWdJ6aAsLMZw1UaGTT9r83x HtDV46zeeqCQAloAkzZncIp7maBtURIWQOCybDgmLJQV1zJc43b0bKceai8dT2YOoOXB yYNA== X-Gm-Message-State: AOAM5332CB2kZ2vjfigu6Q7S+UbxN09RjFYjhzikY0Y/lTLx1Dq0fbyL 5ulhr3Cy5cyv6GlPkJ5hMFlgIw== X-Received: by 2002:ac8:dc9:: with SMTP id t9mr35660938qti.293.1629833453961; Tue, 24 Aug 2021 12:30:53 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id q22sm6382139qtr.95.2021.08.24.12.30.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 12:30:53 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1mIc8K-004ZMK-M2; Tue, 24 Aug 2021 16:30:52 -0300 Date: Tue, 24 Aug 2021 16:30:52 -0300 From: Jason Gunthorpe To: Dave Airlie Cc: John Hubbard , Christian =?utf-8?B?S8O2bmln?= , Gal Pressman , Daniel Vetter , Sumit Semwal , Doug Ledford , "open list:DMA BUFFER SHARING FRAMEWORK" , dri-devel , Linux Kernel Mailing List , linux-rdma , Oded Gabbay , Tomer Tayar , Yossi Leybovich , Alexander Matushevsky , Leon Romanovsky , Jianxin Xiong Subject: Re: [RFC] Make use of non-dynamic dmabuf in RDMA Message-ID: <20210824193052.GF543798@ziepe.ca> References: <0fc94ac0-2bb9-4835-62b8-ea14f85fe512@amazon.com> <20210820143248.GX543798@ziepe.ca> <6b819064-feda-b70b-ea69-eb0a4fca6c0c@amd.com> <20210824173228.GE543798@ziepe.ca> <1d1bd2d0-f467-4808-632b-1cca1174cfd9@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 25, 2021 at 05:15:52AM +1000, Dave Airlie wrote: > On Wed, 25 Aug 2021 at 03:36, John Hubbard wrote: > > > > On 8/24/21 10:32 AM, Jason Gunthorpe wrote: > > ... > > >>> And yes at least for the amdgpu driver we migrate the memory to host > > >>> memory as soon as it is pinned and I would expect that other GPU drivers > > >>> do something similar. > > >> > > >> Well...for many topologies, migrating to host memory will result in a > > >> dramatically slower p2p setup. For that reason, some GPU drivers may > > >> want to allow pinning of video memory in some situations. > > >> > > >> Ideally, you've got modern ODP devices and you don't even need to pin. > > >> But if not, and you still hope to do high performance p2p between a GPU > > >> and a non-ODP Infiniband device, then you would need to leave the pinned > > >> memory in vidmem. > > >> > > >> So I think we don't want to rule out that behavior, right? Or is the > > >> thinking more like, "you're lucky that this old non-ODP setup works at > > >> all, and we'll make it work by routing through host/cpu memory, but it > > >> will be slow"? > > > > > > I think it depends on the user, if the user creates memory which is > > > permanently located on the GPU then it should be pinnable in this way > > > without force migration. But if the memory is inherently migratable > > > then it just cannot be pinned in the GPU at all as we can't > > > indefinately block migration from happening eg if the CPU touches it > > > later or something. > > > > > > > OK. I just want to avoid creating any API-level assumptions that dma_buf_pin() > > necessarily implies or requires migrating to host memory. > > I'm not sure we should be allowing dma_buf_pin at all on > non-migratable memory, what's to stop someone just pinning all the > VRAM and making the GPU unuseable? IMHO the same thinking that prevents pining all of system ram and making the system unusable? GPU isn't so special here. The main restriction is the pinned memory ulimit. For most out-of-the-box cases this is set to something like 64k For the single-user HPC use cases it is made unlimited. > My impression from this is we've designed hardware that didn't > consider the problem, and now to let us use that hardware in horrible > ways we should just allow it to pin all the things. It is more complex than that, HW that can support dynamic memory under *everything* is complicated (and in some cases slow!). As there is only a weak rational to do this, we don't see it in often in the market. Jason