Date:   Mon, 20 Dec 2021 10:31:27 +0100
From:   Daniel Vetter <daniel@ffwll.ch>
To:     Jason Gunthorpe <jgg@ziepe.ca>
Cc:     Christian =?iso-8859-1?Q?K=F6nig?= <christian.koenig@amd.com>,
        Shunsuke Mie <mie@igel.co.jp>,
        Zhu Yanjun <zyjzyj2000@gmail.com>,
        Alex Deucher <alexander.deucher@amd.com>,
        Daniel Vetter <daniel.vetter@ffwll.ch>,
        Doug Ledford <dledford@redhat.com>,
        Jianxin Xiong <jianxin.xiong@intel.com>,
        Leon Romanovsky <leon@kernel.org>,
        Maor Gottlieb <maorg@nvidia.com>,
        Sean Hefty <sean.hefty@intel.com>,
        Sumit Semwal <sumit.semwal@linaro.org>,
        dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org,
        linux-media@vger.kernel.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        linux-rdma <linux-rdma@vger.kernel.org>,
        Damian Hobson-Garcia <dhobsong@igel.co.jp>,
        Takanari Hayama <taki@igel.co.jp>,
        Tomohito Esaki <etom@igel.co.jp>
Subject: Re: [RFC PATCH v4 0/2] RDMA/rxe: Add dma-buf support
Message-ID: <YcBNbypJT3UJ0RG6@phenom.ffwll.local>
Mail-Followup-To: Jason Gunthorpe <jgg@ziepe.ca>,
        Christian =?iso-8859-1?Q?K=F6nig?= <christian.koenig@amd.com>,
        Shunsuke Mie <mie@igel.co.jp>, Zhu Yanjun <zyjzyj2000@gmail.com>,
        Alex Deucher <alexander.deucher@amd.com>,
        Doug Ledford <dledford@redhat.com>,
        Jianxin Xiong <jianxin.xiong@intel.com>,
        Leon Romanovsky <leon@kernel.org>, Maor Gottlieb <maorg@nvidia.com>,
        Sean Hefty <sean.hefty@intel.com>,
        Sumit Semwal <sumit.semwal@linaro.org>,
        dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org,
        linux-media@vger.kernel.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        linux-rdma <linux-rdma@vger.kernel.org>,
        Damian Hobson-Garcia <dhobsong@igel.co.jp>,
        Takanari Hayama <taki@igel.co.jp>, Tomohito Esaki <etom@igel.co.jp>
References: <20211122110817.33319-1-mie@igel.co.jp>
 <CANXvt5oB8_2sDGccSiTMqeLYGi3Vuo-6NnHJ9PGgZZMv=fnUVw@mail.gmail.com>
 <20211207171447.GA6467@ziepe.ca>
 <CANXvt5rCayOcengPr7Z_aFmJaXwWj9VcWZbaHnuHj6=2CkPndA@mail.gmail.com>
 <20211210124204.GG6467@ziepe.ca>
 <880e25ad-4fe9-eacd-a971-993eaea37fc4@amd.com>
 <20211210132656.GH6467@ziepe.ca>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20211210132656.GH6467@ziepe.ca>
Precedence: bulk

On Fri, Dec 10, 2021 at 09:26:56AM -0400, Jason Gunthorpe wrote:
> On Fri, Dec 10, 2021 at 01:47:37PM +0100, Christian König wrote:
> > Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
> > > On Fri, Dec 10, 2021 at 08:29:24PM +0900, Shunsuke Mie wrote:
> > > > Hi Jason,
> > > > Thank you for replying.
> > > > 
> > > > 2021年12月8日(水) 2:14 Jason Gunthorpe <jgg@ziepe.ca>:
> > > > > On Fri, Dec 03, 2021 at 12:51:44PM +0900, Shunsuke Mie wrote:
> > > > > > Hi maintainers,
> > > > > > 
> > > > > > Could you please review this patch series?
> > > > > Why is it RFC?
> > > > > 
> > > > > I'm confused why this is useful?
> > > > > 
> > > > > This can't do copy from MMIO memory, so it shouldn't be compatible
> > > > > with things like Gaudi - does something prevent this?
> > > > I think if an export of the dma-buf supports vmap, CPU is able to access the
> > > > mmio memory.
> > > > 
> > > > Is it wrong? If this is wrong, there is no advantages this changes..
> > > I don't know what the dmabuf folks did, but yes, it is wrong.
> > > 
> > > IOMEM must be touched using only special accessors, some platforms
> > > crash if you don't do this. Even x86 will crash if you touch it with
> > > something like an XMM optimized memcpy.
> > > 
> > > Christian? If the vmap succeeds what rules must the caller use to
> > > access the memory?
> > 
> > See dma-buf-map.h and especially struct dma_buf_map.
> > 
> > MMIO memory is perfectly supported here and actually the most common case.
> 
> Okay that looks sane, but this rxe RFC seems to ignore this
> completely. It stuffs the vaddr directly into a umem which goes to all
> manner of places in the driver.
> 
> ??

dma_buf_map is fairly new and we haven't rolled it out consistently yet.
In the past 10 years we simply yolo'd this :-)

Just an explanation, not an excuse for new code to not use dma_buf_map
consistently now that we fixed this mistake.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch