Received-SPF: pass (google.com: domain of linux-kernel+bounces-58655-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Thu, 8 Feb 2024 16:16:38 -0400
From: Jason Gunthorpe <jgg@ziepe.ca>
To: Konstantin Taranov <kotaranov@microsoft.com>
Cc: Long Li <longli@microsoft.com>,
	Konstantin Taranov <kotaranov@linux.microsoft.com>,
	"sharmaajay@microsoft.com" <sharmaajay@microsoft.com>,
	"leon@kernel.org" <leon@kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH rdma-next v1 1/1] RDMA/mana_ib: Fix bug in creation of
 dma regions
Message-ID: <20240208201638.GZ31743@ziepe.ca>
References: <1707318566-3141-1-git-send-email-kotaranov@linux.microsoft.com>
 <PH7PR21MB326394A06EF49FF286D57B63CE442@PH7PR21MB3263.namprd21.prod.outlook.com>
 <PAXPR83MB0557C2779B1485277FD7E417B4442@PAXPR83MB0557.EURPRD83.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <PAXPR83MB0557C2779B1485277FD7E417B4442@PAXPR83MB0557.EURPRD83.prod.outlook.com>

On Thu, Feb 08, 2024 at 06:53:05PM +0000, Konstantin Taranov wrote:
> > From: Long Li <longli@microsoft.com>
> > Sent: Thursday, 8 February 2024 19:43
> > To: Konstantin Taranov <kotaranov@linux.microsoft.com>; Konstantin
> > Taranov <kotaranov@microsoft.com>; sharmaajay@microsoft.com;
> > jgg@ziepe.ca; leon@kernel.org
> > Cc: linux-rdma@vger.kernel.org; linux-kernel@vger.kernel.org
> > Subject: RE: [PATCH rdma-next v1 1/1] RDMA/mana_ib: Fix bug in creation of
> > dma regions
> > 
> > >
> > >  	/* Hardware requires dma region to align to chosen page size */
> > > -	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, 0);
> > > +	page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, virt);
> > >  	if (!page_sz) {
> > >  		ibdev_dbg(&dev->ib_dev, "failed to find page size.\n");
> > >  		return -ENOMEM;
> > >  	}
> > 
> > How about doing:
> > page_sz = ib_umem_find_best_pgsz(umem, PAGE_SZ_BM, force_zero_offset
> > ? 0 : virt);
> > 
> > Will this work? This can get rid of the following while loop.
> > 
> 
> I do not think so. I mentioned once, that it was failing for me with existing code
> with the 4K-aligned addresses and 8K pages. In this case, we miscalculate the 
> number of pages. So, we think that it is one 8K page, but it is in fact two.

That is a confusing statement.. What is "we" here?

ib_umem_dma_offset() is not always guaranteed to be zero, with a 0
iova. With higher order pages the offset can be within the page, it
generates

  offset = IOVA % pgsz

There are a couple places that do want the offset to be fixed to zero
and have the loop, at this point it would be good to consolidate them
into some common ib_umem_find_best_pgsz_zero_offset() or something.

> > > +
> > > +	if (force_zero_offset) {
> > > +		while (ib_umem_dma_offset(umem, page_sz) && page_sz >
> > > PAGE_SIZE)
> > > +			page_sz /= 2;
> > > +		if (ib_umem_dma_offset(umem, page_sz) != 0) {
> > > +			ibdev_dbg(&dev->ib_dev, "failed to find page size to
> > > force zero offset.\n");
> > > +			return -ENOMEM;
> > > +		}
> > > +	}
> > > +

Yes this doesn't look quite right..

It should flow from the HW capability, the helper you call should be
tightly linked to what the HW can do.

ib_umem_find_best_pgsz() is used for MRs that have the usual
  offset = IOVA % pgsz

We've always created other helpers for other restrictions.

So you should move your "force_zero_offset" into another helper and
describe exactly how the HW works to support the calculation

It is odd to have the offset loop and be using
ib_umem_find_best_pgsz() with some iova, usually you'd use
ib_umem_find_best_pgoff() in those cases, see the other callers.

Jason