Date:   Tue, 28 Feb 2023 11:24:07 -0500
From:   "Liam R. Howlett" <Liam.Howlett@Oracle.com>
To:     Danilo Krummrich <dakr@redhat.com>
Cc:     airlied@gmail.com, daniel@ffwll.ch, tzimmermann@suse.de,
        mripard@kernel.org, corbet@lwn.net, christian.koenig@amd.com,
        bskeggs@redhat.com, matthew.brost@intel.com,
        boris.brezillon@collabora.com, alexdeucher@gmail.com,
        ogabbay@kernel.org, bagasdotme@gmail.com, willy@infradead.org,
        jason@jlekstrand.net, dri-devel@lists.freedesktop.org,
        nouveau@lists.freedesktop.org, linux-doc@vger.kernel.org,
        linux-mm@kvack.org, linux-kernel@vger.kernel.org,
        Dave Airlie <airlied@redhat.com>
Subject: Re: [PATCH drm-next v2 05/16] drm: manager to keep track of GPUs VA
 mappings
Message-ID: <20230228162407.34civk22g5kh7mwt@revolver>
Mail-Followup-To: "Liam R. Howlett" <Liam.Howlett@Oracle.com>,
        Danilo Krummrich <dakr@redhat.com>, airlied@gmail.com,
        daniel@ffwll.ch, tzimmermann@suse.de, mripard@kernel.org,
        corbet@lwn.net, christian.koenig@amd.com, bskeggs@redhat.com,
        matthew.brost@intel.com, boris.brezillon@collabora.com,
        alexdeucher@gmail.com, ogabbay@kernel.org, bagasdotme@gmail.com,
        willy@infradead.org, jason@jlekstrand.net,
        dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org,
        linux-doc@vger.kernel.org, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, Dave Airlie <airlied@redhat.com>
References: <20230217134422.14116-1-dakr@redhat.com>
 <20230217134422.14116-6-dakr@redhat.com>
 <20230221182050.day6z5ge2e3dxerv@revolver>
 <63fd642e.170a0220.f67f6.e248@mx.google.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <63fd642e.170a0220.f67f6.e248@mx.google.com>
User-Agent: NeoMutt/20220429
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?jIS3d3e+a64LlnFm9lhmvxUXOPqkFton2MtlqXGnyohwRyPCNazNFGv/lYYC?=
 =?us-ascii?Q?di3kTkrN1BAheidnTc/R+XTnA+072nxLmPPiwr85RwYPZrCE9nsWGNiY7vb8?=
 =?us-ascii?Q?lMl3LNa07JqOIYdgQmCPtTmzE4kpLHrvQdnDGkkA1+N8b5Vr1CDHRVyJVbcm?=
 =?us-ascii?Q?3rYTvHd4qjaWMG1LYrcz5EF67sUEwGZgv8DDlHy5iDXnKNF4T105YfE//7Q8?=
 =?us-ascii?Q?LHzzhrK6hfTbe9plWdbKCVf6d/PzL8WmZnFlVa6zWBPtPMQp3ufJtFhdwxs2?=
 =?us-ascii?Q?WIyBVcFrq0d2uCHwGqSRxdVuJvA8FBPAOIUR6e4WEnuajyRLR2LcwiWduP1J?=
 =?us-ascii?Q?R5aH0tY0Q8JaIIds4MT9pRpkMW4S4D5bDFLrNPBZXjLTcWhF6RHEoYDmlpRF?=
 =?us-ascii?Q?VbStRnrDQhiqY0KK9htafScQ+GgTeEpQL9H45Z1Nsn6wvFSHSMJH7nyZsyB0?=
 =?us-ascii?Q?jOX/93hj8+xQGuA8krYG14yXnCPBFiPlFHZpeIG1LYP4K8wNmYeWQhmmkX5z?=
 =?us-ascii?Q?tA6e7jLurOp/oQ13JEbzvcLLEVGOykKAIo4HjcS86rceA1cdewtJaViW4TkC?=
 =?us-ascii?Q?gXXA1IGId9J/3QTa0LVxWdw8EVPsfdBvCfCfBLTWVKU9m3niSv4De/Yibw99?=
 =?us-ascii?Q?6ay5myiUaFsa5VL2fS+tqBaXemMFi/9xGxgRO0U/oHGEr+OoiYFSoQyZVO34?=
 =?us-ascii?Q?MfO448c3NqBEy4V5eBlAUywZD/J6HcXyRUTjmY1XVQ6lTiyjjRxs+yL1vVFY?=
 =?us-ascii?Q?qrZWlUo2G4O4sPGQAACUGcEwv0RQJSBhSekz+ePBtczx3zGpFMQPkGL+rWXX?=
 =?us-ascii?Q?ezJC5/c0xSWV27+ZtHbZqoR918t+pl79YwEDzD5RD8NL+nlcvQCtBjjBrpgk?=
 =?us-ascii?Q?fo0daj4sSXKT1oerCmHYbRo5hvy+kB7JNjVTbeOcgr/ky/v+5MBZs00fDKdp?=
 =?us-ascii?Q?DE5lVtjMUC8XlkrQMCO2Rf4fJGv5r3gsll5g3KPW7sYBs8gXo/cx+MJ/k8JE?=
 =?us-ascii?Q?K9BuawIdZXQR2eh9Ztb9Yv07XcL1S1puh6BQGNoJcM+DsN6sPmzhD0c65HTq?=
 =?us-ascii?Q?ZQzZemV/+wEx2gk45Ug3dbhJR0hHH7jQGWIN8W2FoXyGF1k/ucP11eNa82YU?=
 =?us-ascii?Q?Tkml2YfcAxVzByM9uN528dntx5c1jbpLt4V30KtdnUqEWL697jSOxpi8XNU0?=
 =?us-ascii?Q?SNvDSIJYf/oLR+6qogXVh+6WNrXAF5NcTEEupp8/YdPI4U8tLf/oeMGvc+Rt?=
 =?us-ascii?Q?69IRCge6VxLwkAa/kVX/epYDmUl6vv0bOuG7fq6Car8V2Ibu4biYPKbhhxa2?=
 =?us-ascii?Q?cY7gbig18Yb/N6jSYKsdJP9a+jhSWn/JGY9PMaIHpDDqFeb82C/pCI7hSrcp?=
 =?us-ascii?Q?dkLMDAERLXo9c96irDICTS5KPgebUSd8yXVQf1E+18FlhhYUiyl7P6kOQyRl?=
 =?us-ascii?Q?piwPub5bk2aYsryDsQEL80r94xCD2t9GB1fwn/4xCL+kDPKcV9sbAA/O4hd3?=
 =?us-ascii?Q?x+aaH24qOWRl1dENM2p+nbEZPdBfIBbMmfuuxujMj5dr3VWOieyFc0aZVyED?=
 =?us-ascii?Q?h1yncfIiLx4jqlUn+GP6Op0DC0336lkjp7b89b/2MceUeVrf3dvtCzI9mDOh?=
 =?us-ascii?Q?tA=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: =?us-ascii?Q?9RVr1YmEwtXEz09ap4XMvHCnKQ56bScAXwPqRu3Kf4eYsaix48+T59+FuAxA?=
 =?us-ascii?Q?sCOpcmUnM5Cy56MH3rY5ccIS490lYAT8ChJny6dYSu4yKgxR/63P2LE3FQtf?=
 =?us-ascii?Q?KkgnlVI/QRxY1TI1ES4CEgC545kfs8KxyTlgGS5kJYED2SQ566fS4/kAKCdL?=
 =?us-ascii?Q?BSyKRWcXvOZjR6ldw5lShUJVUezmS/uDssVN1tSnHpBqhLFfNZNqxIGCLEOk?=
 =?us-ascii?Q?VAkLub/W7LGmPSOjYoAgLrGZ0iCL0RO2ObMFRU2DewmrgKlaDK5BipQF4ni9?=
 =?us-ascii?Q?Ym7L9yeoY2Rlad45KyGBUbHY6UvmU0J68ny+TuIL2d/V5drhQFhpjrneCvNg?=
 =?us-ascii?Q?h5kx0QPYdri95uIUPVHN0x8YWxt8HvZhS7PQJSVfNjbNCF340XaRf0gONsZ/?=
 =?us-ascii?Q?nTWbv9nDdzvZrDMjs+2dIMQNEAGOcu6i8nD9GxI7PKZg8VpL+uYra8yTfVbk?=
 =?us-ascii?Q?mzYtIh5BUcNCViwRrbZTT98D7Q/4POlchEeEUIpu8FTEYfIGNRxMboR+i17J?=
 =?us-ascii?Q?FskukKmqcA57tT5Pfxfada6QOUECVpI0LMeJ4tCHjr6hEbd1nLwWkqmLZhaq?=
 =?us-ascii?Q?g3ZIpVqP6b9yNOUiFFHeWvNOo8n+e0mL085TfKYNbCNPqJLUDe6fQIpGhN8d?=
 =?us-ascii?Q?XO4JgtVZlZ7918hUGnDY9VIppnZ/fGtYuDMC7ebZiMf1MFwd58CNrXIsYBmK?=
 =?us-ascii?Q?TQEONba0QRCIKPwRkJibUECLrCYV9SBJImDGNxrhRqf8dkc/jGOpcMWcUXXE?=
 =?us-ascii?Q?YsImiFJ/ORaShkFmFKsYt/HtL75hN3TbOzqDf8J2dyBy+e+0fah8yBzb7tZA?=
 =?us-ascii?Q?dUV4SAe4FVyOojMtwgTyhffILZyOt9L0MHPb6tzZSWgKfm1LNYWeJH8L0+jR?=
 =?us-ascii?Q?uyZEbpQYTpWR50A80ckzFZUkPTHs7w8QtWkq2rMXDdt5LLMkYA4zSXpwmWcw?=
 =?us-ascii?Q?tM9r8ezFuoY9sbcbd5UbxFz5dOQZ0yJJG6Q5t0GDFc9ib+33KTuIDi+vYWEg?=
 =?us-ascii?Q?tM/iDbDxBKjvbjdPFvASm6TwJasqTN/9BstKBxwGaL2izhg/jtMYuRuqcNc/?=
 =?us-ascii?Q?ZoXkWkMGFVE028jW8agb5YB+Ga2rG++gSpVY76wdAvc8Ho4RmjBNBija+mr2?=
 =?us-ascii?Q?KBFAL4llSZZ66BGEJ1D2gg69yCRO7d/4jA=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: caab09dd-4a41-4ac8-72ce-08db19a83968
X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Feb 2023 16:24:11.4216
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fZzQGlPaE6eh0kuhh8ebdhhMo0kBWmX7RLpVlKF6p+sdaMv4AEeZDuYxlZoOMjr6xXHslBME2Bsue92JcVXBmQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB4825
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-02-28_13,2023-02-28_03,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0
 suspectscore=0 spamscore=0 adultscore=0 phishscore=0 malwarescore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2302280135
X-Proofpoint-GUID: KiKcDv5AtWpM352P_tBj2AAZYNX5s444
X-Proofpoint-ORIG-GUID: KiKcDv5AtWpM352P_tBj2AAZYNX5s444
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

* Danilo Krummrich <dakr@redhat.com> [230227 21:17]:
> On Tue, Feb 21, 2023 at 01:20:50PM -0500, Liam R. Howlett wrote:
> > * Danilo Krummrich <dakr@redhat.com> [230217 08:45]:
> > > Add infrastructure to keep track of GPU virtual address (VA) mappings
> > > with a decicated VA space manager implementation.
> > > 
> > > New UAPIs, motivated by Vulkan sparse memory bindings graphics drivers
> > > start implementing, allow userspace applications to request multiple and
> > > arbitrary GPU VA mappings of buffer objects. The DRM GPU VA manager is
> > > intended to serve the following purposes in this context.
> > > 
> > > 1) Provide infrastructure to track GPU VA allocations and mappings,
> > >    making use of the maple_tree.
> > > 
> > > 2) Generically connect GPU VA mappings to their backing buffers, in
> > >    particular DRM GEM objects.
> > > 
> > > 3) Provide a common implementation to perform more complex mapping
> > >    operations on the GPU VA space. In particular splitting and merging
> > >    of GPU VA mappings, e.g. for intersecting mapping requests or partial
> > >    unmap requests.
> > > 
> > > Suggested-by: Dave Airlie <airlied@redhat.com>
> > > Signed-off-by: Danilo Krummrich <dakr@redhat.com>
> > > ---
> > >  Documentation/gpu/drm-mm.rst    |   31 +
> > >  drivers/gpu/drm/Makefile        |    1 +
> > >  drivers/gpu/drm/drm_gem.c       |    3 +
> > >  drivers/gpu/drm/drm_gpuva_mgr.c | 1704 +++++++++++++++++++++++++++++++
> > >  include/drm/drm_drv.h           |    6 +
> > >  include/drm/drm_gem.h           |   75 ++
> > >  include/drm/drm_gpuva_mgr.h     |  714 +++++++++++++
> > >  7 files changed, 2534 insertions(+)
> > >  create mode 100644 drivers/gpu/drm/drm_gpuva_mgr.c
> > >  create mode 100644 include/drm/drm_gpuva_mgr.h
> > > 
> > > diff --git a/Documentation/gpu/drm-mm.rst b/Documentation/gpu/drm-mm.rst
> > > index a52e6f4117d6..c9f120cfe730 100644
> > > --- a/Documentation/gpu/drm-mm.rst
> > > +++ b/Documentation/gpu/drm-mm.rst
> > > @@ -466,6 +466,37 @@ DRM MM Range Allocator Function References
> > >  .. kernel-doc:: drivers/gpu/drm/drm_mm.c
> > >     :export:
> > >  
> > ...
> > 
> > > +
> > > +/**
> > > + * drm_gpuva_remove_iter - removes the iterators current element
> > > + * @it: the &drm_gpuva_iterator
> > > + *
> > > + * This removes the element the iterator currently points to.
> > > + */
> > > +void
> > > +drm_gpuva_iter_remove(struct drm_gpuva_iterator *it)
> > > +{
> > > +	mas_erase(&it->mas);
> > > +}
> > > +EXPORT_SYMBOL(drm_gpuva_iter_remove);
> > > +
> > > +/**
> > > + * drm_gpuva_insert - insert a &drm_gpuva
> > > + * @mgr: the &drm_gpuva_manager to insert the &drm_gpuva in
> > > + * @va: the &drm_gpuva to insert
> > > + * @addr: the start address of the GPU VA
> > > + * @range: the range of the GPU VA
> > > + *
> > > + * Insert a &drm_gpuva with a given address and range into a
> > > + * &drm_gpuva_manager.
> > > + *
> > > + * Returns: 0 on success, negative error code on failure.
> > > + */
> > > +int
> > > +drm_gpuva_insert(struct drm_gpuva_manager *mgr,
> > > +		 struct drm_gpuva *va)
> > > +{
> > > +	u64 addr = va->va.addr;
> > > +	u64 range = va->va.range;
> > > +	MA_STATE(mas, &mgr->va_mt, addr, addr + range - 1);
> > > +	struct drm_gpuva_region *reg = NULL;
> > > +	int ret;
> > > +
> > > +	if (unlikely(!drm_gpuva_in_mm_range(mgr, addr, range)))
> > > +		return -EINVAL;
> > > +
> > > +	if (unlikely(drm_gpuva_in_kernel_region(mgr, addr, range)))
> > > +		return -EINVAL;
> > > +
> > > +	if (mgr->flags & DRM_GPUVA_MANAGER_REGIONS) {
> > > +		reg = drm_gpuva_in_region(mgr, addr, range);
> > > +		if (unlikely(!reg))
> > > +			return -EINVAL;
> > > +	}
> > > +
> > 
> > -----
> > 
> > > +	if (unlikely(drm_gpuva_find_first(mgr, addr, range)))
> > > +		return -EEXIST;
> > > +
> > > +	ret = mas_store_gfp(&mas, va, GFP_KERNEL);
> > 
> > mas_walk() will set the internal maple state to the limits to what it
> > finds.  So, instead of an iterator, you can use the walk function and
> > ensure there is a large enough area in the existing NULL:
> > 
> > /*
> >  * Nothing at addr, mas now points to the location where the store would
> >  * happen
> >  */
> > if (mas_walk(&mas))
> > 	return -EEXIST;
> > 
> 
> For some reason mas_walk() finds an entry and hence this function returns
> -EEXIST for the following sequence of insertions.
> 
> A = [0xc0000 - 0xfffff]
> B = [0x0 - 0xbffff]
> 
> Interestingly, inserting B before A works fine.
> 
> I attached a test module that reproduces the issue. I hope its just a stupid
> mistake I just can't spot though.

This is probably my fault in how I explained things, I seem to have had
a bug in my code.

Let me try again.

mas_walk(&mas) will go to the range of mas.index
	It will set mas.index = range_start
	It will set mas.last = range_end
	It will return entry in that range.

Your code is walking to addr (0xc0000, say)
You get NULL
and the range is now: mas.index = 0, mas.last = ULONG_MAX

You set mas.last = 0xc0000 + 0x40000 -1
You store your va in the range of 0 - 0xfffff - This isn't what you want
to do and this is why you are seeing it exists when done in this order.

In the reverse order, your lower limit is fine so it works out.

Try adding a check to ensure the lower range is still accurate as well:
        if (mas.index < addr)                                                                                           
                mas.index = addr;

If you compile with CONFIG_DEBUG_MAPLE_TREE, you can use mt_dump() to
dump the tree for debugging.

I also have some quality of life patches I'm developing to configure the
format of the dump (hex/dec) and a mas_dump() for more information as
well.

> 
> > /* The NULL entry ends at mas.last, make sure there is room */
> > if (mas.last < (addr + range - 1))
> > 	return -EEXIST;
> > 
> > /* Limit the store size to the correct end address, and store */
> >  mas.last = addr + range - 1;
> >  ret = mas_store_gfp(&mas, va, GFP_KERNEL);
> >