64 DRM device nodes is not enough for everyone.
Upgrade it to ~512K (which definitely is more than enough).
To allow testing userspace support for >64 devices, add additional DRM
modparam (force_extended_minors) which causes DRM to skip allocating minors
in 0-192 range.
Additionally - convert minors to use XArray instead of IDR to simplify the
locking.
v1 -> v2:
Don't touch DRM_MINOR_CONTROL and its range (Simon Ser)
v2 -> v3:
Don't use legacy scheme for >=192 minor range (Dave Airlie)
Add modparam for testing (Dave Airlie)
Add lockdep annotation for IDR (Daniel Vetter)
v3 -> v4:
Convert from IDR to XArray (Matthew Wilcox)
v4 -> v5:
Fixup IDR to XArray conversion (Matthew Wilcox)
v5 -> v6:
Also convert Accel to XArray
Rename skip_legacy_minors to force_extended_minors
Michał Winiarski (4):
drm: Use XArray instead of IDR for minors
accel: Use XArray instead of IDR for minors
drm: Expand max DRM device number to full MINORBITS
drm: Introduce force_extended_minors modparam
drivers/accel/drm_accel.c | 110 +++------------------------------
drivers/gpu/drm/drm_drv.c | 105 ++++++++++++++++---------------
drivers/gpu/drm/drm_file.c | 2 +-
drivers/gpu/drm/drm_internal.h | 4 --
include/drm/drm_accel.h | 18 +-----
include/drm/drm_file.h | 5 ++
6 files changed, 69 insertions(+), 175 deletions(-)
--
2.41.0
Having a limit of 64 DRM devices is not good enough for modern world
where we have multi-GPU servers, SR-IOV virtual functions and virtual
devices used for testing.
Let's utilize full minor range for DRM devices.
To avoid regressing the existing userspace, we're still maintaining the
numbering scheme where 0-63 is used for primary, 64-127 is reserved
(formerly for control) and 128-191 is used for render.
For minors >= 192, we're allocating minors dynamically on a first-come,
first-served basis.
Signed-off-by: Michał Winiarski <[email protected]>
---
drivers/gpu/drm/drm_drv.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index 34b60196c443..c2c6e80e6b31 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -121,10 +121,19 @@ static void drm_minor_alloc_release(struct drm_device *dev, void *data)
xa_erase(drm_minor_get_xa(minor->type), minor->index);
}
+/*
+ * DRM used to support 64 devices, for backwards compatibility we need to maintain the
+ * minor allocation scheme where minors 0-63 are primary nodes, 64-127 are control nodes,
+ * and 128-191 are render nodes.
+ * After reaching the limit, we're allocating minors dynamically - first-come, first-serve.
+ * Accel nodes are using a distinct major, so the minors are allocated in continuous 0-MAX
+ * range.
+ */
#define DRM_MINOR_LIMIT(t) ({ \
typeof(t) _t = (t); \
_t == DRM_MINOR_ACCEL ? XA_LIMIT(0, ACCEL_MAX_MINORS) : XA_LIMIT(64 * _t, 64 * _t + 63); \
})
+#define DRM_EXTENDED_MINOR_LIMIT XA_LIMIT(192, (1 << MINORBITS) - 1)
static int drm_minor_alloc(struct drm_device *dev, enum drm_minor_type type)
{
@@ -140,6 +149,9 @@ static int drm_minor_alloc(struct drm_device *dev, enum drm_minor_type type)
r = xa_alloc(drm_minor_get_xa(type), &minor->index,
NULL, DRM_MINOR_LIMIT(type), GFP_KERNEL);
+ if (r == -EBUSY && (type == DRM_MINOR_PRIMARY || type == DRM_MINOR_RENDER))
+ r = xa_alloc(&drm_minors_xa, &minor->index,
+ NULL, DRM_EXTENDED_MINOR_LIMIT, GFP_KERNEL);
if (r < 0)
return r;
--
2.41.0
On Monday, July 24th, 2023 at 23:14, Michał Winiarski <[email protected]> wrote:
> Having a limit of 64 DRM devices is not good enough for modern world
> where we have multi-GPU servers, SR-IOV virtual functions and virtual
> devices used for testing.
> Let's utilize full minor range for DRM devices.
> To avoid regressing the existing userspace, we're still maintaining the
> numbering scheme where 0-63 is used for primary, 64-127 is reserved
> (formerly for control) and 128-191 is used for render.
> For minors >= 192, we're allocating minors dynamically on a first-come,
> first-served basis.
In general the approach looks good to me. Old libdrm will see the new
nodes as nodes with an unknown type when it tries to infer the nod type
from the minor, which is as good as it gets.
We do need patches to stop trying to infer the node type from the minor
in libdrm, though. Emil has suggested using sysfs, which we already do
in a few places in libdrm.
Am 26.07.23 um 20:15 schrieb Simon Ser:
> On Monday, July 24th, 2023 at 23:14, Michał Winiarski <[email protected]> wrote:
>
>> Having a limit of 64 DRM devices is not good enough for modern world
>> where we have multi-GPU servers, SR-IOV virtual functions and virtual
>> devices used for testing.
>> Let's utilize full minor range for DRM devices.
>> To avoid regressing the existing userspace, we're still maintaining the
>> numbering scheme where 0-63 is used for primary, 64-127 is reserved
>> (formerly for control) and 128-191 is used for render.
>> For minors >= 192, we're allocating minors dynamically on a first-come,
>> first-served basis.
> In general the approach looks good to me. Old libdrm will see the new
> nodes as nodes with an unknown type when it tries to infer the nod type
> from the minor, which is as good as it gets.
Yeah, agree. I wouldn't upstream patch #4, but apart from that it looks
like it shouldn't break anything which wasn't broken before.
> We do need patches to stop trying to infer the node type from the minor
> in libdrm, though. Emil has suggested using sysfs, which we already do
> in a few places in libdrm.
That sounds like a really good idea to me as well.
But what do we do with DRM_MAX_MINOR? Change it or keep it and say apps
should use drmGetDevices2() like Emil suggested?
Regards,
Christian.
On Thursday, July 27th, 2023 at 14:01, Christian König <[email protected]> wrote:
> > We do need patches to stop trying to infer the node type from the minor
> > in libdrm, though. Emil has suggested using sysfs, which we already do
> > in a few places in libdrm.
>
> That sounds like a really good idea to me as well.
>
> But what do we do with DRM_MAX_MINOR? Change it or keep it and say apps
> should use drmGetDevices2() like Emil suggested?
DRM_MAX_MINOR has been bumped to 64 now.
With the new minor allocation scheme, DRM_MAX_MINOR is meaningless
because there is no "max minor per type" concept anymore: the minor no
longer contains the type.
So I'd suggest leaving it as-is (so that old apps still continue to
work on systems with < 64 devices like they do today) and mark it as
deprecated.
Am 28.07.23 um 16:22 schrieb Simon Ser:
> On Thursday, July 27th, 2023 at 14:01, Christian König <[email protected]> wrote:
>
>>> We do need patches to stop trying to infer the node type from the minor
>>> in libdrm, though. Emil has suggested using sysfs, which we already do
>>> in a few places in libdrm.
>> That sounds like a really good idea to me as well.
>>
>> But what do we do with DRM_MAX_MINOR? Change it or keep it and say apps
>> should use drmGetDevices2() like Emil suggested?
> DRM_MAX_MINOR has been bumped to 64 now.
>
> With the new minor allocation scheme, DRM_MAX_MINOR is meaningless
> because there is no "max minor per type" concept anymore: the minor no
> longer contains the type.
>
> So I'd suggest leaving it as-is (so that old apps still continue to
> work on systems with < 64 devices like they do today) and mark it as
> deprecated.
Sounds like a plan to me.
Regards,
Christian.
I would like if these kernel patches are accepted by everyone, If yes,
when they can be upstream.
I have a MR for libdrm to support drm nodes type up to 2^MINORBITS
nodes which can work with these patches,
https://gitlab.freedesktop.org/mesa/drm/-/merge_requests/305
Thanks!
James
On 2023-08-08 09:55, Christian König wrote:
> Am 28.07.23 um 16:22 schrieb Simon Ser:
>> On Thursday, July 27th, 2023 at 14:01, Christian König
>> <[email protected]> wrote:
>>
>>>> We do need patches to stop trying to infer the node type from the
>>>> minor
>>>> in libdrm, though. Emil has suggested using sysfs, which we already do
>>>> in a few places in libdrm.
>>> That sounds like a really good idea to me as well.
>>>
>>> But what do we do with DRM_MAX_MINOR? Change it or keep it and say apps
>>> should use drmGetDevices2() like Emil suggested?
>> DRM_MAX_MINOR has been bumped to 64 now.
>>
>> With the new minor allocation scheme, DRM_MAX_MINOR is meaningless
>> because there is no "max minor per type" concept anymore: the minor no
>> longer contains the type.
>>
>> So I'd suggest leaving it as-is (so that old apps still continue to
>> work on systems with < 64 devices like they do today) and mark it as
>> deprecated.
>
> Sounds like a plan to me.
>
> Regards,
> Christian.
> On Jul 24, 2023, at 2:14 PM, Michał Winiarski <[email protected]> wrote:
>
> 64 DRM device nodes is not enough for everyone.
> Upgrade it to ~512K (which definitely is more than enough).
>
> To allow testing userspace support for >64 devices, add additional DRM
> modparam (force_extended_minors) which causes DRM to skip allocating minors
> in 0-192 range.
> Additionally - convert minors to use XArray instead of IDR to simplify the
> locking.
>
> v1 -> v2:
> Don't touch DRM_MINOR_CONTROL and its range (Simon Ser)
>
> v2 -> v3:
> Don't use legacy scheme for >=192 minor range (Dave Airlie)
> Add modparam for testing (Dave Airlie)
> Add lockdep annotation for IDR (Daniel Vetter)
>
> v3 -> v4:
> Convert from IDR to XArray (Matthew Wilcox)
>
> v4 -> v5:
> Fixup IDR to XArray conversion (Matthew Wilcox)
>
> v5 -> v6:
> Also convert Accel to XArray
> Rename skip_legacy_minors to force_extended_minors
>
> Michał Winiarski (4):
> drm: Use XArray instead of IDR for minors
> accel: Use XArray instead of IDR for minors
> drm: Expand max DRM device number to full MINORBITS
> drm: Introduce force_extended_minors modparam
>
> drivers/accel/drm_accel.c | 110 +++------------------------------
> drivers/gpu/drm/drm_drv.c | 105 ++++++++++++++++---------------
> drivers/gpu/drm/drm_file.c | 2 +-
> drivers/gpu/drm/drm_internal.h | 4 --
> include/drm/drm_accel.h | 18 +-----
> include/drm/drm_file.h | 5 ++
> 6 files changed, 69 insertions(+), 175 deletions(-)
>
> --
> 2.41.0
>
Hi Michal,
What is the status on this patch? Did it ever get accepted upstream?
If so, what release? I don’t see the changes in the latest Linux kernel.
I am working on a system that involves a large number of GPUs, where
each GPU consumes a number of DRM devices. As such, I’m easily
exceeding the current limit of 64 in the (6.6) kernel. To workaround this issue,
I have temporarily picked up this patch which is doing the trick, but now
I’m wondering if this patch has seen the light of day in the Linux kernel.
Thanks for any info!
Regards,
Eric