Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 7 Nov 2022 18:52:48 +0100
From:   =?utf-8?Q?Micha=C5=82?= Winiarski <michal.winiarski@intel.com>
To:     Oded Gabbay <ogabbay@kernel.org>
CC:     <dri-devel@lists.freedesktop.org>, <linux-kernel@vger.kernel.org>,
        <intel-gfx@lists.freedesktop.org>, David Airlie <airlied@linux.ie>,
        "Daniel Vetter" <daniel@ffwll.ch>,
        Maarten Lankhorst <maarten.lankhorst@linux.intel.com>,
        Maxime Ripard <mripard@kernel.org>,
        Thomas Zimmermann <tzimmermann@suse.de>,
        Simon Ser <contact@emersion.fr>,
        Matthew Wilcox <willy@infradead.org>
Subject: Re: [PATCH v5 1/3] drm: Use XArray instead of IDR for minors
Message-ID: <20221107175248.np7ptcuzwdkc6vbf@nostramo>
References: <20220911211443.581481-1-michal.winiarski@intel.com>
 <20220911211443.581481-2-michal.winiarski@intel.com>
 <CAFCwf11=hKGm93oF1A5PLhdvdo2ujYJWyt4qAqK-KQQqe6ngDA@mail.gmail.com>
 <CAFCwf12+piEABvv=nxFJCzUU4XdZw5Pu=r=i3aKQGZrBeraFqw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAFCwf12+piEABvv=nxFJCzUU4XdZw5Pu=r=i3aKQGZrBeraFqw@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SnExWFZUbFBSVCswSTlmQXBYTGRKRktGUEV5d2hxSWwzTWhpOWNXT2h0Mk5v?=
 =?utf-8?B?N1pHZDF1dDZmUnIzVkIwaXFVeE5VK3NmMFlzVUdKRVdpaGNqYWdYcVBWZVNn?=
 =?utf-8?B?bTY5UGFrc3FzYlhrTnJjT3l5QlluTWxOYVFZa2hTNDNBblRybnhkTmhaeW5T?=
 =?utf-8?B?aGFUS3UwOXpoM2h2RnZyTDE2elBzRXU1YWlRdzZabDdxRVdKM2ZHQXhNL3Qy?=
 =?utf-8?B?SFAxMDZlbW5TTityZ3Y2b3NtaDVHMlRtNWxOa2gzZDU2MVFXK0o3SUQzcWtm?=
 =?utf-8?B?dE1rajE1MjNzdFozK0UxMHMxeGx2ZCttSWtlZnBtVllET20wa0Ryd0poR1Fl?=
 =?utf-8?B?dFhzVUNWa2JDbndkNS8rdmV5QVpoSURPTnJaSm8ycjJaWGJzT2hjT1dFaTBa?=
 =?utf-8?B?SW5MUEZLdjJndkdWQk45TEpnV2ZVckF4b014Z0U5amcxenFCajJWeDRsV0Vh?=
 =?utf-8?B?NU0xYS9DV1RuSnNwdFdkVlBZV1M4MS9GaTZBSUMyRTlUY3FyMXM4QWlLVHM2?=
 =?utf-8?B?dlBYUTczZ29SRWJZcVIzNnR2VXR6NjVZVVVIVWIyTXp1QnRDM0k1c1ovc29I?=
 =?utf-8?B?TitUNi9idnVOY3VERTc1WVpZdkN4S0xlVW1BYXVBUFpIQVJoVkhmS0hXL3d3?=
 =?utf-8?B?UDdaSFpuUERjYmtIcnFralVleEYvV05KTlc1Rm9OZnZkeEQ5Rlo5ejczMGhx?=
 =?utf-8?B?VDR1a3d2S3hPUFlZTncxNXpJVldUWkc1UzJsRW9lWnlOMmROWGdna01sWloy?=
 =?utf-8?B?c3g4RWlCUk5DL0VlK0Z1QjNLS1lpbi9GM2YrbUJRVjZDdnY4Rno5S1FlbUNo?=
 =?utf-8?B?dkFmNU9EMnd4Q1Q1cTlBYUMxQTBrVUpLYXRtWWE3c0wrUXZtQXNmN1pTa0pI?=
 =?utf-8?B?cnp4YnJmejVMWVBrZXFiVjEzMGs5YmpGMWE5VU1xSFc1cnZtelVOcVNYUHFr?=
 =?utf-8?B?ci9rVWpWc1J0TXNXTWpmTlkxd1pIZXdGQnVCWURvOWxEUmZsUFoyLzdNNEVS?=
 =?utf-8?B?V3Uyam9ZOFRSVnRYVHBrQkRhaW1sZFVBK0kycFc1VnJVNkUzZ1NhVEJ6eVE4?=
 =?utf-8?B?ZUZlZmRvandsdnl0bDNZdVl0R3QzZjgzalFGandlaXAxaFk1TDllNFcyVk16?=
 =?utf-8?B?dlpNR2pGRmRCY0Q1YjVtMXJiRG45MjNJNHBidWJzUWU3VVhhdWRNQ3FuU2xH?=
 =?utf-8?B?UFAvNy9KTk9jQmxOamQydFhQQWdqOUl4NU9rYkhzOEVpNFBnVHBzYVl6b1d0?=
 =?utf-8?B?RDNocnJ3cWU2ZkxYckFTNHlUSDhoYVZxSGNBc1VFREdKOXA2dkNueFFwWEZP?=
 =?utf-8?B?NnBNRUJzdStmUUMxVGp6eVZBa25xVmhVaExrSFk1WVJJS2hvM3d5ZGZ5QjhE?=
 =?utf-8?B?cU55cjJrVW5CdWdZZXhsZzhSTjMvUVg5eFFnMGtzUGRJQzVDcVA3enNCamtx?=
 =?utf-8?B?VEIyY1p1YmwvTXV5RVpKMkRoTHBwdGR0ZHNTNjRjYURLc2RpVEl4MkV2NjUw?=
 =?utf-8?B?ZnBBenZpL3BleE1Bd0RYUkl2SW1UeWE1OXVxSVNaeXpSYVNPVlFoUEk2Ukc3?=
 =?utf-8?B?ejk1d1JBMDVpSURGVzlIdVRSRUdJUzZkQ014MXpUKzc0MEZaTWJyQUxrOXJP?=
 =?utf-8?B?Skw5cnVoNUFSMDhlZFovOFc5OW9MVklwM3hIZ2RxQUcrL21KckRnK1hrT1Bn?=
 =?utf-8?B?WjhXZi8xYXQ2bjZLWm85dVM5MU9aem13cnhjaVM5dW82T1Z1U01mR3lvZXh1?=
 =?utf-8?B?Q3B1dW40dGMwb2VrZmVNQ3g0N0dHc0RyVzl1WXAyeHprdlBSTmRSVFpVdGV2?=
 =?utf-8?B?bUVoUFl4RGxFYzlkV0VMN3A0NHBEODNnV0tJV0VyTHgvWmtxMUQ0WEh6OXdx?=
 =?utf-8?B?N0xNcDBodGxRSUFBQ0Z4b0dZQ2FBNk5PTDRlK1Q4Tmp5OHRFWG9qNk52aXBk?=
 =?utf-8?B?bWk0V2ZqdG9LSC9Ha2s5akt1cjE1bVdzaDBSZXdTbW9BK2QyNDJ1WDRSMk0r?=
 =?utf-8?B?TnpMNjVHUkZBZE4zeWZncE5GMnpQN2VlTXZkTXJTTjhaWlVkMnh5SlZkaGhj?=
 =?utf-8?B?LzdhSE5MdnQ1Njc1TExlNGs3dnZpN28wWWo1R3JVMW83eG5sNkpWaUJtLzRG?=
 =?utf-8?B?Z20yZk1aRmVWWFpHQUlJZW9FTWxpcUtSUFJTdFhLTVA2d0s5ak5SK29WZWRa?=
 =?utf-8?B?Tnc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 275366f1-716d-433b-3db4-08dac0e8e42c
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB5373.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2022 17:52:52.1583
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8+rWe+SzECQVj1+fyp4YSe8RYTt//RUeXjYeYSlKQG0q03t4I28IA3I78MQ73GvqzZm0Nd5nLPtBpzK9YvLfirr+VZOPTEVoSQXTihOxQGg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5784
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sun, Nov 06, 2022 at 04:51:39PM +0200, Oded Gabbay wrote:
> On Wed, Nov 2, 2022 at 4:23 PM Oded Gabbay <ogabbay@kernel.org> wrote:
> >
> > On Mon, Sep 12, 2022 at 12:17 AM Michał Winiarski
> > <michal.winiarski@intel.com> wrote:
> > >
> > > IDR is deprecated, and since XArray manages its own state with internal
> > > locking, it simplifies the locking on DRM side.
> > > Additionally, don't use the IRQ-safe variant, since operating on drm
> > > minor is not done in IRQ context.
> > >
> > > Signed-off-by: Michał Winiarski <michal.winiarski@intel.com>
> > > Suggested-by: Matthew Wilcox <willy@infradead.org>
> > > ---
> > >  drivers/gpu/drm/drm_drv.c | 51 ++++++++++++++-------------------------
> > >  1 file changed, 18 insertions(+), 33 deletions(-)
> > >
> > > diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
> > > index 8214a0b1ab7f..61d24cdcd0f8 100644
> > > --- a/drivers/gpu/drm/drm_drv.c
> > > +++ b/drivers/gpu/drm/drm_drv.c
> > > @@ -34,6 +34,7 @@
> > >  #include <linux/pseudo_fs.h>
> > >  #include <linux/slab.h>
> > >  #include <linux/srcu.h>
> > > +#include <linux/xarray.h>
> > >
> > >  #include <drm/drm_cache.h>
> > >  #include <drm/drm_client.h>
> > > @@ -53,8 +54,7 @@ MODULE_AUTHOR("Gareth Hughes, Leif Delgass, José Fonseca, Jon Smirl");
> > >  MODULE_DESCRIPTION("DRM shared core routines");
> > >  MODULE_LICENSE("GPL and additional rights");
> > >
> > > -static DEFINE_SPINLOCK(drm_minor_lock);
> > > -static struct idr drm_minors_idr;
> > > +static DEFINE_XARRAY_ALLOC(drm_minors_xa);
> > >
> > >  /*
> > >   * If the drm core fails to init for whatever reason,
> > > @@ -98,21 +98,19 @@ static struct drm_minor **drm_minor_get_slot(struct drm_device *dev,
> > >  static void drm_minor_alloc_release(struct drm_device *dev, void *data)
> > >  {
> > >         struct drm_minor *minor = data;
> > > -       unsigned long flags;
> > >
> > >         WARN_ON(dev != minor->dev);
> > >
> > >         put_device(minor->kdev);
> > >
> > > -       spin_lock_irqsave(&drm_minor_lock, flags);
> > > -       idr_remove(&drm_minors_idr, minor->index);
> > > -       spin_unlock_irqrestore(&drm_minor_lock, flags);
> > > +       xa_erase(&drm_minors_xa, minor->index);
> > >  }
> > >
> > > +#define DRM_MINOR_LIMIT(t) ({ typeof(t) _t = (t); XA_LIMIT(64 * _t, 64 * _t + 63); })
> > > +
> > >  static int drm_minor_alloc(struct drm_device *dev, unsigned int type)
> > >  {
> > >         struct drm_minor *minor;
> > > -       unsigned long flags;
> > >         int r;
> > >
> > >         minor = drmm_kzalloc(dev, sizeof(*minor), GFP_KERNEL);
> > > @@ -122,21 +120,10 @@ static int drm_minor_alloc(struct drm_device *dev, unsigned int type)
> > >         minor->type = type;
> > >         minor->dev = dev;
> > >
> > > -       idr_preload(GFP_KERNEL);
> > > -       spin_lock_irqsave(&drm_minor_lock, flags);
> > > -       r = idr_alloc(&drm_minors_idr,
> > > -                     NULL,
> > > -                     64 * type,
> > > -                     64 * (type + 1),
> > > -                     GFP_NOWAIT);
> > > -       spin_unlock_irqrestore(&drm_minor_lock, flags);
> > > -       idr_preload_end();
> > > -
> > > +       r = xa_alloc(&drm_minors_xa, &minor->index, NULL, DRM_MINOR_LIMIT(type), GFP_KERNEL);
> This was GFP_NOWAIT in the original code.

Because we were using spinlock and idr_preload.
We're actually fine with GFP_KERNEL (and we could just use mutex with
idr):
https://lore.kernel.org/dri-devel/20220823210612.296922-3-michal.winiarski@intel.com/

> 
> > >         if (r < 0)
> > >                 return r;
> > >
> > > -       minor->index = r;
> > > -
> > >         r = drmm_add_action_or_reset(dev, drm_minor_alloc_release, minor);
> > >         if (r)
> > >                 return r;
> > > @@ -152,7 +139,7 @@ static int drm_minor_alloc(struct drm_device *dev, unsigned int type)
> > >  static int drm_minor_register(struct drm_device *dev, unsigned int type)
> > >  {
> > >         struct drm_minor *minor;
> > > -       unsigned long flags;
> > > +       void *entry;
> > >         int ret;
> > >
> > >         DRM_DEBUG("\n");
> > > @@ -172,9 +159,12 @@ static int drm_minor_register(struct drm_device *dev, unsigned int type)
> > >                 goto err_debugfs;
> > >
> > >         /* replace NULL with @minor so lookups will succeed from now on */
> > > -       spin_lock_irqsave(&drm_minor_lock, flags);
> > > -       idr_replace(&drm_minors_idr, minor, minor->index);
> > > -       spin_unlock_irqrestore(&drm_minor_lock, flags);
> > > +       entry = xa_cmpxchg(&drm_minors_xa, minor->index, NULL, &minor, GFP_KERNEL);
> > I believe we should pass in "minor", without the &, as &minor will
> > give you the address of the local pointer.
> >
> > Oded
> >
> > > +       if (xa_is_err(entry)) {
> > > +               ret = xa_err(entry);
> > > +               goto err_debugfs;
> > > +       }
> > > +       WARN_ON(entry);
> > >
> > >         DRM_DEBUG("new minor registered %d\n", minor->index);
> > >         return 0;
> > > @@ -187,16 +177,13 @@ static int drm_minor_register(struct drm_device *dev, unsigned int type)
> > >  static void drm_minor_unregister(struct drm_device *dev, unsigned int type)
> > >  {
> > >         struct drm_minor *minor;
> > > -       unsigned long flags;
> > >
> > >         minor = *drm_minor_get_slot(dev, type);
> > >         if (!minor || !device_is_registered(minor->kdev))
> > >                 return;
> > >
> > >         /* replace @minor with NULL so lookups will fail from now on */
> > > -       spin_lock_irqsave(&drm_minor_lock, flags);
> > > -       idr_replace(&drm_minors_idr, NULL, minor->index);
> > > -       spin_unlock_irqrestore(&drm_minor_lock, flags);
> > > +       xa_store(&drm_minors_xa, minor->index, NULL, GFP_KERNEL);
> > >
> > >         device_del(minor->kdev);
> > >         dev_set_drvdata(minor->kdev, NULL); /* safety belt */
> > > @@ -215,13 +202,12 @@ static void drm_minor_unregister(struct drm_device *dev, unsigned int type)
> > >  struct drm_minor *drm_minor_acquire(unsigned int minor_id)
> > >  {
> > >         struct drm_minor *minor;
> > > -       unsigned long flags;
> > >
> > > -       spin_lock_irqsave(&drm_minor_lock, flags);
> > > -       minor = idr_find(&drm_minors_idr, minor_id);
> > > +       xa_lock(&drm_minors_xa);
> > > +       minor = xa_load(&drm_minors_xa, minor_id);
> Did you check this part ? Because this always gives me NULL...
> 
> I tried executing the following code in a dummy driver I wrote:
> 
> static DEFINE_XARRAY_ALLOC(xa_dummy);
> void check_xa(void *pdev)
> {
>   void *entry;
>   int ret, index;
> 
>   ret = xa_alloc(&xa_dummy, &index, NULL, XA_LIMIT(0, 63), GFP_NOWAIT);
>   if (ret < 0)
>       return ret;
> 
>   entry = xa_cmpxchg(&xa_dummy, index, NULL, pdev, GFP_KERNEL);
>   if (xa_is_err(entry))
>        return;
> 
>   xa_lock(&xa_dummy);
>   xa_dev = xa_load(&xa_dummy, index);
>   xa_unlock(&xa_dummy);
> }
> 
> And to my surprise xa_dev is always NULL, when it should be pdev.
> I believe that because we first allocate the entry with NULL, it is
> considered a "zero" entry in the XA.
> And when we replace it, this attribute doesn't change so when we do
> xa_load, the xa code thinks the entry is a "zero" entry and returns
> NULL.
> If that's correct, you need to either fix xarray code or change the
> flow of allocating this in drm.
> 
> If I send a real pointer (just a dummy object I allocated) instead of
> NULL in xa_alloc, and then do xa_cmpxchg with pdev, xa_load returns
> pdev successfully.
> That points to the NULL being problematic in allocating an entry.

Yeah, this was just using xa_store in previous revision.
https://lore.kernel.org/dri-devel/20220906201629.419160-2-michal.winiarski@intel.com/
And I didn't notice the CI failures:
https://patchwork.freedesktop.org/series/108206/
since I was originally hoping for comments on extending DRM minor limit,
XArray conversion was a bit of an extra.
Sorry about that - I just sent a next version using xa_store as a
separate patch (without extending DRM device limit).

Thanks!
-Michał

> 
> Oded
> 
> 
> > >         if (minor)
> > >                 drm_dev_get(minor->dev);
> > > -       spin_unlock_irqrestore(&drm_minor_lock, flags);
> > > +       xa_unlock(&drm_minors_xa);
> > >
> > >         if (!minor) {
> > >                 return ERR_PTR(-ENODEV);
> > > @@ -1037,7 +1023,7 @@ static void drm_core_exit(void)
> > >         unregister_chrdev(DRM_MAJOR, "drm");
> > >         debugfs_remove(drm_debugfs_root);
> > >         drm_sysfs_destroy();
> > > -       idr_destroy(&drm_minors_idr);
> > > +       WARN_ON(!xa_empty(&drm_minors_xa));
> > >         drm_connector_ida_destroy();
> > >  }
> > >
> > > @@ -1046,7 +1032,6 @@ static int __init drm_core_init(void)
> > >         int ret;
> > >
> > >         drm_connector_ida_init();
> > > -       idr_init(&drm_minors_idr);
> > >         drm_memcpy_init_early();
> > >
> > >         ret = drm_sysfs_init();
> > > --
> > > 2.37.3
> > >