Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\))
Subject: Re: [PATCH 4/4] Docs/RCU/rculist_nulls: Drop unnecessary '_release'
 in insert function
From:   Alan Huang <mmpgouride@gmail.com>
In-Reply-To: <46440869-644a-4982-b790-b71b43976c66@paulmck-laptop>
Date:   Sat, 10 Jun 2023 13:37:28 +0800
Cc:     SeongJae Park <sj@kernel.org>,
        Joel Fernandes <joel@joelfernandes.org>, corbet@lwn.net,
        rcu@vger.kernel.org, linux-doc@vger.kernel.org,
        linux-kernel@vger.kernel.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <4071FE01-BEB5-4BEE-A424-F50F0E0E3410@gmail.com>
References: <CAEXW_YQFqW2QcAuHZEhc_GaUaB-=QOS0WgUOizd=FYwtFQ8vag@mail.gmail.com>
 <20230609191206.30465-1-sj@kernel.org>
 <46440869-644a-4982-b790-b71b43976c66@paulmck-laptop>
To:     paulmck@kernel.org
Precedence: bulk

Hi Paul,

> 2023=E5=B9=B46=E6=9C=8810=E6=97=A5 07:42=EF=BC=8CPaul E. McKenney =
<paulmck@kernel.org> =E5=86=99=E9=81=93=EF=BC=9A
>=20
> On Fri, Jun 09, 2023 at 07:12:06PM +0000, SeongJae Park wrote:
>> On Fri, 19 May 2023 14:52:50 -0400 Joel Fernandes =
<joel@joelfernandes.org> wrote:
>>=20
>>> On Thu, May 18, 2023 at 6:40=E2=80=AFPM SeongJae Park =
<sj@kernel.org> wrote:
>>>>=20
>>>> The document says we can avoid extra smp_rmb() in lockless_lookup() =
and
>>>> extra _release() in insert function when hlist_nulls is used.  =
However,
>>>> the example code snippet for the insert function is still using the
>>>> extra _release().  Drop it.
>>>>=20
>>>> Signed-off-by: SeongJae Park <sj@kernel.org>
>>>> ---
>>>> Documentation/RCU/rculist_nulls.rst | 2 +-
>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>=20
>>>> diff --git a/Documentation/RCU/rculist_nulls.rst =
b/Documentation/RCU/rculist_nulls.rst
>>>> index 5cd6f3f8810f..463270273d89 100644
>>>> --- a/Documentation/RCU/rculist_nulls.rst
>>>> +++ b/Documentation/RCU/rculist_nulls.rst
>>>> @@ -191,7 +191,7 @@ scan the list again without harm.
>>>>   obj =3D kmem_cache_alloc(cachep);
>>>>   lock_chain(); // typically a spin_lock()
>>>>   obj->key =3D key;
>>>> -  atomic_set_release(&obj->refcnt, 1); // key before refcnt
>>>> +  atomic_set(&obj->refcnt, 1);
>>>>   /*
>>>>    * insert obj in RCU way (readers might be traversing chain)
>>>>    */
>>>=20
>>> If write to ->refcnt of 1 is reordered with setting of ->key, what
>>> prevents the 'lookup algorithm' from doing a key match (obj->key =3D=3D=

>>> key) before the refcount has been initialized?
>>>=20
>>> Are we sure the reordering mentioned in the document is the same as
>>> the reordering prevented by the atomic_set_release()?
>>=20
>> Paul, may I ask your opinion?
>=20
> The next line of code is this:
>=20
> hlist_nulls_add_head_rcu(&obj->obj_node, list);
>=20
> If I understand the code correctly, obj (and thus *obj) are not
> visible to readers before the hlist_nulls_add_head_rcu().  And
> hlist_nulls_add_head_rcu() uses rcu_assign_pointer() to ensure that
> initialization (including both ->key and ->refcnt) is ordered before
> list insertion.
>=20
> Except that this memory is being allocated from a slab cache that was
> created with SLAB_TYPESAFE_BY_RCU.  This means that there can be =
readers
> who gained a reference before this object was freed, and who still =
hold
> their references.
>=20
> Unfortunately, the implementation of try_get_ref() is not shown.  =
However,
> if ->refcnt is non-zero, this can succeed, and if it succeeds, we need
> the subsequent check of obj->key with key in the lookup algorithm to
> be stable.  For this check to be stable, try_get_ref() needs to use an
> atomic operation with at least acquire semantics =
(kref_get_unless_zero()
> would work), and this must pair with something in the initialization.
>=20
> So I don't see how it is safe to weaken that atomic_set_release() to
> atomic_set(), even on x86.

I totally agree, but only in the case of using hlist_nulls.

That means, atomic_set_release() is not enough in the case without using =
hlist_nulls,
we must ensure that storing to obj->next (in hlist_add_head_rcu) is =
ordered before storing
to obj->key. Otherwise, we can get the new =E2=80=98next' and the old =
=E2=80=98key' in which case we can=E2=80=99t detect
an object movement(from one chain to another).

So, I=E2=80=99m afraid that the atomic_set_release() in insertion =
algorithm without using hlist_nulls should=20
change back to:
=09
	smp_wmb();
	atomic_set(&obj->refcnt, 1);

Thanks,
Alan

>=20
> Or am I missing something subtle here?
>=20
> Thanx, Paul
>=20
>> Thanks,
>> SJ
>>=20
>>>=20
>>> For the other 3 patches, feel free to add:
>>> Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
>>>=20
>>> thanks,
>>>=20
>>> - Joel