Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\))
Subject: Re: [PATCH 4/4] Docs/RCU/rculist_nulls: Drop unnecessary '_release'
 in insert function
From:   Alan Huang <mmpgouride@gmail.com>
In-Reply-To: <20230610002024.80653-1-sj@kernel.org>
Date:   Sat, 10 Jun 2023 13:52:25 +0800
Cc:     "Paul E. McKenney" <paulmck@kernel.org>,
        Joel Fernandes <joel@joelfernandes.org>, corbet@lwn.net,
        rcu@vger.kernel.org, linux-doc@vger.kernel.org,
        linux-kernel@vger.kernel.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <6C469091-6C20-4BBD-B503-F024021C8AE7@gmail.com>
References: <20230610002024.80653-1-sj@kernel.org>
To:     SeongJae Park <sj@kernel.org>
Precedence: bulk

Hi SJ,

> 2023=E5=B9=B46=E6=9C=8810=E6=97=A5 08:20=EF=BC=8CSeongJae Park =
<sj@kernel.org> =E5=86=99=E9=81=93=EF=BC=9A
>=20
> On Fri, 9 Jun 2023 16:42:59 -0700 "Paul E. McKenney" =
<paulmck@kernel.org> wrote:
>=20
>> On Fri, Jun 09, 2023 at 07:12:06PM +0000, SeongJae Park wrote:
>>> On Fri, 19 May 2023 14:52:50 -0400 Joel Fernandes =
<joel@joelfernandes.org> wrote:
>>>=20
>>>> On Thu, May 18, 2023 at 6:40=E2=80=AFPM SeongJae Park =
<sj@kernel.org> wrote:
>>>>>=20
>>>>> The document says we can avoid extra smp_rmb() in =
lockless_lookup() and
>>>>> extra _release() in insert function when hlist_nulls is used.  =
However,
>>>>> the example code snippet for the insert function is still using =
the
>>>>> extra _release().  Drop it.
>>>>>=20
>>>>> Signed-off-by: SeongJae Park <sj@kernel.org>
>>>>> ---
>>>>> Documentation/RCU/rculist_nulls.rst | 2 +-
>>>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>=20
>>>>> diff --git a/Documentation/RCU/rculist_nulls.rst =
b/Documentation/RCU/rculist_nulls.rst
>>>>> index 5cd6f3f8810f..463270273d89 100644
>>>>> --- a/Documentation/RCU/rculist_nulls.rst
>>>>> +++ b/Documentation/RCU/rculist_nulls.rst
>>>>> @@ -191,7 +191,7 @@ scan the list again without harm.
>>>>>   obj =3D kmem_cache_alloc(cachep);
>>>>>   lock_chain(); // typically a spin_lock()
>>>>>   obj->key =3D key;
>>>>> -  atomic_set_release(&obj->refcnt, 1); // key before refcnt
>>>>> +  atomic_set(&obj->refcnt, 1);
>>>>>   /*
>>>>>    * insert obj in RCU way (readers might be traversing chain)
>>>>>    */
>>>>=20
>>>> If write to ->refcnt of 1 is reordered with setting of ->key, what
>>>> prevents the 'lookup algorithm' from doing a key match (obj->key =3D=3D=

>>>> key) before the refcount has been initialized?
>>>>=20
>>>> Are we sure the reordering mentioned in the document is the same as
>>>> the reordering prevented by the atomic_set_release()?
>>>=20
>>> Paul, may I ask your opinion?
>>=20
>> The next line of code is this:
>>=20
>> hlist_nulls_add_head_rcu(&obj->obj_node, list);
>>=20
>> If I understand the code correctly, obj (and thus *obj) are not
>> visible to readers before the hlist_nulls_add_head_rcu().  And
>> hlist_nulls_add_head_rcu() uses rcu_assign_pointer() to ensure that
>> initialization (including both ->key and ->refcnt) is ordered before
>> list insertion.
>>=20
>> Except that this memory is being allocated from a slab cache that was
>> created with SLAB_TYPESAFE_BY_RCU.  This means that there can be =
readers
>> who gained a reference before this object was freed, and who still =
hold
>> their references.
>>=20
>> Unfortunately, the implementation of try_get_ref() is not shown.  =
However,
>> if ->refcnt is non-zero, this can succeed, and if it succeeds, we =
need
>> the subsequent check of obj->key with key in the lookup algorithm to
>> be stable.  For this check to be stable, try_get_ref() needs to use =
an
>> atomic operation with at least acquire semantics =
(kref_get_unless_zero()
>> would work), and this must pair with something in the initialization.
>>=20
>> So I don't see how it is safe to weaken that atomic_set_release() to
>> atomic_set(), even on x86.
>=20
> Thank you for the nice explanation, and I agree.
>=20
>>=20
>> Or am I missing something subtle here?
>=20
> I found the text is saying extra _release() in insert function is not
> needed[1], and I thought it means the atomic_set_release().  Am I =
misreading
> it?  If not, would it be better to fix the text, for example, like =
below?

The original text is:

	=E2=80=9CWith hlist_nulls we can avoid extra smp_rmb() in =
lockless_lookup()
	 and extra smp_wmb() in insert function.=E2=80=9D

We can avoid the extra smp_wmb(), but the _release is required,

As Paul said,

>> Except that this memory is being allocated from a slab cache that was
>> created with SLAB_TYPESAFE_BY_RCU.  This means that there can be =
readers
>> who gained a reference before this object was freed, and who still =
hold
>> their references.

Without the _release, we can get the old =E2=80=98key=E2=80=99 after the =
invocation of
try_get_ref (although try_get_ref noticed the effect of atomic_set).

Thanks,
Alan

>=20
> ```
> --- a/Documentation/RCU/rculist_nulls.rst
> +++ b/Documentation/RCU/rculist_nulls.rst
> @@ -129,8 +129,7 @@ very very fast (before the end of RCU grace =
period)
> Avoiding extra smp_rmb()
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=

>=20
> -With hlist_nulls we can avoid extra smp_rmb() in lockless_lookup()
> -and extra _release() in insert function.
> +With hlist_nulls we can avoid extra smp_rmb() in lockless_lookup().
>=20
> For example, if we choose to store the slot number as the 'nulls'
> end-of-list marker for each slot of the hash table, we can detect
> @@ -182,6 +181,9 @@ scan the list again without harm.
> 2) Insert algorithm
> -------------------
>=20
> +Same to the above one, but uses hlist_nulls_add_head_rcu() instead of
> +hlist_add_head_rcu().
> +
> ::
>=20
>   /*
> @@ -191,7 +193,7 @@ scan the list again without harm.
>   obj =3D kmem_cache_alloc(cachep);
>   lock_chain(); // typically a spin_lock()
>   obj->key =3D key;
> -  atomic_set_release(&obj->refcnt, 1); // key before refcnt
> +  atomic_set(&obj->refcnt, 1);
>   /*
>    * insert obj in RCU way (readers might be traversing chain)
>    */
> ```
>=20
> [1] =
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Do=
cumentation/RCU/rculist_nulls.rst#n133
>=20
>=20
> Thanks,
> SJ
>=20
>>=20
>> Thanx, Paul
>>=20
>>> Thanks,
>>> SJ
>>>=20
>>>>=20
>>>> For the other 3 patches, feel free to add:
>>>> Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>
>>>>=20
>>>> thanks,
>>>>=20
>>>> - Joel