Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Parav Pandit <parav@nvidia.com>
To:     Arnd Bergmann <arnd@arndb.de>,
        "stern@rowland.harvard.edu" <stern@rowland.harvard.edu>,
        "parri.andrea@gmail.com" <parri.andrea@gmail.com>,
        Will Deacon <will@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        "boqun.feng@gmail.com" <boqun.feng@gmail.com>,
        Nicholas Piggin <npiggin@gmail.com>,
        "dhowells@redhat.com" <dhowells@redhat.com>,
        "j.alglave@ucl.ac.uk" <j.alglave@ucl.ac.uk>,
        "luc.maranget@inria.fr" <luc.maranget@inria.fr>,
        "Paul E. McKenney" <paulmck@kernel.org>,
        "akiyks@gmail.com" <akiyks@gmail.com>,
        Dan Lustig <dlustig@nvidia.com>,
        "joel@joelfernandes.org" <joel@joelfernandes.org>,
        Jonathan Corbet <corbet@lwn.net>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Linux-Arch <linux-arch@vger.kernel.org>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>
Subject: RE: [PATCH] locking/memory-barriers.txt: Improve documentation for
 writel() usage
Thread-Topic: [PATCH] locking/memory-barriers.txt: Improve documentation for
 writel() usage
Thread-Index: AQHYyMA9YvMyHkwA3EyvMDv7VnmV1K3gbnaAgAAIXgA=
Date:   Thu, 15 Sep 2022 14:18:15 +0000
Message-ID: <PH0PR12MB5481192DB7B5C6E19683D514DC499@PH0PR12MB5481.namprd12.prod.outlook.com>
References: <20220915050106.650813-1-parav@nvidia.com>
 <96457b14-e196-4f29-be9a-7fa25ac805d9@www.fastmail.com>
In-Reply-To: <96457b14-e196-4f29-be9a-7fa25ac805d9@www.fastmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Ibt43f1gkSpaGW/djU/XnB4sQ+Wy4coJU4YIXMGrvtVikEXYB12nZxkgF35Y?=
 =?us-ascii?Q?qalHc2Z+JeUfZzV0PFlB+KHpIZZzbgZJBJJZ/evC7RKPVgVTN4bauLPrOtFj?=
 =?us-ascii?Q?GvCNzBEWhdNzvQh+eikN8XFWniEfRmngOV4M1hQ87wQ5uIHJbnFiM1YpIB0Q?=
 =?us-ascii?Q?GkLeDzRhFBbR7EThS6mI38VD5Asvn5S25salhvsLdxmYZ5fuE+MdjJQOUFur?=
 =?us-ascii?Q?oT+6XoTBevYLt1fiJx5MaSRB3O4jZbw80RBYcm9eSCK6jCmz4/xj33x29ft3?=
 =?us-ascii?Q?Kc0S4Qu98wjeMWgB3d2QlY+Qlz5ozIPLVgpNX1j9w+brH9JSRalcB17P5xoc?=
 =?us-ascii?Q?+eZlbgGdv55X7hP3FIi8WMUtiGg9QTW7d7djYN5yXxoHN2pkPZR8Nl48taCr?=
 =?us-ascii?Q?HJXdcyf6CRyNGxLnuxnDd67kJMI6S0k2L2OC/8iPh/9gLiYLJZUHSyHWlpjO?=
 =?us-ascii?Q?r+RbiIS4YEk0Kv0F3uuiYD7vjBBw5qTZyX/5JJRtoVMoRay8GAMlQSVI9l1o?=
 =?us-ascii?Q?CxH3qSzovbFjhBPhlg4VIDjPzhyu9GgK9HGwRRD46RFz74iy6Q1qavxS2sqY?=
 =?us-ascii?Q?RSkUNvJ2ahbXXjHw6OnHJyFQueCPi52QU61R0zCWuEdwuSiFJPcmn239nTm1?=
 =?us-ascii?Q?A72PR4EeUzU+fyfWeI7molOCXr2QybbQ5njRpiWu2T5+bxMZbjSIvK1LQnxu?=
 =?us-ascii?Q?4YijEz2C0Am7ra7otlhryBc/4EZia7MED9UhuppfnaD5Fga8WCaOzUNx5rgn?=
 =?us-ascii?Q?42d14nry4PYagz58J5om6sKy9gTDpGlZ1s5YjSHDp1LajxxRxeptfb8xup3t?=
 =?us-ascii?Q?7Xa4tD1q6moqlCRvVdBA5zWQjLam0CzgYJKI/QoMGUE9DV4fioih8wFB1gwM?=
 =?us-ascii?Q?PdkxpOMFM11ZHXicLxmO24BCWUkqSMLT1Wlrz6o7ckjuZ66CnIv68rFWgOp2?=
 =?us-ascii?Q?GJ7SINcvXc7B9HeHXCOWcg/cN1woerXiWfcrAyx3Slf5WUTMQmhymkvoFvKZ?=
 =?us-ascii?Q?G3UiNnE4P75wZo0bKLMgfNpZ0KkBzO54LBSkBpEN0RFsRHXPUCAIfaOC+/KD?=
 =?us-ascii?Q?cp5Rj/IhhJuBzyAKtlJMTa05V3DQDWiewV/W6Vak+jFUpEHkpfr7MDNaOksU?=
 =?us-ascii?Q?DrxlekGNXqpj89a1TsNCV+59Xg1VO9BtCzYceWB8x1T4cfjkPd43MX/0MiIZ?=
 =?us-ascii?Q?9UOzlvlYz8Llv8MXYLR58op/l06yrbWZC3NfF2mH/76a8TAaSX3gYHcneFPC?=
 =?us-ascii?Q?atbErGs938i54jwRNT+MmVcCd19wDDgPBLMnyPvVUsziRO4HyOBAn2djTO8R?=
 =?us-ascii?Q?FkKPYGnghhSHwOighLzthxZcSobt9THmphDH3xDIxInJr4KlUeBATNENoh+q?=
 =?us-ascii?Q?hQRj1ubsl9gAUlXRTmanDesJnCLGlxOIXY4ewnlBFUvf2Azy8ns80+4Kc89J?=
 =?us-ascii?Q?VMGbrGJ5i7kipSdU/yLSOyCYDqvDrInuLZ8EfTqs9zj0jr+atMi3sTtoNPnQ?=
 =?us-ascii?Q?XX4ScyM5VTSiWFyZIgvCnZISKDRO/wNdw3O4z+OMRgWyqMVD9uDsJDE9i1z7?=
 =?us-ascii?Q?HBzCFSs0aC5uTHRIGQ4=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR12MB5481.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9c33f1e7-4004-4c49-15c8-08da97252194
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Sep 2022 14:18:15.8814
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: YwOJM9lbTeta4dUkQ4dTVTV15pXcbML9ziBgSCxWqnJJWmThGtSNyIP/KuKPFwmBtFe+R1XroKVqFlqpK+bM9w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6727
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> From: Arnd Bergmann <arnd@arndb.de>
> Sent: Thursday, September 15, 2022 8:38 AM
>=20
> On Thu, Sep 15, 2022, at 7:01 AM, Parav Pandit wrote:
> > The cited commit [1] describes that when using writel(), explcit wmb()
> > is not needed. However, it should have said that dma_wmb() is not
> > needed.
>=20
> Are you sure? As I understand it, the dma_wmb() only serializes a set of
> memory accesses, but does not serialized against an MMIO access, which
> depending on the CPU architecture may require a different type of barrier=
.
>=20
> E.g. on arm, writel() uses __iowmb(), which like wmb() is defined as "dsb=
(x);
> arm_heavy_mb();", while dma_wmb() is a "dmb(oshst)".

You are right, on arm heavy barrier dsb() is needed, while on arm64, dmb(os=
hst) is sufficient.

So more accurate documentation is to say that=20
'when using writel() a prior IO barrier is not needed ...'

How about that?

It started with my cleanup efforts to two drivers [1] and [2] that had diff=
iculty in using writel() on 32-bit system, and it ended up open coding writ=
el() as wmb() + mlx5_write64().

I am cleaning up the repetitive pattern of,=20
wmb();
mlx5_write64()

Before I fix drivers, I thought to improve the documentation that I can fol=
low. :)

[1] https://elixir.bootlin.com/linux/latest/source/drivers/infiniband/hw/ml=
x5/wr.c#L1042
[2] https://elixir.bootlin.com/linux/latest/source/drivers/net/ethernet/mel=
lanox/mlx5/core/en/txrx.h#L226