Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Praveen Kannoju <praveen.kannoju@oracle.com>
To:     Jason Gunthorpe <jgg@ziepe.ca>
CC:     Leon Romanovsky <leon@kernel.org>,
        Santosh Shilimkar <santosh.shilimkar@oracle.com>,
        "David S . Miller" <davem@davemloft.net>,
        "kuba@kernel.org" <kuba@kernel.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "rds-devel@oss.oracle.com" <rds-devel@oss.oracle.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Rama Nichanamatlu <rama.nichanamatlu@oracle.com>,
        Rajesh Sivaramasubramaniom 
        <rajesh.sivaramasubramaniom@oracle.com>
Subject: RE: [PATCH RFC] rds: ib: Reduce the contention caused by the
 asynchronous workers to flush the mr pool
Thread-Topic: [PATCH RFC] rds: ib: Reduce the contention caused by the
 asynchronous workers to flush the mr pool
Thread-Index: AQHYDHpMX8JHvK1k8kSJh1x30qsksqxo/YyAgAAprgCAAAb9AIAAvOYAgABQL4CAABYHAIAAAX/g
Date:   Wed, 19 Jan 2022 13:12:29 +0000
Message-ID: <PH0PR10MB551565CBAD2FF5CC0D3C69C48C599@PH0PR10MB5515.namprd10.prod.outlook.com>
References: <1642517238-9912-1-git-send-email-praveen.kannoju@oracle.com>
 <53D98F26-FC52-4F3E-9700-ED0312756785@oracle.com>
 <20220118191754.GG8034@ziepe.ca>
 <CEFD48B4-3360-4040-B41A-49B8046D28E8@oracle.com> <Yee2tMJBd4kC8axv@unreal>
 <PH0PR10MB5515E99CA5DF423BDEBB038E8C599@PH0PR10MB5515.namprd10.prod.outlook.com>
 <20220119130450.GJ8034@ziepe.ca>
In-Reply-To: <20220119130450.GJ8034@ziepe.ca>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?AoDG3mbs9Z4TEXO/uxMJRAaTUS3cMBzoEJYJ/wkAHi1jmQpPyUREyN3zG+5p?=
 =?us-ascii?Q?dvVsOcI1sl40N8/1uxyLoPUJPkPsW6QdnuJ8LG3CCT9TQDz8VYTJ2f6jQlJ4?=
 =?us-ascii?Q?qJMkqqN3/4bmTDhYaWM2qwPepOYm8Lv33RAxU1/tuF0s87Ar90roELadCdp7?=
 =?us-ascii?Q?/uHjLyIGq0F22NGiRNaxkNxsDJaxY2MHMwlAnol2yqTC1IM10hgxxHBDlHDv?=
 =?us-ascii?Q?3pRKhKX8r03FHIRjGbP5+S8NIx57P8zsiDjkb97ZhAI4RIeUwWQ7WjvL1F9k?=
 =?us-ascii?Q?+An5Klz7hd1AOErAEd0g4KB9AVhcWSh6cf4CZrH714BciycpgHIUSeFm16oK?=
 =?us-ascii?Q?hIooLu2D1G60v63y7HikqKc2Do7Rb+E3ld9WADU+RUBA42BosKHpEH7kZUEk?=
 =?us-ascii?Q?4jtjC3nedpUWpwtjTPb1YvUXrKidTJfUyv83Toyk2ge6qHcAlM173dvDzbFn?=
 =?us-ascii?Q?BwVCJRf6pgr27nScXk168ITLAW9kD89mbmqH3ot4OsulmlE5tfY53eUYjBj/?=
 =?us-ascii?Q?h5lJkgSOuPTP/3ja8C6bbNx1CKD0TsaYpReHFoaKEwj9Ph7XzPL8FoI1Uvhb?=
 =?us-ascii?Q?dnBr/TivUHSNBB6TGCwMxiLn6Y5sEhJnD0RREBkr3H+fo+yVdXredo6NZNXz?=
 =?us-ascii?Q?4ciBPiasj1CdWffJbdIhU9uedwAGv7oStZZujiZICqE/KCNLnnG74JofqvJK?=
 =?us-ascii?Q?L5gzBYzUJyvJJttgetKZRAyia5MY+92fBR/muEsdlRe1uN4QdpNCyl7Yj4iW?=
 =?us-ascii?Q?OxzJTAtV+YMVylcuA0kfeUqL4OUmGyCLPVuk78+PMO3mACqIOtQZzqJsFP5o?=
 =?us-ascii?Q?gdeU3nxFFl0ermEUMW7m7zq/2IucGCGGwDMxH/1TWwAmhKjQQbbLHG4+DQb3?=
 =?us-ascii?Q?UuvspE2jgwLVoR6OGMP4jc0nBD/w4sYaQgJLPl99hgfvidBbdWQt3v4VDpLJ?=
 =?us-ascii?Q?KNeDo8frN2rrJjMJiEG/wXWbGfbFGPlLHUYK+60kVS2RoTI50zZy08r/5mGs?=
 =?us-ascii?Q?rMfhO19BHJB/F/jEnVHud73dEvlrsQuEZZvZuvQgrDReH8AAtWbKK7juNnuP?=
 =?us-ascii?Q?YgKsOyG4279V3Sq6Dm7s2GKtgxzYra8YXFUt3LF/5abrpqP8alTpXeSAPi3C?=
 =?us-ascii?Q?OpP6s6HJwutxv4P60SgmGTt3WvO1HaA9YYpMWm69QIUq3W6lXC6fYyk7pLO6?=
 =?us-ascii?Q?2gr+49ViWyyHgQPUuPK9576OiXMjWvTeUYHFpR3PicnQ+uqTStgCXZC6uofR?=
 =?us-ascii?Q?5fGYgeRFl7cc0hL7DzfO2N/Y8UszZQtPXoTDlMbz81JUEDzWHlxBTev9xo+S?=
 =?us-ascii?Q?E3zEdc/d6bMzzsQVdo6u8r+PkXIjtt3/efg57pAddgcaqiyTRFyaA6p+FZXZ?=
 =?us-ascii?Q?zv1MAjbQv+wY8Pwhljsd4BHWa+9TiOtT2yoAQWte92AQqP7iB45+j50/qTCy?=
 =?us-ascii?Q?PMfyTi62NkAoxA7ttjYCgjNfR0LFnQRWAT/5qWKF8Nj/BII0UuskIn3Tqzdi?=
 =?us-ascii?Q?xwUGSLRZ+LFj1rES221PIbv9gltwmSWYb2hSpMmKp5pqJRmaMOKdOlXZO6zc?=
 =?us-ascii?Q?ENNTBXnlR/j23eemWC3dmNb2C58PXG8K43boPPfPxqVbjcpchv+iWHgzuJ/D?=
 =?us-ascii?Q?9m3vK0il0A5iVytKMjosCWXmDKDy6sv0caWhkkNqJJuA/5ngboBo+lK3nUL1?=
 =?us-ascii?Q?OmlzXA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR10MB5515.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 313b27a8-33a0-424e-8351-08d9db4d58e7
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Jan 2022 13:12:29.9230
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: aFKrt00wGBVhiHulGm1OcaRTElCA8aVnOhY79sFtCXIW5Jm4hSmRV9OA0gYt1tPpkTaEpJ/KIx/Vl6fuCAyMJaR3/V41xI4ro2khyS8NOhI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2775
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10231 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0
 mlxlogscore=999 adultscore=0 suspectscore=0 malwarescore=0 mlxscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2110150000 definitions=main-2201190076
X-Proofpoint-GUID: XOh0Gxm_PvqUG19x87V1uI1iHhNGiFOj
X-Proofpoint-ORIG-GUID: XOh0Gxm_PvqUG19x87V1uI1iHhNGiFOj
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


-----Original Message-----
From: Jason Gunthorpe [mailto:jgg@ziepe.ca]=20
Sent: 19 January 2022 06:35 PM
To: Praveen Kannoju <praveen.kannoju@oracle.com>
Cc: Leon Romanovsky <leon@kernel.org>; Santosh Shilimkar <santosh.shilimkar=
@oracle.com>; David S . Miller <davem@davemloft.net>; kuba@kernel.org; netd=
ev@vger.kernel.org; linux-rdma@vger.kernel.org; rds-devel@oss.oracle.com; l=
inux-kernel@vger.kernel.org; Rama Nichanamatlu <rama.nichanamatlu@oracle.co=
m>; Rajesh Sivaramasubramaniom <rajesh.sivaramasubramaniom@oracle.com>
Subject: Re: [PATCH RFC] rds: ib: Reduce the contention caused by the async=
hronous workers to flush the mr pool

On Wed, Jan 19, 2022 at 11:46:16AM +0000, Praveen Kannoju wrote:
=20
> 6. Jason, the only function "rds_ib_free_mr" which accesses the=20
> introduced bool variable "flush_ongoing" to spawn a flush worker does=20
> not crucially impact the availability of MR's, because the flush=20
> happens from allocation path as well when necessary.  Hence the=20
> Load-store ordering is not essentially needed here, because of which=20
> we chose smp_rmb() and smp_wmb() over smp_load_acquire() and=20
> smp_store_release().

That seems like a confusing statement, you added barriers which do the same=
 things as acquire/release then say you didn't need acquire release?

I think this is using barriers wrong.

Jason

Jason,=20

Yes, we are using the barriers. I was justifying the usage of smp_rmb() and=
 smp_wmb() over smp_load_acquire() and smp_store_release() in the patch.=20