Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Anand Khoje <anand.a.khoje@oracle.com>
To:     Leon Romanovsky <leon@kernel.org>
CC:     "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "dledford@redhat.com" <dledford@redhat.com>,
        "jgg@ziepe.ca" <jgg@ziepe.ca>,
        Haakon Bugge <haakon.bugge@oracle.com>
Subject: RE: [PATCH v3 3/3] IB/core: Obtain subnet_prefix from cache in IB
 devices.
Thread-Topic: [PATCH v3 3/3] IB/core: Obtain subnet_prefix from cache in IB
 devices.
Thread-Index: AQHXXPQc/GlUWUBsN0yos674dzNO5asLWyGAgAAM10A=
Date:   Wed, 9 Jun 2021 09:26:03 +0000
Message-ID: <MWHPR1001MB2096CA7F29DCF86DE921903EC5369@MWHPR1001MB2096.namprd10.prod.outlook.com>
References: <20210609055534.855-1-anand.a.khoje@oracle.com>
 <20210609055534.855-4-anand.a.khoje@oracle.com> <YMB9gxlKbDvdynUE@unreal>
In-Reply-To: <YMB9gxlKbDvdynUE@unreal>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?fxZhvaUYvmvkellXgX11XOmbEaTRupyY+eT29cuT2zxa9YmhF3cHqcTt4lrZ?=
 =?us-ascii?Q?BCmpB9KzxSpjN0r5javr7gsXCJLpRAllnRS6XjMg0n7WNxJp1wi7vvMF+RoA?=
 =?us-ascii?Q?wk63ZXdO13d0lTpW1p9C9iKMTKJDSUiYxnXiP5W7t932nUjVwSS9Hc+8mt1m?=
 =?us-ascii?Q?kWKKksPervlKJy/hxFCPh/XkkyF/y86DhoYXpZEkyOsdW0QaeCsDePBgkLIj?=
 =?us-ascii?Q?iQwO2HvTt3zUBPGjjHisIksqNVm3vesi/2Uu211jiyOOs/rB2B+nRgF+utA1?=
 =?us-ascii?Q?RXoIDL4kQ2CJni1dH6o2Y/Q9D0wepK1X5BCa/0DD169DjTrK73W2tf3CINi1?=
 =?us-ascii?Q?CrrZubdopFHpRWJeM9GrWvnwMMA8ZGzCvXWiAA8OjAmiwfBS9et0I+I28WON?=
 =?us-ascii?Q?5Bn5yvNVLBAa5n8VPw85iBhl5vnglmyr+xMZldAtKb2R7v8Sg/NXLiaP2s17?=
 =?us-ascii?Q?DJZ4WsM9LOHFt8GGmr/knKuRAx9lJAR9VnYrPqxdn1k3dKtoCZqchg9LG1MW?=
 =?us-ascii?Q?/RehQfd5bmutecAjV94oO7Ic4u/8cfO9KvR/qR8t22aBr8VILGObzhTxh/WW?=
 =?us-ascii?Q?vXEkoFX9MWR9h6jkYwJqHZaayd3jzChVpuJXbXtfaWcNLE8j+GK97Fp7HPB5?=
 =?us-ascii?Q?nYLsJbcN7P4H/+7htBwsshp3nexTw/0TtNvmxvnRyqFLhNKMvLU8osOFBNiR?=
 =?us-ascii?Q?y47D4Qd/Wq9QpUQC7Wnnrpu/vNMdx6FiVdo8Pcox7mVRHUYWqxy3nU41fWHd?=
 =?us-ascii?Q?3TvnnZMkm7uxeMxMzGv+DXxtx1cfOIGxcvfHTlDapWQFXiLegSIov76+tmP/?=
 =?us-ascii?Q?PnawkL8amaU2Swh2wA1YVF1Z6Mjj1vqeVC07xcke3U2FaDkdgMbvaBHF6E2r?=
 =?us-ascii?Q?Uu4qRo8yzdV2Xz66kWFamkvO23gbdkfAA3PZYdrBEGcOydTl7cwrFdZBVY+F?=
 =?us-ascii?Q?3cIIzm6n8gJMGMz9fWqSXgV9d8CzkDBXEGNHrpLe+PWCrUHjHA2YTCy83fsZ?=
 =?us-ascii?Q?x1fMqTFQ1tMwopCKbOcfR0OVG2mR8yfVkKEEn1CfIbhSTGnZh5aOJ3z4lzKK?=
 =?us-ascii?Q?xdA8cWGpgbfEPACyGhguu6Hw9ClLCGLTTXXo9EzWBGL4LZ8I/bu3RY6lH3Md?=
 =?us-ascii?Q?pXbAMlA8YOI+HCUPBQgCn6FAMw/YLgxB42ewjbkSpLLkRmSlj7bvDIc2zls/?=
 =?us-ascii?Q?Xgl6UDSknHvKHdAQtFUe75s5jNRIeJnfAvNd1Tc0TgjHNS6C7ViN5SM8Tn6E?=
 =?us-ascii?Q?64Hp9xXKuGhw4PetdXFOfLyD7BG9CT74RTWcNI9fl7xbRuYgHdF/3AbP9GrV?=
 =?us-ascii?Q?4YU0a+5zetzr5daGnhIuNc7V?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1001MB2096.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ec3a011d-f35c-47ec-8e74-08d92b289a7d
X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jun 2021 09:26:03.8882
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: sEE0EMEPpdNW10nM4A6iDocntYdt9ccIWDxtCAnnZRgPoXohSsnByMRHqLg6btiIGtlCiy+mjC/HfG80+3hTEUZtdfMlSnV665UEU3NPiuc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1998
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10009 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 mlxscore=0
 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000
 definitions=main-2106090042
X-Proofpoint-GUID: hm1-Fk4vP0SLmLV4Ke7ORoKoC2d-CKNJ
X-Proofpoint-ORIG-GUID: hm1-Fk4vP0SLmLV4Ke7ORoKoC2d-CKNJ
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10009 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 spamscore=0
 mlxlogscore=999 phishscore=0 impostorscore=0 suspectscore=0 clxscore=1015
 mlxscore=0 malwarescore=0 priorityscore=1501 lowpriorityscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000
 definitions=main-2106090042
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Leon,

The set_bit()/clear_bit() and enum ib_port_data_flags  has been added as a =
device that can be used for future enhancements.=20
Also, usage of set_bit()/clear_bit() ensures the operations on this bit is =
atomic.

Thanks,
Anand

-----Original Message-----
From: Leon Romanovsky <leon@kernel.org>=20
Sent: Wednesday, June 9, 2021 2:06 PM
To: Anand Khoje <anand.a.khoje@oracle.com>
Cc: linux-rdma@vger.kernel.org; linux-kernel@vger.kernel.org; dledford@redh=
at.com; jgg@ziepe.ca; Haakon Bugge <haakon.bugge@oracle.com>
Subject: Re: [PATCH v3 3/3] IB/core: Obtain subnet_prefix from cache in IB =
devices.

On Wed, Jun 09, 2021 at 11:25:34AM +0530, Anand Khoje wrote:
> ib_query_port() calls device->ops.query_port() to get the port=20
> attributes. The method of querying is device driver specific.
> The same function calls device->ops.query_gid() to get the GID and=20
> extract the subnet_prefix (gid_prefix).
>=20
> The GID and subnet_prefix are stored in a cache. But they do not get=20
> read from the cache if the device is an Infiniband device. The=20
> following change takes advantage of the cached subnet_prefix.
> Testing with RDBMS has shown a significant improvement in performance=20
> with this change.
>=20
> The function ib_cache_is_initialised() is introduced because
> ib_query_port() gets called early in the stage when the cache is not=20
> built while reading port immutable property.
>=20
> In that case, the default GID still gets read from HCA for IB link-=20
> layer devices.
>=20
> Fixes: fad61ad ("IB/core: Add subnet prefix to port info")
> Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
> Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
>=20
> ---
>=20
> v1 -> v2:
>     -	Split the v1 patch in 3 patches as per Leon's suggestion.
>=20
> v2 -> v3:
>     -	Added changes as per Mark Zhang's suggestion of clearing
>     	flags in git_table_cleanup_one().
>=20
> ---
>  drivers/infiniband/core/cache.c  | 7 ++++++- =20
> drivers/infiniband/core/device.c | 9 +++++++++
>  include/rdma/ib_cache.h          | 6 ++++++
>  include/rdma/ib_verbs.h          | 6 ++++++
>  4 files changed, 27 insertions(+), 1 deletion(-)

Why did you use clear_bit/test_bit API? I would expect it for the bitmap, b=
ut for such simple thing, the simple "u8 is_cached_init : 1;"
will do the same trick.

Thanks

>=20
> diff --git a/drivers/infiniband/core/cache.c=20
> b/drivers/infiniband/core/cache.c index e957f0c915a3..94a8653a72c5=20
> 100644
> --- a/drivers/infiniband/core/cache.c
> +++ b/drivers/infiniband/core/cache.c
> @@ -917,9 +917,12 @@ static void gid_table_cleanup_one(struct=20
> ib_device *ib_dev)  {
>  	u32 p;
> =20
> -	rdma_for_each_port (ib_dev, p)
> +	rdma_for_each_port (ib_dev, p) {
> +		clear_bit(IB_PORT_CACHE_INITIALIZED,
> +			&ib_dev->port_data[p].flags);
>  		cleanup_gid_table_port(ib_dev, p,
>  				       ib_dev->port_data[p].cache.gid);
> +	}
>  }
> =20
>  static int gid_table_setup_one(struct ib_device *ib_dev) @@ -1623,6=20
> +1626,8 @@ int ib_cache_setup_one(struct ib_device *device)
>  		err =3D ib_cache_update(device, p, true);
>  		if (err)
>  			return err;
> +		set_bit(IB_PORT_CACHE_INITIALIZED,
> +			&device->port_data[p].flags);
>  	}
> =20
>  	return 0;
> diff --git a/drivers/infiniband/core/device.c=20
> b/drivers/infiniband/core/device.c
> index 595128b26c34..e8e7b0a61411 100644
> --- a/drivers/infiniband/core/device.c
> +++ b/drivers/infiniband/core/device.c
> @@ -2059,6 +2059,15 @@ static int __ib_query_port(struct ib_device *devic=
e,
>  	    IB_LINK_LAYER_INFINIBAND)
>  		return 0;
> =20
> +	if (!ib_cache_is_initialised(device, port_num))
> +		goto query_gid_from_device;
> +
> +	ib_get_cached_subnet_prefix(device, port_num,
> +				    &port_attr->subnet_prefix);
> +
> +	return 0;
> +
> +query_gid_from_device:
>  	err =3D device->ops.query_gid(device, port_num, 0, &gid);
>  	if (err)
>  		return err;
> diff --git a/include/rdma/ib_cache.h b/include/rdma/ib_cache.h index=20
> 226ae3702d8a..1526fc6637eb 100644
> --- a/include/rdma/ib_cache.h
> +++ b/include/rdma/ib_cache.h
> @@ -114,4 +114,10 @@ ssize_t rdma_query_gid_table(struct ib_device *devic=
e,
>  			     struct ib_uverbs_gid_entry *entries,
>  			     size_t max_entries);
> =20
> +static inline bool ib_cache_is_initialised(struct ib_device *device,
> +					  u8 port_num)
> +{
> +	return test_bit(IB_PORT_CACHE_INITIALIZED,
> +			&device->port_data[port_num].flags);
> +}
>  #endif /* _IB_CACHE_H */
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index=20
> 41cbec516424..ad2a55e3a2ee 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -2169,6 +2169,10 @@ struct ib_port_immutable {
>  	u32                           max_mad_size;
>  };
> =20
> +enum ib_port_data_flags {
> +	IB_PORT_CACHE_INITIALIZED =3D 1 << 0,
> +};
> +
>  struct ib_port_data {
>  	struct ib_device *ib_dev;
> =20
> @@ -2178,6 +2182,8 @@ struct ib_port_data {
> =20
>  	spinlock_t netdev_lock;
> =20
> +	unsigned long flags;
> +
>  	struct list_head pkey_list;
> =20
>  	struct ib_port_cache cache;
> --
> 2.27.0
>=20