Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Anand Khoje <anand.a.khoje@oracle.com>
To:     linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org
Cc:     dledford@redhat.com, jgg@ziepe.ca, haakon.bugge@oracle.com,
        leon@kernel.org
Subject: [PATCH v7 for-next 2/2] IB/core: Read subnet_prefix in ib_query_port via cache.
Date:   Mon, 28 Jun 2021 14:59:20 +0530
Message-Id: <20210628092920.1088-3-anand.a.khoje@oracle.com>
In-Reply-To: <20210628092920.1088-1-anand.a.khoje@oracle.com>
References: <20210628092920.1088-1-anand.a.khoje@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from AAKHOJE-T480.in.oracle.com (49.36.45.114) by TY2PR06CA0011.apcprd06.prod.outlook.com (2603:1096:404:42::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4264.18 via Frontend Transport; Mon, 28 Jun 2021 09:29:40 +0000
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: b68a413b-ab26-42b3-4161-08d93a17427c
X-MS-TrafficTypeDiagnostic: CY4PR10MB1607:
X-MS-Exchange-Transport-Forked: True
X-Microsoft-Antispam-PRVS: <CY4PR10MB1607CC4FE021122FFA2D3D1BC5039@CY4PR10MB1607.namprd10.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:5516;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: VvQDy8poWZw2eJ8WBrO/A8SKYGx1RHAYZ2gedo6LYAVz9/JPaonpWrRhXO4/0TCgFCFBU2c/GEto+WhbdUyzQZtz3m562BC9uu9iCZWyg5ifsWVuzPMLIcOtB5DEgavyGM58ffWA4O1zXb1e4ihJLsvKgAdIHJl89zd4Yjcb7g1RXxBnWAVv59i3Eg0dxQjcPBanbVCOKh1eftiKJdUv2SrUO+rvoYOznkOyl1/6C2UJdvlJaQ2wB1crlve47afacIN56NNzSMVFGlOqWXAPweP1+TNXYc/LIkbUoFIjQXqmoTcF8SzkFeKTw6aA0KXoqaJnDgWcLTNsN0nwlvNa5RT7ZzwSdCRP5gXhrXH9VHfD5eKupC58TFiXAzDdlRthioFQO8THXQuTCPnA9gd5+qL7lX1rw0zCfwayEp7fGodfH5zUtETUHe2MmOGWpDOHlQlEA/6Qx1M49Q5NowAyAUmsLMLN52+qYXVpRAkvLAkzoUNcCUdD3M1Ur7slerHINhM8vabVKxZje03EAQE6vLxNk8+DtwtT+5iuAoDyzDs8V4QuOlKvg/nG44ToWEJVlB07zSJVb5h9IKHUKLs9iMe2Q4qDeLJxIzZBhlBBwFeFGic5kU7vi/XN5O/ftHQSJAHX4uHocXh1CNznpbpM+WQllIp9HK8D/X3i6EyFh4sh3fUlaYoXnRoZWgj7s3K19S/BAh5a9A0wTD1L+edcSg==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY4PR1001MB2086.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(396003)(376002)(346002)(366004)(136003)(6486002)(8676002)(956004)(36756003)(2616005)(4326008)(7696005)(103116003)(478600001)(8936002)(5660300002)(2906002)(26005)(52116002)(83380400001)(1076003)(66946007)(186003)(16526019)(66556008)(66476007)(86362001)(38350700002)(6666004)(316002)(1006002)(38100700002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?RKGzwU43e3i1zHKi1lgx8y4OD2y2X7a+x4HWUY1VJqk/fkTWEqglPsxke6ly?=
 =?us-ascii?Q?qmvqqOasZGUZ/KJHLbpM20/4p/432XiBzDo524CiSWjKeARvX7IG/nbgxU53?=
 =?us-ascii?Q?JZ6puqQNE020obTOUbh0aYrPpTQqeNnV5iiO/GhzYNqulNkKHW6NxxFA0bit?=
 =?us-ascii?Q?2Lq8EWlzKJdejwkl7GoDDzbw/GY0riFaA5OtuDkNvuMqpQzdyi2ti3zQDBWy?=
 =?us-ascii?Q?4xt22dtNknaA2A1apHQ3cFq9VhIzCxx6IzaIc8utHgm/sT4bVMF30IMfVtdq?=
 =?us-ascii?Q?czgYlt103Mhp/7tJXnXLzFW6pLr0qoHfDKQfCxNaPCK8QThThsPR763LpaKv?=
 =?us-ascii?Q?Jo7GvxCQBhCSWXgUX1Lqg/AfvqiLSy8btAaVQP44Mv+B5ISmIVKwRWEmOrtH?=
 =?us-ascii?Q?fCmnSG3dqFxebrcms3BESKRTlyYJV3Itko1g+/xeb21YK5ZEz2oZvDhIgzRM?=
 =?us-ascii?Q?HQageYUjOYbQAYFwYVyG6xkshUXs8DHk55dEonyBfxNpEhl4Q2gfSDVbVaGE?=
 =?us-ascii?Q?17ciqkGl84RYkggtKg/4Riokw12OwPf0VSEcBHcLJlBGRrBrzuu6fTTexNLL?=
 =?us-ascii?Q?jDm/FPGnINvbsxCoTuewa103AUsisEbJ/9zP7AYLWIn7LvZkqH28Zf61cd6F?=
 =?us-ascii?Q?xgkKBk262ovxDVqPZwBQtNUMyugYkCtPHi5shzYw4oK0qk4PKwKLsSgZewzI?=
 =?us-ascii?Q?efc0yBpMRA+42NZWalL9/EsxAtNS5mdEgjZHiJDlXqZ4dqJqMqrzAkX0Trek?=
 =?us-ascii?Q?4Tt10OCYH+IBkUSBoDuqLioNazHGMDrv2zdVYyPWxJxToDx007nB852PYiPy?=
 =?us-ascii?Q?IX9ZWKKk2BZZnScYPsyD5gBTANl8Ume6nHorweSolBCLCQdZd9a5lfnPIt6y?=
 =?us-ascii?Q?owbuoy8C6N6Ege+LqYg39x+uEARVSEcjENXw5rHwsE+f24vpLsWiqTTS7cMY?=
 =?us-ascii?Q?ybJ/LAKm+Kvt5aRGMY7tl80+kPF+NMZrpYQe5FMfXyi1DxLhGECs8DpvPelC?=
 =?us-ascii?Q?YJOK/3P8GG8/p3y4EdD24KOQh7ajFS8P+WbFuGomKUJa5+m5TWRV/8fKIPqq?=
 =?us-ascii?Q?8MAn6FMXNSZNEAbBuvAyDc0q7IeWF8FY0QYBem7JcYDVpyixHeY/t8s2bodW?=
 =?us-ascii?Q?pIDHAwtqNimUThGb2RBjPrUN7l7lpCBCcxohtSeGhlENPViMwPutD+idLBw/?=
 =?us-ascii?Q?hfqyDvS/BYKPQbOMAqQjqo4VlWfQfibU0ZWDJYo/pmsXcBYLx5XUnaVshh7t?=
 =?us-ascii?Q?chCGrfs8g4njLGW888al1h5/PF3fdGBXImiNGNp1cEUlSi8mIWtYHJ3yaoBs?=
 =?us-ascii?Q?ylaW1tY+fVNblC80XhnqvGzD?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b68a413b-ab26-42b3-4161-08d93a17427c
X-MS-Exchange-CrossTenant-AuthSource: CY4PR1001MB2086.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Jun 2021 09:29:42.7681
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RvXzGv79u9GiY3MV9X2VTWJTjLuUhYnt9aMCB5oNRsQPAfjHfhDOr6PDL2jlLg2VCfhtzKJP6F2e9A++y1EV6voQ1wyER0l4/P8eFcHH9X4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR10MB1607
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10028 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0
 mlxlogscore=999 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2104190000 definitions=main-2106280065
X-Proofpoint-GUID: uCZ12oyEgwu_rbAWDlx-ELscXx5-wppk
X-Proofpoint-ORIG-GUID: uCZ12oyEgwu_rbAWDlx-ELscXx5-wppk
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

ib_query_port() calls device->ops.query_port() to get the port
attributes. The method of querying is device driver specific.
The same function calls device->ops.query_gid() to get the GID and
extract the subnet_prefix (gid_prefix).

The GID and subnet_prefix are stored in a cache. But they do not get
read from the cache if the device is an Infiniband device. The
following change takes advantage of the cached subnet_prefix.
Testing with RDBMS has shown a significant improvement in performance
with this change.

The variable cache_is_initialized is added because ib_query_port()
gets called early in the stage when cache is not built while reading
port immutable properties.

In that case, the default GID still gets read from HCA for IB link-
layer devices.

Fixes: fad61ad ("IB/core: Add subnet prefix to port info")
Signed-off-by: Anand Khoje <anand.a.khoje@oracle.com>
Signed-off-by: Haakon Bugge <haakon.bugge@oracle.com>
---
 drivers/infiniband/core/cache.c  | 1 +
 drivers/infiniband/core/device.c | 9 +++++++++
 include/rdma/ib_verbs.h          | 1 +
 3 files changed, 11 insertions(+)

diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 929399e..4150043 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -1631,6 +1631,7 @@ int ib_cache_setup_one(struct ib_device *device)
 		err = ib_cache_update(device, p, true, true, true);
 		if (err)
 			return err;
+		device->port_data[p].cache_is_initialized = 1;
 	}
 
 	return 0;
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index fa20b18..e078a48 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2063,6 +2063,15 @@ static int __ib_query_port(struct ib_device *device,
 	    IB_LINK_LAYER_INFINIBAND)
 		return 0;
 
+	if (!device->port_data[port_num].cache_is_initialized)
+		goto query_gid_from_device;
+
+	ib_get_cached_subnet_prefix(device, port_num,
+				    &port_attr->subnet_prefix);
+
+	return 0;
+
+query_gid_from_device:
 	err = device->ops.query_gid(device, port_num, 0, &gid);
 	if (err)
 		return err;
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 371df1c..e692f9b 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2179,6 +2179,7 @@ struct ib_port_data {
 
 	spinlock_t netdev_lock;
 
+	u8 cache_is_initialized:1;
 	struct list_head pkey_list;
 
 	struct ib_port_cache cache;
-- 
1.8.3.1