Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Neil Brown <neilb@suse.de>
CC:     Chuck Lever <cel@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Lorenzo Bianconi <lorenzo@kernel.org>,
        Jeff Layton <jlayton@redhat.com>,
        "david@fromorbit.com" <david@fromorbit.com>
Subject: Re: [PATCH v2 4/9] SUNRPC: Count ingress RPC messages per svc_pool
Thread-Topic: [PATCH v2 4/9] SUNRPC: Count ingress RPC messages per svc_pool
Thread-Index: AQHZrguZSYcSrGelgEOwXadkv8haTK+oxYYAgAARQYCAAAxCgIAA0pEA
Date:   Tue, 4 Jul 2023 15:04:46 +0000
Message-ID: <3A42B6ED-E288-4351-8C12-0C9DD8396E2D@oracle.com>
References: <168842897573.139194.15893960758088950748.stgit@manet.1015granger.net>
 <168842927591.139194.16920372497489479670.stgit@manet.1015granger.net>
 <168843152047.8939.5788535164524204707@noble.neil.brown.name>
 <ZKN6GZ8q9NVLywOJ@manet.1015granger.net>
 <168843785749.8939.9913705917013899427@noble.neil.brown.name>
In-Reply-To: <168843785749.8939.9913705917013899427@noble.neil.brown.name>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ObDTB3xgIDE7dnccvirpYJGHdgS81DPmNo2uA2ChKSA0b0oLbCAQnrmmWvKj?=
 =?us-ascii?Q?SzzoAT1SU9hRR6t8uZD6qQqtxZadISKiBhstue7zo9weps3u61SpqY2ugvkM?=
 =?us-ascii?Q?9aDckupLR24iD2vhml9HN7t7aUAYr4UlaIQj1TUTJPOoArGmmQE2GrTpGox9?=
 =?us-ascii?Q?EhGtJSJtNd2ter0AuMjK0c/UmbjKsX//mPV4EukfIBUr171hwTzJGimUJLPK?=
 =?us-ascii?Q?mw6iSPGp9Kl3FZF204U+83kSnMqqDfbqxGbOyJ1T4kBPTWyLciNJzGwz/AYN?=
 =?us-ascii?Q?5fxwidGnxt7nDR3NldXw8BLIpn1UvcwrnNrlaXNEDHT6rT4TrbdbyydQCgDc?=
 =?us-ascii?Q?DRzWJgNq5mDe2PNA975Y1QnC9DVtVtOAdX3uAJJPk+eIdeCFvdqFHlk/nMC1?=
 =?us-ascii?Q?YmvMpZgh9DD+IHkKmiudN/N/yNOXpBqQ8aBgI1wKOpOFXpQ8rZjBi+g6WNqm?=
 =?us-ascii?Q?jwFws0QEPBD/b5aFOwpRBnRljib5W57Do6whNCj7ApILYtxGD1lMgxbSC6mR?=
 =?us-ascii?Q?IPYWY/mdh+BQJDau9dM0cQdbuAqqp1jIMdeBqbQFJAFONmwFNgKqb4rNDKW0?=
 =?us-ascii?Q?z12ct/iNKgVYtq/xNyHhIxYIsoeuIX9RESv4GidLTViyTcAVHaW1W37Lg9fX?=
 =?us-ascii?Q?7LbqgTDEUIxpaWModHF/ryZxBQ2JzT8S19O7Z2HB0eXURVX4eTLR3BapJh8u?=
 =?us-ascii?Q?u9vVG09MFatFF7J7Owa61Pkt78tzISOn0doMsXP2dSn1hOb9A2GzQk5TOrov?=
 =?us-ascii?Q?LcOsJW+JgKRrfg+lxwpGQGvd4BPnOgmCeHPwTCmsD+yU5Q//ttltGMgtjxeB?=
 =?us-ascii?Q?0XZZ3UaaSGSAzsm3FN6dKQUL1GN9rsg61IibjWs94HwQiiXyeLXw+az2cfEo?=
 =?us-ascii?Q?pTNNguLCVnYpWtP/F3I5G2pnq1EE/J1/Tp1p+G5/YnILiNFK87rgzi2zwr54?=
 =?us-ascii?Q?jr7/gRA8Raqt6ehLEl0y5ULvxD78U8gVqwQzXc2tI29ZVRjeeWB2ilVgXrgz?=
 =?us-ascii?Q?zQ8gL6C7yOqgv3W+GQ1JNorNY6ZRMs+HJFAgyXj/YPCp+f/1qR+gvLCl1Xa/?=
 =?us-ascii?Q?PAOjnDoeM3bsNieOlwASVPsqLA0F+2aBu0EuMF6F/IlKLXQGqYTwMtR24szN?=
 =?us-ascii?Q?HF7FLYjmbcxEbL7cEA50dOP8plWHvjHpgNlMj/W+rA2e+MPnAPU43llUcac3?=
 =?us-ascii?Q?EvcfISse9ee8eQu7ZOfNvHShTUJc0LAw134vff9dgTqfnMXaaYxY9yA7UOws?=
 =?us-ascii?Q?SQx1pbhf+CJzy6b96uG/8Jy5ziWjHtnn8hndpVl17CyFeJSFA9nXizEvFEYm?=
 =?us-ascii?Q?wy4vJaLInMUXpoBcBSgfVN1LZEB3CkvzJXOeUUajeaAf0uUOrpKTnencY4To?=
 =?us-ascii?Q?Rg9zL6F7y0+k4S9UJPSYz4x/Fcnn5JT52B3xrebG2+yBcHP0GNwCQfIOhZli?=
 =?us-ascii?Q?fIZvkpvxF3JzyPeUg8XLwuXSdOfKDn1BrqLGwBGApCemzL2YfWvAnyuQJtPf?=
 =?us-ascii?Q?uFKhKonydUAxe16h8er4NsbhBIVFulQbqCZBko1b8135udl9MUsleQ35flSC?=
 =?us-ascii?Q?F+/VXEumeadUIfe1S8mHeR6B1wHu4ojsFaSYWFKNlpqtbIt1brF1eZwIzuq4?=
 =?us-ascii?Q?bg=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D09D7E5ABED84D41B6799A8ADFA8BC8A@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: KKF+1iratm86+UWV/wkJXXT8EoF/cXjCuyT/CahsMc3XjlJl2kGTr5GMfbLJ3pn3bX0UeJV8PrTSDKwGiW1HoD5dTMkfL4ZK63uWD8J25pC/crK/0jTwUKEwozjbvqlsUoIulGW/G/1LswA6btwmy+Fr76wG9mGRCxybWnNbKOoliAkqAgAwpnmHF7FerDaDHM84vQaEicDmu31IUNXA7JU8pcxbLzfnLIT/9yzJxhTs+f9LtcrQZ8mZZsfnvY52jqxr9QD15Pmhm4l5YhPWHPuKyxHeqDBDH50360tAWVdMyN04W+vfQlhuh4gFYDG6ZCsUXjkJV5rAKYnqmKaBrsoZ1VHPPpky/1r80fwOBxpn2qPXt6+oJ+f0V6U9a46fndaYc+22Jvk0CmLVcfdzU8+xD7lc3DFcWGlT5Rcdj4HYrdvB5AxOv4XQ18tJuZEuQhP3f9cqJBdeznvINgtkXd7XNofRdY6vn2rMa04LXPURhcEIKdFYD1b+8M/nOPQjlRT35DGJ104OQcDRtuFm4yXDyaK+gQdGYqK4JPU+3munB78/UfnT53FXh/BPebKS07xy+b22FtoqSyQUHJ7cmOzELTIMqCKU/DuazZRX+Jtj0XRxKAXIrpDCqRzXh5os/+o6ZK3crIUOnymTebWC1nhSCvzCnSFzOh1RQuKv7F7KytigCJLOBKhpySaer9qJ1DvepQ7+ZcySYHuqOh89i8HnmW0/PDK0KktskjznXNPHe9Fi1UojZid8AezOK0x5u+4IBlKqtzLpbU6gcssoCMcHN8iAMY97EyDBHFI38t5S/ieLNs3kvS+/KIYvaB7Q4FaJFwsJgD7j+JDwf3TiPuwWrtHfaOwA14P1gYM50kJB6yTrj0yTqI3ww8tFCUv0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cd89fdf4-1485-416b-42a5-08db7ca00186
X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jul 2023 15:04:46.4776
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: jUNVzowdKF9nSC86mkz1jUJxU7/UZQVvx/PCEj6myv2gZfIuFmWRYjoxG0HSkwyXR/38Mgju2hZYpGtXcmgUwg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6663
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-07-04_09,2023-07-04_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=925
 suspectscore=0 bulkscore=0 spamscore=0 mlxscore=0 adultscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2305260000 definitions=main-2307040130
X-Proofpoint-GUID: 978epEtqP7RVii4F1jCVYnNq6PVcvHwk
X-Proofpoint-ORIG-GUID: 978epEtqP7RVii4F1jCVYnNq6PVcvHwk
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jul 3, 2023, at 10:30 PM, NeilBrown <neilb@suse.de> wrote:
>=20
> On Tue, 04 Jul 2023, Chuck Lever wrote:
>> On Tue, Jul 04, 2023 at 10:45:20AM +1000, NeilBrown wrote:
>>> On Tue, 04 Jul 2023, Chuck Lever wrote:
>>>> From: Chuck Lever <chuck.lever@oracle.com>
>>>>=20
>>>> To get a sense of the average number of transport enqueue operations
>>>> needed to process an incoming RPC message, re-use the "packets" pool
>>>> stat. Track the number of complete RPC messages processed by each
>>>> thread pool.
>>>=20
>>> If I understand this correctly, then I would say it differently.
>>>=20
>>> I think there are either zero or one transport enqueue operations for
>>> each incoming RPC message.  Is that correct?  So the average would be i=
n
>>> (0,1].
>>> Wouldn't it be more natural to talk about the average number of incomin=
g
>>> RPC messages processed per enqueue operation?  This would be typically
>>> be around 1 on a lightly loaded server and would climb up as things get
>>> busy.=20
>>>=20
>>> Was there a reason you wrote it the way around that you did?
>>=20
>> Yes: more than one enqueue is done per incoming RPC. For example,
>> svc_data_ready() enqueues, and so does svc_xprt_receive().
>>=20
>> If the RPC requires more than one call to ->recvfrom() to complete
>> the receive operation, each one of those calls does an enqueue.
>>=20
>=20
> Ahhhh - that makes sense.  Thanks.
> So its really that a number of transport enqueue operations are needed
> to *receive* the message.  Once it is received, it is then processed
> with no more enqueuing.
>=20
> I was partly thrown by the fact that the series is mostly about the
> queue of threads, but this is about the queue of transports.
> I guess the more times a transport if queued, the more times a thread
> needs to be woken?

Yes: this new metric is an indirect measure of the workload
on the new thread wake-up mechanism.


--
Chuck Lever