Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Neil Brown <neilb@suse.de>
CC:     Jeff Layton <jlayton@redhat.com>, Chuck Lever <cel@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "lorenzo@kernel.org" <lorenzo@kernel.org>,
        "david@fromorbit.com" <david@fromorbit.com>
Subject: Re: [PATCH v3 5/9] SUNRPC: Count pool threads that were awoken but
 found no work to do
Thread-Topic: [PATCH v3 5/9] SUNRPC: Count pool threads that were awoken but
 found no work to do
Thread-Index: AQHZs02GG6qEobU1aE6J2mdZLN/TsK+zlUyAgAAGRYCAALlOgIAAHiiAgAAJngCAAANdAA==
Date:   Tue, 11 Jul 2023 12:29:26 +0000
Message-ID: <ED33222B-7235-4D79-A63F-66CB6405222C@oracle.com>
References: <168900729243.7514.15141312295052254929.stgit@manet.1015granger.net>
 <168900734678.7514.887270657845753276.stgit@manet.1015granger.net>
 <168902815363.8939.4838920400288577480@noble.neil.brown.name>
 <D20E4286-30D2-461D-B856-41D3C53C756C@oracle.com>
 <168906929382.8939.6551236368797730920@noble.neil.brown.name>
 <18061cab9338b08da691e8651e75f8fe8e88b830.camel@redhat.com>
 <168907783478.8939.2524484078705916909@noble.neil.brown.name>
In-Reply-To: <168907783478.8939.2524484078705916909@noble.neil.brown.name>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?QT+9fFUpJZfdRoc4CNisUB6HNjTBLCIB46WA7Z3KQPTtwhW04GsxDLyKgC3Z?=
 =?us-ascii?Q?wbSjXHBvq1LzZlaEuXC1RYz0h+0GHhHeeLRasSAeqdysdBGjVUCiN12rCOIU?=
 =?us-ascii?Q?EoL09LUr6Ym3IKG7JhJfhggKEJX6TqQS00Ukd9qkFl/JJzVqeCmQYnj+uIjy?=
 =?us-ascii?Q?evbIjkkaYlb0Jz89w5UHJp9oTNHpjxRFpc9QL3PcfNhF/eaXgczGa/F/TV3T?=
 =?us-ascii?Q?O662ITPlsVZMxnx/chxhmBCEC2VAy4xcyqr/DDQ8bWaLmjD+TknjMqgviysZ?=
 =?us-ascii?Q?gBtAaQqS1/rPBIEMnT3ywCkCM/NC5O8ds9O8LZe4bqsi7EnRn21EEbuSSq7i?=
 =?us-ascii?Q?uX6bVDn9l7DLA6MzNwq6bZM2Xvbq+ho5lhvA6bS4raj8JWv2lrbMpFzSwSnW?=
 =?us-ascii?Q?TPMzRUA0vIiHgUCqTCt4TK8niAiH0ZXyQ2aQJIwrhUjGsIeigU/4hXCql05i?=
 =?us-ascii?Q?hsTsnApcGZsJTfgPe3n8D94d3vxnF8OE9lIfbm+Tb5CTTRUfJShNwxhXUSYZ?=
 =?us-ascii?Q?mBSWidpwB8W0kEnkGNDLt99/UoXGXY7yPSQ+mt1JfzizWlBm7nbD4QfulFLn?=
 =?us-ascii?Q?egwhCBjqcFeLxmwU8u4iKSKxtI22zymcT4eewvETQqnM6EdJlhh6mS6lajmm?=
 =?us-ascii?Q?4KCgjNCgOtpJlV4czNspq5hY8+3K+n42rD3vHWR7+byGCwyw5t4vGTV80nEV?=
 =?us-ascii?Q?Pco3d8bYIsjSfsxFxbOg0aeWvVlzxhhY6ryoTW0ksuQKukdo2o78t64Nrn0c?=
 =?us-ascii?Q?558yzsxHGIs/YIrV4l+4dvBwg5gguLzA6BXkBqraSQhabxEJK1iEwRtILtLD?=
 =?us-ascii?Q?/yI4+vUdtNnhsgHv1UjcXxF9z8hlfPhSJeTEPrmS0x/5zn5NZg1vft+aiP/L?=
 =?us-ascii?Q?uVr2o7Od/OUCp+dXYjr1Z9lz2vXsaS3ATrT3hA29IU5Nk1wd3j7RrXCFN8/S?=
 =?us-ascii?Q?m28Kt/DHTyX57mBnRre7W2b9Vopq3xlm5OGU53YT5ZPqB7xHgT6ggMPbN0sq?=
 =?us-ascii?Q?7eg3W0pR2teA7Bt2ceqzcBXfraB5Rf/QN/KALZwc745HESRHqulPcO6gvweR?=
 =?us-ascii?Q?8nMT7zB0sGjoF7ufc5wEAbjXBKOo4EuSuDSSuVbz10Moe59k1+ik18dlGnIw?=
 =?us-ascii?Q?+4Txbf48tXkYrMOzXKW8Dd584m6xWslqxzQpkSaon3kFhYa51wYiFmKL8rrU?=
 =?us-ascii?Q?5uzZWvruoT2nwlVN65tIZcBKtOnp4Zh3pOl0ut2NNQMomwrkH2aSYFgH8/uc?=
 =?us-ascii?Q?+c8TQTVis9VBxfjGwwf1y8zKCZNufmN0c8HjUiM2pNBXplfCOB6ja3rEcFFT?=
 =?us-ascii?Q?u4OcXtG9h9J1/3PIHnH2zGlmijCXwwqhxpkCl6pmzUQRbl/lG78HjM+77Q+P?=
 =?us-ascii?Q?p5jMjSAowMtjzWBt8mJJY+paLwhSWJDELq3UuHljUhx5Fav911BUV8mRV7qb?=
 =?us-ascii?Q?5DY2Q5S9akViiBlE1/8Thesph8H08o1Hwp8hUMav8Tkcc+ebz2UAmaSWbt7R?=
 =?us-ascii?Q?VFR8P54E0+vS2zw5UvwJEaeaBzUhVIjjSt02UVld7vVAkGkEfGCt3wBTkEDN?=
 =?us-ascii?Q?857CTYoPzuccNUYBm6G0oggGO71oNnA0Yv/02fozashMrvK0XC3r8y2pd/Jv?=
 =?us-ascii?Q?Jw=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <2128F5415F52D0449AD84D97E6C7FEA3@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 2fAf3B1IHcKh9EyuKyRpO2Md11vAosp4obJ89dxfo2ojvsfJ9180fegP3wUSyDv1AM7pR3IMryUb6lsXYWiGfQAMsZ2gKtcgx6/gMK+8EmRPh3CYg2SCsnUsJLsW05AQCGDm+n9e9gnibadK6h515eMxKlsDAS2VNx4KQLo86CPfUqg1yv9nKd9OMA6Gc0KEshh0fZsMOxPSb7Xx7Osz53iafj48wE4wJBYaGNtpoLW43HCVVlkf38zCD3g1NTn3er8Wo/Tj9YjRgwMGGmdjuD5DBMiBzwj38loTbmMtKv82eI89EK+bBx+oJoiWJ0qmrK+2wAgmJyNN32Gq4npTS0QSrYp2rCdOmGBqa162SVEOXpJBYJwUS0/u4jLbhdoSuAK1N7JbC3H3/1u06Vzla7vScZfYvokWJdzV0n82XsSpVbn0x3XHu7DWS2+TCpJIwOVSIjHDDmpp0ngy1LZXju91kUE6cc5LuDS/7UooaTa85ExYPndAMi6vjyjDb5SEvj1sPkzxTojmDZSPW+wpwFDZBYnbZMrobl3jV1rSoT6BqehvE/Jn9bUSgU0m7MritHh7izdZDcpwtBqLBNXE4csqGZUNdpZXQ9fULAFnsnJDMz2IYawkeyGPEhgD5dTC0zu5EahKuDh5+cLYUJwCO2wsUOVzMyY3724agyFV6rJhkIerp7jNC/5iqe9x6bb7kelHIV2MDWlLJZkNyjOb8XNShDhe5XTkn/CHlpMV/FnNg8YarkycidDK3JzXYOpg6q6bb1Ujf8fyukjB9RN5/LRSOXe4HdLeLUjtKjkEVoB3d6dgebC4YLvlu9bT+766VTk6YCCSxHqN+MglK7D4l7qukynCLp55qjzoq2FKc42DAU/1QuXx5x+vb44f29aP
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2f54eee3-1dad-4705-0b0c-08db820a7757
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Jul 2023 12:29:26.6219
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: mQnibmqqzS25xExyubJXO61yu+0olPAXMVP+ys1rMIhLHAfLk14xjD47n5SLdyqLBwyjQVVSU8OIURJGWhXJYA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR10MB4966
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-07-11_06,2023-07-11_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 malwarescore=0 phishscore=0
 spamscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000
 definitions=main-2307110112
X-Proofpoint-GUID: EqKpY8RNnescAV10zcWgnhgrnH9R9qk6
X-Proofpoint-ORIG-GUID: EqKpY8RNnescAV10zcWgnhgrnH9R9qk6
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jul 11, 2023, at 8:17 AM, NeilBrown <neilb@suse.de> wrote:
>=20
> On Tue, 11 Jul 2023, Jeff Layton wrote:
>> On Tue, 2023-07-11 at 19:54 +1000, NeilBrown wrote:
>>> On Tue, 11 Jul 2023, Chuck Lever III wrote:
>>>>=20
>>>>> On Jul 10, 2023, at 6:29 PM, NeilBrown <neilb@suse.de> wrote:
>>>>>=20
>>>>> On Tue, 11 Jul 2023, Chuck Lever wrote:
>>>>>> From: Chuck Lever <chuck.lever@oracle.com>
>>>>>>=20
>>>>>> Measure a source of thread scheduling inefficiency -- count threads
>>>>>> that were awoken but found that the transport queue had already been
>>>>>> emptied.
>>>>>>=20
>>>>>> An empty transport queue is possible when threads that run between
>>>>>> the wake_up_process() call and the woken thread returning from the
>>>>>> scheduler have pulled all remaining work off the transport queue
>>>>>> using the first svc_xprt_dequeue() in svc_get_next_xprt().
>>>>>=20
>>>>> I'm in two minds about this.  The data being gathered here is
>>>>> potentially useful
>>>>=20
>>>> It's actually pretty shocking: I've measured more than
>>>> 15% of thread wake-ups find no work to do.
>>>=20
>>> That is a bigger number than I would have guessed!
>>>=20
>>=20
>> I'm guessing the idea is that the receiver is waking a thread to do the
>> work, and that races with one that's already running? I'm sure there are
>> ways we can fix that, but it really does seem like we're trying to
>> reinvent workqueues here.
>=20
> True.  But then workqueues seem to reinvent themselves every so often
> too.  Once gets the impression they are trying to meet an enormous
> variety of needs.
> I'm not against trying to see if nfsd could work well in a workqueue
> environment, but I'm not certain it is a good idea.  Maintaining control
> of our own thread pools might be safer.
>=20
> I have a vague memory of looking into this in more detail once and
> deciding that it wasn't a good fit, but I cannot recall or easily deduce
> the reason.  Obviously we would have to give up SIGKILL, but we want to
> do that anyway.
>=20
> Would we want unbound work queues - so they can be scheduled across
> different CPUs?  Are NFSD threads cpu-intensive or not?  I'm not sure.
>=20
> I would be happy to explore a credible attempt at a conversion.

Jeff did one a few years back. It had some subtle performance
problems that we couldn't nail down.

My concern with workqueues is they don't seem well suited to
heavy workloads. There seems to be a lot of lock contention
unless the consumer is very very careful.


>>>>> - but who it is useful to?
>>>>> I think it is primarily useful for us - to understand the behaviour o=
f
>>>>> the implementation so we can know what needs to be optimised.
>>>>> It isn't really of any use to a sysadmin who wants to understand how
>>>>> their system is performing.
>>>>>=20
>>>>> But then .. who are tracepoints for?  Developers or admins?
>>>>> I guess that fact that we feel free to modify them whenever we need
>>>>> means they are primarily for developers?  In which case this is a goo=
d
>>>>> patch, and maybe we'll revert the functionality one day if it turns o=
ut
>>>>> that we can change the implementation so that a thread is never woken
>>>>> when there is no work to do ....
>>>>=20
>>>> A reasonable question to ask. The new "starved" metric
>>>> is similar: possibly useful while we are developing the
>>>> code, but not particularly valuable for system
>>>> administrators.
>>>>=20
>>>> How are the pool_stats used by administrators?
>>>=20
>>> That is a fair question.  Probably not much.
>>> Once upon a time we had stats which could show a histogram how thread
>>> usage.  I used that to decided if the load justified more threads.
>>> But we removed it because it was somewhat expensive and it was argued i=
t
>>> may not be all that useful...
>>> I haven't really looked at any other stats in my work.  Almost always i=
t
>>> is a packet capture that helps me see what is happening when I have an
>>> issue to address.
>>>=20
>>> Maybe I should just accept that stats are primarily for developers and
>>> they can be incredible useful for that purpose, and not worry if admins
>>> might ever need them.
>>>=20
>>>>=20
>>>> (And, why are they in /proc/fs/nfsd/ rather than under
>>>> something RPC-related?)
>>>=20
>>> Maybe because we "owned" /proc/fs/nfsd/, but the RPC-related stuff is
>>> under "net" and we didn't feel so comfortable sticking new stuff there.
>>> Or maybe not.
>>>=20
>>=20
>> --=20
>> Jeff Layton <jlayton@redhat.com>


--
Chuck Lever