Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Christoph Hellwig <hch@lst.de>
CC:     Jens Axboe <axboe@kernel.dk>,
        "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Chuck Lever <cel@kernel.org>
Subject: Re: NFS workload leaves nfsd threads in D state
Thread-Topic: NFS workload leaves nfsd threads in D state
Thread-Index: AQHZscpE2u3hRnymrkaAkGCGMx38d6+ypIIAgABnPoCAABH/AIAAJpqAgAADUYA=
Date:   Mon, 10 Jul 2023 17:40:42 +0000
Message-ID: <0F9A70B1-C6AE-4A8B-8A4B-8DC9ADED73AB@oracle.com>
References: <7A57C7AE-A51A-4254-888B-FE15CA21F9E9@oracle.com>
 <20230710075634.GA30120@lst.de>
 <3F16A14B-F854-41CC-A3CA-87C7946FC277@oracle.com>
 <F610D6B3-876F-4E5D-A3C4-A30F1B81D9B5@oracle.com>
 <20230710172839.GA7190@lst.de>
In-Reply-To: <20230710172839.GA7190@lst.de>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?rru4HnJG6HftsBw3QS47qZuDrU6DmT9I54jveLPB0jFUAqcBvk5C5j2MWle4?=
 =?us-ascii?Q?9F6uc/fpfO98OgeQgYpYViCFQKC2v1guM5L2cP7kES1lOxzw9FCMj82MVpSd?=
 =?us-ascii?Q?rduHZnVCMTsJe9/+6v8WYxqTiPUquKo3HCxrwlZ7iCC/hM0Pv2c+W+8XQlMO?=
 =?us-ascii?Q?0iZl/sf0Jk8a/+xFQOZDe0fUTzkmJ62WQP8xC9RinVdC/amDMGLXy50TA9cW?=
 =?us-ascii?Q?6+l1lQ8jRjnDfdmnjddHVOLVzgf73Fyc97pbcfZIp+WYQ3beFVTzoTIr3w/h?=
 =?us-ascii?Q?yMiZcI64BCeSK3Pk7/b6hJMuQ0mO+Ajn9L38lja11BobklJ/6ZDxMNAT6Kd4?=
 =?us-ascii?Q?7ZRw7cmN0k4IDtWTMCOif9/lk6A72DdZrmDLI3Zo7YzZ8ri3AWYfdoedJ9Ql?=
 =?us-ascii?Q?JrbqNdG5HijaT87nhhu27BeORv6yLek7EZB5kf7UbI5AEPa664sGXLLmknp0?=
 =?us-ascii?Q?+N59NxAsJm1CiaewX/8nIqUVuYEeqVbfWmUmKXp425gwuuONt2xdRtLPsnG9?=
 =?us-ascii?Q?Q1zXYw4HTjoaBiue3O9FZ7KR/tt5o0wljK2YV0h9Pbze6fHY00IlJ9AsTG83?=
 =?us-ascii?Q?zAOzER3dYfK0MobBHiV0v2SpQ3gbf2fkPq+v1N0X7ix4jzNo4s/plThzy11v?=
 =?us-ascii?Q?CT/7keQkbTwdPldiNyeBv+VTTZBkY7IfSnqCKIQnwbbloCyUxcLi3ucl9+uh?=
 =?us-ascii?Q?FVrKsOwOZ5IkHK0oR8uHXEHYuR8gLs7eePgJUPR2ijl3VNdvBy7uSzx7Mvmq?=
 =?us-ascii?Q?jHuHCML8nXcQ+RPlfW6bxXLvhXKoVgZ1AYgjtsXmpPtIA9UOXC6AoF3SXWFB?=
 =?us-ascii?Q?CF9W/MDuYOCQ+P97aRNRbduImhgSqrxrwKnbe+vOs8ONWZKdRBP3UlQC5O3B?=
 =?us-ascii?Q?15ssJC/JvvriVyfbeOSjCcVZA0QGYuf5phGSAeQF/xCASRR3mUKYmSzgG4DU?=
 =?us-ascii?Q?oxBy2a9+iMz3RqJz9PynZk7aDeno92GNfYb6XWgKdQyysTQl+OyX+1FfVz/+?=
 =?us-ascii?Q?5u2URrLpS47cBJmwpqJAbK8dqXmd0ZttZEDIB18rc85KO3z3LwIQYznY8AF4?=
 =?us-ascii?Q?7mSr7+tvTp2mWcrE1OhBA5C2sPJUiOSN7rexroS/gROE9ceUtCl0jE9Wlx/7?=
 =?us-ascii?Q?fiQKLuBCpLxs/0461rmV6i999E80+OWtYenLB+J6v0Hcr8wlbfM2vXhRybwj?=
 =?us-ascii?Q?+eKHPQnSEjCXnwrEKPINTkJ3IGk52/jnJEcm8ChmPefuA193hlnw57KP0VRT?=
 =?us-ascii?Q?bvsOe6p3XR/x81qzyXiUjyVcUYWjwrd/o6hLh6ooC6EVK6wFA0Q8mb7Bf04R?=
 =?us-ascii?Q?RI0j65v6vwnVu1lmU6agsAsdVWqRi8rRh7N55BmnXm9U/sxdSK7LjyytFvo3?=
 =?us-ascii?Q?G1lanJnCJ87jHINnpS7sCTlFgcC1gRsS3m02ruYhfkpX07fHoW3KSQ1ZOObG?=
 =?us-ascii?Q?zhepzEjvFeQF2UUV/2+Y5ub1RVApq5RpFSMJqli8xCC3zlGxpmPLIqXwsz6B?=
 =?us-ascii?Q?AYO1XQ8m8nBtTer2ikesBy1iYS7xYp/qghAvml5AgIXDJqmjb3n/GPtIW+ej?=
 =?us-ascii?Q?ENRdzgWJI03DWqB9ivxmIg3vdWFesLodjcjoiSbbqbBxgtBIYShbMgyNehSa?=
 =?us-ascii?Q?oA=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <1F9ED421A77E2541A3383CFD3AD57E48@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: l6jSB2uPY8ij9txm2PTXTT4s6Fe8UDHrVLhUfjIANH5dnSamZgEryGxOK0QBLtizTH8vat7Q0aYOSyj4PYGrwiUJI8U4couOrMXCtqn8SMMBuB79qq87BCe3Gw96TRMeCp4ZGx5P3yy/LbQzAv78QQzx2x4X3U2gOYUQD39jllCZ7CU7T01rnAfPKe7AR2lejoCT7ZS67T5tCawJXR/97OSgEhJSxm51rDBx5KOqPbM8aWI2o5GkLcugVOveTnnP6JDRQJMRB3rtQ+tEAUnlz9I/8pasTEriJy5VjLE0Ow77l76h8DrHnWb+YX6ym/HWJOb1o3BIzBpmL0fcCUCgA4j/6vrYT/zv8E/ErRVDVHlwACoam5k7SNcasnAzTWIerrrIvgxGSvX2Oe1muYI7a80jolNwvTPMimpuz+kgjxKFxh8AzNqu1t51yUMjfAfOZiZps48p+B3i+Q2rpLJuqfcCOOF7oFHz0+fC2JLm/FWtcIc+Y6S++LoSPZpvIq10q9sxgt3IEA3poAQcxBTDqjGlQZoUb/gkwoCIOOjM54M4QIUIhvkl1FfDsXjYLLpFgcRJDZSTvpomW7Y1L0CcceUQvNSZ9g0JF0U/WAaqJeJqj4c+rDVJ2KskX7DW2wjuPOYKCYRBsxH+UF7OceoXF9B6Y5OVxNCnhKWR+TvWQuwOkH9q1+2Z41L3Gj6VQB6/qd/iwAct2VukQaTCgzDtVYiIxjxbMAiTDzKLuswTNBPL/fBjX5cd9IEMMwS/e2HYjbyuNYG6ar2oZr9r7zAxJjIuX6vobtbVIc/dLvgltLTTUK4zzHZkbC/NvHum8q7AZ3yh2+cOODatAHLI7Uc7Og==
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1bf424c5-d5e2-4f90-c637-08db816cc8c5
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jul 2023 17:40:42.7808
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: vbmhGFcFwRfWEpAg2JtVSGVKwQjViW6qxP0yX3XS8JvAJx2ErnrhF1H7I6OMaFAgmq6vIloeH6z9aosF49g8Mw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6298
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-07-10_13,2023-07-06_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 malwarescore=0 phishscore=0
 spamscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000
 definitions=main-2307100159
X-Proofpoint-GUID: njFEOjbiZ0Q0S2Q8MFiShErRLVuQWyFN
X-Proofpoint-ORIG-GUID: njFEOjbiZ0Q0S2Q8MFiShErRLVuQWyFN
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jul 10, 2023, at 1:28 PM, Christoph Hellwig <hch@lst.de> wrote:
>=20
> Only found a virtualized AHCI setup so far without much success.  Can
> you try this (from Chengming Zhou) in the meantime?
>=20
> diff --git a/block/blk-flush.c b/block/blk-flush.c
> index dba392cf22bec6..5c392a277b9eb2 100644
> --- a/block/blk-flush.c
> +++ b/block/blk-flush.c
> @@ -443,7 +443,7 @@ bool blk_insert_flush(struct request *rq)
> * the post flush, and then just pass the command on.
> */
> blk_rq_init_flush(rq);
> - rq->flush.seq |=3D REQ_FSEQ_POSTFLUSH;
> + rq->flush.seq |=3D REQ_FSEQ_PREFLUSH;
> spin_lock_irq(&fq->mq_flush_lock);
> list_move_tail(&rq->flush.list, &fq->flush_data_in_flight);
> spin_unlock_irq(&fq->mq_flush_lock);

Thanks for the quick response. No change.

--
Chuck Lever