Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Zorro Lang <zlang@redhat.com>
CC:     "fstests@vger.kernel.org" <fstests@vger.kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "djwong@vger.kernel.org" <djwong@vger.kernel.org>,
        "linux-xfs@vger.kernel.org" <linux-xfs@vger.kernel.org>
Subject: Re: generic/650 makes v6.0-rc client unusable
Thread-Topic: generic/650 makes v6.0-rc client unusable
Thread-Index: AQHYv8UPIvgoSU1RX0u5XzLtJ9lDJ63PQWGAgAAunICAAyFTAA==
Date:   Tue, 6 Sep 2022 15:50:47 +0000
Message-ID: <EC58E371-ED4B-4C36-B45E-EF2B861CAE65@oracle.com>
References: <3E21DFEA-8DF7-484B-8122-D578BFF7F9E0@oracle.com>
 <20220904131553.bqdsfbfhmdpuujd3@zlang-mailbox>
 <0AB3B4E6-3B0F-4F04-8618-A3257D820FAA@oracle.com>
In-Reply-To: <0AB3B4E6-3B0F-4F04-8618-A3257D820FAA@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?FdBf4egL1FO3qyA0HvbJFsedN8aMiOB07afSZoWwNWsj4lzdWXGL5MvYyK1R?=
 =?us-ascii?Q?uo5Omy3/6ycJrjZCWshjDPgD9iUE8QTd1rgW1x4rzbcwWlhG1GwLoqdb10Ng?=
 =?us-ascii?Q?Yl5J0yxuU5ov2kqDZixXQIpR1Wl4Pg3IaeIF8RzgFdF6XlWGxGUl2QAOmR3B?=
 =?us-ascii?Q?3d7nx7cxZyw3CJasY4AqDQhM9ACCrJYmMmOg1zm6c5DpkoZGpTj+Bt5eOD3w?=
 =?us-ascii?Q?3lHISaEcszlPNflpFD8iD/msg3yWzdp8w6RZH0GMO3HdrhSRUPyuEHNb/eNy?=
 =?us-ascii?Q?pk00YHlt59BZLoKEz5mcpIYe01htXm+bsFf3iZm5mDuoTvQRxRPrY4rAEMqm?=
 =?us-ascii?Q?3dXt3G8cq9DrtfD81+Igp3l+bZfm5LLLHM7/thq5BqIPO3S2T9TkKHaaFcDv?=
 =?us-ascii?Q?XNEo3NMdpnCI9ciJUN/hNaxA+yNikRi5h4FSYJcoXPLJv0D+v7DKJmEZJ3Ce?=
 =?us-ascii?Q?nC9CqadHbVwqk/Y7LqgER5F7aOjWFHfZ39bfc3FGlPFaFHmXT0xiNpo0KsAo?=
 =?us-ascii?Q?SAZQ9gluegZ05izGvd2IWyeXndQrRVXtP+uzxQ1Q46umMvY82Q+D+GmnUH3A?=
 =?us-ascii?Q?3TYvy21sDeZyCm+D6CAB+L38RH57LCZVGsbDZvgBLX5W1i4vhUHfn5yYOKbz?=
 =?us-ascii?Q?PlX67igTdI3mDQhElaGnaICJVqLQz9KLqg2JjK9vRt4M8tSDkiJlKSGt8V68?=
 =?us-ascii?Q?gcmGbtBurc2QHTHQVdxuAFOs9ntkUMwD/qoQVGPoxnacVspPfOOW21pNoKm2?=
 =?us-ascii?Q?tjI+Hkv9FB58IGzoyy9a4fFb6C61DcpbiJLDc8FgLEkHuFmYv/1+a/vLbyNk?=
 =?us-ascii?Q?1UatHLQFF2AbW1V/vESyS8raRBJUSkgsxeDHXHS3OEeTlYAHTEeecOmFoZD0?=
 =?us-ascii?Q?NmcAs4h8Dl4RIdu+2ayytO6G5vd5TVaInpJX0ZdzTY6N/97RRyqAMtXfoDqM?=
 =?us-ascii?Q?8HEU3nrLr5YQ75uLy+9Dt7eXpE31Rqup7iMXxrFqbRh362eHQNJ7rOmUnjSV?=
 =?us-ascii?Q?x0MOFKontlfcQmnJDdGzTSlBJE3rzKm1boYOWBx17X5f6jJxbR1nq1PNFdMT?=
 =?us-ascii?Q?CJ/0SFkfNc9qB0fEeDCFV1Z059zVNU4gpJ/Ay4ozPinsLfxgbOgPdIGHxfI0?=
 =?us-ascii?Q?Cp0abpq1wQPcQ1ZQ8qFfUf/w+G6etPJGNxVOC1PRkMthnTfzjZJBbCucoYVZ?=
 =?us-ascii?Q?GpzpF+Kgm4gaRM4P+s6Gn44Yl1cttOqKfU7P1jw06y3/DeqFNM5QPeyEA/3F?=
 =?us-ascii?Q?Kt2kadHMzgj1WF6ioTHDVgSJdenXCJeBZ5CElYeak9y0gtlNgtODhn+3Umj8?=
 =?us-ascii?Q?ndYYN2WJHboHnsFZs0q096pRqA0emPl3HjvrjBh/eGlJGJpiurSO5KwxzdeK?=
 =?us-ascii?Q?SXET2LhFAGr/jrYBB5t6jcgpGijj+WQIXl5gaLQd3V3EJzfM2iu5Ucm4TKUJ?=
 =?us-ascii?Q?WaiNKTNecg5w4YK5Z4depFGPdxippVfIXrSOMIid84vq1XO15tvJ8qTAK0nc?=
 =?us-ascii?Q?iioHAjnJTVqkmASL7Ka5yyueyUwQM3dIy0axIfyETimCaOJ/85Tgaal+GSvW?=
 =?us-ascii?Q?LAWYF4ZEE3pnZ1slDENzGzmrC1V4QeJUDdOW/fGvqf/Za9H5wiUImmAgIhRT?=
 =?us-ascii?Q?ug=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <F66F475E2CE5C24A9C6A95D024C643DE@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 41b10499-fcb2-4741-b708-08da901f90dd
X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2022 15:50:47.4556
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: hVQo6Cufd0nFYqV0iTMfK+MK63L4jpFGr5OBkc0AcF3rPg+35mo0gKCsNi1ctJr6JGBGR/hs0X3jcKRukzRA7g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR10MB6255
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1
 definitions=2022-09-06_09,2022-09-06_02,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 phishscore=0
 mlxlogscore=999 spamscore=0 adultscore=0 mlxscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000
 definitions=main-2209060075
X-Proofpoint-GUID: e7g4yzeIBJP3OXuQQELQCR3I1IjWFzVY
X-Proofpoint-ORIG-GUID: e7g4yzeIBJP3OXuQQELQCR3I1IjWFzVY
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Sep 4, 2022, at 12:02 PM, Chuck Lever III <chuck.lever@oracle.com> wro=
te:
>=20
> Hi-
>=20
>> On Sep 4, 2022, at 9:15 AM, Zorro Lang <zlang@redhat.com> wrote:
>>=20
>> On Sat, Sep 03, 2022 at 06:43:29PM +0000, Chuck Lever III wrote:
>>> While investigating some of the other issues that have been
>>> reported lately, I've found that my v6.0-rc3 NFS/TCP client
>>> goes off the rails often (but not always) during generic/650.
>>>=20
>>> This is the test that runs a workload while offlining and
>>> onlining CPUs. My test client has 12 physical cores.
>>>=20
>>> The test appears to start normally, but then after a bit
>>> the NFS server workload drops to zero and the NFS mount
>>> disappears. I can't run programs (sudo, for example) on
>>> the client. Can't log in, even on the console. The console
>>> has a constant stream of "can't rotate log: Input/Output
>>> error" type messages.
>>>=20
>>> I haven't looked further into this yet. Actually I'm not
>>> quite sure where to start looking.
>>>=20
>>> I recently switched this client from a local /home to an
>>> NFS-mounted one, and that's where the xfstests are built
>>> and run from, fwiw.
>>=20
>> If most of users complain generic/650, I'd like to exclude g/650 from th=
e
>> "auto" default run group. Any more points?
>=20
> Well generic/650 was passing for me before v6.0-rc, and IMO
> it is a tough but reasonable test, considering the ubiquitous
> use of workqueues and other scheduling primitives in our
> filesystems.
>=20
> So I think I caught a real bug, but I need a couple more days
> to work it out before deciding generic/650 is throwing false
> negatives and is thus not worth running in the "auto" group.

Following up. I can't reproduce it any more. I've heard more
than one report that this failure can happen on non-NFS
configurations. I'd therefore conclude that I haven't caught
a bug in something I'm actively testing.

Carry on!


> I can't really say whether Ted's failing tests are the
> result of an interaction with the GCE platform or the test
> itself. Ie, his patch might be the right approach -- exclude
> it based on the test platform.

--
Chuck Lever