Received-SPF: softfail (google.com: domain of transitioning linux-nfs-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Trond Myklebust <trondmy@hammerspace.com>
CC:     "kurt@garloff.de" <kurt@garloff.de>,
        "Olga.Kornievskaia@netapp.com" <Olga.Kornievskaia@netapp.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Anna Schumaker <Anna.Schumaker@netapp.com>
Subject: Re: 6f283634 / 1976b2b3 breaks NFS (QNAP/Linux kNFSD)
Thread-Topic: 6f283634 / 1976b2b3 breaks NFS (QNAP/Linux kNFSD)
Thread-Index: AQHYJqo39+LyDrnueE2TsZR4Xaed/KydEsUAgAAiEoCAAImBgIAAFXKAgAOfFICAAEE+AIAABjCA
Date:   Wed, 23 Feb 2022 22:22:18 +0000
Message-ID: <E5300203-2C9B-4873-84EA-CDBEAA1CCDFA@oracle.com>
References: <50bd4b4d-3730-4048-fcce-6c79dfe70acf@garloff.de>
 <8957291b-ecd1-931e-5d0c-7ef20c401e5d@garloff.de>
 <F693AC98-DCB4-4086-AC19-EE1B71DB2551@netapp.com>
 <be851303-b1bb-7d8d-832e-a1a3db529662@garloff.de>
 <10d55787-7b97-8636-9426-73fdeda0a122@garloff.de>
 <FFD47AE3-D6B0-417F-B0CF-D89E6FDB38C5@oracle.com>
 <29c0db53ab6dee067c1b91ba3eeca98310506453.camel@hammerspace.com>
In-Reply-To: <29c0db53ab6dee067c1b91ba3eeca98310506453.camel@hammerspace.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?D1SPutnLuonInkYQnVLcU3X9hv85F/VIPS3Qc8sZpxbFL71RygehVCdIWGIV?=
 =?us-ascii?Q?aukF5Zfd+/7FYnmEjrTu/9vWFet/oktj4YsPcj3yRy1+0fv0/bjV7B08Gq2L?=
 =?us-ascii?Q?y3JsCFkE2QlCWsa1gTNhbVDT5NH2neUEsNZn41MbI36k8dp8yF+030RjRLQ6?=
 =?us-ascii?Q?uk96uXgrLrx6fx7OK82oXGj1o/ChiDvq16w6z4X0ma24eQLIiNXlOef5MTId?=
 =?us-ascii?Q?PrGCzf5mqt8RTHsxLnBnwfoEExottNYf31BDVGzSsOe4b0bupGQ5BDwsA28o?=
 =?us-ascii?Q?KIGS8oFveWVogQswy/p3JVoSxbdwzUVbUJ6ruot5Y5hVkN5lAR3X7y1/4KAs?=
 =?us-ascii?Q?sMDDZiMTdl3VvgOwY5kVi8lM0QIQa0k/AADsEjDR3NRIMbY+OltvrTyhVqQN?=
 =?us-ascii?Q?FfQqd8LQhUO1rJXwNjXfAJcG1fTCpCRwRHe5g7SPe+UiV/FvHkInPN8yZswE?=
 =?us-ascii?Q?Q4/Z3UaWdAFZkhBO/wT8XqMFr5bRYEXXIWmLka/7GKAkwr48vrAeOwgADDEf?=
 =?us-ascii?Q?eaBt+oW6VrTyJhZrnGwEv4dcxFM6ZF+OEoowL6zyiDYPyZAJitu5pPNP+RfK?=
 =?us-ascii?Q?TmYFJZVCmYLyVNZvBbjlaRd5yN8iIB9hjQsMUW0WokWIzLicNOs1TSbqT0fh?=
 =?us-ascii?Q?izrf1HfyXEJxf/4UmqPu1xZTQDpBaTeK1CssD302wXlBsrd6PsHeR7iVFYFZ?=
 =?us-ascii?Q?Np0Y32Gs+UJeBDWpEx9G5AY8Gz9CVvXW05YVt3ZbdXhRHxZ9/eCNXsLQaalC?=
 =?us-ascii?Q?ZABb2UG3Opy643tyZUPzUg7D34P3oKzhTzlCQpfeRclnBuzl6grojQEbYZVT?=
 =?us-ascii?Q?qWc9XKZDoh1WpNsTHJUDV1LuD5Dyxl6mSZxASTtS4Xt4wkmTm24qT1s3IPn8?=
 =?us-ascii?Q?3BDO/VVZG3OgJiHhBvG1jjdpImD7aRwM9LHIbKQon8cahGwRYzDHIc8JsrBO?=
 =?us-ascii?Q?1GLWZ40Df7GNDe39HbnvpoTnz3zO6imLBA3Sv/J46L6lFrDqinjUJCkuCcPZ?=
 =?us-ascii?Q?iREstFIL+LaUOa5rLsFgVFNVbbHAtJEl2uBEOdymYs9z8VSw5UDKdjMQLrhs?=
 =?us-ascii?Q?3JVT4HblSXK8P89FQ9VFhZ2qzBlr671icOVG6/S/M3vQFsPKij2aniU0H0nT?=
 =?us-ascii?Q?bgQ6NB6HccK8qX2lnQTtjGZ39sXoUb90z1nK/aIpy8G5q7XeoeCJF/vGjiKQ?=
 =?us-ascii?Q?/4gbXXkjQlZ+ApN5tQSugh9+4YiCX/oy7Ai/eyF8PRMgYJTAYVorxNkhIWcU?=
 =?us-ascii?Q?cisZ1I6NnMzBStwVWH9Z1ulD2ARWj/278prOWykiyFzTFQKBP/VRaVwQKB8K?=
 =?us-ascii?Q?sI1QpDYD+G8xcYdjuG03bOZeQ4A8mzN4l2yZEOoZJbKRACv3WSvigjU1myA9?=
 =?us-ascii?Q?5mv7VqdtrJRhw16qGDLFa/DUB6wai451Oam6+IQJVjhBTZwXG091dHXbTdez?=
 =?us-ascii?Q?EBwXBMaKO+vjEeYc3sNMafKjrBiUmt5S+JSlyEpJo+JtEoi6UdS3SdVasAkb?=
 =?us-ascii?Q?ymrjcTuhRWDJOlbz1AIBDubF93yO79hg/kKwmQxm/4NlIXW81hOKZXlhGi1F?=
 =?us-ascii?Q?fEsBWCFu4uP/XAGflPSYBwab+vP6U/JuG8RZdz/Hm6sP3gUjoR3jNryaLsHf?=
 =?us-ascii?Q?eOiE3MCZof+5VDjEhsalfiI=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <14C6E3DD2F4E3E4C8DC7C723BEF37024@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 535c9f36-283e-4fc3-0789-08d9f71af3d0
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Feb 2022 22:22:18.0874
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: T2A0wwTe7gelUeMaV+598oGjoLf9+UrGTQ+Xwy4TBnfuKAAIIQLnO0lJZpefFz1wXwsB5/lHnA/4OOT4RxW2FA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR10MB4022
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10267 signatures=681306
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 malwarescore=0
 mlxlogscore=999 adultscore=0 bulkscore=0 phishscore=0 suspectscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2201110000 definitions=main-2202230125
X-Proofpoint-ORIG-GUID: WMP0mzKWSUAP-aZmq9xDKZFjxEtnY9-A
X-Proofpoint-GUID: WMP0mzKWSUAP-aZmq9xDKZFjxEtnY9-A
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Feb 23, 2022, at 5:00 PM, Trond Myklebust <trondmy@hammerspace.com> wr=
ote:
>=20
> On Wed, 2022-02-23 at 18:06 +0000, Chuck Lever III wrote:
>>=20
>>=20
>>> On Feb 21, 2022, at 5:48 AM, Kurt Garloff <kurt@garloff.de> wrote:
>>>=20
>>> Hi,
>>>=20
>>> On 21.02.22 10:31, Kurt Garloff wrote:
>>>> Hi Olga,
>>>>=20
>>>> On 21.02.22 02:19, Kornievskaia, Olga wrote:
>>>>> [...]
>>>>> Is it possible for you to provide a network trace?
>>>>=20
>>>> Yes.
>>>>=20
>>>> Is tcpdump what you'd like to see? wireshark's dumpcap?
>>>> Any NFS specific tracing tools I should be using?
>>>>=20
>>>> One trace with a working kernel and one with the broken one?
>>>=20
>>> Comparing the good and the bad trace ...
>>>=20
>>> mount -t nfs 192.168.155.74:/Public /mnt/Public
>>> against Qnap 4.3.4.xxx NFS v4.1 server.
>>>=20
>>> Both do:
>>>=20
>>> Establish conn
>>> NFS NULL (ack)
>>> NFS EXCHANGE_ID (4.2 -> NFS4ERR_MINOR_VERS_MISMATCH)
>>> Teardown and reestablish
>>> NFS NULL (ack)
>>> NFS EXCAHNGE_ID (4.1 -> ack)
>>> NFS EXCAHNGE_ID (4.1 -> ack)
>>> NFS CREATE_SESSION (ack)
>>> NFS RECLAIM_COMPLETE (CB_NULL, ack)
>>> NFS_SECINFO_NO_NAME (ack)
>>> NFS PUTROOTFH|GETATTR (ack)
>>> NFS GETATTR FH:0x62d40c52 (ack), 8 times
>>> NFS ACCESS FH_ -x62d40c52 (denied md xt dl, alllowed rd lu)
>>> NFS LOOKUP DH:0x62d40c52/Public (ack)
>>> NFS LOOKUP DH:0x62d40c52/Public (ack)
>>> NFS GETATTR FH:0x8ee88cee (ack), 3 times
>>>=20
>>>=20
>>> Now the differences start:
>>>=20
>>> The fixed NFS client repeatedly gets ack back, the broken NFS
>>> client gets
>>>=20
>>> NFS GETATTR FH:0x8ee88cee (NFS4ERR_DELAY), repeating forever (exp.
>>> backoff)
>>=20
>> Any idea why the server is not able to respond properly to
>> the GETATTR request? That seems like the root of the problem.
>>=20
>=20
> The GETATTR is a request for fs_locations in order to probe for
> alternative IP addresses.
>=20
> IIRC, some earlier implementations of knfsd had this response when the
> mountd daemon wasn't configured to expect a referral upcall for that
> location.

knfsd, or mountd? Is there known to be a server-side fix available?


--
Chuck Lever