Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Jeff Layton <jlayton@kernel.org>
CC:     Jiri Slaby <jirislaby@kernel.org>,
        Steve Dickson <SteveD@redhat.com>,
        Chuck Lever <cel@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Neil Brown <neilb@suse.de>,
        Alexander Ahring Oder Aring <aahringo@redhat.com>
Subject: Re: [PATCH v1 26/27] SUNRPC: Set rq_accept_statp inside ->accept
 methods
Thread-Topic: [PATCH v1 26/27] SUNRPC: Set rq_accept_statp inside ->accept
 methods
Thread-Index: AQHZI37QRUQ4TvDJik2GwoQT/5RlW69HhAGAgAA1uwCAFlcegIAAAJUA
Date:   Tue, 16 May 2023 19:25:57 +0000
Message-ID: <0DEDF045-8C95-463D-B5E7-D2A3DF3230FC@oracle.com>
References: <167319499150.7490.2294168831574653380.stgit@bazille.1015granger.net>
 <167319546521.7490.43383592461162363.stgit@bazille.1015granger.net>
 <3e34a2dc-7d72-b719-248f-e78361db8a5b@kernel.org>
 <4AB1ED03-57C8-42C1-9A04-6C224E98EDCC@oracle.com>
 <569be8e6eff7af373e6baaf2170edb8a8c52f262.camel@kernel.org>
In-Reply-To: <569be8e6eff7af373e6baaf2170edb8a8c52f262.camel@kernel.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?iyXn4qcViLctt/NQo58IMLYp7cyDzMtMv8JGA7RWTCjl8rjiGEmu4wNSkxwe?=
 =?us-ascii?Q?/XayTIzRDxucU6v2vBs8kGlRpxHufYri94Gg6n2icpHDBMvh2e2Y8K28by98?=
 =?us-ascii?Q?iNTn2L08R7cntPWyl57CSzFAJVfps6t0tLS1kbkKOCnqRKlOXii6h1m3McSB?=
 =?us-ascii?Q?rSKvf2z6Uxn8ved5jkysJmJH84IWcFlVZvOvkW9ZI+J8MfPB8qxgUFwsOm7T?=
 =?us-ascii?Q?uGhkRojgPTp2rz0FlJ76ggIYMzIMr1Hk6wVmD3+HbTZ1oK/NTCxDKSfvotNw?=
 =?us-ascii?Q?UVplknA+Xok5fuhaHOvJNNS+NOjYSPlPzGka9bfskq3KJU+VfHQqwrwKE4Bs?=
 =?us-ascii?Q?V8VuPOI53+SARBB+/KVlPxsB3d10NuC25d8KbRebvEX77nQJ6+J6Cm+Nz6it?=
 =?us-ascii?Q?+zFgYqnB5frGKketIW/ZjCk3rqVvPWq4AM7ksVMq97BbfPPg8jbzm8IsVlJq?=
 =?us-ascii?Q?M5+X3q/gkvuw6SMj0jJmId/OIiukeCROhd0Kw2Xn/fR4zi5G25JkmBan701u?=
 =?us-ascii?Q?tpHmwi2dR2YrA4fPq3vOJdY6Bt1pEbo7Kq0SSwaOOxK/EVNx2baH/dOSUCBu?=
 =?us-ascii?Q?b13eQLYueMKE4wKoNVACPylg2IFzqoJvcmVVkzDT7MIrWAdcSX0F5vtQ3brt?=
 =?us-ascii?Q?s1OLUnWdpdrh335joe+JgJHaDJNVoOMhdLTSQqTT3qOeN8GfrsdB7Ks7fnsU?=
 =?us-ascii?Q?qvciJMXrKMCZWZTJHEAPEox/o+bJW1dYzkAp3SHTNdH6/GbaGzWqzhcd78cT?=
 =?us-ascii?Q?NGpOjcV8X6fQKdPxQMYdRIQFp/d26RgMGzvi7VT+++w1wz+mAJjrtDbRu9qR?=
 =?us-ascii?Q?FTSkMAthAJ5xx54TQfuDMXxOfNyrlgWJTxQhQv8aZs+RNwHLhxWHrb2Nop31?=
 =?us-ascii?Q?aKHtIUj7QimfryAaxDeHTSta4I3sdoO5nKOzHs+XrQeeB5dMf32QeNMGxb0N?=
 =?us-ascii?Q?itiLKHa51REOWOt2V8cSysuUOGBfDezf5Wt/tav+mQ87sxHXYsQFC6Nka/vQ?=
 =?us-ascii?Q?6WwQ6Od9dQavkYX2qiI8WzgJ0eNccw29+jZuGRwndPrGe2kXppcvB4gKkFLY?=
 =?us-ascii?Q?f41H0IpRAxkOVSRPbCVJi/9wByuK8HxjgtNdzztSB+GoO9L360RNve382ZAU?=
 =?us-ascii?Q?aHfbmvUW2YjnndGh0Ds5Ad7z42t6d7vHPU++S/jEIBeAvTBc+I6Lyi9jtlDG?=
 =?us-ascii?Q?OaQcz4Wy7R5KKyT8WKbY11PuWm0OR8X7sl2YNFaSaZ3cWXf/p2Pr180Pkvg3?=
 =?us-ascii?Q?wEi8AcQSBuP41WI23x8iuBHGzU5OrFGC445bn7DBZ4e6GFpOw/Goa9cVG0+4?=
 =?us-ascii?Q?PR5gdnwRIGNCYe09ejhCIF+MWQe5RmS3Ytn4xjFoCnYif/Z7DMhPYPTxCc2e?=
 =?us-ascii?Q?eowFrEULG932e/QqjoQNhY79tdx7WqFV19ZAVijmTRhESxOQmoW5n5uQFEw/?=
 =?us-ascii?Q?K2IsXKu98Yw0cTYSzj3sd+zXXW9IDuFp6NFJuNfoFG/WFQ+HgHr3tqSm3ua+?=
 =?us-ascii?Q?pAUEgFuV5QfS9gSb3B+97h9G5uzOUPmPrzVFC1L1YnH8Ju9TbVM2k6YG+aV+?=
 =?us-ascii?Q?iyuQHbnzPQ4FHKPaN3S3z3cgXS+zCSdU7ERCTZPpg1aCcocV89JXwkS1vDea?=
 =?us-ascii?Q?Ng=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <BF175BED93F90D438698708C149CF0E0@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: WnZYY6TzeF8DQi6Ny/CJGOr/UUKv08CR0jN4AiOCsAzKkOCj3+TtSr0Z3Sy4uulAAqdF7Z7TD0nMbbKHp6dwMIs6w3x5PUcwOJtqQPJ1OuDhqq/twcboQ8fewT5Qi2jmLYPiJtjokqJdfTHjlsTz7+5OCCt/OWSK53e+MpAee2ulKkBItn4l+6cSZgTH1oZUWMWLacXeG52rdVbgcdOn5rrC/eqrx1X/CEeDGAZGJMnkMKqO5mMvD1OwszPpu8txXTdwYASMll5ZH0q54cy/w/MdeZ7Q0+DMFNdWjMNip3GcgsuHbrlMT6hN3hpIZzQ+/6EIQ0Xn/F1esUcI2psMihnF3irD448rPlU14iBzxLpFcmPSv2fKEbi+FTXKMIwxbKd1FhbHStUebOkrZpl7Cxe7LzaQbP0aJBj89AONFeMLHlNgAcJFTG0dAXWLtfe0E1x2wx6dafDSn6bEmC+iBVifBnywHVtoXHgjrRGVDt25IDc1AMBzbzAEbdYoRpcQL5L3hL0uxLUWemL22ZzcZlFiWZTBcOMjWSDAL0eRD2P2RTvod3c7HFPul+GJYZpfoTW5UWfyvKWhrYdie4cP+0PYmPWbXvCqdgP6+YGgu/xPMXSvB/xPMo4ugzimsOTPpdepCCsfjiUKaZFLqMXH2OgMREH7Y3/lzt+Ej4OoaZrhIwtPwSYjGC6FqzJ2Zxtasl9uNYqbx0aQSKEGx5RkBX6JGIl52tVv+cbz6IuI6Zdln9qWTb0L3qDgYZgiwe3eQMO3OVBufBLunQ/8zEdzONBS27WV/jlZBa9JCvcXozFwxkpOzr4ZefVXNb6IUf2y3KpTxB58xdGAbTYzZh+deQ==
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 568cba78-231b-4f89-0685-08db56435fc3
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 May 2023 19:25:57.1908
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: VqJE31NQJ+TUwunk+UXylXK13SzC9XEuOfKGptjWrAqpErWTvs/l6AaD2xPIr93rFTJjwXxhyg0z1pON8VsSYQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4785
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-05-16_11,2023-05-16_01,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 phishscore=0
 mlxlogscore=999 spamscore=0 adultscore=0 malwarescore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2304280000
 definitions=main-2305160163
X-Proofpoint-GUID: F6M999p_WqOW9UlOTujgZ_dcKt0QYcpL
X-Proofpoint-ORIG-GUID: F6M999p_WqOW9UlOTujgZ_dcKt0QYcpL
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On May 16, 2023, at 3:23 PM, Jeff Layton <jlayton@kernel.org> wrote:
>=20
> On Tue, 2023-05-02 at 14:14 +0000, Chuck Lever III wrote:
>>=20
>>> On May 2, 2023, at 7:01 AM, Jiri Slaby <jirislaby@kernel.org> wrote:
>>>=20
>>> On 08. 01. 23, 17:31, Chuck Lever wrote:
>>>> From: Chuck Lever <chuck.lever@oracle.com>
>>>> To navigate around the space that svcauth_gss_accept() reserves
>>>> for the RPC payload body length and sequence number fields,
>>>> svcauth_gss_release() does a little dance with the reply's
>>>> accept_stat, moving the accept_stat value in the response buffer
>>>> down by two words.
>>>> Instead, let's have the ->accept() methods each set the proper
>>>> final location of the accept_stat to avoid having to move
>>>> things.
>>>=20
>>> Hi,
>>>=20
>>> I bisected to this (4bcf0343e8)
>>=20
>> Assuming you did the bisect on the NFS server's kernel?
>>=20
>>=20
>>> as it breaks nfs3-only servers in 6.3. I.e. /etc/nfs.conf containing:
>>> [nfsd]
>>> vers4=3Dno
>>=20
>> Note: Changing the settings in /etc/nfs.conf had no effect
>> on my server, so I effected the change by stopping the
>> server and poking values into /proc/fs/nfsd/versions by
>> hand.
>>=20
>> Steve?
>>=20
>>=20
>>> The client sees:
>>> mount("10.0.2.15:/tmp", "/mnt", "nfs", 0, "vers=3D4.2,addr=3D10.0.2.15,=
clientad"...) =3D -1 EIO (Input/output error)
>>> write(2, "mount.nfs: mount system call fai"..., 45
>>> mount.nfs: mount system call failed for /mnt
>>>=20
>>> And the kernel says:
>>> nfs4_discover_server_trunking unhandled error -5. Exiting with error EI=
O
>>>=20
>>> I reported in downstream as:
>>> https://bugzilla.suse.com/show_bug.cgi?id=3D1210995
>>>=20
>>> It cannot be reverted cleanly on the top of 6.3.
>>>=20
>>> Any ideas?
>>=20
>> I can reproduce a similar problem. Network capture shows
>> that the server is responding with NFS4ERR_NOENT to the
>> EXCHANGE_ID operation, and the client kernel log says:
>>=20
>>> nfs4_discover_server_trunking unhandled error -121. Exiting with error =
EIO
>>=20
>> That's not the failure mode I expected given the commit
>> you bisected to, so it might not be the same problem you've
>> hit. I'll troubleshoot this and send a fix for testing.
>>=20
>=20
> Alex hit this problem in testing too, and I took a quick look.
>=20
> In the attached capture, the client should have gotten back a
> RPC_PROG_MISMATCH error, but the server has recorded an extra successful
> accept state before encoding the RPC_PROG_MISMATCH error, leading to a
> malformed reply.
>=20
> I think that the problem is that encoding the accept status too early
> means that we can't properly handle failures from the pg_init_request
> call.
>=20
> Chuck, any thoughts on how you'd like to handle this?

With this:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/commit/?h=3Dn=
fsd-fixes&id=3D29cd2927fb914cc53b5ba4f67d2b74695c994ba4

I plan to send the fix to Linus tomorrow.


--
Chuck Lever