Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Al Viro <viro@zeniv.linux.org.uk>
CC:     Benjamin Coddington <bcodding@redhat.com>,
        Olga Kornievskaia <aglo@umich.edu>,
        Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: Is this nfsd kernel oops known?
Thread-Topic: Is this nfsd kernel oops known?
Thread-Index: AQHYvJP9R6JamLcYaUyLf/owehh7vq3HuG2AgALiPACAAGl2gIAAWREAgAdmpICAAAw8AIAAo8KAgACEegCABUMMAIAAAf4A
Date:   Sat, 10 Sep 2022 21:21:11 +0000
Message-ID: <9D6CDF68-6B12-44DE-BC01-3BD0251E7F94@oracle.com>
References: <CAN-5tyGkHd+wEHC5NwQGRuQsJie+aPu0RkWNrp_wFo4e+JcQgA@mail.gmail.com>
 <5c423fdf25e6cedb2dcdbb9c8665d6a9ab4ad4b1.camel@kernel.org>
 <CAN-5tyEOTVDhR6FgP7nPVon76qhKkexaWB8AJ_iBVTp6iYOk1g@mail.gmail.com>
 <11BEA7FE-4CBC-4E5C-9B68-A0310CF1F3BE@oracle.com>
 <CAN-5tyHOugPeTsu+gBJ1tkqawyQDkfHXrO=vQ6vZTTzWJWTqGA@mail.gmail.com>
 <D0A6E504-F2C2-4A5F-BC51-FD3D88A790F0@redhat.com>
 <CAN-5tyHYH7ODzmTK=Maa3NZOSxfcE0mfaWY11+n2htQpya869g@mail.gmail.com>
 <EE9C1D1C-AA5B-48BC-9E3A-8A4523456AEE@oracle.com>
 <25AF9743-A2A2-4AFE-9123-BAD3C8F17655@redhat.com> <Yxz+GhK7nWKcBLcI@ZenIV>
In-Reply-To: <Yxz+GhK7nWKcBLcI@ZenIV>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?QP7sQxZcDr8ZtP9st+Iht20D9lTM7MVnVdmQu0ppTKR6z54CyCpOmDGS4Aj0?=
 =?us-ascii?Q?oQSBH2hxMHNTdgxOtR9si1Pzpmg1BCOAaO7U0afUiS725S8BZjBOzLGDlfoi?=
 =?us-ascii?Q?N7DVRWdsJgMGtE9sF9fNrcZL9rJFzx6GDjAnihFoWbMd2OAuxvsihyMQdgzT?=
 =?us-ascii?Q?AqUBg/cNtCPOPq8p3WW3y+GD8Y25qRD/gNEOmps6Gc6acYpW+jpbI2yyrCq+?=
 =?us-ascii?Q?eC/9f+nXXxx377668X9etMPJxu2imUgq191+Jh/+cYyHIT+YxoYN5Mj6fTGJ?=
 =?us-ascii?Q?OLSoudVuYb78BymeirfoZdyEnb+lX34Nk0vwpWIm/b0BVZZeGq5JqGeeBuXQ?=
 =?us-ascii?Q?vx0ZR3T/+I72loZ7iayoW32JMpFmcbmLdmcXeSHZQ2nQdF5mO+8H6uAGbO36?=
 =?us-ascii?Q?tm3MlSczLmRMDX6d83MYhSMrlLjPcx5K0Jqbx2R9CSiRwlxjiWB0kwZWDvm2?=
 =?us-ascii?Q?H77CRrJWGdI5MzQGIFC/n/G65GwVASsAAUtZ0Ye1J/FMhoNo4YZ/qCmqR2Fj?=
 =?us-ascii?Q?FAZIl7Lxyi4+jdMeg0lyvhSUD6X9KWdm7EzBSf3Ram8IIZjut3kYT86Hiuox?=
 =?us-ascii?Q?dxOmLolc6gto5v4i2nXtnNTbXzLUqhWiNiz/91DT8l1bv80XgasEdxrDXWfR?=
 =?us-ascii?Q?rxLVxzSI48W4D+TOHyLQVZmJxuzzOhoylaROKRMfq8IqsiPRfZGm0YlW7yDN?=
 =?us-ascii?Q?htwTAEurMlVqIAR3bImNPmb0tPpbWU54WCVwPs0noxi/7vqr45P2Y0fTR0IS?=
 =?us-ascii?Q?53d+uuRUbqBsvNGTqdqcQEAM4R6C5MhRRhoEWaN97kErh/j01/rDfvo0VMfx?=
 =?us-ascii?Q?1dTOPJ+JSBJDQ7cNMdxCCNQ1jjlzgO0/XpoOCeby+Jy2CCUlgERJb7F8ZS1v?=
 =?us-ascii?Q?38I0wCtH2bc8Pn1RQWzZ62dgH83UY933AAGx02AE7852h7ZNqTekOEvbv1Lo?=
 =?us-ascii?Q?wNtb+q4aI7IiINJarRmWG8MnJCe1XH1AQIni8xGmZ+UTvNOUtfxP31qATN5V?=
 =?us-ascii?Q?VRLw2hxlDoOfGQsJxPheXBmd/i73LWpELNSXa3rtVsb9XkrX8eGf14CH0WDF?=
 =?us-ascii?Q?74WaAgrYnHHoYVQYdhcnK7tNXJiYKzvsQAiE9/iX/Ziav999a+YZ0dV2niyW?=
 =?us-ascii?Q?FfhM+owWOKG/m4Vm37QhD9CkQ5+9s6PeO186RzNXo0XcxixcjanwvlYRiNUt?=
 =?us-ascii?Q?2y1MSlbc/5QRvkZT8dNrFjXts7pLa4WmNnESpmIcONmNVq6eigIrKp8vjLUd?=
 =?us-ascii?Q?1DLRZtQqwACpBJ+EtPQUDsDNJJ3ICzWp56bzERqIoa96edehwtfWkuGWqONv?=
 =?us-ascii?Q?8Oyolg0Nr5jgXd19ChAZY43G06qtp10ukyjQesR9CftekdqdnTAdjCwgL8x6?=
 =?us-ascii?Q?ia4MXKtgYuHk5yWVvO60IXvL30l8+YHux2Hcb3n4DBfbWRnWzg7IvR48+WA8?=
 =?us-ascii?Q?mTt1cmC/8+pyJAymDUe3eYAavbWP2witFJC4bKccNVlapBut6Wz0GP/UrEhm?=
 =?us-ascii?Q?Ks9BG1VUd1vhmvWCfIJ9iSRlNoIIea2upPTnzlnNiviSH6rtG311v769W6gy?=
 =?us-ascii?Q?6dVIRK8Hx6vO0yD/X19IQKqsWno/6y9KHm+pxZSbOyKkqc5Rm+bq0l6U0amC?=
 =?us-ascii?Q?2w=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E418D1FAFED577429784B89EBBFB5AE5@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 78f17cf9-272a-43c5-c543-08da937262d2
X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Sep 2022 21:21:11.9725
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: l28KzFBLB7p3/ht3t/fXKCxQns+tJBTDEVEFwIiVFrhLCXQ8qwFVhuc37H/zD+0zMcdVjlr5qybWIn6rmxjclQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR10MB4457
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1
 definitions=2022-09-10_10,2022-09-09_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 bulkscore=0 spamscore=0
 mlxscore=0 phishscore=0 suspectscore=0 malwarescore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000
 definitions=main-2209100079
X-Proofpoint-ORIG-GUID: ltOz8Rnk0t6mL5cA1OFjVYUMHfMk1VNo
X-Proofpoint-GUID: ltOz8Rnk0t6mL5cA1OFjVYUMHfMk1VNo
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Sep 10, 2022, at 5:14 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
>=20
> On Wed, Sep 07, 2022 at 08:52:46AM -0400, Benjamin Coddington wrote:
>> On 7 Sep 2022, at 0:58, Chuck Lever III wrote:
>>=20
>>>> On Sep 6, 2022, at 3:12 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>=20
>>>> On Tue, Sep 6, 2022 at 2:28 PM Benjamin Coddington
>>>> <bcodding@redhat.com> wrote:
>>>>>=20
>>>>> On 1 Sep 2022, at 21:27, Olga Kornievskaia wrote:
>>>>>=20
>>>>>> Thanks Chuck. I first, based on a hunch, narrowed down that it's
>>>>>> coming from Al Viro's merge commit. Then I git bisected his
>>>>>> 32patches
>>>>>> to the following commit f0f6b614f83dbae99d283b7b12ab5dd2e04df979
>>>>>=20
>>>>> No crash for me after reverting
>>>>> f0f6b614f83dbae99d283b7b12ab5dd2e04df979.
>>>>=20
>>>> I second that. No crash after a revert here.
>>>=20
>>> I bisected the new xfstests failures to the same commit:
>>>=20
>>> f0f6b614f83dbae99d283b7b12ab5dd2e04df979 is the first bad commit
>>> commit f0f6b614f83dbae99d283b7b12ab5dd2e04df979
>>> Author: Al Viro <viro@zeniv.linux.org.uk>
>>> Date:   Thu Jun 23 17:21:37 2022 -0400
>>>=20
>>>    copy_page_to_iter(): don't split high-order page in case of
>>> ITER_PIPE
>>>=20
>>>    ... just shove it into one pipe_buffer.
>>>=20
>>>    Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
>>>=20
>>> lib/iov_iter.c | 21 ++++++---------------
>>> 1 file changed, 6 insertions(+), 15 deletions(-)
>>>=20
>>=20
>> I've been reliably reproducing this on generic/551 on xfs.  In the case
>> where it crashes, rqstp->rq_res.page_base is positive multiple of PAGE_S=
IZE
>> after getting set in nfsd_splice_actor(), and that with page_len overrun=
s
>> the 256 pages we have.
>>=20
>> With f0f6b614f83d reverted, rqstp->rq_res.page_base is always zero.
>>=20
>> After 47b7fcae419dc and f0f6b614f83d, buf->offset in nfsd_splice_actor c=
an
>> be a positive multiple of PAGE_SIZE, whereas before it was always just t=
he
>> offset into the page.
>>=20
>> Something like this might fix it up:
>>=20
>> diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
>> index 9f486b788ed0..d62963f36f03 100644
>> --- a/fs/nfsd/vfs.c
>> +++ b/fs/nfsd/vfs.c
>> @@ -849,7 +849,7 @@ nfsd_splice_actor(struct pipe_inode_info *pipe, stru=
ct
>> pipe_buffer *buf,
>>=20
>>        svc_rqst_replace_page(rqstp, buf->page);
>>        if (rqstp->rq_res.page_len =3D=3D 0)
>> -               rqstp->rq_res.page_base =3D buf->offset;
>> +               rqstp->rq_res.page_base =3D buf->offset % PAGE_SIZE;
>>        rqstp->rq_res.page_len +=3D sd->len;
>>        return sd->len;
>> }
>>=20
>> .. but we should check with Al about whether this needs to be fixed over=
 in
>> copy_page_to_iter_pipe(),  since I don't think pipe_buffer offset should=
 be
>> greater than PAGE_SIZE.
>>=20
>> Al, any thoughts?
>=20
> pipe_buffer offsets in general can be greater than PAGE_SIZE.  What's mor=
e,
> buffer *size* can be greater than PAGE_SIZE - it really can contain more
> than PAGE_SIZE worth of data.  In that case the page is a compound one, o=
f
> course.
>=20
> Regression is the combination of "splice from regular file to pipe hadn't
> produced that earlier, now it does" and "nfsd never needed to handle that=
".
>=20
> I don't believe that fix is correct.  *IF* you can't deal with compound
> page in sunrpc, you need a loop going by subpages in nfsd_splice_actor(),
> similar to one that used to be in copy_page_to_iter().  Could you try
> the following:
>=20
> nfsd_splice_actor(): handle compound pages
>=20
> pipe_buffer might refer to a compound page (and contain more than a PAGE_=
SIZE
> worth of data).  Theoretically it had been possible since way back, but
> nfsd_splice_actor() hadn't run into that until copy_page_to_iter() change=
.

It's also possible that recent simplifications I've done to the splice
read actor accidentally removed the ability to deal with compound pages.
You might want to review the commit history of nfsd_splice_actor() to
see if there is an historic version that would behave correctly with
the new copy_page_to_iter().

Is the need to deal with CompoundPage documented somewhere? If not,
perhaps nfsd_splice_actor() could mention it so that overzealous
maintainers don't break it again.


> Fortunately, the only thing that changes for compound pages is that we
> need to stuff each relevant subpage in and convert the offset into offset
> in the first subpage.
>=20
> Hopefully-fixes: f0f6b614f83d "copy_page_to_iter(): don't split high-orde=
r page in case of ITER_PIPE"
> Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
> ---
>=20
> diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
> index 9f486b788ed0..b16aed158ba6 100644
> --- a/fs/nfsd/vfs.c
> +++ b/fs/nfsd/vfs.c
> @@ -846,10 +846,14 @@ nfsd_splice_actor(struct pipe_inode_info *pipe, str=
uct pipe_buffer *buf,
> 		  struct splice_desc *sd)
> {
> 	struct svc_rqst *rqstp =3D sd->u.data;
> -
> -	svc_rqst_replace_page(rqstp, buf->page);
> -	if (rqstp->rq_res.page_len =3D=3D 0)
> -		rqstp->rq_res.page_base =3D buf->offset;
> +	struct page *page =3D buf->page;	// may be a compound one
> +	unsigned offset =3D buf->offset;
> +
> +	page +=3D offset / PAGE_SIZE;
> +	for (int i =3D sd->len; i > 0; i -=3D PAGE_SIZE)
> +		svc_rqst_replace_page(rqstp, page++);
> +	if (rqstp->rq_res.page_len =3D=3D 0)	// first call
> +		rqstp->rq_res.page_base =3D offset % PAGE_SIZE;
> 	rqstp->rq_res.page_len +=3D sd->len;
> 	return sd->len;
> }

--
Chuck Lever