Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever <chuck.lever@oracle.com>
To:     Anton Ivanov <anton.ivanov@cambridgegreys.com>
CC:     Salvatore Bonaccorso <carnil@debian.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        "940821@bugs.debian.org" <940821@bugs.debian.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        Bruce Fields <bfields@fieldses.org>
Subject: Re: NFS Caching broken in 4.19.37
Thread-Topic: NFS Caching broken in 4.19.37
Thread-Index: AQHXB8OUt1bYm8JlxU2kOK1wOnml66phej8AgAAAwQA=
Date:   Sat, 20 Feb 2021 20:16:26 +0000
Message-ID: <BEBA9809-373A-4172-B4AD-E19D82E56DB1@oracle.com>
References: <5022bdc4-9f3e-9756-cbca-ada37f88ecc7@cambridgegreys.com>
 <YDFrN0rZAJBbouly@eldamar.lan>
 <af5cebbd-74c9-9345-9fe8-253fb96033f6@cambridgegreys.com>
In-Reply-To: <af5cebbd-74c9-9345-9fe8-253fb96033f6@cambridgegreys.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <46D9364746425442B756E50465B0736E@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: a960cf5c-4a63-454c-3770-08d8d5dc670e
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Feb 2021 20:16:26.9898
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: YwYwFQhSU5xuq5KMLzt41QA6wSMHmW7lX2RQ0Ys7+PqQz+t87cCtjC0n9gM/1SbDJOUDv925Jjy4i8x+uvEynQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4624
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9901 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 adultscore=0 phishscore=0
 malwarescore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000
 definitions=main-2102200188
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9901 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0
 mlxlogscore=999 clxscore=1011 phishscore=0 mlxscore=0 impostorscore=0
 spamscore=0 priorityscore=1501 bulkscore=0 malwarescore=0
 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2009150000 definitions=main-2102200188
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Feb 20, 2021, at 3:13 PM, Anton Ivanov <anton.ivanov@cambridgegreys.co=
m> wrote:
>=20
> On 20/02/2021 20:04, Salvatore Bonaccorso wrote:
>> Hi,
>>=20
>> On Mon, Jul 08, 2019 at 07:19:54PM +0100, Anton Ivanov wrote:
>>> Hi list,
>>>=20
>>> NFS caching appears broken in 4.19.37.
>>>=20
>>> The more cores/threads the easier to reproduce. Tested with identical
>>> results on Ryzen 1600 and 1600X.
>>>=20
>>> 1. Mount an openwrt build tree over NFS v4
>>> 2. Run make -j `cat /proc/cpuinfo | grep vendor | wc -l` ; make clean i=
n a
>>> loop
>>> 3. Result after 3-4 iterations:
>>>=20
>>> State on the client
>>>=20
>>> ls -laF /var/autofs/local/src/openwrt/build_dir/target-mips_24kc_musl/l=
inux-ar71xx_tiny/linux-4.14.125/arch/mips/include/generated/uapi/asm
>>>=20
>>> total 8
>>> drwxr-xr-x 2 anivanov anivanov 4096 Jul  8 11:40 ./
>>> drwxr-xr-x 3 anivanov anivanov 4096 Jul  8 11:40 ../
>>>=20
>>> State as seen on the server (mounted via nfs from localhost):
>>>=20
>>> ls -laF /var/autofs/local/src/openwrt/build_dir/target-mips_24kc_musl/l=
inux-ar71xx_tiny/linux-4.14.125/arch/mips/include/generated/uapi/asm
>>> total 12
>>> drwxr-xr-x 2 anivanov anivanov 4096 Jul  8 11:40 ./
>>> drwxr-xr-x 3 anivanov anivanov 4096 Jul  8 11:40 ../
>>> -rw-r--r-- 1 anivanov anivanov   32 Jul  8 11:40 ipcbuf.h
>>>=20
>>> Actual state on the filesystem:
>>>=20
>>> ls -laF /exports/work/src/openwrt/build_dir/target-mips_24kc_musl/linux=
-ar71xx_tiny/linux-4.14.125/arch/mips/include/generated/uapi/asm
>>> total 12
>>> drwxr-xr-x 2 anivanov anivanov 4096 Jul  8 11:40 ./
>>> drwxr-xr-x 3 anivanov anivanov 4096 Jul  8 11:40 ../
>>> -rw-r--r-- 1 anivanov anivanov   32 Jul  8 11:40 ipcbuf.h
>>>=20
>>> So the client has quite clearly lost the plot. Telling it to drop cache=
s and
>>> re-reading the directory shows the file present.
>>>=20
>>> It is possible to reproduce this using a linux kernel tree too, just ta=
kes
>>> much more iterations - 10+ at least.
>>>=20
>>> Both client and server run 4.19.37 from Debian buster. This is filed as
>>> debian bug 931500. I originally thought it to be autofs related, but IM=
HO it
>>> is actually something fundamentally broken in nfs caching resulting in =
cache
>>> corruption.
>> According to the reporter downstream in Debian, at
>> https://bugs.debian.org/940821#26 thi seem still reproducible with
>> more recent kernels than the initial reported. Is there anything Anton
>> can provide to try to track down the issue?
>>=20
>> Anton, can you reproduce with current stable series?
>=20
> 100% reproducible with any kernel from 4.9 to 5.4, stable or backports. I=
t may exist in earlier versions, but I do not have a machine with anything =
before 4.9 to test at present.

Confirming you are varying client-side kernels. Should the Linux
NFS client maintainers be Cc'd?


> From 1-2 make clean && make  cycles to one afternoon depending on the num=
ber of machine cores. More cores/threads the faster it does it.
>=20
> I tried playing with protocol minor versions, caching options, etc - it i=
s still reproducible for any nfs4 settings as long as there is client side =
caching of metadata.
>=20
> A.
>=20
>>=20
>> Regards,
>> Salvatore
>>=20
>=20
> --=20
> Anton R. Ivanov
> Cambridgegreys Limited. Registered in England. Company Number 10273661
> https://www.cambridgegreys.com/

--
Chuck Lever