Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   DANIEL K FORREST <dforrest@wisc.edu>
To:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Strange NFS file glob behavior
Thread-topic: Strange NFS file glob behavior
Thread-index: AQHYq428dDTNZxBbIk2YruyMT2DYiQ==
Date:   Tue, 9 Aug 2022 01:17:04 +0000
Message-id: <20220809011700.bdiikqngwmxp3abf@cosmos.ssec.wisc.edu>
Reply-to: DANIEL K FORREST <dforrest@wisc.edu>
Accept-Language: en-US
Content-language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?AxqH56p7zj0RrO7+ygVX5iGoDBLEJmPXV/Ike+iyLyoL9eHGa1m60skeN5uK?=
 =?us-ascii?Q?K8m2MwNDAcBQvweUxR1zbq6BrFam5dYWHWSRvQV1QqF2MUQWwkWPtxVthsew?=
 =?us-ascii?Q?jsoHrigYxJ0pejwzI4xXkmf2egNiGx5p/adm9Wh/PCMfGPyVeTLlQV6B5TYz?=
 =?us-ascii?Q?bnl6XiP/BLOFni9Pzvhz5Y7r4XG3DvLGpFIk2cCvpZu8YbqmiBo1tA7O8E2w?=
 =?us-ascii?Q?L5j5AuFvac588VJj5t7SXzIs/Lt4QUFN7tgGSdQqLM1qi02xRldMV0vY0Z6y?=
 =?us-ascii?Q?OfiDy0PRC0NagIa+XtRcsxTh3V8tSAll1MxlNbsHEXGETdUD4poyzwWG1Vc1?=
 =?us-ascii?Q?hnWwxF8J170xpol7LgvVL0gNxutmQH0EdjKIcAH/q68iTtqwT4DvmvZLLS4+?=
 =?us-ascii?Q?3KcH7W91Q4wZ5oobdvZcwB0iUOJpZEceMTZZ3PwdItrJOZhpA11yUkMtEP2H?=
 =?us-ascii?Q?HZTGOecYKKJYNZ1Zldux3OKkPKd0QMSbL4nXa2JrErJ7dLAplkkCHjBNQUk7?=
 =?us-ascii?Q?78I9vfFDz27LUwsdUl9BqQwdlArIhPN+E+fpYJvAv/38syMqFQdCrNJm1dhD?=
 =?us-ascii?Q?4c+IYJxM9lxE4gem/ByGbP7G1f4Fng4rJUYGI/sYdHuX6vyHQxGNkHoHrocx?=
 =?us-ascii?Q?yaCJQeEpK3miT3n5l1jyYN9MfhNanx327hFirjTNQWCNZWdELX9WQCzkJ+RB?=
 =?us-ascii?Q?3QTXGJWU3OEtPPu3PuCYihXWAjds/eBYTkaroClU88dxEb+x50h0TE8Wo+kV?=
 =?us-ascii?Q?Dh2uEAHkNcT4FCwm8Bt5oAeJUGF2makkQMzQgBbcILUdrkcofbpFqmaXwYmu?=
 =?us-ascii?Q?hzI4EMgkFVS5kjaVuYvC5abOjpzMquYng7I+e1qZiABVUi6cd+5PR+thuJ8D?=
 =?us-ascii?Q?6zEmfk++agdmxwDtqrmsgtF2lfh5p2vkQKY95m47h44VxZ/GwX+QhELeebNb?=
 =?us-ascii?Q?tVcNkiCKJOzLb7ITcN0mdk/P9ZVGKgGtdlhEf0pkJW+ScyFeGgPCm0XmQtjr?=
 =?us-ascii?Q?naFVu4R/VgLhxGazLnUOhr3pciuOkggpLcxU6EJ6qZ59zIVRS15O1tt3VNIN?=
 =?us-ascii?Q?+4wSyWLm/aCJ//K8245dtP6tyfrASUhOq01YwMf9NLq3J+zPbRRMHCQonaWT?=
 =?us-ascii?Q?/GehqbTNEsoLoOc9nmyPweE0tOkNkJJd4pUt8w47GuLxI5kv0e7d78Uuh1Sh?=
 =?us-ascii?Q?9Q2iob6LOZrLBKVONpxGdArfu/HoWWBgOqzoWYBmDjZaka6tMZ4a+eofQFcO?=
 =?us-ascii?Q?MaX8o4k7Lm2N7In7svJ34USmalijXVkuxmMtT/Ac2sbnr0WURzhSAFONQMzZ?=
 =?us-ascii?Q?KDK5ZHrWgxecm/zJsy74DHi5cIog0l71QSCsvsMRBKjxoa3KpzbhTm4x13bl?=
 =?us-ascii?Q?frbIvPpverr+2FC2K90hbfhyav7EsniRFODofNcjnN9fHJmSG62JTHgamjne?=
 =?us-ascii?Q?q5t5psLiJYyfYlaPl+eZZmd1AtbFrKP6/GRE/OWHlUQIUobkY6hmiH4Z0BCv?=
 =?us-ascii?Q?MNkKpfTUiSRbQogLf4pqA3qrLoyAI1dpO2euP3wgWzFIQAgJm7M5lgVJgB3y?=
 =?us-ascii?Q?7JP+15kvwPwKz7j4z7fq7rztK1rsd2Z5giNnCC0ZYbgkdSOkKBnZgV3udNUR?=
 =?us-ascii?Q?9Q=3D=3D?=
Content-type: text/plain; charset="us-ascii"
Content-id: <33461D355A88C3419F16B84CCB3F03C1@namprd06.prod.outlook.com>
Content-transfer-encoding: quoted-printable
MIME-version: 1.0
X-OriginatorOrg: wisc.edu
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN7PR06MB7166.namprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f249d10a-af23-4e84-0f05-08da79a4de9a
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2022 01:17:04.2111 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 2ca68321-0eda-4908-88b2-424a8cb4b0f9
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: knYN1yac2CxJ/mAO9wvnvHXm3Ya/JeTSxjvvqMSBq3JQxhuv6FJAD7UNh1Tj15SthJRVIuTXD5t5lIWs2QY2jA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR06MB5759
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

I am seeing a strange glob behavior on NFS that I can't explain.

The server has 16 files, foo/bar{01..16}.

There are other files in foo/, and there are many other processes on
the client accessing files in the directory, but the mount is readonly
so the only create/delete activity is on the server, and it's all
rsync, so create file and rename file, but no file deletions.


When the 16th file is created (random order) processing is triggered
by a message from a different host that is running the rsyncs.

On the client, I run this command:

$ stat -c'%z %n' foo/bar{01..16}

And I see all 16 files.

However, if I immediately follow that command with:

$ stat -c'%z %n' foo/bar*

On rare occasions I see fewer than 16 files.

The missing files are the ones most recently created, they can be seen
by stat when explicitly enumerated, but the shell glob does not see
all of the files.  This test is for verifying a problem with a program
that is also sometimes not seeing files using readdir/glob.


How can all 16 files be seen by stat, but not by readdir/glob?


OS is CentOS 7.9.2009, 3.10.0-1127.19.1.el7.x86_64
NFS mount is version 3, readonly, nordirplus, lookupcache=3Dpos


--=20
Daniel K. Forrest		Space Science and
dforrest@wisc.edu		Engineering Center
(608) 890 - 0558		University of Wisconsin, Madison=