Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Alexander Viro <viro@zeniv.linux.org.uk>, Jan Kara <jack@suse.cz>
Cc:     Stephen Brennan <stephen.s.brennan@oracle.com>,
        linux-kernel@vger.kernel.org, Luis Chamberlain <mcgrof@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        linux-fsdevel@vger.kernel.org, Arnd Bergmann <arnd@arndb.de>,
        Amir Goldstein <amir73il@gmail.com>
Subject: [PATCH v2 0/4] Fix softlockup when adding inotify watch
Date:   Wed,  9 Feb 2022 15:14:02 -0800
Message-Id: <20220209231406.187668-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?KHjjDlKv61GihOmcZqZ4glA4/CdcqwdUZfaBpAyiUwxuPwuLBRjsq2D0hzNP?=
 =?us-ascii?Q?MhivBAG588ZxZn+rwLSsvjEQm9we6gH6AzNfLDni5JCjqzxYA5qxlIWzR7wS?=
 =?us-ascii?Q?f+ktiIBAyYBe8jnbuai1QCVOl9FCJ7bA9QcCB4pS11mpO3bUoz0mpCSWvDfU?=
 =?us-ascii?Q?WI9HOVlBS7ZXz4Rg1+SOgmM2nzUAYTmap69J274lpxIsII1syFYmC0k6hbUk?=
 =?us-ascii?Q?iqnf0I4HyBTmNFUFSQZkXmTSIVHi2mJgee+UW9pidkyR+x/Lk0rxWEQ0xxWl?=
 =?us-ascii?Q?1Y0DgtUnLah/vR1yX96gaUwZnm7qJrHPEDz8n8LFE0mc/WKymVpIm8hpQ+nm?=
 =?us-ascii?Q?3bLDFm9ItYFKU2YzmflQe4bChq0LsEtk23oI8UhV3cYt4gIRp3Oi21lVNP5m?=
 =?us-ascii?Q?1jzLdWLQFf+8DxMEmZX5QdQl6PcHQ4uhkdDDJtMld29B75LpocFP41xl968h?=
 =?us-ascii?Q?GExdEEd/OCqcyBz+p2uDPUy6TS2HA/eeRcVV1boAmMBwbrajBcrK+UDtPDyE?=
 =?us-ascii?Q?bGupEiD2ATBMem0NnYKtZnJhmR53DpqlFsOjtvh9L9NOTnT9SkhnbceyMYXt?=
 =?us-ascii?Q?h2wx7AMIfPre9j0b/D1hdFL023jXkPeiXYKKt/Qvt9StOF/5FKaokcgcIRwR?=
 =?us-ascii?Q?mo/Bjo83O+kP+2z52EQT82/rbW8MTtKEuG1Stfx0px3gYsdbk20lkLD37LpB?=
 =?us-ascii?Q?28XbQWlHGAYbi2diHafYYVcBPdTEV1AUngt1ne33eAgW4vXOpst0iwnhOpHQ?=
 =?us-ascii?Q?mhUoc45fvkc6P68TimzHXlZTS/s9/8HwDqEBE0Gzp13qCZrNfWgNAYAqtQtf?=
 =?us-ascii?Q?/u6ByzgfEsMiyad21W5M8qFMQLD/f0CeBSQp2EVCUyGMTFli+sLmK5iilkwt?=
 =?us-ascii?Q?hTIeGUDn4jqebESm7qY7MWbWbsoMLhG548SzCuxykr8neIqDTMqSm1EUOSpO?=
 =?us-ascii?Q?L8C8Ay16kDVmMg3Mt/knHVciwHavYa/6GgmUDFI6OfmZqlD9ioZc1lyDWVnp?=
 =?us-ascii?Q?uD1q6/ZtqEq+Tf/0q4meYYOUjXBwhi+ds4sYiAPLDcAxu/uOFZGKjC+XcyZ9?=
 =?us-ascii?Q?p0ePC6geW9UIQUd/wBAWOIGQ4HgbfiOJ2d1nUhsZVmjLqPoC8TZGTmRqgoq/?=
 =?us-ascii?Q?12rx+UIr7XcUwwuPvqRhyQL1uctt67bB5r5aS1Lebq1VT2IQxuMhKNcG/tqz?=
 =?us-ascii?Q?uaYwieIgahh2f8E/yXLJhbO/dw7BqPMHNKkXPIvw/76tpbL5fOeFKSDmtWXs?=
 =?us-ascii?Q?iwksgNJM+VTjNq2fBpktyvouxiMiSSaoo1v9cFBYZXC2QBC3P59CA+DqgYHy?=
 =?us-ascii?Q?8lPtRKbkmKOB3qLz/CROEfjm4rTpPI4+6STsGxgaVQif/tlXYE7ll+94v1a7?=
 =?us-ascii?Q?LBrWvEKwUkiej9o1F0QkkzMPJLASKVLqRcDxXZryBg6NnACog/0DaJ5Sz6uB?=
 =?us-ascii?Q?5IbhIR9c+DWc/hOoazg8UwQlzb9cgdp6jEfiS6AW5p2KeGr7EQ9HAhFMGs6L?=
 =?us-ascii?Q?HdpGA4/9LMiZ90q+WmTdFDZt43+W1xhpzViXTn522MYeDuKYOK8EWhsNZQGe?=
 =?us-ascii?Q?STDhaspvcSVEbYVr/y9LdUGx4817qb9J4xL4L5Rc6dThMdeyNqhF8d3EveBH?=
 =?us-ascii?Q?dXx+KK3xFJd1HzI8jhhB03AGQa+VRiHqjUfnD9OLLzy0LY4bRta9KiexLdFg?=
 =?us-ascii?Q?tbAggA=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7c289e84-994b-44de-2540-08d9ec21dffe
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2022 23:14:08.7717
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /Yr4aajQZPU87QIDZJ9sulIphrT+p50CoAN+i/IKLA07L4NLKu0abOClXF6vCo+dgJYn4nc7ou3qPavUZ0bTEiMlo4TfNDPv6jc349R8118=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4884
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10253 signatures=673431
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 bulkscore=0
 malwarescore=0 suspectscore=0 phishscore=0 adultscore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2202090121
X-Proofpoint-ORIG-GUID: bTReoCX5ernxfBYeYrH7sOqheQYwhkF_
X-Proofpoint-GUID: bTReoCX5ernxfBYeYrH7sOqheQYwhkF_
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Al et al,

When a system with large amounts of memory has several millions of
negative dentries in a single directory, a softlockup can occur while
adding an inotify watch:

 watchdog: BUG: soft lockup - CPU#20 stuck for 9s! [inotifywait:9528]
 CPU: 20 PID: 9528 Comm: inotifywait Kdump: loaded Not tainted 5.16.0-rc4.20211208.el8uek.rc1.x86_64 #1
 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
 RIP: 0010:__fsnotify_update_child_dentry_flags+0xad/0x120
 Call Trace:
  <TASK>
  fsnotify_add_mark_locked+0x113/0x160
  inotify_new_watch+0x130/0x190
  inotify_update_watch+0x11a/0x140
  __x64_sys_inotify_add_watch+0xef/0x140
  do_syscall_64+0x3b/0x90
  entry_SYSCALL_64_after_hwframe+0x44/0xae

This patch series is a modified version of the following:
https://lore.kernel.org/linux-fsdevel/1611235185-1685-1-git-send-email-gautham.ananthakrishna@oracle.com/

The strategy employed by this series is to move negative dentries to the
end of the d_subdirs list, and mark them with a flag as "tail negative".
Then, readers of the d_subdirs list, which are only interested in
positive dentries, can stop reading once they reach the first tail
negative dentry. By applying this patch, I'm able to avoid the above
softlockup caused by 200 million negative dentries on my test system.
Inotify watches are set up nearly instantly.

Previously, Al expressed concern for:

1. Possible memory corruption due to use of lock_parent() in
sweep_negative(), see patch 01 for fix.
2. The previous patch didn't catch all ways a negative dentry could
become positive (d_add, d_instantiate_new), see patch 01.
3. The previous series contained a new negative dentry limit, which
capped the negative dentry count at around 3 per hash bucket. I've
dropped this patch from the series.

Patches 2-4 are unmodified from the previous posting.

In v1 of the patch, a warning was triggered by patch 1:
https://lore.kernel.org/linux-fsdevel/20211218081736.GA1071@xsang-OptiPlex-9020/

I reproduced this warning, and verified it no longer occurs with my patch on
5.17 rc kernels. In particular, commit 29044dae2e74 ("fsnotify: fix fsnotify
hooks in pseudo filesystems") resolves the warning, which I verified on the
5.16 branch that the 0day bot tested. It seems that nfsdfs was using d_delete
to remove some pseudo-filesystem dentries, rather than d_drop, but it
expected there to never be negative dentries. I don't believe that
warning reflected an error in this patch series.

v2:
- explain the nfsd warning
- remove sweep_negative() call from __d_add - rely on dput() for that


Konstantin Khlebnikov (2):
  dcache: add action D_WALK_SKIP_SIBLINGS to d_walk()
  dcache: stop walking siblings if remaining dentries all negative

Stephen Brennan (2):
  dcache: sweep cached negative dentries to the end of list of siblings
  fsnotify: stop walking child dentries if remaining tail is negative

 fs/dcache.c            | 101 +++++++++++++++++++++++++++++++++++++++--
 fs/libfs.c             |   3 ++
 fs/notify/fsnotify.c   |   6 ++-
 include/linux/dcache.h |   6 +++
 4 files changed, 110 insertions(+), 6 deletions(-)

-- 
2.30.2