Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Jan Kara <jack@suse.cz>, Alexander Viro <viro@zeniv.linux.org.uk>
Cc:     Amir Goldstein <amir73il@gmail.com>, linux-fsdevel@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: [PATCH 0/2] fsnotify: fix softlockups iterating over d_subdirs
Date:   Mon, 17 Oct 2022 21:12:31 -0700
Message-Id: <20221018041233.376977-1-stephen.s.brennan@oracle.com>
In-Reply-To: <20221013222719.277923-1-stephen.s.brennan@oracle.com>
References: <20221013222719.277923-1-stephen.s.brennan@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?EtugbzTMJ/2NSMfos6aA0heY+es0LZZxh0OhymWNlH6AAeTl8SRTgq/2hiYu?=
 =?us-ascii?Q?RN1JPIpmDiBvimEQsD8ly0jLDej+BgSTh4zpFNgY3RYNQ6A0DCz028wZzYN5?=
 =?us-ascii?Q?taAcq8VZQBWv6JLx8+WOVlaEuutuKX2qPXdGYSfrynAOYz8g3k4BgDiP3A7a?=
 =?us-ascii?Q?SBECtSzVlJUwQmBZft8yw3ogNojQxGNJNKXp/8Eo5OC93XKB8JXn60gtBkrP?=
 =?us-ascii?Q?Kpsc51Vb+WVP775/Ppi4S7H8RY9dzuhJufIkyPgUG/PQfRINz31QyAvOz+ld?=
 =?us-ascii?Q?bt0kuD9PNtAxbcQAPusAM8osqgZ/QJmA7rz/CQC34mHEX7MWDcaOq5OIrq6e?=
 =?us-ascii?Q?Ep9d1pyocgvvPA54qPMXzl9qOQ/JeLidJAfQvexGqI1DIs0KHp/H8XZJZ3uz?=
 =?us-ascii?Q?gTyaSjtSe8f9fg6HdSuezDD5IWNCVlglYhOJLl3qB+HW6iNEYDNspU4xYZIp?=
 =?us-ascii?Q?uawmILHKLTuZ7S6/FJmzjXHEAdbvax/wbfiBdYSWF3Y6TeAC745ekFSwy2PH?=
 =?us-ascii?Q?XJagHmOCsPj27x22DS0O1s6GdpYqsW33H1nav1TOfPBozWAtxTnlGeTXqD93?=
 =?us-ascii?Q?+oOvIy0YcFxtbrNTEZN57YyfDFCw2llmUx7F6a3293QSAsuRz5af4tIgDeG/?=
 =?us-ascii?Q?OYtiNC5225MQ/BnfxLrJcu8TpnF5kWgRcWmOPCMBwJ0RPPKC22oj27mP+XsT?=
 =?us-ascii?Q?mi/aix8f+1qch+kLEb8BD1tZWV1d6Z+QEEUc9y/ZEzqyzCAa87GpdjCTFpQc?=
 =?us-ascii?Q?FtZaPbWtWXKM5tYAHZikFI+WLdLC1RHQoHxJ25cy5NkQGFhaguHqsacxzq+c?=
 =?us-ascii?Q?eP6QqL+wZL+i6d8lqCHSRD6gJ0XWIGQaltldYcE96sWw22c07K3Uv2EjnWBl?=
 =?us-ascii?Q?t+qrFt4Hw0S5gQ7Kc+KXLHdDb3uKtwUzZ5gMa+MU7jEYLLlzGp6lYwU83Urt?=
 =?us-ascii?Q?eilmgn/zHdzXTrFsf/nIzjUiyCBe+CfXJV8D4GP43W5Zxri8NjoEfgTlkSLh?=
 =?us-ascii?Q?cnY12+EYBMD5Hgho1//iYXPKR+INZaUxNNJWRjT4lZhh4D+ZpyO8FXSwh4tk?=
 =?us-ascii?Q?LgIZJpCQtnwltTS/QweFaZfOZ3KsiCx/d510wySoScI03jbkssqxFUDq1fTe?=
 =?us-ascii?Q?q9YmVPLq58dt/GDCGnuZ0ctRNCOYSO38/L48w64S1jK9wHi9urg5qY35x50u?=
 =?us-ascii?Q?uA9qSepYjL/7qaUTMiQNLCDgkJAgRGjB1Mth8ak7A79IWOBEv/YV16xBrFzs?=
 =?us-ascii?Q?d8UlCFZEEpiLCzW+4zdl08jcFWmmY1m4aU+Xz7ECJbkP2MtMFXvlEOHKSzf4?=
 =?us-ascii?Q?kgsTWYaagDVjvZtR8zT/Qr6OIzVpUHukA0Akvp/mV+VE9tgVkKZkSK0SAsFx?=
 =?us-ascii?Q?hVcij6ETnihDv/M6nJ5ZuIexBt8P7TlWDmx2s8WKABIpCCiYJbNvd3g7YQ1Z?=
 =?us-ascii?Q?H0ym/uw0tVRIs5S+HXI3hWGaARKRBWsjXTtsOlKxjRcLjiyQhhEw9yoIUlZ4?=
 =?us-ascii?Q?hZfT4xDzn3h7dWLbjrWwCNv8SPN1umMHsjTwwOUiRWIih4GpxFL14/Y4EQp9?=
 =?us-ascii?Q?roShbbRSJXE2B3VMs0asPjyywmS/i2Qk8LbUQlqeEza3Bs7FtC5xECr/avgI?=
 =?us-ascii?Q?HA=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: da74a9cb-a302-470e-e5a3-08dab0befc07
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2022 04:12:34.7340
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: TqYhD6zKv3PwzxxKRj/9qXyJkii7FcY322Y2flgvnMbkut96p+G5AzfJH2bXTvxEpKP0jXD7vsY8rX04zjjnt8/kwrZLCZgx+XXqbdycZRg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4567
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-10-17_13,2022-10-17_02,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 bulkscore=0
 malwarescore=0 phishscore=0 mlxlogscore=999 spamscore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000
 definitions=main-2210180022
X-Proofpoint-ORIG-GUID: wKKcWA8bcF1MbkxVvXx1I5e9-8zA-rRH
X-Proofpoint-GUID: wKKcWA8bcF1MbkxVvXx1I5e9-8zA-rRH
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jan, Amir, Al,

Here's my first shot at implementing what we discussed. I tested it using the
negative dentry creation tool I mentioned in my previous message, with a similar
workflow. Rather than having a bunch of threads accessing the directory to
create that "thundering herd" of CPUs in __fsnotify_update_child_dentry_flags, I
just started a lot of inotifywait tasks:

1. Create 100 million negative dentries in a dir
2. Use trace-cmd to watch __fsnotify_update_child_dentry_flags:
   trace-cmd start -p function_graph -l __fsnotify_update_child_dentry_flags
   sudo cat /sys/kernel/debug/tracing/trace_pipe
3. Run a lot of inotifywait tasks: for i in {1..10} inotifywait $dir & done

With step #3, I see only one execution of __fsnotify_update_child_dentry_flags.
Once that completes, all the inotifywait tasks say "Watches established".
Similarly, once an access occurs in the directory, a single
__fsnotify_update_child_dentry_flags execution occurs, and all the tasks exit.
In short: it works great!

However, while testing this, I've observed a dentry still in use warning during
unmount of rpc_pipefs on the "nfs" dentry during shutdown. NFS is of course in
use, and I assume that fsnotify must have been used to trigger this. The error
is not there on mainline without my patch so it's definitely caused by this
code. I'll continue debugging it but I wanted to share my first take on this so
you could take a look.

[ 1595.197339] BUG: Dentry 000000005f5e7197{i=67,n=nfs}  still in use (2) [unmount of rpc_pipefs rpc_pipefs]

Thanks!
Stephen

Stephen Brennan (2):
  fsnotify: Protect i_fsnotify_mask and child flags with inode rwsem
  fsnotify: allow sleepable child flag update

 fs/notify/fsnotify.c | 84 +++++++++++++++++++++++++++++++-------------
 fs/notify/mark.c     | 55 ++++++++++++++++++++++-------
 2 files changed, 103 insertions(+), 36 deletions(-)

-- 
2.34.1