Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Amir Goldstein <amir73il@gmail.com>
Cc:     Jan Kara <jack@suse.cz>, Alexander Viro <viro@zeniv.linux.org.uk>,
        linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH 0/2] fsnotify: fix softlockups iterating over d_subdirs
In-Reply-To: <CAOQ4uxhwFGddgJP5xPYDysoa4GFPYu6Bj7rgHVXTEuZk+QKYQQ@mail.gmail.com>
References: <20221013222719.277923-1-stephen.s.brennan@oracle.com>
 <20221018041233.376977-1-stephen.s.brennan@oracle.com>
 <CAOQ4uxgnW1An-3FJvUfYoixeycZ0w=XDfU0fh6RdV4KM9DzX_g@mail.gmail.com>
 <87edv44rll.fsf@oracle.com>
 <CAOQ4uxhwFGddgJP5xPYDysoa4GFPYu6Bj7rgHVXTEuZk+QKYQQ@mail.gmail.com>
Date:   Thu, 27 Oct 2022 15:06:48 -0700
Message-ID: <87czachqfb.fsf@oracle.com>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?q+fXoMuhbgTvIqYeZAvS8Oah0vfK4nWNIhRdYeuwRIi+mYLBMwSHsNzd9QWB?=
 =?us-ascii?Q?e42Gdxn9dITJ/u1A7W/2vB6gyzIyfYkWGuerIzS96R0HA8dqIPUR7mzMID1R?=
 =?us-ascii?Q?4kct9pOn53j+gCJG0RBpa32lpRSbzBtGWmeBUN+niHbeHT8Kyf5IEFO2fXxe?=
 =?us-ascii?Q?m6aGLtXpdKHHaz4QjYHkghy9U7pVLD2uF6VhyFeyqOdrlZxTLL9AIz51Wfdk?=
 =?us-ascii?Q?0sPP2hJfq0qQ4RaGe4d1wnY8XGpByqifiD8eE2ZsLGau62HRbi3Y+xlXBUM1?=
 =?us-ascii?Q?xRZYrJOWeRWRMnbHhs5isGo7N+qB80CgOvnyvCMd1sXpg9X9DXzpJ4AEknXB?=
 =?us-ascii?Q?6LsiEUnaymUrXMANlMsRQrrzJnH7BOBDmVPqlcdzwZk54yP0q4sYknHHDf5y?=
 =?us-ascii?Q?bo+6HWdV15fgoHhqjXIaSpv5C2Y4ztTclpTOHMK/dSa4pfZYzl3RQc4cVqX6?=
 =?us-ascii?Q?hsuswFbDOX/Wr3yY/rVwXcIfZk1811E+Ve60vrvLF3gZt+VgnDwljPPxYgFV?=
 =?us-ascii?Q?oiFxaZm/Pvvrbuk/6KL59DZ+KymkrTvjWUiggXWccNx2Kq8XSMWvs1xTKU6c?=
 =?us-ascii?Q?dDWlAjZE87oVLs0vvyZaWNyyNgBdrkmwn0bktja5i0MemdnTzoNChiIZCnac?=
 =?us-ascii?Q?lENtDnAjSMhZ9jQw8yoE4xI3PELOdyBxeroFqwHcKgouk7Iy4+NEQVxQVhHP?=
 =?us-ascii?Q?nwzuY6EmBC4LI0nQfCT/RphVF2wnrF5pX9DwqthSitcIVGF8QyZKNr3u3s41?=
 =?us-ascii?Q?HH6sVZaGM9NJnC6fbq7AeRHV2VQsRvB4g4ObHwdigaetsDciToB80DwttS3x?=
 =?us-ascii?Q?Osm5Ay66FgGiG6m4rBGmvEo5BP98baTxWnj6XQZxovlV1b5fegtC+nkXYhP6?=
 =?us-ascii?Q?mon4XYWEaLIMgUfG9iayOSKlbHL/SSnRKvil17HSxJLbf773qelzeZZnjv+H?=
 =?us-ascii?Q?0vTS5+DqR5dj+WtFM9iZuyDFHJ4zIFRo/zwdcq4nC0TKjsxYSWRUS2YzvnXD?=
 =?us-ascii?Q?Vh89t8mCZEMZeaZyklg376ot1PxHXM0+xP9PHfCXI3wNDVb8y4pLNFTQm9/7?=
 =?us-ascii?Q?1jiIC8fXHfhmvRA1JDAX1HuWUtfXJyHE7coQMTP/BYwzOq3RGw8WmF1c5qza?=
 =?us-ascii?Q?2ee7ktNG/WCxWnZ8t/dgNqd0VEXY2bpTd2Dtj3XmH1QgsJTJGYwKpscwl8uy?=
 =?us-ascii?Q?TM6FzcGYm1w2/8YKcyt9q7Nxgs7l3PdWO4eRCeAi9PnBIij09A0F4+1ooLVr?=
 =?us-ascii?Q?d7UGCOnpsbDASzhC6GN0UxEFzQpfXBPIUV7UXCKjK4lin3I2yFw3ZhmmTpgw?=
 =?us-ascii?Q?yIapO3Z7i+fmiYNadtsGs7mADzmDvJ8HO1I9gr7ANaTmBv/WmMVh8vfa4V51?=
 =?us-ascii?Q?MCAN/qJ3RQ9b2UkAtzxbOh4s9c1Gm857bTiKuzlcHlXeuW1Gc2Mqk7gy+m9I?=
 =?us-ascii?Q?hWwXD7XPKybvUO2NSC+JC/D41yil4/ndxSxllcSeMo4tiNdQeKJiF3A1gdp5?=
 =?us-ascii?Q?1UfxN/LDcbOIrEADd9Vxtt01RZz+g6+9pfkp0NTIfc4Jlu1ntHC4skUmOPnd?=
 =?us-ascii?Q?EwAfjF+50AtIwW+8nxHMXLlM7KiV1vfCEtCXbBCQ1TlvqwWwiEZjSYfEY08E?=
 =?us-ascii?Q?sQ=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5153559f-4e24-4533-5637-08dab8679142
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2022 22:06:58.6563
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: TW1GoI4L+jMjq/6OB7bOqVDsFjuHAz+dcRG9p2s0422zgLY0dMNgCXm6aH1Fxzump80COW0SKEgeBegd95hWLjr/h7BQJP3yDC7J7tBNM0c=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5551
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-10-27_07,2022-10-27_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0
 phishscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000
 definitions=main-2210270124
X-Proofpoint-GUID: F-MxDzEN7zyUMMzJBKjZsKkVsIXx2c9q
X-Proofpoint-ORIG-GUID: F-MxDzEN7zyUMMzJBKjZsKkVsIXx2c9q
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Amir Goldstein <amir73il@gmail.com> writes:
> On Wed, Oct 19, 2022 at 2:52 AM Stephen Brennan
> <stephen.s.brennan@oracle.com> wrote:
>>
>> Amir Goldstein <amir73il@gmail.com> writes:
>> > On Tue, Oct 18, 2022 at 7:12 AM Stephen Brennan
>> > <stephen.s.brennan@oracle.com> wrote:
>> >>
>> >> Hi Jan, Amir, Al,
>> >>
>> >> Here's my first shot at implementing what we discussed. I tested it using the
>> >> negative dentry creation tool I mentioned in my previous message, with a similar
>> >> workflow. Rather than having a bunch of threads accessing the directory to
>> >> create that "thundering herd" of CPUs in __fsnotify_update_child_dentry_flags, I
>> >> just started a lot of inotifywait tasks:
>> >>
>> >> 1. Create 100 million negative dentries in a dir
>> >> 2. Use trace-cmd to watch __fsnotify_update_child_dentry_flags:
>> >>    trace-cmd start -p function_graph -l __fsnotify_update_child_dentry_flags
>> >>    sudo cat /sys/kernel/debug/tracing/trace_pipe
>> >> 3. Run a lot of inotifywait tasks: for i in {1..10} inotifywait $dir & done
>> >>
>> >> With step #3, I see only one execution of __fsnotify_update_child_dentry_flags.
>> >> Once that completes, all the inotifywait tasks say "Watches established".
>> >> Similarly, once an access occurs in the directory, a single
>> >> __fsnotify_update_child_dentry_flags execution occurs, and all the tasks exit.
>> >> In short: it works great!
>> >>
>> >> However, while testing this, I've observed a dentry still in use warning during
>> >> unmount of rpc_pipefs on the "nfs" dentry during shutdown. NFS is of course in
>> >> use, and I assume that fsnotify must have been used to trigger this. The error
>> >> is not there on mainline without my patch so it's definitely caused by this
>> >> code. I'll continue debugging it but I wanted to share my first take on this so
>> >> you could take a look.
>> >>
>> >> [ 1595.197339] BUG: Dentry 000000005f5e7197{i=67,n=nfs}  still in use (2) [unmount of rpc_pipefs rpc_pipefs]
>> >>
>> >
>> > Hmm, the assumption we made about partial stability of d_subdirs
>> > under dir inode lock looks incorrect for rpc_pipefs.
>> > None of the functions that update the rpc_pipefs dcache take the parent
>> > inode lock.
>>
>> That may be, but I'm confused how that would trigger this issue. If I'm
>> understanding correctly, this warning indicates a reference counting
>> bug.
>
> Yes.
> On generic_shutdown_super() there should be no more
> references to dentries.
>
>>
>> If __fsnotify_update_child_dentry_flags() had gone to sleep and the list
>> were edited, then it seems like there could be only two possibilities
>> that could cause bugs:
>>
>> 1. The dentry we slept holding a reference to was removed from the list,
>> and maybe moved to a different one, or just removed. If that were the
>> case, we're quite unlucky, because we'll start looping indefinitely as
>> we'll never get back to the beginning of the list, or worse.
>>
>> 2. A dentry adjacent to the one we held a reference to was removed. In
>> that case, our dentry's d_child pointers should get rearranged, and when
>> we wake, we should see those updates and continue.
>>
>> In neither of those cases do I understand where we could have done a
>> dget() unpaired with a dput(), which is what seemingly would trigger
>> this issue.
>>
>
> I got the same impression.

Well I feel stupid. The reason behind this seems to be... that
d_find_any_alias() returns a reference to the dentry, and I promptly
leaked that. I'll have it fixed in v3 which I'm going through testing
now.

Stephen