Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Stephen Brennan <stephen.s.brennan@oracle.com>
To:     Jan Kara <jack@suse.cz>
Cc:     Jan Kara <jack@suse.cz>, linux-fsdevel@vger.kernel.org,
        linux-kernel@vger.kernel.org, Amir Goldstein <amir73il@gmail.com>,
        Al Viro <viro@zeniv.linux.org.uk>
Subject: Re: [PATCH v3 0/3] fsnotify: fix softlockups iterating over d_subdirs
In-Reply-To: <20221101175144.yu3l5qo5gfwfpatt@quack3>
References: <20221021010310.29521-1-stephen.s.brennan@oracle.com>
 <20221028001016.332663-1-stephen.s.brennan@oracle.com>
 <20221101175144.yu3l5qo5gfwfpatt@quack3>
Date:   Tue, 01 Nov 2022 13:48:54 -0700
Message-ID: <877d0eh03t.fsf@oracle.com>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?QR93s5mkCF7ktzvUuKc3BNQ2M2BJjHiDmfLAi4s62YOuPcFt2eCJ/N3lnwuF?=
 =?us-ascii?Q?OXvAS2WwgXFCqB5LAMUqUKht7qEbjDLJzUiK5mzxW48/MNapS2r+lt/Q3gDK?=
 =?us-ascii?Q?nxH+DoaTRZmnbHzrCfVVs8Jk0a3hdDdOaYWbMiUokmT9n58coKKt6I1C7ApD?=
 =?us-ascii?Q?uz+aiCk+uIZCoCvPv2NBokTrP3CMhy984kWyXlqcofkho5uPszXXYjWsQqCV?=
 =?us-ascii?Q?Q2etnTW9cNE+7qZWPCQWedR/cO/XcMSOEqU8kdew7wUBxONktZvrOmUxQiMJ?=
 =?us-ascii?Q?tyVKRPmnH/BMI/TSWzCLEnIYYjsQV9XI5ltOboWgdSrBCJCTy9AWwycTJUWA?=
 =?us-ascii?Q?C8+8QkhwXooqVKzlEhkzIDIL+A6eKVHWvEAZaUSlHF7F8rFIBwojYs58eKfu?=
 =?us-ascii?Q?iQoCvcxLjHwTk5BeGM/BJeWBav++/+oEJ0fygo+Nc57J5r3ojh65JXmcbevb?=
 =?us-ascii?Q?cxEFehyqhLPi506BEm7zHf+K1Wg9+vOvGjEtLaE/NPyVMHSYirGH/6jnebHA?=
 =?us-ascii?Q?MctMyz8PPG1Z1dBBdpYDgakxGL4C//PiTH+wfQHQo6Xxrh/cDHrMsN5K80ba?=
 =?us-ascii?Q?gJtnX9uwsBRlDu2G+Lb9GbkEZGNHRjnrbghAi6PaG26A8iV/qu2fkn5+867w?=
 =?us-ascii?Q?RmUub0FovhjcDRD9IaSUP/WmkA/9axd0mna5Ud6i6Nq7YDac2qDmjMpXTQyj?=
 =?us-ascii?Q?INQocwn3dOPeWfkV8xOuT+oEnwCD+K7j4F9MSkvUvkysjRT8BKait+QaXnSd?=
 =?us-ascii?Q?dLP8LA5gepM0Ch4VgQN987zdBmOFqieuiwYFTcG/PiIigvw1ULdOskEN0Tih?=
 =?us-ascii?Q?Qf6h/4fcQ6Cjld+2usyRNvYDgZDhttefrzbEBR/7GkUyh/de9Tmv8a+MNoS2?=
 =?us-ascii?Q?zvqpDSGtWqPNUOabs7HcrCgEk4W1PAjpX+hsP8WYUmwSmDhc9uDZjsog33H3?=
 =?us-ascii?Q?yKZ8VK4S/yE0kNyHg8DyfKl4JYRYH1reeO//H69XBtdKXORyJFnSIuXmd8G6?=
 =?us-ascii?Q?3vEDVveuzeIq+T7dXOgNQ54GuZogoAnS9lg1Kp5gW1w1aWvEKQvOOMQKAnwL?=
 =?us-ascii?Q?ljfG1cuRvAN48Qw/pxL5CQUjf6fEweoJJP3D2xMM7mWtevwCTLPO/RPGs3t2?=
 =?us-ascii?Q?6Vnxoz9Wbx6okF0BvEmgbzQV4peEFqqpM73y6JqqDHDVdBKm+ZkJWrJvsSZ8?=
 =?us-ascii?Q?/U69yImPCNU3lp/D6lAR9SlwoakD2Kn68SiuslgwHqgah5ZprSDTkm5LGfsT?=
 =?us-ascii?Q?B6BgoJTcvAv0mxqF8IoS5rkLonI21kykEa/YNF7MQKpUxHRzWaOs67hJFat7?=
 =?us-ascii?Q?0xBMECy3+ZWreVO0BY/+eW2rNYb3w45+RnnL3x+KN545zEFdcrDDRnl1VuPN?=
 =?us-ascii?Q?9vRifeiSya79F8UsITmQ6HjZdXLMald+1o+RsI4WkXcrA4SL3HgwfMmijH7l?=
 =?us-ascii?Q?vl8ZEkSuaDYsjA33V7wrbgrXPSXM8SCTyaLBVre5bCYL05Zdr9L3JXqgph5T?=
 =?us-ascii?Q?DNPH+6qC6c9ngfPnkU5WeSHy2e5YKcTP08eHH04IgeGieNHkit+rX8QGeLrS?=
 =?us-ascii?Q?+71vGytnEptJ7wrL1LAD1KhUSXmbQIkc5C3NSFuc0vLRELbfd7hhivOf1W8q?=
 =?us-ascii?Q?gw=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 52cc1fc0-8de2-42f4-42a8-08dabc4a7ebd
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4166.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Nov 2022 20:48:56.8013
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +fVK6YKh1oRuoorRXBFGsocRNg3df5sI65uHEmUzTdOjlrPEOwQbZWxW+8iDoj6AIsKASHbXWVArZBy4pTyCB+nPYM3iFjmNcZG9WYh+iVA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB6687
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-11-01_10,2022-11-01_02,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 phishscore=0
 mlxlogscore=999 suspectscore=0 spamscore=0 malwarescore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000
 definitions=main-2211010146
X-Proofpoint-ORIG-GUID: _8hO3evGhIct6qeznL_K_FI6WfdFIJO2
X-Proofpoint-GUID: _8hO3evGhIct6qeznL_K_FI6WfdFIJO2
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Jan,

Jan Kara <jack@suse.cz> writes:
> Hi Stephen!
>
> On Thu 27-10-22 17:10:13, Stephen Brennan wrote:
>> Here is v3 of the patch series. I've taken all of the feedback,
>> thanks Amir, Christian, Hilf, et al. Differences are noted in each
>> patch.
>> 
>> I caught an obvious and silly dentry reference leak: d_find_any_alias()
>> returns a reference, which I never called dput() on. With that change, I
>> no longer see the rpc_pipefs issue, but I do think I need more testing
>> and thinking through the third patch. Al, I'd love your feedback on that
>> one especially.
>> 
>> Thanks,
>> Stephen
>> 
>> Stephen Brennan (3):
>>   fsnotify: Use d_find_any_alias to get dentry associated with inode
>>   fsnotify: Protect i_fsnotify_mask and child flags with inode rwsem
>>   fsnotify: allow sleepable child flag update
>
> Thanks for the patches Stephen and I'm sorry for replying somewhat late.

Absolutely no worries, these things take time. Thanks for taking a look!

> The first patch is a nobrainer. The other two patches ... complicate things
> somewhat more complicated than I'd like. I guess I can live with them if we
> don't find a better solution but I'd like to discuss a bit more about
> alternatives.

Understood!

> So what would happen if we just clear DCACHE_FSNOTIFY_PARENT_WATCHED in
> __fsnotify_parent() for the dentry which triggered the event and does not
> have watched parent anymore and never bother with full children walk? I
> suppose your contention problems will be gone, we'll just pay the price of
> dget_parent() + fsnotify_inode_watches_children() for each child that
> falsely triggers instead of for only one. Maybe that's not too bad? After
> all any event upto this moment triggered this overhead as well...

This is an interesting idea. It came across my mind but I don't think I
considered it seriously because I assumed that it was too big a change.
But I suppose in the process I created an even bigger change :P

The false positive dget_parent() + fsnotify_inode_watches_children()
shouldn't be too bad. I could see a situation where there's a lot of
random accesses within a directory, where the dget_parent() could cause
some contention over the parent dentry. But to be fair, the performance
would have been the same or worse while fsnotify was active in that
case, and the contention would go away as most of the dentries get their
flags cleared. So I don't think this is a problem.

> Am I missing something?

I think there's one thing missed here. I understand you'd like to get
rid of the extra flag in the connector. But the advantage of the flag is
avoiding duplicate work by saving a bit of state. Suppose that a mark is
added to a connector, which causes fsnotify_inode_watches_children() to
become true. Then, any subsequent call to fsnotify_recalc_mask() must
call __fsnotify_update_child_dentry_flags(), even though the child
dentry flags don't need to be updated: they're already set. For (very)
large directories, this can take a few seconds, which means that we're
doing a few extra seconds of work each time a new mark is added to or
removed from a connector in that case. I can't imagine that's a super
common workload though, and I don't know if my customers do that (my
guess would be no).

For an example of a test workload where this would make a difference:
one of my test cases is to create a directory with millions of negative
dentries, and then run "for i in {1..20}; do inotifywait $DIR & done".
With the series as-is, only the first task needs to do the child flag
update. With your proposed alternative, each task would re-do the
update.

If we were to manage to get __fsnotify_update_child_dentry_flags() to
work correctly, and safely, with the ability to drop the d_lock and
sleep, then I think that wouldn't be too much of a problem, because then
other spinlock users of the directory will get a chance to grab it, and
so we don't risk softlockups. Without the sleepable iterations, it would
be marginally worse than the current solution, but I can't really
comment _how_ much worse because like I said, it doesn't sound like a
frequent usage pattern.

I think I have a _slight_ preference for the current design, but I see
the appeal of simpler code, and your design would still improve things a
lot! Maybe Amir has an opinion too, but of course I'll defer to what you
want.

Thanks,
Stephen