Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751710AbdCNWaE (ORCPT ); Tue, 14 Mar 2017 18:30:04 -0400 Received: from mail-ot0-f171.google.com ([74.125.82.171]:33968 "EHLO mail-ot0-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750959AbdCNWaC (ORCPT ); Tue, 14 Mar 2017 18:30:02 -0400 MIME-Version: 1.0 In-Reply-To: <20170314144846.hlrcuwroeopnbna5@rgvaio> References: <20170314124152.w53wga4abqpfml7v@rgvaio> <20170314144846.hlrcuwroeopnbna5@rgvaio> From: Amir Goldstein Date: Wed, 15 Mar 2017 00:30:00 +0200 Message-ID: Subject: Re: [RFC 1/2] fanotify: new event FAN_MODIFY_DIR To: =?UTF-8?B?RmlsaXAgxaB0xJtkcm9uc2vDvQ==?= Cc: linux-fsdevel , linux-kernel , Jan Kara , Alexander Viro Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by mail.home.local id v2EMU7Qu000608 Content-Length: 2904 Lines: 67 On Tue, Mar 14, 2017 at 4:48 PM, Filip Štědronský wrote: > Hi, > > On Tue, Mar 14, 2017 at 03:55:19PM +0200, Amir Goldstein wrote: >> Please let me know if that is sufficient for your needs >> or if you need me to prepare a version that delivers filename events >> without filename info, therefore allowing to merge filename events. >> >> Sounds to me like if you get an event FAN_DELETE, "aaa", >> your implementation can update the db directly without having >> to scan the directory, so it should be useful. >> For your consideration. > > both approaches might be useful and there are tradeoffs. Direct updates > save us from rescanning but give more events and more context switches. > > On the other hand, with an unlimited queue and well-mergable events I > could e.g. sleep for five seconds each time after emptying the queue, > thus giving the events more potential to merge and reducing context > switches. > > But one risks blowing up the queue. However, I have some ideas. > Basically we could implement some kind of "bulk read" mode for fanotify > that would pass events to userspace only when > (a) a given event count thresold is hit (e.g. half the queue limit > if queue is limited), or > (b) the oldest event is older than a specified maximum latency. That > might vary from 1 second to 5 minutes depending on specific > application needs (e.g. background backup does not need to be > low-latency). > > This would greatly reduce extra context switches when watching > a large portion (or whole) of the file system. All of this presumably > could be implemented on top of your suggested patches. > > Either way, I suggest that implementing the nameless filename events > is a good idea. I do not know whether they will be the best choice > for my application but they probably will be useful for some > applications. > Done and pushed all branches. Added 2 new patches to the fanotify_dentry set: ce0c223 fanotify: allow merge of filename events cd872d1 fanotify: make filename info optional for filename events You should test with branch fanotify_sb, using super block watch, because filename events are not supported in mount scope. You can follow user API used by my fanotify_demo.c program. Tested with my fanotify_demo test both without FAN_EVENT_INFO_NAME and without FAN_EVENT_INFO_FH. Without FAN_EVENT_INFO_NAME more filename events are merged, but in my test that runs fs ops by a script, most events do not get merged because of different pid value. I guess that for the filesync/indexer use case, we would need to add a new FAN_EVENT_INFO_NOPID flag to loose the irrelevant pid information in order to be able to merge a lot more filename event on a directory. I also plan to add FAN_EVENT_INFO_NOFD, to get rid of redundant fd (when INFO_FH is available) and then be able to drop the refcount on the victim object. Cheers, Amir.