I was trying to maintain a local and remote directory in sync with
lsync, using inotify.
I happen to have >4M files and >400k directories... running over
/proc/sys/fs/inotify/max_user_watches
What if inotify could just provide all events (limited for instance by
euid), which can be filtered by the application requesting
inotification?
Would something like that be possible?
BTW, I'm not a kernel expert, and not following this list... just
wanted to throw in this idea. Pls Cc: personally.
Thanks,
Jos
On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote:
> I was trying to maintain a local and remote directory in sync with
> lsync, using inotify.
> I happen to have >4M files and >400k directories... running over
> /proc/sys/fs/inotify/max_user_watches
Would fanotify perhaps be a better interface to use?
--
Len Sorensen
On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen
<[email protected]> wrote:
> On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote:
>> I was trying to maintain a local and remote directory in sync with
>> lsync, using inotify.
>> I happen to have >4M files and >400k directories... running over
>> /proc/sys/fs/inotify/max_user_watches
>
> Would fanotify perhaps be a better interface to use?
(One of us is misunderstanding fanotify; it might be me.)
Did you look at fanotify closely? I don't think it could be used for
this task -- does not notify linka dn unlink events, difficult to set
up recursive monitoring, etc.
Cheers,
Michael
On Fri, Apr 18, 2014 at 06:15:40AM +0200, Michael Kerrisk wrote:
> (One of us is misunderstanding fanotify; it might be me.)
No I think you are right, and that the misunderstanding is on my part.
I had not realized that fanotify did not provide create/delete events,
only access events.
> Did you look at fanotify closely? I don't think it could be used for
> this task -- does not notify linka dn unlink events, difficult to set
> up recursive monitoring, etc.
It would appear you are correct.
--
Len Sorensen
On Fri, Apr 18, 2014 at 6:15 AM, Michael Kerrisk <[email protected]> wrote:
> On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen
> <[email protected]> wrote:
>> On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote:
>>> I was trying to maintain a local and remote directory in sync with
>>> lsync, using inotify.
>>> I happen to have >4M files and >400k directories... running over
>>> /proc/sys/fs/inotify/max_user_watches
>>
>> Would fanotify perhaps be a better interface to use?
>
> (One of us is misunderstanding fanotify; it might be me.)
>
> Did you look at fanotify closely? I don't think it could be used for
> this task -- does not notify linka dn unlink events, difficult to set
> up recursive monitoring, etc.
Does recursive monitoring even work with inotify?
Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
mkdir() raced against the thread which installes the new watches.
--
Thanks,
//richard
On Mon, Apr 21, 2014 at 12:22 AM, Richard Weinberger
<[email protected]> wrote:
> On Fri, Apr 18, 2014 at 6:15 AM, Michael Kerrisk <[email protected]> wrote:
>> On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen
>> <[email protected]> wrote:
>>> On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote:
>>>> I was trying to maintain a local and remote directory in sync with
>>>> lsync, using inotify.
>>>> I happen to have >4M files and >400k directories... running over
>>>> /proc/sys/fs/inotify/max_user_watches
>>>
>>> Would fanotify perhaps be a better interface to use?
>>
>> (One of us is misunderstanding fanotify; it might be me.)
>>
>> Did you look at fanotify closely? I don't think it could be used for
>> this task -- does not notify linka dn unlink events, difficult to set
>> up recursive monitoring, etc.
>
> Does recursive monitoring even work with inotify?
> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
> mkdir() raced against the thread which installes the new watches.
As I understand it, you have to program to deal with the races (rescan
directories after adding watches). I recently did a lot of work
updating the inotify(7) man page to discuss all the issues that I know
of, and their remedies. If I missed anything, I'd appreciate a note on
it, so that it can be added. See
http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
Cheers,
Michael
--
Michael Kerrisk Linux man-pages maintainer;
http://www.kernel.org/doc/man-pages/
Author of "The Linux Programming Interface", http://blog.man7.org/
Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>> Does recursive monitoring even work with inotify?
>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>> mkdir() raced against the thread which installes the new watches.
>
> As I understand it, you have to program to deal with the races (rescan
> directories after adding watches). I recently did a lot of work
> updating the inotify(7) man page to discuss all the issues that I know
> of, and their remedies. If I missed anything, I'd appreciate a note on
> it, so that it can be added. See
> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
I'm aware of the rescan hack, but in my case it does not help
because my program must not miss any event.
Currently I'm using a fuse overlay filesystem to log everything.
Not perfect but works... :-)
Thanks,
//richard
On 04/21/2014 10:42 AM, Richard Weinberger wrote:
> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>> Does recursive monitoring even work with inotify?
>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>> mkdir() raced against the thread which installes the new watches.
>>
>> As I understand it, you have to program to deal with the races (rescan
>> directories after adding watches). I recently did a lot of work
>> updating the inotify(7) man page to discuss all the issues that I know
>> of, and their remedies. If I missed anything, I'd appreciate a note on
>> it, so that it can be added. See
>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
>
> I'm aware of the rescan hack, but in my case it does not help
> because my program must not miss any event.
Then, we're understand the same thing: you're out of luck :-}.
> Currently I'm using a fuse overlay filesystem to log everything.
> Not perfect but works... :-)
Interesting notion. I need to get to grips with FUSE...
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
Am 21.04.2014 15:31, schrieb Michael Kerrisk (man-pages):
> On 04/21/2014 10:42 AM, Richard Weinberger wrote:
>> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>>> Does recursive monitoring even work with inotify?
>>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>>> mkdir() raced against the thread which installes the new watches.
>>>
>>> As I understand it, you have to program to deal with the races (rescan
>>> directories after adding watches). I recently did a lot of work
>>> updating the inotify(7) man page to discuss all the issues that I know
>>> of, and their remedies. If I missed anything, I'd appreciate a note on
>>> it, so that it can be added. See
>>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
>>
>> I'm aware of the rescan hack, but in my case it does not help
>> because my program must not miss any event.
>
> Then, we're understand the same thing: you're out of luck :-}.
>
>> Currently I'm using a fuse overlay filesystem to log everything.
>> Not perfect but works... :-)
>
> Interesting notion. I need to get to grips with FUSE...
e.g. http://loggedfs.sourceforge.net/
Thanks,
//richard
Seems like also 'mv' events are also not supported by fanotify, like
creation/deletion.
Therefore not an option (for lsyncd) probably. Maybe loggedfs is (if
time permits...).
Thanks for the discussion.
Jos
On Mon, Apr 21, 2014 at 5:01 PM, Richard Weinberger <[email protected]> wrote:
>
>
> Am 21.04.2014 15:31, schrieb Michael Kerrisk (man-pages):
>> On 04/21/2014 10:42 AM, Richard Weinberger wrote:
>>> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>>>> Does recursive monitoring even work with inotify?
>>>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>>>> mkdir() raced against the thread which installes the new watches.
>>>>
>>>> As I understand it, you have to program to deal with the races (rescan
>>>> directories after adding watches). I recently did a lot of work
>>>> updating the inotify(7) man page to discuss all the issues that I know
>>>> of, and their remedies. If I missed anything, I'd appreciate a note on
>>>> it, so that it can be added. See
>>>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
>>>
>>> I'm aware of the rescan hack, but in my case it does not help
>>> because my program must not miss any event.
>>
>> Then, we're understand the same thing: you're out of luck :-}.
>>
>>> Currently I'm using a fuse overlay filesystem to log everything.
>>> Not perfect but works... :-)
>>
>> Interesting notion. I need to get to grips with FUSE...
>
> e.g. http://loggedfs.sourceforge.net/
>
> Thanks,
> //richard