Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752605AbaGLTHJ (ORCPT ); Sat, 12 Jul 2014 15:07:09 -0400 Received: from mail-qg0-f52.google.com ([209.85.192.52]:44469 "EHLO mail-qg0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751438AbaGLTHG convert rfc822-to-8bit (ORCPT ); Sat, 12 Jul 2014 15:07:06 -0400 MIME-Version: 1.0 Reply-To: mtk.manpages@gmail.com In-Reply-To: <20140404124338.GA26806@quack.suse.cz> References: <20140403205236.GE14107@quack.suse.cz> <533E60D6.2000704@gmail.com> <20140404124338.GA26806@quack.suse.cz> From: "Michael Kerrisk (man-pages)" Date: Sat, 12 Jul 2014 21:06:45 +0200 Message-ID: Subject: Re: Things I wish I'd known about Inotify To: Jan Kara Cc: John McCutchan , Robert Love , Eric Paris , Lennart Poettering , Radu Voicilas , daniel@veillard.com, Christoph Hellwig , Vegard Nossum , "linux-fsdevel@vger.kernel.org" , linux-man , gamin-list@gnome.org, lkml , inotify-tools-general@lists.sourceforge.net, jake@lwn.net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Late follow up on this thread..., since another question occurred in discussions with Jake. On Fri, Apr 4, 2014 at 2:43 PM, Jan Kara wrote: > On Fri 04-04-14 09:35:50, Michael Kerrisk (man-pages) wrote: >> On 04/03/2014 10:52 PM, Jan Kara wrote: >> > On Thu 03-04-14 08:34:44, Michael Kerrisk (man-pages) wrote: [...] >> >> Dealing with rename() events >> >> The IN_MOVED_FROM and IN_MOVED_TO events that are generated by >> >> rename(2) are usually available as consecutive events when read‐ >> >> ing from the inotify file descriptor. However, this is not guar‐ >> >> anteed. If multiple processes are triggering events for moni‐ >> >> tored objects, then (on rare occasions) an arbitrary number of >> >> other events may appear between the IN_MOVED_FROM and IN_MOVED_TO >> >> events. >> >> >> >> Matching up the IN_MOVED_FROM and IN_MOVED_TO event pair gener‐ >> >> ated by rename(2) is thus inherently racy. (Don't forget that if >> >> an object is renamed outside of a monitored directory, there may >> >> not even be an IN_MOVED_TO event.) Heuristic approaches (e.g., >> >> assume the events are always consecutive) can be used to ensure a >> >> match in most cases, but will inevitably miss some cases, causing >> >> the application to perceive the IN_MOVED_FROM and IN_MOVED_TO >> >> events as being unrelated. If watch descriptors are destroyed >> >> and re-created as a result, then those watch descriptors will be >> >> inconsistent with the watch descriptors in any pending events. >> >> (Re-creating the inotify file descriptor and rebuilding the cache >> >> may be useful to deal with this scenario.) >> > Well, but there's 'cookie' value meant exactly for matching up >> > IN_MOVED_FROM and IN_MOVED_TO events. And 'cookie' is guaranteed to be >> > unique at least within the inotify instance (in fact currently it is unique >> > within the whole system but I don't think we want to give that promise). >> >> Yes, that's already assumed by my discussion above (its described elsewhere >> in the page). But your comment makes me think I should add a few words to >> remind the reader of that fact. I'll do that. > Yes, that would be good. > >> But, the point is that even with the cookie, matching the events is >> nontrivial, since: >> >> * There may not even be an IN_MOVED_FROM event >> * There may be an arbitrary number of other events in between the >> IN_MOVED_FROM and the IN_MOVED_TO. >> >> Therefore, one has to use heuristic approaches such as "allow at least >> N millisconds" or "check the next N events" to see if there is an >> IN_MOVED_FROM that matches the IN_MOVED_TO. I can't see any way around >> that being inherently racy. (It's unfortunate that the kernel can't >> provide a guarantee that the two events are always consecutive, since >> that would simply user space's life considerably.) > Yeah, it's unpleasant but doing that would be quite costly/complex at the > kernel side. And the race would in the worst case lead to application > thinking there's been file moved outside of watched area & a file moved > somewhere else inside the watched area. So the application will have to > possibly inspect that file. That doesn't seem too bad. One further question. The IN_MOVED_FROM+IN_MOVED_TO pair may not be guaranteed to be contiguous in the read buffer, but is their insertion in the event queue guaranteed to be atomic from a user-space point of view? That is to say: having read an IN_MOVED_FROM event, does user space have the guarantee that if there is an IN_MOVED_TO event, then it will already be in the queue? The reason I ask is that this would affect how user space might try to read the IN_MOVED_TO event. If there is no such guarantee, then a read() (or select()/poll()) with (small) timeout is needed. If such a guarantee is provided, then a nonblocking read() would suffice. Cheers, Michael PS I just now found this code by John McCutchan https://git.gnome.org/browse/gnome-vfs/tree/modules/inotify-kernel.c#n570 which suggests that the insertion of the event pair is not atomic w.r.t. user space. Still, I wonder if there is any definitive statement about this. -- Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/