Received: by 2002:a6b:fb09:0:0:0:0:0 with SMTP id h9csp647514iog; Fri, 24 Jun 2022 10:55:17 -0700 (PDT) X-Google-Smtp-Source: AGRyM1sTIkHXsngckNMCUKY3dYOIxmmJ4YQwB0GqMlbyKStge9kFE+amKN/gdd5JeF7DHm+a+MYl X-Received: by 2002:a05:6402:360d:b0:435:710a:2531 with SMTP id el13-20020a056402360d00b00435710a2531mr297812edb.377.1656093317701; Fri, 24 Jun 2022 10:55:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1656093317; cv=none; d=google.com; s=arc-20160816; b=GDSJ6bzavQJMznJpBTdTaeBGE3uBLtkz8jALBOoKWv9TTqvX7VdHFlBqneZMDxQXT7 YDasizPqDtJQUTyKJQH6Y2OgWi4KcwQY6eXDLyaHDr1WH4+hf5Vs2gLcRXrjfb2/EFG8 s40NPExbFkiDmF9yozXg4cV7Tvyi//lmQow2ig5J7wP+ycF28TNuqQ3q3T+47EbIkga1 cQ373/7pi25wSXr1iYanW7iNN71bCHI7L0CAF5/oKPDeoA4wb5Dy3dNgu8uU3OPq7Zjv 6chxr3hrvGVUyoeCsygZsoqD6nahtil+YqSwrk6K/oeOs4nSz+3GlLltZaglvA9KewZV no2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=TqUnppsLw+ptEFOWt96xqRoxkSJ1xKv0wtk/TcEKVps=; b=Twos4rQDa5ALGxKcefEp8/BLOvAvEpeZqMSjOXCsnPSxW/qzlue/szrfkqX0P+lE37 Kg9wGdUMPju4HJ8RBEH0Z88U+IPSXa7Uig6FwpkE21ktXShz5YkEtSEq8TyDxy9Iy+q5 32/nprQkR4W9Ygcnk+iAqgw4csNSthi2pxE4h861rAdduxBos27bl5yLNPYglEsmdT+N aBGM3Pry3EhxzJx1h4+oZLS8Fqo9ffZC2TmgA3E/S+6Nqf1aNipo5Q5RYYovSFv8AoFw LyRy4tK9wmPrmuYGFzfYQ6id7qem+huTBA1j6qjK4OMJ5ya4XpCrTv23XZKhWVoj5/Fd dYdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BFVrBJQ8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t19-20020a50ab53000000b004359a63823esi3538897edc.225.2022.06.24.10.54.51; Fri, 24 Jun 2022 10:55:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BFVrBJQ8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231249AbiFXRhM (ORCPT + 99 others); Fri, 24 Jun 2022 13:37:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231162AbiFXRhJ (ORCPT ); Fri, 24 Jun 2022 13:37:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 64AB262C13 for ; Fri, 24 Jun 2022 10:37:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1656092226; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=TqUnppsLw+ptEFOWt96xqRoxkSJ1xKv0wtk/TcEKVps=; b=BFVrBJQ8bT/BPBVGa4c75qhAS4AJy4MEv76CafLHiIWRngrNDiatuvtwp/WTrPDg+91+Lj ZcTB0EspBYkmdIzRQB5FtUt00NhNDrZ6Z57EqemqGtHHaJPxBlg3Fyz/2Kti7aYdIwXMF4 e/Cw8aoo1MZEhdUZqNZ8EjusYGuFJEs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-352-NlKausoxPTGBlsCfhDCECA-1; Fri, 24 Jun 2022 13:37:01 -0400 X-MC-Unique: NlKausoxPTGBlsCfhDCECA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B88F729AB442; Fri, 24 Jun 2022 17:37:00 +0000 (UTC) Received: from horse.redhat.com (unknown [10.22.9.80]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0226A492C3B; Fri, 24 Jun 2022 17:36:59 +0000 (UTC) Received: by horse.redhat.com (Postfix, from userid 10451) id B7C8E2209F9; Fri, 24 Jun 2022 13:36:59 -0400 (EDT) Date: Fri, 24 Jun 2022 13:36:59 -0400 From: Vivek Goyal To: Tycho Andersen Cc: Eric Biederman , Christian Brauner , Miklos Szeredi , fuse-devel@lists.sourceforge.net, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: strange interaction between fuse + pidns Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jun 23, 2022 at 05:41:17PM -0600, Tycho Andersen wrote: > On Thu, Jun 23, 2022 at 05:55:20PM -0400, Vivek Goyal wrote: > > So in this case single process is client as well as server. IOW, one > > thread is fuse server servicing fuse requests and other thread is fuse > > client accessing fuse filesystem? > > Yes. Probably an abuse of the API and something people Should Not Do, > but as you say the kernel still shouldn't lock up like this. > > > > since the thread has a copy of > > > the fd table with an fd pointing to the same fuse device, the reference > > > count isn't decremented to zero in fuse_dev_release(), and the task hangs > > > forever. > > > > So why did fuse server thread stop responding to fuse messages. Why > > did it not complete flush. > > In this particular case I think it's because the application crashed > for unrelated reasons and tried to exit the pidns, hitting this > problem. > > > BTW, unkillable wait happens on ly fc->no_interrupt = 1. And this seems > > to be set only if server probably some previous interrupt request > > returned -ENOSYS. > > > > fuse_dev_do_write() { > > else if (oh.error == -ENOSYS) > > fc->no_interrupt = 1; > > } > > > > So a simple workaround might be for server to implement support for > > interrupting requests. > > Yes, but that is the libfuse default IIUC. Looking at libfuse code. I understand low level API interface and for that looks like generic code itself will take care of this (without needing support from filesystem). libfuse/lib/fuse_lowlevel.c do_interrupt(). > > > Having said that, this does sounds like a problem and probably should > > be fixed at kernel level. > > > > > > > > diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c > > > index 0e537e580dc1..c604dfcaec26 100644 > > > --- a/fs/fuse/dev.c > > > +++ b/fs/fuse/dev.c > > > @@ -297,7 +297,6 @@ void fuse_request_end(struct fuse_req *req) > > > spin_unlock(&fiq->lock); > > > } > > > WARN_ON(test_bit(FR_PENDING, &req->flags)); > > > - WARN_ON(test_bit(FR_SENT, &req->flags)); > > > if (test_bit(FR_BACKGROUND, &req->flags)) { > > > spin_lock(&fc->bg_lock); > > > clear_bit(FR_BACKGROUND, &req->flags); > > > @@ -381,30 +380,33 @@ static void request_wait_answer(struct fuse_req *req) > > > queue_interrupt(req); > > > } > > > > > > - if (!test_bit(FR_FORCE, &req->flags)) { > > > - /* Only fatal signals may interrupt this */ > > > - err = wait_event_killable(req->waitq, > > > - test_bit(FR_FINISHED, &req->flags)); > > > - if (!err) > > > - return; > > > + /* Only fatal signals may interrupt this */ > > > + err = wait_event_killable(req->waitq, > > > + test_bit(FR_FINISHED, &req->flags)); > > > > Trying to do a fatal signal killable wait sounds reasonable. But I am > > not sure about the history. > > > > - Why FORCE requests can't do killable wait. > > - Why flush needs to have FORCE flag set. > > args->force implies a few other things besides this killable wait in > fuse_simple_request(), most notably: > > req = fuse_request_alloc(fm, GFP_KERNEL | __GFP_NOFAIL); > > and > > __set_bit(FR_WAITING, &req->flags); FR_WAITING stuff is common between both type of requests. We set it in fuse_get_req() as well which is called for non-force requests. So there seem to be only two key difference. - We allocate request with flag __GFP_NOFAIL for force. So don't want memory allocation to fail. - And this special casing of non-killable wait. Miklos probably will have more thoughts on this. Thanks Vivek > > seems like it probably can be invoked from some non-user/atomic > context somehow? > > > > + if (!err) > > > + return; > > > > > > - spin_lock(&fiq->lock); > > > - /* Request is not yet in userspace, bail out */ > > > - if (test_bit(FR_PENDING, &req->flags)) { > > > - list_del(&req->list); > > > - spin_unlock(&fiq->lock); > > > - __fuse_put_request(req); > > > - req->out.h.error = -EINTR; > > > - return; > > > - } > > > + spin_lock(&fiq->lock); > > > + /* Request is not yet in userspace, bail out */ > > > + if (test_bit(FR_PENDING, &req->flags)) { > > > + list_del(&req->list); > > > spin_unlock(&fiq->lock); > > > + __fuse_put_request(req); > > > + req->out.h.error = -EINTR; > > > + return; > > > } > > > + spin_unlock(&fiq->lock); > > > > > > /* > > > - * Either request is already in userspace, or it was forced. > > > - * Wait it out. > > > + * Womp womp. We sent a request to userspace and now we're getting > > > + * killed. > > > */ > > > - wait_event(req->waitq, test_bit(FR_FINISHED, &req->flags)); > > > + set_bit(FR_INTERRUPTED, &req->flags); > > > + /* matches barrier in fuse_dev_do_read() */ > > > + smp_mb__after_atomic(); > > > + /* request *must* be FR_SENT here, because we ignored FR_PENDING before */ > > > + WARN_ON(!test_bit(FR_SENT, &req->flags)); > > > + queue_interrupt(req); > > > } > > > > > > static void __fuse_request_send(struct fuse_req *req) > > > > > > avaialble as a full patch here: > > > https://github.com/tych0/linux/commit/81b9ff4c8c1af24f6544945da808dbf69a1293f7 > > > > > > but now things are even weirder. Tasks are stuck at the killable wait, but with > > > a SIGKILL pending for the thread group. > > > > That's strange. No idea what's going on. > > Thanks for taking a look. This is where it falls apart for me. In > principle the patch seems simple, but this sleeping behavior is beyond > my understanding. > > Tycho >