Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 13 Apr 2022 13:20:53 +0200
From:   Greg KH <gregkh@linuxfoundation.org>
To:     Yao Hongbo <yaohongbo@linux.alibaba.com>
Cc:     "Michael S. Tsirkin" <mst@redhat.com>,
        alikernel-developer@linux.alibaba.com, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] uio/uio_pci_generic: Introduce refcnt on open/release
Message-ID: <YlayFROf5P294P/P@kroah.com>
References: <1649833302-27299-1-git-send-email-yaohongbo@linux.alibaba.com>
 <YlZ8vZ9RX5i7mWNk@kroah.com>
 <20220413044246-mutt-send-email-mst@kernel.org>
 <ebd1b238-6e48-6561-93ab-f562096b1c05@linux.alibaba.com>
 <YlabT7+Hqc3h62AT@kroah.com>
 <5a80c065-e811-018e-6c35-01c12b194c94@linux.alibaba.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <5a80c065-e811-018e-6c35-01c12b194c94@linux.alibaba.com>
Precedence: bulk

On Wed, Apr 13, 2022 at 07:09:57PM +0800, Yao Hongbo wrote:
> 
> 在 2022/4/13 下午5:43, Greg KH 写道:
> > On Wed, Apr 13, 2022 at 05:25:40PM +0800, Yao Hongbo wrote:
> > > 在 2022/4/13 下午4:51, Michael S. Tsirkin 写道:
> > > > On Wed, Apr 13, 2022 at 09:33:17AM +0200, Greg KH wrote:
> > > > > On Wed, Apr 13, 2022 at 03:01:42PM +0800, Yao Hongbo wrote:
> > > > > > If two userspace programs both open the PCI UIO fd, when one
> > > > > > of the program exits uncleanly, the other will cause IO hang
> > > > > > due to bus-mastering disabled.
> > > > > > 
> > > > > > It's a common usage for spdk/dpdk to use UIO. So, introduce refcnt
> > > > > > to avoid such problems.
> > > > > Why do you have multiple userspace programs opening the same device?
> > > > > Shouldn't they coordinate?
> > > > Or to restate, I think the question is, why not open the device
> > > > once and pass the FD around?
> > > Hmm, it will have the same result, no matter  whether opening the same
> > > device or pass the FD around.
> > How?  You only open once, and close once.  Where is the multiple closes?
> > 
> > > Our expectation is that even if the primary process exits abnormally,  the
> > > second process can still send
> > > 
> > > or receive data.
> > Then use the same file descriptor.
> 
> 
> Yes, we can use the same file descriptor.
> 
> but since the pcie bus-master  has been disabled by the primary process,
> 
> the seconday process cannot continue to operate.

Really?  With the same file descriptor?  Try it and see.  release should
only be called when the file descriptor is closed.

> > > The impact of disabling pci bus-master is relatively large, and we should
> > > make some restrictions on
> > > this behavior.
> > Why?  UIO is "you better really really know what you are doing to use
> > this interface", right?  Just duplicate the fd and pass it around if you
> > must have multiple accesses to the same device.
> > 
> > And again, this will be a functional change.  How can you handle your
> > userspace on older kernels if you make this change?
> 
> Without this change, our userspace cannot work properly on older kernels.

What change broke your userspace?

> Our userspace only use the "multi process mode" feature of the spdk.
> 
> The SPDK links:
> https://spdk.io/doc/app_overview.html
> 
> "Multi process mode
> When --shm-id is specified, the application is started in multi-process
> mode.
> 
> Applications using the same shm-id share their memory and NVMe devices.
> 
> The first app to start with a given id becomes a primary process, with the
> rest,
> 
> called secondary processes, only attaching to it. When the primary process
> exits,
> 
> the secondary ones continue to operate, but no new processes can be attached
> 
> at this point. All processes within the same shm-id group must use the same
> --single-file-segments setting."

Please work with the spdk users, I know nothing about that mess, sorry.

greg k-h