MIME-Version: 1.0
References: <0d9f1b89-9374-747b-3fb0-b4b28ad0ace1@redhat.com>
 <CACGkMEv=+D+Es4sfde_X7F0zspVdy4Rs1Wi9qfCudsznsUrOTQ@mail.gmail.com>
 <20221229020553-mutt-send-email-mst@kernel.org> <CACGkMEs5s3Muo+4OfjaLK_P76rTdPhjQdTwykRNGOecAWnt+8g@mail.gmail.com>
 <20221229030633-mutt-send-email-mst@kernel.org> <CACGkMEukqZX=6yz1yCj+psHp5c+ZGVVuEYTUssfRCTQZgVWS6g@mail.gmail.com>
 <20230127053112-mutt-send-email-mst@kernel.org> <CACGkMEsZs=6TaeSUnu_9Rf+38uisi6ViHyM50=2+ut3Wze2S1g@mail.gmail.com>
 <20230129022809-mutt-send-email-mst@kernel.org> <CACGkMEuya+_2P8d4hokoyL_LKGdVzyCC1nDwOCdZb0=+2rjKPQ@mail.gmail.com>
 <20230130003334-mutt-send-email-mst@kernel.org>
In-Reply-To: <20230130003334-mutt-send-email-mst@kernel.org>
From:   Jason Wang <jasowang@redhat.com>
Date:   Mon, 30 Jan 2023 15:44:24 +0800
Message-ID: <CACGkMEu0v-kbh2vKvcDRoMsRoXwidPnQhiFetYPY-tXOAVScsg@mail.gmail.com>
Subject: Re: [PATCH 3/4] virtio_ring: introduce a per virtqueue waitqueue
To:     "Michael S. Tsirkin" <mst@redhat.com>
Cc:     davem@davemloft.net, edumazet@google.com, kuba@kernel.org,
        pabeni@redhat.com, virtualization@lists.linux-foundation.org,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        maxime.coquelin@redhat.com, alvaro.karsz@solid-run.com,
        eperezma@redhat.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, Jan 30, 2023 at 1:43 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jan 30, 2023 at 10:53:54AM +0800, Jason Wang wrote:
> > On Sun, Jan 29, 2023 at 3:30 PM Michael S. Tsirkin <mst@redhat.com> wro=
te:
> > >
> > > On Sun, Jan 29, 2023 at 01:48:49PM +0800, Jason Wang wrote:
> > > > On Fri, Jan 27, 2023 at 6:35 PM Michael S. Tsirkin <mst@redhat.com>=
 wrote:
> > > > >
> > > > > On Fri, Dec 30, 2022 at 11:43:08AM +0800, Jason Wang wrote:
> > > > > > On Thu, Dec 29, 2022 at 4:10 PM Michael S. Tsirkin <mst@redhat.=
com> wrote:
> > > > > > >
> > > > > > > On Thu, Dec 29, 2022 at 04:04:13PM +0800, Jason Wang wrote:
> > > > > > > > On Thu, Dec 29, 2022 at 3:07 PM Michael S. Tsirkin <mst@red=
hat.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Dec 28, 2022 at 07:53:08PM +0800, Jason Wang wrot=
e:
> > > > > > > > > > On Wed, Dec 28, 2022 at 2:34 PM Jason Wang <jasowang@re=
dhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > =E5=9C=A8 2022/12/27 17:38, Michael S. Tsirkin =E5=86=
=99=E9=81=93:
> > > > > > > > > > > > On Tue, Dec 27, 2022 at 05:12:58PM +0800, Jason Wan=
g wrote:
> > > > > > > > > > > >> =E5=9C=A8 2022/12/27 15:33, Michael S. Tsirkin =E5=
=86=99=E9=81=93:
> > > > > > > > > > > >>> On Tue, Dec 27, 2022 at 12:30:35PM +0800, Jason W=
ang wrote:
> > > > > > > > > > > >>>>> But device is still going and will later use th=
e buffers.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Same for timeout really.
> > > > > > > > > > > >>>> Avoiding infinite wait/poll is one of the goals,=
 another is to sleep.
> > > > > > > > > > > >>>> If we think the timeout is hard, we can start fr=
om the wait.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Thanks
> > > > > > > > > > > >>> If the goal is to avoid disrupting traffic while =
CVQ is in use,
> > > > > > > > > > > >>> that sounds more reasonable. E.g. someone is turn=
ing on promisc,
> > > > > > > > > > > >>> a spike in CPU usage might be unwelcome.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Yes, this would be more obvious is UP is used.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>> things we should be careful to address then:
> > > > > > > > > > > >>> 1- debugging. Currently it's easy to see a warnin=
g if CPU is stuck
> > > > > > > > > > > >>>      in a loop for a while, and we also get a bac=
ktrace.
> > > > > > > > > > > >>>      E.g. with this - how do we know who has the =
RTNL?
> > > > > > > > > > > >>>      We need to integrate with kernel/watchdog.c =
for good results
> > > > > > > > > > > >>>      and to make sure policy is consistent.
> > > > > > > > > > > >>
> > > > > > > > > > > >> That's fine, will consider this.
> > > > > > > > > >
> > > > > > > > > > So after some investigation, it seems the watchdog.c do=
esn't help. The
> > > > > > > > > > only export helper is touch_softlockup_watchdog() which=
 tries to avoid
> > > > > > > > > > triggering the lockups warning for the known slow path.
> > > > > > > > >
> > > > > > > > > I never said you can just use existing exporting APIs. Yo=
u'll have to
> > > > > > > > > write new ones :)
> > > > > > > >
> > > > > > > > Ok, I thought you wanted to trigger similar warnings as a w=
atchdog.
> > > > > > > >
> > > > > > > > Btw, I wonder what kind of logic you want here. If we switc=
h to using
> > > > > > > > sleep, there won't be soft lockup anymore. A simple wait + =
timeout +
> > > > > > > > warning seems sufficient?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > >
> > > > > > > I'd like to avoid need to teach users new APIs. So watchdog s=
etup to apply
> > > > > > > to this driver. The warning can be different.
> > > > > >
> > > > > > Right, so it looks to me the only possible setup is the
> > > > > > watchdog_thres. I plan to trigger the warning every watchdog_th=
res * 2
> > > > > > second (as softlockup did).
> > > > > >
> > > > > > And I think it would still make sense to fail, we can start wit=
h a
> > > > > > very long timeout like 1 minutes and break the device. Does thi=
s make
> > > > > > sense?
> > > > > >
> > > > > > Thanks
> > > > >
> > > > > I'd say we need to make this manageable then.
> > > >
> > > > Did you mean something like sysfs or module parameters?
> > >
> > > No I'd say pass it with an ioctl.
> > >
> > > > > Can't we do it normally
> > > > > e.g. react to an interrupt to return to userspace?
> > > >
> > > > I didn't get the meaning of this. Sorry.
> > > >
> > > > Thanks
> > >
> > > Standard way to handle things that can timeout and where userspace
> > > did not supply the time is to block until an interrupt
> > > then return EINTR.
> >
> > Well this seems to be a huge change, ioctl(2) doesn't say it can
> > return EINTR now.
>
> the one on fedora 37 does not but it says:
>        No single standard.  Arguments, returns, and semantics of ioctl() =
vary according to the device driver in question (the call  is
>        used as a catch-all for operations that don't cleanly fit the UNIX=
 stream I/O model).
>
> so it depends on the device e.g. for a streams device it does:
> https://pubs.opengroup.org/onlinepubs/9699919799/functions/ioctl.html
> has EINTR.

Ok, I saw signal(7) also mention about EINTR for ioctl(2):

"""
       If  a  blocked call to one of the following interfaces is
interrupted by a signal handler, then the call is automatically
restarted after the signal handler re=E2=80=90
       turns if the SA_RESTART flag was used; otherwise the call fails
with the error EINTR:

       * read(2), readv(2), write(2), writev(2), and ioctl(2) calls on
"slow" devices.  A "slow" device is one where the I/O call may block
for an indefinite time, for
         example,  a  terminal,  pipe, or socket.  If an I/O call on a
slow device has already transferred some data by the time it is
interrupted by a signal handler,
         then the call will return a success status (normally, the
number of bytes transferred).  Note that a (local) disk is not a slow
device according to this defi=E2=80=90
         nition; I/O operations on disk devices are not interrupted by sign=
als.
"""

>
>
>
> > Actually, a driver timeout is used by other drivers when using
> > controlq/adminq (e.g i40e). Starting from a sane value (e.g 1 minutes
> > to avoid false negatives) seems to be a good first step.
>
> Well because it's specific hardware so timeout matches what it can
> promise.  virtio spec does not give guarantees.  One issue is with
> software implementations. At the moment I can set a breakpoint in qemu
> or vhost user backend and nothing bad happens in just continues.

Yes but it should be no difference from using a kgdb to debug i40e drivers.

>
>
> > > Userspace controls the timeout by
> > > using e.g. alarm(2).
> >
> > Not used in iproute2 after a git grep.
> >
> > Thanks
>
> No need for iproute2 to do it user can just do it from shell. Or user can=
 just press CTRL-C.

Yes, but iproute2 needs to deal with EINTR, that is the challenge
part, if we simply return an error, the next cvq command might get
confused.

Thanks

>
> > >
> > >
> > > > >
> > > > >
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > > > And before the patch, we end up with a real infinite lo=
op which could
> > > > > > > > > > be caught by RCU stall detector which is not the case o=
f the sleep.
> > > > > > > > > > What we can do is probably do a periodic netdev_err().
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > Only with a bad device.
> > > > > > > > >
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>> 2- overhead. In a very common scenario when devic=
e is in hypervisor,
> > > > > > > > > > > >>>      programming timers etc has a very high overh=
ead, at bootup
> > > > > > > > > > > >>>      lots of CVQ commands are run and slowing boo=
t down is not nice.
> > > > > > > > > > > >>>      let's poll for a bit before waiting?
> > > > > > > > > > > >>
> > > > > > > > > > > >> Then we go back to the question of choosing a good=
 timeout for poll. And
> > > > > > > > > > > >> poll seems problematic in the case of UP, schedule=
r might not have the
> > > > > > > > > > > >> chance to run.
> > > > > > > > > > > > Poll just a bit :) Seriously I don't know, but at l=
east check once
> > > > > > > > > > > > after kick.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I think it is what the current code did where the con=
dition will be
> > > > > > > > > > > check before trying to sleep in the wait_event().
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >>> 3- suprise removal. need to wake up thread in som=
e way. what about
> > > > > > > > > > > >>>      other cases of device breakage - is there a =
chance this
> > > > > > > > > > > >>>      introduces new bugs around that? at least en=
umerate them please.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The current code did:
> > > > > > > > > > > >>
> > > > > > > > > > > >> 1) check for vq->broken
> > > > > > > > > > > >> 2) wakeup during BAD_RING()
> > > > > > > > > > > >>
> > > > > > > > > > > >> So we won't end up with a never woke up process wh=
ich should be fine.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks
> > > > > > > > > > > >
> > > > > > > > > > > > BTW BAD_RING on removal will trigger dev_err. Not s=
ure that is a good
> > > > > > > > > > > > idea - can cause crashes if kernel panics on error.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yes, it's better to use __virtqueue_break() instead.
> > > > > > > > > > >
> > > > > > > > > > > But consider we will start from a wait first, I will =
limit the changes
> > > > > > > > > > > in virtio-net without bothering virtio core.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >>>
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>