Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Feedback-ID: i1568416f:Fastmail
Date:   Mon, 21 Nov 2022 17:16:37 +0100
From:   Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= 
        <marmarek@invisiblethingslab.com>
To:     Jason Andryuk <jandryuk@gmail.com>
Cc:     linux-kernel@vger.kernel.org, Juergen Gross <jgross@suse.com>,
        Stefano Stabellini <sstabellini@kernel.org>,
        Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>,
        Jan Beulich <jbeulich@suse.com>,
        "moderated list:XEN HYPERVISOR INTERFACE" 
        <xen-devel@lists.xenproject.org>
Subject: Re: [PATCH v3] xen-pciback: Consider INTx disabled when MSI/MSI-X is
 enabled
Message-ID: <Y3ukZqLaTqzjY/ro@mail-itl>
References: <20221118154931.1928298-1-marmarek@invisiblethingslab.com>
 <CAKf6xpuCxftyQ+PKN_ffJ0onsSxcT8kVSwkM7Z10pfjqf0XFgA@mail.gmail.com>
 <Y3f9O0S8kVXZ+py+@mail-itl>
 <CAKf6xpvUcR=rta6SD7mw_pvgQJTxKjN5VHpC6x-zCAiaJ=pKfg@mail.gmail.com>
 <Y3kFYeml1CPL318u@mail-itl>
 <CAKf6xpts-u_JUtTGnzhxmRHZYVFnVMksTERZWS8EC+7BGsoViw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
        protocol="application/pgp-signature"; boundary="Zq9ZSVbg9oNu91zv"
Content-Disposition: inline
In-Reply-To: <CAKf6xpts-u_JUtTGnzhxmRHZYVFnVMksTERZWS8EC+7BGsoViw@mail.gmail.com>
Precedence: bulk


--Zq9ZSVbg9oNu91zv
Content-Type: text/plain; protected-headers=v1; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Date: Mon, 21 Nov 2022 17:16:37 +0100
From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= <marmarek@invisiblethingslab.com>
To: Jason Andryuk <jandryuk@gmail.com>
Cc: linux-kernel@vger.kernel.org, Juergen Gross <jgross@suse.com>,
	Stefano Stabellini <sstabellini@kernel.org>,
	Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>,
	Jan Beulich <jbeulich@suse.com>,
	"moderated list:XEN HYPERVISOR INTERFACE" <xen-devel@lists.xenproject.org>
Subject: Re: [PATCH v3] xen-pciback: Consider INTx disabled when MSI/MSI-X is
 enabled

On Mon, Nov 21, 2022 at 10:41:34AM -0500, Jason Andryuk wrote:
> On Sat, Nov 19, 2022 at 11:33 AM Marek Marczykowski-G=C3=B3recki
> <marmarek@invisiblethingslab.com> wrote:
> >
> > On Sat, Nov 19, 2022 at 09:36:54AM -0500, Jason Andryuk wrote:
> > > Hi, Marek,
> > >
> > > On Fri, Nov 18, 2022 at 4:46 PM Marek Marczykowski-G=C3=B3recki
> > > <marmarek@invisiblethingslab.com> wrote:
> > > >
> > > > On Fri, Nov 18, 2022 at 03:46:47PM -0500, Jason Andryuk wrote:
> > > > > I was trying to test your xen-pciback v3 patch, and I am having
> > > > > assignment fail consistently now.  It is actually failing to
> > > > > quarantine to domIO in the first place, which matches the failure=
 from
> > > > > the other day (when I more carefully read through the logs).  It =
now
> > > > > consistently fails to quarantine on every boot unlike the other d=
ay
> > > > > where it happened once.
> > > >
> > > > Does this include the very first assignment too, or only after doma=
in
> > > > reboot? If the latter, maybe some cleanup missed clearing MASKALL?
> > >
> > > It's the quarantine during dom0 boot that fails.  Later assignment
> > > during VM boot fails.  I tried warm reboots and cold boots and it
> > > happened both times.
> > >
> > > I also modified my initrd to halt in there and checked the config
> > > space.  MASKALL wasn't set at that time.  I need to double check -
> > > MASKALL may have been unset after dom0 booted in that case.
> > >
> > > I'll test more to figure when and how MASKALL is getting set.
>=20
> I'm testing with a laptop without a battery.  It seems MASKALL remains
> set when rebooting or when left plugged in.
>=20
> From unplugged, a cold boot doesn't have MASKALL set and the network vm b=
oots.
>=20
> After that, rebooting the laptop leaves MASKALL set on the NIC when
> the laptop reboots.   NIC assignment fails.
>=20
> Shutdown and later boot while left plugged in keeps MASKALL set.  NIC
> assignment fails.  I have only tested this scenario for short periods
> of time, so I don't know if it would eventually clear after a longer
> time.

That's interesting, seems like firmware is not resetting the device
properly. Maybe related to enabled wake on lan?

Anyway, resetting the device at domain create/destroy is AFAIR normally
done by pciback (at the instruction by the toolstack). Should it maybe
be done when assigning to pciback initially too? Or maybe in this
specific case, device reset doesn't properly clear MASKALL, so pciback
should clear it explicitly (after ensuring the MSI-X enable is cleared
too)?

--=20
Best Regards,
Marek Marczykowski-G=C3=B3recki
Invisible Things Lab

--Zq9ZSVbg9oNu91zv
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmN7pGYACgkQ24/THMrX
1yxERQf9H+wGG9EoUn9lEqAbJsW6Pe0L4+RgIEU63aMDb1AWJH8jNYkUWaNM5X6A
goktstxeX/Y3luFyhQujlu4+SyqVZdouIzyB2Y6LUf+GVo9a2uKVVYR1OpFAcoEM
w4vCPhDx4T6IE1iAWxLmLJzRz5c1BOEp7srCrDMCCShggoiKpgGrOeXaKuu1rzPU
sSj+BTJcnR/TT+REL1HQkClQIzA4+bfjQyfU3uBKcMzS3IEJdb5BEu/ApHcWO/i7
k0ciDq9j3iBxBZ7tE7CbsY3eM8mYuQUnF8PjA2T/9R3gge0P2pogZmYGxfHsfjbA
7arbj3nv9o75HQoDNWT+a10mfYgsDw==
=haAo
-----END PGP SIGNATURE-----

--Zq9ZSVbg9oNu91zv--