Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Feedback-ID: i1568416f:Fastmail
Date:   Mon, 28 Nov 2022 14:44:39 +0100
From:   Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= 
        <marmarek@invisiblethingslab.com>
To:     Jason Andryuk <jandryuk@gmail.com>
Cc:     linux-kernel@vger.kernel.org, Juergen Gross <jgross@suse.com>,
        Stefano Stabellini <sstabellini@kernel.org>,
        Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>,
        Jan Beulich <jbeulich@suse.com>,
        "moderated list:XEN HYPERVISOR INTERFACE" 
        <xen-devel@lists.xenproject.org>
Subject: Re: [PATCH v3] xen-pciback: Consider INTx disabled when MSI/MSI-X is
 enabled
Message-ID: <Y4S7R2dN7kpJnPpY@mail-itl>
References: <20221118154931.1928298-1-marmarek@invisiblethingslab.com>
 <CAKf6xpuCxftyQ+PKN_ffJ0onsSxcT8kVSwkM7Z10pfjqf0XFgA@mail.gmail.com>
 <Y3f9O0S8kVXZ+py+@mail-itl>
 <CAKf6xpvUcR=rta6SD7mw_pvgQJTxKjN5VHpC6x-zCAiaJ=pKfg@mail.gmail.com>
 <Y3kFYeml1CPL318u@mail-itl>
 <CAKf6xpts-u_JUtTGnzhxmRHZYVFnVMksTERZWS8EC+7BGsoViw@mail.gmail.com>
 <Y3ukZqLaTqzjY/ro@mail-itl>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
        protocol="application/pgp-signature"; boundary="aozQvSbj7iUV5V+i"
Content-Disposition: inline
In-Reply-To: <Y3ukZqLaTqzjY/ro@mail-itl>
Precedence: bulk


--aozQvSbj7iUV5V+i
Content-Type: text/plain; protected-headers=v1; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Date: Mon, 28 Nov 2022 14:44:39 +0100
From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= <marmarek@invisiblethingslab.com>
To: Jason Andryuk <jandryuk@gmail.com>
Cc: linux-kernel@vger.kernel.org, Juergen Gross <jgross@suse.com>,
	Stefano Stabellini <sstabellini@kernel.org>,
	Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>,
	Jan Beulich <jbeulich@suse.com>,
	"moderated list:XEN HYPERVISOR INTERFACE" <xen-devel@lists.xenproject.org>
Subject: Re: [PATCH v3] xen-pciback: Consider INTx disabled when MSI/MSI-X is
 enabled

On Mon, Nov 21, 2022 at 05:16:37PM +0100, Marek Marczykowski-G=C3=B3recki w=
rote:
> On Mon, Nov 21, 2022 at 10:41:34AM -0500, Jason Andryuk wrote:
> > On Sat, Nov 19, 2022 at 11:33 AM Marek Marczykowski-G=C3=B3recki
> > <marmarek@invisiblethingslab.com> wrote:
> > >
> > > On Sat, Nov 19, 2022 at 09:36:54AM -0500, Jason Andryuk wrote:
> > > > Hi, Marek,
> > > >
> > > > On Fri, Nov 18, 2022 at 4:46 PM Marek Marczykowski-G=C3=B3recki
> > > > <marmarek@invisiblethingslab.com> wrote:
> > > > >
> > > > > On Fri, Nov 18, 2022 at 03:46:47PM -0500, Jason Andryuk wrote:
> > > > > > I was trying to test your xen-pciback v3 patch, and I am having
> > > > > > assignment fail consistently now.  It is actually failing to
> > > > > > quarantine to domIO in the first place, which matches the failu=
re from
> > > > > > the other day (when I more carefully read through the logs).  I=
t now
> > > > > > consistently fails to quarantine on every boot unlike the other=
 day
> > > > > > where it happened once.
> > > > >
> > > > > Does this include the very first assignment too, or only after do=
main
> > > > > reboot? If the latter, maybe some cleanup missed clearing MASKALL?
> > > >
> > > > It's the quarantine during dom0 boot that fails.  Later assignment
> > > > during VM boot fails.  I tried warm reboots and cold boots and it
> > > > happened both times.
> > > >
> > > > I also modified my initrd to halt in there and checked the config
> > > > space.  MASKALL wasn't set at that time.  I need to double check -
> > > > MASKALL may have been unset after dom0 booted in that case.
> > > >
> > > > I'll test more to figure when and how MASKALL is getting set.
> >=20
> > I'm testing with a laptop without a battery.  It seems MASKALL remains
> > set when rebooting or when left plugged in.
> >=20
> > From unplugged, a cold boot doesn't have MASKALL set and the network vm=
 boots.
> >=20
> > After that, rebooting the laptop leaves MASKALL set on the NIC when
> > the laptop reboots.   NIC assignment fails.
> >=20
> > Shutdown and later boot while left plugged in keeps MASKALL set.  NIC
> > assignment fails.  I have only tested this scenario for short periods
> > of time, so I don't know if it would eventually clear after a longer
> > time.
>=20
> That's interesting, seems like firmware is not resetting the device
> properly. Maybe related to enabled wake on lan?
>=20
> Anyway, resetting the device at domain create/destroy is AFAIR normally
> done by pciback (at the instruction by the toolstack). Should it maybe
> be done when assigning to pciback initially too? Or maybe in this
> specific case, device reset doesn't properly clear MASKALL, so pciback
> should clear it explicitly (after ensuring the MSI-X enable is cleared
> too)?

Can you check if `echo 1 > /sys/bus/pci/devices/$SBDF/reset` clears
MASKALL on this device?

I'm tempted to add libxl__device_pci_reset() as part of
libxl__device_pci_assignable_add().

--=20
Best Regards,
Marek Marczykowski-G=C3=B3recki
Invisible Things Lab

--aozQvSbj7iUV5V+i
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmOEu0cACgkQ24/THMrX
1ywlKQgAjmX9f7oEMMKhPvTU9Cwer1pxYt/jaRfYJThGAZvH/4w7VkyNInSN2u6G
7obNR47Rju6OXU1j7JOO5uW6DDQ6oo4tyRzYUpJl3a0QVSRpTgSN32eisomrGSSB
1/TU7XFcKS+ZBXJaRqq6whhdiXwFRuPhuO96pd4JVcxRoL2ocGzS/3B01Vv4OF5N
0bamXgj9FFQvUcXHcQRWzwj/mjOShwe/dYeDjaAmCf4kKJBAX5TJg3mEu3e7yz3d
MiKHnU0bLSOC38cW3/XMs16NH3cMYD48ickM7Eca5PmZAX7wQrAeCJiVBuH8IHPR
ZlSWFUeLsMnImI4snXIT4aex+c/2Mw==
=Yg9M
-----END PGP SIGNATURE-----

--aozQvSbj7iUV5V+i--