Subject: [PATCH] PCI/MSI: skip masking MSI on Xen PV

When running as Xen PV guest, masking MSI is a responsibility of the
hypervisor. Guest has no write access to relevant BAR at all - when it
tries to, it results in a crash like this:

BUG: unable to handle page fault for address: ffffc9004069100c
#PF: supervisor write access in kernel mode
#PF: error_code(0x0003) - permissions violation
PGD 18f1c067 P4D 18f1c067 PUD 4dbd067 PMD 4fba067 PTE 80100000febd4075
Oops: 0003 [#1] SMP NOPTI
CPU: 0 PID: 234 Comm: kworker/0:2 Tainted: G W 5.14.0-rc7-1.fc32.qubes.x86_64 #15
Workqueue: events work_for_cpu_fn
RIP: e030:__pci_enable_msix_range.part.0+0x26b/0x5f0
Code: 2f 96 ff 48 89 44 24 28 48 89 c7 48 85 c0 0f 84 f6 01 00 00 45 0f b7 f6 48 8d 40 0c ba 01 00 00 00 49 c1 e6 04 4a 8d 4c 37 1c <89> 10 48 83 c0 10 48 39 c1 75 f5 41 0f b6 44 24 6a 84 c0 0f 84 48
RSP: e02b:ffffc9004018bd50 EFLAGS: 00010212
RAX: ffffc9004069100c RBX: ffff88800ed412f8 RCX: ffffc9004069105c
RDX: 0000000000000001 RSI: 00000000000febd4 RDI: ffffc90040691000
RBP: 0000000000000003 R08: 0000000000000000 R09: 00000000febd404f
R10: 0000000000007ff0 R11: ffff88800ee8ae40 R12: ffff88800ed41000
R13: 0000000000000000 R14: 0000000000000040 R15: 00000000feba0000
FS: 0000000000000000(0000) GS:ffff888018400000(0000) knlGS:0000000000000000
CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff8000007f5ea0 CR3: 0000000012f6a000 CR4: 0000000000000660
Call Trace:
e1000e_set_interrupt_capability+0xbf/0xd0 [e1000e]
e1000_probe+0x41f/0xdb0 [e1000e]
local_pci_probe+0x42/0x80
(...)

There is pci_msi_ignore_mask variable for bypassing MSI masking on Xen
PV, but msix_mask_all() missed checking it. Add the check there too.

Fixes: 7d5ec3d36123 ("PCI/MSI: Mask all unused MSI-X entries")
Cc: [email protected]
Signed-off-by: Marek Marczykowski-Górecki <[email protected]>
---
Cc: [email protected]
---
drivers/pci/msi.c | 3 +++
1 file changed, 3 insertions(+)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index e5e75331b415..3a9f4f8ad8f9 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -776,6 +776,9 @@ static void msix_mask_all(void __iomem *base, int tsize)
u32 ctrl = PCI_MSIX_ENTRY_CTRL_MASKBIT;
int i;

+ if (pci_msi_ignore_mask)
+ return;
+
for (i = 0; i < tsize; i++, base += PCI_MSIX_ENTRY_SIZE)
writel(ctrl, base + PCI_MSIX_ENTRY_VECTOR_CTRL);
}
--
2.31.1


2021-08-26 14:57:28

by Bjorn Helgaas

[permalink] [raw]
Subject: Re: [PATCH] PCI/MSI: skip masking MSI on Xen PV

If/when you repost this, please run "git log --oneline
drivers/pci/msi.c" and follow the convention of capitalizing the
subject line.

Also, I think this patch refers specifically to MSI-X, not MSI, so
please update the subject line and the "masking MSI" below to reflect
that.

On Thu, Aug 26, 2021 at 03:43:37PM +0200, Marek Marczykowski-G?recki wrote:
> When running as Xen PV guest, masking MSI is a responsibility of the
> hypervisor. Guest has no write access to relevant BAR at all - when it
> tries to, it results in a crash like this:
>
> BUG: unable to handle page fault for address: ffffc9004069100c
> #PF: supervisor write access in kernel mode
> #PF: error_code(0x0003) - permissions violation
> PGD 18f1c067 P4D 18f1c067 PUD 4dbd067 PMD 4fba067 PTE 80100000febd4075
> Oops: 0003 [#1] SMP NOPTI
> CPU: 0 PID: 234 Comm: kworker/0:2 Tainted: G W 5.14.0-rc7-1.fc32.qubes.x86_64 #15
> Workqueue: events work_for_cpu_fn
> RIP: e030:__pci_enable_msix_range.part.0+0x26b/0x5f0
> Code: 2f 96 ff 48 89 44 24 28 48 89 c7 48 85 c0 0f 84 f6 01 00 00 45 0f b7 f6 48 8d 40 0c ba 01 00 00 00 49 c1 e6 04 4a 8d 4c 37 1c <89> 10 48 83 c0 10 48 39 c1 75 f5 41 0f b6 44 24 6a 84 c0 0f 84 48
> RSP: e02b:ffffc9004018bd50 EFLAGS: 00010212
> RAX: ffffc9004069100c RBX: ffff88800ed412f8 RCX: ffffc9004069105c
> RDX: 0000000000000001 RSI: 00000000000febd4 RDI: ffffc90040691000
> RBP: 0000000000000003 R08: 0000000000000000 R09: 00000000febd404f
> R10: 0000000000007ff0 R11: ffff88800ee8ae40 R12: ffff88800ed41000
> R13: 0000000000000000 R14: 0000000000000040 R15: 00000000feba0000
> FS: 0000000000000000(0000) GS:ffff888018400000(0000) knlGS:0000000000000000
> CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffff8000007f5ea0 CR3: 0000000012f6a000 CR4: 0000000000000660
> Call Trace:
> e1000e_set_interrupt_capability+0xbf/0xd0 [e1000e]
> e1000_probe+0x41f/0xdb0 [e1000e]
> local_pci_probe+0x42/0x80
> (...)
>
> There is pci_msi_ignore_mask variable for bypassing MSI masking on Xen
> PV, but msix_mask_all() missed checking it. Add the check there too.
>
> Fixes: 7d5ec3d36123 ("PCI/MSI: Mask all unused MSI-X entries")
> Cc: [email protected]

7d5ec3d36123 appeared in v5.14-rc6, so if this fix is merged before
v5.14, the stable tag will be unnecessary. But we are running out of
time there.

> Signed-off-by: Marek Marczykowski-G?recki <[email protected]>
> ---
> Cc: [email protected]
> ---
> drivers/pci/msi.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index e5e75331b415..3a9f4f8ad8f9 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -776,6 +776,9 @@ static void msix_mask_all(void __iomem *base, int tsize)
> u32 ctrl = PCI_MSIX_ENTRY_CTRL_MASKBIT;
> int i;
>
> + if (pci_msi_ignore_mask)
> + return;
> +
> for (i = 0; i < tsize; i++, base += PCI_MSIX_ENTRY_SIZE)
> writel(ctrl, base + PCI_MSIX_ENTRY_VECTOR_CTRL);
> }
> --
> 2.31.1
>

Subject: Re: [PATCH] PCI/MSI: skip masking MSI on Xen PV

On Thu, Aug 26, 2021 at 09:55:32AM -0500, Bjorn Helgaas wrote:
> If/when you repost this, please run "git log --oneline
> drivers/pci/msi.c" and follow the convention of capitalizing the
> subject line.
>
> Also, I think this patch refers specifically to MSI-X, not MSI, so
> please update the subject line and the "masking MSI" below to reflect
> that.

Sure, thanks for pointing this out. Is the patch otherwise ok? Should I
post v2 with just updated commit message?

> On Thu, Aug 26, 2021 at 03:43:37PM +0200, Marek Marczykowski-Górecki wrote:
> > When running as Xen PV guest, masking MSI is a responsibility of the
> > hypervisor. Guest has no write access to relevant BAR at all - when it
> > tries to, it results in a crash like this:
> >
> > BUG: unable to handle page fault for address: ffffc9004069100c
> > #PF: supervisor write access in kernel mode
> > #PF: error_code(0x0003) - permissions violation
> > PGD 18f1c067 P4D 18f1c067 PUD 4dbd067 PMD 4fba067 PTE 80100000febd4075
> > Oops: 0003 [#1] SMP NOPTI
> > CPU: 0 PID: 234 Comm: kworker/0:2 Tainted: G W 5.14.0-rc7-1.fc32.qubes.x86_64 #15
> > Workqueue: events work_for_cpu_fn
> > RIP: e030:__pci_enable_msix_range.part.0+0x26b/0x5f0
> > Code: 2f 96 ff 48 89 44 24 28 48 89 c7 48 85 c0 0f 84 f6 01 00 00 45 0f b7 f6 48 8d 40 0c ba 01 00 00 00 49 c1 e6 04 4a 8d 4c 37 1c <89> 10 48 83 c0 10 48 39 c1 75 f5 41 0f b6 44 24 6a 84 c0 0f 84 48
> > RSP: e02b:ffffc9004018bd50 EFLAGS: 00010212
> > RAX: ffffc9004069100c RBX: ffff88800ed412f8 RCX: ffffc9004069105c
> > RDX: 0000000000000001 RSI: 00000000000febd4 RDI: ffffc90040691000
> > RBP: 0000000000000003 R08: 0000000000000000 R09: 00000000febd404f
> > R10: 0000000000007ff0 R11: ffff88800ee8ae40 R12: ffff88800ed41000
> > R13: 0000000000000000 R14: 0000000000000040 R15: 00000000feba0000
> > FS: 0000000000000000(0000) GS:ffff888018400000(0000) knlGS:0000000000000000
> > CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: ffff8000007f5ea0 CR3: 0000000012f6a000 CR4: 0000000000000660
> > Call Trace:
> > e1000e_set_interrupt_capability+0xbf/0xd0 [e1000e]
> > e1000_probe+0x41f/0xdb0 [e1000e]
> > local_pci_probe+0x42/0x80
> > (...)
> >
> > There is pci_msi_ignore_mask variable for bypassing MSI masking on Xen
> > PV, but msix_mask_all() missed checking it. Add the check there too.
> >
> > Fixes: 7d5ec3d36123 ("PCI/MSI: Mask all unused MSI-X entries")
> > Cc: [email protected]
>
> 7d5ec3d36123 appeared in v5.14-rc6, so if this fix is merged before
> v5.14, the stable tag will be unnecessary. But we are running out of
> time there.

7d5ec3d36123 was already backported to stable branches (at least 5.10
and 5.4), and in fact this is how I discovered the issue...

> > Signed-off-by: Marek Marczykowski-Górecki <[email protected]>
> > ---
> > Cc: [email protected]
> > ---
> > drivers/pci/msi.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> > index e5e75331b415..3a9f4f8ad8f9 100644
> > --- a/drivers/pci/msi.c
> > +++ b/drivers/pci/msi.c
> > @@ -776,6 +776,9 @@ static void msix_mask_all(void __iomem *base, int tsize)
> > u32 ctrl = PCI_MSIX_ENTRY_CTRL_MASKBIT;
> > int i;
> >
> > + if (pci_msi_ignore_mask)
> > + return;
> > +
> > for (i = 0; i < tsize; i++, base += PCI_MSIX_ENTRY_SIZE)
> > writel(ctrl, base + PCI_MSIX_ENTRY_VECTOR_CTRL);
> > }
> > --
> > 2.31.1
> >

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachments:
(No filename) (3.59 kB)
 signature.asc (499.00 B)
 Download all attachments

2021-08-26 16:58:28

by Bjorn Helgaas

[permalink] [raw]
Subject: Re: [PATCH] PCI/MSI: skip masking MSI on Xen PV

On Thu, Aug 26, 2021 at 06:36:49PM +0200, Marek Marczykowski-G?recki wrote:
> On Thu, Aug 26, 2021 at 09:55:32AM -0500, Bjorn Helgaas wrote:
> > If/when you repost this, please run "git log --oneline
> > drivers/pci/msi.c" and follow the convention of capitalizing the
> > subject line.
> >
> > Also, I think this patch refers specifically to MSI-X, not MSI, so
> > please update the subject line and the "masking MSI" below to reflect
> > that.
>
> Sure, thanks for pointing this out. Is the patch otherwise ok? Should I
> post v2 with just updated commit message?

Wouldn't hurt to post a v2. I don't have any objections to the patch,
but ultimately up to Thomas.

> > On Thu, Aug 26, 2021 at 03:43:37PM +0200, Marek Marczykowski-G?recki wrote:
> > > When running as Xen PV guest, masking MSI is a responsibility of the
> > > hypervisor. Guest has no write access to relevant BAR at all - when it
> > > tries to, it results in a crash like this:
> > >
> > > BUG: unable to handle page fault for address: ffffc9004069100c
> > > #PF: supervisor write access in kernel mode
> > > #PF: error_code(0x0003) - permissions violation
> > > PGD 18f1c067 P4D 18f1c067 PUD 4dbd067 PMD 4fba067 PTE 80100000febd4075
> > > Oops: 0003 [#1] SMP NOPTI
> > > CPU: 0 PID: 234 Comm: kworker/0:2 Tainted: G W 5.14.0-rc7-1.fc32.qubes.x86_64 #15
> > > Workqueue: events work_for_cpu_fn
> > > RIP: e030:__pci_enable_msix_range.part.0+0x26b/0x5f0
> > > Code: 2f 96 ff 48 89 44 24 28 48 89 c7 48 85 c0 0f 84 f6 01 00 00 45 0f b7 f6 48 8d 40 0c ba 01 00 00 00 49 c1 e6 04 4a 8d 4c 37 1c <89> 10 48 83 c0 10 48 39 c1 75 f5 41 0f b6 44 24 6a 84 c0 0f 84 48
> > > RSP: e02b:ffffc9004018bd50 EFLAGS: 00010212
> > > RAX: ffffc9004069100c RBX: ffff88800ed412f8 RCX: ffffc9004069105c
> > > RDX: 0000000000000001 RSI: 00000000000febd4 RDI: ffffc90040691000
> > > RBP: 0000000000000003 R08: 0000000000000000 R09: 00000000febd404f
> > > R10: 0000000000007ff0 R11: ffff88800ee8ae40 R12: ffff88800ed41000
> > > R13: 0000000000000000 R14: 0000000000000040 R15: 00000000feba0000
> > > FS: 0000000000000000(0000) GS:ffff888018400000(0000) knlGS:0000000000000000
> > > CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > CR2: ffff8000007f5ea0 CR3: 0000000012f6a000 CR4: 0000000000000660
> > > Call Trace:
> > > e1000e_set_interrupt_capability+0xbf/0xd0 [e1000e]
> > > e1000_probe+0x41f/0xdb0 [e1000e]
> > > local_pci_probe+0x42/0x80
> > > (...)
> > >
> > > There is pci_msi_ignore_mask variable for bypassing MSI masking on Xen
> > > PV, but msix_mask_all() missed checking it. Add the check there too.
> > >
> > > Fixes: 7d5ec3d36123 ("PCI/MSI: Mask all unused MSI-X entries")
> > > Cc: [email protected]
> >
> > 7d5ec3d36123 appeared in v5.14-rc6, so if this fix is merged before
> > v5.14, the stable tag will be unnecessary. But we are running out of
> > time there.
>
> 7d5ec3d36123 was already backported to stable branches (at least 5.10
> and 5.4), and in fact this is how I discovered the issue...

Oh, right, of course. Sorry :)

> > > Signed-off-by: Marek Marczykowski-G?recki <[email protected]>
> > > ---
> > > Cc: [email protected]
> > > ---
> > > drivers/pci/msi.c | 3 +++
> > > 1 file changed, 3 insertions(+)
> > >
> > > diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> > > index e5e75331b415..3a9f4f8ad8f9 100644
> > > --- a/drivers/pci/msi.c
> > > +++ b/drivers/pci/msi.c
> > > @@ -776,6 +776,9 @@ static void msix_mask_all(void __iomem *base, int tsize)
> > > u32 ctrl = PCI_MSIX_ENTRY_CTRL_MASKBIT;
> > > int i;
> > >
> > > + if (pci_msi_ignore_mask)
> > > + return;
> > > +
> > > for (i = 0; i < tsize; i++, base += PCI_MSIX_ENTRY_SIZE)
> > > writel(ctrl, base + PCI_MSIX_ENTRY_VECTOR_CTRL);
> > > }
> > > --
> > > 2.31.1
> > >
>
> --
> Best Regards,
> Marek Marczykowski-G?recki
> Invisible Things Lab