IOMMU harms performance signficantly when we run very fast networking
workloads. This is a limitation in hardware based on our observation, so
we'd like to disable the IOMMU force on, but we do want to use TBOOT and
we can sacrifice the DMA security bought by IOMMU. I must admit I know
nothing about TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is
totally ok.
So introduce a new boot option to disable the force on. It's kind of
silly we need to run into intel_iommu_init even without force on, but we
need to disable TBOOT PMR registers. For system without the boot option,
nothing is changed.
Signed-off-by: Shaohua Li <[email protected]>
---
arch/x86/kernel/tboot.c | 3 +++
drivers/iommu/intel-iommu.c | 21 ++++++++++++++++++++-
include/linux/dma_remapping.h | 1 +
3 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kernel/tboot.c b/arch/x86/kernel/tboot.c
index b868fa1..edbdfe6 100644
--- a/arch/x86/kernel/tboot.c
+++ b/arch/x86/kernel/tboot.c
@@ -510,6 +510,9 @@ int tboot_force_iommu(void)
if (!tboot_enabled())
return 0;
+ if (!intel_iommu_tboot_noforce)
+ return 1;
+
if (no_iommu || swiotlb || dmar_disabled)
pr_warning("Forcing Intel-IOMMU to enabled\n");
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 238ad34..737dfa7 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -183,6 +183,7 @@ static int rwbf_quirk;
* (used when kernel is launched w/ TXT)
*/
static int force_on = 0;
+int intel_iommu_tboot_noforce;
/*
* 0: Present
@@ -607,6 +608,10 @@ static int __init intel_iommu_setup(char *str)
"Intel-IOMMU: enable pre-production PASID support\n");
intel_iommu_pasid28 = 1;
iommu_identity_mapping |= IDENTMAP_GFX;
+ } else if (!strncmp(str, "tboot_noforce", 13)) {
+ printk(KERN_INFO
+ "Intel-IOMMU: not forcing on after tboot\n");
+ intel_iommu_tboot_noforce = 1;
}
str += strcspn(str, ",");
@@ -4840,8 +4845,22 @@ int __init intel_iommu_init(void)
goto out_free_dmar;
}
- if (no_iommu || dmar_disabled)
+ if (no_iommu || dmar_disabled) {
+ /*
+ * We exit the function here to ensure IOMMU's remapping and
+ * mempool aren't setup, which means that the IOMMU's PMRs
+ * won't be disabled via the call to init_dmars(). So disable
+ * it explicitly here. The PMRs were setup by tboot prior to
+ * calling SENTER, but the kernel is expected to reset/tear
+ * down the PMRs.
+ */
+ if (intel_iommu_tboot_noforce) {
+ for_each_iommu(iommu, drhd)
+ iommu_disable_protect_mem_regions(iommu);
+ }
+
goto out_free_dmar;
+ }
if (list_empty(&dmar_rmrr_units))
pr_info("No RMRR found\n");
diff --git a/include/linux/dma_remapping.h b/include/linux/dma_remapping.h
index 187c102..9088407 100644
--- a/include/linux/dma_remapping.h
+++ b/include/linux/dma_remapping.h
@@ -39,6 +39,7 @@ extern int iommu_calculate_agaw(struct intel_iommu *iommu);
extern int iommu_calculate_max_sagaw(struct intel_iommu *iommu);
extern int dmar_disabled;
extern int intel_iommu_enabled;
+extern int intel_iommu_tboot_noforce;
#else
static inline int iommu_calculate_agaw(struct intel_iommu *iommu)
{
--
2.9.3
Hi Shaohua,
On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote:
> IOMMU harms performance signficantly when we run very fast networking
> workloads. This is a limitation in hardware based on our observation, so
> we'd like to disable the IOMMU force on, but we do want to use TBOOT and
> we can sacrifice the DMA security bought by IOMMU. I must admit I know
> nothing about TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is
> totally ok.
Can you elaborate a bit more on the setup where the IOMMU still harms
network performance? With the recent scalability improvements I measured
only a minimal impact on 10GBit networking.
Joerg
On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote:
> Hi Shaohua,
>
> On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote:
> > IOMMU harms performance signficantly when we run very fast networking
> > workloads. This is a limitation in hardware based on our observation, so
> > we'd like to disable the IOMMU force on, but we do want to use TBOOT and
> > we can sacrifice the DMA security bought by IOMMU. I must admit I know
> > nothing about TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is
> > totally ok.
>
> Can you elaborate a bit more on the setup where the IOMMU still harms
> network performance? With the recent scalability improvements I measured
> only a minimal impact on 10GBit networking.
Hi,
It's 40GB networking doing XDP test. Software overhead is almost unaware, but
it's the IOTLB miss (based on our analysis) which kills the performance. We
observed the same performance issue even with software passthrough (identity
mapping), only the hardware passthrough survives. The pps with iommu (with
software passthrough) is only about ~30% of that without it.
Thanks,
Shaohua
On Wed, Mar 22, 2017 at 07:50:55AM -0400, Shaohua Li wrote:
> On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote:
> > Hi Shaohua,
> >
> > On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote:
> > > IOMMU harms performance signficantly when we run very fast networking
> > > workloads. This is a limitation in hardware based on our observation, so
> > > we'd like to disable the IOMMU force on, but we do want to use TBOOT and
> > > we can sacrifice the DMA security bought by IOMMU. I must admit I know
> > > nothing about TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is
> > > totally ok.
> >
> > Can you elaborate a bit more on the setup where the IOMMU still harms
> > network performance? With the recent scalability improvements I measured
> > only a minimal impact on 10GBit networking.
> Hi,
>
> It's 40GB networking doing XDP test. Software overhead is almost unaware, but
> it's the IOTLB miss (based on our analysis) which kills the performance. We
> observed the same performance issue even with software passthrough (identity
> mapping), only the hardware passthrough survives. The pps with iommu (with
> software passthrough) is only about ~30% of that without it.
Hi,
Any update on this?
Thanks,
Shaohua
On Mon, Apr 03, 2017 at 12:19:28PM -0700, Shaohua Li wrote:
> On Wed, Mar 22, 2017 at 07:50:55AM -0400, Shaohua Li wrote:
> > On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote:
> > > Hi Shaohua,
> > >
> > > On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote:
> > > > IOMMU harms performance signficantly when we run very fast networking
> > > > workloads. This is a limitation in hardware based on our observation, so
> > > > we'd like to disable the IOMMU force on, but we do want to use TBOOT and
> > > > we can sacrifice the DMA security bought by IOMMU. I must admit I know
> > > > nothing about TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is
> > > > totally ok.
> > >
> > > Can you elaborate a bit more on the setup where the IOMMU still harms
> > > network performance? With the recent scalability improvements I measured
> > > only a minimal impact on 10GBit networking.
> > Hi,
> >
> > It's 40GB networking doing XDP test. Software overhead is almost unaware, but
> > it's the IOTLB miss (based on our analysis) which kills the performance. We
> > observed the same performance issue even with software passthrough (identity
> > mapping), only the hardware passthrough survives. The pps with iommu (with
> > software passthrough) is only about ~30% of that without it.
>
> Any update on this?
An explicit Ack from the tboot guys would be good to have.
Joerg
Hi Shaohua,
One question, did you still see the network performance penalty when Linux kernel cmdline intel_iommu was set to off ( intel_iommu=off) ?
Thanks,
-ning
-----Original Message-----
From: Joerg Roedel [mailto:[email protected]]
Sent: Friday, April 07, 2017 3:09 AM
To: Shaohua Li <[email protected]>
Cc: [email protected]; Wei, Gang <[email protected]>; [email protected]; [email protected]; [email protected]; Sun, Ning <[email protected]>; [email protected]; Eydelberg, Alex <[email protected]>
Subject: Re: [RFC] x86/tboot: add an option to disable iommu force on
On Mon, Apr 03, 2017 at 12:19:28PM -0700, Shaohua Li wrote:
> On Wed, Mar 22, 2017 at 07:50:55AM -0400, Shaohua Li wrote:
> > On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote:
> > > Hi Shaohua,
> > >
> > > On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote:
> > > > IOMMU harms performance signficantly when we run very fast
> > > > networking workloads. This is a limitation in hardware based on
> > > > our observation, so we'd like to disable the IOMMU force on, but
> > > > we do want to use TBOOT and we can sacrifice the DMA security
> > > > bought by IOMMU. I must admit I know nothing about TBOOT, but
> > > > TBOOT guys (cc-ed) think not eabling IOMMU is totally ok.
> > >
> > > Can you elaborate a bit more on the setup where the IOMMU still
> > > harms network performance? With the recent scalability
> > > improvements I measured only a minimal impact on 10GBit networking.
> > Hi,
> >
> > It's 40GB networking doing XDP test. Software overhead is almost
> > unaware, but it's the IOTLB miss (based on our analysis) which kills
> > the performance. We observed the same performance issue even with
> > software passthrough (identity mapping), only the hardware
> > passthrough survives. The pps with iommu (with software passthrough) is only about ~30% of that without it.
>
> Any update on this?
An explicit Ack from the tboot guys would be good to have.
Joerg
On Fri, Apr 07, 2017 at 09:49:52PM +0000, Sun, Ning wrote:
> Hi Shaohua,
>
> One question, did you still see the network performance penalty when Linux kernel cmdline intel_iommu was set to off ( intel_iommu=off) ?
the boot parameter has no effect, it runs very early and set dmar_disable=1.
The tboot code (tboot_force_iommu) runs later and force dmar_disabled = 0.
Thanks,
Shaohua
> Thanks,
> -ning
>
> -----Original Message-----
> From: Joerg Roedel [mailto:[email protected]]
> Sent: Friday, April 07, 2017 3:09 AM
> To: Shaohua Li <[email protected]>
> Cc: [email protected]; Wei, Gang <[email protected]>; [email protected]; [email protected]; [email protected]; Sun, Ning <[email protected]>; [email protected]; Eydelberg, Alex <[email protected]>
> Subject: Re: [RFC] x86/tboot: add an option to disable iommu force on
>
> On Mon, Apr 03, 2017 at 12:19:28PM -0700, Shaohua Li wrote:
> > On Wed, Mar 22, 2017 at 07:50:55AM -0400, Shaohua Li wrote:
> > > On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote:
> > > > Hi Shaohua,
> > > >
> > > > On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote:
> > > > > IOMMU harms performance signficantly when we run very fast
> > > > > networking workloads. This is a limitation in hardware based on
> > > > > our observation, so we'd like to disable the IOMMU force on, but
> > > > > we do want to use TBOOT and we can sacrifice the DMA security
> > > > > bought by IOMMU. I must admit I know nothing about TBOOT, but
> > > > > TBOOT guys (cc-ed) think not eabling IOMMU is totally ok.
> > > >
> > > > Can you elaborate a bit more on the setup where the IOMMU still
> > > > harms network performance? With the recent scalability
> > > > improvements I measured only a minimal impact on 10GBit networking.
> > > Hi,
> > >
> > > It's 40GB networking doing XDP test. Software overhead is almost
> > > unaware, but it's the IOTLB miss (based on our analysis) which kills
> > > the performance. We observed the same performance issue even with
> > > software passthrough (identity mapping), only the hardware
> > > passthrough survives. The pps with iommu (with software passthrough) is only about ~30% of that without it.
> >
> > Any update on this?
>
> An explicit Ack from the tboot guys would be good to have.
>
>
> Joerg
>
>From tboot perspective, it is ok to add the option "tboot_noforce" to Linux kernel Intel_iommu parameter for those performance hungry tboot users, so long as the users are aware of the security implication behind of this option.
Thanks,
-ning
-----Original Message-----
From: Shaohua Li [mailto:[email protected]]
Sent: Sunday, April 09, 2017 9:31 PM
To: Sun, Ning <[email protected]>
Cc: Joerg Roedel <[email protected]>; [email protected]; Wei, Gang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Eydelberg, Alex <[email protected]>
Subject: Re: [RFC] x86/tboot: add an option to disable iommu force on
On Fri, Apr 07, 2017 at 09:49:52PM +0000, Sun, Ning wrote:
> Hi Shaohua,
>
> One question, did you still see the network performance penalty when Linux kernel cmdline intel_iommu was set to off ( intel_iommu=off) ?
the boot parameter has no effect, it runs very early and set dmar_disable=1.
The tboot code (tboot_force_iommu) runs later and force dmar_disabled = 0.
Thanks,
Shaohua
> Thanks,
> -ning
>
> -----Original Message-----
> From: Joerg Roedel [mailto:[email protected]]
> Sent: Friday, April 07, 2017 3:09 AM
> To: Shaohua Li <[email protected]>
> Cc: [email protected]; Wei, Gang <[email protected]>;
> [email protected]; [email protected]; [email protected]; Sun, Ning
> <[email protected]>; [email protected]; Eydelberg, Alex
> <[email protected]>
> Subject: Re: [RFC] x86/tboot: add an option to disable iommu force on
>
> On Mon, Apr 03, 2017 at 12:19:28PM -0700, Shaohua Li wrote:
> > On Wed, Mar 22, 2017 at 07:50:55AM -0400, Shaohua Li wrote:
> > > On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote:
> > > > Hi Shaohua,
> > > >
> > > > On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote:
> > > > > IOMMU harms performance signficantly when we run very fast
> > > > > networking workloads. This is a limitation in hardware based
> > > > > on our observation, so we'd like to disable the IOMMU force
> > > > > on, but we do want to use TBOOT and we can sacrifice the DMA
> > > > > security bought by IOMMU. I must admit I know nothing about
> > > > > TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is totally ok.
> > > >
> > > > Can you elaborate a bit more on the setup where the IOMMU still
> > > > harms network performance? With the recent scalability
> > > > improvements I measured only a minimal impact on 10GBit networking.
> > > Hi,
> > >
> > > It's 40GB networking doing XDP test. Software overhead is almost
> > > unaware, but it's the IOTLB miss (based on our analysis) which
> > > kills the performance. We observed the same performance issue even
> > > with software passthrough (identity mapping), only the hardware
> > > passthrough survives. The pps with iommu (with software passthrough) is only about ~30% of that without it.
> >
> > Any update on this?
>
> An explicit Ack from the tboot guys would be good to have.
>
>
> Joerg
>
Hi Joerg,
Is Ning's answer sufficient to justify merging the patch?
Thanks,
Shaohua
On Mon, Apr 10, 2017 at 09:28:46PM +0000, Sun, Ning wrote:
> From tboot perspective, it is ok to add the option "tboot_noforce" to Linux kernel Intel_iommu parameter for those performance hungry tboot users, so long as the users are aware of the security implication behind of this option.
>
> Thanks,
> -ning
>
> -----Original Message-----
> From: Shaohua Li [mailto:[email protected]]
> Sent: Sunday, April 09, 2017 9:31 PM
> To: Sun, Ning <[email protected]>
> Cc: Joerg Roedel <[email protected]>; [email protected]; Wei, Gang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Eydelberg, Alex <[email protected]>
> Subject: Re: [RFC] x86/tboot: add an option to disable iommu force on
>
> On Fri, Apr 07, 2017 at 09:49:52PM +0000, Sun, Ning wrote:
> > Hi Shaohua,
> >
> > One question, did you still see the network performance penalty when Linux kernel cmdline intel_iommu was set to off ( intel_iommu=off) ?
>
> the boot parameter has no effect, it runs very early and set dmar_disable=1.
> The tboot code (tboot_force_iommu) runs later and force dmar_disabled = 0.
>
> Thanks,
> Shaohua
>
> > Thanks,
> > -ning
> >
> > -----Original Message-----
> > From: Joerg Roedel [mailto:[email protected]]
> > Sent: Friday, April 07, 2017 3:09 AM
> > To: Shaohua Li <[email protected]>
> > Cc: [email protected]; Wei, Gang <[email protected]>;
> > [email protected]; [email protected]; [email protected]; Sun, Ning
> > <[email protected]>; [email protected]; Eydelberg, Alex
> > <[email protected]>
> > Subject: Re: [RFC] x86/tboot: add an option to disable iommu force on
> >
> > On Mon, Apr 03, 2017 at 12:19:28PM -0700, Shaohua Li wrote:
> > > On Wed, Mar 22, 2017 at 07:50:55AM -0400, Shaohua Li wrote:
> > > > On Wed, Mar 22, 2017 at 11:49:00AM +0100, Joerg Roedel wrote:
> > > > > Hi Shaohua,
> > > > >
> > > > > On Tue, Mar 21, 2017 at 11:37:51AM -0700, Shaohua Li wrote:
> > > > > > IOMMU harms performance signficantly when we run very fast
> > > > > > networking workloads. This is a limitation in hardware based
> > > > > > on our observation, so we'd like to disable the IOMMU force
> > > > > > on, but we do want to use TBOOT and we can sacrifice the DMA
> > > > > > security bought by IOMMU. I must admit I know nothing about
> > > > > > TBOOT, but TBOOT guys (cc-ed) think not eabling IOMMU is totally ok.
> > > > >
> > > > > Can you elaborate a bit more on the setup where the IOMMU still
> > > > > harms network performance? With the recent scalability
> > > > > improvements I measured only a minimal impact on 10GBit networking.
> > > > Hi,
> > > >
> > > > It's 40GB networking doing XDP test. Software overhead is almost
> > > > unaware, but it's the IOTLB miss (based on our analysis) which
> > > > kills the performance. We observed the same performance issue even
> > > > with software passthrough (identity mapping), only the hardware
> > > > passthrough survives. The pps with iommu (with software passthrough) is only about ~30% of that without it.
> > >
> > > Any update on this?
> >
> > An explicit Ack from the tboot guys would be good to have.
> >
> >
> > Joerg
> >
On Mon, Apr 24, 2017 at 09:50:59AM -0700, Shaohua Li wrote:
> Hi Joerg,
>
> Is Ning's answer sufficient to justify merging the patch?
Yes, I will take it if you repost without the RFC tag and when you add
documentation for the new command-line parameter.
I think a kernel-log message about the potential security impact also
makes sense.
Joerg