IOMMU implementations now sometimes bounce memory through SWIOTLB to
achieve cacheline alignment [1], or prevent DMA attacks by untrusted
devices [2]. These uses of SWIOTLB differ conceptually from historical
use which was a solution to the problem of device addressing
limitations that prevent DMA to some portion of system memory
(typically beyond 4 GiB). IOMMUs also solve the problem of device
addressing limitations and therefore eliminate the need for SWIOTLB for
that purpose. However as mentioned, IOMMUs can use SWIOTLB for other
purposes.
The swiotlb kernel command line parameter does not impact IOMMU related
use of SWIOTLB, and that is intentional. IOMMUs cannot be forced to use
SWIOTLB for all buffers. Update the documentation for the swiotlb
parameter to clarify that SWIOTLB use can only be forced in scenarios
where an IOMMU is not involved.
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]/
Signed-off-by: T.J. Mercier <[email protected]>
---
Documentation/admin-guide/kernel-parameters.txt | 1 +
Documentation/arch/x86/x86_64/boot-options.rst | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 213d0719e2b7..84c582ac246c 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -6486,6 +6486,7 @@
to a power of 2.
force -- force using of bounce buffers even if they
wouldn't be automatically used by the kernel
+ where a hardware IOMMU is not involved
noforce -- Never use bounce buffers (for debugging)
switches= [HW,M68k,EARLY]
diff --git a/Documentation/arch/x86/x86_64/boot-options.rst b/Documentation/arch/x86/x86_64/boot-options.rst
index 137432d34109..066b4bc81583 100644
--- a/Documentation/arch/x86/x86_64/boot-options.rst
+++ b/Documentation/arch/x86/x86_64/boot-options.rst
@@ -285,7 +285,7 @@ iommu options only relevant to the AMD GART hardware IOMMU:
Always panic when IOMMU overflows.
iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
-implementation:
+implementation where a hardware IOMMU is not involved:
swiotlb=<slots>[,force,noforce]
<slots>
--
2.45.0.rc1.225.g2a3ae87e7f-goog
On Fri, 3 May 2024 18:35:26 +0000
"T.J. Mercier" <[email protected]> wrote:
> IOMMU implementations now sometimes bounce memory through SWIOTLB to
> achieve cacheline alignment [1], or prevent DMA attacks by untrusted
> devices [2]. These uses of SWIOTLB differ conceptually from historical
> use which was a solution to the problem of device addressing
> limitations that prevent DMA to some portion of system memory
> (typically beyond 4 GiB). IOMMUs also solve the problem of device
> addressing limitations and therefore eliminate the need for SWIOTLB for
> that purpose. However as mentioned, IOMMUs can use SWIOTLB for other
> purposes.
>
> The swiotlb kernel command line parameter does not impact IOMMU related
> use of SWIOTLB, and that is intentional. IOMMUs cannot be forced to use
> SWIOTLB for all buffers. Update the documentation for the swiotlb
> parameter to clarify that SWIOTLB use can only be forced in scenarios
> where an IOMMU is not involved.
>
> [1] https://lore.kernel.org/all/[email protected]
> [2] https://lore.kernel.org/all/[email protected]/
> Signed-off-by: T.J. Mercier <[email protected]>
> ---
> Documentation/admin-guide/kernel-parameters.txt | 1 +
> Documentation/arch/x86/x86_64/boot-options.rst | 2 +-
> 2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 213d0719e2b7..84c582ac246c 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -6486,6 +6486,7 @@
> to a power of 2.
> force -- force using of bounce buffers even if they
> wouldn't be automatically used by the kernel
> + where a hardware IOMMU is not involved
> noforce -- Never use bounce buffers (for debugging)
>
> switches= [HW,M68k,EARLY]
Yes, this part is correct. SWIOTLB cannot be forced if there is an IOMMU.
> diff --git a/Documentation/arch/x86/x86_64/boot-options.rst b/Documentation/arch/x86/x86_64/boot-options.rst
> index 137432d34109..066b4bc81583 100644
> --- a/Documentation/arch/x86/x86_64/boot-options.rst
> +++ b/Documentation/arch/x86/x86_64/boot-options.rst
> @@ -285,7 +285,7 @@ iommu options only relevant to the AMD GART hardware IOMMU:
> Always panic when IOMMU overflows.
>
> iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
> -implementation:
> +implementation where a hardware IOMMU is not involved:
>
> swiotlb=<slots>[,force,noforce]
> <slots>
But this part needs some improvement. The "swiotlb" option is not
entirely ignored if there is a hardware IOMMU. For example, the size of
the SWIOTLB can be adjusted using "swiotlb=<slots>", and since SWIOTLB
can be used by IOMMUs for other purposes (as you correctly note in the
commit message), this setting is relevant even where a hardware IOMMU
is involved.
Petr T
On Sat, May 4, 2024 at 1:16 AM Petr Tesařík <[email protected]> wrote:
>
> On Fri, 3 May 2024 18:35:26 +0000
> "T.J. Mercier" <[email protected]> wrote:
>
> > IOMMU implementations now sometimes bounce memory through SWIOTLB to
> > achieve cacheline alignment [1], or prevent DMA attacks by untrusted
> > devices [2]. These uses of SWIOTLB differ conceptually from historical
> > use which was a solution to the problem of device addressing
> > limitations that prevent DMA to some portion of system memory
> > (typically beyond 4 GiB). IOMMUs also solve the problem of device
> > addressing limitations and therefore eliminate the need for SWIOTLB for
> > that purpose. However as mentioned, IOMMUs can use SWIOTLB for other
> > purposes.
> >
> > The swiotlb kernel command line parameter does not impact IOMMU related
> > use of SWIOTLB, and that is intentional. IOMMUs cannot be forced to use
> > SWIOTLB for all buffers. Update the documentation for the swiotlb
> > parameter to clarify that SWIOTLB use can only be forced in scenarios
> > where an IOMMU is not involved.
> >
> > [1] https://lore.kernel.org/all/[email protected]
> > [2] https://lore.kernel.org/all/[email protected]/
> > Signed-off-by: T.J. Mercier <[email protected]>
> > ---
> > Documentation/admin-guide/kernel-parameters.txt | 1 +
> > Documentation/arch/x86/x86_64/boot-options.rst | 2 +-
> > 2 files changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > index 213d0719e2b7..84c582ac246c 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -6486,6 +6486,7 @@
> > to a power of 2.
> > force -- force using of bounce buffers even if they
> > wouldn't be automatically used by the kernel
> > + where a hardware IOMMU is not involved
> > noforce -- Never use bounce buffers (for debugging)
> >
> > switches= [HW,M68k,EARLY]
>
> Yes, this part is correct. SWIOTLB cannot be forced if there is an IOMMU.
>
> > diff --git a/Documentation/arch/x86/x86_64/boot-options.rst b/Documentation/arch/x86/x86_64/boot-options.rst
> > index 137432d34109..066b4bc81583 100644
> > --- a/Documentation/arch/x86/x86_64/boot-options.rst
> > +++ b/Documentation/arch/x86/x86_64/boot-options.rst
> > @@ -285,7 +285,7 @@ iommu options only relevant to the AMD GART hardware IOMMU:
> > Always panic when IOMMU overflows.
> >
> > iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
> > -implementation:
> > +implementation where a hardware IOMMU is not involved:
> >
> > swiotlb=<slots>[,force,noforce]
> > <slots>
>
> But this part needs some improvement. The "swiotlb" option is not
> entirely ignored if there is a hardware IOMMU. For example, the size of
> the SWIOTLB can be adjusted using "swiotlb=<slots>", and since SWIOTLB
> can be used by IOMMUs for other purposes (as you correctly note in the
> commit message), this setting is relevant even where a hardware IOMMU
> is involved.
>
> Petr T
Thanks. I think I should also update the commit message:
"The swiotlb=force kernel command line parameter does not impact IOMMU
related use of SWIOTLB"
and title:
"Clarify swiotlb=force option applies only to dma-direct"
As for fixing boot-options.txt, I think it makes the most sense to
expand on just the force option rather than the section summary like
above:
force
Force all IO through the software TLB.
+ Hardware IOMMU implementations can use SWIOTLB bounce buffering in
+ some circumstances, but they cannot be forced to always use them, so
+ this option only has an effect when no hardware IOMMU is involved.
noforce
From: T.J. Mercier <[email protected]> Sent: Monday, May 6, 2024 10:23 AM
>
> On Sat, May 4, 2024 at 1:16 AM Petr Tesařík <[email protected]> wrote:
> >
> > On Fri, 3 May 2024 18:35:26 +0000
> > "T.J. Mercier" <[email protected]> wrote:
> >
> > > IOMMU implementations now sometimes bounce memory through SWIOTLB to
> > > achieve cacheline alignment [1], or prevent DMA attacks by untrusted
> > > devices [2]. These uses of SWIOTLB differ conceptually from historical
> > > use which was a solution to the problem of device addressing
> > > limitations that prevent DMA to some portion of system memory
> > > (typically beyond 4 GiB). IOMMUs also solve the problem of device
> > > addressing limitations and therefore eliminate the need for SWIOTLB for
> > > that purpose. However as mentioned, IOMMUs can use SWIOTLB for other
> > > purposes.
> > >
> > > The swiotlb kernel command line parameter does not impact IOMMU related
> > > use of SWIOTLB, and that is intentional. IOMMUs cannot be forced to use
> > > SWIOTLB for all buffers. Update the documentation for the swiotlb
> > > parameter to clarify that SWIOTLB use can only be forced in scenarios
> > > where an IOMMU is not involved.
> > >
> > > [1] https://lore.kernel.org/all/[email protected]/
> > > [2] https://lore.kernel.org/all/[email protected]/
> > > Signed-off-by: T.J. Mercier <[email protected]>
> > > ---
> > > Documentation/admin-guide/kernel-parameters.txt | 1 +
> > > Documentation/arch/x86/x86_64/boot-options.rst | 2 +-
> > > 2 files changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > > index 213d0719e2b7..84c582ac246c 100644
> > > --- a/Documentation/admin-guide/kernel-parameters.txt
> > > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > > @@ -6486,6 +6486,7 @@
> > > to a power of 2.
> > > force -- force using of bounce buffers even if they
> > > wouldn't be automatically used by the kernel
> > > + where a hardware IOMMU is not involved
> > > noforce -- Never use bounce buffers (for debugging)
> > >
> > > switches= [HW,M68k,EARLY]
> >
> > Yes, this part is correct. SWIOTLB cannot be forced if there is an IOMMU.
> >
> > > diff --git a/Documentation/arch/x86/x86_64/boot-options.rst b/Documentation/arch/x86/x86_64/boot-options.rst
> > > index 137432d34109..066b4bc81583 100644
> > > --- a/Documentation/arch/x86/x86_64/boot-options.rst
> > > +++ b/Documentation/arch/x86/x86_64/boot-options.rst
> > > @@ -285,7 +285,7 @@ iommu options only relevant to the AMD GART hardware IOMMU:
> > > Always panic when IOMMU overflows.
> > >
> > > iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
> > > -implementation:
> > > +implementation where a hardware IOMMU is not involved:
> > >
> > > swiotlb=<slots>[,force,noforce]
> > > <slots>
> >
> > But this part needs some improvement. The "swiotlb" option is not
> > entirely ignored if there is a hardware IOMMU. For example, the size of
> > the SWIOTLB can be adjusted using "swiotlb=<slots>", and since SWIOTLB
> > can be used by IOMMUs for other purposes (as you correctly note in the
> > commit message), this setting is relevant even where a hardware IOMMU
> > is involved.
> >
> > Petr T
>
> Thanks. I think I should also update the commit message:
> "The swiotlb=force kernel command line parameter does not impact IOMMU
> related use of SWIOTLB"
> and title:
> "Clarify swiotlb=force option applies only to dma-direct"
>
> As for fixing boot-options.txt, I think it makes the most sense to
> expand on just the force option rather than the section summary like
> above:
> force
> Force all IO through the software TLB.
> + Hardware IOMMU implementations can use SWIOTLB bounce buffering in
> + some circumstances, but they cannot be forced to always use them, so
> + this option only has an effect when no hardware IOMMU is involved.
> noforce
Note also that the documentation for swiotlb= in boot-options.rst is somewhat
out-of-date. It doesn't have the optional second integer parameter to specify
the number of "areas" that have their own lock. Perhaps that could be fixed
at the same time?
Michael
On Mon, May 6, 2024 at 12:58 PM Michael Kelley <[email protected]> wrote:
>
> From: T.J. Mercier <[email protected]> Sent: Monday, May 6, 2024 10:23 AM
> >
> > On Sat, May 4, 2024 at 1:16 AM Petr Tesařík <[email protected]> wrote:
> > >
> > > On Fri, 3 May 2024 18:35:26 +0000
> > > "T.J. Mercier" <[email protected]> wrote:
> > >
> > > > IOMMU implementations now sometimes bounce memory through SWIOTLB to
> > > > achieve cacheline alignment [1], or prevent DMA attacks by untrusted
> > > > devices [2]. These uses of SWIOTLB differ conceptually from historical
> > > > use which was a solution to the problem of device addressing
> > > > limitations that prevent DMA to some portion of system memory
> > > > (typically beyond 4 GiB). IOMMUs also solve the problem of device
> > > > addressing limitations and therefore eliminate the need for SWIOTLB for
> > > > that purpose. However as mentioned, IOMMUs can use SWIOTLB for other
> > > > purposes.
> > > >
> > > > The swiotlb kernel command line parameter does not impact IOMMU related
> > > > use of SWIOTLB, and that is intentional. IOMMUs cannot be forced to use
> > > > SWIOTLB for all buffers. Update the documentation for the swiotlb
> > > > parameter to clarify that SWIOTLB use can only be forced in scenarios
> > > > where an IOMMU is not involved.
> > > >
> > > > [1] https://lore.kernel.org/all/[email protected]/
> > > > [2] https://lore.kernel.org/all/[email protected]/
> > > > Signed-off-by: T.J. Mercier <[email protected]>
> > > > ---
> > > > Documentation/admin-guide/kernel-parameters.txt | 1 +
> > > > Documentation/arch/x86/x86_64/boot-options.rst | 2 +-
> > > > 2 files changed, 2 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > > > index 213d0719e2b7..84c582ac246c 100644
> > > > --- a/Documentation/admin-guide/kernel-parameters.txt
> > > > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > > > @@ -6486,6 +6486,7 @@
> > > > to a power of 2.
> > > > force -- force using of bounce buffers even if they
> > > > wouldn't be automatically used by the kernel
> > > > + where a hardware IOMMU is not involved
> > > > noforce -- Never use bounce buffers (for debugging)
> > > >
> > > > switches= [HW,M68k,EARLY]
> > >
> > > Yes, this part is correct. SWIOTLB cannot be forced if there is an IOMMU.
> > >
> > > > diff --git a/Documentation/arch/x86/x86_64/boot-options.rst b/Documentation/arch/x86/x86_64/boot-options.rst
> > > > index 137432d34109..066b4bc81583 100644
> > > > --- a/Documentation/arch/x86/x86_64/boot-options.rst
> > > > +++ b/Documentation/arch/x86/x86_64/boot-options.rst
> > > > @@ -285,7 +285,7 @@ iommu options only relevant to the AMD GART hardware IOMMU:
> > > > Always panic when IOMMU overflows.
> > > >
> > > > iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
> > > > -implementation:
> > > > +implementation where a hardware IOMMU is not involved:
> > > >
> > > > swiotlb=<slots>[,force,noforce]
> > > > <slots>
> > >
> > > But this part needs some improvement. The "swiotlb" option is not
> > > entirely ignored if there is a hardware IOMMU. For example, the size of
> > > the SWIOTLB can be adjusted using "swiotlb=<slots>", and since SWIOTLB
> > > can be used by IOMMUs for other purposes (as you correctly note in the
> > > commit message), this setting is relevant even where a hardware IOMMU
> > > is involved.
> > >
> > > Petr T
> >
> > Thanks. I think I should also update the commit message:
> > "The swiotlb=force kernel command line parameter does not impact IOMMU
> > related use of SWIOTLB"
> > and title:
> > "Clarify swiotlb=force option applies only to dma-direct"
> >
> > As for fixing boot-options.txt, I think it makes the most sense to
> > expand on just the force option rather than the section summary like
> > above:
> > force
> > Force all IO through the software TLB.
> > + Hardware IOMMU implementations can use SWIOTLB bounce buffering in
> > + some circumstances, but they cannot be forced to always use them, so
> > + this option only has an effect when no hardware IOMMU is involved.
> > noforce
>
> Note also that the documentation for swiotlb= in boot-options.rst is somewhat
> out-of-date. It doesn't have the optional second integer parameter to specify
> the number of "areas" that have their own lock. Perhaps that could be fixed
> at the same time?
>
> Michael
Thanks, I could add this as a second patch.
I also noticed that several of the iommu options (soft, allowed,
nofullflush, panic) are not listed in the set of possible options at
the top of the IOMMU section. Rebooting is the only other section that
does that, so I'm wondering if we're better off getting rid of it to
be like other sections.