Received-SPF: pass (google.com: domain of linux-kernel+bounces-164929-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Wed, 1 May 2024 06:49:10 +0200
From: Christoph Hellwig <hch@lst.de>
To: Petr =?utf-8?B?VGVzYcWZw61r?= <petr@tesarici.cz>
Cc: mhkelley58@gmail.com, mhklinux@outlook.com, robin.murphy@arm.com,
	joro@8bytes.org, will@kernel.org, hch@lst.de,
	m.szyprowski@samsung.com, corbet@lwn.net, iommu@lists.linux.dev,
	linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
	roberto.sassu@huaweicloud.com
Subject: Re: [PATCH v3 1/1] Documentation/core-api: Add swiotlb
 documentation
Message-ID: <20240501044910.GD31252@lst.de>
References: <20240429151337.1069470-1-mhklinux@outlook.com> <20240430132413.69797af1@mordecai>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20240430132413.69797af1@mordecai>
User-Agent: Mutt/1.5.17 (2007-11-01)

On Tue, Apr 30, 2024 at 01:24:13PM +0200, Petr Tesařík wrote:
> > +swiotlb was originally created to handle DMA for devices with addressing
> > +limitations. As physical memory sizes grew beyond 4 GiB, some devices could
> > +only provide 32-bit DMA addresses. By allocating bounce buffer memory below
> > +the 4 GiB line, these devices with addressing limitations could still work and
> > +do DMA.
> 
> IIRC the origins are even older and bounce buffers were used to
> overcome the design flaws inherited all the way from the original IBM
> PC.

[correct, but for swiotlb largely irrelevant PC addressing bits snipped]

swiotlb was added with the merge of the ia64 port to address 32-bit
addressing limitations.  The 1MB addressing limitations of the PC did
and still do of course exist, but weren't dealt with in any coherent
fashion, and still aren't.  Swiotlb isn't related to them.

> > +data to/from the original target memory buffer. The CPU copying bridges between
> > +the unencrypted and the encrypted memory. This use of bounce buffers allows
> > +existing device drivers to "just work" in a CoCo VM, with no modifications
> > +needed to handle the memory encryption complexity.
> 
> This part might be misleading. It sounds as if SWIOTLB would not be
> needed if drivers were smarter. But IIUC that's not the case. SWIOTLB
> is used for streaming DMA, where device drivers have little control
> over the physical placement of a DMA buffer. For example, when a
> process allocates some memory, the kernel cannot know that this memory
> will be later passed to a write(2) syscall to do direct I/O of a
> properly aligned buffer that can go all the way down to the NVMe driver
> with zero copy.

I think the statement in the text is fine and easy to understand.  CoCo
drivers could instead always map the memory unencrypted (which would have
no so nice security and performance properties) or use fixed ringbuffers
in shared unencrypted memory (which would require a different driver
architecture).

> > +block. Hence the default memory pool for swiotlb allocations must be
> > +pre-allocated at boot time (but see Dynamic swiotlb below). Because swiotlb
> > +allocations must be physically contiguous, the entire default memory pool is
> > +allocated as a single contiguous block.
> 
> Allocations must be contiguous in target device's DMA address space. In
> practice this is achieved by being contiguous in CPU physical address
> space (aka "physically contiguous"), but there might be subtle
> differences, e.g. in a virtualized environment.
> 
> Now that I'm thinking about it, leave the paragraph as is, and I'll
> update it if I write the code for it.

Heh.  The only think making cpu non-contiguous address space contiguous
for a device is an iommu.  And when we have that we only use swiotlb
for unaligned iommu pages, so I'm not sure how we'd ever get there.