From: Eric Auger <eric.auger@linaro.org>
Subject: Re: [PATCH v6 0/7] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part
 1/3: iommu changes
To: Alex Williamson <alex.williamson@redhat.com>
References: <1459757222-2668-1-git-send-email-eric.auger@linaro.org>
 <20160406171534.794c6824@t450s.home>
Cc: eric.auger@st.com, robin.murphy@arm.com, will.deacon@arm.com,
        joro@8bytes.org, tglx@linutronix.de, jason@lakedaemon.net,
        marc.zyngier@arm.com, christoffer.dall@linaro.org,
        linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu,
        kvm@vger.kernel.org, suravee.suthikulpanit@amd.com, patches@linaro.org,
        linux-kernel@vger.kernel.org, Manish.Jaggi@caviumnetworks.com,
        Bharat.Bhushan@freescale.com, pranav.sawargaonkar@gmail.com,
        p.fedin@samsung.com, iommu@lists.linux-foundation.org,
        Jean-Philippe.Brucker@arm.com, julien.grall@arm.com
Message-ID: <5706528B.2010906@linaro.org>
Date: Thu, 7 Apr 2016 14:28:59 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
In-Reply-To: <20160406171534.794c6824@t450s.home>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 6095
Lines: 143

Hi Alex,
On 04/07/2016 01:15 AM, Alex Williamson wrote:
> On Mon,  4 Apr 2016 08:06:55 +0000
> Eric Auger <eric.auger@linaro.org> wrote:
> 
>> This series introduces the dma-reserved-iommu api used to:
>> - create/destroy an iova domain dedicated to reserved iova bindings
>> - map/unmap physical addresses onto reserved IOVAs.
>> - unmap and destroy all IOVA reserved bindings
> 
> Why are we making the decision to have an unbalanced map vs unmap, we
> can create individual mappings, but only unmap the whole thing and
> start over?  That's a strange interface.  Thanks,
The "individual" balanced unmap also exists (iommu_put_reserved_iova)
and this is the "normal" path. This happens on msi_domain_deactivate
(and possibly on msi_domain_set_affinity).

I added iommu_unmap_reserved to handle the case where the userspace
registers a reserved iova domain and fails to unregister it. In that
case one need to handle the cleanup on kernel-side and I chose to
implement this on vfio_iommu_type1 release. All the reserved IOMMU
bindings get destroyed on that event.

Any advice to handle this situation?

Best Regards

Eric

> 
> Alex
>  
>> Currently reserved IOVAs are meant to map MSI physical doorbells. A single
>> reserved domain does exit per domain.
>>
>> Also a new domain attribute is introduced to signal whether the MSI
>> addresses must be mapped in the IOMMU
>>
>> VFIO subsystem is supposed to create/destroy the iommu reserved domain.
>>
>> When the MSI sub-system is about to handle an MSI physical address
>> that needs to be bound, it uses the dma-reserved_iommu API to map/unmap
>> the address. Since several drivers are likely to use the same doorbell,
>> a reference counting must exist on the bindings. An RB-tree indexed by PA
>> is used.
>>
>> More details & context can be found at:
>> http://www.linaro.org/blog/core-dump/kvm-pciemsi-passthrough-armarm64/
>>
>> Best Regards
>>
>> Eric
>>
>> Git: complete series available at
>> https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.6-rc1-pcie-passthrough-v6
>>
>> History:
>>
>> RFC v5 -> patch v6:
>> - split to ease the review process
>> - in dma-reserved-api use a spin lock instead of a mutex (reported by
>>   Jean-Philippe)
>> - revisit iommu_get_reserved_iova API to pass a size parameter upon
>>   Marc's request
>> - Consistently use the page order passed when creating the iova domain.
>> - init reserved_binding_list (reported by Julien)
>>
>> RFC v4 -> RFC v5:
>> - take into account Thomas' comments on MSI related patches
>>   - split "msi: IOMMU map the doorbell address when needed"
>>   - increase readability and add comments
>>   - fix style issues
>>  - split "iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute"
>>  - platform ITS now advertises IOMMU_CAP_INTR_REMAP
>>  - fix compilation issue with CONFIG_IOMMU API unset
>>  - arm-smmu-v3 now advertises DOMAIN_ATTR_MSI_MAPPING
>>
>> RFC v3 -> v4:
>> - Move doorbell mapping/unmapping in msi.c
>> - fix ref count issue on set_affinity: in case of a change in the address
>>   the previous address is decremented
>> - doorbell map/unmap now is done on msi composition. Should allow the use
>>   case for platform MSI controllers
>> - create dma-reserved-iommu.h/c exposing/implementing a new API dedicated
>>   to reserved IOVA management (looking like dma-iommu glue)
>> - series reordering to ease the review:
>>   - first part is related to IOMMU
>>   - second related to MSI sub-system
>>   - third related to VFIO (except arm-smmu IOMMU_CAP_INTR_REMAP removal)
>> - expose the number of requested IOVA pages through VFIO_IOMMU_GET_INFO
>>   [this partially addresses Marc's comments on iommu_get/put_single_reserved
>>    size/alignment problematic - which I did not ignore - but I don't know
>>    how much I can do at the moment]
>>
>> RFC v2 -> RFC v3:
>> - should fix wrong handling of some CONFIG combinations:
>>   CONFIG_IOVA, CONFIG_IOMMU_API, CONFIG_PCI_MSI_IRQ_DOMAIN
>> - fix MSI_FLAG_IRQ_REMAPPING setting in GICv3 ITS (although not tested)
>>
>> PATCH v1 -> RFC v2:
>> - reverted to RFC since it looks more reasonable ;-) the code is split
>>   between VFIO, IOMMU, MSI controller and I am not sure I did the right
>>   choices. Also API need to be further discussed.
>> - iova API usage in arm-smmu.c.
>> - MSI controller natively programs the MSI addr with either the PA or IOVA.
>>   This is not done anymore in vfio-pci driver as suggested by Alex.
>> - check irq remapping capability of the group
>>
>> RFC v1 [2] -> PATCH v1:
>> - use the existing dma map/unmap ioctl interface with a flag to register a
>>   reserved IOVA range. Use the legacy Rb to store this special vfio_dma.
>> - a single reserved IOVA contiguous region now is allowed
>> - use of an RB tree indexed by PA to store allocated reserved slots
>> - use of a vfio_domain iova_domain to manage iova allocation within the
>>   window provided by the userspace
>> - vfio alloc_map/unmap_free take a vfio_group handle
>> - vfio_group handle is cached in vfio_pci_device
>> - add ref counting to bindings
>> - user modality enabled at the end of the series
>>
>>
>> Eric Auger (7):
>>   iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute
>>   iommu/arm-smmu: advertise DOMAIN_ATTR_MSI_MAPPING attribute
>>   iommu: introduce a reserved iova cookie
>>   dma-reserved-iommu: alloc/free_reserved_iova_domain
>>   dma-reserved-iommu: reserved binding rb-tree and helpers
>>   dma-reserved-iommu: iommu_get/put_single_reserved
>>   dma-reserved-iommu: iommu_unmap_reserved
>>
>>  drivers/iommu/Kconfig              |   8 +
>>  drivers/iommu/Makefile             |   1 +
>>  drivers/iommu/arm-smmu-v3.c        |   2 +
>>  drivers/iommu/arm-smmu.c           |   2 +
>>  drivers/iommu/dma-reserved-iommu.c | 321 +++++++++++++++++++++++++++++++++++++
>>  drivers/iommu/iommu.c              |   2 +
>>  include/linux/dma-reserved-iommu.h |  80 +++++++++
>>  include/linux/iommu.h              |   7 +
>>  8 files changed, 423 insertions(+)
>>  create mode 100644 drivers/iommu/dma-reserved-iommu.c
>>  create mode 100644 include/linux/dma-reserved-iommu.h
>>
>