Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <67ca2485-8a87-f3e6-8eb9-6dfd6ebc0017@intel.com>
Date:   Fri, 11 Aug 2023 09:12:25 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.13.0
Subject: Re: [PATCH 0/5] Share sva domains with all devices bound to a mm
Content-Language: en-US
To:     Jason Gunthorpe <jgg@ziepe.ca>,
        "Tian, Kevin" <kevin.tian@intel.com>
CC:     Lu Baolu <baolu.lu@linux.intel.com>,
        Michael Shavit <mshavit@google.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <20230808074944.7825-1-tina.zhang@intel.com>
 <BN9PR11MB527638E7361714F8AA5FD4448C12A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <bfc2eb22-54af-dab8-72e3-a142470590c3@intel.com>
 <BN9PR11MB52760CE856F69A3FFD7EE2C88C13A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZNUQA01c8zIvHBQn@ziepe.ca>
From:   Tina Zhang <tina.zhang@intel.com>
In-Reply-To: <ZNUQA01c8zIvHBQn@ziepe.ca>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WlYvdHl0UFNQaG5hK2dqcFlkejJubGZiTUJ4cmxiSHp3VXJvMDNYRFFRUG1Y?=
 =?utf-8?B?WllYR2ZsYXVzTzkzeXlLeWZkd0lkdlM4Z0dlTEx0SmhTbHpDVndmQmxkenlD?=
 =?utf-8?B?S2NNVEtTZ2tiTlltMjgraTVZK3BhaWtyTGpLdEZ0cGNaS0x1VXBIQW9pdllu?=
 =?utf-8?B?Zk5NM3AzTkgvSE9RU3J1dHFxWW1MYWFOQVRrOEZhMlhUZEgxRjU5N3pqa1kw?=
 =?utf-8?B?dE5hdEFPRWhtSkZudElpZ1V5cFRxSVh6YVpQSmw0R1orRjN0NlFFZzBBS1da?=
 =?utf-8?B?ZytuSzhxZG1NNnVtcjd3SVNrYyttNE9QTGlGZUJOSi9HSVQ4bkhKZEdRcEVK?=
 =?utf-8?B?ekMxbFpFTS8vZmVpVzV5RzRPVnhjMDJxcVlIUUNBK0NaN3pCWC9UK0sxQkpv?=
 =?utf-8?B?bVpDRy9kMm9TQXlkc3NIVW0wWk9ycHhTV2tIdEVydWxLSFdFeDg4Y1MrSDVI?=
 =?utf-8?B?U0NlZ0ZFSTc0bml5UFFWaXBTdVd3VU9JcU5ZakNZTzgzeFRlcWpFeWpxVnV5?=
 =?utf-8?B?ZldWWkhBZWtvV2RvdEJkWGVwZUJMSDE5R2U5NVk2Vzc1TURSZEJwSmxyT1d3?=
 =?utf-8?B?TEZlZEtjOTFJcE5mdnZGd20vaFFNcnVjcXFka3BUUEVrRXFLcm9lb1BKRFY0?=
 =?utf-8?B?bHB5WVg3YU92TDF2bzR5UkZyMjdJdnFKbVZKN1lRWlgvN1lCSk9DVlhlOUZy?=
 =?utf-8?B?a1pGN3JPM2hnNlYyR25DL3EvZ21UK3BRVTRMZVFpMXBsNnNwU0F4M0M4MXlV?=
 =?utf-8?B?UDNTa29naFFLcUtyREw2T2NpVEgxYmlzKzBBNzh1cThweUk5eGpkK3FMZ29H?=
 =?utf-8?B?c2d2SC9hN2tqZUZmRHlYQ0lCU21tMG5TMlRmMzRnSGc5NXM2bHBvaU1Zcitw?=
 =?utf-8?B?cFl3aWxobW40M3g2S3NINWkzaklnalp4TGVUZTdhV01yK2J4b3B2cFp0Z3A3?=
 =?utf-8?B?SnN6Sm5LT0d6eURGaitHUnNkL2xybUNVK2g1c1dxMW9nNDZBdWhJcTdrekhu?=
 =?utf-8?B?T3Nkb28xcVA1eUVxYjNXZFBHODFxTHpmamJHUXprOGF3bEFKb3lWOHYxb1FT?=
 =?utf-8?B?WHAzVEVCMXZrTWtNcjZSTzl6UkVxbGhDbW1HQXpIU1NCY1JSM1ZydlVmYVVr?=
 =?utf-8?B?SDFtNzZmOCszZC8yT29nM3QyU08yKy9hTCs4YXBwb0FuNno1RHJubXVBNmxL?=
 =?utf-8?B?amd0b0JZb1RsMzFVdVlKQy94ak9tYzVKbGlPT1J1T0ZqbHV2aW5ZUGZCSmlu?=
 =?utf-8?B?NHNuakRyZkZsa1hQQVNEKy9tRnBjdS93ajJoK2ZuOThzMDliMmg3MkhFakE4?=
 =?utf-8?B?bkZzSlVTa0ExUllVVm5aMlpzVDJnaWUvcVdrdkhRVERqRmpKVDhGUlZmTnNy?=
 =?utf-8?B?Zi9ycnYxOVNoSGdiQW5SUXZmTmtaTHhNQ1gyY0tIdFIza1FuaVRBbHNnVzhm?=
 =?utf-8?B?My9YOHB5UHRMckpWcm1xRjEwaUV1NXU2bG5LWnFJL2gxQVJmRDRHZFdURlE3?=
 =?utf-8?B?YnpZd3lwNGVnUmVsRmhocnE5SGVsNkNna2p2Smo4a3IvaWVyRm1kcmt6T0dF?=
 =?utf-8?B?YzVmQzFUSkhuZHgra2NDTkxIT3ZCVElxRDhUekw0OW9hQm5kREl3VGJEcHln?=
 =?utf-8?B?VHhtTnNFeW8zWmJPZkZyNldBYzJJOHBibGlTcmdIazY0YlRzbkZZeDNDRjRR?=
 =?utf-8?B?ejZHaUN3VUsyZUF1Wnk3Y3A3M3M1dHcxM1BLNFpYaW43QWNYT0Vnd2NITGJh?=
 =?utf-8?B?TktodHEvWHZWU3VScitzUjg0c0w4MG9NQVFnT1VhWG9OWGFEK1JjOE1SZldm?=
 =?utf-8?B?SXJxeWp1WXZ0eEhsbEozc2UwbDdtQTdZSWJHY1NPWmFPOVVDbmlyN25hdUN5?=
 =?utf-8?B?bEI3aDdVZEhuUWw4UFVIaUpGWVhpUXNNSWhZbTM4SzdHNDRaMTdabUplR21J?=
 =?utf-8?B?amEzVGpBREV3NCtoa2NPTlV0bEZjdHE1OGRMT1NlWENzWmtraGJNMGIxa0dn?=
 =?utf-8?B?Q1RHdFRtVldTaitmTURYRWg3TGJjYTNldlNNeFNJQ3J1a09CUUY4c041R2sw?=
 =?utf-8?B?cFNnOTFacWdHcGFrNkJlMzdnOTBoK0lUY0lBQ09WcGJaWE8rNGVmc2ZlS3Jx?=
 =?utf-8?B?SStad1o5MldHaE5xUURMeDJyOEg5dzNmWEl1ZU5iRTNDcVpZVFRiZHhBbk0y?=
 =?utf-8?B?VGc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 011109b6-89c5-4a0d-1798-08db9a080b88
X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5881.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Aug 2023 01:12:35.0213
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7f3A3jjWGeyezaT04zjccPm/Kf1G2h4LZGIVocbmc6Bi06oY0yEDOd4B7yQDRcfr+JpERqZY3iUKAiBGl8ZWNQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6032
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi,

On 8/11/23 00:27, Jason Gunthorpe wrote:
> On Thu, Aug 10, 2023 at 07:49:11AM +0000, Tian, Kevin wrote:
>>> From: Zhang, Tina <tina.zhang@intel.com>
>>> Sent: Thursday, August 10, 2023 9:32 AM
>>>
>>> Hi,
>>>
>>> On 8/9/23 17:41, Tian, Kevin wrote:
>>>>> From: Zhang, Tina <tina.zhang@intel.com>
>>>>> Sent: Tuesday, August 8, 2023 3:50 PM
>>>>>
>>>>> A sva domain's lifetime begins with binding a device to a mm and ends
>>>>> by releasing all the bound devices from that sva domain. Technically,
>>>>> there could be more than one sva domain identified by the mm PASID for
>>>>> the use of bound devices issuing DMA transactions.
>>>>
>>>> Could you elaborate it with some concrete examples which motivate
>>>> this change?
>>> The motivation is to remove the superfluous IOTLB invalidation in
>>> current VT-d driver.
>>>
>>> Currently, in VT-d driver, due to lacking shared sva domain info, in
>>> intel_flush_svm_range(), both iotlb and dev-tlb invalidation operations
>>> are performed per-device. However, difference devices could be behind
>>> one IOMMU (e.g., four devices are behind one IOMMU) and invoking iotlb
>>> per-device gives us more iotlb invalidation than necessary (4 iotlb
>>> invalidation instead of 1). This issue may give more performance impact
>>> when in a virtual machine guest, as currently we have one virtual VT-d
>>> for in front of those virtual devices.
>>>
>>>
>>> This patch fixes this issue by attaching shared sva domain information
>>> to mm, so that it can be utilized in the mm_notifier_ops callbacks.
>>>
>>
>> that is one of the motivations. e.g. another one as Jason suggested
>> is to cleanup to decouple the common sva logic from enqcmd. Both
>> should be mentioned in next version cover letter.
> 
> I also want to purge all the de-duplication and refcounting code
> around mm's and sva_binds from the drivers. Eg see the mess this makes
> of SMMUv3.
> 
> Core code provides a single iommu_domain per-mm for SVA. Driver can
> rely on this optimization and does not need to de-duplicate.
> 
> Single domain tracks all attachments. Driver can optimize using that
> information by de-duplicating (eg ASID invalidation vs ATC
> invalidation)
> 
> After this we need to fix the domain allocation op to add a
> 'alloc_domain_sva(dev, mm_struct)' op so that the drivers can setup
> their SVA domains fully in a nice lock-safe environment.
Agree. These can be added to the to-do list.

Regards,
-Tina
> 
> Jason