Received-SPF: pass (google.com: domain of linux-crypto-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <5771ea99-eef7-7321-dd67-4c42c0cbb721@amd.com>
Date:   Fri, 23 Dec 2022 17:45:24 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.5.0
Subject: Re: [PATCH v2] crypto: ccp - Allocate TEE ring and cmd buffer using
 DMA APIs
To:     Jeremi Piotrowski <jpiotrowski@linux.microsoft.com>
Cc:     Tom Lendacky <thomas.lendacky@amd.com>,
        John Allen <john.allen@amd.com>,
        Herbert Xu <herbert@gondor.apana.org.au>,
        "David S . Miller" <davem@davemloft.net>,
        Sumit Semwal <sumit.semwal@linaro.org>,
        =?UTF-8?Q?Christian_K=c3=b6nig?= <christian.koenig@amd.com>,
        linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org,
        linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org,
        linaro-mm-sig@lists.linaro.org,
        Mythri PK <Mythri.Pandeshwarakrishna@amd.com>,
        Jeshwanth <JESHWANTHKUMAR.NK@amd.com>,
        Devaraj Rangasamy <Devaraj.Rangasamy@amd.com>,
        stable@vger.kernel.org, Jens Wiklander <jens.wiklander@linaro.org>
References: <651349f55060767a9a51316c966c1e5daa57a644.1670919979.git.Rijo-john.Thomas@amd.com>
 <20221215132917.GA11061@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
Content-Language: en-US
From:   Rijo Thomas <Rijo-john.Thomas@amd.com>
In-Reply-To: <20221215132917.GA11061@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?R3k3eHF5K2N3Nmgvb29xWlJFMUlDZ1JoVEZXei9CNTlPWXdtYWdyRnAwRE0z?=
 =?utf-8?B?U0FISTgwdEJKMjNpOWhsSFJrMWViNElxQ3N1d2UwbzhEanVhV2xiSytzTGlx?=
 =?utf-8?B?Y3JTYm9RS0Z0bldUVjNBcEZBcFNNUmtZcU1BUGxpZGFhbkJXV0FLMHlhOEFx?=
 =?utf-8?B?ME8yZkNPbWQ2anJWWTFwaVBaNVRGalVWWTJYeDF3Nmg5VnMvaEFOMzd2VW02?=
 =?utf-8?B?ZkRzWXJxSVpQeFlNdW5ta0RMVG43QTZLcE8wamsxcWxmUlFYNHFaRjNzRHV1?=
 =?utf-8?B?NjVYdHpwclZTQmo1MDNpRmUyVkFqeGNqVXcwKzdUZ0EyT1VLUkRoTHczaEpn?=
 =?utf-8?B?R0JpdEM1QS9IbTRyc29FQitoUXRiQy82bXhPWDlkSSt5NWNUa3NVMVJlcE40?=
 =?utf-8?B?cFd4bDBHYTJ2SXF1VmlTa3hCOUM1MnhnclYvZEhWNDY4azlHRTZWcHhjOTI1?=
 =?utf-8?B?TzhmRmZoL1BubVRkZUdQVW95MytmYllpbWJYeTM0Y2J4U2trbDIrcTgxWUZu?=
 =?utf-8?B?d0NtMUp6NnBUSDdhd2ZGZUJxSGUyaVlyL0VwSjkwYVliNDY5Y0Q3TGNXeGM2?=
 =?utf-8?B?Zkkyek5JUzJ6TVViNHY0MW1PSjQ0SFN2eXVZSTg4eVV1UFJlK3YrSGFRRzhC?=
 =?utf-8?B?Vno5VThIQ0x3WDZYYk84K1lUcGg0OGdLRGQ1eEh3cTJDemRGNTVXKzlmWVFT?=
 =?utf-8?B?aW10WW1YOFBsN0UrVFgreTVqTEZVVi9lRmJ0bzAxZndmMFZNQjdLMzVJQnF6?=
 =?utf-8?B?SUVraC83aGdtenQyRi9EcjRYbjRmMGRySHVzUHlRdWlVOWpFd0xjay9xcklt?=
 =?utf-8?B?dFlBaHBkbUNwY1MweE4zWDIxWUt0ejVsblJ1QTFJSXpxUjZoUy9DWnFOckZK?=
 =?utf-8?B?OWVpYU1HendEcmNxbGQ1VVMvMEdnaU94emhvWnQ1cm5veUlJQStqMmtTRG1n?=
 =?utf-8?B?VXR5QTJKMVV3SlZsdEpVb1cxZEFJK0svQnFGME1XcERiaktMZFEvR2hiSDZL?=
 =?utf-8?B?MXZ0TEwzS0RDOERJODdiYUtmcVU3WFJtb1VNM1RORVQ0K2ZwUmhoSmlPTjZw?=
 =?utf-8?B?a2VHNWJEdDh4RVMzaHI5L1VnRml2N0pnSWFmVHVvMVYyYU5ZdGF0cktYR1VK?=
 =?utf-8?B?ajZqUkZiYjhEMGFwYkVtOWVNQXAvc2xVTUZkM1piSzJQeGRTNXpUV09kS1lu?=
 =?utf-8?B?ekw0amRvb0VMWHF3cDNLR1R6Z01ucHd0ZDc1UVpGWmlIVTRnbmNwOEZHNDh3?=
 =?utf-8?B?NE1RdEtvT2lLR2xXVEdZWWpFaVJnc1F5SzhieGs4K2pLcmpEY2MvN2NFaHVD?=
 =?utf-8?B?QnhCRUxxKzRWTFJCMlcwWVRvQmszZkNxeTVhQ1dsWHVrOStndWZaaXNwK1RF?=
 =?utf-8?B?RkttVTl4cnFES2dEd3pOektSUXQrM1YzNjZMR29uR2FtSExXejdOR1ZWMlRx?=
 =?utf-8?B?ckdHSExkL0JnYnFIdElCNW9TcjVESTQvdTQ2TjJNb3VsT3hJRzlCeXU3UXRN?=
 =?utf-8?B?QzNkZWRNcW1ON014Z210WEZrSWJxbUwydjVpSXdNV1Z1V3BVaWFScWplZjBy?=
 =?utf-8?B?VzJTQnZieDFxQThqM0I2aFZZK1lGNTNCanREcHQxWEFXR2l0dnJrQStKSWJn?=
 =?utf-8?B?K2JYMG1qclo1c211WW1IdmVPakxiVUJJNlBDRll0ZVhZV3FSaWV0SW9XVGtx?=
 =?utf-8?B?ZmtVak1Pci93elVtZ090elZ1alFXa1NUVnBFWGVjVTFBeHdWWTFzRDFmbEZB?=
 =?utf-8?B?TndJWnUrVVluWkdKTkp6Sm1kN094YTJEZ0VjdmxQRm9vblRORUYzSUlKc25N?=
 =?utf-8?B?aGkvdTlIZXVqWjd4dHVjV083MVVmQmJNMFZmQ3piU2FBMnRTcUh5R0kxWkdH?=
 =?utf-8?B?TmR2Wkl0cjRENlFkakhUc3hMUS9OLzl4dDBWd3Q5TGRoc1djOTcvYkQ2TTdi?=
 =?utf-8?B?Sy9QbWlxN1lrZllyQWkrRDB2MTB3S0QzbHdKTVd4TjhTcXhEa2NRYjEyZmhE?=
 =?utf-8?B?N1pZclpaUS95LzhrVFpXamE4Qk40elBrR3g2VTYveFZBaStLQXRsSTJ4WmQx?=
 =?utf-8?B?OXJzUTdpZ2xZTUdzazlaNjUraS9VNDJ6cWUzcTFibUQ3TWo2N0N5SXY3QjJI?=
 =?utf-8?Q?qhLAIG9C56ZDHGMt6ULD+KEM2?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 24867849-0cad-4b78-00e8-08dae4df66c8
X-MS-Exchange-CrossTenant-AuthSource: CH0PR12MB5346.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Dec 2022 12:15:38.4256
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: nqAa8Ib5d8VxlUU64jhrdTWvSbnmNtTyhlVzBM187JlcXlGPJacPSoUotbx8W+w1URJ8nJavwZE6npYhCkwBdQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7150
X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-crypto.vger.kernel.org>
X-Mailing-List: linux-crypto@vger.kernel.org


On 12/15/2022 6:59 PM, Jeremi Piotrowski wrote:
> On Tue, Dec 13, 2022 at 04:40:27PM +0530, Rijo Thomas wrote:
>> For AMD Secure Processor (ASP) to map and access TEE ring buffer, the
>> ring buffer address sent by host to ASP must be a real physical
>> address and the pages must be physically contiguous.
>>
>> In a virtualized environment though, when the driver is running in a
>> guest VM, the pages allocated by __get_free_pages() may not be
>> contiguous in the host (or machine) physical address space. Guests
>> will see a guest (or pseudo) physical address and not the actual host
>> (or machine) physical address. The TEE running on ASP cannot decipher
>> pseudo physical addresses. It needs host or machine physical address.
>>
>> To resolve this problem, use DMA APIs for allocating buffers that must
>> be shared with TEE. This will ensure that the pages are contiguous in
>> host (or machine) address space. If the DMA handle is an IOVA,
>> translate it into a physical address before sending it to ASP.
>>
>> This patch also exports two APIs (one for buffer allocation and
>> another to free the buffer). This API can be used by AMD-TEE driver to
>> share buffers with TEE.
>>
>> Fixes: 33960acccfbd ("crypto: ccp - add TEE support for Raven Ridge")
>> Cc: Tom Lendacky <thomas.lendacky@amd.com>
>> Cc: stable@vger.kernel.org
>> Signed-off-by: Rijo Thomas <Rijo-john.Thomas@amd.com>
>> Co-developed-by: Jeshwanth <JESHWANTHKUMAR.NK@amd.com>
>> Signed-off-by: Jeshwanth <JESHWANTHKUMAR.NK@amd.com>
>> Reviewed-by: Devaraj Rangasamy <Devaraj.Rangasamy@amd.com>
>> ---
>> v2:
>>  * Removed references to dma_buffer.
>>  * If psp_init() fails, clear reference to master device.
>>  * Handle gfp flags within psp_tee_alloc_buffer() instead of passing it as
>>    a function argument.
>>  * Added comments within psp_tee_alloc_buffer() to serve as future
>>    documentation.
>>
>>  drivers/crypto/ccp/psp-dev.c |  13 ++--
>>  drivers/crypto/ccp/tee-dev.c | 124 +++++++++++++++++++++++------------
>>  drivers/crypto/ccp/tee-dev.h |   9 +--
>>  include/linux/psp-tee.h      |  49 ++++++++++++++
>>  4 files changed, 142 insertions(+), 53 deletions(-)
>>
>> diff --git a/drivers/crypto/ccp/psp-dev.c b/drivers/crypto/ccp/psp-dev.c
>> index c9c741ac8442..380f5caaa550 100644
>> --- a/drivers/crypto/ccp/psp-dev.c
>> +++ b/drivers/crypto/ccp/psp-dev.c
>> @@ -161,13 +161,13 @@ int psp_dev_init(struct sp_device *sp)
>>  		goto e_err;
>>  	}
>>
>> -	ret = psp_init(psp);
>> -	if (ret)
>> -		goto e_irq;
>> -
>>  	if (sp->set_psp_master_device)
>>  		sp->set_psp_master_device(sp);
>>
>> +	ret = psp_init(psp);
>> +	if (ret)
>> +		goto e_clear;
>> +
>>  	/* Enable interrupt */
>>  	iowrite32(-1, psp->io_regs + psp->vdata->inten_reg);
>>
>> @@ -175,7 +175,10 @@ int psp_dev_init(struct sp_device *sp)
>>
>>  	return 0;
>>
>> -e_irq:
>> +e_clear:
>> +	if (sp->clear_psp_master_device)
>> +		sp->clear_psp_master_device(sp);
>> +
>>  	sp_free_psp_irq(psp->sp, psp);
>>  e_err:
>>  	sp->psp_data = NULL;
>> diff --git a/drivers/crypto/ccp/tee-dev.c b/drivers/crypto/ccp/tee-dev.c
>> index 5c9d47f3be37..5c43e6e166f1 100644
>> --- a/drivers/crypto/ccp/tee-dev.c
>> +++ b/drivers/crypto/ccp/tee-dev.c
>> @@ -12,8 +12,9 @@
>>  #include <linux/mutex.h>
>>  #include <linux/delay.h>
>>  #include <linux/slab.h>
>> +#include <linux/dma-direct.h>
>> +#include <linux/iommu.h>
>>  #include <linux/gfp.h>
>> -#include <linux/psp-sev.h>
>>  #include <linux/psp-tee.h>
>>
>>  #include "psp-dev.h"
>> @@ -21,25 +22,73 @@
>>
>>  static bool psp_dead;
>>
>> +struct psp_tee_buffer *psp_tee_alloc_buffer(unsigned long size)
>> +{
>> +	struct psp_device *psp = psp_get_master_device();
>> +	struct psp_tee_buffer *buf;
>> +	struct iommu_domain *dom;
>> +
>> +	if (!psp || !size)
>> +		return NULL;
>> +
>> +	buf = kzalloc(sizeof(*buf), GFP_KERNEL);
>> +	if (!buf)
>> +		return NULL;
>> +
>> +	/* The pages allocated for PSP Trusted OS must be physically
>> +	 * contiguous in host (or machine) address space. Therefore,
>> +	 * use DMA API to allocate memory.
>> +	 */
>> +
>> +	buf->vaddr = dma_alloc_coherent(psp->dev, size, &buf->dma,
>> +					GFP_KERNEL | __GFP_ZERO);
> 
> dma_alloc_coherent memory is just as contiguous as __get_free_pages, and
> calling dma_alloc_coherent from a guest does not guarantee that the memory is
> contiguous in host memory either. The memory would look contiguous from the
> device point of view thanks to the IOMMU though (in both cases). So this is not
> about being contiguous but other properties that you might rely on (dma mask
> most likely, or coherent if you're not running this on x86?).
> 
> Can you confirm why this fixes things and update the comment to reflect that.
> 

I see what you are saying.

We verified this in Xen Dom0 PV guest, where dma_alloc_coherent() returned a memory
that is contiguous in machine address space, and the machine address was returned
in the dma handle (buf->dma).

>> +	if (!buf->vaddr || !buf->dma) {
>> +		kfree(buf);
>> +		return NULL;
>> +	}
>> +
>> +	buf->size = size;
>> +
>> +	/* Check whether IOMMU is present. If present, convert IOVA to
>> +	 * physical address. In the absence of IOMMU, the DMA address
>> +	 * is actually the physical address.
>> +	 */
>> +
>> +	dom = iommu_get_domain_for_dev(psp->dev);
>> +	if (dom)
>> +		buf->paddr = iommu_iova_to_phys(dom, buf->dma);
>> +	else
>> +		buf->paddr = buf->dma;
> 
> This is confusing: you're storing GPA for the guest and HPA in case of the
> host, to pass to the device. Let's talk about the host case.
> 
> a) the device is behind an IOMMU. The DMA API gives you an IOVA, and the device
> should be using the IOVA to access memory (because it's behind an IOMMU).
> b) the device is not behind an IOMMU. The DMA API gives you a PA, the device
> uses a PA.
> 
> But in case a) you're extracting the PA, which means your device can bypass the
> IOMMU, in which case the system should not think that it is behind an IOMMU. So
> how does this work?
> 

PSP Trusted OS maps memory without using IOMMU (it bypasses IOMMU). Hence, we need to pass system
physical address to PSP. That's why we cannot pass IOVA or GPA to PSP.

We are re-evaluating our solution to handle other scenarios as well (not just Xen Dom0 PV).

Thanks,
Rijo

> Jeremi
> 
>> +
>> +	return buf;
>> +}
>> +EXPORT_SYMBOL(psp_tee_alloc_buffer);
>> +
>