Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Cc:     brijesh.singh@amd.com, joro@8bytes.org, Jon.Grimm@amd.com
Subject: Re: [PATCH v2] iommu/amd: Enforce 4k mapping for certain IOMMU data
 structures
To:     Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org,
        Will Deacon <will@kernel.org>
References: <20201105145832.3065-1-suravee.suthikulpanit@amd.com>
 <c189684a-27e5-c0c2-1629-063b9fb16957@amd.com>
From:   Brijesh Singh <brijesh.singh@amd.com>
Message-ID: <35c6f7d8-f889-8c3c-2e01-1a9226babf0a@amd.com>
Date:   Thu, 19 Nov 2020 22:19:01 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
 Gecko/20100101 Thunderbird/78.4.0
In-Reply-To: <c189684a-27e5-c0c2-1629-063b9fb16957@amd.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from Brijeshs-MacBook-Pro.local (70.112.153.56) by SA9PR13CA0062.namprd13.prod.outlook.com (2603:10b6:806:23::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.13 via Frontend Transport; Fri, 20 Nov 2020 04:19:02 +0000
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-HT: Tenant
X-MS-Office365-Filtering-Correlation-Id: 4ba7db2f-69eb-4dcc-c2af-08d88d0b6962
X-MS-TrafficTypeDiagnostic: SA0PR12MB4413:
X-MS-Exchange-Transport-Forked: True
X-Microsoft-Antispam-PRVS: <SA0PR12MB441333C64B650A40AA274C89E5FF0@SA0PR12MB4413.namprd12.prod.outlook.com>
X-MS-Oob-TLC-OOBClassifiers: OLM:9508;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: +0APLS97wMbKvj0UFNCBcHMHWwAfu/TQDwfh28ZYIjEnase23SFygRs1mH+j5dY3jplmnrd9BheGON4a//ifAGwldA9BFKmeJFTjyddayilwcTe/7X5JEWq9UcY+M/1Jz0shFuATcrTKB8XmcoEIms0p1n9m27BrAjfCiB0MmQRwCZXdpsQMeJq8aqRJL4rq2XYN8wczUAjAuNCUPL++NbVnsvgr+PBqrh9tHE7u8qzY+2rjHNF4d55YyAP+4WXRdF1LWhM65LSQSVDddKw5GbfUOnfA5YvGomLi2XyFzjexJ8jOlmRpb4DjbKGDsD3Xs0dhLph76rYTn/k0uGU5kJaVNT9crviIQ5D+lz481S6tH7fFD260vt7D7yeR7bYUnc4qE6Bj4rm3YJFELcsBxSPL7+5pF+nbysoQGt3iPEd0mGkAaAv7Sdj6vzRw4ssp2qOMD0ftaXE69gaUvEDGpw==
X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR12MB2718.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(396003)(136003)(366004)(376002)(346002)(2616005)(86362001)(956004)(66476007)(31686004)(44832011)(6512007)(2906002)(31696002)(66946007)(478600001)(52116002)(4326008)(53546011)(6506007)(5660300002)(36756003)(66556008)(83380400001)(26005)(16526019)(8676002)(316002)(8936002)(186003)(110136005)(6486002)(43740500002);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData: SaQCS2ZyEKo7R775N/pbl6cXunTcY/J+4m1cD7RGkKHMH+F5WL+I+ERB8WHeFl/XM8QS7U+Pd+mIMqbiKhMEr9VAJ0cidE1USC7y4VxHkpGeMfsXAKgmdshfrnPPeqEnPaENVLYStzrq1e57N9h64+zcTMoLtgFXNnpKux3LHzLhkjmqAZ9XIcKJqAc55mi9UZHkvClT6LP1hm+TPcfATfjmB7tB0n/woMpP70DJGmVRFw+yqrIaSgEpZf+kR8eDZoq5PfewcOrgV2e6vkc3H/XfqBsgPNHQyLPCv2FPULHQcwTUgmEKzx8YDlTtGj/Xtmkgrnn8+rdK/4EXpMnAuwt4+uSpJv94vdBiF8TALibz50YXk1h42BLmBTwce+LkQOqRJjP0STv0iw6YG1H84aGlzjpCPVcbm9xQ+4BZHFli/jm8jtUIm1p6W6rZk3zvQXG6OYyTtPGHWYEoqXnSCUQgZ9+c4n3rKI+D80bOAaD1kqkJI6orHCehvwib9mIUnOyTn9HDi0YApGtY8YlgHWsda+YTQpan5phgUl92WroXEL0MciGaK0tRSrrtLrbdC+CSt7Rv4wi7rnPlzX+uRZK+7mmdo04GK16H6TLP1ffOO+K97rtvA5cLE7HRtEooud+WQnOV2danlaSzuxa/gw==
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4ba7db2f-69eb-4dcc-c2af-08d88d0b6962
X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB2718.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Nov 2020 04:19:02.8869
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SL4lEg4H3QfEeWCY2VO3ns+8FN8u1mMW2rcJ5N0yC2HAfDUxz/Ks8daWdaEpllrfIDV73rS/fQtt5GSdvokjIw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4413
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 11/19/20 8:30 PM, Suravee Suthikulpanit wrote:
> Will,
>
> To answer your questions from v1 thread.
>
> On 11/18/20 5:57 AM, Will Deacon wrote:
> > On 11/5/20 9:58 PM, Suravee Suthikulpanit wrote:
> >> AMD IOMMU requires 4k-aligned pages for the event log, the PPR log,
> >> and the completion wait write-back regions. However, when allocating
> >> the pages, they could be part of large mapping (e.g. 2M) page.
> >> This causes #PF due to the SNP RMP hardware enforces the check based
> >> on the page level for these data structures.
> >
> > Please could you include an example backtrace here?
>
> Unfortunately, we don't actually have the backtrace available here.
> This information is based on the SEV-SNP specification.
>
> >> So, fix by calling set_memory_4k() on the allocated pages.
> >
> > I think I'm missing something here. set_memory_4k() will break the
> kernel
> > linear mapping up into page granular mappings, but the IOMMU isn't
> using
> > that mapping, right?
>
> That's correct. This does not affect the IOMMU, but it affects the PSP
> FW.
>
> > It's just using the physical address returned by
> iommu_virt_to_phys(), so why does it matter?
> >
> > Just be nice to capture some of this rationale in the log,
> especially as
> > I'm not familiar with this device.
>
> According to the AMD SEV-SNP white paper
> (https://www.amd.com/system/files/TechDocs/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf),
> the Reverse Map Table (RMP) contains one entry for every 4K page of
> DRAM that may be used by the VM. In this case, the pages allocated by
> the IOMMU driver are added as 4K entries in the RMP table by the
> SEV-SNP FW.
>
> During the page table walk, the RMP checks if the page is owned by the
> hypervisor. Without calling set_memory_4k() to break the mapping up
> into 4K pages, pages could end up being part of large mapping (e.g. 2M
> page), in which the page access would be denied and result in #PF.


Since the page is added as a 4K page in the RMP table by the SEV-SNP FW,
so we need to split the physmap to ensure that this page will be access
with a 4K mapping from the x86. If the page is part of large page then
write access will cause a RMP violation (i.e #PF), this is because SNP
hardware enforce that the CPU page level walk must match with page-level
programmed in the RMP table.


>
> >> Fixes: commit c69d89aff393 ("iommu/amd: Use 4K page for completion
> wait write-back semaphore")
> >
> > I couldn't figure out how that commit could cause this problem.
> Please can
> > you explain that to me?
>
> Hope this helps clarify. If so, I'll update the commit log and send
> out V3.
>
> Thanks,
> Suravee