Subject: Re: [PATCH] x86/mm: Fix PAGE_KERNEL_IO removal breakage
From:   Tom Lendacky <thomas.lendacky@amd.com>
To:     Lucas De Marchi <lucas.demarchi@intel.com>,
        Joerg Roedel <joro@8bytes.org>
Cc:     x86@kernel.org, Dave Hansen <dave.hansen@linux.intel.com>,
        Borislav Petkov <bp@alien8.de>, Ingo Molnar <mingo@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>, hpa@zytor.com,
        Andy Lutomirski <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Brijesh Singh <brijesh.singh@amd.com>,
        linux-kernel@vger.kernel.org, Joerg Roedel <jroedel@suse.de>
References: <20211202144646.23186-1-joro@8bytes.org>
 <20211202155452.jh4qnvpx52r3od67@ldmartin-desk2>
 <20211203002513.fa43j6uvsn2ho4mm@ldmartin-desk2>
 <35ba181f-c182-aa74-07e2-fcff94bf345a@amd.com>
Message-ID: <45962757-d539-0caf-1031-9340ca4ce0e0@amd.com>
Date:   Fri, 3 Dec 2021 11:32:45 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
In-Reply-To: <35ba181f-c182-aa74-07e2-fcff94bf345a@amd.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aStHWWlNTzdaWXRjL3pYVnczYjdZS1p0RlBJSTZoUVdIbkJnYTZBVGNuVnoz?=
 =?utf-8?B?eWMzRjdzZGhxa0t3cXNqQ25mZjBmM25WVzIxa3IrVlhzc2VYeG90UkJmc3BU?=
 =?utf-8?B?YURMNVkweWVDbk12U0dYYUFjV3RxSDlTR1l1SmtQeTBtNmEwRVYyZWJqOHhL?=
 =?utf-8?B?WjBDb3NxeGJXdm1iK0E1ckhVY0I4TW4yNDlVeDJEYnc4OXZ0NGlUS0lBK2xX?=
 =?utf-8?B?RjJKTDVsbEdkS0hla1paNUY0RnZzcWlrVWxCODNlWE8yWFcxRTR6NzgxbVZC?=
 =?utf-8?B?N0x1M1ZhcVlMWE1YMnFTVWlzczNldnNTNEpuekJiY0gwbHZIMDZuek8wN01s?=
 =?utf-8?B?OGFDQUpSQzhOMWtWRTkxSldWaUdwcjlyYVJlNlQrSCtiWVEwd1JjbGxZMlNU?=
 =?utf-8?B?R2R4MmlUN0VXYlJGdm9pdDNzVFZTTlUyUTkxcGlFZUpURWwySzQ1RnVRNHNj?=
 =?utf-8?B?NEFiOXVDZ3VGWEtNdGorbkxyWEpJYUdpODNLN3Y5Z09vMjJNYWk0UnlkNHcx?=
 =?utf-8?B?Nm5Bckk5aGFzenJ2czIwK0ZIcUlmcW5ScWFEQXZERE55aElZL2l3NXV1QTND?=
 =?utf-8?B?WG5XcGpYWjVScmlYdkF3UTNRenc3eko1ZHU1d3ZrMjlRYnlYaTZwOVJCdmNs?=
 =?utf-8?B?YXFTMHdFZnpSendXaGJUby9SZnZpSThDM2pTYjJ0UzFCYWgzWUlrSnpWV0hk?=
 =?utf-8?B?SnFqRDlPbjdMK1RHbC83L0xyT1RNS2hDQUxkbjAxVmw2OEF2YkZuWjhDMXZP?=
 =?utf-8?B?V05GaUxMR3U1L0RwQnJTL2IwbXdVY2dyZTUyTjd3N0pIUXNHVWZWakFZblJF?=
 =?utf-8?B?VmhVNFJmeTdZWkUrdzNCaGpFNW5MRnh2L0dTbVBmN0NiS280SDZTNGp0QVdv?=
 =?utf-8?B?cjZBNkVSa08wZ2xVZmVlS3VOSVJwcng0djV3Z25DcXFnaWxjSXlGWis5S3lo?=
 =?utf-8?B?aTNwZ0Y4NDlVRU92ckl4QjIwVnFFUUNMdCtyMStiYXh2TW9jNlROdmVCbW4w?=
 =?utf-8?B?Z0dVYXgybFlQK0RxdjYyK1lZeU4rcTl5a2pzNGRjT000eEsyT3ZiT3hIeGEw?=
 =?utf-8?B?NThoeTZ4eTlRUmRnbHNxZFA3UlRaSHpXTUZuUGxEVThNcGF1VmYvd2M3SDJX?=
 =?utf-8?B?NkN5eHV6bGlCdkdGcWJieDdzeGE2T3ROMVZkOUhjckFVUFpSVUMrdzRscHU2?=
 =?utf-8?B?VGdCRkhmZVI3VlMxUWRpb0NRQUFUTDdtb3pTems0aVhBQ29Pbis5d1kvQlhj?=
 =?utf-8?B?ejdHRnVZWWhzL0tCUVR6NlV2cHBMclIvdFRPK3ArQzd2YzJqWkI3eW56dWtu?=
 =?utf-8?B?UjlEd1Y1ME5vQW43T1h3RUdtbTZ2ZWpXbjQ4QzV2dHpoSnBobXgvclMwN2Fx?=
 =?utf-8?B?RThVYmx4L0lCeHQ0NURsSm9OWCthVUpQTXpGcUowZFNEc2ZsYXI4S3IreThN?=
 =?utf-8?B?NWRiNDU3bjlHbDlJVU8xR08zUXU2UmZEWmlsNEJKVGJsZTVKV25jRXlHNWo1?=
 =?utf-8?B?WmVlMUFXTDRUMW4yeVJPQUxJaHVjQzN2azM4QWEyOTdvc0ZlODQzSXZDKzFN?=
 =?utf-8?B?aVMxY1I3Sk1udWJ0dzlBbFhZWkxvYjFodncwZ2NJS0h5QTV4ajRJR3BsUVIr?=
 =?utf-8?B?TEN5VlMzRnM2TG4wYmJnVzZMcHBYREYzVlMvSm0zd3hzUnNHalY2SlZTSEh6?=
 =?utf-8?B?SHdvbWtjQnR5eXB2aHZLNHJ1TUVqRWVTZXFBSWhFWEtZVDRxUVlwRlJUMUV1?=
 =?utf-8?B?N01lam01QVFFMVQzZ0R4UHllQUF6cmhKYVhseStqTzM5aXozb1N6bHZHbzNz?=
 =?utf-8?B?ZWx6cEIwSnFsR2txYU5EekdHVmVtOU1yV1dWcm1xWnJERmxsOENZTzZvWGk0?=
 =?utf-8?B?N3NHclNDR1hjRlhYbkhhODRaZHA1eWhjd291akhKUm5HU1BhODZlaGI0ZXla?=
 =?utf-8?B?NnZaZWtKbWNYcVE1ZlJuMnBwaC9MYlZSVlJPVnM2Vkk0SzZoTVRrSVJ4WnJW?=
 =?utf-8?B?R2N2TEswT3l1VzF5dU41RHVWekxCK3l4bmc4bHcwbk9IMERRUzRZeTVNMXhz?=
 =?utf-8?B?cFM2NXhsOGN0UWkrbnZRY0wvaDN1NW5oWThPWUw5TFJTb2I4b3J0TjZBSDcy?=
 =?utf-8?B?dGJvV1dYMjUyc3BQSThJWmdncUVzVmcvSWFlajErTmJ0NkZrWmlqVHI0N29r?=
 =?utf-8?Q?4wM672RqUSqxK4C4K5TvGg8=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 12015517-6dca-45a8-05e1-08d9b682ec82
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5229.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Dec 2021 17:32:48.0957
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JOVLsqRBwE2WAsfJyOQFPFAS0UQlu1FSZHr/aDEyZwB1LIfI53SNMQBB2ZAPRvoJiAEFGV8WMZniMimngQ87NA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5133
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 12/3/21 11:28 AM, Tom Lendacky wrote:
> On 12/2/21 6:25 PM, Lucas De Marchi wrote:
>> On Thu, Dec 02, 2021 at 07:55:14AM -0800, Lucas De Marchi wrote:
>>> On Thu, Dec 02, 2021 at 03:46:46PM +0100, Joerg Roedel wrote:
>>>> From: Joerg Roedel <jroedel@suse.de>
>>>>
>>>> The removal of PAGE_KERNEL_IO broke SEV-ES because it changed the
>>>> mapping of ioremap and some fixmap areas (like the local APIC page)
>>>> from unencrypted to encrypted. Change those mappings back to
>>>> be unencrypted.
>>>>
>>>> Cc: Lucas De Marchi <lucas.demarchi@intel.com>
>>>> Fixes: 27dff0f58bde ("x86/mm: Nuke PAGE_KERNEL_IO")
>>>> Signed-off-by: Joerg Roedel <jroedel@suse.de>
>>>
>>> oops, missed the fact PAGE_KERNEL had `| ENC` while PAGE_KERENL_IO
>>> didn't have it. Thanks for the fixup.
>>
>> on a second thought, the fact that PAGE_KERNEL is _not_ the same as
>> PAGE_KERNEL_IO, completely invalidates those 2 patches I sent. It seems
>> I screwed it up big here.
>>
>> About the first patch,
>> 6b2a2138cf36 ("drm/i915/gem: Stop using PAGE_KERNEL_IO"),
>> I didn't notice any regression on the i915
>> side though. Is it safe to keep it? Otherwise we are probably better
>> off reverting everything.

Is i915 for just integrated graphics? In which case SME/SEV-ES aren't 
available on Intel.

Thanks,
Tom

>>
>> I'm wondering why the addition of memory encryption
>> in 21729f81ce8a ("x86/mm: Provide general kernel support for memory 
>> encryption")
>> didn't break io_mapping_init_wc() though as it had already done a
>> s/PAGE_KERNEL_IO/PAGE_KERNEL/ in commit
>> ac96b5566926 ("io-mapping.h: s/PAGE_KERNEL_IO/PAGE_KERNEL/")
> 
> If I follow it correctly, since SME/SEV-ES are X86_64 only, 
> io_mapping_init_wc() takes the ioremap_wc() path which uses 
> PAGE_KERNEL_IO. iomap_create_wc() is only called when 
> CONFIG_HAVE_ATOMIC_IOMAP is set, which isn't set for X86_64.
> 
> Thanks,
> Tom
> 
>>
>> thanks
>> Lucas De Marchi
>>
>>>
>>> Reviewed-by: Lucas De Marchi <lucas.demarchi@intel.com>
>>>
>>> Lucas De Marchi
>>>
>>>> ---
>>>> arch/x86/include/asm/fixmap.h        |  2 +-
>>>> arch/x86/include/asm/pgtable_types.h | 21 +++++++++++----------
>>>> arch/x86/mm/ioremap.c                |  2 +-
>>>> 3 files changed, 13 insertions(+), 12 deletions(-)
>>>>
>>>> diff --git a/arch/x86/include/asm/fixmap.h 
>>>> b/arch/x86/include/asm/fixmap.h
>>>> index 5e186a69db10..a2eaf265f784 100644
>>>> --- a/arch/x86/include/asm/fixmap.h
>>>> +++ b/arch/x86/include/asm/fixmap.h
>>>> @@ -173,7 +173,7 @@ static inline void __set_fixmap(enum 
>>>> fixed_addresses idx,
>>>> * supported for MMIO addresses, so make sure that the memory encryption
>>>> * mask is not part of the page attributes.
>>>> */
>>>> -#define FIXMAP_PAGE_NOCACHE PAGE_KERNEL_NOCACHE
>>>> +#define FIXMAP_PAGE_NOCACHE PAGE_KERNEL_NOCACHE_NOENC
>>>>
>>>> /*
>>>> * Early memremap routines used for in-place encryption. The mappings 
>>>> created
>>>> diff --git a/arch/x86/include/asm/pgtable_types.h 
>>>> b/arch/x86/include/asm/pgtable_types.h
>>>> index a87224767ff3..fc9b6995cb22 100644
>>>> --- a/arch/x86/include/asm/pgtable_types.h
>>>> +++ b/arch/x86/include/asm/pgtable_types.h
>>>> @@ -208,16 +208,17 @@ enum page_cache_mode {
>>>>
>>>> #define __pgprot_mask(x)    __pgprot((x) & __default_kernel_pte_mask)
>>>>
>>>> -#define PAGE_KERNEL        __pgprot_mask(__PAGE_KERNEL            | 
>>>> _ENC)
>>>> -#define PAGE_KERNEL_NOENC    __pgprot_mask(__PAGE_KERNEL |    0)
>>>> -#define PAGE_KERNEL_RO        __pgprot_mask(__PAGE_KERNEL_RO         
>>>> | _ENC)
>>>> -#define PAGE_KERNEL_EXEC    __pgprot_mask(__PAGE_KERNEL_EXEC       | 
>>>> _ENC)
>>>> -#define PAGE_KERNEL_EXEC_NOENC __pgprot_mask(__PAGE_KERNEL_EXEC       
>>>> |    0)
>>>> -#define PAGE_KERNEL_ROX        __pgprot_mask(__PAGE_KERNEL_ROX | _ENC)
>>>> -#define PAGE_KERNEL_NOCACHE    __pgprot_mask(__PAGE_KERNEL_NOCACHE | 
>>>> _ENC)
>>>> -#define PAGE_KERNEL_LARGE    __pgprot_mask(__PAGE_KERNEL_LARGE      | 
>>>> _ENC)
>>>> -#define PAGE_KERNEL_LARGE_EXEC __pgprot_mask(__PAGE_KERNEL_LARGE_EXEC 
>>>> | _ENC)
>>>> -#define PAGE_KERNEL_VVAR    __pgprot_mask(__PAGE_KERNEL_VVAR       | 
>>>> _ENC)
>>>> +#define PAGE_KERNEL            __pgprot_mask(__PAGE_KERNEL | _ENC)
>>>> +#define PAGE_KERNEL_NOENC __pgprot_mask(__PAGE_KERNEL            |    0)
>>>> +#define PAGE_KERNEL_RO __pgprot_mask(__PAGE_KERNEL_RO         | _ENC)
>>>> +#define PAGE_KERNEL_EXEC        __pgprot_mask(__PAGE_KERNEL_EXEC | _ENC)
>>>> +#define PAGE_KERNEL_EXEC_NOENC __pgprot_mask(__PAGE_KERNEL_EXEC       
>>>> |    0)
>>>> +#define PAGE_KERNEL_ROX __pgprot_mask(__PAGE_KERNEL_ROX        | _ENC)
>>>> +#define PAGE_KERNEL_NOCACHE __pgprot_mask(__PAGE_KERNEL_NOCACHE    | 
>>>> _ENC)
>>>> +#define PAGE_KERNEL_NOCACHE_NOENC 
>>>> __pgprot_mask(__PAGE_KERNEL_NOCACHE    |    0)
>>>> +#define PAGE_KERNEL_LARGE __pgprot_mask(__PAGE_KERNEL_LARGE      | _ENC)
>>>> +#define PAGE_KERNEL_LARGE_EXEC __pgprot_mask(__PAGE_KERNEL_LARGE_EXEC 
>>>> | _ENC)
>>>> +#define PAGE_KERNEL_VVAR        __pgprot_mask(__PAGE_KERNEL_VVAR | _ENC)
>>>>
>>>> #endif    /* __ASSEMBLY__ */
>>>>
>>>> diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c
>>>> index 3102dda4b152..4fe8d43d53bb 100644
>>>> --- a/arch/x86/mm/ioremap.c
>>>> +++ b/arch/x86/mm/ioremap.c
>>>> @@ -243,7 +243,7 @@ __ioremap_caller(resource_size_t phys_addr, 
>>>> unsigned long size,
>>>>      * make sure the memory encryption attribute is enabled in the
>>>>      * resulting mapping.
>>>>      */
>>>> -    prot = PAGE_KERNEL;
>>>> +    prot = PAGE_KERNEL_NOENC;
>>>>     if ((io_desc.flags & IORES_MAP_ENCRYPTED) || encrypted)
>>>>         prot = pgprot_encrypted(prot);
>>>>
>>>> -- 
>>>> 2.34.0
>>>>