Subject: Re: [PATCH] x86/mm: Fix PAGE_KERNEL_IO removal breakage
To:     Lucas De Marchi <lucas.demarchi@intel.com>,
        Joerg Roedel <joro@8bytes.org>
Cc:     x86@kernel.org, Dave Hansen <dave.hansen@linux.intel.com>,
        Borislav Petkov <bp@alien8.de>, Ingo Molnar <mingo@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>, hpa@zytor.com,
        Andy Lutomirski <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        Brijesh Singh <brijesh.singh@amd.com>,
        linux-kernel@vger.kernel.org, Joerg Roedel <jroedel@suse.de>
References: <20211202144646.23186-1-joro@8bytes.org>
 <20211202155452.jh4qnvpx52r3od67@ldmartin-desk2>
 <20211203002513.fa43j6uvsn2ho4mm@ldmartin-desk2>
From:   Tom Lendacky <thomas.lendacky@amd.com>
Message-ID: <35ba181f-c182-aa74-07e2-fcff94bf345a@amd.com>
Date:   Fri, 3 Dec 2021 11:28:58 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
In-Reply-To: <20211203002513.fa43j6uvsn2ho4mm@ldmartin-desk2>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?djBLZmtkalZMMWlNZVdDenFmWkc4WE1ZZHc0cUZGTUU5bTBOd3h6OWFtQnAr?=
 =?utf-8?B?UXYvSEs4aUtEelhwZDh5dmhJd0RLbzhSQUl1a2FDdnV0NUhFaVBBYlBFZS9L?=
 =?utf-8?B?R0JvamszWXlZL2R4UVlPYkx0ZHZTOGgvNGZvS0ZsUkFWK3hRVEZBelptSDNF?=
 =?utf-8?B?V0phbWlmSHUvUnBWVEdobmpsNEUyVzBJOGQxOGh6eHk3Znk2Qm02M1pXeDBp?=
 =?utf-8?B?ejBLMXA1ZHpCVXZjWks4Q1o0aWk1d094MnVML3pxN3dQV1RoYzREcmlFS0ZS?=
 =?utf-8?B?ZDlvMU1YbDhZbmtNVmsvMEJnOERqMEZGSzEwMkRvLzdaTWlZZzQ2MnVkKzhZ?=
 =?utf-8?B?aDdQWEl0ZkNlTU1iM0R0MEN3RFRVME11QW1rYUV6UkU0YWpvRU14MVlMN294?=
 =?utf-8?B?UFlZdXdIMWRvaW5YM054eWI5Qk9DUEQ2dTJobFJDY3JXYS9BNDJ2eVJBZmhQ?=
 =?utf-8?B?K2oxdXJVdnpBTU52K1hiTnVmUk1UbEJXMGgrZmthczF4WG1vWjVnK3VENHMv?=
 =?utf-8?B?cWNmTlozV0d2YWlCV0JtMkIyZlgrdWNoY1hNRHRwSllzZmpmSm1YY3pPMjhy?=
 =?utf-8?B?bHMwU2hUL1gySkZtdGljbGJQZ3ByVmxjUDYyYjYwQ2ZtZ1FnS3ZOV05LZjBh?=
 =?utf-8?B?T2QxS2FDZytRYzlLbUFPeXFQSzljcHY4L1BrWVZZamFvU2dUbTJJZmhNZ1Jh?=
 =?utf-8?B?eGNoUWtMeldxVTF4dFp0K0pvay9XS3NLcEttRkx1MzF6ZzRHdmp6L1NjSEQ1?=
 =?utf-8?B?WSttZ3p5RVZoaHhEVlM5KzdUSlkrdHQyQ3FwMmFnZE92cGQ1SmErUGtROURK?=
 =?utf-8?B?VGhoVlc4UVlaNEdPVVQzMVVXUGJDMEM2bFdrNHhoMmsrcmhqWlphd1BLc3do?=
 =?utf-8?B?QVF5dmk1RENnck5GcVI4b1loakp3UDdrR3o4QktsTW1NTGJoYk9qdnNrTDRy?=
 =?utf-8?B?anloNnBMMlRDcU1HS0gzMXVxbDV6OE5XVlJqYWl1c0E4cGxZelFCWlRoSUNP?=
 =?utf-8?B?K0g1N01HMWM4SG15Z1RCOXNRTGlScTdyN3BLRXNhdmpsRGJsTnRjUjhLcTFr?=
 =?utf-8?B?RzY0blBQcGMzNGwvdUR6dUx4UTB6MzF6NTN3N1ZaUnBXaytGRmMvRWZxNURv?=
 =?utf-8?B?SGRuTmI5TWlNUmx4aEE1NDk3MG5HeGhlNGNNVDhXU1UyaFdSejk0YjlGUGIw?=
 =?utf-8?B?T2lXcXhkT3lWSXV4WEVra1ZPbks4U1FPQklWRVd6VHBUN1JCUk1IQ1NXeXcx?=
 =?utf-8?B?WFdSMFpaK3NhYW9Uekw1T2hQaTN2b09MT2RXbzhuUVNmYmJZZGJoTW8ya3VV?=
 =?utf-8?B?K2t5cU84UkV0TGRyWkZkOGs3dURXUkY4aUpDWjE5eDdDQmhiREc2dTBtVjhV?=
 =?utf-8?B?Z1ZNQm1Gald4djlReG5KNTFIVFhSbFg2K2NnK050cTBpUnh3aWN3eEx6M2w5?=
 =?utf-8?B?VmttaC9VV1pLTEFiSUZ2alJkeDJSdHdUR2YwTzVrd0M2cFpJRk8ybFduekg5?=
 =?utf-8?B?c0xJM2llRkh2Mm5teGoxd1Q5WGpiOWVXU0dnRmllV2h4K092OTArL3ZxN2Ux?=
 =?utf-8?B?NjJlS0VnWkNFeStQV29tYXFrTmFwYXg4aVRBR3BscGY1R3FCNmd1eGJWbHdh?=
 =?utf-8?B?Mm1rS0tsb1BKQXZNLzdUTEtNTmludkZBekZYazBiVFVtaW1KdEtNV3puRVRn?=
 =?utf-8?B?dGtSL3NWb28vOVNhZWxQOXRjaHFUeGI1REhTT0h5d0xFQmZKSC9VNmszeU0y?=
 =?utf-8?B?WUN2ZU5XY3R2TzQ5V1pJMllsc0ZnVHlmRk1JVGFHeXpXcEd2L0o0SWJwNVBP?=
 =?utf-8?B?dm9UQ0s4QXJzbFAwek9NSk5rTDBxY2o3QmRPeFloTjVvYm01aVVUNXdZbXRY?=
 =?utf-8?B?ZjRpN29yNWZOZi9KYVBsd0s1TkpsMHpOWThJM3FIa2hTQXBGUmFLbXhHQmdu?=
 =?utf-8?B?ak1wTkl6WW9lQVdvWDZuMFRJdmNsbmhBOE1rTGhpNlY1Q3pNMWs1ZlZJNVVl?=
 =?utf-8?B?WUJRL3hGS3owY2NQMjlHY0ttVFdNbXlieU9iV0JIZjFHWS8xUzlocXV2ZDNI?=
 =?utf-8?B?aEZ0c245QUc5SkpJSUNmOWRPMFYzbGZKaWxua1lYUFk4K1kxRU8xdlNST3lw?=
 =?utf-8?B?NFg5MWNDQThwY3d4c2gvQnpFZjRPbjNTTXJlWW1ERWhvcEd4MzBmMG1Ia08y?=
 =?utf-8?Q?Zj00mvuRGi4dTv45HIODRIc=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4d6d8fb5-d39a-460e-57f7-08d9b6826553
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5229.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Dec 2021 17:29:01.4089
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3g+MNlovRMK8q1yTsnBMK4bilEXfgAzGeNrBaQzoNmO6v52CTnuwfxkp+an9kZ052cpsUpq3aTthaRA70/nttg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5133
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 12/2/21 6:25 PM, Lucas De Marchi wrote:
> On Thu, Dec 02, 2021 at 07:55:14AM -0800, Lucas De Marchi wrote:
>> On Thu, Dec 02, 2021 at 03:46:46PM +0100, Joerg Roedel wrote:
>>> From: Joerg Roedel <jroedel@suse.de>
>>>
>>> The removal of PAGE_KERNEL_IO broke SEV-ES because it changed the
>>> mapping of ioremap and some fixmap areas (like the local APIC page)
>>> from unencrypted to encrypted. Change those mappings back to
>>> be unencrypted.
>>>
>>> Cc: Lucas De Marchi <lucas.demarchi@intel.com>
>>> Fixes: 27dff0f58bde ("x86/mm: Nuke PAGE_KERNEL_IO")
>>> Signed-off-by: Joerg Roedel <jroedel@suse.de>
>>
>> oops, missed the fact PAGE_KERNEL had `| ENC` while PAGE_KERENL_IO
>> didn't have it. Thanks for the fixup.
> 
> on a second thought, the fact that PAGE_KERNEL is _not_ the same as
> PAGE_KERNEL_IO, completely invalidates those 2 patches I sent. It seems
> I screwed it up big here.
> 
> About the first patch,
> 6b2a2138cf36 ("drm/i915/gem: Stop using PAGE_KERNEL_IO"),
> I didn't notice any regression on the i915
> side though. Is it safe to keep it? Otherwise we are probably better
> off reverting everything.
> 
> I'm wondering why the addition of memory encryption
> in 21729f81ce8a ("x86/mm: Provide general kernel support for memory 
> encryption")
> didn't break io_mapping_init_wc() though as it had already done a
> s/PAGE_KERNEL_IO/PAGE_KERNEL/ in commit
> ac96b5566926 ("io-mapping.h: s/PAGE_KERNEL_IO/PAGE_KERNEL/")

If I follow it correctly, since SME/SEV-ES are X86_64 only, 
io_mapping_init_wc() takes the ioremap_wc() path which uses 
PAGE_KERNEL_IO. iomap_create_wc() is only called when 
CONFIG_HAVE_ATOMIC_IOMAP is set, which isn't set for X86_64.

Thanks,
Tom

> 
> thanks
> Lucas De Marchi
> 
>>
>> Reviewed-by: Lucas De Marchi <lucas.demarchi@intel.com>
>>
>> Lucas De Marchi
>>
>>> ---
>>> arch/x86/include/asm/fixmap.h        |  2 +-
>>> arch/x86/include/asm/pgtable_types.h | 21 +++++++++++----------
>>> arch/x86/mm/ioremap.c                |  2 +-
>>> 3 files changed, 13 insertions(+), 12 deletions(-)
>>>
>>> diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h
>>> index 5e186a69db10..a2eaf265f784 100644
>>> --- a/arch/x86/include/asm/fixmap.h
>>> +++ b/arch/x86/include/asm/fixmap.h
>>> @@ -173,7 +173,7 @@ static inline void __set_fixmap(enum 
>>> fixed_addresses idx,
>>> * supported for MMIO addresses, so make sure that the memory encryption
>>> * mask is not part of the page attributes.
>>> */
>>> -#define FIXMAP_PAGE_NOCACHE PAGE_KERNEL_NOCACHE
>>> +#define FIXMAP_PAGE_NOCACHE PAGE_KERNEL_NOCACHE_NOENC
>>>
>>> /*
>>> * Early memremap routines used for in-place encryption. The mappings 
>>> created
>>> diff --git a/arch/x86/include/asm/pgtable_types.h 
>>> b/arch/x86/include/asm/pgtable_types.h
>>> index a87224767ff3..fc9b6995cb22 100644
>>> --- a/arch/x86/include/asm/pgtable_types.h
>>> +++ b/arch/x86/include/asm/pgtable_types.h
>>> @@ -208,16 +208,17 @@ enum page_cache_mode {
>>>
>>> #define __pgprot_mask(x)    __pgprot((x) & __default_kernel_pte_mask)
>>>
>>> -#define PAGE_KERNEL        __pgprot_mask(__PAGE_KERNEL            | _ENC)
>>> -#define PAGE_KERNEL_NOENC    __pgprot_mask(__PAGE_KERNEL            
>>> |    0)
>>> -#define PAGE_KERNEL_RO        __pgprot_mask(__PAGE_KERNEL_RO         | 
>>> _ENC)
>>> -#define PAGE_KERNEL_EXEC    __pgprot_mask(__PAGE_KERNEL_EXEC       | 
>>> _ENC)
>>> -#define PAGE_KERNEL_EXEC_NOENC    
>>> __pgprot_mask(__PAGE_KERNEL_EXEC       |    0)
>>> -#define PAGE_KERNEL_ROX        __pgprot_mask(__PAGE_KERNEL_ROX        
>>> | _ENC)
>>> -#define PAGE_KERNEL_NOCACHE    __pgprot_mask(__PAGE_KERNEL_NOCACHE    
>>> | _ENC)
>>> -#define PAGE_KERNEL_LARGE    __pgprot_mask(__PAGE_KERNEL_LARGE      | 
>>> _ENC)
>>> -#define PAGE_KERNEL_LARGE_EXEC    
>>> __pgprot_mask(__PAGE_KERNEL_LARGE_EXEC | _ENC)
>>> -#define PAGE_KERNEL_VVAR    __pgprot_mask(__PAGE_KERNEL_VVAR       | 
>>> _ENC)
>>> +#define PAGE_KERNEL            __pgprot_mask(__PAGE_KERNEL            
>>> | _ENC)
>>> +#define PAGE_KERNEL_NOENC        
>>> __pgprot_mask(__PAGE_KERNEL            |    0)
>>> +#define PAGE_KERNEL_RO            
>>> __pgprot_mask(__PAGE_KERNEL_RO         | _ENC)
>>> +#define PAGE_KERNEL_EXEC        __pgprot_mask(__PAGE_KERNEL_EXEC       
>>> | _ENC)
>>> +#define PAGE_KERNEL_EXEC_NOENC        
>>> __pgprot_mask(__PAGE_KERNEL_EXEC       |    0)
>>> +#define PAGE_KERNEL_ROX            
>>> __pgprot_mask(__PAGE_KERNEL_ROX        | _ENC)
>>> +#define PAGE_KERNEL_NOCACHE        
>>> __pgprot_mask(__PAGE_KERNEL_NOCACHE    | _ENC)
>>> +#define PAGE_KERNEL_NOCACHE_NOENC    
>>> __pgprot_mask(__PAGE_KERNEL_NOCACHE    |    0)
>>> +#define PAGE_KERNEL_LARGE        
>>> __pgprot_mask(__PAGE_KERNEL_LARGE      | _ENC)
>>> +#define PAGE_KERNEL_LARGE_EXEC        
>>> __pgprot_mask(__PAGE_KERNEL_LARGE_EXEC | _ENC)
>>> +#define PAGE_KERNEL_VVAR        __pgprot_mask(__PAGE_KERNEL_VVAR       
>>> | _ENC)
>>>
>>> #endif    /* __ASSEMBLY__ */
>>>
>>> diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c
>>> index 3102dda4b152..4fe8d43d53bb 100644
>>> --- a/arch/x86/mm/ioremap.c
>>> +++ b/arch/x86/mm/ioremap.c
>>> @@ -243,7 +243,7 @@ __ioremap_caller(resource_size_t phys_addr, 
>>> unsigned long size,
>>>      * make sure the memory encryption attribute is enabled in the
>>>      * resulting mapping.
>>>      */
>>> -    prot = PAGE_KERNEL;
>>> +    prot = PAGE_KERNEL_NOENC;
>>>     if ((io_desc.flags & IORES_MAP_ENCRYPTED) || encrypted)
>>>         prot = pgprot_encrypted(prot);
>>>
>>> -- 
>>> 2.34.0
>>>