Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <f8fb2b4f-305f-6873-3ef8-e8d5d45e862d@amd.com>
Date:   Tue, 16 May 2023 14:44:00 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.10.0
Subject: Re: [PATCHv11 1/9] mm: Add support for unaccepted memory
Content-Language: en-US
To:     "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
        Borislav Petkov <bp@alien8.de>,
        Andy Lutomirski <luto@kernel.org>,
        Dave Hansen <dave.hansen@intel.com>,
        Sean Christopherson <seanjc@google.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        Joerg Roedel <jroedel@suse.de>,
        Ard Biesheuvel <ardb@kernel.org>
Cc:     Andi Kleen <ak@linux.intel.com>,
        Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        David Rientjes <rientjes@google.com>,
        Vlastimil Babka <vbabka@suse.cz>,
        Thomas Gleixner <tglx@linutronix.de>,
        Peter Zijlstra <peterz@infradead.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Ingo Molnar <mingo@redhat.com>,
        Dario Faggioli <dfaggioli@suse.com>,
        Mike Rapoport <rppt@kernel.org>,
        David Hildenbrand <david@redhat.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        marcelo.cerri@canonical.com, tim.gardner@canonical.com,
        khalid.elmously@canonical.com, philip.cox@canonical.com,
        aarcange@redhat.com, peterx@redhat.com, x86@kernel.org,
        linux-mm@kvack.org, linux-coco@lists.linux.dev,
        linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org,
        Mike Rapoport <rppt@linux.ibm.com>
References: <20230513220418.19357-1-kirill.shutemov@linux.intel.com>
 <20230513220418.19357-2-kirill.shutemov@linux.intel.com>
From:   Tom Lendacky <thomas.lendacky@amd.com>
In-Reply-To: <20230513220418.19357-2-kirill.shutemov@linux.intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M21tU2pGTEZaZTZtV0lVTEJqaE84U3ZSbXVpZThRbFR1WEUvUTUzaHFUUlM1?=
 =?utf-8?B?VFdhbE9ER3NxWFRxdFBzNnlKSU8zdENqRXRFWTBxdXFjWlpuUi9ZclNWVlFu?=
 =?utf-8?B?SThINWZrUWtFemJXakdiUktsNXRnd0RjSE8yYW1GTzJxYXJ3aUJxOU92bmQx?=
 =?utf-8?B?N3Y4TkVqQy9CSzJpb1JKVjUvc1FweU5iV05vYTJwRHJIZmNCS09VR0ZLMWNO?=
 =?utf-8?B?QmR1d1FIdENWMEtsT1krcTVOT3JYaU52SGNYcm1reTVOWUU5RUR0NVBjL1JS?=
 =?utf-8?B?alI1S2Ira3kxKzd0MFdRSVdhTUppZlJNbEVYQ1paOGFjQytSMnJSSGluOE5r?=
 =?utf-8?B?cStNY1Y2d1Y0VzYvOExCYThCUUZvZUNTakZaU3VQTlFnOGpUT0svVUxBRXlC?=
 =?utf-8?B?MnUyMWMyVnFEZHp3YXJ2SEV3L1ptb2VzYXY3NkZwbC9ETmdDRzJLYStPdnRl?=
 =?utf-8?B?Q0pCZGRma2JMUXVTbTVKWUFrTDI5U1VoZ2J5VWtzd3FXdkVTR1FmSGxOZ3lH?=
 =?utf-8?B?SHlQQXNTa09Kejh0TlZodHBRUUlBdXFsK01FR0lKdDhGWk5FYnFsR2VWTE1N?=
 =?utf-8?B?TDZUeXFlV0NWNG1pZnRsbENlMVVLTFU1ZnM5aWliQ3J5ZVF1UmpjTjRQQzVN?=
 =?utf-8?B?dldwRTE5dklWWFBzY3p4Uk1zVTV4Rm5IZ1NuTzB6UEMrYVdhenZiSzJkWHJH?=
 =?utf-8?B?TkRPMWd2ZXEwUUtxNGJRc0VCcG5iNU12K2J0cVU4VmR0TG9OOVpzWW13enVZ?=
 =?utf-8?B?ZjZNQnQ3TElNZEU5WXAxejFTR3ptblFSRmN4bGhjM2FoV1AvMjZIbldsVFNp?=
 =?utf-8?B?TzdsMHcrd2pZSExrRFEzK3U3S1l3ZDRJb1lySkdmNDhOUm8rWDFicWh4d1hU?=
 =?utf-8?B?Tm9ZTUtVMjQzb3NIaENPbFhqMkdWN3VnM09XMGg1aXcvNVA4WWJ6dEEwTlVv?=
 =?utf-8?B?dFJzQTV0OXdkdVByZjZHMnNEK2IwZThNY0kvck42R3N5Uml6Z3dOR0VTY01C?=
 =?utf-8?B?ZkpqNjVpUnFwazlmWFZDNk5ROFM2TXliMWZ1cFR2aEtBOWVVSUlyZCtzV1d0?=
 =?utf-8?B?emtWTkxkRlZINkRXVTlYeWxzNW5zSEhGYmlHeDViY2RCUHQ5Wm1DSmJWOWNW?=
 =?utf-8?B?VFlsNk1YNzQrTGFIb1IvbUs3RGJoWGdBbjU3dHRaSnM1TlkvV2ZvQUpjZVBw?=
 =?utf-8?B?VHZkdjVWVjZPcWVBZGR5Sm5MMzRHYTJJZGZ2TkZ5SDA0S1pyMHVyOG5RMEdW?=
 =?utf-8?B?d3V4dCtnenU5R2p3bmwyTk5jZlBrL2UwSmdXbGpvdEU3N09GbkVoa2pQSVUw?=
 =?utf-8?B?MGx4TG0vT0dmSm1PSXVsWUVzNGFLNUZzdStPVVBhR1M1eVMzZy9uZ3lXbWRa?=
 =?utf-8?B?UVEyTHovOGNRajFMOS9MQitkYU1reW53MERPd1oyeGJtdEhhODJ0c29kaUw5?=
 =?utf-8?B?NS9kUVdOOVkvVkMyUmpYL2NNSmN5VDNFMjlmb29xcktYNExjNXhGcEZJNTVN?=
 =?utf-8?B?ejBzeTkxWGo1MU5VREZkN2tlaHVURlVvTk5hV1NHR2Z1K2ZBN0RCY3dHb01P?=
 =?utf-8?B?VzNYT1hzZzR6c2g1VzBodmwzMUpEYUxkU3JZNHZTRTY3dEd1MTZpdEYrd2ti?=
 =?utf-8?B?WEVFdDhUdCsrWTV3OXRXZ1VsTDliVUxjaEdTZUlRT1hVYkYzREdSRjhiak9R?=
 =?utf-8?B?QUZIcVpaUjQyZEp3R0VsZmRmTGdVdDArNlR3Zk9pbTM1eE84dlhzNFN4Mzky?=
 =?utf-8?B?YVBFMGZDc0RPaEs1UzRYRkhwWlRXU1pWTjBsbE5JWUlSYS9xeThVZDdvVG0x?=
 =?utf-8?B?eUJ2eXF4RlNNVzVjMkRHMEpvbVVQdzVLbmVPNGYyQSthcnpKSmtZNXUxeTFR?=
 =?utf-8?B?VG1reTNxV2hjUnVyaW4wRmtxc1E5cXFBYVUwSlpyVDk1bWs3Z3RldFlUdk9G?=
 =?utf-8?B?cHhZdHppeFk0RlBVT0RGUHJZMUxtTVdKcnh1dVl2ZElxTlVET3Ric1FiakF2?=
 =?utf-8?B?Mjd5UWMyRis3Yld0WEF4YTAwVW9xZjIyRjZ4V3phVVRoRDBJb3o3bk1lN0FJ?=
 =?utf-8?B?cXZFK01yWmwwLzl4K0M5Wm0wbW9aTTEvZlNXRDk0clZlekxZTi9KZ29MNDgz?=
 =?utf-8?Q?guTH77tgQd6w8LOpo865HXv92?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c6b6d100-91da-4ac5-9b61-08db5645e741
X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5229.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2023 19:44:03.8045
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EfvOy6NXbi6GJGbdJosBsss950tcCqSpgDmqoyB19GZFuTKMWMNmzLVQ6xJWjID4VxPRK6oCO4T4IZCyhUIE/g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR12MB5494
X-Spam-Status: No, score=-3.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 5/13/23 17:04, Kirill A. Shutemov wrote:
> UEFI Specification version 2.9 introduces the concept of memory
> acceptance. Some Virtual Machine platforms, such as Intel TDX or AMD
> SEV-SNP, require memory to be accepted before it can be used by the
> guest. Accepting happens via a protocol specific to the Virtual Machine
> platform.
> 
> There are several ways kernel can deal with unaccepted memory:
> 
>   1. Accept all the memory during the boot. It is easy to implement and
>      it doesn't have runtime cost once the system is booted. The downside
>      is very long boot time.
> 
>      Accept can be parallelized to multiple CPUs to keep it manageable
>      (i.e. via DEFERRED_STRUCT_PAGE_INIT), but it tends to saturate
>      memory bandwidth and does not scale beyond the point.
> 
>   2. Accept a block of memory on the first use. It requires more
>      infrastructure and changes in page allocator to make it work, but
>      it provides good boot time.
> 
>      On-demand memory accept means latency spikes every time kernel steps
>      onto a new memory block. The spikes will go away once workload data
>      set size gets stabilized or all memory gets accepted.
> 
>   3. Accept all memory in background. Introduce a thread (or multiple)
>      that gets memory accepted proactively. It will minimize time the
>      system experience latency spikes on memory allocation while keeping
>      low boot time.
> 
>      This approach cannot function on its own. It is an extension of #2:
>      background memory acceptance requires functional scheduler, but the
>      page allocator may need to tap into unaccepted memory before that.
> 
>      The downside of the approach is that these threads also steal CPU
>      cycles and memory bandwidth from the user's workload and may hurt
>      user experience.
> 
> The patch implements #1 and #2 for now. #2 is the default. Some
> workloads may want to use #1 with accept_memory=eager in kernel
> command line. #3 can be implemented later based on user's demands.
> 
> Support of unaccepted memory requires a few changes in core-mm code:
> 
>    - memblock has to accept memory on allocation;
> 
>    - page allocator has to accept memory on the first allocation of the
>      page;
> 
> Memblock change is trivial.
> 
> The page allocator is modified to accept pages. New memory gets accepted
> before putting pages on free lists. It is done lazily: only accept new
> pages when we run out of already accepted memory. The memory gets
> accepted until the high watermark is reached.
> 
> EFI code will provide two helpers if the platform supports unaccepted
> memory:
> 
>   - accept_memory() makes a range of physical addresses accepted.
> 
>   - range_contains_unaccepted_memory() checks anything within the range
>     of physical addresses requires acceptance.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> Acked-by: Mike Rapoport <rppt@linux.ibm.com>	# memblock
> Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
> ---
>   drivers/base/node.c    |   7 ++
>   fs/proc/meminfo.c      |   5 ++
>   include/linux/mm.h     |  19 +++++
>   include/linux/mmzone.h |   8 ++
>   mm/internal.h          |   1 +
>   mm/memblock.c          |   9 +++
>   mm/mm_init.c           |   7 ++
>   mm/page_alloc.c        | 173 +++++++++++++++++++++++++++++++++++++++++
>   mm/vmstat.c            |   3 +
>   9 files changed, 232 insertions(+)
> 

> diff --git a/mm/internal.h b/mm/internal.h
> index 68410c6d97ac..b1db7ba5f57d 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -1099,4 +1099,5 @@ struct vma_prepare {
>   	struct vm_area_struct *remove;
>   	struct vm_area_struct *remove2;
>   };
> +

Looks like an unintentional change.

>   #endif	/* __MM_INTERNAL_H */
> diff --git a/mm/memblock.c b/mm/memblock.c
> index 3feafea06ab2..50b921119600 100644
> --- a/mm/memblock.c
> +++ b/mm/memblock.c
> @@ -1436,6 +1436,15 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,
>   		 */
>   		kmemleak_alloc_phys(found, size, 0);
>   
> +	/*
> +	 * Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP,
> +	 * require memory to be accepted before it can be used by the
> +	 * guest.
> +	 *
> +	 * Accept the memory of the allocated buffer.
> +	 */
> +	accept_memory(found, found + size);

I'm not an mm or memblock expert, but do we need to worry about freed 
memory from memblock_phys_free() being possibly doubly accepted? A double 
acceptance will trigger a guest termination on SNP.

Thanks,
Tom

> +
>   	return found;
>   }
>