Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <110f1aa0-7fcd-1287-701a-89c2203f0ac2@amd.com>
Date:   Mon, 24 Jul 2023 12:08:40 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Subject: Re: [RFC PATCH v11 00/29] KVM: guest_memfd() and per-page attributes
To:     Sean Christopherson <seanjc@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Marc Zyngier <maz@kernel.org>,
        Oliver Upton <oliver.upton@linux.dev>,
        Huacai Chen <chenhuacai@kernel.org>,
        Michael Ellerman <mpe@ellerman.id.au>,
        Anup Patel <anup@brainfault.org>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        "Matthew Wilcox (Oracle)" <willy@infradead.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Paul Moore <paul@paul-moore.com>,
        James Morris <jmorris@namei.org>,
        "Serge E. Hallyn" <serge@hallyn.com>
Cc:     kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
        kvmarm@lists.linux.dev, linux-mips@vger.kernel.org,
        linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org,
        linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org,
        linux-mm@kvack.org, linux-security-module@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Chao Peng <chao.p.peng@linux.intel.com>,
        Fuad Tabba <tabba@google.com>,
        Jarkko Sakkinen <jarkko@kernel.org>,
        Yu Zhang <yu.c.zhang@linux.intel.com>,
        Vishal Annapurve <vannapurve@google.com>,
        Ackerley Tng <ackerleytng@google.com>,
        Maciej Szmigiero <mail@maciej.szmigiero.name>,
        Vlastimil Babka <vbabka@suse.cz>,
        David Hildenbrand <david@redhat.com>,
        Quentin Perret <qperret@google.com>,
        Michael Roth <michael.roth@amd.com>,
        Wang <wei.w.wang@intel.com>,
        Liam Merwick <liam.merwick@oracle.com>,
        Isaku Yamahata <isaku.yamahata@gmail.com>,
        "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>
References: <20230718234512.1690985-1-seanjc@google.com>
Content-Language: en-US
From:   "Nikunj A. Dadhania" <nikunj@amd.com>
In-Reply-To: <20230718234512.1690985-1-seanjc@google.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?S1ZoRnlBUDJyR1dyWEs5b0ZscndEVWJXa2NFczU5bDFYV05rSW5nYXNEZXdm?=
 =?utf-8?B?OERoOWdWYkkxK3VENjJVbXEyNkkxYXVEVjBLZmpPL0VhL1cwZzVvY1drR1JT?=
 =?utf-8?B?N1hjSVhZQmUyNzh3SUdhY0VBVUJWVHNjVTkvajRUQ29xL1ZSM2tBZy9sREdj?=
 =?utf-8?B?UFpldzlBOVVINVRZM2lVOHFmWDV5MlMrOWttNmhXem1pOHZkeDI1Q3VJQ2FB?=
 =?utf-8?B?RjV3Vm84bkFqM1g5WWgwWkw1ZUtWeXFLdVkvSzZGTkZ3NGFQQkp0RGxTMWhx?=
 =?utf-8?B?SVprQTNOV3RJUG1ESUpaU2JoMFlvcDEvSHRoYXBBWmYyWWxJamRsQUdXd1Ur?=
 =?utf-8?B?TllKekMwVUNpQlNYdFU1NHRaR24xaG5CQWN5Wks4aUV2T21EbC9wS0pnZElu?=
 =?utf-8?B?V1ZzeUw1VjlsYnZGODRIRFd4Wm9iTGVtbldZcDQzKy81WTBTUGIyeG5STHZT?=
 =?utf-8?B?djEyNWo1WTZkdmJaYmI1TFdMMGNpdEMxSmZ4WGhYSmUxYXhiVk9sSEcyQkJM?=
 =?utf-8?B?b25FWFhqK1dlRGNWaVhKTXY4K1dnVE9wUjdYcE5Ed3NxTXNOeTlRb3IzSzkx?=
 =?utf-8?B?Wnp4THY1TTd4VHVzcWhRNnJQU2FTZmxnMlVETXNIUWpRZndzN0NmdjFRd3h1?=
 =?utf-8?B?a1laUW1EK0dKOHJjVExpTkZkT0tSUUZBZzVISHVOZFJhNG9CcUZHcnQzUWJR?=
 =?utf-8?B?VUZXVzNaanp4b3o0UlprdlhJNjVPSjY5ZnlXQStCeEE3Tk1nNng3R0R5OUN3?=
 =?utf-8?B?RWQ1RkpEK1ViQ2Nua040UnJORWx4WHZRamVFMFcxcGFvem9BbUhrcU4wZms0?=
 =?utf-8?B?WlRWZ2QzTmg4dzZPdmpMYWhBUjhZUm8rZ25iV3ZCYkI5ejk3eEpiRXJGZWV5?=
 =?utf-8?B?dWVOMU16VUFKaitGTmhKUDU3eFRPMFVZSkRoT3R0Vk9vWFAzdjBTNWhUR0xO?=
 =?utf-8?B?cEJ2K2trMDMxdXFIWGE3elRlU3BEbm1wdmhnalVCaTdPaFE4VnVDcEhYcUNk?=
 =?utf-8?B?bDkrWUdCUlFqcTdjNVJuaXpuMkFqT0dDT0RXZUxlUTRqdFE4MmF5bit5MnBz?=
 =?utf-8?B?QmUrMkc3ejA3M3NGNDJpU2x6b1BTZDZRQXlpQ1Z3ZmZna1RkVnNNUTloSHdo?=
 =?utf-8?B?MmplQmN3emREVTc3QjM0NzljU1ZxZDZwY0p3RWJKbjhCa0xrT2FLWlAzRmVU?=
 =?utf-8?B?VFZtQnlMdXg5aGc2SWdvb1FVVms1Wkp1YTd3YWYrOFlHTHJRWWZNcTViZEVE?=
 =?utf-8?B?U2hJVGFzVjVSOHhDK0t5dklTNGFmRmkvZk53bTdpYWE4U1E1VTdEa0VRTXNl?=
 =?utf-8?B?dU4wa3JjK040UW5ZeEpQNzl5bkUyakhaYTBBRWVpU2YwMXFXeTZhdDlxY1hM?=
 =?utf-8?B?VFMyc2xuN3ZOd1Q3VS8veXVpQVhkTSt3aG1MMGxOWDg5ZGZiVkpHeldmWkg2?=
 =?utf-8?B?OUpmUE1lUnRXT2RIWUI1VCtyZFJUcm5lbkpRb29nekt2SUV0RmZtakdFbGQz?=
 =?utf-8?B?QjFXWC9MRDAzNGFUMHpicHFSTFJxZEdUK2hNTGNYcHJaTzVFZGc5NzNlUjFm?=
 =?utf-8?B?dzFIOEZqTzJrcFRDNXFjR3NlMHBNbXlkcUhHQnZqL3kzRUZRYnlMQ1ZscW1Z?=
 =?utf-8?B?UVVHV2p4T1ZRa0haTFQ2Zkh1UGZuNThEaVNhOUFTcEh4WW9aa0szYzJYTGJ1?=
 =?utf-8?B?RWpWdmFxY2l3Yk9EdHZDY0hpNHNuTU9KVStXZzZwT0pCNCt2NmtZYjFNVGkw?=
 =?utf-8?B?eEszanhYZmwwb0RUL1JOZ2Vkc3FzT1V3NUVscm9Mdk1oTjE3QmQ4bXNkL0Fk?=
 =?utf-8?B?b1MzYzJqWXBNcnlaVDdNUzBSUFVZVnZQdlNEMnYrUEFBbDdOZnVSSlNXQ3dp?=
 =?utf-8?B?V0tYS2pScVZlTmlsWC9MYTFId2NiOXVxbHhqYWF6TCtLdlVnSGNQYS9XbGxw?=
 =?utf-8?B?NG1HcWsxeXlNVFpzbC9ma3dNK1JjRG9iNXE5ZGxlL0dzNWltMWd6bXZPMWV6?=
 =?utf-8?B?aExPMnZadU0zZXoxMmtlRzVaMUhZYzJDL2h2WTRHMkIyaTMzdi9aa2VKNTE5?=
 =?utf-8?B?cHpHUGdydDVLZVBrSFhlQms5Yko4NVRnbk4rVHNLMk12aFFIVzR0QkwrUjRR?=
 =?utf-8?Q?amJpMnW8DV/IBiVn2HDv8+u+r?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 076b3286-626b-46fb-6874-08db8c10ac96
X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB6309.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jul 2023 06:39:05.1204
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: UzVeJFK0RtlE0mOaAY9S50k+ogzzjG0MuHLg60uHtqbm99Rz/tgFNZ7/z7dMuV5otSKvWBXRaXMFk7RMCFmatQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB8045
X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,NICE_REPLY_A,
        SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 7/19/2023 5:14 AM, Sean Christopherson wrote:
> This is the next iteration of implementing fd-based (instead of vma-based)
> memory for KVM guests.  If you want the full background of why we are doing
> this, please go read the v10 cover letter[1].
> 
> The biggest change from v10 is to implement the backing storage in KVM
> itself, and expose it via a KVM ioctl() instead of a "generic" sycall.
> See link[2] for details on why we pivoted to a KVM-specific approach.
> 
> Key word is "biggest".  Relative to v10, there are many big changes.
> Highlights below (I can't remember everything that got changed at
> this point).
> 
> Tagged RFC as there are a lot of empty changelogs, and a lot of missing
> documentation.  And ideally, we'll have even more tests before merging.
> There are also several gaps/opens (to be discussed in tomorrow's PUCK).

As per our discussion on the PUCK call, here are the memory/NUMA accounting 
related observations that I had while working on SNP guest secure page migration:

* gmem allocations are currently treated as file page allocations
  accounted to the kernel and not to the QEMU process. 
  
  Starting an SNP guest with 40G memory with memory interleave between
  Node2 and Node3

  $ numactl -i 2,3 ./bootg_snp.sh

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 242179 root      20   0   40.4g  99580  51676 S  78.0   0.0   0:56.58 qemu-system-x86

  -> Incorrect process resident memory and shared memory is reported

  Accounting of the memory happens in the host page fault handler path,
  but for private guest pages we will never hit that.

* NUMA allocation does use the process mempolicy for appropriate node 
  allocation (Node2 and Node3), but they again do not get attributed to 
  the QEMU process

  Every 1.0s: sudo numastat  -m -p qemu-system-x86 | egrep -i "qemu|PID|Node|Filepage"   gomati: Mon Jul 24 11:51:34 2023

  Per-node process memory usage (in MBs)
  PID                               Node 0          Node 1          Node 2          Node 3           Total
  242179 (qemu-system-x86)           21.14            1.61           39.44           39.38          101.57
  Per-node system memory usage (in MBs):
                            Node 0          Node 1          Node 2          Node 3           Total
  FilePages                2475.63         2395.83        23999.46        23373.22        52244.14


* Most of the memory accounting relies on the VMAs and as private-fd of 
  gmem doesn't have a VMA(and that was the design goal), user-space fails 
  to attribute the memory appropriately to the process.

  /proc/<qemu pid>/numa_maps
  7f528be00000 interleave:2-3 file=/memfd:memory-backend-memfd-shared\040(deleted) anon=1070 dirty=1070 mapped=1987 mapmax=256 active=1956 N2=582 N3=1405 kernelpagesize_kB=4
  7f5c90200000 interleave:2-3 file=/memfd:rom-backend-memfd-shared\040(deleted)
  7f5c90400000 interleave:2-3 file=/memfd:rom-backend-memfd-shared\040(deleted) dirty=32 active=0 N2=32 kernelpagesize_kB=4
  7f5c90800000 interleave:2-3 file=/memfd:rom-backend-memfd-shared\040(deleted) dirty=892 active=0 N2=512 N3=380 kernelpagesize_kB=4

  /proc/<qemu pid>/smaps
  7f528be00000-7f5c8be00000 rw-p 00000000 00:01 26629                      /memfd:memory-backend-memfd-shared (deleted)
  7f5c90200000-7f5c90220000 rw-s 00000000 00:01 44033                      /memfd:rom-backend-memfd-shared (deleted)
  7f5c90400000-7f5c90420000 rw-s 00000000 00:01 44032                      /memfd:rom-backend-memfd-shared (deleted)
  7f5c90800000-7f5c90b7c000 rw-s 00000000 00:01 1025                       /memfd:rom-backend-memfd-shared (deleted)

* QEMU based NUMA bindings will not work. Memory backend uses mbind() 
  to set the policy for a particular virtual memory range but gmem 
  private-FD does not have a virtual memory range visible in the host.

Regards,
Nikunj