Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 10 Jul 2023 10:58:14 -0700
In-Reply-To: <0c32f845-aad0-3059-2efa-9f6e3bb3affb@intel.com>
Mime-Version: 1.0
References: <0c9639db604a0670eeae5343d456e43d06b35d39.camel@intel.com>
 <20230630092615.GD2533791@hirez.programming.kicks-ass.net>
 <2659d6eef84f008635ba300f4712501ac88cef2c.camel@intel.com>
 <20230630183020.GA4253@hirez.programming.kicks-ass.net> <20230630190514.GH3436214@ls.amr.corp.intel.com>
 <ZJ9IKALhz1Q6ogu1@google.com> <20230704165836.GB462772@hirez.programming.kicks-ass.net>
 <1a8099e2-da28-6b2a-7b5a-1d6346b7f95d@intel.com> <20230705145750.GD4253@hirez.programming.kicks-ass.net>
 <0c32f845-aad0-3059-2efa-9f6e3bb3affb@intel.com>
Message-ID: <ZKxGtl7ErXCG5joz@google.com>
Subject: Re: [PATCH v12 07/22] x86/virt/tdx: Add skeleton to enable TDX on demand
From:   Sean Christopherson <seanjc@google.com>
To:     Dave Hansen <dave.hansen@intel.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Isaku Yamahata <isaku.yamahata@gmail.com>,
        Kai Huang <kai.huang@intel.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        Ashok Raj <ashok.raj@intel.com>,
        Tony Luck <tony.luck@intel.com>,
        "david@redhat.com" <david@redhat.com>,
        "bagasdotme@gmail.com" <bagasdotme@gmail.com>,
        "ak@linux.intel.com" <ak@linux.intel.com>,
        Rafael J Wysocki <rafael.j.wysocki@intel.com>,
        "kirill.shutemov@linux.intel.com" <kirill.shutemov@linux.intel.com>,
        Reinette Chatre <reinette.chatre@intel.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        Isaku Yamahata <isaku.yamahata@intel.com>,
        "nik.borisov@suse.com" <nik.borisov@suse.com>,
        "hpa@zytor.com" <hpa@zytor.com>, Sagi Shahar <sagis@google.com>,
        "imammedo@redhat.com" <imammedo@redhat.com>,
        "bp@alien8.de" <bp@alien8.de>, Chao Gao <chao.gao@intel.com>,
        Len Brown <len.brown@intel.com>,
        "sathyanarayanan.kuppuswamy@linux.intel.com" 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        Ying Huang <ying.huang@intel.com>,
        Dan J Williams <dan.j.williams@intel.com>,
        "x86@kernel.org" <x86@kernel.org>
Content-Type: text/plain; charset="us-ascii"
Precedence: bulk

On Thu, Jul 06, 2023, Dave Hansen wrote:
> On 7/5/23 07:57, Peter Zijlstra wrote:
> > On Wed, Jul 05, 2023 at 07:34:06AM -0700, Dave Hansen wrote:
> >> On 7/4/23 09:58, Peter Zijlstra wrote:
> >>> If we have concerns about allocating the PAMT array, can't we use CMA
> >>> for this? Allocate the whole thing at boot as CMA such that when not
> >>> used for TDX it can be used for regular things like userspace and
> >>> filecache pages?
> >> I never thought of CMA as being super reliable.  Maybe it's improved
> >> over the years.
> >>
> >> KVM also has a rather nasty habit of pinning pages, like for device
> >> passthrough.  I suspect that means that we'll have one of two scenarios:
> >>
> >>  1. CMA works great, but the TDX/CMA area is unusable for KVM because
> >>     it's pinning all its pages and they just get moved out of the CMA
> >>     area immediately.  The CMA area is effectively wasted.
> >>  2. CMA sucks, and users get sporadic TDX failures when they wait a long
> >>     time to run a TDX guest after boot.  Users just work around the CMA
> >>     support by starting up TDX guests at boot or demanding a module
> >>     parameter be set.  Hacking in CMA support was a waste.
> >>
> >> Am I just too much of a pessimist?
> > Well, if CMA still sucks, then that needs fixing. If CMA works, but we
> > have a circular fail in that KVM needs to long-term pin the PAMT pages
> > but long-term pin is evicted from CMA (the whole point of long-term pin,
> > after all), then surely we can break that cycle somehow, since in this
> > case the purpose of the CMA is being able to grab that memory chunk when
> > we needs it.
> > 
> > That is, either way around is just a matter of a little code, no?
> 
> It's not a circular dependency, it's conflicting requirements.
> 
> CMA makes memory more available, but only in the face of unpinned pages.
> 
> KVM can pin lots of pages, even outside of TDX-based VMs.
> 
> So we either need to change how CMA works fundamentally or stop KVM from
> pinning pages.

Nit, I think you're conflating KVM with VFIO and/or IOMMU code.  Device passhthrough
does pin large chunks of memory, but KVM itself isn't involved or even aware of
the pins.

HugeTLB is another case where CMA will be effectively used to serve guest memory,
but again KVM isn't the thing doing the pinning.