Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 12 Jun 2023 10:58:30 +0300
From:   "kirill.shutemov@linux.intel.com" <kirill.shutemov@linux.intel.com>
To:     "Huang, Kai" <kai.huang@intel.com>
Cc:     "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "Hansen, Dave" <dave.hansen@intel.com>,
        "david@redhat.com" <david@redhat.com>,
        "bagasdotme@gmail.com" <bagasdotme@gmail.com>,
        "ak@linux.intel.com" <ak@linux.intel.com>,
        "Wysocki, Rafael J" <rafael.j.wysocki@intel.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Chatre, Reinette" <reinette.chatre@intel.com>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "pbonzini@redhat.com" <pbonzini@redhat.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "Yamahata, Isaku" <isaku.yamahata@intel.com>,
        "Luck, Tony" <tony.luck@intel.com>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "Shahar, Sagi" <sagis@google.com>,
        "imammedo@redhat.com" <imammedo@redhat.com>,
        "Gao, Chao" <chao.gao@intel.com>,
        "Brown, Len" <len.brown@intel.com>,
        "sathyanarayanan.kuppuswamy@linux.intel.com" 
        <sathyanarayanan.kuppuswamy@linux.intel.com>,
        "Huang, Ying" <ying.huang@intel.com>,
        "Williams, Dan J" <dan.j.williams@intel.com>
Subject: Re: [PATCH v11 18/20] x86: Handle TDX erratum to reset TDX private
 memory during kexec() and reboot
Message-ID: <20230612075830.jbrdd6ysz4qq7wdf@box.shutemov.name>
References: <cover.1685887183.git.kai.huang@intel.com>
 <5aa7506d4fedbf625e3fe8ceeb88af3be1ce97ea.1685887183.git.kai.huang@intel.com>
 <20230609132301.uvvp27yr5kpenl6f@box.shutemov.name>
 <58f34b4b81b6d6b37d3386dec0f073e6eb7a97ff.camel@intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <58f34b4b81b6d6b37d3386dec0f073e6eb7a97ff.camel@intel.com>
Precedence: bulk

On Mon, Jun 12, 2023 at 03:06:48AM +0000, Huang, Kai wrote:
> On Fri, 2023-06-09 at 16:23 +0300, kirill.shutemov@linux.intel.com wrote:
> > On Mon, Jun 05, 2023 at 02:27:31AM +1200, Kai Huang wrote:
> > > diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c
> > > index 8ff07256a515..0aa413b712e8 100644
> > > --- a/arch/x86/virt/vmx/tdx/tdx.c
> > > +++ b/arch/x86/virt/vmx/tdx/tdx.c
> > > @@ -587,6 +587,14 @@ static int tdmr_set_up_pamt(struct tdmr_info *tdmr,
> > >  		tdmr_pamt_base += pamt_size[pgsz];
> > >  	}
> > >  
> > > +	/*
> > > +	 * tdx_memory_shutdown() also reads TDMR's PAMT during
> > > +	 * kexec() or reboot, which could happen at anytime, even
> > > +	 * during this particular code.  Make sure pamt_4k_base
> > > +	 * is firstly set otherwise tdx_memory_shutdown() may
> > > +	 * get an invalid PAMT base when it sees a valid number
> > > +	 * of PAMT pages.
> > > +	 */
> > 
> > Hmm? What prevents compiler from messing this up. It can reorder as it
> > wishes, no?
> 
> Hmm.. Right. Sorry I missed.
> 
> > 
> > Maybe add a proper locking? Anything that prevent preemption would do,
> > right?
> > 
> > >  	tdmr->pamt_4k_base = pamt_base[TDX_PS_4K];
> > >  	tdmr->pamt_4k_size = pamt_size[TDX_PS_4K];
> > >  	tdmr->pamt_2m_base = pamt_base[TDX_PS_2M];
> > 
> 
> I think a simple memory barrier will do.  How does below look?
> 
> --- a/arch/x86/virt/vmx/tdx/tdx.c
> +++ b/arch/x86/virt/vmx/tdx/tdx.c
> @@ -591,11 +591,12 @@ static int tdmr_set_up_pamt(struct tdmr_info *tdmr,
>          * tdx_memory_shutdown() also reads TDMR's PAMT during
>          * kexec() or reboot, which could happen at anytime, even
>          * during this particular code.  Make sure pamt_4k_base
> -        * is firstly set otherwise tdx_memory_shutdown() may
> -        * get an invalid PAMT base when it sees a valid number
> -        * of PAMT pages.
> +        * is firstly set and place a __mb() after it otherwise
> +        * tdx_memory_shutdown() may get an invalid PAMT base
> +        * when it sees a valid number of PAMT pages.
>          */
>         tdmr->pamt_4k_base = pamt_base[TDX_PS_4K];
> +       __mb();

If you want to play with barriers, assign pamt_4k_base the last with
smp_store_release() and read it first in tdmr_get_pamt() with
smp_load_acquire(). If it is non-zero, all pamt_* fields are valid.

Or just drop this non-sense and use a spin lock for serialization.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov