Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Wed, 7 Aug 2019 22:12:14 +0300
From:   Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
To:     Jethro Beekman <jethro@fortanix.com>
Cc:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "x86@kernel.org" <x86@kernel.org>,
        "linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "dave.hansen@intel.com" <dave.hansen@intel.com>,
        "sean.j.christopherson@intel.com" <sean.j.christopherson@intel.com>,
        "nhorman@redhat.com" <nhorman@redhat.com>,
        "npmccallum@redhat.com" <npmccallum@redhat.com>,
        "serge.ayoun@intel.com" <serge.ayoun@intel.com>,
        "shay.katz-zamir@intel.com" <shay.katz-zamir@intel.com>,
        "haitao.huang@intel.com" <haitao.huang@intel.com>,
        "andriy.shevchenko@linux.intel.com" 
        <andriy.shevchenko@linux.intel.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "kai.svahn@intel.com" <kai.svahn@intel.com>,
        "bp@alien8.de" <bp@alien8.de>,
        "josh@joshtriplett.org" <josh@joshtriplett.org>,
        "luto@kernel.org" <luto@kernel.org>,
        "kai.huang@intel.com" <kai.huang@intel.com>,
        "rientjes@google.com" <rientjes@google.com>,
        "cedric.xing@intel.com" <cedric.xing@intel.com>
Subject: Re: [PATCH v21 18/28] x86/sgx: Add swapping code to the core and SGX
 driver
Message-ID: <20190807191214.45xnr5y5jmplyups@linux.intel.com>
References: <20190713170804.2340-1-jarkko.sakkinen@linux.intel.com>
 <20190713170804.2340-19-jarkko.sakkinen@linux.intel.com>
 <ba0aa229-d764-4e26-5de3-044fe28ddf61@fortanix.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ba0aa229-d764-4e26-5de3-044fe28ddf61@fortanix.com>
Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
User-Agent: NeoMutt/20180716
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Aug 07, 2019 at 06:33:32AM +0000, Jethro Beekman wrote:
> On 2019-07-13 10:07, Jarkko Sakkinen wrote:
> > Because the kernel is untrusted, swapping pages in/out of the Enclave
> > Page Cache (EPC) has specialized requirements:
> > 
> > * The kernel cannot directly access EPC memory, i.e. cannot copy data
> >    to/from the EPC.
> > * To evict a page from the EPC, the kernel must "prove" to hardware that
> >    are no valid TLB entries for said page since a stale TLB entry would
> >    allow an attacker to bypass SGX access controls.
> > * When loading a page back into the EPC, hardware must be able to verify
> >    the integrity and freshness of the data.
> > * When loading an enclave page, e.g. regular pages and Thread Control
> >    Structures (TCS), hardware must be able to associate the page with a
> >    Secure Enclave Control Structure (SECS).
> > 
> > To satisfy the above requirements, the CPU provides dedicated ENCLS
> > functions to support paging data in/out of the EPC:
> > 
> > * EBLOCK:   Mark a page as blocked in the EPC Map (EPCM).  Attempting
> >              to access a blocked page that misses the TLB will fault.
> > * ETRACK:   Activate blocking tracking.  Hardware verifies that all
> >              translations for pages marked as "blocked" have been flushed
> > 	    from the TLB.
> > * EPA:      Add version array page to the EPC.  As the name suggests, a
> >              VA page is an 512-entry array of version numbers that are
> > 	    used to uniquely identify pages evicted from the EPC.
> > * EWB:      Write back a page from EPC to memory, e.g. RAM.  Software
> >              must supply a VA slot, memory to hold the a Paging Crypto
> > 	    Metadata (PCMD) of the page and obviously backing for the
> > 	    evicted page.
> > * ELD{B,U}: Load a page in {un}blocked state from memory to EPC.  The
> >              driver only uses the ELDU variant as there is no use case
> > 	    for loading a page as "blocked" in a bare metal environment.
> > 
> > To top things off, all of the above ENCLS functions are subject to
> > strict concurrency rules, e.g. many operations will #GP fault if two
> > or more operations attempt to access common pages/structures.
> > 
> > To put it succinctly, paging in/out of the EPC requires coordinating
> > with the SGX driver where all of an enclave's tracking resides.  But,
> > simply shoving all reclaim logic into the driver is not desirable as
> > doing so has unwanted long term implications:
> > 
> > * Oversubscribing EPC to KVM guests, i.e. virtualizing SGX in KVM and
> >    swapping a guest's EPC pages (without the guest's cooperation) needs
> >    the same high level flows for reclaim but has painfully different
> >    semantics in the details.
> > * Accounting EPC, i.e. adding an EPC cgroup controller, is desirable
> >    as EPC is effectively a specialized memory type and even more scarce
> >    than system memory.  Providing a single touchpoint for EPC accounting
> >    regardless of end consumer greatly simplifies the EPC controller.
> > * Allowing the userspace-facing driver to be built as a loaded module
> >    is desirable, e.g. for debug, testing and development.  The cgroup
> >    infrastructure does not support dependencies on loadable modules.
> > * Separating EPC swapping from the driver once it has been tightly
> >    coupled to the driver is non-trivial (speaking from experience).
> 
> Some of these points seem stale now.

Thanks for spotting. I'll do a full edit for the commit message and try
to make it more short and punctual.

/Jarkko