Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Fri, 11 Mar 2022 14:33:14 +0200
From:   Jarkko Sakkinen <jarkko@kernel.org>
To:     Haitao Huang <haitao.huang@linux.intel.com>,
        "Chatre, Reinette" <reinette.chatre@intel.com>
Cc:     "Dhanraj, Vijay" <vijay.dhanraj@intel.com>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "bp@alien8.de" <bp@alien8.de>,
        "Lutomirski, Andy" <luto@kernel.org>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
        "x86@kernel.org" <x86@kernel.org>,
        "Christopherson,, Sean" <seanjc@google.com>,
        "Huang, Kai" <kai.huang@intel.com>,
        "Zhang, Cathy" <cathy.zhang@intel.com>,
        "Xing, Cedric" <cedric.xing@intel.com>,
        "Huang, Haitao" <haitao.huang@intel.com>,
        "Shanahan, Mark" <mark.shanahan@intel.com>,
        "hpa@zytor.com" <hpa@zytor.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH V2 16/32] x86/sgx: Support restricting of enclave page
 permissions
Message-ID: <YitBinMgBpPbqful@iki.fi>
References: <cover.1644274683.git.reinette.chatre@intel.com>
 <4ce06608b5351f65f4e6bc6fc87c88a71215a2e7.1644274683.git.reinette.chatre@intel.com>
 <YhLhoMFPyOFZ2fsX@iki.fi>
 <DM8PR11MB55917F499CDF4CC7D426B0A7F63C9@DM8PR11MB5591.namprd11.prod.outlook.com>
 <YimWaAqEnXHbLdjh@iki.fi>
 <op.1itu5vkewjvjmi@hhuan26-mobl1.mshome.net>
 <Yis8LV99mORcLYs6@iki.fi>
 <Yis9rA8uC/0bmWCF@iki.fi>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Yis9rA8uC/0bmWCF@iki.fi>
Precedence: bulk

On Fri, Mar 11, 2022 at 02:16:47PM +0200, Jarkko Sakkinen wrote:
> On Fri, Mar 11, 2022 at 02:10:24PM +0200, Jarkko Sakkinen wrote:
> > On Thu, Mar 10, 2022 at 12:33:20PM -0600, Haitao Huang wrote:
> > > Hi Jarkko
> > > 
> > > I have some trouble understanding the sequences below.
> > > 
> > > On Thu, 10 Mar 2022 00:10:48 -0600, Jarkko Sakkinen <jarkko@kernel.org>
> > > wrote:
> > > 
> > > > On Wed, Feb 23, 2022 at 07:21:50PM +0000, Dhanraj, Vijay wrote:
> > > > > Hi All,
> > > > > 
> > > > > Regarding the recent update of splitting the page permissions change
> > > > > request into two IOCTLS (RELAX and RESTRICT), can we combine them into
> > > > > one? That is, revert to how it was done in the v1 version?
> > > > > 
> > > > > Why? Currently in Gramine (a library OS for unmodified applications,
> > > > > https://gramineproject.io/) with the new proposed change, one needs to
> > > > > store the page permission for each page or range of pages. And for every
> > > > > request of `mmap` or `mprotect`, Gramine would have to do a lookup
> > > > > of the
> > > > > page permissions for the request range and then call the respective
> > > > > IOCTL
> > > > > either RESTRICT or RELAX. This seems a little overwhelming.
> > > > > 
> > > > > Request: Instead, can we do `MODPE`,  call `RESTRICT` IOCTL, and then do
> > > > > an `EACCEPT` irrespective of RELAX or RESTRICT page permission request?
> > > > > With this approach, we can avoid storing  page permissions and simplify
> > > > > the implementation.
> > > > > 
> > > > > I understand RESTRICT IOCTL would do a `MODPR` and trigger `ETRACK`
> > > > > flows
> > > > > to do TLB shootdowns which might not be needed for RELAX IOCTL but I am
> > > > > not sure what will be the performance impact. Is there any data point to
> > > > > see the performance impact?
> > > > > 
> > > > > Thanks,
> > > > > -Vijay
> > > > 
> > > > This should get better in the next versuin. "relax" is gone. And for
> > > > dynamic EAUG'd pages only VMA and EPCM permissions matter, i.e.
> > > > internal vm_max_prot_bits is set to RWX.
> > > > 
> > > > I patched the existing series eno
> > > > 
> > > > For Enarx I'm using the following patterns.
> > > > 
> > > > Shim mmap() handler:
> > > > 1. Ask host for mmap() syscall.
> > > > 2. Construct secinfo matching the protection bits.
> > > > 3. For each page in the address range: EACCEPTCOPY with a
> > > >    zero page.
> > > 
> > > For EACCEPTCOPY to work, I believe PTE.RW is required for the target page.
> > > So this only works for mmap(..., RW) or mmap(...,RWX).
> > 
> > I use it only with EAUG.
> > 
> > > So that gives you pages with RW/RWX.
> > > 
> > > To change permissions of any of those pages from RW/RWX to R/RX , you need
> > > call ENCLAVE_RESTRICT_PERMISSIONS ioctl with R or with PROT_NONE. you can't
> > > just do EMODPE.
> > > 
> > > so for RW->R, you either:
> > > 
> > > 1)EMODPR(EPCM.NONE)
> > > 2)EACCEPT(EPCM.NONE)
> > > 3)EMODPE(R) -- not sure this would work as spec says EMODPE requires "Read
> > > access permitted by enclave"
> > > 
> > > or:
> > > 
> > > 1)EMODPR(EPCM.PROT_R)
> > > 2)EACCEPT(EPCM.PROT_R)
> > 
> > I checked from SDM and you're correct.
> > 
> > Then the appropriate thing is to reset to R.
> > 
> > > > Shim mprotect() handler:
> > > > 1. Ask host for mprotect() syscall.
> > > > 2. For each page in the address range: EACCEPT with PROT_NONE
> > > >    secinfo and EMODPE with the secinfo having the prot bits.
> > > 
> > > EACCEPT requires PTE.R. And EAUG'd pages will always initialized with
> > > EPCM.RW,
> > > so EACCEPT(EPCM.PROT_NONE) will fail with SGX_PAGE_ATTRIBUTES_MISMATCH.
> > 
> > Ditto.
> > 
> > > > Backend mprotect() handler:
> > > > 1. Invoke ENCLAVE_RESTRICT_PERMISSIONS ioctl for the address
> > > >    range with PROT_NONE.
> > > > 2. Invoke real mprotect() syscall.
> > > > 
> > > Note #1 can only be done after EACCEPT. MODPR is not allowed for pending
> > > pages.
> > 
> > Yes, and that's what I'm doing. After that shim does EACCEPT's in a loop.
> > 
> > Reinette, the ioctl should already check that either R or W is set in
> > secinfo and return -EACCES.
> > 
> > I.e.
> > 
> > (* Check for misconfigured SECINFO flags*)
> > IF ( (SCRATCH_SECINFO reserved fields are not zero ) or
> > (SCRATCH_SECINFO.FLAGS.R is 0 and SCRATCH_SECINFO.FLAGS.W is not 0) )
> > THEN #GP(0); FI;
> > 
> > I was testing this and wondering why my enclave #GP's, and then I checked
> > SDM after reading Haitao's response. So clearly check in kernel side is
> > needed.
> 
> I would consider also adding such check "add pages". It's our least common
> denominator.
> 
> If we can assume that at least R is there for every enclave page, then it
> gives invariant that enables EMODPR with R all the time.

Since EAUG is done already in the #PF handler, so must be EMODPR. Otherwise
we do things incosistently [*]. One being in #PF handler and other being
ioctl is unacceptable.

Moving EMODPR to #PF handler would be trivial:

1. In mprotect() callback unmap PTE's for
   the range.
2. In #PF handler, EMODPR with read permissions.

This is something that would be understandable for the user space. The only
API ever required would be EMODPE for permission changes. You could
basically implement the whole thing for EPCM inside enclave with no ioctls
required.

That would leave only ioctls to the series:
1. SGX_IOC_ENCLAVE_MODIFY_TYPE
2. SGX_IOO_ENCLAVE_REMOVE_PAGES

[*] For me stick to #PF handler for EAUG is fine for the first mainline
version. The API side is factors more critical.

BR, Jarkko