Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: [PATCH v12 00/18] Enable FSGSBASE instructions
To:     Andy Lutomirski <luto@amacapital.net>,
        Thomas Gleixner <tglx@linutronix.de>,
        Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>,
        Andi Kleen <ak@linux.intel.com>,
        Sasha Levin <sashal@kernel.org>, linux-kernel@vger.kernel.org,
        bp@alien8.de, luto@kernel.org, hpa@zytor.com,
        dave.hansen@intel.com, tony.luck@intel.com,
        ravi.v.shankar@intel.com, chang.seok.bae@intel.com
References: <e9a0a521-104b-5c3a-a689-78f878e73d31@cs.unc.edu>
 <7A3EBAB0-B3B3-4CB7-AA6A-FDF29D03E30D@amacapital.net>
 <20200529152756.GA7452@invisiblethingslab.com>
From:   Don Porter <porter@cs.unc.edu>
Message-ID: <ef8bbdff-e891-bee3-677d-3606474ecc10@cs.unc.edu>
Date:   Thu, 25 Jun 2020 11:27:28 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0)
 Gecko/20100101 Thunderbird/68.9.0
MIME-Version: 1.0
In-Reply-To: <20200529152756.GA7452@invisiblethingslab.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 5/29/20 11:27 AM, Wojtek Porczyk wrote:
> On Thu, May 28, 2020 at 11:38:01AM -0700, Andy Lutomirski wrote:
>> One useful test for the actual kernel patches would be to run your SGX
>> workload on a loaded core.  That is, do something like taskset -c
>> 0 graphene_thing and, simultaneously, write a trivial infinite loop program
>> and run that under taskset -c 0 as well. For good measure, you could have
>> perf top or perf record running at the same time.  Look for kernel errors,
>> but also look for any evidence of your workload malfunctioning.
> 
> We currently run as part of CI several workloads[1], among them LTP tests[2],
> and sometimes it's not pretty, because we encounter stability problems in
> Graphene+SGX even without the patchset. We'll pick some stable subset and
> will let know. Right now we'll have to retool CI for custom kernels, which
> will take some back and forth with uni's admins.
> 
> [1] https://github.com/oscarlab/graphene/tree/master/Examples
> [2] https://github.com/oscarlab/graphene/tree/master/LibOS/shim/test/ltp
> 

Following up: we have been running a patched 5.7 kernel with v12 of this 
series on one of our CI workers.  As Wojtek mentions, infrastructure and 
other orthogonal issues took some time.

We have run our complete SGX testing pipelines successfully several 
times with no issues: no errors in Graphene or suspicious kernel messages.

I also did Andy's suggested test:
* Graphene running nginx pinned to core 0
* infinite loop on core 0
* perf top running
* Exercised with non-SGX apache bench several times (~10 minutes of 
testing time) also from core 0

Again, no apparent issues, nothing in dmesg.  I ran a similar setup with 
our SGX-specific Graphene (PAL) unit tests.  Same story: everything 
looks good.

Let us know if we can be of any more help here.

Thanks,
Don