Received-SPF: pass (google.com: domain of linux-crypto-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Cc:     brijesh.singh@amd.com, x86@kernel.org,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        linux-coco@lists.linux.dev, linux-mm@kvack.org,
        linux-crypto@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Joerg Roedel <jroedel@suse.de>,
        Tom Lendacky <thomas.lendacky@amd.com>,
        "H. Peter Anvin" <hpa@zytor.com>, Ard Biesheuvel <ardb@kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Jim Mattson <jmattson@google.com>,
        Andy Lutomirski <luto@kernel.org>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Sergio Lopez <slp@redhat.com>, Peter Gonda <pgonda@google.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>,
        David Rientjes <rientjes@google.com>,
        Dov Murik <dovmurik@linux.ibm.com>,
        Tobin Feldman-Fitzthum <tobin@ibm.com>,
        Borislav Petkov <bp@alien8.de>,
        Michael Roth <michael.roth@amd.com>,
        Vlastimil Babka <vbabka@suse.cz>,
        "Kirill A . Shutemov" <kirill@shutemov.name>,
        Andi Kleen <ak@linux.intel.com>, tony.luck@intel.com,
        marcorr@google.com, sathyanarayanan.kuppuswamy@linux.intel.com
Subject: Re: [PATCH Part2 v5 37/45] KVM: SVM: Add support to handle MSR based
 Page State Change VMGEXIT
To:     Sean Christopherson <seanjc@google.com>
References: <20210820155918.7518-1-brijesh.singh@amd.com>
 <20210820155918.7518-38-brijesh.singh@amd.com> <YWYCrQX4ZzwUVZCe@google.com>
 <a677423e-fa24-5b6a-785f-4cbdf0ebee37@amd.com> <YWcWW7eikkWSmCkH@google.com>
From:   Brijesh Singh <brijesh.singh@amd.com>
Message-ID: <b4067e4e-6901-265e-49f8-0142b52f5819@amd.com>
Date:   Wed, 13 Oct 2021 12:49:03 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
 Gecko/20100101 Thunderbird/78.14.0
In-Reply-To: <YWcWW7eikkWSmCkH@google.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Content-Language: en-US
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a0pVNk9Vck9Ha3B0WitldEJicGpwUndLSFlraVdtTUVOc2EwZHh4aW5XbFZO?=
 =?utf-8?B?VXBqZWplWEZ1UUZpbWZnYzNBL05VNTFaVkhleEFnVEdBUnllbE9DVldEbEhk?=
 =?utf-8?B?VjFxZTByc0QveGExbE80bzZ3aHA2NExuV3BVQWlnc0NWQU42cmFqWnZaYXNM?=
 =?utf-8?B?ZXE1YUtVYmtxcHZFd3R5Q2EzTFNyVFVzQk1iZW51OHZra0tKOUloWFl4Y2ZE?=
 =?utf-8?B?NXc5RHpWR0FjODFMZFdsSjZVU3RnMG93UjA0eHdDWkhoVmJqZnNiYzJoZWwv?=
 =?utf-8?B?dk9HUUZxdC80bjBsRlMvQlg3ZTl1OVJ0Sk9ONE1TMm0rTWVSZUFYM0NFcEx0?=
 =?utf-8?B?SjdWb21NZElvR1lXWkp6cWVJMjRTZ1lZNnFLNml5NWp4R3FTN0krSFByZEt3?=
 =?utf-8?B?TCtpM1o1T1hXTE8vSHA1UU8yTzhSL0hMSTBUY25EWlJ4RmM3T3FTRTBpZzNL?=
 =?utf-8?B?VFk1YjBJb0YxZDVmZytSeE5aWitGbk1wbW1tQy9qUTNIZ0YwOG1JSlFITGd4?=
 =?utf-8?B?Yk5QNHlHYXBCRkZhTUJmZm1rK2h2VWE0OVVtdjNJYm03NXk1YWxKY3hqbFF6?=
 =?utf-8?B?enJNbHl1N1o4ejlRZVVYR0I1MEtvR0llalRwZ294SjR0emNqSnB0STB5Q1J1?=
 =?utf-8?B?WUw3WDV5NmxCK3owZENwOSt4d2luNjJHNFFHQk5lbUh2eW5xTFBjVXVRYkUx?=
 =?utf-8?B?Y3M5N1NML3FxN3Fja3VzWmlDY0xFTzk2d2dVWERJMlh4ZEJzT3psK3NBZFVM?=
 =?utf-8?B?RmE1K1lnaGNTcG1xVGZwSG5rcmVZMkI3b3pTai9OWHE0WENDcE15VXhtTU5q?=
 =?utf-8?B?OGhEUGg1U2JhdTcwRXhGRWpacWl1SVQvdGluem5rU3hVZEZzbXZrL3hrSDUr?=
 =?utf-8?B?QmVMTUdwc2hpM0h4d3E0YjBHZyt5d1JGY3BNTWl6UzNRS2dQMERlYXNlbmNO?=
 =?utf-8?B?TURiTGdWUjgvZnhvTXZpVHNQdjhVUTI5Wmt0VGtEaXlSZ0ROMU1EZjhEK21M?=
 =?utf-8?B?SG9qRWJJRFQ0YkFDTFZlMlc0VUx1WDEvQ1Ura044ZE0raVVlQVJMSlNYZm03?=
 =?utf-8?B?UlVZeTY1ODIxMFZmWGQ5NlkybjI4cDh5eEx1dDVaZWROaDBpUE9CR0t3TGFB?=
 =?utf-8?B?Mjc5SlhlQ0wyMm9NTHlvVGRtMnRUWFhmTmh2WnBCUjFaVlI3aWZQVTlZQ3Mx?=
 =?utf-8?B?bVZFaU9vZXR5L1RZTlFRN1pvb1hQSkNyei9FMkJ3Qi8yWUNkS3lqa0E3ZXpy?=
 =?utf-8?B?Qjh4bUIxbzFwTGZ5NmpQa2RVNUdJcXdIcTJKeGZmOElTcTRrSGFrWEpmRWxV?=
 =?utf-8?B?M1JGemFEM0xIYmpBcTVpZlhOclp2TjNSYXhZSGZyK1NaZ1VtaVhJR25KQWZq?=
 =?utf-8?B?QnR1NDdQQnFZY21KTHRZN0VqQ0xjMGpXc0RjVzZXZUROajdzbnFKMHJrNWhs?=
 =?utf-8?B?WVpjY3VoUUw5VTFPY0J2VE02cHBSUVlHSUNHbmJOL3doMEVEL2c1V294Qmsr?=
 =?utf-8?B?Q2lCZUxmcm1QUHhCWkROdDA1ejNxWGhWaVlEV1VOakl1QkpESFo0UXVDdTJn?=
 =?utf-8?B?NWNWYmtaRUNINzE1YnRNeE52Wm5aRk9hakNKYitndTZac3RRdFZXVlNRbklk?=
 =?utf-8?B?aTd5WVhLeW1FNVZ1Q2lDMGFSaS8yQVcvVkJybVFoNjdrZzA0d2pWZWlxNXVn?=
 =?utf-8?B?NDNnbzBkZVNSQ3prZ2NwOGx6R1pzRTBEOVkyQlRzS1FsZ051aWpIZGlUUkNw?=
 =?utf-8?Q?mpgr4oqqXwLhBdaE7KJvqzfq5+sSbuLpm2yUOph?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4c5fa4c2-0248-4983-cfb1-08d98e71c378
X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB2718.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Oct 2021 17:49:11.4327
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: a0pswrodz6FM7rzrxzVgZzQtG83wqny/AaN97lPdWwP+T/APrZWbfnOpZVQfdOPTlct/SJLkxLu8JWH850X0eg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2368
Precedence: bulk
List-ID: <linux-crypto.vger.kernel.org>
X-Mailing-List: linux-crypto@vger.kernel.org


On 10/13/21 10:24 AM, Sean Christopherson wrote:
> On Wed, Oct 13, 2021, Brijesh Singh wrote:
>>> The more I look at this, the more strongly I feel that private <=> shared conversions
>>> belong in the MMU, and that KVM's SPTEs should be the single source of truth for
>>> shared vs. private.  E.g. add a SPTE_TDP_PRIVATE_MASK in the software available bits.
>>> I believe the only hiccup is the snafu where not zapping _all_ SPTEs on memslot
>>> deletion breaks QEMU+VFIO+GPU, i.e. KVM would lose its canonical info on unrelated
>>> memslot deletion.
>>>
>>> But that is a solvable problem.  Ideally the bug, wherever it is, would be root
>>> caused and fixed.  I believe Peter (and Marc?) is going to work on reproducing
>>> the bug.
>> We have been also setting up VM with Qemu + VFIO + GPU usecase to repro
>> the bug on AMD HW and so far we no luck in reproducing it. Will continue
>> stressing the system to recreate it. Lets hope that Peter (and Marc) can
>> easily recreate on Intel HW so that we can work towards fixing it.
> Are you trying on a modern kernel?  If so, double check that nx_huge_pages is off,
> turning that on caused the bug to disappear.  It should be off for AMD systems,
> but it's worth checking.

Yes, this is a recent kernel. I will double check the nx_huge_pages is off.


>>>> +		if (!rc) {
>>>> +			/*
>>>> +			 * This may happen if another vCPU unmapped the page
>>>> +			 * before we acquire the lock. Retry the PSC.
>>>> +			 */
>>>> +			write_unlock(&kvm->mmu_lock);
>>>> +			return 0;
>>> How will the caller (guest?) know to retry the PSC if KVM returns "success"?
>> If a guest is adhering to the GHCB spec then it will see that hypervisor
>> has not processed all the entry and it should retry the PSC.
> But AFAICT that information isn't passed to the guest.  Even in this single-page
> MSR-based case, the caller will say "all good" on a return of 0.
>
> The "full" path is more obvious, as the caller clearly continues to process
> entries unless there's an actual failure.
>
> +       for (; cur <= end; cur++) {
> +               entry = &info->entries[cur];
> +               gpa = gfn_to_gpa(entry->gfn);
> +               level = RMP_TO_X86_PG_LEVEL(entry->pagesize);
> +               op = entry->operation;
> +
> +               if (!IS_ALIGNED(gpa, page_level_size(level))) {
> +                       rc = PSC_INVALID_ENTRY;
> +                       goto out;
> +               }
> +
> +               rc = __snp_handle_page_state_change(vcpu, op, gpa, level);
> +               if (rc)
> +                       goto out;
> +       }
>
Please see the guest kernel patch #19 [1]. In spec there is no special
code for the retry. The guest will look at the PSC hdr to determine how
many entries were processed by the hypervisor (in this particular case a
0). And at time the guest can do whatever it wants. In the case of Linux
guest, we retry the PSC.

[1]
https://lore.kernel.org/linux-mm/20211008180453.462291-20-brijesh.singh@amd.com/

thanks