On 21/11/2023 21:49, David Woodhouse wrote:
> On Tue, 2023-11-21 at 18:02 +0000, Paul Durrant wrote:
>> From: Paul Durrant <[email protected]>
>>
>> There is need for the existing kvm_gpc_XXX() functions to be exported. Clean
>> up now before additional functions are added in subsequent patches.
>>
>
> I think you mean "no need".
>

Oh yes. Will fix.

>> Signed-off-by: Paul Durrant <[email protected]>
>
> Reviewed-by: David Woodhouse <[email protected]>
>

Thanks,

Paul

2023-11-22 08:58:23

by Xu Yilun

[permalink] [raw]

Subject: Re: [PATCH v8 07/15] KVM: pfncache: include page offset in uhva and use it consistently

On Tue, Nov 21, 2023 at 06:02:15PM +0000, Paul Durrant wrote:
> From: Paul Durrant <[email protected]>
>
> Currently the pfncache page offset is sometimes determined using the gpa
> and sometimes the khva, whilst the uhva is always page-aligned. After a
> subsequent patch is applied the gpa will not always be valid so adjust
> the code to include the page offset in the uhva and use it consistently
> as the source of truth.
>
> Also, where a page-aligned address is required, use PAGE_ALIGN_DOWN()
> for clarity.
>
> Signed-off-by: Paul Durrant <[email protected]>
> ---
> Cc: Sean Christopherson <[email protected]>
> Cc: Paolo Bonzini <[email protected]>
> Cc: David Woodhouse <[email protected]>
>
> v8:
> - New in this version.
> ---
> virt/kvm/pfncache.c | 27 +++++++++++++++++++--------
> 1 file changed, 19 insertions(+), 8 deletions(-)
>
> diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
> index 0eeb034d0674..c545f6246501 100644
> --- a/virt/kvm/pfncache.c
> +++ b/virt/kvm/pfncache.c
> @@ -48,10 +48,10 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
> if (!gpc->active)
> return false;
>
> - if (offset_in_page(gpc->gpa) + len > PAGE_SIZE)
> + if (gpc->generation != slots->generation || kvm_is_error_hva(gpc->uhva))
> return false;
>
> - if (gpc->generation != slots->generation || kvm_is_error_hva(gpc->uhva))
> + if (offset_in_page(gpc->uhva) + len > PAGE_SIZE)
> return false;
>
> if (!gpc->valid)
> @@ -119,7 +119,7 @@ static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long mmu_s
> static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
> {
> /* Note, the new page offset may be different than the old! */
> - void *old_khva = gpc->khva - offset_in_page(gpc->khva);
> + void *old_khva = (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva);
> kvm_pfn_t new_pfn = KVM_PFN_ERR_FAULT;
> void *new_khva = NULL;
> unsigned long mmu_seq;
> @@ -192,7 +192,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
>
> gpc->valid = true;
> gpc->pfn = new_pfn;
> - gpc->khva = new_khva + offset_in_page(gpc->gpa);
> + gpc->khva = new_khva + offset_in_page(gpc->uhva);
>
> /*
> * Put the reference to the _new_ pfn. The pfn is now tracked by the
> @@ -215,8 +215,8 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa,
> struct kvm_memslots *slots = kvm_memslots(gpc->kvm);
> unsigned long page_offset = offset_in_page(gpa);
> bool unmap_old = false;
> - unsigned long old_uhva;
> kvm_pfn_t old_pfn;
> + bool hva_change = false;
> void *old_khva;
> int ret;
>
> @@ -242,8 +242,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa,
> }
>
> old_pfn = gpc->pfn;
> - old_khva = gpc->khva - offset_in_page(gpc->khva);
> - old_uhva = gpc->uhva;
> + old_khva = (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva);
>
> /* If the userspace HVA is invalid, refresh that first */
> if (gpc->gpa != gpa || gpc->generation != slots->generation ||
> @@ -259,13 +258,25 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa,
> ret = -EFAULT;
> goto out;
> }
> +
> + hva_change = true;
> + } else {
> + /*
> + * No need to do any re-mapping if the only thing that has
> + * changed is the page offset. Just page align it to allow the
> + * new offset to be added in.

I don't understand how the uhva('s offset) could be changed when both gpa and
slot are not changed. Maybe I have no knowledge of xen, but in later
patch you said your uhva would never change...

Thanks,
Yilun

> + */
> + gpc->uhva = PAGE_ALIGN_DOWN(gpc->uhva);
> }
>
> + /* Note: the offset must be correct before calling hva_to_pfn_retry() */
> + gpc->uhva += page_offset;
> +
> /*
> * If the userspace HVA changed or the PFN was already invalid,
> * drop the lock and do the HVA to PFN lookup again.
> */
> - if (!gpc->valid || old_uhva != gpc->uhva) {
> + if (!gpc->valid || hva_change) {
> ret = hva_to_pfn_retry(gpc);
> } else {
> /*
> --
> 2.39.2
>
>

2023-11-22 09:13:09

On Tue, 2023-11-21 at 22:53 +0000, David Woodhouse wrote:
> On Tue, 2023-11-21 at 18:02 +0000, Paul Durrant wrote:
> > From: Paul Durrant <[email protected]>
> >
> > If the guest sets an explicit vcpu_info GPA then, for any of the first 32
> > vCPUs, the content of the default vcpu_info in the shared_info page must be
> > copied into the new location. Because this copy may race with event
> > delivery (which updates the 'evtchn_pending_sel' field in vcpu_info) there
> > needs to be a way to defer that until the copy is complete.
> > Happily there is already a shadow of 'evtchn_pending_sel' in kvm_vcpu_xen
> > that is used in atomic context if the vcpu_info PFN cache has been
> > invalidated so that the update of vcpu_info can be deferred until the
> > cache can be refreshed (on vCPU thread's the way back into guest context).
> >
> > Also use this shadow if the vcpu_info cache has been *deactivated*, so that
> > the VMM can safely copy the vcpu_info content and then re-activate the
> > cache with the new GPA. To do this, stop considering an inactive vcpu_info
> > cache as a hard error in kvm_xen_set_evtchn_fast().
> >
> > Signed-off-by: Paul Durrant <[email protected]>
> > Reviewed-by: David Woodhouse <[email protected]>
>
> Wait, didn't we realise that this leaves the bits set in the shadow
> evtchn_pending_sel that get lost on migration?
>
> The point in your previous patch which split out a shiny new
> set_shinfo_evtchn_pending() function was that you could then *call*
> that function to ensure that the corresponding index bit was set on the
> destination host after migration, if the bit in the shinfo is.
>
> So we'd do that from kvm_xen_setup_evtchn(), kvm_xen_eventfd_assign(),
> and when setting KVM_XEN_VCPU_ATTR_TYPE_TIMER.
>
> if (bit_is_set_in_shinfo)
> set_shinfo_evtchn_pending()

I mean set_vcpu_info_evtchn_pending() of course. And we probably want
to extend the xen_shinfo_test to test it, by setting the bit in the
shinfo to mark the event as pending, and then doing each of

• Set up timer (a bit like in TEST_TIMER_RESTORE at line 817).
• Add incoming eventfd with KVM_SET_GSI_ROUTING (cf. line 563)
• Add IPI with KVM_XEN_ATTR_TYPE_EVTCHN (cf. line 597)

Each of those should set the index bit in the vcpu_info immediately if
the evtchn port is already set (and unmasked) in the shinfo.

(Ignore this part if you're cleverer than me or have had more coffee…)

It took me a moment to get my head around the different setups we have
for event channels, but that's because the standard KVM_SET_GSI_ROUTING
one is for *incoming* events, just as we would for MSIs, and we use the
standard way of attaching an eventfd to an incoming GSI/MSI/evtchn.

The KVM_XEN_ATTR_TYPE_EVTCHN one is for *outbound* events where the
guest does an EVTCHNOP_send. That can *raise* events on an eventfd, or
it can be an IPI or loopback interdomain port, which is the case we
need to test.

Attachments:

smime.p7s (5.83 kB)

2023-11-22 10:58:13

by Paul Durrant

[permalink] [raw]

Subject: Re: [PATCH v8 15/15] KVM: xen: allow vcpu_info content to be 'safely' copied

On 22/11/2023 10:39, David Woodhouse wrote:
> On Tue, 2023-11-21 at 22:53 +0000, David Woodhouse wrote:
>> On Tue, 2023-11-21 at 18:02 +0000, Paul Durrant wrote:
>>> From: Paul Durrant <[email protected]>
>>>
>>> If the guest sets an explicit vcpu_info GPA then, for any of the first 32
>>> vCPUs, the content of the default vcpu_info in the shared_info page must be
>>> copied into the new location. Because this copy may race with event
>>> delivery (which updates the 'evtchn_pending_sel' field in vcpu_info) there
>>> needs to be a way to defer that until the copy is complete.
>>> Happily there is already a shadow of 'evtchn_pending_sel' in kvm_vcpu_xen
>>> that is used in atomic context if the vcpu_info PFN cache has been
>>> invalidated so that the update of vcpu_info can be deferred until the
>>> cache can be refreshed (on vCPU thread's the way back into guest context).
>>>
>>> Also use this shadow if the vcpu_info cache has been *deactivated*, so that
>>> the VMM can safely copy the vcpu_info content and then re-activate the
>>> cache with the new GPA. To do this, stop considering an inactive vcpu_info
>>> cache as a hard error in kvm_xen_set_evtchn_fast().
>>>
>>> Signed-off-by: Paul Durrant <[email protected]>
>>> Reviewed-by: David Woodhouse <[email protected]>
>>
>> Wait, didn't we realise that this leaves the bits set in the shadow
>> evtchn_pending_sel that get lost on migration?
>>

Indeed we did not, but that's not something that *this* patch, or even
this series, is dealing with. We also know that setting the 'width' of
shared_info has some issues, but again, can we keep that for other
patches? The series is at v9 and has already suffered a fair amount of
scope-creep.

Paul

2023-11-22 11:27:07

On Wed, 2023-11-22 at 22:27 +0800, Xu Yilun wrote:
> On Wed, Nov 22, 2023 at 09:12:18AM +0000, David Woodhouse wrote:
> > On Wed, 2023-11-22 at 16:54 +0800, Xu Yilun wrote:
> > >
> > > > @@ -259,13 +258,25 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa,
> > > >                         ret = -EFAULT;
> > > >                         goto out;
> > > >                 }
> > > > +
> > > > +               hva_change = true;
> > > > +       } else {
> > > > +               /*
> > > > +                * No need to do any re-mapping if the only thing that has
> > > > +                * changed is the page offset. Just page align it to allow the
> > > > +                * new offset to be added in.
> > >
> > > I don't understand how the uhva('s offset) could be changed when both gpa and
> > > slot are not changed. Maybe I have no knowledge of xen, but in later
> > > patch you said your uhva would never change...
> >
> > It doesn't change on a normal refresh with kvm_gpc_refresh(), which is
> > just for revalidation after memslot changes or MMU invalidation.
> >
> > But it can change if the gpc is being reinitialized with a new address
> > (perhaps because the guest has made another hypercall to set the
> > address, etc.)
> >
> > That new address could happen to be in the *same* page as the previous
>
> In this case, the lower bits of new gpa should be different to gpc->gpa,
> so will hit "if (gpc->gpa != gpa ...)" branch.

I think that 'if (gpc->gpa != gpa); branch is also gratuitously
refreshing when it doesn't need to; it only needs to refresh if the
*aligned* gpas don't match.

But it was like that already, so I won't heckle Paul any further :)

Attachments:

smime.p7s (5.83 kB)

2023-11-22 15:53:26

by Paul Durrant

[permalink] [raw]

Subject: Re: [PATCH v8 07/15] KVM: pfncache: include page offset in uhva and use it consistently

On 22/11/2023 15:42, David Woodhouse wrote:
> On Wed, 2023-11-22 at 22:27 +0800, Xu Yilun wrote:
>> On Wed, Nov 22, 2023 at 09:12:18AM +0000, David Woodhouse wrote:
>>> On Wed, 2023-11-22 at 16:54 +0800, Xu Yilun wrote:
>>>>
>>>>> @@ -259,13 +258,25 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa,
>>>>>                         ret = -EFAULT;
>>>>>                         goto out;
>>>>>                 }
>>>>> +
>>>>> +               hva_change = true;
>>>>> +       } else {
>>>>> +               /*
>>>>> +                * No need to do any re-mapping if the only thing that has
>>>>> +                * changed is the page offset. Just page align it to allow the
>>>>> +                * new offset to be added in.
>>>>
>>>> I don't understand how the uhva('s offset) could be changed when both gpa and
>>>> slot are not changed. Maybe I have no knowledge of xen, but in later
>>>> patch you said your uhva would never change...
>>>
>>> It doesn't change on a normal refresh with kvm_gpc_refresh(), which is
>>> just for revalidation after memslot changes or MMU invalidation.
>>>
>>> But it can change if the gpc is being reinitialized with a new address
>>> (perhaps because the guest has made another hypercall to set the
>>> address, etc.)
>>>
>>> That new address could happen to be in the *same* page as the previous
>>
>> In this case, the lower bits of new gpa should be different to gpc->gpa,
>> so will hit "if (gpc->gpa != gpa ...)" branch.
>
> I think that 'if (gpc->gpa != gpa); branch is also gratuitously
> refreshing when it doesn't need to; it only needs to refresh if the
> *aligned* gpas don't match.
>

I did look at that but decided that gfn_to_hva_memslot() was
sufficiently lightweight that it was not really worth optimising.

> But it was like that already, so I won't heckle Paul any further :)

I appreciate it! :-)

Paul

2023-11-27 23:36:21

by Sean Christopherson

[permalink] [raw]

Subject: Re: [PATCH v8 05/15] KVM: pfncache: remove KVM_GUEST_USES_PFN usage

On Tue, Nov 21, 2023, David Woodhouse wrote:
> On Tue, 2023-11-21 at 18:02 +0000, Paul Durrant wrote:
> > From: Paul Durrant <[email protected]>
> >
> > As noted in [1] the KVM_GUEST_USES_PFN usage flag is never set by any
> > callers of kvm_gpc_init(), which also makes the 'vcpu' argument redundant.
> > Moreover, all existing callers specify KVM_HOST_USES_PFN so the usage
> > check in hva_to_pfn_retry() and hence the 'usage' argument to
> > kvm_gpc_init() are also redundant.
> > Remove the pfn_cache_usage enumeration and remove the redundant arguments,
> > fields of struct gfn_to_hva_cache, and all the related code.
> >
> > [1] https://lore.kernel.org/all/[email protected]/
> >
> > Signed-off-by: Paul Durrant <[email protected]>
>
> I think it's https://lore.kernel.org/all/[email protected]/

Yeah, that's the more important link.

> which is the key reference. I'm not sure I'm 100% on board, but I never
> got round to replying to Sean's email because it was one of those "put
> up or shut up situations" and I didn't have the bandwidth to actually
> write the code to prove my point.
>
> I think it *is* important to support non-pinned pages. There's a reason
> we even made the vapic page migratable. We want to support memory
> hotplug, we want to cope with machine checks telling us to move certain
> pages (which I suppose is memory hotplug). See commit 38b9917350cb
> ("kvm: vmx: Implement set_apic_access_page_addr") for example.

The vAPIC page is slightly different in that it effectively never opened a window
for page migration, i.e. once a vCPU was created that page was stuck. For nested
virtualization pages, the probability of being able to migrate a page at any given
time might be relatively low, but it's extremely unlikely for a page to be pinned
for the entire lifetime of a (L1) VM.

> I agree that in the first round of the nVMX code there were bugs. And
> sure, of *course* it isn't sufficient to wire up the invalidation
> without either a KVM_REQ_SOMETHIMG to put it back, or just a *check* on
> the corresponding gpc on the way back into the guest. We'd have worked
> that out.

Maybe. I spent most of a day, maybe longer, hacking at the nVMX code and was
unable to get line of sight to an end result that I felt would be worth pursuing.

I'm definitely not saying it's impossible, and I'm not dead set against
re-introducing KVM_GUEST_USES_PFN or similar, but a complete solution crosses the
threshold where it's unreasonable to ask/expect someone to pick up the work in
order to get their code/series merged.

Which is effectively what you said below, I just wanted to explain why I'm pushing
to remove KVM_GUEST_USES_PFN, and to say that if you or someone else were to write
the code it wouldn't be an automatic nak.

> And yes, the gpc has had bugs as we implemented it, but the point was
> that we got to something which *is* working, and forms a usable
> building block.
>
> So I'm not really sold on the idea of ditching KVM_GUEST_USES_PFN. I
> think we could get it working, and I think it's worth it. But my
> opinion is worth very little unless I express it in 'diff -up' form
> instead of prose, and reverting this particular patch is the least of
> my barriers to doing so, so reluctantly...