* Zachary Amsden ([email protected]) wrote:
> >1) can't use stack based args, so have to allocate each data structure,
> >which could conceivably fail unless it's some fixed buffer.
>
> We use a fixed buffer that is private to our VMI layer. It's a per-cpu
> packing struct for hypercalls. Dynamically allocating from the kernel
> inside the interface layer is a really great way to get into a whole lot
> of trouble.
Heh, indeed that's why I asked. per-cpu buffer means ROM state knows
which vcpu is current. How is this done in OS agnostic method w/out
trapping to hypervisor? Some shared data that ROM and VMM know about,
and VMM updates as it schedules each vcpu?
> >2) complicates the rom implementation slightly where implementation of
> >each deferrable part of the API needs to have switch (am I deferred or
> >not) to then build the batch, or make direct hypercall.
>
> This is an overhead that is easily absorbed by the gain. The direct
> hypercalls are mostly either always direct, or always queued. The page
> table updates already have conditional logic to do the right thing, and
> Xen doesn't require the queueing of these anymore anyways. And the
> flush happens at an explicit point. The best approach can still be fine
> tuned. You could have separate VMI calls for queued vs. non-queued
> operation. But that greatly bloats the interface and doesn't make sense
> for everything. I believe the best solution is to annotate this in the
> VMI call itself. Consider the VMI call number, not as an integer, but
> as an identifier tuple. Perhaps I'm going overboard here. Perhaps not.
>
> 31--------24-23---------16-15--------8-7-----------0
> | family | call number | reserved | annotation |
> ---------------------------------------------------
I agree with your final assessment, needs more threshing out. It does
feel a bit overkill at first blush. I worry about these semantic
changes as an annotation instead of explicit API update. But I guess
we still have more work on finding the right actual interface, not just
the possible ways to annotate the calls.
thanks,
-chris
Chris Wright wrote:
> * Zachary Amsden ([email protected]) wrote:
>
>>> 1) can't use stack based args, so have to allocate each data structure,
>>> which could conceivably fail unless it's some fixed buffer.
>>>
>> We use a fixed buffer that is private to our VMI layer. It's a per-cpu
>> packing struct for hypercalls. Dynamically allocating from the kernel
>> inside the interface layer is a really great way to get into a whole lot
>> of trouble.
>>
>
> Heh, indeed that's why I asked. per-cpu buffer means ROM state knows
> which vcpu is current. How is this done in OS agnostic method w/out
> trapping to hypervisor? Some shared data that ROM and VMM know about,
> and VMM updates as it schedules each vcpu?
>
Yes, we have private mappings per CPU. I don't think that is as
feasible on Xen, since it requires the hypervisor to support a per-CPU
PD shadow for each root. But alternative implementations are possible
using segmentation. The primary advantage is that you don't need to
call back from the interface layer to disable preemption for per-CPU
data access.
It turns out to be really easy if you add the loadsegment / savesegment
macros to the VMI interface, and require the kernel to abstain from
using, say, the GS segment. I think this is the path we are going down
for the VMI on Xen 3 port.
> I agree with your final assessment, needs more threshing out. It does
> feel a bit overkill at first blush. I worry about these semantic
> changes as an annotation instead of explicit API update. But I guess
> we still have more work on finding the right actual interface, not just
> the possible ways to annotate the calls.
>
Yes, lets focus on finding the right interface for now - and just leave
the door open a bit for the future.
Cheers,
Zach
Chris Wright wrote:
> * Zachary Amsden ([email protected]) wrote:
>
>>>1) can't use stack based args, so have to allocate each data structure,
>>>which could conceivably fail unless it's some fixed buffer.
>>
>>We use a fixed buffer that is private to our VMI layer. It's a per-cpu
>>packing struct for hypercalls. Dynamically allocating from the kernel
>>inside the interface layer is a really great way to get into a whole lot
>>of trouble.
>
>
> Heh, indeed that's why I asked. per-cpu buffer means ROM state knows
> which vcpu is current. How is this done in OS agnostic method w/out
> trapping to hypervisor? Some shared data that ROM and VMM know about,
> and VMM updates as it schedules each vcpu?
Each VCPU gets a private data area at the same linear address. The VMM
constructs private page table shadows for each VCPU, and the shadows magically
contain the right mappings for that VCPU's private data area.
Other hypervisor implementations (especially those that don't make use of shadow
page tables) would have to come up with something along the lines that you're
suggesting.
Dan.