Alan Cox wrote:
> On Tue, 31 Mar 2009 14:43:55 -0400
> Gregory Haskins <[email protected]> wrote:
>
>
>> The ioapic code currently privately manages the mapping between irq
>> and vector. This results in some layering violations as the support
>> for certain MSI operations need this info. As a result, the MSI
>> code itself was moved to the ioapic module. This is not really
>> optimal.
>>
>
> This appears to have been muddled in with the vnet patches ?
>
Its needed for the kvm-connector patches later in the series, so it was
included intentionally.

On that topic, I probably should have had a TOC of some kind. Hmm..let
me hack one together now:

Patch 1: Stand-alone "shared-memory signal" construct, used by various
components in vbus/venet
Patches 2-5: Basic vbus infrastructure
Patches 6-7: IOQ construct, similar to virtio-ring. Used to overlay
ring-like behavior over the shm interface in vbus
Patches 8-12: virtual-ethernet front and backends
Patch 13: io-apic work to expose the irq-vector in x86, needed for
dynirq support
Patches 14-16: KVM host side support
Patch 17: KVM guest side support

Sorry for the confusion :(

Regards,
-Greg

Attachments:

signature.asc (257.00 B)
OpenPGP digital signature

2009-03-31 20:00:49

Avi Kivity wrote:
> Gregory Haskins wrote:
>>> - works with all guests
>>> - supports hotplug/hotunplug, udev, sysfs, module autoloading, ...
>>> - supported in all OSes
>>> - someone else maintains it
>>>
>> These points are all valid, and I really struggled with this particular
>> part of the design. The entire vbus design only requires one IRQ for
>> the entire guest,
>
> Won't this have scaling issues? One IRQ means one target vcpu.
> Whereas I'd like virtio devices to span multiple queues, each queue
> with its own MSI IRQ.
Hmm..you know I hadnt really thought of it that way, but you have a
point. To clarify, my design actually uses one IRQ per "eventq", where
we can have an arbitrary number of eventq's defined (note: today I only
define one eventq, however). An eventq is actually a shm-ring construct
where I can pass events up to the host like "device added" or "ring X
signaled". Each individual device based virtio-ring would then
aggregates "signal" events onto this eventq mechanism to actually inject
events to the host. Only the eventq itself injects an actual IRQ to the
assigned vcpu.

My intended use of multiple eventqs was for prioritization of different
rings. For instance, we could define 8 priority levels, each with its
own ring/irq. That way, a virtio-net that supports something like
802.1p could define 8 virtio-rings, one for each priority level.

But this scheme is more targeted at prioritization than per vcpu
irq-balancing. I support the eventq construct I proposed could still be
used in this fashion since each has its own routable IRQ. However, I
would have to think about that some more because it is beyond the design
spec.

The good news is that the decision to use the "eventq+irq" approach is
completely contained in the kvm-host+guest.patch. We could easily
switch to a 1:1 irq:shm-signal if we wanted to, and the device/drivers
would work exactly the same without modification.

> Also, the single IRQ handler will need to scan for all potential IRQ
> sources. Even if implemented carefully, this will cause many
> cacheline bounces.
Well, no, I think this part is covered. As mentioned above, we use a
queuing technique so there is no scanning needed. Ultimately I would
love to adapt a similar technique to optionally replace the LAPIC. That
way we can avoid the EOI trap and just consume the next interrupt (if
applicable) from the shm-ring.

>
>> so its conceivable that I could present a simple
>> "dummy" PCI device with some "VBUS" type PCI-ID, just to piggy back on
>> the IRQ routing logic. Then userspace could simply pass the IRQ routing
>> info down to the kernel with an ioctl, or something similar.
>>
>
> Xen does something similar, I believe.
>
>> I think ultimately I was trying to stay away from PCI in general because
>> I want to support environments that do not have PCI. However, for the
>> kvm-transport case (at least on x86) this isnt really a constraint.
>>
>>
>
> s/PCI/the native IRQ solution for your platform/. virtio has the same
> problem; on s390 we use the native (if that word ever applies to s390)
> interrupt and device discovery mechanism.

yeah, I agree. We can contain the "exposure" of PCI to just platforms
within KVM that care about it.

-Greg

Attachments:

signature.asc (257.00 B)
OpenPGP digital signature

2009-03-31 20:35:54

Avi Kivity wrote:
> Gregory Haskins wrote:
>> This interface provides a bidirectional shared-memory based signaling
>> mechanism. It can be used by any entities which desire efficient
>> communication via shared memory. The implementation details of the
>> signaling are abstracted so that they may transcend a wide variety
>> of locale boundaries (e.g. userspace/kernel, guest/host, etc).
>>
>> The shm_signal mechanism supports event masking as well as spurious
>> event delivery mitigation.
>> +
>> +/*
>> + *---------
>> + * The following structures represent data that is shared across
>> boundaries
>> + * which may be quite disparate from one another (e.g. Windows vs
>> Linux,
>> + * 32 vs 64 bit, etc). Therefore, care has been taken to make sure
>> they
>> + * present data in a manner that is independent of the environment.
>> + *-----------
>> + */
>> +
>> +#define SHM_SIGNAL_MAGIC 0x58fa39df
>> +#define SHM_SIGNAL_VER 1
>> +
>> +struct shm_signal_irq {
>> + __u8 enabled;
>> + __u8 pending;
>> + __u8 dirty;
>> +};
>>
>
> Some ABIs may choose to pad this, suggest explicit padding.

Yeah, good idea. What is the official way to do this these days? Are
GCC pragmas allowed?

>
>> +
>> +enum shm_signal_locality {
>> + shm_locality_north,
>> + shm_locality_south,
>> +};
>> +
>> +struct shm_signal_desc {
>> + __u32 magic;
>> + __u32 ver;
>> + struct shm_signal_irq irq[2];
>> +};
>>
>
> Similarly, this should be padded to 0 (mod 8).
Ack

>
> Instead of versions, I prefer feature flags which can be independently
> enabled or disabled.

Totally agreed. If you look, most of the ABI has a type of "NEGCAP"
(negotiate capabilities) feature. The version number is a contingency
plan in case I still have to break it for whatever reason. I will
always opt for the feature bits over bumping the version when its
feasible (especially if/when this is actually in the field).

>
>> +
>> +/* --- END SHARED STRUCTURES --- */
>> +
>> +#ifdef __KERNEL__
>> +
>> +#include <linux/interrupt.h>
>> +
>> +struct shm_signal_notifier {
>> + void (*signal)(struct shm_signal_notifier *);
>> +};
>>
>
> This means "->inject() has been called from the other side"?

Yep
>
> (reading below I see this is so. not used to reading well commented
> code... :)

:)

>
>> +
>> +struct shm_signal;
>> +
>> +struct shm_signal_ops {
>> + int (*inject)(struct shm_signal *s);
>> + void (*fault)(struct shm_signal *s, const char *fmt, ...);
>>
>
> Eww. Must we involve strings and printf formats?

This is still somewhat of a immature part of the design. Its supposed
to be used so that by default, its a panic. But on the host side, we
can do something like inject a machine-check. That way malicious/broken
guests cannot (should not? ;) be able to take down the host. Note today
I do not map this to anything other than the default panic, so this
needs some love.

But given the asynchronous nature of the fault, I want to be sure we
have decent accounting to avoid bug reports like "silent MCE kills the
guest" ;) At least this way, we can log the fault string somewhere to
get a clue.

>
>> + void (*release)(struct shm_signal *s);
>> +};
>> +
>> +/*
>> + * signaling protocol:
>> + *
>> + * each side of the shm_signal has an "irq" structure with the
>> following
>> + * fields:
>> + *
>> + * - enabled: controlled by shm_signal_enable/disable() to
>> mask/unmask
>> + * the notification locally
>> + * - dirty: indicates if the shared-memory is dirty or clean.
>> This
>> + * is updated regardless of the enabled/pending state
>> so that
>> + * the state is always accurately tracked.
>> + * - pending: indicates if a signal is pending to the remote locale.
>> + * This allows us to determine if a
>> remote-notification is
>> + * already in flight to optimize spurious
>> notifications away.
>> + */
>>
>
> When you overlay a ring on top of this, won't the ring indexes convey
> the same information as ->dirty?

I agree that the information may be redundant with components of the
broader shm state. However, we need this state at this level of scope
in order to function optimally, so I dont think its a huge deal to have
this here as well. Afterall, the shm_signal library can only assess its
internal state. We would have to teach it how to glean the broader
state through some mechanism otherwise (callback, perhaps), but I don't
think its worth it.

-Greg

>
>

Attachments:

signature.asc (257.00 B)
OpenPGP digital signature

2009-03-31 20:59:01

by Avi Kivity

[permalink] [raw]

Subject: Re: [RFC PATCH 15/17] kvm: add dynamic IRQ support

Gregory Haskins wrote:
>> Won't this have scaling issues? One IRQ means one target vcpu.
>> Whereas I'd like virtio devices to span multiple queues, each queue
>> with its own MSI IRQ.
>>
> Hmm..you know I hadnt really thought of it that way, but you have a
> point. To clarify, my design actually uses one IRQ per "eventq", where
> we can have an arbitrary number of eventq's defined (note: today I only
> define one eventq, however). An eventq is actually a shm-ring construct
> where I can pass events up to the host like "device added" or "ring X
> signaled". Each individual device based virtio-ring would then
> aggregates "signal" events onto this eventq mechanism to actually inject
> events to the host. Only the eventq itself injects an actual IRQ to the
> assigned vcpu.
>

You will get get cachelines bounced around when events from different
devices are added to the queue. On the plus side, a single injection
can contain interrupts for multiple devices.

I'm not sure how useful this coalescing is; certainly you will never see
it on microbenchmarks, but that doesn't mean it's not useful.

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

2009-03-31 21:05:36

by Avi Kivity

[permalink] [raw]

Subject: Re: [RFC PATCH 01/17] shm-signal: shared-memory signals

Gregory Haskins wrote:
>>> +struct shm_signal_irq {
>>> + __u8 enabled;
>>> + __u8 pending;
>>> + __u8 dirty;
>>> +};
>>>
>>>
>> Some ABIs may choose to pad this, suggest explicit padding.
>>
>
> Yeah, good idea. What is the official way to do this these days? Are
> GCC pragmas allowed?
>
>

I just add a __u8 pad[5] in such cases.

>>> +
>>> +struct shm_signal;
>>> +
>>> +struct shm_signal_ops {
>>> + int (*inject)(struct shm_signal *s);
>>> + void (*fault)(struct shm_signal *s, const char *fmt, ...);
>>>
>>>
>> Eww. Must we involve strings and printf formats?
>>
>
> This is still somewhat of a immature part of the design. Its supposed
> to be used so that by default, its a panic. But on the host side, we
> can do something like inject a machine-check. That way malicious/broken
> guests cannot (should not? ;) be able to take down the host. Note today
> I do not map this to anything other than the default panic, so this
> needs some love.
>
> But given the asynchronous nature of the fault, I want to be sure we
> have decent accounting to avoid bug reports like "silent MCE kills the
> guest" ;) At least this way, we can log the fault string somewhere to
> get a clue.
>

I see.

This raises a point I've been thinking of - the symmetrical nature of
the API vs the assymetrical nature of guest/host or user/kernel
interfaces. This is most pronounced in ->inject(); in the host->guest
direction this is async (host can continue processing while the guest is
handling the interrupt), whereas in the guest->host direction it is
synchronous (the guest is blocked while the host is processing the call,
unless the host explicitly hands off work to a different thread).

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

2009-04-01 06:09:36

by Rusty Russell

2009-04-01 16:59:42

by Andi Kleen

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

On Wed, Apr 01, 2009 at 10:19:49AM -0400, Gregory Haskins wrote:
> >>
> >
> > But surely you must have some specific use case in mind? Something
> > that it does better than the various methods that are available
> > today. Or rather there must be some problem you're trying
> > to solve. I'm just not sure what that problem exactly is.
> >
> Performance. We are trying to create a high performance IO infrastructure.

Ok. So the goal is to bypass user space qemu completely for better
performance. Can you please put this into the initial patch
description?

> So the administrator can then set these attributes as
> desired to manipulate the configuration of the instance of the device,
> on a per device basis.

How would the guest learn of any changes in there?

I think the interesting part would be how e.g. a vnet device
would be connected to the outside interfaces.

> So the admin would instantiate this "vdisk" device and do:
>
> 'echo /path/to/my/exported/disk.dat > /sys/vbus/devices/foo/src_path'

So it would act like a loop device? Would you reuse the loop device
or write something new?

How about VFS mount name spaces?

-Andi
--
[email protected] -- Speaking for myself only.

2009-04-01 18:46:20

by Anthony Liguori

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

2009-04-01 20:28:07

Anthony Liguori wrote:
> Andi Kleen wrote:
>> On Wed, Apr 01, 2009 at 10:19:49AM -0400, Gregory Haskins wrote:
>>
>>>>>
>>>> But surely you must have some specific use case in mind? Something
>>>> that it does better than the various methods that are available
>>>> today. Or rather there must be some problem you're trying
>>>> to solve. I'm just not sure what that problem exactly is.
>>>>
>>> Performance. We are trying to create a high performance IO
>>> infrastructure.
>>>
>>
>> Ok. So the goal is to bypass user space qemu completely for better
>> performance. Can you please put this into the initial patch
>> description?
>>
>
> FWIW, there's nothing that prevents in-kernel back ends with virtio so
> vbus certainly isn't required for in-kernel backends.

I think there is a slight disconnect here. This is *exactly* what I am
trying to do. You can of course do this many ways, and I am not denying
it could be done a different way than the path I have chosen. One
extreme would be to just slam a virtio-net specific chunk of code
directly into kvm on the host. Another extreme would be to build a
generic framework into Linux for declaring arbitrary IO types,
integrating it with kvm (as well as other environments such as lguest,
userspace, etc), and building a virtio-net model on top of that.

So in case it is not obvious at this point, I have gone with the latter
approach. I wanted to make sure it wasn't kvm specific or something
like pci specific so it had the broadest applicability to a range of
environments. So that is why the design is the way it is. I understand
that this approach is technically "harder/more-complex" than the "slam
virtio-net into kvm" approach, but I've already done that work. All we
need to do now is agree on the details ;)

>
>
> That said, I don't think we're bound today by the fact that we're in
> userspace.
You will *always* be bound by the fact that you are in userspace. Its
purely a question of "how much" and "does anyone care". Right now,
the anwer is "a lot (roughly 45x slower)" and "at least Greg's customers
do". I have no doubt that this can and will change/improve in the
future. But it will always be true that no matter how much userspace
improves, the kernel based solution will always be faster. Its simple
physics. I'm cutting out the middleman to ultimately reach the same
destination as the userspace path, so userspace can never be equal.

I agree that the "does anyone care" part of the equation will approach
zero as the latency difference shrinks across some threshold (probably
the single microsecond range), but I will believe that is even possible
when I see it ;)

Regards,
-Greg

Attachments:

signature.asc (257.00 B)
OpenPGP digital signature

2009-04-01 21:09:46

by Gregory Haskins

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Chris Wright wrote:
> And more stuff in the kernel can come at the potential cost of weakening
> protection/isolation.
>
Note that the design of vbus should prevent any weakening...though if
you see a hole, please point it out.

(On that front, note that I still have some hardening to do, such as not
calling BUG_ON() in venet-tap if the ring is in a funk, etc)

Regards,
-Greg

Attachments:

Gregory Haskins wrote:

>> virtio is already non-kvm-specific (lguest uses it) and
>> non-pci-specific (s390 uses it).
>>
>
> Ok, then to be more specific, I need it to be more generic than it
> already is. For instance, I need it to be able to integrate with
> shm_signals.

Why?

>> If you have a good exit mitigation scheme you can cut exits by a
>> factor of 100; so the userspace exit costs are cut by the same
>> factor. If you have good copyless networking APIs you can cut the
>> cost of copies to zero (well, to the cost of get_user_pages_fast(),
>> but a kernel solution needs that too).
>>
>
> "exit mitigation' schemes are for bandwidth, not latency. For latency
> it all comes down to how fast you can signal in both directions. If
> someone is going to do a stand-alone request-reply, its generally always
> going to be at least one hypercall and one rx-interrupt. So your speed
> will be governed by your signal path, not your buffer bandwidth.
>

The userspace path is longer by 2 microseconds (for two additional
heavyweight exits) and a few syscalls. I don't think that's worthy of
putting all the code in the kernel.

--
error compiling committee.c: too many arguments to function

2009-04-02 11:48:45

by Avi Kivity

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Gregory Haskins wrote:

>> There is no choice. Exiting from the guest to the kernel to userspace
>> is prohibitively expensive, you can't do that on every packet.
>>
>>
>
> Now you are making my point ;) This is part of the cost of your
> signaling path, and it directly adds to your latency time.

It adds a microsecond. The kvm overhead of putting things in userspace
is low enough, I don't know why people keep mentioning it. The problem
is the kernel/user networking interfaces.

> You can't
> buffer packets here if the guest is only going to send one and wait for
> a response and expect that to perform well. And this is precisely what
> drove me to look at avoiding going back to userspace in the first place.
>

We're not buffering any packets. What we lack is a way to tell the
guest that we're done processing all packets in the ring (IOW, re-enable
notifications).

--
error compiling committee.c: too many arguments to function

2009-04-02 11:59:56

by Avi Kivity

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Gregory Haskins wrote:

>> Why does a kernel solution not need to know when a packet is transmitted?
>>
>>
>
> You do not need to know when the packet is copied (which I currently
> do). You only need it for zero-copy (of which I would like to support,
> but as I understand it there are problems with the reliability of proper
> callback (i.e. skb->destructor).
>
> Its "fire and forget" :)
>

It's more of a "schedule and forget" which I think brings you the win.
The host disables notifications and schedules the actual tx work (rx
from the host's perspective). So now the guest and host continue
producing and consuming packets in parallel. So long as the guest is
faster (due to the host being throttled?), notifications continue to be
disabled.

If you changed your rx_isr() to process the packets immediately instead
of scheduling, I think throughput would drop dramatically.

Mark had a similar change for virtio. Mark?

--
error compiling committee.c: too many arguments to function

2009-04-02 12:13:43

by Rusty Russell

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
> You do not need to know when the packet is copied (which I currently
> do). You only need it for zero-copy (of which I would like to support,
> but as I understand it there are problems with the reliability of proper
> callback (i.e. skb->destructor).

But if you have a UP guest, there will *never* be another packet in the queue
at this point, since it wasn't running.

As Avi said, you can do the processing in another thread and go back to the
guest; lguest pre-virtio did a hacky "weak" wakeup to ensure the guest ran
again before the thread did for exactly this kind of reason.

While Avi's point about a "powerful enough userspace API" is probably valid,
I don't think it's going to happen. It's almost certainly less code to put a
virtio_net server in the kernel, than it is to create such a powerful
interface (see vringfd & tap). And that interface would have one user in
practice.

So, let's roll out a kernel virtio_net server. Anyone?
Rusty.

2009-04-02 12:20:37

by Gregory Haskins

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Avi Kivity wrote:
> Gregory Haskins wrote:
>
>
>
>>> virtio is already non-kvm-specific (lguest uses it) and
>>> non-pci-specific (s390 uses it).
>>>
>>
>> Ok, then to be more specific, I need it to be more generic than it
>> already is. For instance, I need it to be able to integrate with
>> shm_signals.
>
> Why?
Well, shm_signals is what I designed to be the event mechanism for vbus
devices. One of the design criteria of shm_signal is that it should
support a variety of environments, such as kvm, but also something like
userspace apps. So I cannot make assumptions about things like "pci
interrupts", etc.

So if I want to use it in vbus, virtio-ring has to be able to use them,
as opposed to what it does today. Part of this would be a natural fit
for the "kick()" callback in virtio, but there are other problems. For
one, virtio-ring (IIUC) does its own event-masking directly in the
virtio metadata. However, really I want the higher layer ring-overlay
to do its masking in terms of the lower-layered shm_signal in order to
work the way I envision this stuff. If you look at the IOQ
implementation, this is exactly what it does.

To be clear, and Ive stated this in the past: venet is just an example
of this generic, in-kernel concept. We plan on doing much much more
with all this. One of the things we are working on is have userspace
clients be able to access this too, with an ultimately goal of
supporting things like having guest-userspace doing bypass, rdma, etc.
We are not there yet, though...only the kvm-host to guest kernel is
currently functional and is thus the working example.

I totally "get" the attraction to doing things in userspace. Its
contained, naturally isolated, easily supports migration, etc. Its also
a penalty. Bare-metal userspace apps have a direct path to the kernel
IO. I want to give guest the same advantage. Some people will care
more about things like migration than performance, and that is fine.
But others will certainly care more about performance, and that is what
we are trying to address.

>
>
>
>>> If you have a good exit mitigation scheme you can cut exits by a
>>> factor of 100; so the userspace exit costs are cut by the same
>>> factor. If you have good copyless networking APIs you can cut the
>>> cost of copies to zero (well, to the cost of get_user_pages_fast(),
>>> but a kernel solution needs that too).
>>>
>>
>> "exit mitigation' schemes are for bandwidth, not latency. For latency
>> it all comes down to how fast you can signal in both directions. If
>> someone is going to do a stand-alone request-reply, its generally always
>> going to be at least one hypercall and one rx-interrupt. So your speed
>> will be governed by your signal path, not your buffer bandwidth.
>>
>
> The userspace path is longer by 2 microseconds (for two additional
> heavyweight exits) and a few syscalls. I don't think that's worthy of
> putting all the code in the kernel.

By your own words, the exit to userspace is "prohibitively expensive",
so that is either true or its not. If its 2 microseconds, show me. We
need the rtt time to go from a "kick" PIO all the way to queue a packet
on the egress hardware and return. That is going to define your
latency. If you can do this such that you can do something like ICMP
ping in 65us (or anything close to a few dozen microseconds of this),
I'll shut-up about how much I think the current path sucks ;) Even so,
I still propose the concept of a frame-work for in-kernel devices for
all the other reasons I mentioned above.

-Greg

Attachments:

signature.asc (257.00 B)
OpenPGP digital signature

2009-04-02 12:28:46

Gregory Haskins wrote:
> Rusty Russell wrote:
>
>> On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
>>
>>
>>> You do not need to know when the packet is copied (which I currently
>>> do). You only need it for zero-copy (of which I would like to support,
>>> but as I understand it there are problems with the reliability of proper
>>> callback (i.e. skb->destructor).
>>>
>>>
>> But if you have a UP guest,
>>
>
> I assume you mean UP host ;)
>
>

I think Rusty did mean a UP guest, and without schedule-and-forget.

> Hmm..well I was hoping to be able to work with you guys to make my
> proposal fit this role. If there is no interest in that, I hope that my
> infrastructure itself may still be considered for merging (in *some*
> tree, not -kvm per se) as I would prefer to not maintain it out of tree
> if it can be avoided.

The problem is that we already have virtio guest drivers going several
kernel versions back, as well as Windows drivers. We can't keep
changing the infrastructure under people's feet.

--
error compiling committee.c: too many arguments to function

2009-04-02 13:08:27

by Avi Kivity

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Gregory Haskins wrote:
> Avi Kivity wrote:
>
>> My 'prohibitively expensive' is true only if you exit every packet.
>>
>>
>>
>
> Understood, but yet you need to do this if you want something like iSCSI
> READ transactions to have as low-latency as possible.
>

Dunno, two microseconds is too much? The wire imposes much more.

--
error compiling committee.c: too many arguments to function

2009-04-02 13:20:15

Avi Kivity wrote:
> Gregory Haskins wrote:
>> Avi Kivity wrote:
>>
>>> My 'prohibitively expensive' is true only if you exit every packet.
>>>
>>>
>>>
>>
>> Understood, but yet you need to do this if you want something like iSCSI
>> READ transactions to have as low-latency as possible.
>>
>
> Dunno, two microseconds is too much? The wire imposes much more.
>

No, but thats not what we are talking about. You said signaling on
every packet is prohibitively expensive. I am saying signaling on every
packet is required for decent latency. So is it prohibitively expensive
or not?

I think most would agree that adding 2us is not bad, but so far that is
an unproven theory that the IO path in question only adds 2us. And we
are not just looking at the rate at which we can enter and exit the
guest...we need the whole path...from the PIO kick to the dev_xmit() on
the egress hardware, to the ingress and rx-injection. This includes any
and all penalties associated with the path, even if they are imposed by
something like the design of tun-tap.

Right now its way way way worse than 2us. In fact, at my last reading
this was more like 3060us (3125-65). So shorten that 3125 to 67 (while
maintaining line-rate) and I will be impressed. Heck, shorten it to
80us and I will be impressed.

-Greg

Attachments:

signature.asc (257.00 B)
OpenPGP digital signature

2009-04-02 13:46:22

by Avi Kivity

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Gregory Haskins wrote:
> Avi Kivity wrote:
>
>> Gregory Haskins wrote:
>>
>>> Avi Kivity wrote:
>>>
>>>
>>>> My 'prohibitively expensive' is true only if you exit every packet.
>>>>
>>>>
>>>>
>>>>
>>> Understood, but yet you need to do this if you want something like iSCSI
>>> READ transactions to have as low-latency as possible.
>>>
>>>
>> Dunno, two microseconds is too much? The wire imposes much more.
>>
>>
>
> No, but thats not what we are talking about. You said signaling on
> every packet is prohibitively expensive. I am saying signaling on every
> packet is required for decent latency. So is it prohibitively expensive
> or not?
>

We're heading dangerously into the word-game area. Let's not do that.

If you have a high throughput workload with many packets per seconds
then an exit per packet (whether to userspace or to the kernel) is
expensive. So you do exit mitigation. Latency is not important since
the packets are going to sit in the output queue anyway.

If you have a request-response workload with the wire idle and latency
critical, then there's no problem having an exit per packet because (a)
there aren't that many packets and (b) the guest isn't doing any
batching, so guest overhead will swamp the hypervisor overhead.

If you have a low latency request-response workload mixed with a high
throughput workload, then you aren't going to get low latency since your
low latency packets will sit on the queue behind the high throughput
packets. You can fix that with multiqueue and then you're back to one
of the scenarios above.

> I think most would agree that adding 2us is not bad, but so far that is
> an unproven theory that the IO path in question only adds 2us. And we
> are not just looking at the rate at which we can enter and exit the
> guest...we need the whole path...from the PIO kick to the dev_xmit() on
> the egress hardware, to the ingress and rx-injection. This includes any
> and all penalties associated with the path, even if they are imposed by
> something like the design of tun-tap.
>

Correct, we need to look at the whole path. That's why the wishing well
is clogged with my 'give me a better userspace interface' emails.

> Right now its way way way worse than 2us. In fact, at my last reading
> this was more like 3060us (3125-65). So shorten that 3125 to 67 (while
> maintaining line-rate) and I will be impressed. Heck, shorten it to
> 80us and I will be impressed.
>

The 3060us thing is a timer, not cpu time. We aren't starting a JVM for
each packet. We could remove it given a notification API, or
duplicating the sched-and-forget thing, like Rusty did with lguest or
Mark with qemu.

--
error compiling committee.c: too many arguments to function

2009-04-02 14:03:24

by Gregory Haskins

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Avi Kivity wrote:
> Gregory Haskins wrote:
>> Avi Kivity wrote:
>>
>>> Gregory Haskins wrote:
>>>
>>>> Rusty Russell wrote:
>>>>
>>>>
>>>>> On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
>>>>>
>>>>>> You do not need to know when the packet is copied (which I currently
>>>>>> do). You only need it for zero-copy (of which I would like to
>>>>>> support,
>>>>>> but as I understand it there are problems with the reliability of
>>>>>> proper
>>>>>> callback (i.e. skb->destructor).
>>>>>>
>>>>> But if you have a UP guest,
>>>>>
>>>> I assume you mean UP host ;)
>>>>
>>>>
>>> I think Rusty did mean a UP guest, and without schedule-and-forget.
>>>
>> That doesnt make sense to me, tho. All the testing I did was a UP
>> guest, actually. Why would I be constrained to run without the
>> scheduling unless the host was also UP?
>>
>
> You aren't constrained. And your numbers show it works.
>
>>>
>>> The problem is that we already have virtio guest drivers going several
>>> kernel versions back, as well as Windows drivers. We can't keep
>>> changing the infrastructure under people's feet.
>>>
>>
>> Well, IIUC the virtio code itself declares the ABI as unstable, so there
>> technically *is* an out if we really wanted one. But I certainly
>> understand the desire to not change this ABI if at all possible, and
>> thus the resistance here.
>>
>
> virtio is a stable ABI.

Dang! Scratch that.
>
>> However, theres still the possibility we can make this work in an ABI
>> friendly way with cap-bits, or other such features. For instance, the
>> virtio-net driver could register both with pci and vbus-proxy and
>> instantiate a device with a slightly different ops structure for each or
>> something. Alternatively we could write a host-side shim to expose vbus
>> devices as pci devices or something like that.
>>
>
> Sounds complicated...

Well, the first solution would be relatively trivial...at least on the
guest side. All the other infrastructure is done and included in the
series I sent out. The changes to the virtio-net driver on the guest
itself would be minimal. The bigger effort would be converting
venet-tap to use virtio-ring instead of IOQ. But this would arguably be
less work than starting a virtio-net backend module from scratch because
you would have to not only code up the entire virtio-net backend, but
also all the pci emulation and irq routing stuff that is required (and
is already done by the vbus infrastructure). Here all the major pieces
are in place, just the xmit and rx routines need to be converted to
virtio-isms.

For the second option, I agree. Its probably too nasty and it would be
better if there was just either a virtio-net to kvm-host hack, or a more
pci oriented version of a vbus-like framework.

That said, there is certainly nothing wrong with having an alternate
option. There is plenty of precedent for having different drivers for
different subsystems, etc, even if there is overlap. Heck, even KVM has
realtek, e1000, and virtio-net, etc. Would our kvm community be willing
to work with me to get these patches merged? I am perfectly willing to
maintain them. That said, the general infrastructure should probably
not live in -kvm (perhaps -tip, -mm, or -next, etc is more
appropriate). So a good plan might be to shoot for the core going into
a more general upstream tree. When/if that happens, then the kvm
community could consider the kvm specific parts, etc. I realize this is
all pending review acceptance by everyone involved...

-Greg

Attachments:

signature.asc (257.00 B)
OpenPGP digital signature

2009-04-02 14:22:44

Avi Kivity wrote:
> Gregory Haskins wrote:
>>> vbus (if I understand it right) is a whole package of things:
>>>
>>> - a way to enumerate, discover, and manage devices
>>>
>>
>> Yes
>>
>>> That part duplicates PCI
>>>
>>
>> Yes, but the important thing to point out is it doesn't *replace*
>> PCI. It simply an alternative.
>>
>
> Does it offer substantial benefits over PCI? If not, it's just extra
> code.

First of all, do you think I would spend time designing it if I didn't
think so? :)

Second of all, I want to use vbus for other things that do not speak PCI
natively (like userspace for instance...and if I am gleaning this
correctly, lguest doesnt either).

PCI sounds good at first, but I believe its a false economy. It was
designed, of course, to be a hardware solution, so it carries all this
baggage derived from hardware constraints that simply do not exist in a
pure software world and that have to be emulated. Things like the fixed
length and centrally managed PCI-IDs, PIO config cycles, BARs,
pci-irq-routing, etc. While emulation of PCI is invaluable for
executing unmodified guest, its not strictly necessary from a
paravirtual software perspective...PV software is inherently already
aware of its context and can therefore use the best mechanism
appropriate from a broader selection of choices.

If we insist that PCI is the only interface we can support and we want
to do something, say, in the kernel for instance, we have to have either
something like the ICH model in the kernel (and really all of the pci
chipset models that qemu supports), or a hacky hybrid userspace/kernel
solution. I think this is what you are advocating, but im sorry. IMO
that's just gross and unecessary gunk. Lets stop beating around the
bush and just define the 4-5 hypercall verbs we need and be done with
it. :)

FYI: The guest support for this is not really *that* much code IMO.

drivers/vbus/proxy/Makefile | 2
drivers/vbus/proxy/kvm.c | 726 +++++++++++++++++

and plus, I'll gladly maintain it :)

I mean, its not like new buses do not get defined from time to time.
Should the computing industry stop coming up with new bus types because
they are afraid that the windows ABI only speaks PCI? No, they just
develop a new driver for whatever the bus is and be done with it. This
is really no different.

>
> Note that virtio is not tied to PCI, so "vbus is generic" doesn't count.
Well, preserving the existing virtio-net on x86 ABI is tied to PCI,
which is what I was referring to. Sorry for the confusion.

>
>>> and it would be pretty hard to convince me we need to move to
>>> something new
>>>
>>
>> But thats just it. You don't *need* to move. The two can coexist side
>> by side peacefully. "vbus" just ends up being another device that may
>> or may not be present, and that may or may not have devices on it. In
>> fact, during all this testing I was booting my guest with "eth0" as
>> virtio-net, and "eth1" as venet. The both worked totally fine and
>> harmoniously. The guest simply discovers if vbus is supported via a
>> cpuid feature bit and dynamically adds it if present.
>>
>
> I meant, move the development effort, testing, installed base, Windows
> drivers.

Again, I will maintain this feature, and its completely off to the
side. Turn it off in the config, or do not enable it in qemu and its
like it never existed. Worst case is it gets reverted if you don't like
it. Aside from the last few kvm specific patches, the rest is no
different than the greater linux environment. E.g. if I update the
venet driver upstream, its conceptually no different than someone else
updating e1000, right?

>
>>
>>> . virtio-pci (a) works,
>>>
>> And it will continue to work
>>
>
> So why add something new?

I was hoping this was becoming clear by now, but apparently I am doing a
poor job of articulating things. :( I think we got bogged down in the
802.x performance discussion and lost sight of what we are trying to
accomplish with the core infrastructure.

So this core vbus infrastructure is for generic, in-kernel IO models.
As a first pass, we have implemented a kvm-connector, which lets kvm
guest kernels have access to the bus. We also have a userspace
connector (which I haven't pushed yet due to remaining issues being
ironed out) which allows userspace applications to interact with the
devices as well. As a prototype, we built "venet" to show how it all works.

In the future, we want to use this infrastructure to build IO models for
various things like high performance fabrics and guest bypass
technologies, etc. For instance, guest userspace connections to RDMA
devices in the kernel, etc.

>
>>
>>> (b) works on Windows.
>>>
>>
>> virtio will continue to work on windows, as well. And if one of my
>> customers wants vbus support on windows and is willing to pay us to
>> develop it, we can support *it* there as well.
>>
>
> I don't want to develop and support both virtio and vbus. And I
> certainly don't want to depend on your customers.

So don't. Ill maintain the drivers and the infrastructure. All we are
talking here is the possible acceptance of my kvm-connector patches
*after* the broader LKML community accepts the core infrastructure,
assuming that happens.

You can always just state that you do not support enabling the feature.
Bug reports with it enabled go to me, etc.

If that is still not acceptable and you are ultimately not interested in
any kind of merge/collaboration: At the very least, I hope we can get
some very trivial patches in for registering things like the
KVM_CAP_VBUS bits for vbus so I can present a stable ABI to anyone
downstream from me. Those things have been shifting on me a lot lately ;)

>
>
>>> - a different way of doing interrupts
>>>
>> Yeah, but this is ok. And I am not against doing that mod we talked
>> about earlier where I replace dynirq with a pci shim to represent the
>> vbus. Question about that: does userspace support emulation of MSI
>> interrupts?
>
> Yes, this is new. See the interrupt routing stuff I mentioned. It's
> probably only in kvm.git, not even in 2.6.30.
Cool, will check out, thanks.

>
>> I would probably prefer it if I could keep the vbus IRQ (or
>> IRQs when I support MQ) from being shared. It seems registering the
>> vbus as an MSI device would be more conducive to avoiding this.
>>
>
> I still think you want one MSI per device rather than one MSI per
> vbus, to avoid scaling problems on large guest. After Herbert's let
> loose on the code, one MSI per queue.

This is trivial for me to support with just a few tweaks to the kvm
host/guest connector patches.

>
>
>
>>> - a different ring layout, and splitting notifications from the ring
>>>
>> Again, virtio will continue to work. And if we cannot find a way to
>> collapse virtio and ioq together in a way that everyone agrees on, there
>> is no harm in having two. I have no problem saying I will maintain
>> IOQ. There is plenty of precedent for multiple ways to do the same
>> thing.
>>
>
> IMO we should just steal whatever makes ioq better, and credit you in
> some file no one reads. We get backwards compatibility, Windows
> support, continuity, etc.
>
>>> I don't see the huge win here
>>>
>>> - placing the host part in the host kernel
>>>
>>> Nothing vbus-specific here.
>>>
>>
>> Well, it depends on what you want. Do you want a implementation that is
>> virtio-net, kvm, and pci specific while being hardcoded in?
>
> No. virtio is already not kvm or pci specific. Definitely all the
> pci emulation parts will remain in user space.

blech :)

-Greg

Attachments:

signature.asc (257.00 B)
OpenPGP digital signature

2009-04-02 18:11:23

by Avi Kivity

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Gregory Haskins wrote:
>
>>> 2) the vbus-proxy and kvm-guest patch go away
>>> 3) the kvm-host patch changes to work with coordination from the
>>> userspace-pci emulation for things like MSI routing
>>> 4) qemu will know to create some MSI shim 1:1 with whatever it
>>> instantiates on the bus (and can communicate changes
>>>
>>>
>> Don't userstand. What's this MSI shim?
>>
>
> Well, if the device model was an object in vbus down in the kernel, yet
> PCI emulation was up in qemu, presumably we would want something to
> handle things like PCI config-cycles up in userspace. Like, for
> instance, if the guest re-routes the MSI. The shim/proxy would handle
> the config-cycle, and then turn around and do an ioctl to the kernel to
> configure the change with the in-kernel device model (or the irq
> infrastructure, as required).
>

Right, this is how it should work. All the gunk in userspace.

> But, TBH, I haven't really looked into whats actually required to make
> this work yet. I am just spitballing to try to find a compromise.
>

One thing I thought of trying to get this generic is to use file
descriptors as irq handles. So:

- userspace exposes a PCI device (same as today)
- guest configures its PCI IRQ (using MSI if it supports it)
- userspace handles this by calling KVM_IRQ_FD which converts the irq to
a file descriptor
- userspace passes this fd to the kernel, or another userspace process
- end user triggers guest irqs by writing to this fd

We could do the same with hypercalls:

- guest and host userspace negotiate hypercall use through PCI config space
- userspace passes an fd to the kernel
- whenever the guest issues an hypercall, the kernel writes the
arguments to the fd
- other end (in kernel or userspace) processes the hypercall

> No, you are confusing the front-end and back-end again ;)
>
> The back-end remains, and holds the device models as before. This is
> the "vbus core". Today the front-end interacts with the hypervisor to
> render "vbus" specific devices. The proposal is to eliminate the
> front-end, and have the back end render the objects on the bus as PCI
> devices to the guest. I am not sure if I can make it work, yet. It
> needs more thought.
>

It seems to me this already exists, it's the qemu device model.

The host kernel doesn't need any knowledge of how the devices are
connected, even if it does implement some of them.

>> . I don't think you've yet set down what its advantages are. Being
>> pure and clean doesn't count, unless you rip out PCI from all existing
>> installed hardware and from Windows.
>>
>
> You are being overly dramatic. No one has ever said we are talking
> about ripping something out. In fact, I've explicitly stated that PCI
> can coexist peacefully. Having more than one bus in a system is
> certainly not without precedent (PCI, scsi, usb, etc).
>
> Rather, PCI is PCI, and will always be. PCI was designed as a
> software-to-hardware interface. It works well for its intention. When
> we do full emulation of guests, we still do PCI so that all that
> software that was designed to work software-to-hardware still continue
> to work, even though technically its now software-to-software. When we
> do PV, on the other hand, we no longer need to pretend it is
> software-to-hardware. We can continue to use an interface designed for
> software-to-hardware if we choose, or we can use something else such as
> an interface designed specifically for software-to-software.
>
> As I have stated, PCI was designed with hardware constraints in mind.
> What if I don't want to be governed by those constraints?

I'd agree with all this if I actually saw a constraint in PCI. But I don't.

> What if I
> don't want an interrupt per device (I don't)?

Don't. Though I thing you do, even multiple interrupts per device.

> What do I need BARs for
> (I don't)?

Don't use them.

> Is a PCI PIO address relevant to me (no, hypercalls are more
> direct)? Etc. Its crap I dont need.
>

So use hypercalls.

> All I really need is a way to a) discover and enumerate devices,
> preferably dynamically (hotswap), and b) a way to communicate with those
> devices. I think you are overstating the the importance that PCI plays
> in (a), and are overstating the complexity associated with doing an
> alternative.

Given that we have PCI, why would we do an alternative?

It works, it works with Windows, the nasty stuff is in userspace. Why
expend effort on an alternative? Instead make it go faster.

> I think you are understating the level of hackiness
> required to continue to support PCI as we move to new paradigms, like
> in-kernel models.

The kernel need know nothing about PCI, so I don't see how you work this
out.

> And I think I have already stated that I can
> establish a higher degree of flexibility, and arguably, performance for
> (b).

You've stated it, but failed to provide arguments for it.

--
error compiling committee.c: too many arguments to function

2009-04-05 10:51:22

by Avi Kivity

[permalink] [raw]

Subject: Re: [RFC PATCH 00/17] virtual-bus

Gregory Haskins wrote:
>> You don't gain simplicity by adding things.
>>
>
> But you are failing to account for the fact that we still have to add
> something for PCI if we go with something like the in-kernel model. Its
> nice for the userspace side because a) it was already in qemu, and b) we
> need it for proper guest support. But we don't presumably have it for
> this new thing, so something has to be created (unless this support is
> somehow already there and I don't know it?)
>

No, a virtio server in the kernel would know nothing about PCI.
Userspace would handle the PCI interface and configure the kernel. That
way we can reuse the kernel part for lguest and s390.

>> Optimization:
>>
>> Most of PCI (in our context) deals with configuration. So removing it
>> doesn't optimize anything, unless you're counting hotplugs-per-second
>> or something.
>>
>
> Most, but not all ;) (Sorry, you left the window open on that one).
>
> What about IRQ routing?

That's already in the kernel.

> What if I want to coalesce interrupts to
> minimize injection overhead? How do I do that in PCI?
>

It has nothing to do with PCI. It has to do with the device/guest
protocol. And virtio already does that (badly, in the case of network tx).

> How do I route those interrupts in an arbitrarily nested fashion, say,
> to a guest userspace?
>

That's a guest problem. kvm delivers an interrupt; if the guest knows
how to service it in userspace, great.

> What about scale? What if Herbet decides to implement a 2048 ring MQ
> device ;) Theres no great way to do that in x86 with PCI, yet I can do
> it in vbus. (And yes, I know, this is ridiculous..just wanting to get
> you thinking)
>

I don't see why you can't do 2048 (or even 2049) rings with PCI. You'd
point some config space address at a 'ring descriptor table' and that's it.

>> There is be no problem supporting an in-kernel host virtio endpoint
>> with the existing guest/host ABI. Nothing in the ABI assumes the host
>> endpoint is in userspace. Nothing in the implementation requires us
>> to move any of the PCI stuff into the kernel.
>>
> Well, thats not really true. If the device is a PCI device, there is
> *some* stuff that has to go into the kernel. Not an ICH model or
> anything, but at least an ability to interact with userspace for
> config-space changes, etc.
>

Config space changes go to userspace anyway. You'd need an interface to
let userspace configure the kernel, but that's true for every device in
the kernel. And you don't want to let the guest configure the kernel
directly, you want userspace to be able to keep control of things.

>> To avoid reiterating, please be specific about these advantages.
>>
> We are both reading the same thread, right?
>

Using different languages?

>> Last time we measured, hypercall overhead was the same as pio
>> overhead. Both vmx and svm decode pio completely (except for string
>> pio ...)
>>
> Not on my woodcrests last time I looked, but I'll check again.
>

On woodcrests too. See vmx.c:handle_io().

>> True, PCI interrupts suck. But this was fixed with MSI. Why fix it
>> again?
>>
>
> As I stated, I don't like the constraints in place even by MSI (though
> that is definately a step in the right direction).
>

Which constraints?

> With vbus I can have a device that has an arbitrary number of shm
> regions (limited by memory, of course),

So you can with PCI.

> each with an arbitrarily routed
> signal path that is limited by a u64, even on x86.

There are still only 224 vectors per vcpu.

> Each region can be
> signaled bidirectionally and masked with a simple local memory write.
> They can be declared on the fly, allowing for the easy expression of
> things like nested devices or or other dynamic resources. The can be
> routed across various topologies, such as IRQs or posix signals, even
> across multiple hops in a single path.
>
> How do I do that in PCI?
>

Not what this nesting means. If I understand the rest, I think you can
do it.

> What does masking an interrupt look like?

It's a protocol between the device and the guest. PCI doesn't specify
it. So you can use a bit in shared memory if you like.

> Again, for the nested case?
>

What's that?

> Interrupt acknowledgment cycles?
>

Standard for the platform. Again it's outside the scope of PCI.

>>> One of my primary design objectives with vbus was to a) reduce the
>>> signaling as much as possible, and b) reduce the cost of signaling.
>>> That is why I do things like use explicit hypercalls, aggregated
>>> interrupts, bidir napi to mitigate signaling, the shm_signal::pending
>>> mitigation, and avoiding going to userspace by running in the kernel.
>>> All of these things together help to form what I envision would be a
>>> maximum performance transport. Not all of these tricks are
>>> interdependent (for instance, the bidir + full-duplex threading that I
>>> do can be done in userspace too, as discussed). They are just the
>>> collective design elements that I think we need to make a guest perform
>>> very close to its peak. That is what I am after.
>>>
>>>
>>>
>> None of these require vbus. They can all be done with PCI.
>>
> Well, first of all: Not really.

Really? I think every network card+driver do this bidir napi thing.
napi was invented for real network cards, IIUC.

> Second of all, even if you *could* do
> this all with PCI, its not really PCI anymore. So the question I have
> is: whats the value in still using it? For the discovery? Its not very
> hard to do discovery. I wrote that whole part in a few hours and it
> worked the first time I ran it.
>

Yes, for the discovery. And so it could work on all guests, not just
Linux 2.6.31+.

> What about that interrupt model I keep talking about? How do you work
> around that? How do I nest these to support bypass?
>

I'm lost, sorry.

>> What constraints? Please be specific.
>>
>
> Avi, I have been. Is this an exercise to see how much you can get me to
> type? ;)
>

I know I'd lose this, so no. I'm really puzzled what you think we'd
gain by departing from PCI (other than having a nice clean code base,
which I don't think helps because we get to maintain both PCI and the
new code base).

>> I'm not saying anything about what the advantages are worth and how
>> they compare to the cost. I'm asking what are the advantages. Please
>> don't just assert them into existence.
>>
>
> Thats an unfair statement, Avi. Now I would say you are playing word-games.
>

I genuinely don't see them. I'm not being deliberately stupid.

>> All of this overhead is incurred at configuration time. All the
>> complexity already exists
>>
>
> So you already have the ability to represent PCI devices that are in the
> kernel? Is this the device-assignment infrastructure? Cool! Wouldn't
> this still need to be adapted to work with software devices? If not,
> then I take back the statements that they both add more host code and
> agree that vbus is simply the one adding more.
>

Of course it would need to be adapted, but nothing in the core. For
example, virtio-net.c would need to communicate with its kernel
counterpart to tell it what it's configuration is, and to start and stop
it (so we could do live migration).

We wouldn't need to make any changes to hw/pci.c, for example.

It's similar to how the in-kernel lapic and ioapic are integrated with qemu.

>> so we gain nothing by adding a competing implementation. And making
>> the guest complex in order to simplify the host is a pretty bad
>> tradeoff considering we maintain one host but want to support many
>> guests.
>>
>> It's good to look forward, but in the vbus-dominated universe, what do
>> we have that we don't have now? Besides simplicity.
>>
>
> A unified framework for declaring virtual resources directly in the
> kernel, yet still retaining the natural isolation that we get in
> userspace.

That's not an advantage. "directly in the kernel" doesn't buy the user
anything.

> The ability to support guests that don't have PCI.

Already have that. See lguest and s390.

> The
> ability to support things that are not guests.

So would a PCI implementation, as long as PCI is only in userspace.

> The ability to support
> things that are not supported by PCI, like less hardware-centric signal
> path routing.

What's that?

> The ability to signal across more than just IRQs.

You can't, either with or without vbus. You have to honour guest cli.
You might do a Xen-like alternative implementation of interrupts, but
that's bound to be slow since you have to access guest stack directly
and switch stacks instead of letting the hardware do it for you. And of
course forget about Windows.

> The
> ability for nesting (e.g. guest-userspace talking to host-kernel, etc).
>

That's a guest problem. If the guest kernel gives guest userspace
access, guest userspace can have a go too, PCI or not.

In fact, even today guest userspace controls a PCI device - the X server
runs in userspace and talks to the cirrus PCI device.

> I recognize that this has no bearing on whether you, or anyone else
> cares about these features. But it certainly has features beyond what
> he have with PCI, and I hope that is clear now.
>

With the exception of "less hardware-centric signal path routing", which
I did not understand, I don't think you demonstrated any advantage.

>>> Ive already said this is low on my list, but it could always be added if
>>> someone cares that much
>>>
>>>
>> That's unreasonable. Windows is an important workload.
>>
>
> Well, this is all GPL, right. I mean, was KVM 100% complete when it was
> proposed? Accepted? I am hoping to get some help building the parts of
> this infrastructure from anyone interested in the community. If Windows
> support is truly important and someone cares, it will get built soon enough.
>
> I pushed it out now because I have enough working to be useful in of
> itself and to get a review. But its certainly not done.
>

You are proposing a major break from what we have now. While you've
demonstrated very nice performance numbers, it cannot be undertaken lightly.

This is how I see our options:

- continue to develop virtio, taking the performance improvements from venet

IMO this is the best course. We do what we have to do to get better
performance, perhaps by implementing a server in the kernel. The
Windows drivers continue to work. Linux 2.6.older+ continue to work.
Older hosts continue to work (with the userspace virtio
implementation). Performance improves.

- drop virtio, switch to vbus

That's probably the worst course. Windows drivers stop working until
further notice. Older hosts stop working. Older guests stop working.
The only combination that works is 2.6.31+ on 2.6.31+.

- move virtio to maintenance mode, start developing vbus

Older guests use virtio, older hosts use virtio, if we have a new guest
on new host we use vbus. Start porting the Windows drivers to vbus.
Start porting block drivers and host to vbus. Same for balloon.

While workable, it increases the maintenance burden significantly as
well as user confusion. I don't think we'd be justified in moving in
this direction unless there was a compelling reason, which I don't see
right now.
>
>
>> Of course we need to. RHEL 4/5 and their equivalents will live for a
>> long time as guests. Customers will expect good performance.
>>
>
> Okay, easy enough from my perspective. However, I didn't realize it was
> very common to backport new features to enterprise distros like this. I
> have a sneaking suspicion we wouldn't really need to worry about this as
> the project managers for those products would probably never allow it.
> But in the event that it was necessary, I think it wouldn't be horrendous.
>

As it happens, RHEL 5.3 has backported virtio drivers.

>>> So does virtio also do demand loading in the backend?
>>>
>> Given that it's entirely in userspace, yes.
>>
>
> Ah, right. How does that work our of curiosity? Do you have to do a
> syscall for every page you want to read?
>

No, you just read or write it through pointers. Syscalls that access
userspace work too (like read() or write()).

>>> Hmm. I suppose
>>> we could do this, but it will definitely affect the performance
>>> somewhat. I was thinking that the pages needed for the basic shm
>>> components should be minimal, so this is a good tradeoff to vmap them in
>>> and only demand load the payload.
>>>
>>>
>> This is negotiable :) I won't insist on it, only strongly recommend
>> it. copy_to_user() should be pretty fast.
>>
>
> It probably is, but generally we cant use it since we are not in the
> same context when we need to do the copy (copy_to/from_user assume
> "current" is proper).

Right.

> Thats ok, there are ways to do what you request
> without explicitly using c_t_u().
>

How?

If we can't, vmap() is fine.

--
error compiling committee.c: too many arguments to function

2009-04-05 14:13:48

by Anthony Liguori

[permalink] [raw]

On Thu, 2009-04-02 at 13:18 -0500, Anthony Liguori wrote:
> Avi Kivity wrote:
> > Anthony Liguori wrote:
> >>> I don't think we even need that to end this debate. I'm convinced
> >>> we have a bug somewhere. Even disabling TX mitigation, I see a ping
> >>> latency of around 300ns whereas it's only 50ns on the host. This
> >>> defies logic so I'm now looking to isolate why that is.
> >>
> >> I'm down to 90us. Obviously, s/ns/us/g above. The exec.c changes
> >> were the big winner... I hate qemu sometimes.
>
> Anyway, if we're able to send this many packets, I suspect we'll be able
> to also handle much higher throughputs without TX mitigation so that's
> what I'm going to look at now.

Anthony,

Any news on this? I'm anxious to see virtio-net performance on par with
the virtual-bus results. Thanks,

Alex