VMware Tools installer requires the upstream driver to be compiled
as a module in order to detect its presence and avoid installing
our own version on top of it. To avoid surprises with 2 versions
of the driver being installed and fighting with each other, let's
force the driver to be compiled as a module unless user selects
CONFIG_EMBEDDED.
Signed-off-by: Dmitry Torokhov <[email protected]>
---
Andrew,
Please consider forwarding this to Linus in your next batch.
Thanks,
Dmitry
drivers/misc/Kconfig | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index 26386a9..bb5b6ea 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -339,7 +339,7 @@ config TI_DAC7512
config VMWARE_BALLOON
tristate "VMware Balloon Driver"
- depends on X86
+ depends on X86 && (EMBEDDED || m)
help
This is VMware physical memory management driver which acts
like a "balloon" that can be inflated to reclaim physical pages
--
1.7.0
Dmitry Torokhov <[email protected]> wrote:
>
> VMware Tools installer requires the upstream driver to be compiled
> as a module in order to detect its presence and avoid installing
> our own version on top of it. To avoid surprises with 2 versions
> of the driver being installed and fighting with each other, let's
> force the driver to be compiled as a module unless user selects
> CONFIG_EMBEDDED.
>
*barf*
This surely is a problem in the installer and not the kernel? Can you
not nosey around in /sys/class/misc or where-ever your driver appears?
If it does not, then I would probably suggest a patch to your balloon
driver that dumps some details in there, including module version
information.
Eugh.
--
Alexander Clouter
.sigmonster says: Who goeth a-borrowing goeth a-sorrowing.
-- Thomas Tusser
On Tue, 29 June 2010 Alexander Clouter <[email protected]> wrote:
> Dmitry Torokhov <[email protected]> wrote:
> >
> > VMware Tools installer requires the upstream driver to be compiled
> > as a module in order to detect its presence and avoid installing
> > our own version on top of it. To avoid surprises with 2 versions
> > of the driver being installed and fighting with each other, let's
> > force the driver to be compiled as a module unless user selects
> > CONFIG_EMBEDDED.
> >
> *barf*
>
> This surely is a problem in the installer and not the kernel? Can you
> not nosey around in /sys/class/misc or where-ever your driver appears?
> If it does not, then I would probably suggest a patch to your balloon
> driver that dumps some details in there, including module version
> information.
>
> Eugh.
In addition, the installer may check under /sys/module/ for it (as for
any/most code that can be built as a module), even for built-in code.
(if balloon driver does not show up there when built-in it would be
better to get it to show up there)
Bruno
Hi Alexander,
On Tuesday, June 29, 2010 01:27:43 am Alexander Clouter wrote:
> Dmitry Torokhov <[email protected]> wrote:
> > VMware Tools installer requires the upstream driver to be compiled
> > as a module in order to detect its presence and avoid installing
> > our own version on top of it. To avoid surprises with 2 versions
> > of the driver being installed and fighting with each other, let's
> > force the driver to be compiled as a module unless user selects
> > CONFIG_EMBEDDED.
>
> *barf*
>
> This surely is a problem in the installer and not the kernel? Can you
> not nosey around in /sys/class/misc or where-ever your driver appears?
The driver does not "appear" anywhere at the moment as it does not export
any interfaces to userland. It only communicates with the hypervisor.
Unfortunately the kernel, with the exception of module parameters which
this driver does not have at the moment, does not populate
/sys/module/<modulename> for modules built into the kernel and so
installer can not rely on this data. Also, the module might be disabled
(blacklisted) by the system administrator and thus being absent from the
kernel. In such scenario we also so not want to install our version of the
driver.
> If it does not, then I would probably suggest a patch to your balloon
> driver that dumps some details in there, including module version
> information.
>
> Eugh.
Exactly, eugh. I do not believe that dumping some unnecessary data in
sysfs is better than making driver a module.
Also, besides the installer logic, we prefer having the driver being
compiled by default as a module so that we can deliver truly urgent
fixes to customers without need of recompiling the kernel/wait for
distribution to roll out the updates/needing to reboot the box.
Thanks.
--
Dmitry
On Tuesday, June 29, 2010 09:28:59 am Bruno Pr?mont wrote:
> On Tue, 29 June 2010 Alexander Clouter <[email protected]> wrote:
> > Dmitry Torokhov <[email protected]> wrote:
> > > VMware Tools installer requires the upstream driver to be compiled
> > > as a module in order to detect its presence and avoid installing
> > > our own version on top of it. To avoid surprises with 2 versions
> > > of the driver being installed and fighting with each other, let's
> > > force the driver to be compiled as a module unless user selects
> > > CONFIG_EMBEDDED.
> >
> > *barf*
> >
> > This surely is a problem in the installer and not the kernel? Can you
> > not nosey around in /sys/class/misc or where-ever your driver appears?
> > If it does not, then I would probably suggest a patch to your balloon
> > driver that dumps some details in there, including module version
> > information.
> >
> > Eugh.
>
> In addition, the installer may check under /sys/module/ for it (as for
> any/most code that can be built as a module), even for built-in code.
> (if balloon driver does not show up there when built-in it would be
> better to get it to show up there)
>
When driver is built-in the only thing exported in /sys/module/XXX are
module parameters.
We also need to handle scenario when module is not loaded into the
kernel.
Thanks.
--
Dmitry
> From: [email protected] [mailto:linux-kernel-
> [email protected]] On Behalf Of Dmitry Torokhov
> Sent: June 28, 2010 7:01 PM
> VMware Tools installer requires the upstream driver to be compiled
> as a module in order to detect its presence and avoid installing
> our own version on top of it. To avoid surprises with 2 versions
> of the driver being installed and fighting with each other, let's
> force the driver to be compiled as a module unless user selects
> CONFIG_EMBEDDED.
>
> config VMWARE_BALLOON
> tristate "VMware Balloon Driver"
> - depends on X86
> + depends on X86 && (EMBEDDED || m)
None of the following questions are related to this patch.
Q1)Does vmtools handle pvscsi correctly?
Q2)In case if a VM wants to be a good citizen, is there a way for a
guest to know about the balloon-event?
Q3)What if an app mlock's its memory resources and driver's have
pinned down their pages then how does inflation work?
regards
Chetan Loke
Hi Chetan,
On Wednesday, June 30, 2010 11:42:53 am Chetan Loke wrote:
> > From: [email protected] [mailto:linux-kernel-
> > [email protected]] On Behalf Of Dmitry Torokhov
> > Sent: June 28, 2010 7:01 PM
> > VMware Tools installer requires the upstream driver to be compiled
> > as a module in order to detect its presence and avoid installing
> > our own version on top of it. To avoid surprises with 2 versions
> > of the driver being installed and fighting with each other, let's
> > force the driver to be compiled as a module unless user selects
> > CONFIG_EMBEDDED.
> >
> > config VMWARE_BALLOON
> >
> > tristate "VMware Balloon Driver"
> >
> > - depends on X86
> > + depends on X86 && (EMBEDDED || m)
>
> None of the following questions are related to this patch.
>
> Q1)Does vmtools handle pvscsi correctly?
>
Yes, as long as it compiled as a module or installer will not overwrite
distribution-supplied version unless user explicitly requests installer
to clobber it.
So far distributions have not tried building their kernels with pvscsi
or vmxnet3 built-in, but did so with our ballon driver, which prompted
this particular change.
> Q2)In case if a VM wants to be a good citizen, is there a way for a
> guest to know about the balloon-event?
I am not sure I follow. Ballooning supposed to be as transparent as
possible...
>
> Q3)What if an app mlock's its memory resources and driver's have
> pinned down their pages then how does inflation work?
We will inflate as much as we can. Obviously if there are no more
memory balloon may not grow to its full target size.
Balloon driver communicates to the hypervisor the total amount of
memory in the guest, we may want to adjust that number by subtracting
memory allocated by the kernel, mlocked memory and so on, but it is
not done currently.
Thanks.
--
Dmitry
>
>
> regards
> Chetan Loke
Hello Dmitry,
On Wed, Jun 30, 2010 at 3:27 PM, Dmitry Torokhov <[email protected]> wrote:
> Hi Chetan,
>
> On Wednesday, June 30, 2010 11:42:53 am Chetan Loke wrote:
>> Q1)Does vmtools handle pvscsi correctly?
>>
>
> Yes, as long as it compiled as a module or installer will not overwrite
> distribution-supplied version unless user explicitly requests installer
> to clobber it.
>
perfect.
> So far distributions have not tried building their kernels with pvscsi
> or vmxnet3 built-in, but did so with our ballon driver, which prompted
> this particular change.
>
We are building iso's which will then be used to build/create an ESX
appliance. So we would need the pvscsi driver from the start. vNICs
will be populated post-install. At which point vmxnet[2/3] will
kick-in via vmtools.
>> Q2)In case if a VM wants to be a good citizen, is there a way for a
>> guest to know about the balloon-event?
>
> I am not sure I follow. Ballooning supposed to be as transparent as
> possible...
>
This is too product specific. I will send you an email separately.
>> Q3)What if an app mlock's its memory resources and driver's have
>> pinned down their pages then how does inflation work?
>
> We will inflate as much as we can. Obviously if there are no more
> memory balloon may not grow to its full target size.
>
> Balloon driver communicates to the hypervisor the total amount of
> memory in the guest, we may want to adjust that number by subtracting
> memory allocated by the kernel, mlocked memory and so on, but it is
> not done currently.
Ok.
I'm stuck with one question -
A) Ballooning will trigger guest's native memory management policy.
A.1) So this could mean guest might swap it's pages on it's vdisk, correct?
Consider this setup -
B) VM1..VMn have backing store(data and OS partitions) on LUNs(SAN).
Further, data LUNs are mounted as RDMs. I chose RDMs just to keep it
simple.
C) Say there's memory pressure. How? Well, few VM's are blasting I/O
to the LUNs. Plus, a backup triggered. Plus, whatever else happened.
C.1) VM's now seem to need more and more memory.
C.2) hypervisors block-layer/other-layers also need more memory.
C.3) Hypervisor's memory-management algorithm kicks-in.
......
C.3.x) Ballooning triggers - now some VM's (excluding the ones
from C.1) are giving up memory and if A.1) above is true then the
guest's pages will be swapped out on the LUNs via
hypervisor's SCSI-LLDD. But look at C.2) above. Is
this a soft-deadlock?
Oh, it's a linux-guest and if C.1) timesout then the guest will send
aborts and eventually a LUN reset ;).
In this particular case, if my suspicion is valid and if all the
signatures match(swap is out on the SAN, block-congestion etc) then
the balloon driver could just bail out.
> Thanks.
> --
> Dmitry
Thanks
Chetan Loke
On Wednesday, June 30, 2010 02:26:40 pm Chetan Loke wrote:
> Hello Dmitry,
>
> On Wed, Jun 30, 2010 at 3:27 PM, Dmitry Torokhov <[email protected]> wrote:
> > Hi Chetan,
> >
> > On Wednesday, June 30, 2010 11:42:53 am Chetan Loke wrote:
> >> Q1)Does vmtools handle pvscsi correctly?
> >
> > Yes, as long as it compiled as a module or installer will not overwrite
> > distribution-supplied version unless user explicitly requests installer
> > to clobber it.
>
> perfect.
>
> > So far distributions have not tried building their kernels with pvscsi
> > or vmxnet3 built-in, but did so with our ballon driver, which prompted
> > this particular change.
>
> We are building iso's which will then be used to build/create an ESX
> appliance. So we would need the pvscsi driver from the start.
Well, with typical setup, even though pvscsi is a module, as long as it
is in initramfs it will still be loaded automatically. If you are building
truly custom appliance and require pvscsi built-in you'll have to modify
the tools installer script.
> vNICs
> will be populated post-install. At which point vmxnet[2/3] will
> kick-in via vmtools.
Depending on what you base your appliance vmxnet3 might be already in
the kernel along with pvscsi.
>
> >> Q2)In case if a VM wants to be a good citizen, is there a way for a
> >> guest to know about the balloon-event?
> >
> > I am not sure I follow. Ballooning supposed to be as transparent as
> > possible...
>
> This is too product specific. I will send you an email separately.
>
OK.
> >> Q3)What if an app mlock's its memory resources and driver's have
> >> pinned down their pages then how does inflation work?
> >
> > We will inflate as much as we can. Obviously if there are no more
> > memory balloon may not grow to its full target size.
> >
> > Balloon driver communicates to the hypervisor the total amount of
> > memory in the guest, we may want to adjust that number by subtracting
> > memory allocated by the kernel, mlocked memory and so on, but it is
> > not done currently.
>
> Ok.
>
> I'm stuck with one question -
>
> A) Ballooning will trigger guest's native memory management policy.
> A.1) So this could mean guest might swap it's pages on it's vdisk,
> correct?
>
Yes.
> Consider this setup -
> B) VM1..VMn have backing store(data and OS partitions) on LUNs(SAN).
> Further, data LUNs are mounted as RDMs. I chose RDMs just to keep it
> simple.
> C) Say there's memory pressure. How? Well, few VM's are blasting I/O
> to the LUNs. Plus, a backup triggered. Plus, whatever else happened.
> C.1) VM's now seem to need more and more memory.
> C.2) hypervisors block-layer/other-layers also need more memory.
> C.3) Hypervisor's memory-management algorithm kicks-in.
> ......
> C.3.x) Ballooning triggers - now some VM's (excluding the ones
> from C.1) are giving up memory and if A.1) above is true then the
> guest's pages will be swapped out on the LUNs via
> hypervisor's SCSI-LLDD. But look at C.2) above. Is
> this a soft-deadlock?
>
If there is no memory something will have to give up. If you look at
the ballon driver you will see that when it switches from non-sleeping
to sleeping allocations or otherwise starts getting allocation errors
it will throttle the inflation rates to give the box a "breather" and
not choke it completely right then and there.
> Oh, it's a linux-guest and if C.1) timesout then the guest will send
> aborts and eventually a LUN reset ;).
>
> In this particular case, if my suspicion is valid and if all the
> signatures match(swap is out on the SAN, block-congestion etc) then
> the balloon driver could just bail out.
>
Yes, it is not guaranteed that ballon will reach this target, and in
this case host itself might start swapping causing severe performance
issues.
Realistically it all boils down to this: even though you may overcommit
you still have to adequately provision your hosts so they could handle
the load.
Thanks.
--
Dmitry
On Tue, 29 Jun 2010 09:40:38 -0700
Dmitry Torokhov <[email protected]> wrote:
> On Tuesday, June 29, 2010 09:28:59 am Bruno Pr__mont wrote:
> > On Tue, 29 June 2010 Alexander Clouter <[email protected]> wrote:
> > > Dmitry Torokhov <[email protected]> wrote:
> > > > VMware Tools installer requires the upstream driver to be compiled
> > > > as a module in order to detect its presence and avoid installing
> > > > our own version on top of it. To avoid surprises with 2 versions
> > > > of the driver being installed and fighting with each other, let's
> > > > force the driver to be compiled as a module unless user selects
> > > > CONFIG_EMBEDDED.
> > >
> > > *barf*
> > >
> > > This surely is a problem in the installer and not the kernel? Can you
> > > not nosey around in /sys/class/misc or where-ever your driver appears?
> > > If it does not, then I would probably suggest a patch to your balloon
> > > driver that dumps some details in there, including module version
> > > information.
> > >
> > > Eugh.
> >
> > In addition, the installer may check under /sys/module/ for it (as for
> > any/most code that can be built as a module), even for built-in code.
> > (if balloon driver does not show up there when built-in it would be
> > better to get it to show up there)
> >
>
> When driver is built-in the only thing exported in /sys/module/XXX are
> module parameters.
>
> We also need to handle scenario when module is not loaded into the
> kernel.
>
- check for the /sys/module directory.
- if that failed, modprobe the driver
- if that failed, error out
The patch really is pretty icky. Adding restrictions to kernel
configuration to work around problems in some installer program?
On Thursday, July 01, 2010 03:18:35 pm Andrew Morton wrote:
> On Tue, 29 Jun 2010 09:40:38 -0700
>
> Dmitry Torokhov <[email protected]> wrote:
> > On Tuesday, June 29, 2010 09:28:59 am Bruno Pr__mont wrote:
> > > On Tue, 29 June 2010 Alexander Clouter <[email protected]> wrote:
> > > > Dmitry Torokhov <[email protected]> wrote:
> > > > > VMware Tools installer requires the upstream driver to be compiled
> > > > > as a module in order to detect its presence and avoid installing
> > > > > our own version on top of it. To avoid surprises with 2 versions
> > > > > of the driver being installed and fighting with each other, let's
> > > > > force the driver to be compiled as a module unless user selects
> > > > > CONFIG_EMBEDDED.
> > > >
> > > > *barf*
> > > >
> > > > This surely is a problem in the installer and not the kernel? Can
> > > > you not nosey around in /sys/class/misc or where-ever your driver
> > > > appears? If it does not, then I would probably suggest a patch to
> > > > your balloon driver that dumps some details in there, including
> > > > module version information.
> > > >
> > > > Eugh.
> > >
> > > In addition, the installer may check under /sys/module/ for it (as for
> > > any/most code that can be built as a module), even for built-in code.
> > > (if balloon driver does not show up there when built-in it would be
> > > better to get it to show up there)
> >
> > When driver is built-in the only thing exported in /sys/module/XXX are
> > module parameters.
> >
> > We also need to handle scenario when module is not loaded into the
> > kernel.
>
> - check for the /sys/module directory.
Empty.
>
> - if that failed, modprobe the driver
>
Succeeds since the driver name changed (we renamed it to vmware_balloon
before submitting into mainline to avoid confusion based on our experience
with pvscsi; the existing one in the wild is called vmmemctl).
Now we have 2 drivers fighting. There is no backing device and so driver
core will not save us by refusing to bind to already claimed device.
> - if that failed, error out
>
> The patch really is pretty icky. Adding restrictions to kernel
> configuration to work around problems in some installer program?
It is probably not the prettiest patch, I agree, but users still have
an option of building this stuff in if they care that much
(CONFIG_EMBEDDED).
Thanks.
--
Dmitry
On Thu, 1 Jul 2010 15:31:50 -0700
Dmitry Torokhov <[email protected]> wrote:
> On Thursday, July 01, 2010 03:18:35 pm Andrew Morton wrote:
> > On Tue, 29 Jun 2010 09:40:38 -0700
> >
> > Dmitry Torokhov <[email protected]> wrote:
> > > On Tuesday, June 29, 2010 09:28:59 am Bruno Pr__mont wrote:
> > > > On Tue, 29 June 2010 Alexander Clouter <[email protected]> wrote:
> > > > > Dmitry Torokhov <[email protected]> wrote:
> > > > > > VMware Tools installer requires the upstream driver to be compiled
> > > > > > as a module in order to detect its presence and avoid installing
> > > > > > our own version on top of it. To avoid surprises with 2 versions
> > > > > > of the driver being installed and fighting with each other, let's
> > > > > > force the driver to be compiled as a module unless user selects
> > > > > > CONFIG_EMBEDDED.
> > > > >
> > > > > *barf*
> > > > >
> > > > > This surely is a problem in the installer and not the kernel? Can
> > > > > you not nosey around in /sys/class/misc or where-ever your driver
> > > > > appears? If it does not, then I would probably suggest a patch to
> > > > > your balloon driver that dumps some details in there, including
> > > > > module version information.
> > > > >
> > > > > Eugh.
> > > >
> > > > In addition, the installer may check under /sys/module/ for it (as for
> > > > any/most code that can be built as a module), even for built-in code.
> > > > (if balloon driver does not show up there when built-in it would be
> > > > better to get it to show up there)
> > >
> > > When driver is built-in the only thing exported in /sys/module/XXX are
> > > module parameters.
> > >
> > > We also need to handle scenario when module is not loaded into the
> > > kernel.
> >
> > - check for the /sys/module directory.
>
> Empty.
But the presence of the empty /sys/module/vmware_balloon tells you that
the driver is loaded?
Confused.
> >
> > - if that failed, modprobe the driver
> >
>
> Succeeds since the driver name changed (we renamed it to vmware_balloon
> before submitting into mainline to avoid confusion based on our experience
> with pvscsi; the existing one in the wild is called vmmemctl).
>
> Now we have 2 drivers fighting. There is no backing device and so driver
> core will not save us by refusing to bind to already claimed device.
If vmware_balloon is present in /sys/modules or is loaded, don't load
vmmemctl. And vice versa.
I dunno - it's silly for me to sit here proposing solutions. it's
better that you do it!
On Thursday, July 01, 2010 03:43:47 pm Andrew Morton wrote:
> On Thu, 1 Jul 2010 15:31:50 -0700
>
> Dmitry Torokhov <[email protected]> wrote:
> > On Thursday, July 01, 2010 03:18:35 pm Andrew Morton wrote:
> > > On Tue, 29 Jun 2010 09:40:38 -0700
> > >
> > > Dmitry Torokhov <[email protected]> wrote:
> > > > On Tuesday, June 29, 2010 09:28:59 am Bruno Pr__mont wrote:
> > > > > On Tue, 29 June 2010 Alexander Clouter <[email protected]> wrote:
> > > > > > Dmitry Torokhov <[email protected]> wrote:
> > > > > > > VMware Tools installer requires the upstream driver to be
> > > > > > > compiled as a module in order to detect its presence and avoid
> > > > > > > installing our own version on top of it. To avoid surprises
> > > > > > > with 2 versions of the driver being installed and fighting
> > > > > > > with each other, let's force the driver to be compiled as a
> > > > > > > module unless user selects CONFIG_EMBEDDED.
> > > > > >
> > > > > > *barf*
> > > > > >
> > > > > > This surely is a problem in the installer and not the kernel?
> > > > > > Can you not nosey around in /sys/class/misc or where-ever your
> > > > > > driver appears? If it does not, then I would probably suggest a
> > > > > > patch to your balloon driver that dumps some details in there,
> > > > > > including module version information.
> > > > > >
> > > > > > Eugh.
> > > > >
> > > > > In addition, the installer may check under /sys/module/ for it (as
> > > > > for any/most code that can be built as a module), even for
> > > > > built-in code. (if balloon driver does not show up there when
> > > > > built-in it would be better to get it to show up there)
> > > >
> > > > When driver is built-in the only thing exported in /sys/module/XXX
> > > > are module parameters.
> > > >
> > > > We also need to handle scenario when module is not loaded into the
> > > > kernel.
> > >
> > > - check for the /sys/module directory.
> >
> > Empty.
>
> But the presence of the empty /sys/module/vmware_balloon tells you that
> the driver is loaded?
>
> Confused.
No, unfortunately a kernel module that does not use module parameters
is invisible when it is built in. If you check your box you will most
likely not see /sys/module/evdev, /sys/module/input, and so forth.
>
> > > - if that failed, modprobe the driver
> >
> > Succeeds since the driver name changed (we renamed it to vmware_balloon
> > before submitting into mainline to avoid confusion based on our
> > experience with pvscsi; the existing one in the wild is called
> > vmmemctl).
> >
> > Now we have 2 drivers fighting. There is no backing device and so driver
> > core will not save us by refusing to bind to already claimed device.
>
> If vmware_balloon is present in /sys/modules or is loaded, don't load
> vmmemctl. And vice versa.
>
> I dunno - it's silly for me to sit here proposing solutions. it's
> better that you do it!
Unfortunately I do not have a good solution at the moment. I guess we'll
have to work with distributions to make sure they keep it as a module
(it also makes most sense for them since not everyone runs on our
platform).
Thanks.
--
Dmitry
Dmitry Torokhov <[email protected]> wrote:
>
>> > Now we have 2 drivers fighting. There is no backing device and so driver
>> > core will not save us by refusing to bind to already claimed device.
>>
>> If vmware_balloon is present in /sys/modules or is loaded, don't load
>> vmmemctl. And vice versa.
>>
>> I dunno - it's silly for me to sit here proposing solutions. it's
>> better that you do it!
>
> Unfortunately I do not have a good solution at the moment. I guess we'll
> have to work with distributions to make sure they keep it as a module
> (it also makes most sense for them since not everyone runs on our
> platform).
>
I cannot seriously believe you are considering a viable solution is
"everyone[1] must abide by these rules otherwise our installer might
barf". The only benefactor of this patch is your installer and the
effect is an undocumented and peculiar constraint on a kernel module.
Seriously, add sometime so that you get something in /sys/modules (maybe
it's time for something in /sys/class?) or maybe do something so that
you have: VMWARE_BALLOON_CMD(STATUS, ...) where the guest can say if
there is already something ballooning for it. Surely the guest should
be aware if there is more than one balloon driver at play?
I think a friend of mine summed it up rather well: "Fixing the kernel
instead of fixing the VMWare installer is an inspired move".
Cheers
[1] the dropdown menu on distrowatch lists 319 distrubutions
--
Alexander Clouter
.sigmonster says: May the bluebird of happiness twiddle your bits.