lkml,
http://www.sourceforge.net/project/showfiles.php?group_id=64580
I am announcing a sourceforge project for developing support in Linux
kernel for Advanced TCA (PICMG 3.0) architecture. Advanced TCA is a
technology where boards exist in a chassis and can either be processor
nodes or storage nodes. All boards in the chassis are connected by
FibreChannel and Ethernet. The blades can be hot added or hot removed
while the Linux processor nodes are active, meaning, that the SCSI
subsystem must add devices on insertion request and remove devices on
ejection requests. Further the typical /dev/sda naming of devices is
not appropriate since device nodes can change depending on the insertion
order of disks.
These patches are for Linux 2.4.19 and work with the Qlogic 2300
FibreChannel driver and at this point mostly support hotswap of the disk
subsystem.
The patches consist of a SCSI hotswap patch for 2.4 kernel and QLA 2300
support.
The patches also consist of a GA Mapper library which maps fibrechannel
WWNs to devices in devfs filesystems such as /dev/scsi/chassis/slot/disc.
The sourceforge project also contains a userland library for each patch
and userland applications such that these operations can be scripted.
Advanced TCA uses IPMI, so this project will use the IPMI driver being
developed by Corey Minyard.
To be posted are userland or kernel hotswap managers. I've not decided
how to do this yet, so I'll post the bits when they are done.
I welcome comments questions or code to be added to the sourceforge project.
Thanks,
-steve
On Mon, Oct 14, 2002 at 11:42:15AM -0700, Steven Dake wrote:
> lkml,
>
> http://www.sourceforge.net/project/showfiles.php?group_id=64580
>
> I am announcing a sourceforge project for developing support in Linux
> kernel for Advanced TCA (PICMG 3.0) architecture. Advanced TCA is a
> technology where boards exist in a chassis and can either be processor
> nodes or storage nodes. All boards in the chassis are connected by
> FibreChannel and Ethernet. The blades can be hot added or hot removed
> while the Linux processor nodes are active, meaning, that the SCSI
> subsystem must add devices on insertion request and remove devices on
> ejection requests. Further the typical /dev/sda naming of devices is
> not appropriate since device nodes can change depending on the insertion
> order of disks.
>
> These patches are for Linux 2.4.19 and work with the Qlogic 2300
> FibreChannel driver and at this point mostly support hotswap of the disk
> subsystem.
Some questions:
- is there a public spec for this architecture?
- are you going to be generating a 2.5 version of this so that
this feature can be added to the main kernel tree?
- Why don't you use the existing kernel way of notifying
userspace of hotplug events, through /sbin/hotplug?
- You create a lot of new ioctls, which is not nice. You should
probably do what was done for the pci hotplug subsystem, and
create a ram based filesystem for this subsystem. That way
you don't need to have a /dev node, and the userspace tools
become dirt simple.
thanks,
greg k-h
On Mon, Oct 14, 2002 at 11:42:15AM -0700, Steven Dake wrote:
>
> I welcome comments questions or code to be added to the sourceforge project.
Hi,
Here's some specific comments on your different patches.
--- linux-2.4.19/Documentation/Configure.help Fri Aug 2 17:39:42 2002
+++ linux-2.4.19-gamap/Documentation/Configure.help Mon Oct 14 11:16:22 2002
@@ -6795,6 +6795,19 @@
module if your root file system (the one containing the directory /)
is located on a SCSI device.
+Geographical Address Mapping support
+CONFIG_GAMAP
+ This driver will provide device filesystem dynamic mapping of WWNs
+ to their logical geographical address. The result is that block
+ storage devices can be accessed using:
+
+ /dev/scsi/chassis/slot/host
+
+ This feature also works after an fdisk updating the appropriate files
+ to match the state of the system.
+
+ This feature is only supported by the Qlogic V6 driver.
+
In looking at this patch (and the others) it looks like you are relying
on devfs being in the system. Is this true? What about the other 99%
of machines out there with devfs disabled?
diff -uNr linux-2.4.19/Documentation/devices.txt linux-2.4.19-gamap/Documentation/devices.txt
--- linux-2.4.19/Documentation/devices.txt Wed Nov 7 15:46:01 2001
+++ linux-2.4.19-gamap/Documentation/devices.txt Mon Oct 14 11:06:00 2002
@@ -417,8 +417,8 @@
218 = /dev/kchuid Inter-process chuid control
219 = /dev/modems/mwave MWave modem firmware upload
220 = /dev/mptctl Message passing technology (MPT) control
- 221 = /dev/mvista/hssdsi Montavista PICMG hot swap system driver
- 222 = /dev/mvista/hasi Montavista PICMG high availability
+ 221 = /dev/scsi/hotswap MontaVista SCSI/FC hotswap driver
+ 222 = /dev/scsi/gamap MontaVista SCSI/FC geographical address mapping driver
240-255 Reserved for local use
11 char Raw keyboard device
You are re-using minors that have previously been reserved. Does this
mean Montavista is dropping their PICMG patches?
diff -uNr linux-2.4.19/drivers/scsi/gamap.c linux-2.4.19-gamap/drivers/scsi/gamap.c
--- linux-2.4.19/drivers/scsi/gamap.c Wed Dec 31 17:00:00 1969
+++ linux-2.4.19-gamap/drivers/scsi/gamap.c Mon Oct 14 11:06:00 2002
<snip>
+#define __KERNEL_SYSCALLS__
+
+#include <linux/unistd.h>
Any reason for this? I didn't see any internal syscalls being made, but
I might have missed them.
+int ioctl_gamap_getga_from_fc_wwn (unsigned long parameters);
+int ioctl_gamap_getwwn_from_ga (unsigned long parameters);
+int ioctl_gamap_insert_by_scsi_id (unsigned long parameters);
+int ioctl_gamap_remove_by_scsi_id (unsigned long parameters);
+int ioctl_gamap_insert_by_fc_wwn (unsigned long parameters);
+int ioctl_gamap_remove_by_fc_wwn (unsigned long parameters);
These should all be static (along with being a little shorter).
+int gamap_insert_by_fc_wwn (int chassis, int slot, unsigned long long wwn) {
+int i = 0;
+struct Scsi_Host *hba_p;
+Scsi_Device *scsi_device;
+int host, channel, lun, id;
+char devname[32];
+struct gendisk *gendisk;
+int part;
+
+ /*
+ * Ensure entry not already present in map
+ * chassis and slot are a match, wwn is seperate match
+ */
This style is all through the patches. It is a SCSI way of defining
local variables? I'd really recommend indenting them to follow the rest
of the kernel style.
+static int gamap_ioctl (struct inode *inode,
+ struct file *file,
+ unsigned int cmd,
+ unsigned long parameters) {
+
+int result = -EINVAL;
+
+ switch (cmd) {
So you let any user do an ioctl command on the device (this goes for the
other patch too.) You should restrict this to CAP_SYS_ADMIN users only.
All of these ioctl commands look like they can easily be done through a
ramfs like interface.
+EXPORT_SYMBOL (gamap_getga_from_fc_wwn);
+EXPORT_SYMBOL (gamap_getwwn_from_ga);
+EXPORT_SYMBOL (gamap_insert_by_scsi_id);
+EXPORT_SYMBOL (gamap_remove_by_scsi_id);
+EXPORT_SYMBOL (gamap_insert_by_fc_wwn);
+EXPORT_SYMBOL (gamap_remove_by_fc_wwn);
I don't see any other code using these functions. Is there a need to
export them?
diff -uNr linux-2.4.19/fs/partitions/check.c linux-2.4.19-gamap/fs/partitions/check.c
--- linux-2.4.19/fs/partitions/check.c Fri Aug 2 17:39:45 2002
+++ linux-2.4.19-gamap/fs/partitions/check.c Mon Oct 14 11:06:21 2002
@@ -334,6 +334,35 @@
}
#endif /* CONFIG_DEVFS_FS */
+#ifdef CONFIG_GAMAP
+static void devfs_gamap_register_partition (struct gendisk *gendisk, int minor, int part) {
+char devname[16];
You shouldn't use a #ifdef within this .c file. I think you could move
it to your specific file, and then use #ifdef within a .h file. This
also goes for your other change to this file.
diff -uNr linux-2.4.19/include/linux/genhd.h linux-2.4.19-gamap/include/linux/genhd.h
--- linux-2.4.19/include/linux/genhd.h Fri Aug 2 17:39:45 2002
+++ linux-2.4.19-gamap/include/linux/genhd.h Mon Oct 14 11:14:20 2002
@@ -63,6 +63,8 @@
unsigned long nr_sects;
devfs_handle_t de; /* primary (master) devfs entry */
int number; /* stupid old code wastes space */
+ devfs_handle_t de_gamap; /* ga map device entry */
+
/* Performance stats: */
unsigned int ios_in_flight;
@@ -98,6 +100,8 @@
struct gendisk *next;
struct block_device_operations *fops;
+ devfs_handle_t de_gadir; /* GA device entry directory chassis/slot/host */
+ devfs_handle_t de_gamap[17]; /* for GA Mapping, need 17 entries (disc + 16 parts ) */
devfs_handle_t *de_arr; /* one per physical disc */
char *flags; /* one per physical disc */
};
Ouch, do you really need all of these handles?
As the gendisk interface has been cleaned up a _lot_ in 2.5, I'm not so
sure how well this implementation will now work.
Pretty much the same comments apply for your scsi-hotswap-main.patch
(devfs reliance, coding style, loads of ioctls, long ioctl function
names, global functions that don't need to be, etc.)
I also noticed that this code is included in the CGL CVS tree. What is
MontaVista's (and yours) future plans for this feature? Do you want it
in the main kernel tree, and are you willing to port it to 2.5?
thanks,
greg k-h
Greg,
Thanks for the comments.
Responses below.
Greg KH wrote:
>On Mon, Oct 14, 2002 at 11:42:15AM -0700, Steven Dake wrote:
>
>
>>I welcome comments questions or code to be added to the sourceforge project.
>>
>>
>
>Hi,
>
>Here's some specific comments on your different patches.
>
>--- linux-2.4.19/Documentation/Configure.help Fri Aug 2 17:39:42 2002
>+++ linux-2.4.19-gamap/Documentation/Configure.help Mon Oct 14 11:16:22 2002
>@@ -6795,6 +6795,19 @@
> module if your root file system (the one containing the directory /)
> is located on a SCSI device.
>
>+Geographical Address Mapping support
>+CONFIG_GAMAP
>+ This driver will provide device filesystem dynamic mapping of WWNs
>+ to their logical geographical address. The result is that block
>+ storage devices can be accessed using:
>+
>+ /dev/scsi/chassis/slot/host
>+
>+ This feature also works after an fdisk updating the appropriate files
>+ to match the state of the system.
>+
>+ This feature is only supported by the Qlogic V6 driver.
>+
>
>In looking at this patch (and the others) it looks like you are relying
>on devfs being in the system. Is this true? What about the other 99%
>of machines out there with devfs disabled?
>
>
Hopefully these machines that have devfs disabled aren't ATCA machines
:) It is true that devfs is REQUIRED for the GA Mapping. It isn't
required for the scsi hotswap feature. The reason it is required for
gamapping is that devices must be defineable at runtime during hot swap
events and /dev/sda is inappropriate for several reasons as named access
to a device. I can't see any other way to do it, although I could
provide some library in userspace that translates /dev/sda into
chassis/slots. The downside of this technique is now the user must use
some utility to find out which device to access when using fdisk, etc
instead of just using the device directly.
>diff -uNr linux-2.4.19/Documentation/devices.txt linux-2.4.19-gamap/Documentation/devices.txt
>--- linux-2.4.19/Documentation/devices.txt Wed Nov 7 15:46:01 2001
>+++ linux-2.4.19-gamap/Documentation/devices.txt Mon Oct 14 11:06:00 2002
>@@ -417,8 +417,8 @@
> 218 = /dev/kchuid Inter-process chuid control
> 219 = /dev/modems/mwave MWave modem firmware upload
> 220 = /dev/mptctl Message passing technology (MPT) control
>- 221 = /dev/mvista/hssdsi Montavista PICMG hot swap system driver
>- 222 = /dev/mvista/hasi Montavista PICMG high availability
>+ 221 = /dev/scsi/hotswap MontaVista SCSI/FC hotswap driver
>+ 222 = /dev/scsi/gamap MontaVista SCSI/FC geographical address mapping driver
> 240-255 Reserved for local use
>
> 11 char Raw keyboard device
>
>You are re-using minors that have previously been reserved. Does this
>mean Montavista is dropping their PICMG patches?
>
>
I had requested these minors in our rev1 of the PICMG 2.12 hotswap
patches. During a rework of the picmg patches another developer dropped
the minors and nobody is using the old code, so I thought I could reuse
them. This is probably bad mojo, but I think its ok :)
>diff -uNr linux-2.4.19/drivers/scsi/gamap.c linux-2.4.19-gamap/drivers/scsi/gamap.c
>--- linux-2.4.19/drivers/scsi/gamap.c Wed Dec 31 17:00:00 1969
>+++ linux-2.4.19-gamap/drivers/scsi/gamap.c Mon Oct 14 11:06:00 2002
><snip>
>+#define __KERNEL_SYSCALLS__
>+
>+#include <linux/unistd.h>
>
>Any reason for this? I didn't see any internal syscalls being made, but
>I might have missed them.
>
>
thanks i'll remove it. sloppy on my part.
>+int ioctl_gamap_getga_from_fc_wwn (unsigned long parameters);
>+int ioctl_gamap_getwwn_from_ga (unsigned long parameters);
>+int ioctl_gamap_insert_by_scsi_id (unsigned long parameters);
>+int ioctl_gamap_remove_by_scsi_id (unsigned long parameters);
>+int ioctl_gamap_insert_by_fc_wwn (unsigned long parameters);
>+int ioctl_gamap_remove_by_fc_wwn (unsigned long parameters);
>
>These should all be static (along with being a little shorter).
>
>
thanks i'll fix.
>+int gamap_insert_by_fc_wwn (int chassis, int slot, unsigned long long wwn) {
>+int i = 0;
>+struct Scsi_Host *hba_p;
>+Scsi_Device *scsi_device;
>+int host, channel, lun, id;
>+char devname[32];
>+struct gendisk *gendisk;
>+int part;
>+
>+ /*
>+ * Ensure entry not already present in map
>+ * chassis and slot are a match, wwn is seperate match
>+ */
>
>This style is all through the patches. It is a SCSI way of defining
>local variables? I'd really recommend indenting them to follow the rest
>of the kernel style.
>
>
Ok thanks will do.
>+static int gamap_ioctl (struct inode *inode,
>+ struct file *file,
>+ unsigned int cmd,
>+ unsigned long parameters) {
>+
>+int result = -EINVAL;
>+
>+ switch (cmd) {
>
>So you let any user do an ioctl command on the device (this goes for the
>other patch too.) You should restrict this to CAP_SYS_ADMIN users only.
>
>
I thought about this, and determined that access control can happen
through permissions on the device. I'll probably add something like
this though. We definately don't want typical users hotswap removing a
device!
>All of these ioctl commands look like they can easily be done through a
>ramfs like interface.
>
>+EXPORT_SYMBOL (gamap_getga_from_fc_wwn);
>+EXPORT_SYMBOL (gamap_getwwn_from_ga);
>+EXPORT_SYMBOL (gamap_insert_by_scsi_id);
>+EXPORT_SYMBOL (gamap_remove_by_scsi_id);
>+EXPORT_SYMBOL (gamap_insert_by_fc_wwn);
>+EXPORT_SYMBOL (gamap_remove_by_fc_wwn);
>
>I don't see any other code using these functions. Is there a need to
>export them?
>
>
At this point, there isn't anything using them. I am working on a
hotswap manager, that may be in kernel space (for performance reasons)
that may use these interfaces. I'm also working on a SAFTE Hotswap
processor module (ie; drivers/scsi/sp.c) for the SCSI subsystem that
uses these interfaces. (Safte is a hotswap standard for SCSI chassis).
>diff -uNr linux-2.4.19/fs/partitions/check.c linux-2.4.19-gamap/fs/partitions/check.c
>--- linux-2.4.19/fs/partitions/check.c Fri Aug 2 17:39:45 2002
>+++ linux-2.4.19-gamap/fs/partitions/check.c Mon Oct 14 11:06:21 2002
>@@ -334,6 +334,35 @@
> }
> #endif /* CONFIG_DEVFS_FS */
>
>+#ifdef CONFIG_GAMAP
>+static void devfs_gamap_register_partition (struct gendisk *gendisk, int minor, int part) {
>+char devname[16];
>
>You shouldn't use a #ifdef within this .c file. I think you could move
>it to your specific file, and then use #ifdef within a .h file. This
>also goes for your other change to this file.
>
>
I'm not so sure about this. The reason this file is patched is to
support things like fdisk changes (rexporting partitions after an fdisk)
and initial setup. I could provide an "alternate" check.c but would
rather not have to maintain the bug fixes from check.c to check_alt.c.
devfs_gamap_register_partition could be in a seperate file, but
devfs_register_partition is a generic function used by other parts of
the kernel that needs ifdef patches (or two seperate implementations).
Would two seperate implementations one in check.c (unpatched) and one in
gamap.c (GAMAP version) with the implementation in check.c ifndef'ed out
if GAMAP is defined be more palatable?
>diff -uNr linux-2.4.19/include/linux/genhd.h linux-2.4.19-gamap/include/linux/genhd.h
>--- linux-2.4.19/include/linux/genhd.h Fri Aug 2 17:39:45 2002
>+++ linux-2.4.19-gamap/include/linux/genhd.h Mon Oct 14 11:14:20 2002
>@@ -63,6 +63,8 @@
> unsigned long nr_sects;
> devfs_handle_t de; /* primary (master) devfs entry */
> int number; /* stupid old code wastes space */
>+ devfs_handle_t de_gamap; /* ga map device entry */
>+
>
> /* Performance stats: */
> unsigned int ios_in_flight;
>@@ -98,6 +100,8 @@
> struct gendisk *next;
> struct block_device_operations *fops;
>
>+ devfs_handle_t de_gadir; /* GA device entry directory chassis/slot/host */
>+ devfs_handle_t de_gamap[17]; /* for GA Mapping, need 17 entries (disc + 16 parts ) */
> devfs_handle_t *de_arr; /* one per physical disc */
> char *flags; /* one per physical disc */
> };
>
>
>Ouch, do you really need all of these handles?
>
>
maybe not. I think there is still a little bit of slop in the patches.
As you can tell, I've not code reviewed the driver yet (thats what you
guys are for :) and there are some things that need improvement.
>As the gendisk interface has been cleaned up a _lot_ in 2.5, I'm not so
>sure how well this implementation will now work.
>
>Pretty much the same comments apply for your scsi-hotswap-main.patch
>(devfs reliance, coding style, loads of ioctls, long ioctl function
>names, global functions that don't need to be, etc.)
>
>
the scsi-hotswap-main patch can use devfs, but shouldn't rely on it.
Ioctls are only to provide user space access to the kernel features.
global functions, as in global in the symbol table (via EXPORT_SYMBOL?)
Thanks for the comments i"ll try to apply to the scsi-hotswap patches as
well.
>I also noticed that this code is included in the CGL CVS tree. What is
>MontaVista's (and yours) future plans for this feature? Do you want it
>in the main kernel tree, and are you willing to port it to 2.5?
>
>
I'd love to see ATCA support in the main kernel (if it would be
accepted). Maintaining patches against 2.4 is ok (since it has slowed
down to bug fixes mostly). I really don't want to maintain these
patches against every minor release of 2.5 for now until 2.6 is
stabilized if I can help it. I think these sorts of things belong in
the kernel, atleast the SCSI hotswap patches.
These patches also depend on the Qlogic QLA 2300 driver which isn't in
the kernel. Would this also be included? We can talk to the QLogic
guys and see if they want to submit their driver to the 2.5 trees...
If you can suggest a route for getting these patches accepted into the
2.5 trees (beyond a port to 2.5 and the above suggested changes) i'd
love to hear them.
>thanks,
>
>greg k-h
>
>
thanks
-steve
>
>
>
>
Greg,
Responses below.
Greg KH wrote:
>On Mon, Oct 14, 2002 at 11:42:15AM -0700, Steven Dake wrote:
>
>
>>lkml,
>>
>>http://www.sourceforge.net/project/showfiles.php?group_id=64580
>>
>>I am announcing a sourceforge project for developing support in Linux
>>kernel for Advanced TCA (PICMG 3.0) architecture. Advanced TCA is a
>>technology where boards exist in a chassis and can either be processor
>>nodes or storage nodes. All boards in the chassis are connected by
>>FibreChannel and Ethernet. The blades can be hot added or hot removed
>>while the Linux processor nodes are active, meaning, that the SCSI
>>subsystem must add devices on insertion request and remove devices on
>>ejection requests. Further the typical /dev/sda naming of devices is
>>not appropriate since device nodes can change depending on the insertion
>>order of disks.
>>
>>These patches are for Linux 2.4.19 and work with the Qlogic 2300
>>FibreChannel driver and at this point mostly support hotswap of the disk
>>subsystem.
>>
>>
>
>Some questions:
> - is there a public spec for this architecture?
>
>
The spec hasn't ratified yet and I don't have a copy (I only have
pre-spec hardware). I think distribution is limited to PICMG members
once a spec is available, but I'm not sure. Who needs specs anyway :)
> - are you going to be generating a 2.5 version of this so that
> this feature can be added to the main kernel tree?
>
If you think it would be accepted, I'd spend the time making 2.5 kernel
patches. Beyond your other comments, any suggestions to get it accepted?
> - Why don't you use the existing kernel way of notifying
> userspace of hotplug events, through /sbin/hotplug?
>
The hotplug events occur through IPMI (a system management interface
specification) messages. I'm not sure if the hotswap manager will go in
the kernel or not, but if it were in the kernel, it could use
/sbin/hotplug to notify management software of hotswap events (which
would allow the scsi hotswap commands to be used to add or remove
devices). Initially I am going to probably do a user space manager
since its simpler and I think that sort of thing probably belongs in
user space. It will access the IPMI driver, read hotswap events from
the IPMI driver, and swap in and out devices and map/unmap devices via
the ga mapper.
Perhaps what is really needed is a kernel driver that uses the IPMI
driver kernel interface to pump disk device hotswap messages through
/sbin/hotplug. After I get a userspace implementation working (which is
easier to debug and test) I can start work on something like this. What
would you think of that? The nice thing about using /sbin/hotplug is
more things can be scripted like automatically removing a MD disk if the
hotswapped device is part of an MD device.
I've not started on this component yet and am just figuring out the IPMI
messaging at this point. Any comments you have on how to best integrate
this into the current hotplug system would be highly welcomed.
> - You create a lot of new ioctls, which is not nice. You should
> probably do what was done for the pci hotplug subsystem, and
> create a ram based filesystem for this subsystem. That way
> you don't need to have a /dev node, and the userspace tools
> become dirt simple.
>
>
I'll have to look at that. I'm not familiar with the ram based
filesystem. Could you point me to a source file that uses some of the
interfaces?
Thanks
-steve
>thanks,
>
>greg k-h
>
>
>
>
>
On 10/16/02 01:38, Steven Dake wrote:
> At this point, there isn't anything using them. I am working on a
> hotswap manager, that may be in kernel space (for performance reasons)
> that may use these interfaces. I'm also working on a SAFTE Hotswap
> processor module (ie; drivers/scsi/sp.c) for the SCSI subsystem that
> uses these interfaces. (Safte is a hotswap standard for SCSI chassis).
Do you really want to have SAF-TE polling in the kernel?
This can easily be accomplished in userspace using sg.
safte-monitor <http://gort.metaparadigm.com/safte-monitor/> can already
provde disk insertion, removal notifications in userspace and already
supports calling out to a script with the physical slot location information
(and with tweaks to the code, scsi device of the disk inserted)
There is even code in safte-monitor to identify the wwn of the devices
in each slot, although it needs updating to the latest qlogic ioctl
interface (or hbaapi).
~mc
Michael Clark wrote:
> On 10/16/02 01:38, Steven Dake wrote:
>
>> At this point, there isn't anything using them. I am working on a
>> hotswap manager, that may be in kernel space (for performance
>> reasons) that may use these interfaces. I'm also working on a SAFTE
>> Hotswap processor module (ie; drivers/scsi/sp.c) for the SCSI
>> subsystem that uses these interfaces. (Safte is a hotswap standard
>> for SCSI chassis).
>
>
> Do you really want to have SAF-TE polling in the kernel?
Good code thanks for the pointer.
When I searched months ago, there wasn't anything out there.
Safte polling in the kernel isn't inherently bad and could be tied into
the hotplug mechanism.
Making SAFTE hotswap available via SG would also work but system
performance would be bad at small poll intervals (like 100 msec).
Thanks
-steve
>
> This can easily be accomplished in userspace using sg.
>
> safte-monitor <http://gort.metaparadigm.com/safte-monitor/> can already
> provde disk insertion, removal notifications in userspace and already
> supports calling out to a script with the physical slot location
> information
> (and with tweaks to the code, scsi device of the disk inserted)
>
> There is even code in safte-monitor to identify the wwn of the devices
> in each slot, although it needs updating to the latest qlogic ioctl
> interface (or hbaapi).
>
> ~mc
>
>
>
On Tue, Oct 15, 2002 at 12:28:59PM -0700, Steven Dake wrote:
>
>
> Safte polling in the kernel isn't inherently bad and could be tied into
> the hotplug mechanism.
>
> Making SAFTE hotswap available via SG would also work but system
> performance would be bad at small poll intervals (like 100 msec).
Is there a real nead to get hotplug notification any faster than that?
And yes, it should all be done in userspace, whenever possible :)
thanks,
greg k-h
On Tue, Oct 15, 2002 at 11:39:02AM -0700, Steven Dake wrote:
> The spec hasn't ratified yet and I don't have a copy (I only have
> pre-spec hardware). I think distribution is limited to PICMG members
> once a spec is available, but I'm not sure. Who needs specs anyway :)
Heh, so are there any other devices besides the qlogic device that
support this?
> > - are you going to be generating a 2.5 version of this so that
> > this feature can be added to the main kernel tree?
> >
> If you think it would be accepted, I'd spend the time making 2.5 kernel
> patches. Beyond your other comments, any suggestions to get it accepted?
I think those comments are a great start, fix all of them, and I'd be
glad to look at the code again.
> > - Why don't you use the existing kernel way of notifying
> > userspace of hotplug events, through /sbin/hotplug?
> >
> The hotplug events occur through IPMI (a system management interface
> specification) messages. I'm not sure if the hotswap manager will go in
> the kernel or not, but if it were in the kernel, it could use
> /sbin/hotplug to notify management software of hotswap events (which
> would allow the scsi hotswap commands to be used to add or remove
> devices). Initially I am going to probably do a user space manager
> since its simpler and I think that sort of thing probably belongs in
> user space. It will access the IPMI driver, read hotswap events from
> the IPMI driver, and swap in and out devices and map/unmap devices via
> the ga mapper.
Hm, sounds like the IPMI driver needs to be generating /sbin/hotplug
events itself. That way everything could be done in userspace, right?
> Perhaps what is really needed is a kernel driver that uses the IPMI
> driver kernel interface to pump disk device hotswap messages through
> /sbin/hotplug.
Could the IPMI core do that itself?
> After I get a userspace implementation working (which is
> easier to debug and test) I can start work on something like this. What
> would you think of that? The nice thing about using /sbin/hotplug is
> more things can be scripted like automatically removing a MD disk if the
> hotswapped device is part of an MD device.
>
> I've not started on this component yet and am just figuring out the IPMI
> messaging at this point. Any comments you have on how to best integrate
> this into the current hotplug system would be highly welcomed.
I don't know a thing about IPMI. Feel free to ask questions here, or on
the linux-hotplug-devel list if you want to.
> > - You create a lot of new ioctls, which is not nice. You should
> > probably do what was done for the pci hotplug subsystem, and
> > create a ram based filesystem for this subsystem. That way
> > you don't need to have a /dev node, and the userspace tools
> > become dirt simple.
> >
> >
> I'll have to look at that. I'm not familiar with the ram based
> filesystem. Could you point me to a source file that uses some of the
> interfaces?
In the 2.4 kernel tree take a look at:
drivers/hotplug/pci_hotplug_core.c
and there's an article at:
http://www.linuxjournal.com/article.php?sid=5633
on how some of that stuff works.
In the 2.5 kernel, things are much easier, with the libfs code. Take a
look at:
drivers/hotplug/pci_hotplug_core.c
drivers/usb/core/inode.c
fs/driverfs/inode.c
for 3 different implementations of ramfs based file systems.
Hope this helps,
greg k-h
The data/telecoms I've talked to require disk hotswap times of less then
20 msec from notification of hotwap to blue led (a light used to
indicate the device can be removed). They would like 10 msec if it
could be done. This is because of how long it takes on a surprise
extraction for the hardware to send the signal vs the user to disconnect
the hardware.
For legacy systems such as SAFTE hotswap, polling through sg at 10 msec
intervals would be extremely painful because of all the context
switches. A timer scheduled every 10 msec to send out a SCSI message
and handle a response if there is a hotswap event is a much better course.
Thanks
-steve
Greg KH wrote:
>On Tue, Oct 15, 2002 at 12:28:59PM -0700, Steven Dake wrote:
>
>
>>Safte polling in the kernel isn't inherently bad and could be tied into
>>the hotplug mechanism.
>>
>>Making SAFTE hotswap available via SG would also work but system
>>performance would be bad at small poll intervals (like 100 msec).
>>
>>
>
>Is there a real nead to get hotplug notification any faster than that?
>
>And yes, it should all be done in userspace, whenever possible :)
>
>thanks,
>
>greg k-h
>
>
>
>
>
On Tue, Oct 15, 2002 at 10:38:47AM -0700, Steven Dake wrote:
> >In looking at this patch (and the others) it looks like you are relying
> >on devfs being in the system. Is this true? What about the other 99%
> >of machines out there with devfs disabled?
> >
> >
> Hopefully these machines that have devfs disabled aren't ATCA machines
> :) It is true that devfs is REQUIRED for the GA Mapping.
Given that there are a number of kernel developers working their
respective asses off trying to get devfs out of the kernel (by obsoleting
it), I would not really recommend tying your driver to it if you want it
to be around for very long :)
> It isn't
> required for the scsi hotswap feature. The reason it is required for
> gamapping is that devices must be defineable at runtime during hot swap
> events and /dev/sda is inappropriate for several reasons as named access
> to a device. I can't see any other way to do it, although I could
> provide some library in userspace that translates /dev/sda into
> chassis/slots. The downside of this technique is now the user must use
> some utility to find out which device to access when using fdisk, etc
> instead of just using the device directly.
Take a look at the driverfs tree in 2.5, and see how you could tie into
that. I think everything you need is there, if not, please let us know.
> >You are re-using minors that have previously been reserved. Does this
> >mean Montavista is dropping their PICMG patches?
> >
> I had requested these minors in our rev1 of the PICMG 2.12 hotswap
> patches. During a rework of the picmg patches another developer dropped
> the minors and nobody is using the old code, so I thought I could reuse
> them. This is probably bad mojo, but I think its ok :)
Hm, if the picmg code is dropped, why is it in the CGL release? And a
lot of code in the picmg patch was good, I'd hate to see it go away :(
But I agree, the picmg minors are not needed (it should tie into the
existing pci hotplug interface, and not use a minor), so as long as you
talked the pcimg people into it, that's fine.
> At this point, there isn't anything using them. I am working on a
> hotswap manager, that may be in kernel space (for performance reasons)
> that may use these interfaces. I'm also working on a SAFTE Hotswap
> processor module (ie; drivers/scsi/sp.c) for the SCSI subsystem that
> uses these interfaces. (Safte is a hotswap standard for SCSI chassis).
As I mentioned previously, this can probably be done in userspace (unless
you have some unreasonable performance reasons, what are the
requirements?)
> >You shouldn't use a #ifdef within this .c file. I think you could move
> >it to your specific file, and then use #ifdef within a .h file. This
> >also goes for your other change to this file.
> >
> >
> I'm not so sure about this. The reason this file is patched is to
> support things like fdisk changes (rexporting partitions after an fdisk)
> and initial setup. I could provide an "alternate" check.c but would
> rather not have to maintain the bug fixes from check.c to check_alt.c.
> devfs_gamap_register_partition could be in a seperate file, but
> devfs_register_partition is a generic function used by other parts of
> the kernel that needs ifdef patches (or two seperate implementations).
Hm, no I don't think you need to do that. Just put your logic in a
function, and provide a null function if your CONFIG variable is not
present.
> maybe not. I think there is still a little bit of slop in the patches.
> As you can tell, I've not code reviewed the driver yet (thats what you
> guys are for :) and there are some things that need improvement.
Ah, didn't realize we were the first line of quality control :)
> >As the gendisk interface has been cleaned up a _lot_ in 2.5, I'm not so
> >sure how well this implementation will now work.
> >
> >Pretty much the same comments apply for your scsi-hotswap-main.patch
> >(devfs reliance, coding style, loads of ioctls, long ioctl function
> >names, global functions that don't need to be, etc.)
> >
> >
> the scsi-hotswap-main patch can use devfs, but shouldn't rely on it.
> Ioctls are only to provide user space access to the kernel features.
> global functions, as in global in the symbol table (via EXPORT_SYMBOL?)
ioctls should be not used, instead a ram based fs like I mentioned in
the previous reply.
> >I also noticed that this code is included in the CGL CVS tree. What is
> >MontaVista's (and yours) future plans for this feature? Do you want it
> >in the main kernel tree, and are you willing to port it to 2.5?
> >
> >
> I'd love to see ATCA support in the main kernel (if it would be
> accepted). Maintaining patches against 2.4 is ok (since it has slowed
> down to bug fixes mostly). I really don't want to maintain these
> patches against every minor release of 2.5 for now until 2.6 is
> stabilized if I can help it. I think these sorts of things belong in
> the kernel, atleast the SCSI hotswap patches.
I agree, come up with a 2.5 version, and let us see what it looks like.
> These patches also depend on the Qlogic QLA 2300 driver which isn't in
> the kernel. Would this also be included? We can talk to the QLogic
> guys and see if they want to submit their driver to the 2.5 trees...
That's up to the QLogic people, I'm just too confused with the vast
number of different drivers from them :)
> If you can suggest a route for getting these patches accepted into the
> 2.5 trees (beyond a port to 2.5 and the above suggested changes) i'd
> love to hear them.
That's a good start.
thanks,
greg k-h
On Tue, Oct 15, 2002 at 01:46:34PM -0700, Steven Dake wrote:
> The data/telecoms I've talked to require disk hotswap times of less then
> 20 msec from notification of hotwap to blue led (a light used to
> indicate the device can be removed). They would like 10 msec if it
> could be done. This is because of how long it takes on a surprise
> extraction for the hardware to send the signal vs the user to disconnect
> the hardware.
But what starts the "notification of hotswap"? Is this driven by the
user somehow, or is it a hardware event that happens out of the blue?
> For legacy systems such as SAFTE hotswap, polling through sg at 10 msec
> intervals would be extremely painful because of all the context
> switches. A timer scheduled every 10 msec to send out a SCSI message
> and handle a response if there is a hotswap event is a much better course.
What generates the hotswap event?
thanks,
greg k-h
On Tue, Oct 15, 2002 at 02:07:43PM -0700, Steven Dake wrote:
>
>
> Greg KH wrote:
>
> >On Tue, Oct 15, 2002 at 01:46:34PM -0700, Steven Dake wrote:
> >
> >
> >>The data/telecoms I've talked to require disk hotswap times of less then
> >>20 msec from notification of hotwap to blue led (a light used to
> >>indicate the device can be removed). They would like 10 msec if it
> >>could be done. This is because of how long it takes on a surprise
> >>extraction for the hardware to send the signal vs the user to disconnect
> >>the hardware.
> >>
> >>
> >
> >But what starts the "notification of hotswap"? Is this driven by the
> >user somehow, or is it a hardware event that happens out of the blue?
> >
> >
> In the case of Advanced TCA, an IPMI message is sent to the CPU blade
> indicating the hotswap button is pressed on the front panel of a disk
> blade. The hotswap manager software unmaps the GA address, removes the
> device from the linux kernel via the scsi-hotswap-main stuff, and sends
> another IPMI message to the disk node telling it to light its "blue
> led". The user removes the disk. Insertion is easier.
So if userspace gets the event that a button was pressed, it can decide
to light up the led after is spins the disk down, right?
> In this case, the hotswap button on the front panel is used to indicate
> a hotswap event. There is talk of making the removal of the board
> indicate a hotswap event (surprise extraction) because the technicians
> don't wait for the blue led to remove the boards occasionally and the
> system should be able to handle this use case.
Hotplug PCI works much the same way. A user could just walk up, pop the
slot, and pull out the card without waiting for the LED to go out. We
don't care about that, as the user was obviously stupid in doing such a
thing. The spec even states something like this :)
> >>For legacy systems such as SAFTE hotswap, polling through sg at 10 msec
> >>intervals would be extremely painful because of all the context
> >>switches. A timer scheduled every 10 msec to send out a SCSI message
> >>and handle a response if there is a hotswap event is a much better course.
> >>
> >>
> >
> >What generates the hotswap event?
> >
> >
> In the case of SAFTE, a SCSI processor (ASIC) is polled by some polling
> interval about the state of the SAFTE (SCSI) backplane. When the state
> changes, software generates a hotswap event and removes the device.
So polling can be done within the kernel, right? Then notify userspace
of the event, which can decide what to do. Sound ok?
Or do you think this should be like the pci hotplug code, in that when a
slot is opened (or button pressed, depending on the hardware), the
kernel should scramble as fast as possible to unload the driver, and
shut down the power to the card? Then when it is finished, notify
userspace of what just happened.
thanks,
greg k-h
Greg KH wrote:
>On Tue, Oct 15, 2002 at 01:46:34PM -0700, Steven Dake wrote:
>
>
>>The data/telecoms I've talked to require disk hotswap times of less then
>>20 msec from notification of hotwap to blue led (a light used to
>>indicate the device can be removed). They would like 10 msec if it
>>could be done. This is because of how long it takes on a surprise
>>extraction for the hardware to send the signal vs the user to disconnect
>>the hardware.
>>
>>
>
>But what starts the "notification of hotswap"? Is this driven by the
>user somehow, or is it a hardware event that happens out of the blue?
>
>
In the case of Advanced TCA, an IPMI message is sent to the CPU blade
indicating the hotswap button is pressed on the front panel of a disk
blade. The hotswap manager software unmaps the GA address, removes the
device from the linux kernel via the scsi-hotswap-main stuff, and sends
another IPMI message to the disk node telling it to light its "blue
led". The user removes the disk. Insertion is easier.
In this case, the hotswap button on the front panel is used to indicate
a hotswap event. There is talk of making the removal of the board
indicate a hotswap event (surprise extraction) because the technicians
don't wait for the blue led to remove the boards occasionally and the
system should be able to handle this use case.
>>For legacy systems such as SAFTE hotswap, polling through sg at 10 msec
>>intervals would be extremely painful because of all the context
>>switches. A timer scheduled every 10 msec to send out a SCSI message
>>and handle a response if there is a hotswap event is a much better course.
>>
>>
>
>What generates the hotswap event?
>
>
In the case of SAFTE, a SCSI processor (ASIC) is polled by some polling
interval about the state of the SAFTE (SCSI) backplane. When the state
changes, software generates a hotswap event and removes the device.
Thanks
-steve
>
>
Greg KH wrote:
>On Tue, Oct 15, 2002 at 02:07:43PM -0700, Steven Dake wrote:
>
>
>>Greg KH wrote:
>>
>>
>>
>>>On Tue, Oct 15, 2002 at 01:46:34PM -0700, Steven Dake wrote:
>>>
>>>
>>>
>>>
>>>>The data/telecoms I've talked to require disk hotswap times of less then
>>>>20 msec from notification of hotwap to blue led (a light used to
>>>>indicate the device can be removed). They would like 10 msec if it
>>>>could be done. This is because of how long it takes on a surprise
>>>>extraction for the hardware to send the signal vs the user to disconnect
>>>>the hardware.
>>>>
>>>>
>>>>
>>>>
>>>But what starts the "notification of hotswap"? Is this driven by the
>>>user somehow, or is it a hardware event that happens out of the blue?
>>>
>>>
>>>
>>>
>>In the case of Advanced TCA, an IPMI message is sent to the CPU blade
>>indicating the hotswap button is pressed on the front panel of a disk
>>blade. The hotswap manager software unmaps the GA address, removes the
>>device from the linux kernel via the scsi-hotswap-main stuff, and sends
>>another IPMI message to the disk node telling it to light its "blue
>>led". The user removes the disk. Insertion is easier.
>>
>>
>
>So if userspace gets the event that a button was pressed, it can decide
>to light up the led after is spins the disk down, right?
>
>
The userspace app of some sort should light up the led after its call to
scsi hotswap remove. I believe the correct thing to do is that the scsi
hotswap kernel code should spin down the disk (which it doesn't do
today) since its a function of the bus protocol to spin up/down disks.
>
>
>>In this case, the hotswap button on the front panel is used to indicate
>>a hotswap event. There is talk of making the removal of the board
>>indicate a hotswap event (surprise extraction) because the technicians
>>don't wait for the blue led to remove the boards occasionally and the
>>system should be able to handle this use case.
>>
>>
>
>Hotplug PCI works much the same way. A user could just walk up, pop the
>slot, and pull out the card without waiting for the LED to go out. We
>don't care about that, as the user was obviously stupid in doing such a
>thing. The spec even states something like this :)
>
>
unforutnately those picky telecoms want exactly this type of thing to
not cause system crashes. Evil I say. The bad user should just be
smacked with a clue-by-4.
>
>
>>>>For legacy systems such as SAFTE hotswap, polling through sg at 10 msec
>>>>intervals would be extremely painful because of all the context
>>>>switches. A timer scheduled every 10 msec to send out a SCSI message
>>>>and handle a response if there is a hotswap event is a much better course.
>>>>
>>>>
>>>>
>>>>
>>>What generates the hotswap event?
>>>
>>>
>>>
>>>
>>In the case of SAFTE, a SCSI processor (ASIC) is polled by some polling
>>interval about the state of the SAFTE (SCSI) backplane. When the state
>>changes, software generates a hotswap event and removes the device.
>>
>>
>
>So polling can be done within the kernel, right? Then notify userspace
>of the event, which can decide what to do. Sound ok?
>
>
This is what I planned to do with the drivers/scsi/sp.c code or perhaps
do the hotswap directly. This is in my work queue for sometime in
December so I've not thought about it much yet :)
>Or do you think this should be like the pci hotplug code, in that when a
>slot is opened (or button pressed, depending on the hardware), the
>kernel should scramble as fast as possible to unload the driver, and
>shut down the power to the card? Then when it is finished, notify
>userspace of what just happened.
>
I think if there isn't polling from userland, response times of 20msec
can be obtained for disk devices. In our case, the backplane only has
serial connections so there isn't a typical driver controlling the
interface. They are talking about adding support for PCI Express (3gio)
to the PICMG3 spec, so this may become an issue in which case unloading
the driver could probably happen from userland or kernel although kernel
would probably give better response times. I've not thought about it
much since PCI Express seems a long way out.
I know our PICMG 2.12 driver unloads PCI drivers controlling cPCI
backplane devices via a userspace daemon that retrieves a hotswap event
via a kernel event manager, turns off the interface (in the case of
networking) and unloads the driver if there are no more devices
controlled by it.
In the case of SAFTE there is no event-driven hotswap (hotswaps are
driven by a poll routine) so a userland poll for hotswap removal
wouldn't work and this is probably better serviced by a kernel
thread/timer of some sort. PCI is a whole nother beast which I'll leave
to the PICMG2.12 experts and PCI hotplug experts (you) :)
Safte is a pretty old technology and I'm only working on it because some
platforms I support have a build in SAFTE ASIC.
>
>thanks,
>
>greg k-h
>
>
>
>
>
On Tue, Oct 15, 2002 at 02:34:50PM -0700, Steven Dake wrote:
> >Given that there are a number of kernel developers working their
> >respective asses off trying to get devfs out of the kernel (by obsoleting
> >it), I would not really recommend tying your driver to it if you want it
> >to be around for very long :)
> >
> >
> What is replacing devfs? Atleast for kernel 2.4, devfs is a good
> solution. Is something new in 2.5 used to provide device nodes at
> kernel runtime?
A combination of /sbin/hotplug notification, and information in the
driverfs tree will enable userspace to create and remove device nodes
whereever and with whatever name they want to.
> We didn't drop picmg2.12 support at all. What was dropped was the
> initial prototype implementation that used those minors (and has been
> replaced by a better implementation that doesn't use them). PICMG 2.12
> support (as well as redundant system slot) is still supported in the CGL
> tress and in MontaVista Linux. I don't know much about picmg 2.12
> implementation, another developer @ Mvista is doing the work.
My main complaint (as the pci hotplug maintainer) is that Mvista isn't
trying to work with the community for their code. I had a number of
good emails with the main developer (at the time), but unfortunatly I
haven't heard anything in quite some time. Any proding you might
provide in this area would be greatly appreciated.
Otherwise Mvista is going to be supporting that patch outside of the
main tree for forever, and that can get old and expensive over time.
> >As I mentioned previously, this can probably be done in userspace (unless
> >you have some unreasonable performance reasons, what are the
> >requirements?)
> >
> >
> 20 msec from hotswap request to disk extraction.
Yes, that sounds pretty unreasonable :) What drives such a quick
request turnaround time?
> The functionality is intertwined in devfs_register_partition. I don't
> think I understand what you mean here, could you expand in mail?
Hm, ok, let me go dig up the patch again, ugh. Ok, I don't really know
how, but it could proably be done cleaner :)
> Ok and I'll take a look at using ramfs and driverfs instead of devfs and
> ioctls in both patches for the 2.5 tree. We may probably keep the patch
> for the 2.4 tree since I dont think there is driverfs or ramfs in 2.4.
Yes, ramfs is in 2.4, and the pci hotplug core uses a filesystem just
like this in the 2.4 kernel. But if you want to keep your patch out of
the kernel, it's fine with me :)
> >That's up to the QLogic people, I'm just too confused with the vast
> >number of different drivers from them :)
> >
> >
> Mark Bellon @ MontaVista is working with them to support their driver in
> MontaVista Linux. We have made a number of quality improvements and
> submitted them back to Qlogic. We have talked to their development team
> and they would be willing to submit their Qlogic 6 driver to the Linux
> kernel. We can do the integration work here of including it in the
> kernel. Should I submit a patch to the list ?
Here, and/or to the linux-scsi mailing list.
thanks,
greg k-h
On 10/16/02 04:46, Steven Dake wrote:
> The data/telecoms I've talked to require disk hotswap times of less then
> 20 msec from notification of hotwap to blue led (a light used to
> indicate the device can be removed). They would like 10 msec if it
> could be done. This is because of how long it takes on a surprise
> extraction for the hardware to send the signal vs the user to disconnect
> the hardware.
I'm just surprised that the SAF-TE processers will respond this quickly.
> For legacy systems such as SAFTE hotswap, polling through sg at 10 msec
> intervals would be extremely painful because of all the context
> switches. A timer scheduled every 10 msec to send out a SCSI message
> and handle a response if there is a hotswap event is a much better course.
100 context switchs a second isn't that much is it?
I'll adjust safte-monitor from its default 2 second polling down
to 10msec and see what the result is.
~mc