Hi hotplug folks,
I'm interested in hotplug features, especially physical insertion/removal. I'd
like to propose here a new sysfs tree for hotplug. Please comment on my
proposal, because this is a request for comments:-)
1. Current sysfs for hotplug
----------------------------
In 2.6, there are some sysfs files for hotplug. For instance, PCI has that
kind of files to control hotplug features. PCI related sysfs entries for hotplug
are created under /sys/bus/pci/slots/:
/sys/bus/pci/slots/<slot#>/<files>
There are some files under <slot#> directory. Some of them are supposed
to expose kenrel information, the others are supposed to control hotplug.
Recently, CPU and memory hotplug are being worked actively. According
to their patch, they seem to be trying to create sysfs files under the following
directory.
/sys/devices/system/
2. Problem
----------
Recent large machines have many PCI devices and some boards that
contain devices (e.g. CPU, memory, and/or I/O devices). A certain PCI
device (PCI1) might be connected with other one (PCI2), which means that
there is a dependency between PCI1 and PCI2. Similarly, there is a
dependency between the board and devices attached on the board.
In this situation, we need to know the dependency first when we want to
hot-remove such devices or board. However, I don't think that the flat sysfs
layout represents a dependency between each device quite well.
3. Suggestion
-------------
To solve the problem, I'd like to propose the following idea.
["hotplug" directory]
This directory is to represent a hierarchy of hotpluggable devices.
"hotpluggable device" means a device that can be powered off and
removed physically from the system running. The hierarchy describes a
dependency between each device. This directory would be placed, like:
/sys/devices/hotplug
Any systems that enable hotplug (e.g. ACPI, DLPAR) can create their
own directory right under the "hotplug" directory, like:
/sys/devices/hotplug/acpi
/sys/devices/hotplug/dlpar
Each of systems can create directories and files under the own directory,
and these directories should be easy for user to use.
[ACPI based Hotplug Case]
I think that ACPI is one of the systems tha know dependencies of devices.
So, I'd like to show an example here how ACPI based hotplug can work
under the "hotplug" directory. (I still have a lot of things to think, though.)
In ACPI based hotplug, "hotpluggable device" could be defined as follows:
A device that its device object in ACPI namespace has _EJ0 and _STA
method, and the _STA returns 0x0F (present).
[Hotpluggable device (simplified ASL)]
Scope (\_SB_) {
Device (DEV1) {
Method (_EJ0) {...}
Method (_STA) {... return 0x0F}
}
...
The above hotpluggable device is represented under the "hotplug" directory,
like:
/sys/devices/hotplug/acpi/DEV1
Plus, a control file, "eject", which is used for invoking a corresponding
device hot-removal handler, is created under each hotpluggable devices'
directory, like:
/sys/devices/hotplug/acpi/DEV1/eject
The directory "DEV1" and the file "eject" are deleted when the device is
removed. A point here is:
While a device is attached to the system, a corresponding directory
appears on sysfs. When the device is removed from the system, the
directory is also deleted.
As mentioned above, basically a directory is created only for hotpluggable
devices. However, there is a special case that a directory is created for
devices that include a hotpluggable device.
/sys/devices/hotplug/acpi/DEV2/DEV1/eject
(where, DEV2 is not a hotpluggable device)
In addition, the special directory is deleted when the child hotpluggable
device is removed.
Let's take a look at a sample here.
Assume that you have a large machine containing many devices, and the
devices are defined in ACPI namespace like below.
[Simplified ASL]
Scope (\_SB_) {
Device (DEV1) {
Method (_EJ0) {...}
Method (_STA, 0) { Return (0x0F) }
Device (DEV2) {
Method (_STA, 0) { Return (0x0F) }
}
Device (DEV3) {
Method (_EJ0) {...}
Method (_STA, 0) { Return (0x0F) }
}
Device (DEV4) {
Method (_STA, 0) { Return (0x0F) }
Device (DEV5) {
Method (_EJ0) {...}
Method (_STA, 0) { Return (0x00) }
}
Device (DEV6) {
Method (_EJ0) {...}
Method (_STA, 0) { Return (0x0F) }
}
}
}
}
In this case, "hotplug" tree looks like:
sys/devices/hotplug/acpi/
`-DEV1/
|-eject
|
|-DEV3/
| `-eject
|
`-DEV4/
`-DEV6/
`-eject
Any comments or questions are welcomed.
Thanks,
Kei
On Thu, Apr 15, 2004 at 05:09:39PM +0900, Keiichiro Tokunaga wrote:
>
> 1. Current sysfs for hotplug
> ----------------------------
> In 2.6, there are some sysfs files for hotplug. For instance, PCI has that
> kind of files to control hotplug features. PCI related sysfs entries for hotplug
> are created under /sys/bus/pci/slots/:
>
> /sys/bus/pci/slots/<slot#>/<files>
>
> There are some files under <slot#> directory. Some of them are supposed
> to expose kenrel information, the others are supposed to control hotplug.
> Recently, CPU and memory hotplug are being worked actively. According
> to their patch, they seem to be trying to create sysfs files under the following
> directory.
>
> /sys/devices/system/
That's correct, as cpus and memory are system devices, while pci slots
belong in the pci bus directory in sysfs.
> 2. Problem
There is no problem :)
> Recent large machines have many PCI devices and some boards that
> contain devices (e.g. CPU, memory, and/or I/O devices). A certain PCI
> device (PCI1) might be connected with other one (PCI2), which means that
> there is a dependency between PCI1 and PCI2.
You have this today? On what platform? This is the first I have heard
of this. If needed, we can merely change the pci hotplug core to allow
a hierarchy of pci slots. Will that solve your problem?
> 3. Suggestion
> -------------
> To solve the problem, I'd like to propose the following idea.
>
> ["hotplug" directory]
> This directory is to represent a hierarchy of hotpluggable devices.
Hm, no. What about usb, firewire, scsi and any other future bus that
can be "hotpluggable". The kernel doesn't treat them differently, and
we shouldn't either.
> "hotpluggable device" means a device that can be powered off and
> removed physically from the system running. The hierarchy describes a
> dependency between each device. This directory would be placed, like:
>
> /sys/devices/hotplug
>
> Any systems that enable hotplug (e.g. ACPI, DLPAR) can create their
> own directory right under the "hotplug" directory, like:
>
> /sys/devices/hotplug/acpi
> /sys/devices/hotplug/dlpar
>
> Each of systems can create directories and files under the own directory,
> and these directories should be easy for user to use.
>
>
> [ACPI based Hotplug Case]
> I think that ACPI is one of the systems tha know dependencies of devices.
But it doesn't know about all devices in the system (like USB, firewire
and others), so this would quickly break down. I also don't like
creating a solution that is so hard-wired for one firmware type like
ACPI. What about Open Firmware based machines? Pure BIOS machines? No
firmware at all machines? The current sysfs trees work just fine for
all of them, without users having to figure out what the access type the
kernel uses to get to the devices.
thanks,
greg k-h
On Fri, Apr 16, 2004 at 03:34:36PM -0700, Greg KH wrote:
> > Recent large machines have many PCI devices and some boards that
> > contain devices (e.g. CPU, memory, and/or I/O devices). A certain PCI
> > device (PCI1) might be connected with other one (PCI2), which means that
> > there is a dependency between PCI1 and PCI2.
>
> You have this today?
I interpreted his comments to mean PCI-PCI Bridges.
eg something like a 4-port NIC which usually has a PCI-PCI bridge
to "isolate" multiple PCI devices (NICs):
+-[60]---01.0-[61]--+-04.0 Digital Equipment Corporation DECchip 21142/43
| +-05.0 Digital Equipment Corporation DECchip 21142/43
| +-06.0 Digital Equipment Corporation DECchip 21142/43
| \-07.0 Digital Equipment Corporation DECchip 21142/43
...
I thought this was already handled though so I may be misunderstanding.
Keiichiro, an example would be very helpful in understanding.
...
> Hm, no. What about usb, firewire, scsi and any other future bus that
> can be "hotpluggable". The kernel doesn't treat them differently, and
> we shouldn't either.
SCSI has a heirarchy as well. Ie LUNs can be removed without
removing the target (RAID controllers). Normal JBOD use equates
LUNs and targets.
grant
Hi Greg,
Thanks for the comments:)
Greg KH wrote:
> > 2. Problem
>
> There is no problem :)
>
> > Recent large machines have many PCI devices and some boards that
> > contain devices (e.g. CPU, memory, and/or I/O devices). A certain PCI
> > device (PCI1) might be connected with other one (PCI2), which means that
> > there is a dependency between PCI1 and PCI2.
>
> You have this today? On what platform? This is the first I have heard
> of this. If needed, we can merely change the pci hotplug core to allow
> a hierarchy of pci slots. Will that solve your problem?
I meant that a P2P bridge (that has hotpluggable slots) and a PCI device would
have such a dependency. As you suggeted, if the PCI hotplug core is changed
that way, the dependency would be represented in sysfs quite well:) However,
a board that contains CPU, memory and/or I/O devices still doesn't have a
directory in sysfs to represent dependencies... Actually, I'm focusing on hotplug
features for that kind of the boards, and making a patch that enables it. That
patch will be coming out soom.
> > 3. Suggestion
> > -------------
> > To solve the problem, I'd like to propose the following idea.
> >
> > ["hotplug" directory]
> > This directory is to represent a hierarchy of hotpluggable devices.
>
> Hm, no. What about usb, firewire, scsi and any other future bus that
> can be "hotpluggable". The kernel doesn't treat them differently, and
> we shouldn't either.
>
> > "hotpluggable device" means a device that can be powered off and
> > removed physically from the system running. The hierarchy describes a
> > dependency between each device. This directory would be placed, like:
> >
> > /sys/devices/hotplug
> >
> > Any systems that enable hotplug (e.g. ACPI, DLPAR) can create their
> > own directory right under the "hotplug" directory, like:
> >
> > /sys/devices/hotplug/acpi
> > /sys/devices/hotplug/dlpar
> >
> > Each of systems can create directories and files under the own directory,
> > and these directories should be easy for user to use.
> >
> >
> > [ACPI based Hotplug Case]
> > I think that ACPI is one of the systems tha know dependencies of devices.
>
> But it doesn't know about all devices in the system (like USB, firewire
> and others), so this would quickly break down. I also don't like
> creating a solution that is so hard-wired for one firmware type like
> ACPI. What about Open Firmware based machines? Pure BIOS machines? No
> firmware at all machines? The current sysfs trees work just fine for
> all of them, without users having to figure out what the access type the
> kernel uses to get to the devices.
That's right. /sys/devices/hotplug/ACPI/ tree becomes hard-wired one. I was
thinking to define the board by using ACPI (as a "generic container device" in
ACPI namespace). Therefore, if there is the new tree I proposed in the kernel,
it would be easy to represent the hierarchy, and a directory for the board
appears in the new tree. So I thought that we could put an control file to
invoke the board hotplug and an information file under the directory.
(Actually, I've made a rough patch for the new tree and it seems to work fine:)
I also thought that interface for hotplug could be unified so that it would become
easier for user to use.
However, it's a hard-wired way and the current sysfs trees work fine for all of
devices as you mentioned. Now I have just one thing necessary to sysfs.
That's a directory and files for the board. Should I abstract the "board" and
introduce a new directory for board under /sys/devices/system/, like NUMA
node directory? (e.g. /sys/devices/system/board/) The control file, the
information file, and etc could be created under the directory, like
/sys/devices/hotplug/board/board0/eject. If it's possible, there might be less
impact to the kernel. I'd appreciate it if you would comment on this :)
Thanks,
Kei
Hi Grant,
Thanks for the comments:)
Grant wrote:
> On Fri, Apr 16, 2004 at 03:34:36PM -0700, Greg KH wrote:
> > > Recent large machines have many PCI devices and some boards that
> > > contain devices (e.g. CPU, memory, and/or I/O devices). A certain PCI
> > > device (PCI1) might be connected with other one (PCI2), which means that
> > > there is a dependency between PCI1 and PCI2.
> >
> > You have this today?
>
> I interpreted his comments to mean PCI-PCI Bridges.
> eg something like a 4-port NIC which usually has a PCI-PCI bridge
> to "isolate" multiple PCI devices (NICs):
> +-[60]---01.0-[61]--+-04.0 Digital Equipment Corporation DECchip 21142/43
> | +-05.0 Digital Equipment Corporation DECchip 21142/43
> | +-06.0 Digital Equipment Corporation DECchip 21142/43
> | \-07.0 Digital Equipment Corporation DECchip 21142/43
> ...
>
> I thought this was already handled though so I may be misunderstanding.
> Keiichiro, an example would be very helpful in understanding.
As in an email I sent to Greg, P2P bridge that has hotpluggable slots need
to be represented in a hierarchy style. I don't think that kind of P2P bridge
is handled yet.
Thanks,
Kei
On Fri, Apr 23, 2004 at 09:18:16PM +0900, Keiichiro Tokunaga wrote:
> That's right. /sys/devices/hotplug/ACPI/ tree becomes hard-wired one. I was
> thinking to define the board by using ACPI (as a "generic container device" in
> ACPI namespace). Therefore, if there is the new tree I proposed in the kernel,
> it would be easy to represent the hierarchy, and a directory for the board
> appears in the new tree. So I thought that we could put an control file to
> invoke the board hotplug and an information file under the directory.
> (Actually, I've made a rough patch for the new tree and it seems to work fine:)
> I also thought that interface for hotplug could be unified so that it would become
> easier for user to use.
Have you seen Alex Williamson's patch to fill out the /sys/firmware/acpi tree?
http://marc.theaimsgroup.com/?l=linux-kernel&m=108239031314885&w=2
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain
On Fri, Apr 23, 2004 at 09:18:16PM +0900, Keiichiro Tokunaga wrote:
It is EXTREMELY rude to crosspost between closed and open lists. Take
the lhcs-devel list off the cc list in all further posts. Thank you.
--
"Next the statesmen will invent cheap lies, putting the blame upon
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince
himself that the war is just, and will thank God for the better sleep
he enjoys after this process of grotesque self-deception." -- Mark Twain
On Fri, 23 Apr 2004 13:30:57 +0100
Matthew Wilcox <[email protected]> wrote:
> On Fri, Apr 23, 2004 at 09:18:16PM +0900, Keiichiro Tokunaga wrote:
> > That's right. /sys/devices/hotplug/ACPI/ tree becomes hard-wired one. I was
> > thinking to define the board by using ACPI (as a "generic container device" in
> > ACPI namespace). Therefore, if there is the new tree I proposed in the kernel,
> > it would be easy to represent the hierarchy, and a directory for the board
> > appears in the new tree. So I thought that we could put an control file to
> > invoke the board hotplug and an information file under the directory.
> > (Actually, I've made a rough patch for the new tree and it seems to work fine:)
> > I also thought that interface for hotplug could be unified so that it would become
> > easier for user to use.
>
> Have you seen Alex Williamson's patch to fill out the /sys/firmware/acpi tree?
> http://marc.theaimsgroup.com/?l=linux-kernel&m=108239031314885&w=2
Yes, I have. I understand he's trying to expose objects and control methods
under /sys/firmware/acpi/namespace/ACPI/ tree and user can access it. Actually,
I'm interested in hotplug part he mentioned and wondering if it's possible to
run _EJ0 method without any hotplug processing. That's what I know so far,
since I didn't know he has already relased the new patch. Thanks for the
information, I'll take a look at that :)
I'm sorry for the crossposting. I just wanted to hear from every hotplug related
people as possible... Now I'm sending this to you and lkml.
Thanks,
Kei
On Fri, Apr 23, 2004 at 09:18:16PM +0900, Keiichiro Tokunaga wrote:
> Greg KH wrote:
> > > 2. Problem
> >
> > There is no problem :)
> >
> > > Recent large machines have many PCI devices and some boards that
> > > contain devices (e.g. CPU, memory, and/or I/O devices). A certain PCI
> > > device (PCI1) might be connected with other one (PCI2), which means that
> > > there is a dependency between PCI1 and PCI2.
> >
> > You have this today? On what platform? This is the first I have heard
> > of this. If needed, we can merely change the pci hotplug core to allow
> > a hierarchy of pci slots. Will that solve your problem?
>
>
> I meant that a P2P bridge (that has hotpluggable slots) and a PCI device would
> have such a dependency.
But you don't need to show that for any reason, right? Today, pci
slots behind a pluggable P2P bridge work just fine. I can remove the
entire drawer full of pci slots and they all go away properly, and if I
add a new drawer of pci slots, they all show up.
Why do you want to show that hierarchy? Who cares about it?
> As you suggeted, if the PCI hotplug core is changed that way, the
> dependency would be represented in sysfs quite well:)
I don't think the PCI Hotplug core needs to change today to support
these kinds of devices based on the hardware I have seen. I have spoken
to some people from other companies, and they also agree with me. But
if you have some different kind of hardware that really needs this, I'm
open to ideas.
> However, a board that contains CPU, memory and/or I/O devices still
> doesn't have a directory in sysfs to represent dependencies...
Why would it need to?
Right now you can determine the CPU that a pci bus is attached to
through sysfs. As for the CPU and memory depiction, talk to the NUMA
developers. They have been trying for years now to come up with a way
to do this in a portable and proper manner, and so far have failed :(
>
> That's right. /sys/devices/hotplug/ACPI/ tree becomes hard-wired one. I was
> thinking to define the board by using ACPI (as a "generic container device" in
> ACPI namespace). Therefore, if there is the new tree I proposed in the kernel,
> it would be easy to represent the hierarchy, and a directory for the board
> appears in the new tree. So I thought that we could put an control file to
> invoke the board hotplug and an information file under the directory.
> (Actually, I've made a rough patch for the new tree and it seems to work fine:)
Patches are better seen than spoken about in the abstract :) Please
post them if you have them...
> I also thought that interface for hotplug could be unified so that it would become
> easier for user to use.
But the user doesn't care about ACPI. Actually they want nothing to do
with ACPI namespaces :) They just want to be able to add and remove
their different devices, be them memory, cpus, or pci slots, right?
I'd point you to the recent ACPI sysfs patches on linux-kernel for more
information on what people are trying to do in that area.
> However, it's a hard-wired way and the current sysfs trees work fine for all of
> devices as you mentioned. Now I have just one thing necessary to sysfs.
> That's a directory and files for the board. Should I abstract the "board" and
> introduce a new directory for board under /sys/devices/system/, like NUMA
> node directory? (e.g. /sys/devices/system/board/) The control file, the
> information file, and etc could be created under the directory, like
> /sys/devices/hotplug/board/board0/eject. If it's possible, there might be less
> impact to the kernel. I'd appreciate it if you would comment on this :)
But writing to that "eject" file would not be able to go and turn off
the different CPU's, memory sticks, and pci slots, right? That still
needs to be done from userspace.
thanks,
greg k-h
On Fri, 23 Apr 2004 13:07:51 -0700, Greg KH <[email protected]> wrote:
> On Fri, Apr 23, 2004 at 09:18:16PM +0900, Keiichiro Tokunaga wrote:
> > Greg KH wrote:
> > > > 2. Problem
> > >
> > > There is no problem :)
> > >
> > > > Recent large machines have many PCI devices and some boards that
> > > > contain devices (e.g. CPU, memory, and/or I/O devices). A certain PCI
> > > > device (PCI1) might be connected with other one (PCI2), which means that
> > > > there is a dependency between PCI1 and PCI2.
> > >
> > > You have this today? On what platform? This is the first I have heard
> > > of this. If needed, we can merely change the pci hotplug core to allow
> > > a hierarchy of pci slots. Will that solve your problem?
> >
> >
> > I meant that a P2P bridge (that has hotpluggable slots) and a PCI device would
> > have such a dependency.
>
> But you don't need to show that for any reason, right? Today, pci
> slots behind a pluggable P2P bridge work just fine. I can remove the
> entire drawer full of pci slots and they all go away properly, and if I
> add a new drawer of pci slots, they all show up.
>
> Why do you want to show that hierarchy? Who cares about it?
>
> > As you suggeted, if the PCI hotplug core is changed that way, the
> > dependency would be represented in sysfs quite well:)
>
> I don't think the PCI Hotplug core needs to change today to support
> these kinds of devices based on the hardware I have seen. I have spoken
> to some people from other companies, and they also agree with me. But
> if you have some different kind of hardware that really needs this, I'm
> open to ideas.
I have to say two things here. First of all, I misunderstood that
PCI slots behind a pluggable P2P bridge didn't work. I'm sorry
about it. Second of all, I proposed the hierarchy just because I
thought it's nice for user, but I don't push this any more:). I agree
with you that the PCI Hotplug core doesn't need to change.
> > However, a board that contains CPU, memory and/or I/O devices still
> > doesn't have a directory in sysfs to represent dependencies...
>
> Why would it need to?
Please see my comment on this part below.
> Right now you can determine the CPU that a pci bus is attached to
> through sysfs. As for the CPU and memory depiction, talk to the NUMA
> developers. They have been trying for years now to come up with a way
> to do this in a portable and proper manner, and so far have failed :(
>
> >
> > That's right. /sys/devices/hotplug/ACPI/ tree becomes hard-wired one. I was
> > thinking to define the board by using ACPI (as a "generic container device" in
> > ACPI namespace). Therefore, if there is the new tree I proposed in the kernel,
> > it would be easy to represent the hierarchy, and a directory for the board
> > appears in the new tree. So I thought that we could put an control file to
> > invoke the board hotplug and an information file under the directory.
> > (Actually, I've made a rough patch for the new tree and it seems to work fine:)
>
> Patches are better seen than spoken about in the abstract :) Please
> post them if you have them...
>
> > I also thought that interface for hotplug could be unified so that it would become
> > easier for user to use.
>
> But the user doesn't care about ACPI. Actually they want nothing to do
> with ACPI namespaces :) They just want to be able to add and remove
> their different devices, be them memory, cpus, or pci slots, right?
>
> I'd point you to the recent ACPI sysfs patches on linux-kernel for more
> information on what people are trying to do in that area.
>
> > However, it's a hard-wired way and the current sysfs trees work fine for all of
> > devices as you mentioned. Now I have just one thing necessary to sysfs.
> > That's a directory and files for the board. Should I abstract the "board" and
> > introduce a new directory for board under /sys/devices/system/, like NUMA
> > node directory? (e.g. /sys/devices/system/board/) The control file, the
> > information file, and etc could be created under the directory, like
> > /sys/devices/hotplug/board/board0/eject. If it's possible, there might be less
> > impact to the kernel. I'd appreciate it if you would comment on this :)
>
> But writing to that "eject" file would not be able to go and turn off
> the different CPU's, memory sticks, and pci slots, right? That still
> needs to be done from userspace.
I think it depends on a hotplug driver that is invoked when writing to
a "eject" file. In the board case, a board hotplug driver (I'm making)
handles those CPUs, memory, and PCI slots on the board. So my
story for board hotplug is:
- user checks/knows what resources are on the board (dependency)
- user writes to the "eject" file of the board properly (invocation)
I'm expecting the first one would be available with sysfs (e.g. a hierarchy,
an information file, etc). Plus, the "eject" file would be created in sysfs.
Thanks,
Kei
On Mon, Apr 26, 2004 at 02:58:08PM +0900, Keiichiro Tokunaga wrote:
>
> I think it depends on a hotplug driver that is invoked when writing to
> a "eject" file. In the board case, a board hotplug driver (I'm making)
> handles those CPUs, memory, and PCI slots on the board. So my
> story for board hotplug is:
>
> - user checks/knows what resources are on the board (dependency)
> - user writes to the "eject" file of the board properly (invocation)
Why not make a program that does all of this from userspace? It would
turn off the proper CPUs, memory, and pci slots for a specific "board".
Otherwise you are going to have to either:
- hook the current CPU, memory, and pci hotplug code to allow it
to be called from within the kernel
- have your kernel code write to a the sysfs files from within
kernelspace.
Neither of which are acceptable things :(
thanks,
greg k-h