Greg KH, on 10/22/2010 10:54 PM wrote:
> On Fri, Oct 22, 2010 at 10:40:34PM +0400, Vladislav Bolkhovitin wrote:
>>>> + unsigned int tgt_kobj_initialized:1;
>>>
>>> It's the middle of the merge window, and I'm about to go on vacation, so
>>> I didn't read this patch after this line.
>>>
>>> It's obvious that this patch is wrong, you shouldn't need to worry about
>>> this. And even if you did, you don't need this flag.
>>>
>>> Why are you trying to do something that no one else needs? Why make
>>> things harder than they have to be.
>>
>> I tried to explain that to you in http://lkml.org/lkml/2010/10/14/291
>> and mentioned there the need to create this flag to track
>> half-initialized kobjects. You agreed
>> (http://lkml.org/lkml/2010/10/14/299) that don't return half-initialized
>> objects is a regular kernel practice, but then requested to strictly
>> bound the larger object freeing to its kobject release(), which means
>> that all SYSFS creating functions now have to return half-initialized
>> SYSFS hierarchy in case of any error. Hence the flag to track it.
>
> I agreed that you needed to do something about it, not that this is the
> correct way to do it.
>
> Think for a second as to why your code path looks different from EVERY
> other kobject user in the kernel. Perhaps it is incorrect? You don't
> need all this completion mess, in fact, it's wrong.
>
> Just do what everyone else does please, as that is the simpler, and
> correct, way to do it.
Hello Greg,
Why SCST objects are different from most other kernel objects and why
they can't be implemented the same way as them is exactly what I'm
trying to explain you. Let me try again from a bit different angle.
SCST objects are different from the most other kernel objects, because
they are very complex, hence complex to initialize and delete with
dependencies in order how initialization and delete actions should be
performed. Particularly, SCST objects have a lot of attributes and
sub-objects, so their kobjects can't be inited and exposed to SYSFS or
removed from it until they reach some point during initialization or
delete correspondingly, otherwise their attributes' reads and writes can
crash.
I can elaborate all those on examples, if you request.
I understand you are very busy and appreciate very much your review and
comments, so be glad to implement what you are requesting. But I don't
see how to implement that in an acceptable manner with neither the
completion-based delete as now, nor with x_initialized flag as in the
previous patch.
Note, SCST's kobjects are not the only kobjects in the kernel using the
completion based delete. See, for instance, ktype_cpufreq, ktype_cpuidle
or ext4_ktype.
Also, elevator (struct elevator_queue) uses "registered" flag to see if
it was added to the SYSFS.
Thus, you can see, both the completion and the flag approaches already
used in the kernel. Hence, I believe, the way how SCST is doing it
should also be acceptable.
I'm not a person who, as you, probably, guessed, at first doing, only
then thinking and reading docs. I before writing a code line at first
read all the available docs and carefully consider possible
alternatives, so there isn't a well thought bit in SCST.
I believe, the completions aren't mess, they are refinement.
> Oh, and why are you using a kobject at all anyway? Shouldn't you be
> using a 'struct device'?
We need only to represent our internal SCST objects on the SYSFS. The
objects are not devices, but special entities for special purposes. For
instance, struct scst_session is a representation of SCSI I_T nexuses.
Struct scst_tgt_dev - I_T_L nexuses. Another example is struct scst_acg,
which is a representation of per initiator access control groups to
determine which initiators can see which LUNs. Hence, for such purpose
kobjects are fully sufficient.
Thanks,
Vlad
On Mon, Nov 08, 2010 at 10:58:37PM +0300, Vladislav Bolkhovitin wrote:
> Greg KH, on 10/22/2010 10:54 PM wrote:
> > On Fri, Oct 22, 2010 at 10:40:34PM +0400, Vladislav Bolkhovitin wrote:
> >>>> + unsigned int tgt_kobj_initialized:1;
> >>>
> >>> It's the middle of the merge window, and I'm about to go on vacation, so
> >>> I didn't read this patch after this line.
> >>>
> >>> It's obvious that this patch is wrong, you shouldn't need to worry about
> >>> this. And even if you did, you don't need this flag.
> >>>
> >>> Why are you trying to do something that no one else needs? Why make
> >>> things harder than they have to be.
> >>
> >> I tried to explain that to you in http://lkml.org/lkml/2010/10/14/291
> >> and mentioned there the need to create this flag to track
> >> half-initialized kobjects. You agreed
> >> (http://lkml.org/lkml/2010/10/14/299) that don't return half-initialized
> >> objects is a regular kernel practice, but then requested to strictly
> >> bound the larger object freeing to its kobject release(), which means
> >> that all SYSFS creating functions now have to return half-initialized
> >> SYSFS hierarchy in case of any error. Hence the flag to track it.
> >
> > I agreed that you needed to do something about it, not that this is the
> > correct way to do it.
> >
> > Think for a second as to why your code path looks different from EVERY
> > other kobject user in the kernel. Perhaps it is incorrect? You don't
> > need all this completion mess, in fact, it's wrong.
> >
> > Just do what everyone else does please, as that is the simpler, and
> > correct, way to do it.
>
> Hello Greg,
>
> Why SCST objects are different from most other kernel objects and why
> they can't be implemented the same way as them is exactly what I'm
> trying to explain you. Let me try again from a bit different angle.
I'm sorry, but I just don't buy it.
> SCST objects are different from the most other kernel objects, because
> they are very complex, hence complex to initialize and delete with
> dependencies in order how initialization and delete actions should be
> performed.
Then don't abuse kobjects with this "different" type of kobject, as that
is not how the kobject code was designed to be used.
It was only designed to be used with the "sane" type of kernel objects
:)
> Particularly, SCST objects have a lot of attributes and sub-objects,
> so their kobjects can't be inited and exposed to SYSFS or removed from
> it until they reach some point during initialization or delete
> correspondingly, otherwise their attributes' reads and writes can
> crash.
That sounds like an implementation error. No other kernel code has that
problem from what I can see.
> Note, SCST's kobjects are not the only kobjects in the kernel using the
> completion based delete. See, for instance, ktype_cpufreq, ktype_cpuidle
> or ext4_ktype.
ext4 is using this to get stuff under the /sys/fs/ext4 location. And
even there, I don't think it is using kobjects correctly, but I really
don't want to go audit that code at the moment.
cpufreq and cpuidle is also probably incorrect, but again, I don't feel
like auditing it right now.
Basically, don't copy bad examples as a valid usage model for the code.
> Also, elevator (struct elevator_queue) uses "registered" flag to see if
> it was added to the SYSFS.
It's wrong and should be fixed then.
> > Oh, and why are you using a kobject at all anyway? Shouldn't you be
> > using a 'struct device'?
>
> We need only to represent our internal SCST objects on the SYSFS. The
> objects are not devices, but special entities for special purposes. For
> instance, struct scst_session is a representation of SCSI I_T nexuses.
> Struct scst_tgt_dev - I_T_L nexuses. Another example is struct scst_acg,
> which is a representation of per initiator access control groups to
> determine which initiators can see which LUNs. Hence, for such purpose
> kobjects are fully sufficient.
No, you should be using a struct device as you are putting stuff into
the device tree. NEVER put a kobject into the device tree, that is just
wrong and will cause problems.
thanks,
greg k-h
Greg KH, on 11/09/2010 03:28 AM wrote:
> On Mon, Nov 08, 2010 at 10:58:37PM +0300, Vladislav Bolkhovitin wrote:
>> Greg KH, on 10/22/2010 10:54 PM wrote:
>>> On Fri, Oct 22, 2010 at 10:40:34PM +0400, Vladislav Bolkhovitin wrote:
>>>>>> + unsigned int tgt_kobj_initialized:1;
>>>>>
>>>>> It's the middle of the merge window, and I'm about to go on vacation, so
>>>>> I didn't read this patch after this line.
>>>>>
>>>>> It's obvious that this patch is wrong, you shouldn't need to worry about
>>>>> this. And even if you did, you don't need this flag.
>>>>>
>>>>> Why are you trying to do something that no one else needs? Why make
>>>>> things harder than they have to be.
>>>>
>>>> I tried to explain that to you in http://lkml.org/lkml/2010/10/14/291
>>>> and mentioned there the need to create this flag to track
>>>> half-initialized kobjects. You agreed
>>>> (http://lkml.org/lkml/2010/10/14/299) that don't return half-initialized
>>>> objects is a regular kernel practice, but then requested to strictly
>>>> bound the larger object freeing to its kobject release(), which means
>>>> that all SYSFS creating functions now have to return half-initialized
>>>> SYSFS hierarchy in case of any error. Hence the flag to track it.
>>>
>>> I agreed that you needed to do something about it, not that this is the
>>> correct way to do it.
>>>
>>> Think for a second as to why your code path looks different from EVERY
>>> other kobject user in the kernel. Perhaps it is incorrect? You don't
>>> need all this completion mess, in fact, it's wrong.
>>>
>>> Just do what everyone else does please, as that is the simpler, and
>>> correct, way to do it.
>>
>> Hello Greg,
>>
>> Why SCST objects are different from most other kernel objects and why
>> they can't be implemented the same way as them is exactly what I'm
>> trying to explain you. Let me try again from a bit different angle.
>
> I'm sorry, but I just don't buy it.
>
>> SCST objects are different from the most other kernel objects, because
>> they are very complex, hence complex to initialize and delete with
>> dependencies in order how initialization and delete actions should be
>> performed.
>
> Then don't abuse kobjects with this "different" type of kobject, as that
> is not how the kobject code was designed to be used.
>
> It was only designed to be used with the "sane" type of kernel objects
> :)
SCST objects are generally 1:1 mapping of SCSI Architecture Model (SAM)
objects. Is SAM insane and should have been designed with kobjects in
mind? ;)
>> Particularly, SCST objects have a lot of attributes and sub-objects,
>> so their kobjects can't be inited and exposed to SYSFS or removed from
>> it until they reach some point during initialization or delete
>> correspondingly, otherwise their attributes' reads and writes can
>> crash.
>
> That sounds like an implementation error. No other kernel code has that
> problem from what I can see.
>
>> Note, SCST's kobjects are not the only kobjects in the kernel using the
>> completion based delete. See, for instance, ktype_cpufreq, ktype_cpuidle
>> or ext4_ktype.
>
> ext4 is using this to get stuff under the /sys/fs/ext4 location.
"To get stuff under /sys/..." is exactly for what SCST needs kobjects.
SCST objects are user interface agnostic and can be equally well exposed
to user space via SYSFS, PROCFS, or anything else, like CONFIGFS or
custom IOCTLs.
> And
> even there, I don't think it is using kobjects correctly, but I really
> don't want to go audit that code at the moment.
>
> cpufreq and cpuidle is also probably incorrect, but again, I don't feel
> like auditing it right now.
Sorry, but what is incorrect in the working implementation without any
bugs doing its job in the simplest, smallest and clearest way?
If those objects remade to free themselves in the kobjects release(),
what value would it add to them? Would the implementation be simpler,
smaller or clearer? Not, I believe, new implementation would be only
bigger and less clear. So, what's the point to do it to make the code worse?
>>> Oh, and why are you using a kobject at all anyway? Shouldn't you be
>>> using a 'struct device'?
>>
>> We need only to represent our internal SCST objects on the SYSFS. The
>> objects are not devices, but special entities for special purposes. For
>> instance, struct scst_session is a representation of SCSI I_T nexuses.
>> Struct scst_tgt_dev - I_T_L nexuses. Another example is struct scst_acg,
>> which is a representation of per initiator access control groups to
>> determine which initiators can see which LUNs. Hence, for such purpose
>> kobjects are fully sufficient.
>
> No, you should be using a struct device as you are putting stuff into
> the device tree. NEVER put a kobject into the device tree, that is just
> wrong and will cause problems.
Hmm, we never put any stuff into the device tree. Why do you think we
do? None of SCST objects has any relation to any hardware device, hence
there's no need to touch the device tree.
Thanks,
Vlad
On 11/09/2010 10:06 PM, Vladislav Bolkhovitin wrote:
>
> Sorry, but what is incorrect in the working implementation without any
> bugs doing its job in the simplest, smallest and clearest way?
>
> If those objects remade to free themselves in the kobjects release(),
> what value would it add to them? Would the implementation be simpler,
> smaller or clearer? Not, I believe, new implementation would be only
> bigger and less clear. So, what's the point to do it to make the code worse?
>
Totally theoretically speaking, since I have not inspected the code.
If today you wait for the count to reach zero, then unregister
and send an event to some other subsystem to free the object.
Is it not the same as if you take an extra refcount, unregister and
send the event at count=1. Then at that other place decrement the last
count to cause the object to be freed.
I agree that it is hard to do lockless. what some places do is have
an extra kref. The kobj has a single ref on it. everything takes the
other kref. when that reaches zero the unregister and event fires
and at free you decrement the only kobj ref to deallocate. This is one
way. In some situations you can manage with a single counter it all
depends.
Boaz
Boaz Harrosh, on 11/10/2010 12:58 PM wrote:
> On 11/09/2010 10:06 PM, Vladislav Bolkhovitin wrote:
>>
>> Sorry, but what is incorrect in the working implementation without any
>> bugs doing its job in the simplest, smallest and clearest way?
>>
>> If those objects remade to free themselves in the kobjects release(),
>> what value would it add to them? Would the implementation be simpler,
>> smaller or clearer? Not, I believe, new implementation would be only
>> bigger and less clear. So, what's the point to do it to make the code worse?
>>
>
> Totally theoretically speaking, since I have not inspected the code.
>
> If today you wait for the count to reach zero, then unregister
> and send an event to some other subsystem to free the object.
>
> Is it not the same as if you take an extra refcount, unregister and
> send the event at count=1. Then at that other place decrement the last
> count to cause the object to be freed.
>
> I agree that it is hard to do lockless. what some places do is have
> an extra kref. The kobj has a single ref on it. everything takes the
> other kref. when that reaches zero the unregister and event fires
> and at free you decrement the only kobj ref to deallocate. This is one
> way. In some situations you can manage with a single counter it all
> depends.
Thanks for sharing your thoughts with us. But the question isn't about
if it's possible to implement what we need locklessly. The question is
in two approaches how to synchronously delete objects with entries on SYSFS:
1. struct object_x {
...
struct kobject kobj;
struct completion *release_completion;
};
static void x_release(struct kobject *kobj)
{
struct object_x *x;
struct completion *c;
x = container_of(kobj, struct object_x, kobj);
c = x->release_completion;
kfree(x);
complete_all(c);
}
void del_object(struct object_x *x)
{
DECLARE_COMPLETION_ONSTACK(completion);
...
x->release_completion = &completion;
kobject_put(&x->kobj);
wait_for_completion(&completion);
}
and
2. struct object_x {
...
struct kobject kobj;
struct completion release_completion;
};
static void x_release(struct kobject *kobj)
{
struct object_x *x;
x = container_of(kobj, struct object_x, kobj);
complete_all(&x->release_completion);
}
void del_object(struct object_x *x)
{
...
kobject_put(&x->kobj);
wait_for_completion(&completion);
...
kfree(x);
}
Greg asserts that (1) is the only correct approach while (2) is
incorrect, and I'm trying to justify that (2) is correct too and
sometimes could be better, i.e. simpler and clearer, because it
decouples object_x from SYSFS and its kobj. Then kobj becomes an
ordinary member of struct object_x without any special treatment and
with the same lifetime rules as other members of struct object_x. While
in (1) all lifetime of struct object_x is strictly attached to kobj, so
it needs be specially handled with additional code for that if struct
object_x has many other members which needed to be initialized/deleted
_before and after_ kobj as we have in SCST.
Vlad
On 11/10/10 12:19 PM, Vladislav Bolkhovitin wrote:
> Boaz Harrosh, on 11/10/2010 12:58 PM wrote:
>> On 11/09/2010 10:06 PM, Vladislav Bolkhovitin wrote:
>>>
>>> Sorry, but what is incorrect in the working implementation without any
>>> bugs doing its job in the simplest, smallest and clearest way?
>>>
>>> If those objects remade to free themselves in the kobjects release(),
>>> what value would it add to them? Would the implementation be simpler,
>>> smaller or clearer? Not, I believe, new implementation would be only
>>> bigger and less clear. So, what's the point to do it to make the code worse?
>>>
>>
>> Totally theoretically speaking, since I have not inspected the code.
>>
>> If today you wait for the count to reach zero, then unregister
>> and send an event to some other subsystem to free the object.
>>
>> Is it not the same as if you take an extra refcount, unregister and
>> send the event at count=1. Then at that other place decrement the last
>> count to cause the object to be freed.
>>
>> I agree that it is hard to do lockless. what some places do is have
>> an extra kref. The kobj has a single ref on it. everything takes the
>> other kref. when that reaches zero the unregister and event fires
>> and at free you decrement the only kobj ref to deallocate. This is one
>> way. In some situations you can manage with a single counter it all
>> depends.
>
> Thanks for sharing your thoughts with us. But the question isn't about
> if it's possible to implement what we need locklessly. The question is
> in two approaches how to synchronously delete objects with entries on SYSFS:
>
> 1. struct object_x {
> ...
> struct kobject kobj;
> struct completion *release_completion;
> };
>
> static void x_release(struct kobject *kobj)
> {
> struct object_x *x;
> struct completion *c;
>
> x = container_of(kobj, struct object_x, kobj);
> c = x->release_completion;
> kfree(x);
> complete_all(c);
> }
>
> void del_object(struct object_x *x)
> {
> DECLARE_COMPLETION_ONSTACK(completion);
>
> ...
> x->release_completion = &completion;
> kobject_put(&x->kobj);
> wait_for_completion(&completion);
> }
>
> and
>
> 2. struct object_x {
> ...
> struct kobject kobj;
> struct completion release_completion;
> };
>
> static void x_release(struct kobject *kobj)
> {
> struct object_x *x;
>
> x = container_of(kobj, struct object_x, kobj);
> complete_all(&x->release_completion);
> }
>
> void del_object(struct object_x *x)
> {
> ...
> kobject_put(&x->kobj);
> wait_for_completion(&completion);
> ...
> kfree(x);
> }
I'll admit I don't understand this all that well, but
why not just have x_release() (based on (2))
do free(x), and have del_object
do the kobject_put(&x->kobj) as its very last thing?
Then you don't need the completion.
>
> Greg asserts that (1) is the only correct approach while (2) is
> incorrect, and I'm trying to justify that (2) is correct too and
> sometimes could be better, i.e. simpler and clearer, because it
> decouples object_x from SYSFS and its kobj. Then kobj becomes an
> ordinary member of struct object_x without any special treatment and
> with the same lifetime rules as other members of struct object_x. While
> in (1) all lifetime of struct object_x is strictly attached to kobj, so
> it needs be specially handled with additional code for that if struct
> object_x has many other members which needed to be initialized/deleted
> _before and after_ kobj as we have in SCST.
>
> Vlad
Joe Eykholt, on 11/10/2010 11:29 PM wrote:
>> Thanks for sharing your thoughts with us. But the question isn't about
>> if it's possible to implement what we need locklessly. The question is
>> in two approaches how to synchronously delete objects with entries on SYSFS:
>>
>> 1. struct object_x {
>> ...
>> struct kobject kobj;
>> struct completion *release_completion;
>> };
>>
>> static void x_release(struct kobject *kobj)
>> {
>> struct object_x *x;
>> struct completion *c;
>>
>> x = container_of(kobj, struct object_x, kobj);
>> c = x->release_completion;
>> kfree(x);
>> complete_all(c);
>> }
>>
>> void del_object(struct object_x *x)
>> {
>> DECLARE_COMPLETION_ONSTACK(completion);
>>
>> ...
>> x->release_completion = &completion;
>> kobject_put(&x->kobj);
>> wait_for_completion(&completion);
>> }
>>
>> and
>>
>> 2. struct object_x {
>> ...
>> struct kobject kobj;
>> struct completion release_completion;
>> };
>>
>> static void x_release(struct kobject *kobj)
>> {
>> struct object_x *x;
>>
>> x = container_of(kobj, struct object_x, kobj);
>> complete_all(&x->release_completion);
>> }
>>
>> void del_object(struct object_x *x)
>> {
>> ...
>> kobject_put(&x->kobj);
>> wait_for_completion(&completion);
>> ...
>> kfree(x);
>> }
>
> I'll admit I don't understand this all that well, but
> why not just have x_release() (based on (2))
> do free(x), and have del_object
> do the kobject_put(&x->kobj) as its very last thing?
> Then you don't need the completion.
We are discussing _synchronous_ delete of x, so need to wait until
x->kobj released, hence the completion is needed in both cases.
For instance, the sync delete is needed for targets to let the
corresponding target driver be safely unloaded after its target
unregistered.
Vlad
On 11/10/10 12:29 PM, Joe Eykholt wrote:
>
>
> On 11/10/10 12:19 PM, Vladislav Bolkhovitin wrote:
>> Boaz Harrosh, on 11/10/2010 12:58 PM wrote:
>>> On 11/09/2010 10:06 PM, Vladislav Bolkhovitin wrote:
>>>>
>>>> Sorry, but what is incorrect in the working implementation without any
>>>> bugs doing its job in the simplest, smallest and clearest way?
>>>>
>>>> If those objects remade to free themselves in the kobjects release(),
>>>> what value would it add to them? Would the implementation be simpler,
>>>> smaller or clearer? Not, I believe, new implementation would be only
>>>> bigger and less clear. So, what's the point to do it to make the code worse?
>>>>
>>>
>>> Totally theoretically speaking, since I have not inspected the code.
>>>
>>> If today you wait for the count to reach zero, then unregister
>>> and send an event to some other subsystem to free the object.
>>>
>>> Is it not the same as if you take an extra refcount, unregister and
>>> send the event at count=1. Then at that other place decrement the last
>>> count to cause the object to be freed.
>>>
>>> I agree that it is hard to do lockless. what some places do is have
>>> an extra kref. The kobj has a single ref on it. everything takes the
>>> other kref. when that reaches zero the unregister and event fires
>>> and at free you decrement the only kobj ref to deallocate. This is one
>>> way. In some situations you can manage with a single counter it all
>>> depends.
>>
>> Thanks for sharing your thoughts with us. But the question isn't about
>> if it's possible to implement what we need locklessly. The question is
>> in two approaches how to synchronously delete objects with entries on SYSFS:
>>
>> 1. struct object_x {
>> ...
>> struct kobject kobj;
>> struct completion *release_completion;
>> };
>>
>> static void x_release(struct kobject *kobj)
>> {
>> struct object_x *x;
>> struct completion *c;
>>
>> x = container_of(kobj, struct object_x, kobj);
>> c = x->release_completion;
>> kfree(x);
>> complete_all(c);
>> }
>>
>> void del_object(struct object_x *x)
>> {
>> DECLARE_COMPLETION_ONSTACK(completion);
>>
>> ...
>> x->release_completion = &completion;
>> kobject_put(&x->kobj);
>> wait_for_completion(&completion);
>> }
>>
>> and
>>
>> 2. struct object_x {
>> ...
>> struct kobject kobj;
>> struct completion release_completion;
>> };
>>
>> static void x_release(struct kobject *kobj)
>> {
>> struct object_x *x;
>>
>> x = container_of(kobj, struct object_x, kobj);
>> complete_all(&x->release_completion);
>> }
>>
>> void del_object(struct object_x *x)
>> {
>> ...
>> kobject_put(&x->kobj);
>> wait_for_completion(&completion);
>> ...
>> kfree(x);
>> }
>
> I'll admit I don't understand this all that well, but
> why not just have x_release() (based on (2))
> do free(x), and have del_object
> do the kobject_put(&x->kobj) as its very last thing?
> Then you don't need the completion.
Ah, well to answer my own question, I guess that's (1).
Never mind.
Joe
>> Greg asserts that (1) is the only correct approach while (2) is
>> incorrect, and I'm trying to justify that (2) is correct too and
>> sometimes could be better, i.e. simpler and clearer, because it
>> decouples object_x from SYSFS and its kobj. Then kobj becomes an
>> ordinary member of struct object_x without any special treatment and
>> with the same lifetime rules as other members of struct object_x. While
>> in (1) all lifetime of struct object_x is strictly attached to kobj, so
>> it needs be specially handled with additional code for that if struct
>> object_x has many other members which needed to be initialized/deleted
>> _before and after_ kobj as we have in SCST.
>>
>> Vlad
On 11/10/2010 10:19 PM, Vladislav Bolkhovitin wrote:
> Boaz Harrosh, on 11/10/2010 12:58 PM wrote:
>> On 11/09/2010 10:06 PM, Vladislav Bolkhovitin wrote:
>>>
>>> Sorry, but what is incorrect in the working implementation without any
>>> bugs doing its job in the simplest, smallest and clearest way?
>>>
>>> If those objects remade to free themselves in the kobjects release(),
>>> what value would it add to them? Would the implementation be simpler,
>>> smaller or clearer? Not, I believe, new implementation would be only
>>> bigger and less clear. So, what's the point to do it to make the code worse?
>>>
>>
>> Totally theoretically speaking, since I have not inspected the code.
>>
>> If today you wait for the count to reach zero, then unregister
>> and send an event to some other subsystem to free the object.
>>
>> Is it not the same as if you take an extra refcount, unregister and
>> send the event at count=1. Then at that other place decrement the last
>> count to cause the object to be freed.
>>
>> I agree that it is hard to do lockless. what some places do is have
>> an extra kref. The kobj has a single ref on it. everything takes the
>> other kref. when that reaches zero the unregister and event fires
>> and at free you decrement the only kobj ref to deallocate. This is one
>> way. In some situations you can manage with a single counter it all
>> depends.
>
> Thanks for sharing your thoughts with us. But the question isn't about
> if it's possible to implement what we need locklessly. The question is
> in two approaches how to synchronously delete objects with entries on SYSFS:
>
Thanks for putting up an example we can now speak more specifically.
(And saved me the time to actually look at code)
I'll first comment on your code below and I have some questions, please
see if I understood you correctly. Later below I'll try to explain what I
meant.
> 1. struct object_x {
> ...
> struct kobject kobj;
> struct completion *release_completion;
release_completion is only to be used by del_object!
> };
>
> static void x_release(struct kobject *kobj)
This one is put on the kobj.release Right?
> {
> struct object_x *x;
> struct completion *c;
>
> x = container_of(kobj, struct object_x, kobj);
> c = x->release_completion;
> kfree(x);
> complete_all(c);
> }
>
I don't see the unregister of the object_x.kobj, where do
you do this one in x_release or del_object below?
> void del_object(struct object_x *x)
> {
> DECLARE_COMPLETION_ONSTACK(completion);
>
> ...
> x->release_completion = &completion;
> kobject_put(&x->kobj);
This put might not be the last put on the object, IOs in flight
and/or open files might have extra reference on the object.
We release our initial ref, and below wait for all operations
to complete. (Is there a matter of timeout like files not closing?)
> wait_for_completion(&completion);
> }
>
> and
>
> 2. struct object_x {
> ...
> struct kobject kobj;
> struct completion release_completion;
> };
>
> static void x_release(struct kobject *kobj)
> {
> struct object_x *x;
>
> x = container_of(kobj, struct object_x, kobj);
> complete_all(&x->release_completion);
> }
>
> void del_object(struct object_x *x)
> {
DECLARE_COMPLETION_ONSTACK(completion);
x->release_completion = &completion;
Right?
> ...
> kobject_put(&x->kobj);
> wait_for_completion(&completion);
> ...
> kfree(x);
> }
>
> Greg asserts that (1) is the only correct approach while (2) is
> incorrect, and I'm trying to justify that (2) is correct too and
> sometimes could be better, i.e. simpler and clearer, because it
> decouples object_x from SYSFS and its kobj. Then kobj becomes an
> ordinary member of struct object_x without any special treatment and
> with the same lifetime rules as other members of struct object_x. While
> in (1) all lifetime of struct object_x is strictly attached to kobj, so
> it needs be specially handled with additional code for that if struct
> object_x has many other members which needed to be initialized/deleted
> _before and after_ kobj as we have in SCST.
>
> Vlad
One possibility (There are others)
3. struct object_x {
...
struct kref kref;
struct kobject kobj;
struct completion *release_completion;
};
Every body takes kref_put(&object_x.kref) and kref_get(&object_x.kref)
I hope you have x_get/x_put, Yes?
static void x_kref_release(struct kref *kref)
{
struct object_x *x = container_of(kref, struct object_x, kref);
complete_all(&x->release_completion);
}
static void x_obj_release(struct kobject *kobj)
{
struct object_x *x = container_of(kobj, struct object_x, kobj);
kfree(x);
}
int x_put(struct object_x *x)
{
return kref_put(&x->kref, x_kref_release);
}
void del_object(struct object_x *x)
{
DECLARE_COMPLETION_ONSTACK(completion);
...
x->release_completion = &completion;
x_put(x)
wait_for_completion(&completion);
kobject_put(&x->kobj);
}
Or
4. Exactly Like 3 but without the extra kref member
Only x_put() changes and x_kref_release() now receives
an x_object
int x_put(struct object_x *x)
{
if (kobject_put(&x->kobj) == 1)
// Like above [3] x_kref_release()
x_kref_release(x);
}
Note that in 4 you don't actually have a kref member, and that you have
one extra ref on kobj from the beginning. In del_object above the first
x_put(x) makes it possible to reach the "1" count and then the final
kobject_put(&x->kobj); frees the object.
(You need to be carfull with [4] because it must have a refcount==2 before
you expose it to any IO or sysfs.)
So this is what I meant.
Cheers
Boaz
On Thu, Nov 11, 2010 at 11:59:28AM +0200, Boaz Harrosh wrote:
> 4. Exactly Like 3 but without the extra kref member
> Only x_put() changes and x_kref_release() now receives
> an x_object
>
> int x_put(struct object_x *x)
> {
> if (kobject_put(&x->kobj) == 1)
> // Like above [3] x_kref_release()
> x_kref_release(x);
> }
This is racy, please never do this.
thanks,
greg k-h
On 11/11/2010 02:04 PM, Greg KH wrote:
> On Thu, Nov 11, 2010 at 11:59:28AM +0200, Boaz Harrosh wrote:
>> 4. Exactly Like 3 but without the extra kref member
>> Only x_put() changes and x_kref_release() now receives
>> an x_object
>>
>> int x_put(struct object_x *x)
>> {
>> if (kobject_put(&x->kobj) == 1)
>> // Like above [3] x_kref_release()
>> x_kref_release(x);
>> }
>
> This is racy, please never do this.
>
The last ref belongs to the core code. 1 means there are no
more external clients on the object. So it can not race with
decrements. But I guess there is a possibility that it can
race with new increments. If it is the case that new increments
can only come from, say, sysfs access, then if we call the
x_put() == 1 after we are unregistered from sysfs and no new
users are allowed then the counter can only go down and we
have the last reference. No?
Like I said option 4 is delicate it must be done carefully.
> thanks,
>
> greg k-h
Thanks
Boaz
On Thu, Nov 11, 2010 at 04:05:43PM +0200, Boaz Harrosh wrote:
> On 11/11/2010 02:04 PM, Greg KH wrote:
> > On Thu, Nov 11, 2010 at 11:59:28AM +0200, Boaz Harrosh wrote:
> >> 4. Exactly Like 3 but without the extra kref member
> >> Only x_put() changes and x_kref_release() now receives
> >> an x_object
> >>
> >> int x_put(struct object_x *x)
> >> {
> >> if (kobject_put(&x->kobj) == 1)
> >> // Like above [3] x_kref_release()
> >> x_kref_release(x);
> >> }
> >
> > This is racy, please never do this.
> >
>
> The last ref belongs to the core code. 1 means there are no
> more external clients on the object. So it can not race with
> decrements. But I guess there is a possibility that it can
> race with new increments.
Exactly.
> If it is the case that new increments
> can only come from, say, sysfs access, then if we call the
> x_put() == 1 after we are unregistered from sysfs and no new
> users are allowed then the counter can only go down and we
> have the last reference. No?
Just don't do this, it's not worth it and will break over time when
others mess with the code.
Also note that kobject_put() does not even return a value, so the code
above will not even compile, let alone work.
thanks,
greg k-h
On 11/11/2010 04:16 PM, Greg KH wrote:
> On Thu, Nov 11, 2010 at 04:05:43PM +0200, Boaz Harrosh wrote:
>> On 11/11/2010 02:04 PM, Greg KH wrote:
>>> On Thu, Nov 11, 2010 at 11:59:28AM +0200, Boaz Harrosh wrote:
>>>> 4. Exactly Like 3 but without the extra kref member
>>>> Only x_put() changes and x_kref_release() now receives
>>>> an x_object
>>>>
>>>> int x_put(struct object_x *x)
>>>> {
>>>> if (kobject_put(&x->kobj) == 1)
>>>> // Like above [3] x_kref_release()
>>>> x_kref_release(x);
>>>> }
>>>
>>> This is racy, please never do this.
>>>
>>
>> The last ref belongs to the core code. 1 means there are no
>> more external clients on the object. So it can not race with
>> decrements. But I guess there is a possibility that it can
>> race with new increments.
>
> Exactly.
>
>> If it is the case that new increments
>> can only come from, say, sysfs access, then if we call the
>> x_put() == 1 after we are unregistered from sysfs and no new
>> users are allowed then the counter can only go down and we
>> have the last reference. No?
>
> Just don't do this, it's not worth it and will break over time when
> others mess with the code.
>
> Also note that kobject_put() does not even return a value, so the code
> above will not even compile, let alone work.
>
OK Point taken, it is fragile. So there is option [3] then, with the extra
kref. I think I've seen other places with this approach.
> thanks,
>
> greg k-h
Thanks
Boaz
Boaz Harrosh, on 11/11/2010 12:59 PM wrote:
>> static void x_release(struct kobject *kobj)
Yes. Precisely speaking, of its kobj_type.
> This one is put on the kobj.release Right?
>
>> {
>> struct object_x *x;
>> struct completion *c;
>>
>> x = container_of(kobj, struct object_x, kobj);
>> c = x->release_completion;
>> kfree(x);
>> complete_all(c);
>> }
>>
>
> I don't see the unregister of the object_x.kobj, where do
> you do this one in x_release or del_object below?
Which unregister? Put for object_x.kobj is in del_object()
>> void del_object(struct object_x *x)
>> {
>> DECLARE_COMPLETION_ONSTACK(completion);
>>
>> ...
>> x->release_completion = &completion;
>> kobject_put(&x->kobj);
>
> This put might not be the last put on the object, IOs in flight
> and/or open files might have extra reference on the object.
> We release our initial ref, and below wait for all operations
> to complete. (Is there a matter of timeout like files not closing?)
This is the last internal put. All other references are from outsiders.
So, we are waiting for all them to put before we go on.
> One possibility (There are others)
>
> 3. struct object_x {
> ...
> struct kref kref;
> struct kobject kobj;
> struct completion *release_completion;
> };
>
> Every body takes kref_put(&object_x.kref) and kref_get(&object_x.kref)
> I hope you have x_get/x_put, Yes?
>
> static void x_kref_release(struct kref *kref)
> {
> struct object_x *x = container_of(kref, struct object_x, kref);
>
> complete_all(&x->release_completion);
> }
>
> static void x_obj_release(struct kobject *kobj)
> {
> struct object_x *x = container_of(kobj, struct object_x, kobj);
>
> kfree(x);
> }
>
> int x_put(struct object_x *x)
> {
> return kref_put(&x->kref, x_kref_release);
> }
>
> void del_object(struct object_x *x)
> {
> DECLARE_COMPLETION_ONSTACK(completion);
>
> ...
> x->release_completion = &completion;
> x_put(x)
> wait_for_completion(&completion);
> kobject_put(&x->kobj);
> }
>
> Or
>
> 4. Exactly Like 3 but without the extra kref member
> Only x_put() changes and x_kref_release() now receives
> an x_object
>
> int x_put(struct object_x *x)
> {
> if (kobject_put(&x->kobj) == 1)
> // Like above [3] x_kref_release()
> x_kref_release(x);
> }
>
> Note that in 4 you don't actually have a kref member, and that you have
> one extra ref on kobj from the beginning. In del_object above the first
> x_put(x) makes it possible to reach the "1" count and then the final
> kobject_put(&x->kobj); frees the object.
> (You need to be carfull with [4] because it must have a refcount==2 before
> you expose it to any IO or sysfs.)
>
> So this is what I meant.
OK, I see. You know, all non-trivial things can be done in >1 correct
way ;) (Although (4) is not too correct as Greg already wrote)
Vlad
On Thu, Nov 11, 2010 at 11:50:01PM +0300, Vladislav Bolkhovitin wrote:
> Boaz Harrosh, on 11/11/2010 12:59 PM wrote:
> >> static void x_release(struct kobject *kobj)
>
> Yes. Precisely speaking, of its kobj_type.
>
> > This one is put on the kobj.release Right?
> >
> >> {
> >> struct object_x *x;
> >> struct completion *c;
> >>
> >> x = container_of(kobj, struct object_x, kobj);
> >> c = x->release_completion;
> >> kfree(x);
> >> complete_all(c);
> >> }
> >>
> >
> > I don't see the unregister of the object_x.kobj, where do
> > you do this one in x_release or del_object below?
>
> Which unregister? Put for object_x.kobj is in del_object()
>
> >> void del_object(struct object_x *x)
> >> {
> >> DECLARE_COMPLETION_ONSTACK(completion);
> >>
> >> ...
> >> x->release_completion = &completion;
> >> kobject_put(&x->kobj);
> >
> > This put might not be the last put on the object, IOs in flight
> > and/or open files might have extra reference on the object.
> > We release our initial ref, and below wait for all operations
> > to complete. (Is there a matter of timeout like files not closing?)
>
> This is the last internal put. All other references are from outsiders.
> So, we are waiting for all them to put before we go on.
>
The question is why do you need to wait here? I presume it is module
unloading path, but then it is quite bad - you can easily wedge your
subsystem if you make something to take a reference to your kobject
while module is ytying to be unloaded. Back when sysfs attributes tied
kobjects the easiest thing was to do:
rmmod <module> < / sys/devices/..../attribute
If you are done with the kobject - just proceed with what you were doing
and let it die its own peaceful death some time later. You just need to
make sure release code sticks around to free it and your subsystem core
can be tasked with this. Use module counter to prevent unloading of the
subsystem core until all kobjects belonging to the subsystem are
destroyed.
--
Dmitry
On Fri, Nov 12, 2010 at 2:23 AM, Dmitry Torokhov
<[email protected]> wrote:
> On Thu, Nov 11, 2010 at 11:50:01PM +0300, Vladislav Bolkhovitin wrote:
> > [ ... ]
> >
> > This is the last internal put. All other references are from outsiders.
> > So, we are waiting for all them to put before we go on.
>
> The question is why do you need to wait here? I presume it is module
> unloading path, but then it is quite bad - you can easily wedge your
> subsystem if you make something to take a reference to your kobject
> while module is trying to be unloaded. Back when sysfs attributes tied
> kobjects the easiest thing was to do:
>
> ? ? ? ?rmmod <module> < / sys/devices/..../attribute
>
> If you are done with the kobject - just proceed with what you were doing
> and let it die its own peaceful death some time later. You just need to
> make sure release code sticks around to free it and your subsystem core
> can be tasked with this. Use module counter to prevent unloading of the
> subsystem core until all kobjects belonging to the subsystem are
> destroyed.
Do you mean keeping a kref object in the kernel module, invoking
kref_get() every time a kobject has been created and invoking
kref_put() from the kobject/ktype release method ? That would help to
reduce the race window but would not eliminate all races: as soon as
the last kref_put() has been invoked from the release method, the
module can get unloaded. And module unloading involves freeing all
module code sections, including the section that contains the
implementation of the release method. Which is a race condition.
I'm not sure that it is even possible with the current kobject
implementation to solve this race. I haven't found any information
about this race in Documentation/kobject.txt. And it seems to me that
the code in samples/kobject/kobject-example.c is vulnerable to this
race: methods like foo_show() and foo_store() can access statically
allocated memory ("static int foo") after the module has been
unloaded. Although the race window is small, this makes me wonder
whether module unloading been overlooked at the time the kobject
subsystem has been designed and implemented ?
Bart.
On Fri, Nov 12, 2010 at 01:09:48PM +0100, Bart Van Assche wrote:
> On Fri, Nov 12, 2010 at 2:23 AM, Dmitry Torokhov
> <[email protected]> wrote:
> > On Thu, Nov 11, 2010 at 11:50:01PM +0300, Vladislav Bolkhovitin wrote:
> > > [ ... ]
> > >
> > > This is the last internal put. All other references are from outsiders.
> > > So, we are waiting for all them to put before we go on.
> >
> > The question is why do you need to wait here? I presume it is module
> > unloading path, but then it is quite bad - you can easily wedge your
> > subsystem if you make something to take a reference to your kobject
> > while module is trying to be unloaded. Back when sysfs attributes tied
> > kobjects the easiest thing was to do:
> >
> > ? ? ? ?rmmod <module> < / sys/devices/..../attribute
> >
> > If you are done with the kobject - just proceed with what you were doing
> > and let it die its own peaceful death some time later. You just need to
> > make sure release code sticks around to free it and your subsystem core
> > can be tasked with this. Use module counter to prevent unloading of the
> > subsystem core until all kobjects belonging to the subsystem are
> > destroyed.
>
> Do you mean keeping a kref object in the kernel module, invoking
> kref_get() every time a kobject has been created and invoking
> kref_put() from the kobject/ktype release method ? That would help to
> reduce the race window but would not eliminate all races: as soon as
> the last kref_put() has been invoked from the release method, the
> module can get unloaded. And module unloading involves freeing all
> module code sections, including the section that contains the
> implementation of the release method. Which is a race condition.
No, you do not add a kref, but rather manipulate module use counter:
static void blah_blah_release(struct kobject *kobj)
{
struct blah_blah *b = to_blah_blah(kobj);
...
kfree(kobj);
module_put(THIS_MODULE);
}
int blah_blah_register(struct blah_blah *blah)
{
...
__module_get(THIS_MODULE);
...
return 0;
}
The above should reside in subsystem _core_ and it will pin the core
module until last kobject belonging to the subsystem is released.
Once all users are gone module counter will go to 0 and rmmod will
allow core to unload. Note that no new kobjects will be created while
module usage count is 0 because there are no users of the core - all of
them have to be unloaded already, otherwise module loader would have
bumped up usage count as well.
>
> I'm not sure that it is even possible with the current kobject
> implementation to solve this race.
It is possible and it is solved in most (all?) mainline subsystems.
> I haven't found any information
> about this race in Documentation/kobject.txt. And it seems to me that
> the code in samples/kobject/kobject-example.c is vulnerable to this
> race: methods like foo_show() and foo_store() can access statically
> allocated memory ("static int foo") after the module has been
> unloaded. Although the race window is small, this makes me wonder
> whether module unloading been overlooked at the time the kobject
> subsystem has been designed and implemented ?
>
> Bart.
--
Dmitry
On Fri, Nov 12, 2010 at 7:44 PM, Dmitry Torokhov
<[email protected]> wrote:
> [ ... ]
>
> No, you do not add a kref, but rather manipulate module use counter:
>
> static void blah_blah_release(struct kobject *kobj)
> {
> ? ? ? ?struct blah_blah *b = to_blah_blah(kobj);
>
> ? ? ? ?...
> ? ? ? ?kfree(kobj);
>
> ? ? ? ?module_put(THIS_MODULE);
> }
>
> int blah_blah_register(struct blah_blah *blah)
> {
> ? ? ? ?...
>
> ? ? ? ?__module_get(THIS_MODULE);
>
> ? ? ? ?...
>
> ? ? ? ?return 0;
> }
>
> The above should reside in subsystem _core_ and it will pin the core
> module until last kobject belonging to the subsystem is released.
> Once all users are gone module counter will go to 0 and rmmod will
> allow core to unload. Note that no new kobjects will be created while
> module usage count is 0 because there are no users of the core - all of
> them have to be unloaded already, otherwise module loader would have
> bumped up usage count as well.
>
> > I'm not sure that it is even possible with the current kobject
> > implementation to solve this race.
>
> It is possible and it is solved in most (all?) mainline subsystems.
Thanks for replying, but sorry, it's still not clear to me. The use
counter of which module should be manipulated ? Manipulating the use
counter of the module that contains the kobject/ktype callback
function implementations would make it impossible to unload that
module because rmmod refuses to unload any module whose use counter is
above zero. And manipulating the use counter of a parent module would
not help in any way.
It would help if you could tell us where in the kernel tree we can
find a good example. I have tried to find such an example, but all I
found are examples of kobject release method implementations in which
no attempt is made to prevent the aforementioned race:
* iscsi_boot_kobj_release() in drivers/scsi/iscsi_boot_sysfs.c
* pkt_kobj_release() in drivers/block/pktcdvd.c
Bart.
Dmitry Torokhov, on 11/12/2010 04:23 AM wrote:
>>> This put might not be the last put on the object, IOs in flight
>>> and/or open files might have extra reference on the object.
>>> We release our initial ref, and below wait for all operations
>>> to complete. (Is there a matter of timeout like files not closing?)
>>
>> This is the last internal put. All other references are from outsiders.
>> So, we are waiting for all them to put before we go on.
>>
>
> The question is why do you need to wait here? I presume it is module
> unloading path, but then it is quite bad - you can easily wedge your
> subsystem if you make something to take a reference to your kobject
> while module is ytying to be unloaded. Back when sysfs attributes tied
> kobjects the easiest thing was to do:
>
> rmmod <module> < / sys/devices/..../attribute
>
> If you are done with the kobject - just proceed with what you were doing
> and let it die its own peaceful death some time later. You just need to
> make sure release code sticks around to free it and your subsystem core
> can be tasked with this. Use module counter to prevent unloading of the
> subsystem core until all kobjects belonging to the subsystem are
> destroyed.
This is a very good question. During implementation I spent a lot of
time working on it.
In fact, the first implementation was asynchronous similarly as you
proposing, i.e. it just proceed with what you were doing and let it die
its own peaceful death some time later. But soon the implementation was
becoming so complicated, so it started getting out of control. For
instance, some of the tasks to solve with this approach were:
1. What to do if another SCST object is being created with the same name
as supposed to be deleted, but not completely dead yet?
2. What to do if a dieing object is found on some list and reference for
is supposed to be taken? If the object deleted from the list before it
marked dieing, i.e. the latest internal put() done, it made additional
problems during deleting it after the latest external put done.
...
So, I decided to reimplement it to be completely synchronous. SYSFS
authors did really great job and thanks to the excellent internal SYSFS
design and implementation it is absolutely safe. See:
[root@tgt ~]# modprobe scst
[root@tgt ~]# cd /sys/kernel/scst_tgt/
[root@tgt scst_tgt]# ls -l
total 0
drwxr-xr-x 4 root root 0 Nov 13 21:31 devices
drwxr-xr-x 2 root root 0 Nov 13 21:31 handlers
-r--r--r-- 1 root root 4096 Nov 13 21:30 last_sysfs_mgmt_res
-rw-r--r-- 1 root root 4096 Nov 13 21:30 setup_id
drwxr-xr-x 5 root root 0 Nov 13 21:31 sgv
drwxr-xr-x 2 root root 0 Nov 13 21:31 targets
-rw-r--r-- 1 root root 4096 Nov 13 21:30 threads
-rw-r--r-- 1 root root 4096 Nov 13 21:30 trace_level
-r--r--r-- 1 root root 4096 Nov 13 21:30 version
[root@tgt scst_tgt]# cat version
2.1.0-pre1
EXTRACHECKS
DEBUG
[root@tgt scst_tgt]# rmmod scst </sys/kernel/scst_tgt/version
[root@tgt scst_tgt]# ls -l
total 0
[root@tgt scst_tgt]# pwd
/sys/kernel/scst_tgt
[root@tgt scst_tgt]# lsmod
Module Size Used by
scsi_debug 65188 0
w83627hf 22424 0
hwmon_vid 2207 1 w83627hf
adm1021 6189 0
binfmt_misc 6229 1
xfs 673142 1
exportfs 3143 1 xfs
dm_mirror 12069 0
dm_region_hash 8703 1 dm_mirror
dm_log 8345 2 dm_mirror,dm_region_hash
dm_mod 63511 2 dm_mirror,dm_log
pci_slot 3378 0
hed 1758 0
floppy 52718 0
uhci_hcd 21459 0
sg 25181 0
e1000 128475 0
i2c_i801 8756 0
pcspkr 1442 0
i2c_core 22319 2 adm1021,i2c_i801
e7xxx_edac 3463 0
parport_pc 25439 0
parport 29682 1 parport_pc
Everything works fine.
This is because SYSFS doesn't hold references for the corresponding
kobjects for every open file handle. It holds references only when
show() and store() functions called. So, everything is under control and
a malicious user can do nothing to hold a reference forever.
Vlad
On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
> So, I decided to reimplement it to be completely synchronous. SYSFS
> authors did really great job and thanks to the excellent internal SYSFS
> design and implementation it is absolutely safe. See:
>
> [root@tgt ~]# modprobe scst
> [root@tgt ~]# cd /sys/kernel/scst_tgt/
Sorry, but no, you can't put this in /sys/kernel/ without getting the
approval of the sysfs maintainer.
I really don't understand why you are using kobjects in the first place,
why isn't this in the main device tree in the kernel, using 'struct
device'?
In the end, I guess it really doesn't matter as this code isn't getting
merged so I shouldn't worry about it, right?
thanks,
greg k-h
On Sat, Nov 13, 2010 at 03:59:38PM -0800, Greg KH wrote:
> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
> > So, I decided to reimplement it to be completely synchronous. SYSFS
> > authors did really great job and thanks to the excellent internal SYSFS
> > design and implementation it is absolutely safe. See:
> >
> > [root@tgt ~]# modprobe scst
> > [root@tgt ~]# cd /sys/kernel/scst_tgt/
>
> Sorry, but no, you can't put this in /sys/kernel/ without getting the
> approval of the sysfs maintainer.
>
> I really don't understand why you are using kobjects in the first place,
> why isn't this in the main device tree in the kernel, using 'struct
> device'?
>
It is my understanding that Vlad is able to reflect the topology by
manipulating sysfs objects there.
> In the end, I guess it really doesn't matter as this code isn't getting
> merged so I shouldn't worry about it, right?
>
This is quite unfortunate as I still have not seen the public comparison
of the 2 implementations and the lists of benefits and shortfalls for
both of them.
--
Dmitry
On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
> Dmitry Torokhov, on 11/12/2010 04:23 AM wrote:
> >>> This put might not be the last put on the object, IOs in flight
> >>> and/or open files might have extra reference on the object.
> >>> We release our initial ref, and below wait for all operations
> >>> to complete. (Is there a matter of timeout like files not closing?)
> >>
> >> This is the last internal put. All other references are from outsiders.
> >> So, we are waiting for all them to put before we go on.
> >>
> >
> > The question is why do you need to wait here? I presume it is module
> > unloading path, but then it is quite bad - you can easily wedge your
> > subsystem if you make something to take a reference to your kobject
> > while module is ytying to be unloaded. Back when sysfs attributes tied
> > kobjects the easiest thing was to do:
> >
> > rmmod <module> < / sys/devices/..../attribute
> >
> > If you are done with the kobject - just proceed with what you were doing
> > and let it die its own peaceful death some time later. You just need to
> > make sure release code sticks around to free it and your subsystem core
> > can be tasked with this. Use module counter to prevent unloading of the
> > subsystem core until all kobjects belonging to the subsystem are
> > destroyed.
>
> This is a very good question. During implementation I spent a lot of
> time working on it.
>
> In fact, the first implementation was asynchronous similarly as you
> proposing, i.e. it just proceed with what you were doing and let it die
> its own peaceful death some time later. But soon the implementation was
> becoming so complicated, so it started getting out of control. For
> instance, some of the tasks to solve with this approach were:
>
> 1. What to do if another SCST object is being created with the same name
> as supposed to be deleted, but not completely dead yet?
The same rules as with files - the object disappears from the
"directories" so no new users can get it but is not destroyed till last
reference is gone.
>
> 2. What to do if a dieing object is found on some list and reference for
> is supposed to be taken? If the object deleted from the list before it
> marked dieing, i.e. the latest internal put() done, it made additional
> problems during deleting it after the latest external put done.
You delete the object from the list, then mark it as dead, notify users,
drop refcount. No new users will get it (as it is not on the list
anymore) and existing ones should notice that it is dead and stop using
it.
>
> ...
>
> So, I decided to reimplement it to be completely synchronous. SYSFS
> authors did really great job and thanks to the excellent internal SYSFS
> design and implementation it is absolutely safe. See:
>
> [root@tgt ~]# modprobe scst
> [root@tgt ~]# cd /sys/kernel/scst_tgt/
> [root@tgt scst_tgt]# ls -l
> total 0
> drwxr-xr-x 4 root root 0 Nov 13 21:31 devices
> drwxr-xr-x 2 root root 0 Nov 13 21:31 handlers
> -r--r--r-- 1 root root 4096 Nov 13 21:30 last_sysfs_mgmt_res
> -rw-r--r-- 1 root root 4096 Nov 13 21:30 setup_id
> drwxr-xr-x 5 root root 0 Nov 13 21:31 sgv
> drwxr-xr-x 2 root root 0 Nov 13 21:31 targets
> -rw-r--r-- 1 root root 4096 Nov 13 21:30 threads
> -rw-r--r-- 1 root root 4096 Nov 13 21:30 trace_level
> -r--r--r-- 1 root root 4096 Nov 13 21:30 version
> [root@tgt scst_tgt]# cat version
> 2.1.0-pre1
> EXTRACHECKS
> DEBUG
> [root@tgt scst_tgt]# rmmod scst </sys/kernel/scst_tgt/version
> [root@tgt scst_tgt]# ls -l
> total 0
> [root@tgt scst_tgt]# pwd
> /sys/kernel/scst_tgt
> [root@tgt scst_tgt]# lsmod
> Module Size Used by
> scsi_debug 65188 0
> w83627hf 22424 0
> hwmon_vid 2207 1 w83627hf
> adm1021 6189 0
> binfmt_misc 6229 1
> xfs 673142 1
> exportfs 3143 1 xfs
> dm_mirror 12069 0
> dm_region_hash 8703 1 dm_mirror
> dm_log 8345 2 dm_mirror,dm_region_hash
> dm_mod 63511 2 dm_mirror,dm_log
> pci_slot 3378 0
> hed 1758 0
> floppy 52718 0
> uhci_hcd 21459 0
> sg 25181 0
> e1000 128475 0
> i2c_i801 8756 0
> pcspkr 1442 0
> i2c_core 22319 2 adm1021,i2c_i801
> e7xxx_edac 3463 0
> parport_pc 25439 0
> parport 29682 1 parport_pc
>
> Everything works fine.
>
> This is because SYSFS doesn't hold references for the corresponding
> kobjects for every open file handle. It holds references only when
> show() and store() functions called. So, everything is under control and
> a malicious user can do nothing to hold a reference forever.
Right, Tejun plugged this particular (and very annoying) attributes
behavior, but that does not mean that this is the only way kobject's
reference might be pinned.
--
Dmitry
On 11/14/2010 01:59 AM, Greg KH wrote:
> In the end, I guess it really doesn't matter as this code isn't getting
> merged so I shouldn't worry about it, right?
>
This is not nice and is uncharacteristic of you.
This project, even though out-of-tree, is an old and mature project that
has many users. These are all *Linux* users. The authors and community
have come to us for help, and advice on making this code acceptable for
mainline and hardening the code the way, only one project on the planet
can do, the Linux community. I think it is our courtesy and obligation
to the Linux users of this Project to comment where they are doing wrong
and where they should do better.
It is not of their choice to be out-of-tree. It is ours. The least we can
do. Is give then some assistance if we can, and have 5 minutes of our time.
All these issues we were discussing are interesting and are real Kernel
problems. For instance the last comment you made was that for such a dynamic
system and life time problems, and functionality. A better and expected
solution might be the device tree and not sysfs. And for such big additions
the sysfs maintainer must give his blessings. This is most valuable information
regardless of if we accept their code or not at the end.
(And we better explain ourselves well when we don't)
> thanks,
>
> greg k-h
Sincerely yours
Boaz
On Mon, Nov 15, 2010 at 11:46:38AM +0200, Boaz Harrosh wrote:
> On 11/14/2010 01:59 AM, Greg KH wrote:
> > In the end, I guess it really doesn't matter as this code isn't getting
> > merged so I shouldn't worry about it, right?
> >
>
> This is not nice and is uncharacteristic of you.
Why, I'm not allowed to get frustrated at repeated attempts to get the
original poster to change their code to something that is acceptable and
just give up and walk away?
Why not?
> This project, even though out-of-tree, is an old and mature project that
> has many users. These are all *Linux* users. The authors and community
> have come to us for help, and advice on making this code acceptable for
> mainline and hardening the code the way, only one project on the planet
> can do, the Linux community. I think it is our courtesy and obligation
> to the Linux users of this Project to comment where they are doing wrong
> and where they should do better.
It is also the job of the kernel community to say "No, what you are
doing is wrong, please don't do that."
And that's what I'm doing here.
> It is not of their choice to be out-of-tree. It is ours. The least we can
> do. Is give then some assistance if we can, and have 5 minutes of our time.
I have given _way_ more than 5 minutes of my time already.
> All these issues we were discussing are interesting and are real Kernel
> problems. For instance the last comment you made was that for such a dynamic
> system and life time problems, and functionality. A better and expected
> solution might be the device tree and not sysfs.
Yes, that is what I have been saying for a while now.
Again:
This code is using kobjects incorrectly.
This code should not be using kobjects.
this is my last response to this thread now, and I'm sure you can
understand why.
thanks,
greg k-h
On 11/15/2010 06:16 PM, Greg KH wrote:
> On Mon, Nov 15, 2010 at 11:46:38AM +0200, Boaz Harrosh wrote:
>> All these issues we were discussing are interesting and are real Kernel
>> problems. For instance the last comment you made was that for such a dynamic
>> system and life time problems, and functionality. A better and expected
>> solution might be the device tree and not sysfs.
>
> Yes, that is what I have been saying for a while now.
>
> Again:
> This code is using kobjects incorrectly.
> This code should not be using kobjects.
>
> this is my last response to this thread now, and I'm sure you can
> understand why.
>
> thanks,
>
> greg k-h
Thank you Greg for your time and most valuable input.
I'm sorry for not understanding your position. I needed the
clear cut statement:
This code should not be using kobjects. i.e not belong in sysfs
SCST guys. This sounds pretty clear cut to me. Sysfs was not built
in mind for such dynamic systems, and it will cause never ending
conflicts with future maintenance of sysfs vs SCST.
Perhaps consider a new alternative like the device tree as Greg suggested
or maybe finally accept the harsh realities of ConfigFS, and come join us
in the LIO project. SCST is a most valuable project and community which
we would like to join forces with in making Linux the best.
Lets call it Linux-Target and unify all our efforts.
Boaz
On Sun, Nov 14, 2010 at 12:59 AM, Greg KH <[email protected]> wrote:
>
> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
> > So, I decided to reimplement it to be completely synchronous. SYSFS
> > authors did really great job and thanks to the excellent internal SYSFS
> > design and implementation it is absolutely safe. See:
> >
> > [root@tgt ~]# modprobe scst
> > [root@tgt ~]# cd /sys/kernel/scst_tgt/
>
> Sorry, but no, you can't put this in /sys/kernel/ without getting the
> approval of the sysfs maintainer.
>
> I really don't understand why you are using kobjects in the first place,
> why isn't this in the main device tree in the kernel, using 'struct
> device'?
We might have missed something, but as far as we know it has not yet
been explained in this thread why using 'struct device' would be an
advantage over using 'struct kobject'. All I can see are the
disadvantages of such a transition: instead of having a single
hierarchy that represents all SCST-related information, there would be
multiple, and the hierarchical relationship between objects would be
lost. Also, during startup, once all SCST-related kernel modules have
been loaded, configuration happens by writing values to individual
sysfs variables. There is a user-space tool included with SCST that
not only can restore a configuration from a file but also can save an
existing configuration to file. I'm afraid that saving an existing
configuration would be made considerably more difficult by
transforming the single SCST sysfs-tree into multiple. Below you can
find an example of a sysfs-tree created by SCST:
# (cd /sys/kernel/scst_tgt && find | cut -c3-)
threads
setup_id
trace_level
version
last_sysfs_mgmt_res
targets
targets/ib_srpt
targets/ib_srpt/ib_srpt_target_0
targets/ib_srpt/ib_srpt_target_0/enabled
targets/ib_srpt/ib_srpt_target_0/sessions
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/initiator_name
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/req_lim
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/req_lim_delta
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/luns
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun0
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun0/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun1
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun1/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun2
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun2/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun255
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun255/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun3
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun3/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun4
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun4/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun5
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun5/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun6
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun6/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun7
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun7/active_commands
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun8
targets/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun8/active_commands
targets/ib_srpt/ib_srpt_target_0/luns
targets/ib_srpt/ib_srpt_target_0/luns/mgmt
targets/ib_srpt/ib_srpt_target_0/luns/0
targets/ib_srpt/ib_srpt_target_0/luns/0/read_only
targets/ib_srpt/ib_srpt_target_0/luns/0/device
targets/ib_srpt/ib_srpt_target_0/luns/1
targets/ib_srpt/ib_srpt_target_0/luns/1/read_only
targets/ib_srpt/ib_srpt_target_0/luns/1/device
targets/ib_srpt/ib_srpt_target_0/luns/2
targets/ib_srpt/ib_srpt_target_0/luns/2/read_only
targets/ib_srpt/ib_srpt_target_0/luns/2/device
targets/ib_srpt/ib_srpt_target_0/luns/255
targets/ib_srpt/ib_srpt_target_0/luns/255/read_only
targets/ib_srpt/ib_srpt_target_0/luns/255/device
targets/ib_srpt/ib_srpt_target_0/luns/3
targets/ib_srpt/ib_srpt_target_0/luns/3/read_only
targets/ib_srpt/ib_srpt_target_0/luns/3/device
targets/ib_srpt/ib_srpt_target_0/luns/4
targets/ib_srpt/ib_srpt_target_0/luns/4/read_only
targets/ib_srpt/ib_srpt_target_0/luns/4/device
targets/ib_srpt/ib_srpt_target_0/luns/5
targets/ib_srpt/ib_srpt_target_0/luns/5/read_only
targets/ib_srpt/ib_srpt_target_0/luns/5/device
targets/ib_srpt/ib_srpt_target_0/luns/6
targets/ib_srpt/ib_srpt_target_0/luns/6/read_only
targets/ib_srpt/ib_srpt_target_0/luns/6/device
targets/ib_srpt/ib_srpt_target_0/luns/7
targets/ib_srpt/ib_srpt_target_0/luns/7/read_only
targets/ib_srpt/ib_srpt_target_0/luns/7/device
targets/ib_srpt/ib_srpt_target_0/luns/8
targets/ib_srpt/ib_srpt_target_0/luns/8/read_only
targets/ib_srpt/ib_srpt_target_0/luns/8/device
targets/ib_srpt/ib_srpt_target_0/ini_groups
targets/ib_srpt/ib_srpt_target_0/ini_groups/mgmt
targets/ib_srpt/ib_srpt_target_0/rel_tgt_id
targets/ib_srpt/ib_srpt_target_0/addr_method
targets/ib_srpt/ib_srpt_target_0/io_grouping_type
targets/ib_srpt/ib_srpt_target_0/cpu_mask
targets/ib_srpt/ib_srpt_target_1
targets/ib_srpt/ib_srpt_target_1/enabled
targets/ib_srpt/ib_srpt_target_1/sessions
targets/ib_srpt/ib_srpt_target_1/luns
targets/ib_srpt/ib_srpt_target_1/luns/mgmt
targets/ib_srpt/ib_srpt_target_1/luns/0
targets/ib_srpt/ib_srpt_target_1/luns/0/read_only
targets/ib_srpt/ib_srpt_target_1/luns/0/device
targets/ib_srpt/ib_srpt_target_1/luns/1
targets/ib_srpt/ib_srpt_target_1/luns/1/read_only
targets/ib_srpt/ib_srpt_target_1/luns/1/device
targets/ib_srpt/ib_srpt_target_1/luns/2
targets/ib_srpt/ib_srpt_target_1/luns/2/read_only
targets/ib_srpt/ib_srpt_target_1/luns/2/device
targets/ib_srpt/ib_srpt_target_1/luns/255
targets/ib_srpt/ib_srpt_target_1/luns/255/read_only
targets/ib_srpt/ib_srpt_target_1/luns/255/device
targets/ib_srpt/ib_srpt_target_1/luns/3
targets/ib_srpt/ib_srpt_target_1/luns/3/read_only
targets/ib_srpt/ib_srpt_target_1/luns/3/device
targets/ib_srpt/ib_srpt_target_1/luns/4
targets/ib_srpt/ib_srpt_target_1/luns/4/read_only
targets/ib_srpt/ib_srpt_target_1/luns/4/device
targets/ib_srpt/ib_srpt_target_1/luns/5
targets/ib_srpt/ib_srpt_target_1/luns/5/read_only
targets/ib_srpt/ib_srpt_target_1/luns/5/device
targets/ib_srpt/ib_srpt_target_1/luns/6
targets/ib_srpt/ib_srpt_target_1/luns/6/read_only
targets/ib_srpt/ib_srpt_target_1/luns/6/device
targets/ib_srpt/ib_srpt_target_1/luns/7
targets/ib_srpt/ib_srpt_target_1/luns/7/read_only
targets/ib_srpt/ib_srpt_target_1/luns/7/device
targets/ib_srpt/ib_srpt_target_1/luns/8
targets/ib_srpt/ib_srpt_target_1/luns/8/read_only
targets/ib_srpt/ib_srpt_target_1/luns/8/device
targets/ib_srpt/ib_srpt_target_1/ini_groups
targets/ib_srpt/ib_srpt_target_1/ini_groups/mgmt
targets/ib_srpt/ib_srpt_target_1/rel_tgt_id
targets/ib_srpt/ib_srpt_target_1/addr_method
targets/ib_srpt/ib_srpt_target_1/io_grouping_type
targets/ib_srpt/ib_srpt_target_1/cpu_mask
targets/iscsi
targets/iscsi/mgmt
targets/iscsi/version
targets/iscsi/open_state
targets/iscsi/trace_level
targets/iscsi/iSNSServer
targets/iscsi/enabled
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/enabled
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/0
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/0/read_only
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/0/device
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/mgmt
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/ini_groups
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/ini_groups/mgmt
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/rel_tgt_id
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/addr_method
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/io_grouping_type
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/cpu_mask
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/tid
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/per_portal_acl
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/redirect
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/InitialR2T
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/ImmediateData
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxRecvDataSegmentLength
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxXmitDataSegmentLength
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxBurstLength
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/FirstBurstLength
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxOutstandingR2T
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/HeaderDigest
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/DataDigest
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/QueuedCommands
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/RspTimeout
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/NopInInterval
targets/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxSessions
targets/scst_local
targets/scst_local/mgmt
targets/scst_local/version
targets/scst_local/stats
targets/scst_local/trace_level
targets/scst_local/scst_local_tgt
targets/scst_local/scst_local_tgt/sessions
targets/scst_local/scst_local_tgt/sessions/scst_local_host
targets/scst_local/scst_local_tgt/sessions/scst_local_host/commands
targets/scst_local/scst_local_tgt/sessions/scst_local_host/active_commands
targets/scst_local/scst_local_tgt/sessions/scst_local_host/initiator_name
targets/scst_local/scst_local_tgt/sessions/scst_local_host/transport_id
targets/scst_local/scst_local_tgt/sessions/scst_local_host/luns
targets/scst_local/scst_local_tgt/sessions/scst_local_host/host
targets/scst_local/scst_local_tgt/luns
targets/scst_local/scst_local_tgt/luns/mgmt
targets/scst_local/scst_local_tgt/ini_groups
targets/scst_local/scst_local_tgt/ini_groups/mgmt
targets/scst_local/scst_local_tgt/rel_tgt_id
targets/scst_local/scst_local_tgt/addr_method
targets/scst_local/scst_local_tgt/io_grouping_type
targets/scst_local/scst_local_tgt/cpu_mask
targets/scst_local/scst_local_tgt/scsi_transport_version
targets/scst_local/scst_local_tgt/phys_transport_version
devices
devices/2:0:0:0
devices/2:0:0:0/type
devices/2:0:0:0/exported
devices/2:0:0:0/scsi_device
devices/2:0:1:0
devices/2:0:1:0/type
devices/2:0:1:0/exported
devices/2:0:1:0/scsi_device
devices/3:0:0:0
devices/3:0:0:0/type
devices/3:0:0:0/exported
devices/3:0:0:0/scsi_device
devices/disk01
devices/disk01/type
devices/disk01/exported
devices/disk01/exported/export0
devices/disk01/exported/export1
devices/disk01/exported/export2
devices/disk01/dump_prs
devices/disk01/handler
devices/disk01/threads_num
devices/disk01/threads_pool_type
devices/disk01/size_mb
devices/disk01/blocksize
devices/disk01/read_only
devices/disk01/write_through
devices/disk01/thin_provisioned
devices/disk01/nv_cache
devices/disk01/o_direct
devices/disk01/removable
devices/disk01/filename
devices/disk01/resync_size
devices/disk01/t10_dev_id
devices/disk01/usn
devices/disk02
devices/disk02/type
devices/disk02/exported
devices/disk02/exported/export0
devices/disk02/exported/export1
devices/disk02/dump_prs
devices/disk02/handler
devices/disk02/threads_num
devices/disk02/threads_pool_type
devices/disk02/size_mb
devices/disk02/blocksize
devices/disk02/read_only
devices/disk02/write_through
devices/disk02/thin_provisioned
devices/disk02/nv_cache
devices/disk02/o_direct
devices/disk02/removable
devices/disk02/filename
devices/disk02/resync_size
devices/disk02/t10_dev_id
devices/disk02/usn
devices/disk03
devices/disk03/type
devices/disk03/exported
devices/disk03/exported/export0
devices/disk03/exported/export1
devices/disk03/dump_prs
devices/disk03/handler
devices/disk03/threads_num
devices/disk03/threads_pool_type
devices/disk03/size_mb
devices/disk03/blocksize
devices/disk03/read_only
devices/disk03/write_through
devices/disk03/thin_provisioned
devices/disk03/nv_cache
devices/disk03/o_direct
devices/disk03/removable
devices/disk03/filename
devices/disk03/resync_size
devices/disk03/t10_dev_id
devices/disk03/usn
devices/disk04
devices/disk04/type
devices/disk04/exported
devices/disk04/exported/export0
devices/disk04/exported/export1
devices/disk04/dump_prs
devices/disk04/handler
devices/disk04/threads_num
devices/disk04/threads_pool_type
devices/disk04/size_mb
devices/disk04/blocksize
devices/disk04/read_only
devices/disk04/write_through
devices/disk04/thin_provisioned
devices/disk04/nv_cache
devices/disk04/o_direct
devices/disk04/removable
devices/disk04/filename
devices/disk04/resync_size
devices/disk04/t10_dev_id
devices/disk04/usn
devices/disk05
devices/disk05/type
devices/disk05/exported
devices/disk05/exported/export0
devices/disk05/exported/export1
devices/disk05/dump_prs
devices/disk05/handler
devices/disk05/threads_num
devices/disk05/threads_pool_type
devices/disk05/size_mb
devices/disk05/blocksize
devices/disk05/read_only
devices/disk05/write_through
devices/disk05/thin_provisioned
devices/disk05/nv_cache
devices/disk05/o_direct
devices/disk05/removable
devices/disk05/filename
devices/disk05/resync_size
devices/disk05/t10_dev_id
devices/disk05/usn
devices/disk06
devices/disk06/type
devices/disk06/exported
devices/disk06/exported/export0
devices/disk06/exported/export1
devices/disk06/dump_prs
devices/disk06/handler
devices/disk06/threads_num
devices/disk06/threads_pool_type
devices/disk06/size_mb
devices/disk06/blocksize
devices/disk06/read_only
devices/disk06/write_through
devices/disk06/thin_provisioned
devices/disk06/nv_cache
devices/disk06/o_direct
devices/disk06/removable
devices/disk06/filename
devices/disk06/resync_size
devices/disk06/t10_dev_id
devices/disk06/usn
devices/disk07
devices/disk07/type
devices/disk07/exported
devices/disk07/exported/export0
devices/disk07/exported/export1
devices/disk07/dump_prs
devices/disk07/handler
devices/disk07/threads_num
devices/disk07/threads_pool_type
devices/disk07/size_mb
devices/disk07/blocksize
devices/disk07/read_only
devices/disk07/write_through
devices/disk07/thin_provisioned
devices/disk07/nv_cache
devices/disk07/o_direct
devices/disk07/removable
devices/disk07/filename
devices/disk07/resync_size
devices/disk07/t10_dev_id
devices/disk07/usn
devices/disk08
devices/disk08/type
devices/disk08/exported
devices/disk08/exported/export0
devices/disk08/exported/export1
devices/disk08/dump_prs
devices/disk08/handler
devices/disk08/threads_num
devices/disk08/threads_pool_type
devices/disk08/size_mb
devices/disk08/blocksize
devices/disk08/read_only
devices/disk08/write_through
devices/disk08/thin_provisioned
devices/disk08/nv_cache
devices/disk08/o_direct
devices/disk08/removable
devices/disk08/filename
devices/disk08/resync_size
devices/disk08/t10_dev_id
devices/disk08/usn
devices/disk09
devices/disk09/type
devices/disk09/exported
devices/disk09/exported/export0
devices/disk09/exported/export1
devices/disk09/dump_prs
devices/disk09/handler
devices/disk09/threads_num
devices/disk09/threads_pool_type
devices/disk09/size_mb
devices/disk09/blocksize
devices/disk09/read_only
devices/disk09/removable
devices/disk09/t10_dev_id
devices/disk09/usn
devices/disk10
devices/disk10/type
devices/disk10/exported
devices/disk10/exported/export0
devices/disk10/exported/export1
devices/disk10/dump_prs
devices/disk10/handler
devices/disk10/threads_num
devices/disk10/threads_pool_type
devices/disk10/size_mb
devices/disk10/blocksize
devices/disk10/read_only
devices/disk10/removable
devices/disk10/t10_dev_id
devices/disk10/usn
sgv
sgv/global_stats
sgv/sgv
sgv/sgv/stats
sgv/sgv-clust
sgv/sgv-clust/stats
sgv/sgv-dma
sgv/sgv-dma/stats
handlers
handlers/vdisk_fileio
handlers/vdisk_fileio/type
handlers/vdisk_fileio/mgmt
handlers/vdisk_fileio/trace_level
handlers/vdisk_fileio/disk01
handlers/vdisk_fileio/disk02
handlers/vdisk_fileio/disk03
handlers/vdisk_fileio/disk04
handlers/vdisk_fileio/disk05
handlers/vdisk_fileio/disk06
handlers/vdisk_fileio/disk07
handlers/vdisk_fileio/disk08
handlers/vdisk_blockio
handlers/vdisk_blockio/type
handlers/vdisk_blockio/mgmt
handlers/vdisk_blockio/trace_level
handlers/vdisk_nullio
handlers/vdisk_nullio/type
handlers/vdisk_nullio/mgmt
handlers/vdisk_nullio/trace_level
handlers/vdisk_nullio/disk09
handlers/vdisk_nullio/disk10
handlers/vcdrom
handlers/vcdrom/type
handlers/vcdrom/mgmt
handlers/vcdrom/trace_level
Bart.
On Mon, Nov 15, 2010 at 6:19 PM, Boaz Harrosh <[email protected]> wrote:
> [ ... ]
> Perhaps consider a new alternative like the device tree as Greg suggested
> or maybe finally accept the harsh realities of ConfigFS.
I think that Vlad has already explained several times why ConfigFS is
not suited for the needs of a storage target: a storage target must
not only be able to accept configuration information from userspace
but must also be able to create new directories and file nodes itself.
See e.g. this message from October 6:
http://kerneltrap.org/mailarchive/linux-kernel/2010/10/6/4628664.
Bart.
On Mon, Nov 15, 2010 at 7:59 AM, Dmitry Torokhov
<[email protected]> wrote:
> [ ... ]
> This is quite unfortunate as I still have not seen the public comparison
> of the 2 implementations and the lists of benefits and shortfalls for
> both of them.
The reason why we are proposing SCST for upstream inclusion are:
- It has a large user base.
- The project is known for its high performance, low latency I/O and
excellent stability.
- A feature-wise overview can be found here:
http://scst.sourceforge.net/comparison.html.
Bart.
On Mon, Nov 15, 2010 at 06:45:24PM +0100, Bart Van Assche wrote:
> On Sun, Nov 14, 2010 at 12:59 AM, Greg KH <[email protected]> wrote:
> >
> > On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
> > > So, I decided to reimplement it to be completely synchronous. SYSFS
> > > authors did really great job and thanks to the excellent internal SYSFS
> > > design and implementation it is absolutely safe. See:
> > >
> > > [root@tgt ~]# modprobe scst
> > > [root@tgt ~]# cd /sys/kernel/scst_tgt/
> >
> > Sorry, but no, you can't put this in /sys/kernel/ without getting the
> > approval of the sysfs maintainer.
> >
> > I really don't understand why you are using kobjects in the first place,
> > why isn't this in the main device tree in the kernel, using 'struct
> > device'?
>
> We might have missed something, but as far as we know it has not yet
> been explained in this thread why using 'struct device' would be an
> advantage over using 'struct kobject'.
It's very simple.
You want your device to show up in the global device tree in the kernel,
not off to one side, unconnected to anything else.
Please use 'struct device', it is what you want to do here.
good luck,
greg k-h
<Trimming CC list>
On Mon, 2010-11-15 at 18:49 +0100, Bart Van Assche wrote:
> On Mon, Nov 15, 2010 at 6:19 PM, Boaz Harrosh <[email protected]> wrote:
> > [ ... ]
> > Perhaps consider a new alternative like the device tree as Greg suggested
> > or maybe finally accept the harsh realities of ConfigFS.
>
> I think that Vlad has already explained several times why ConfigFS is
> not suited for the needs of a storage target: a storage target must
> not only be able to accept configuration information from userspace
> but must also be able to create new directories and file nodes itself.
> See e.g. this message from October 6:
> http://kerneltrap.org/mailarchive/linux-kernel/2010/10/6/4628664.
>
Sorry, but this post explains nothing but a single misguided and
uninformed opinion, with no hard facts on the actual usage of a native
configfs control plane within target mode infrastructure.
The hard facts are:
Using configfs works to drive the generation/release, real time
configuration, dependancy relationships because:
*) configfs represents the individual target data structures who's
creation/deletion are driven from enterily userspace.
*) The parent/child relationships of dependant data structures is
handled transparently to the configfs consumer (eg: no hard requirement
internal reference counting)
*) The module reference counting of target core -> fabric module is
handled transparently to configfs consumers *and* TCM fabric modules
*) The current implementation of TCM/ConfigFS contains no global locks.
Each /sys/kernel/config/target/$FABRIC_1 operates independently of
/sys/kernel/config/target/$FABRIC_2
*) Expecting target fabric module developers to (by-hand) add struct
config_groups, struct kobjects or anything else directly to their fabric
module code is really clumsy and stupid. This problem was solved
earlier this year in TCM v4.0 with the advent of:
http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/target_core_fabric_configfs.c;hb=refs/heads/lio-4.0
What this code means is that target fabric module developers no longer
has to duplicate complex control path code, and all of the interesting
port attributes (like ALUA secondary access state for example) are
transparent to new fabric modules, and 'just work'. We can even
generate *functional* configfs skeletons for new fabric modules using
tcm_mod_builder.py discussed here:
http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=Documentation/target/tcm_mod_builder.txt;hb=refs/heads/lio-4.0
For the specific case of the target control path, until someone from
SCST can present some hard facts *and* running code about the target
mode configfs vs. sysfs debate, the discussion is like comparing oil and
water.
Again, please do not take this as the LIO maintainer saying 'you should
stop working on SCST code XYZ', because that is *not* what I am saying.
We are simply stating that we already have a fully functional configfs
backend and generic target fabric independent infrastrucutre that has
not only been reviewed no less than five times this year on
linux-scsi/LKML, it has actually been developed *on* linux-scsi, patch
by patch in full view of the configfs maintainer and other interested
parties.
This is the reason why the v4.0 code has progressed as fast as it has; a
constant feedback loop from upstream developers who have been involved
since the start of the development process. By virtue of this fact, we
where able to make some early design decisions based on feedback from
the community from the original proposed design and running prototype
code.
So, while I can very much apperciate Boaz's desire to attempt to
reconcile the relationship between myself + other TCM/LIO developers
with the SCST development community, but I don't think this will happen
without a definate change to the SCST development workflow and
interaction with 'outside of project' developers.
--nab
Greg KH, on 11/14/2010 02:59 AM wrote:
> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
>> So, I decided to reimplement it to be completely synchronous. SYSFS
>> authors did really great job and thanks to the excellent internal SYSFS
>> design and implementation it is absolutely safe. See:
>>
>> [root@tgt ~]# modprobe scst
>> [root@tgt ~]# cd /sys/kernel/scst_tgt/
>
> Sorry, but no, you can't put this in /sys/kernel/ without getting the
> approval of the sysfs maintainer.
Hmm, I thought we had the approvement in the "[RFC]: SCST sysfs layout"
thread (http://osdir.com/ml/linux-kernel/2009-04/msg07822.html). Particularly,
in the message http://osdir.com/ml/linux-kernel/2009-04/msg08557.html.
But, looks like it isn't so and I should have asked you, the SYSFS maintainer.
Sorry.
Could you consider it, please?
You can find detail description why SCST needs such layout below.
> I really don't understand why you are using kobjects in the first place,
> why isn't this in the main device tree in the kernel, using 'struct
> device'?
I'll try to explain. It's a bit long story involving deep SCSI target specific
knowledge, but I'll try to make it as simple and short as possible.
SCST is a SCSI _target_ side subsystem, i.e. it is a _server_ side sybsystem.
It exports SCSI-speaking devices, not using them. You can consider it as an NFS
server. What usually meant by "SCSI subsystem" is SCSI _initiator_ subsystem,
i.e. client side subsystem (like NFS client), which is using SCSI-speaking
devices by sending SCSI commands to them.
Any SCSI-speaking protocol can be used with SCST: parallel (wide) SCSI, Fibre Channel,
iSCSI, SAS, SRP, iSER, etc. (Also, non-SCSI speaking protocols, like AoE and
NBD can be used, but that's another story.)
Strictly as required by SCSI Architecture Model (SAM [1]), SCST doesn't deal with
hardware devices. The closest to hardware things SCST deals with are SCSI target
ports and SCSI target devices.
SCSI target port is an abstract concept reflecting path through which SCST
exports devices. You can consider it as an IP network (network, not interface!)
through which an NFS server's exports can be used. For instance, for iSCSI
such ports are iSCSI targets. For Fibre Channel - virtual or hardware Fibre
Channel ports.
SCSI target device is an abstract concept, which provides a way to reach real storage
(files, block devices, etc.) and contains internal state information (reservations,
configuration parameters, etc.). You can consider it as an NFS export. Please don't
confuse it with SCSI _initiator_ device, which is a place to generate SCSI commands
and send them via one or more SCSI initiator ports (MPIO). On the target side
they will be accepted via one or more SCSI target ports, then sent by the
corresponding SCSI target device to back storage device (file, block device, etc.).
So, there is no place in SCST to make Linux devices and use struct device. It's up to
SCST target drivers to create Linux devices for target hardware, if they need it,
which is rare. For instance, scst_local driver make SCST's SCSI target devices be
available as SCSI initiator, i.e. regular, devices by creating all the necessary
devices and SYSFS infrastructure for them:
Load SCST modules:
[root@tgt ~]# modprobe scst
[root@tgt ~]# modprobe scst_vdisk
Create SCSI target device "blockio" with /dev/sda5 as backstorage:
[root@tgt ~]# echo "add_device blockio filename=/dev/sda5" >/sys/kernel/scst_tgt/handlers/vdisk_blockio/mgmt
Check current SCSI devices:
[root@tgt ~]# lsscsi
[1:0:0:0] disk SEAGATE ST373455LW 0003 /dev/sda
[2:0:0:0] disk Linux scsi_debug 0004 /dev/sdb
Check that host3 doesn't exist:
[root@tgt ~]# cd /sys/class/scsi_host/host3
bash: cd: /sys/class/scsi_host/host3: No such file or directory
Load scst_local target driver:
[root@tgt ~]# modprobe scst_local
Create SCSI target port "scst_local_tgt" with SCSI host "scst_local_host" (host3):
[root@tgt ~]# echo "add_target scst_local_tgt session_name=scst_local_host" >/sys/kernel/scst_tgt/targets/scst_local/mgmt
Add "blockio" as LUN 0:
[root@tgt ~]# echo "add blockio 0" >/sys/kernel/scst_tgt/targets/scst_local/scst_local_tgt/luns/mgmt
See new local SCSI device 3:0:0:0 (/dev/sdc):
[root@tgt ~]# lsscsi
[1:0:0:0] disk SEAGATE ST373455LW 0003 /dev/sda
[2:0:0:0] disk Linux scsi_debug 0004 /dev/sdb
[3:0:0:0] disk SCST_BIO blockio 210 /dev/sdc
[root@tgt ~]# cd /sys/class/scsi_host/host3
[root@tgt host3]# ll
total 0
-rw-r--r-- 1 root root 4096 Nov 16 00:22 active_mode
-r--r--r-- 1 root root 4096 Nov 16 00:22 can_queue
-r--r--r-- 1 root root 4096 Nov 16 00:22 cmd_per_lun
lrwxrwxrwx 1 root root 0 Nov 16 00:07 device -> ../../../devices/scst_local/scst_local_host/host3
-r--r--r-- 1 root root 4096 Nov 16 00:22 host_busy
drwxr-xr-x 2 root root 0 Nov 16 00:22 power
-r--r--r-- 1 root root 4096 Nov 16 00:22 proc_name
-r--r--r-- 1 root root 4096 Nov 16 00:22 prot_capabilities
-r--r--r-- 1 root root 4096 Nov 16 00:22 prot_guard_type
--w------- 1 root root 4096 Nov 16 00:22 scan
-r--r--r-- 1 root root 4096 Nov 16 00:22 sg_tablesize
-rw-r--r-- 1 root root 4096 Nov 16 00:22 state
lrwxrwxrwx 1 root root 0 Nov 16 00:22 subsystem -> ../../scsi_host
-rw-r--r-- 1 root root 4096 Nov 16 00:22 supported_mode
-rw-r--r-- 1 root root 4096 Nov 16 00:22 uevent
-r--r--r-- 1 root root 4096 Nov 16 00:22 unchecked_isa_dma
-r--r--r-- 1 root root 4096 Nov 16 00:22 unique_id
Hopefully, it will make a bit clearer, why SCST can't use struct device, but uses
struct kobject, and why it needs a special place in the SYSFS tree to attach to.
Thanks,
Vlad
[1] You can download a copy of SAM from http://www.t10.org/drafts.htm. Exact version doesn't matter.
Dmitry Torokhov, on 11/15/2010 10:04 AM wrote:
>> 1. What to do if another SCST object is being created with the same name
>> as supposed to be deleted, but not completely dead yet?
>
> The same rules as with files - the object disappears from the
> "directories" so no new users can get it but is not destroyed till last
> reference is gone.
>
>>
>> 2. What to do if a dieing object is found on some list and reference for
>> is supposed to be taken? If the object deleted from the list before it
>> marked dieing, i.e. the latest internal put() done, it made additional
>> problems during deleting it after the latest external put done.
>
> You delete the object from the list, then mark it as dead, notify users,
> drop refcount. No new users will get it (as it is not on the list
> anymore) and existing ones should notice that it is dead and stop using
> it.
Those are good in theory, but on practice, you know, devil is in the
details..
>> This is because SYSFS doesn't hold references for the corresponding
>> kobjects for every open file handle. It holds references only when
>> show() and store() functions called. So, everything is under control and
>> a malicious user can do nothing to hold a reference forever.
>
> Right, Tejun plugged this particular (and very annoying) attributes
> behavior
This behavior isn't annoying, it's GREAT, because it allows to use SYSFS
simply and reliably.
>, but that does not mean that this is the only way kobject's
> reference might be pinned.
Could you be more specific and point out on exact ways for that? From my
quite deep SYSFS source code study I see such cases should not exist.
Thanks,
Vlad
Dmitry Torokhov, on 11/15/2010 09:59 AM wrote:
> On Sat, Nov 13, 2010 at 03:59:38PM -0800, Greg KH wrote:
>> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
>>> So, I decided to reimplement it to be completely synchronous. SYSFS
>>> authors did really great job and thanks to the excellent internal SYSFS
>>> design and implementation it is absolutely safe. See:
>>>
>>> [root@tgt ~]# modprobe scst
>>> [root@tgt ~]# cd /sys/kernel/scst_tgt/
>>
>> Sorry, but no, you can't put this in /sys/kernel/ without getting the
>> approval of the sysfs maintainer.
>>
>> I really don't understand why you are using kobjects in the first place,
>> why isn't this in the main device tree in the kernel, using 'struct
>> device'?
>
> It is my understanding that Vlad is able to reflect the topology by
> manipulating sysfs objects there.
Correct. As I wrote in the previous e-mail, SCST doesn't deal with
devices, so doesn't have a need to use struct device.
>> In the end, I guess it really doesn't matter as this code isn't getting
>> merged so I shouldn't worry about it, right?
>>
>
> This is quite unfortunate as I still have not seen the public comparison
> of the 2 implementations and the lists of benefits and shortfalls for
> both of them.
Indeed, it is unfortunate :(. Undercover political games continue...
Vlad
Greg KH, on 11/15/2010 07:16 PM wrote:
> Why, I'm not allowed to get frustrated at repeated attempts to get the
> original poster to change their code to something that is acceptable and
> just give up and walk away?
>
> Why not?
Hmm, frankly, I decided that you agreed with my arguments..
As I wrote, I'm willing to make any changes you requests. I only asked
why this should be done.
I really don't understand why we and other similar in-kernel developers
should treat kobjects in the different way than any other subobjects of
our outer objects and make for them _additional code_ to specially treat
them as life-time center (http://lkml.org/lkml/2010/11/10/421)? You have
not explained it anywhere in any doc I can find.
This is just small "why" question. Greg, don't we have a right to ask
this before go on?
>> This project, even though out-of-tree, is an old and mature project that
>> has many users. These are all *Linux* users. The authors and community
>> have come to us for help, and advice on making this code acceptable for
>> mainline and hardening the code the way, only one project on the planet
>> can do, the Linux community. I think it is our courtesy and obligation
>> to the Linux users of this Project to comment where they are doing wrong
>> and where they should do better.
>
> It is also the job of the kernel community to say "No, what you are
> doing is wrong, please don't do that."
>
> And that's what I'm doing here.
>
>> It is not of their choice to be out-of-tree. It is ours. The least we can
>> do. Is give then some assistance if we can, and have 5 minutes of our time.
>
> I have given _way_ more than 5 minutes of my time already.
We appreciated it very much.
>> All these issues we were discussing are interesting and are real Kernel
>> problems. For instance the last comment you made was that for such a dynamic
>> system and life time problems, and functionality. A better and expected
>> solution might be the device tree and not sysfs.
>
> Yes, that is what I have been saying for a while now.
>
> Again:
> This code is using kobjects incorrectly.
> This code should not be using kobjects.
>
> this is my last response to this thread now, and I'm sure you can
> understand why.
It is REALLY frustrating you are refusing to explain why. I guess, I'm
too stupid to figure out that alone. Don't you want we rise as highly
skilled kernel developers? I believe, not only SCST developers are very
interested to know background behind particular moves in the kernel.
Thanks,
Vlad
Boaz Harrosh, on 11/15/2010 08:19 PM wrote:
> On 11/15/2010 06:16 PM, Greg KH wrote:
>> On Mon, Nov 15, 2010 at 11:46:38AM +0200, Boaz Harrosh wrote:
>>> All these issues we were discussing are interesting and are real Kernel
>>> problems. For instance the last comment you made was that for such a dynamic
>>> system and life time problems, and functionality. A better and expected
>>> solution might be the device tree and not sysfs.
>>
>> Yes, that is what I have been saying for a while now.
>>
>> Again:
>> This code is using kobjects incorrectly.
>> This code should not be using kobjects.
>>
>> this is my last response to this thread now, and I'm sure you can
>> understand why.
>>
>> thanks,
>>
>> greg k-h
>
> Thank you Greg for your time and most valuable input.
> I'm sorry for not understanding your position. I needed the
> clear cut statement:
>
> This code should not be using kobjects. i.e not belong in sysfs
>
> SCST guys. This sounds pretty clear cut to me. Sysfs was not built
> in mind for such dynamic systems, and it will cause never ending
> conflicts with future maintenance of sysfs vs SCST.
As I explained in the previous e-mail, I believe, SYSFS perfectly suits
SCST and SCST perfectly suits SYSFS.
If you think it isn't so, let's discuss each showstopper for that, one
after one.
> Lets call it Linux-Target and unify all our efforts.
Looks like a great idea!
Thanks,
Vlad
Greg KH, on 11/15/2010 09:44 PM wrote:
> On Mon, Nov 15, 2010 at 06:45:24PM +0100, Bart Van Assche wrote:
>> On Sun, Nov 14, 2010 at 12:59 AM, Greg KH <[email protected]> wrote:
>>>
>>> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
>>>> So, I decided to reimplement it to be completely synchronous. SYSFS
>>>> authors did really great job and thanks to the excellent internal SYSFS
>>>> design and implementation it is absolutely safe. See:
>>>>
>>>> [root@tgt ~]# modprobe scst
>>>> [root@tgt ~]# cd /sys/kernel/scst_tgt/
>>>
>>> Sorry, but no, you can't put this in /sys/kernel/ without getting the
>>> approval of the sysfs maintainer.
>>>
>>> I really don't understand why you are using kobjects in the first place,
>>> why isn't this in the main device tree in the kernel, using 'struct
>>> device'?
>>
>> We might have missed something, but as far as we know it has not yet
>> been explained in this thread why using 'struct device' would be an
>> advantage over using 'struct kobject'.
>
> It's very simple.
>
> You want your device to show up in the global device tree in the kernel,
> not off to one side, unconnected to anything else.
>
> Please use 'struct device', it is what you want to do here.
But we don't have any device to show up in the global device tree! We
don't have any devices in the struct device's understanding at all!
Vlad
On Mon, Nov 15, 2010 at 11:37:28PM +0300, Vladislav Bolkhovitin wrote:
> Dmitry Torokhov, on 11/15/2010 10:04 AM wrote:
>
> >> This is because SYSFS doesn't hold references for the corresponding
> >> kobjects for every open file handle. It holds references only when
> >> show() and store() functions called. So, everything is under control and
> >> a malicious user can do nothing to hold a reference forever.
> >
> > Right, Tejun plugged this particular (and very annoying) attributes
> > behavior
>
> This behavior isn't annoying, it's GREAT, because it allows to use SYSFS
> simply and reliably.
Right, I mean that _before_ Tejun plugged that hole the behavior _was_
annoying.
>
> >, but that does not mean that this is the only way kobject's
> > reference might be pinned.
>
> Could you be more specific and point out on exact ways for that? From my
> quite deep SYSFS source code study I see such cases should not exist.
While I do not know offhand I am sure there are such scenarios. Isn't
there any way for the users that you are waiting on descend back into
your module that is waiting for kobject removal and get stuck on some
resource?
Even if it isn't possible now the scheme is quite fragile. Kobjects are
refcounted so work with them appropriately (rely on refcount, do not
wait, etc).
--
Dmitry
On Mon, Nov 15, 2010 at 11:39:48PM +0300, Vladislav Bolkhovitin wrote:
> Greg KH, on 11/15/2010 09:44 PM wrote:
> > On Mon, Nov 15, 2010 at 06:45:24PM +0100, Bart Van Assche wrote:
> >> On Sun, Nov 14, 2010 at 12:59 AM, Greg KH <[email protected]> wrote:
> >>>
> >>> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
> >>>> So, I decided to reimplement it to be completely synchronous. SYSFS
> >>>> authors did really great job and thanks to the excellent internal SYSFS
> >>>> design and implementation it is absolutely safe. See:
> >>>>
> >>>> [root@tgt ~]# modprobe scst
> >>>> [root@tgt ~]# cd /sys/kernel/scst_tgt/
> >>>
> >>> Sorry, but no, you can't put this in /sys/kernel/ without getting the
> >>> approval of the sysfs maintainer.
> >>>
> >>> I really don't understand why you are using kobjects in the first place,
> >>> why isn't this in the main device tree in the kernel, using 'struct
> >>> device'?
> >>
> >> We might have missed something, but as far as we know it has not yet
> >> been explained in this thread why using 'struct device' would be an
> >> advantage over using 'struct kobject'.
> >
> > It's very simple.
> >
> > You want your device to show up in the global device tree in the kernel,
> > not off to one side, unconnected to anything else.
> >
> > Please use 'struct device', it is what you want to do here.
>
> But we don't have any device to show up in the global device tree!
Not true at all.
> We don't have any devices in the struct device's understanding at all!
Then create them just like you are doing so for your kobject use.
The first device would be the root one, and then everything trickles
down from there.
And use configfs for your configuration stuff, that's what it is there
for, and if it doesn't somehow work properly for you, please work with
the configfs developers to fix that up.
thanks,
greg k-h
On 11/15/10 2:13 PM, Greg KH wrote:
> On Mon, Nov 15, 2010 at 11:39:48PM +0300, Vladislav Bolkhovitin wrote:
>> Greg KH, on 11/15/2010 09:44 PM wrote:
>> > On Mon, Nov 15, 2010 at 06:45:24PM +0100, Bart Van Assche wrote:
>> >> On Sun, Nov 14, 2010 at 12:59 AM, Greg KH <[email protected]> wrote:
>> >>>
>> >>> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
>> >>>> So, I decided to reimplement it to be completely synchronous. SYSFS
>> >>>> authors did really great job and thanks to the excellent internal SYSFS
>> >>>> design and implementation it is absolutely safe. See:
>> >>>>
>> >>>> [root@tgt ~]# modprobe scst
>> >>>> [root@tgt ~]# cd /sys/kernel/scst_tgt/
>> >>>
>> >>> Sorry, but no, you can't put this in /sys/kernel/ without getting the
>> >>> approval of the sysfs maintainer.
>> >>>
>> >>> I really don't understand why you are using kobjects in the first place,
>> >>> why isn't this in the main device tree in the kernel, using 'struct
>> >>> device'?
>> >>
>> >> We might have missed something, but as far as we know it has not yet
>> >> been explained in this thread why using 'struct device' would be an
>> >> advantage over using 'struct kobject'.
>> >
>> > It's very simple.
>> >
>> > You want your device to show up in the global device tree in the kernel,
>> > not off to one side, unconnected to anything else.
>> >
>> > Please use 'struct device', it is what you want to do here.
>>
>> But we don't have any device to show up in the global device tree!
>
> Not true at all.
>
>> We don't have any devices in the struct device's understanding at all!
>
> Then create them just like you are doing so for your kobject use.
>
> The first device would be the root one, and then everything trickles
> down from there.
>
> And use configfs for your configuration stuff, that's what it is there
> for, and if it doesn't somehow work properly for you, please work with
> the configfs developers to fix that up.
>
> thanks,
>
> greg k-h
I don't have any opinion on the above, but I don't see why sysfs can't be
used for configuration as well as its other roles. It seems to me wasteful
to require configfs to be used in order to change configuration when
sysfs works fine for this.
Here are a couple of existing examples where sysfs is used in a role that
would seem similar to SCST's usage:
1) scsi_transport_fc already has an sysfs file, fc_host/vport_create, which
can be written to create new fc_host and scsi_host instances.
2) fcoe.ko uses a write to the sysfs file /sys/module/fcoe/parameters/create
to start the protocol on a particular ethernet interface. There's another
file for destroy.
I'll bet there are other examples in other subsystems, and I don't think
there is anything wrong with the above usages of sysfs.
I agree with Vladislav's point that configfs doesn't work instead of sysfs
because configs doesn't make it easy for the kernel side to create nodes for
dynamic information like connected initiators and statistics.
So, if the above examples are considered a misuse of sysfs,
SCST would need to use both sysfs and configfs. It would use configfs to
do configuration, and sysfs to display and access the dynamic information
like connected initiators. That seems a minor role for configfs,
which is easily handled in sysfs as SCST currently does.
Just my two cents.
Joe
On Mon, 2010-11-15 at 21:04 -0800, Joe Eykholt wrote:
>
> On 11/15/10 2:13 PM, Greg KH wrote:
> > On Mon, Nov 15, 2010 at 11:39:48PM +0300, Vladislav Bolkhovitin wrote:
> >> Greg KH, on 11/15/2010 09:44 PM wrote:
> >> > On Mon, Nov 15, 2010 at 06:45:24PM +0100, Bart Van Assche wrote:
> >> >> On Sun, Nov 14, 2010 at 12:59 AM, Greg KH <[email protected]> wrote:
> >> >>>
> >> >>> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
> >> >>>> So, I decided to reimplement it to be completely synchronous. SYSFS
> >> >>>> authors did really great job and thanks to the excellent internal SYSFS
> >> >>>> design and implementation it is absolutely safe. See:
> >> >>>>
> >> >>>> [root@tgt ~]# modprobe scst
> >> >>>> [root@tgt ~]# cd /sys/kernel/scst_tgt/
> >> >>>
> >> >>> Sorry, but no, you can't put this in /sys/kernel/ without getting the
> >> >>> approval of the sysfs maintainer.
> >> >>>
> >> >>> I really don't understand why you are using kobjects in the first place,
> >> >>> why isn't this in the main device tree in the kernel, using 'struct
> >> >>> device'?
> >> >>
> >> >> We might have missed something, but as far as we know it has not yet
> >> >> been explained in this thread why using 'struct device' would be an
> >> >> advantage over using 'struct kobject'.
> >> >
> >> > It's very simple.
> >> >
> >> > You want your device to show up in the global device tree in the kernel,
> >> > not off to one side, unconnected to anything else.
> >> >
> >> > Please use 'struct device', it is what you want to do here.
> >>
> >> But we don't have any device to show up in the global device tree!
> >
> > Not true at all.
> >
> >> We don't have any devices in the struct device's understanding at all!
> >
> > Then create them just like you are doing so for your kobject use.
> >
> > The first device would be the root one, and then everything trickles
> > down from there.
> >
> > And use configfs for your configuration stuff, that's what it is there
> > for, and if it doesn't somehow work properly for you, please work with
> > the configfs developers to fix that up.
> >
> > thanks,
> >
> > greg k-h
>
Greetings Joe,
> I don't have any opinion on the above, but I don't see why sysfs can't be
> used for configuration as well as its other roles. It seems to me wasteful
> to require configfs to be used in order to change configuration when
> sysfs works fine for this.
>
> Here are a couple of existing examples where sysfs is used in a role that
> would seem similar to SCST's usage:
>
> 1) scsi_transport_fc already has an sysfs file, fc_host/vport_create, which
> can be written to create new fc_host and scsi_host instances.
>
> 2) fcoe.ko uses a write to the sysfs file /sys/module/fcoe/parameters/create
> to start the protocol on a particular ethernet interface. There's another
> file for destroy.
>
> I'll bet there are other examples in other subsystems, and I don't think
> there is anything wrong with the above usages of sysfs.
>
> I agree with Vladislav's point that configfs doesn't work instead of sysfs
> because configs doesn't make it easy for the kernel side to create nodes for
> dynamic information like connected initiators and statistics.
I disagree that conversion of 'demo mode' struct se_node_acls to explict
userspace syscall driven configfs registered struct
se_node_acl->acl_group is overly complex, or otherwise difficult in
context of a configfs consumer for a transition demo mode -> explict
NodeACL via:
mkdir -p /sys/kernel/config/target/$FABRIC/$WWN/$TPGT/acls/$INITIATOR_WWPN.
Note the current v4.0 code supports this for LIO-Target and TCM/FC
fabric module code, and will just work 'out of the box' for v4.0
compatible fabric modules using target_core_fabric_configfs.c logic.
As for statistics point, this is also currently handled by TCM in struct
se_node_acl in target_core_mib.c:scsi_auth_intr_seq_show() for both
explict NodeACL and TPG context 'demo-mode' generated struct se_node_acl
attached to a struct se_portal_group fabric endpoint.
> So, if the above examples are considered a misuse of sysfs,
> SCST would need to use both sysfs and configfs. It would use configfs to
> do configuration, and sysfs to display and access the dynamic information
> like connected initiators. That seems a minor role for configfs,
> which is easily handled in sysfs as SCST currently does.
>
> Just my two cents.
>
I think the question is more along the lines of: does sysfs have a hard
requirement for target mode using native TCM/ConfigFS logic.?. Early in
the development of TCM/LIO v3.0 I thought that driving creation of
configfs struct_groups from kernel code was in fact useful for my own
development and for future mainline code, but with the help of foresight
of jlbec, gregkh and others with our modern v4.0 code I am no longer
convinced we need any direct kernel level interaction between native
v4.0 target mode configfs code and existing sysfs code.
So far using interpreted userspace python/shell code has served this
purpose really quite well in terms of interaction between target mode
configfs and existing sysfs design, and I do not currently see any
inherient limitations from a userspace code driven persective for a full
native configfs target implementation. I would be happy to be proven
wrong on this point, but so far the original decision to drive target
mode completely from userspace with native configfs code has in fact
proven to be a better decision than a hybrid kernel level configfs +
sysfs implementation from the perspective of the userspace developer
driving /sys/kernel/config/target/* code.
So on this basis I am currently not interested in moving TCM/LIO code
away from a native configfs control plane for the v4.0 release, but I am
happy to discuss positive benefits that a hybrid implementation can
potentially provide to our existing TCM/LIO userspace ecosystem.
Best,
--nab
On Mon, Nov 15, 2010 at 11:13 PM, Greg KH <[email protected]> wrote:
> [ ... ]
> And use configfs for your configuration stuff, that's what it is there
> for, and if it doesn't somehow work properly for you, please work with
> the configfs developers to fix that up.
Hello Greg,
The reason why we use sysfs instead of configfs is that we do not only
want to export kernel objects to user space but because we also want
to allow configuration from user space. As far as I can see configfs
has been designed to allow configuration from user space only and not
for exporting kernel objects. A quote from
Documentation/filesystems/configfs/configfs.txt:
12 [What is configfs?]
13
14 configfs is a ram-based filesystem that provides the converse of
15 sysfs's functionality. Where sysfs is a filesystem-based view of
16 kernel objects, configfs is a filesystem-based manager of kernel
17 objects, or config_items.
Bart.
On Mon, 15 Nov 2010 21:04:19 -0800
Joe Eykholt <[email protected]> wrote:
>
>
> On 11/15/10 2:13 PM, Greg KH wrote:
> > On Mon, Nov 15, 2010 at 11:39:48PM +0300, Vladislav Bolkhovitin wrote:
> >> Greg KH, on 11/15/2010 09:44 PM wrote:
> >> > On Mon, Nov 15, 2010 at 06:45:24PM +0100, Bart Van Assche wrote:
> >> >> On Sun, Nov 14, 2010 at 12:59 AM, Greg KH <[email protected]> wrote:
> >> >>>
> >> >>> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
> >> >>>> So, I decided to reimplement it to be completely synchronous. SYSFS
> >> >>>> authors did really great job and thanks to the excellent internal SYSFS
> >> >>>> design and implementation it is absolutely safe. See:
> >> >>>>
> >> >>>> [root@tgt ~]# modprobe scst
> >> >>>> [root@tgt ~]# cd /sys/kernel/scst_tgt/
> >> >>>
> >> >>> Sorry, but no, you can't put this in /sys/kernel/ without getting the
> >> >>> approval of the sysfs maintainer.
> >> >>>
> >> >>> I really don't understand why you are using kobjects in the first place,
> >> >>> why isn't this in the main device tree in the kernel, using 'struct
> >> >>> device'?
> >> >>
> >> >> We might have missed something, but as far as we know it has not yet
> >> >> been explained in this thread why using 'struct device' would be an
> >> >> advantage over using 'struct kobject'.
> >> >
> >> > It's very simple.
> >> >
> >> > You want your device to show up in the global device tree in the kernel,
> >> > not off to one side, unconnected to anything else.
> >> >
> >> > Please use 'struct device', it is what you want to do here.
> >>
> >> But we don't have any device to show up in the global device tree!
> >
> > Not true at all.
> >
> >> We don't have any devices in the struct device's understanding at all!
> >
> > Then create them just like you are doing so for your kobject use.
> >
> > The first device would be the root one, and then everything trickles
> > down from there.
> >
> > And use configfs for your configuration stuff, that's what it is there
> > for, and if it doesn't somehow work properly for you, please work with
> > the configfs developers to fix that up.
> >
> > thanks,
> >
> > greg k-h
>
> I don't have any opinion on the above, but I don't see why sysfs can't be
> used for configuration as well as its other roles. It seems to me wasteful
> to require configfs to be used in order to change configuration when
> sysfs works fine for this.
Well, I'm not involved here.. but to me it makes sense. It's just a
useful principle in software design. You don't take something that is
designed for one purpose and mis-use it. It will grow warts and
mutate, stealing flexibility.
it's one of the reasons many software projects become
unmaintainable... nobody can change anything anymore because the
once simple design has proliferated into a nine headed monster...
> Just my two cents.
My two cents too.
>
> Joe
Flo
I'm just an outsider - but maybe my perspective has value - it seems there are two sides to this debate:
1) sysfs is great for scst due to certain stability concerns and code concerns
2) sysfs is bad for scst due to the intended role of sysfs and its namespace
Maybe I misunderstand -
But if both sides have merit then wouldn't a compromise be appropriate?
Maybe the sensical compromise is to use sysfs code to create a new namespace that would fit this purpose? It seems that I am also hearing that the alternatives to sysfs aren't always adequate - so why not use sysfs, but have a place where it's appropriate to use it?
Apologies in advance if I'm just way off base here...
- Richard Williams-
Nicholas A. Bellinger, on 11/15/2010 11:19 PM wrote:
>> I think that Vlad has already explained several times why ConfigFS is
>> not suited for the needs of a storage target: a storage target must
>> not only be able to accept configuration information from userspace
>> but must also be able to create new directories and file nodes itself.
>> See e.g. this message from October 6:
>> http://kerneltrap.org/mailarchive/linux-kernel/2010/10/6/4628664.
>
> Sorry, but this post explains nothing but a single misguided and
> uninformed opinion, with no hard facts on the actual usage of a native
> configfs control plane within target mode infrastructure.
What is "misguided and uninformed opinion"? That you can't with ConfigFS
serve real life needs for target driver developers and can't create
targets for hardware target ports from withing the kernel and instead
enforce users to clumsy workarounds to somehow magically know their
names to manually perform "mkdir target_name"?
The same problem exists for all other objects where you need to create
ConfigFS entries (directories), like for per-session/per-initiator
statistics or default ACLs.
ConfigFS is just too simple to serve real life needs of a SCSI target
subsystem.
Vlad
Dmitry Torokhov, on 11/16/2010 12:14 AM wrote:
>> Could you be more specific and point out on exact ways for that? From my
>> quite deep SYSFS source code study I see such cases should not exist.
>
> While I do not know offhand I am sure there are such scenarios. Isn't
> there any way for the users that you are waiting on descend back into
> your module that is waiting for kobject removal and get stuck on some
> resource?
No, I don't see any, because SYSFS implements atomic "all or nothing"
behavior on destroy, which is pretty bulletproof.
> Even if it isn't possible now the scheme is quite fragile. Kobjects are
> refcounted so work with them appropriately (rely on refcount, do not
> wait, etc).
The same is true for other SCST objects. For instance, a target can't be
destroyed until there are commands from it being processed. So, kobjects
are only one of the objects we wait for all their ref counters reach
zero, hence addition of kobjects to SCST objects changed nothing in this
area.
Vlad
(Since this discussion goes to a quite fundamental scope, I let myself
to add Joel Becker and Linus Torvalds on CC)
Richard Williams, on 11/16/2010 02:59 PM wrote:
> I'm just an outsider - but maybe my perspective has value - it seems
> there are two sides to this debate:
>
> 1) sysfs is great for scst due to certain stability concerns and code
> concerns
> 2) sysfs is bad for scst due to the intended role of sysfs
> and its namespace
Your questions are very good, so let's summarize what we need to serve
the needs of a SCSI target subsystem (not necessary SCST) and see what
can fit them.
So, the needs:
1. Be capable to represent to user space internal configuration to let
user space be able to see and analyze it, including various statistics.
2. Let user space manage the internal configuration.
3. Desired: possibility to send to user space events about important
internal actions, like I/O failures, which may need user space
intervention to recover, like switching from active to passive role in a
cluster.
So, what can we do with ConfigFS:
(1): Only partially, because by design ConfigFS isn't supposed to
represent internal configuration, it can only manage it. Extending
ConfigFS to be capable to do that would be, in my understanding, a
strong violation of its purpose and, hence, design and if went this way
eventually ConfigFS would become just a duplication of the SYSFS
functionality.
(2): ConfigFS can do that. This is exactly for what it was designed and
implemented. But in this particular application it would have some
limitations derived from (1): to manage harware-related entries a user
should magically know from somewhen names of those entries to create
them by "mkdir" command.
For instance, consider a user has a Fibre Channel HBA and want to use it
in the target mode. Before he can configure it, he should somehow know
its ports names and for each of them run:
# mkdir /sys/kernel/config/.../50:50:00:00:00:00:00:11
# mkdir /sys/kernel/config/.../50:50:00:00:00:00:00:12
...
where 50:50:00:00:00:00:00:1x are the ports' names. Only after that
those ports appear on the ConfigFS and can be managed.
(3): No events at all.
Now consider SYSFS:
(1): Easily. This is exactly for what it was designed and implemented.
(2): Possible without any limitations and side effects.
(3): Also possible.
So, why not use SYSFS if it suits all the needs _without_ any additional
effort and patches?
Other alternatives? A set of custom IOCTLs? One more configuration FS? I
believe, those would be quite disgusting for all.
> Maybe I misunderstand - But if both sides have merit then wouldn't a
> compromise be appropriate?
>
> Maybe the sensical compromise is to use sysfs code to create a new
> namespace that would fit this purpose? It seems that I am also
> hearing that the alternatives to sysfs aren't always adequate - so
> why not use sysfs, but have a place where it's appropriate to use it?
This is exactly what we are proposing: to use SYSFS in additional
namespace /sys/kernel/scst_tgt.
As far as I can see, only Greg is against it. Greg keeps his reasons
private, so I can only guess that Greg is against extending usage of
SYSFS (note, _usage_, not implementation! Everything needed long ago
implemented.) beyond the scope it was originally designed around 10
years ago. But SYSFS is already widely used this way in the kernel, as
Joe illustrated, hence there is demand for it. People need it. So, why
not to just acknowledge this fact and go ahead the simplest and most
useful for both users and developers way?
Thanks,
Vlad
Greg KH, on 11/16/2010 01:13 AM wrote:
>>> Please use 'struct device', it is what you want to do here.
>>
>> But we don't have any device to show up in the global device tree!
>
> Not true at all.
Why?
Greg, sorry, you keep writing as if we all are idiots, but keep refusing
to explain us, idiots, why we can't do what we did. It isn't very
constructive, is it?
>> We don't have any devices in the struct device's understanding at all!
>
> Then create them just like you are doing so for your kobject use.
>
> The first device would be the root one, and then everything trickles
> down from there.
Sorry, I can't understand what you mean. What would be the purpose of
this configuration and what benefits it would bring?
Anyway, let's forget about SCSI. Do you believe that struct device
should be created for all IP addresses on which an NFS server listen and
for all exports it exports?
> And use configfs for your configuration stuff, that's what it is there
> for, and if it doesn't somehow work properly for you, please work with
> the configfs developers to fix that up.
I explained why it isn't a too good option in another e-mail.
Thanks,
Vlad
Joe Eykholt, on 11/16/2010 08:04 AM wrote:
>> On Mon, Nov 15, 2010 at 11:39:48PM +0300, Vladislav Bolkhovitin wrote:
>>> Greg KH, on 11/15/2010 09:44 PM wrote:
>>>> On Mon, Nov 15, 2010 at 06:45:24PM +0100, Bart Van Assche wrote:
>>>>> On Sun, Nov 14, 2010 at 12:59 AM, Greg KH <[email protected]> wrote:
>>>>>>
>>>>>> On Sat, Nov 13, 2010 at 08:20:18PM +0300, Vladislav Bolkhovitin wrote:
>>>>>>> So, I decided to reimplement it to be completely synchronous. SYSFS
>>>>>>> authors did really great job and thanks to the excellent internal SYSFS
>>>>>>> design and implementation it is absolutely safe. See:
>>>>>>>
>>>>>>> [root@tgt ~]# modprobe scst
>>>>>>> [root@tgt ~]# cd /sys/kernel/scst_tgt/
>>>>>>
>>>>>> Sorry, but no, you can't put this in /sys/kernel/ without getting the
>>>>>> approval of the sysfs maintainer.
>>>>>>
>>>>>> I really don't understand why you are using kobjects in the first place,
>>>>>> why isn't this in the main device tree in the kernel, using 'struct
>>>>>> device'?
>>>>>
>>>>> We might have missed something, but as far as we know it has not yet
>>>>> been explained in this thread why using 'struct device' would be an
>>>>> advantage over using 'struct kobject'.
>>>>
>>>> It's very simple.
>>>>
>>>> You want your device to show up in the global device tree in the kernel,
>>>> not off to one side, unconnected to anything else.
>>>>
>>>> Please use 'struct device', it is what you want to do here.
>>>
>>> But we don't have any device to show up in the global device tree!
>>
>> Not true at all.
>>
>>> We don't have any devices in the struct device's understanding at all!
>>
>> Then create them just like you are doing so for your kobject use.
>>
>> The first device would be the root one, and then everything trickles
>> down from there.
>>
>> And use configfs for your configuration stuff, that's what it is there
>> for, and if it doesn't somehow work properly for you, please work with
>> the configfs developers to fix that up.
>>
>> thanks,
>>
>> greg k-h
>
> I don't have any opinion on the above, but I don't see why sysfs can't be
> used for configuration as well as its other roles. It seems to me wasteful
> to require configfs to be used in order to change configuration when
> sysfs works fine for this.
>
> Here are a couple of existing examples where sysfs is used in a role that
> would seem similar to SCST's usage:
>
> 1) scsi_transport_fc already has an sysfs file, fc_host/vport_create, which
> can be written to create new fc_host and scsi_host instances.
>
> 2) fcoe.ko uses a write to the sysfs file /sys/module/fcoe/parameters/create
> to start the protocol on a particular ethernet interface. There's another
> file for destroy.
>
> I'll bet there are other examples in other subsystems, and I don't think
> there is anything wrong with the above usages of sysfs.
>
> I agree with Vladislav's point that configfs doesn't work instead of sysfs
> because configs doesn't make it easy for the kernel side to create nodes for
> dynamic information like connected initiators and statistics.
> So, if the above examples are considered a misuse of sysfs,
> SCST would need to use both sysfs and configfs. It would use configfs to
> do configuration, and sysfs to display and access the dynamic information
> like connected initiators. That seems a minor role for configfs,
> which is easily handled in sysfs as SCST currently does.
>
> Just my two cents.
Thank you, Joe, for those good examples. I may not know for what SYSFS
was _originally_ designed, but the current reality is that SYSFS is
_widely_ used to both represent the kernel internal configuration as
well as to change it. And SCST is just following that.
After all, if SYSFS supposed to only represent the kernel internal
configuration, why wasn't it from the beginning made read-only?
Vlad
Vladislav Bolkhovitin, on 11/16/2010 04:17 PM wrote:
> Your questions are very good, so let's summarize what we need to serve
> the needs of a SCSI target subsystem (not necessary SCST) and see what
> can fit them.
>
> So, the needs:
>
> 1. Be capable to represent to user space internal configuration to let
> user space be able to see and analyze it, including various statistics.
>
> 2. Let user space manage the internal configuration.
>
> 3. Desired: possibility to send to user space events about important
> internal actions, like I/O failures, which may need user space
> intervention to recover, like switching from active to passive role in a
> cluster.
>
> So, what can we do with ConfigFS:
>
> (1): Only partially, because by design ConfigFS isn't supposed to
> represent internal configuration, it can only manage it. Extending
> ConfigFS to be capable to do that would be, in my understanding, a
> strong violation of its purpose and, hence, design and if went this way
> eventually ConfigFS would become just a duplication of the SYSFS
> functionality.
>
> (2): ConfigFS can do that. This is exactly for what it was designed and
> implemented. But in this particular application it would have some
> limitations derived from (1): to manage harware-related entries a user
> should magically know from somewhen names of those entries to create
> them by "mkdir" command.
>
> For instance, consider a user has a Fibre Channel HBA and want to use it
> in the target mode. Before he can configure it, he should somehow know
> its ports names and for each of them run:
>
> # mkdir /sys/kernel/config/.../50:50:00:00:00:00:00:11
> # mkdir /sys/kernel/config/.../50:50:00:00:00:00:00:12
> ...
>
> where 50:50:00:00:00:00:00:1x are the ports' names. Only after that
> those ports appear on the ConfigFS and can be managed.
>
> (3): No events at all.
>
> Now consider SYSFS:
>
> (1): Easily. This is exactly for what it was designed and implemented.
>
> (2): Possible without any limitations and side effects.
>
> (3): Also possible.
>
> So, why not use SYSFS if it suits all the needs _without_ any additional
> effort and patches?
>
> Other alternatives? A set of custom IOCTLs? One more configuration FS? I
> believe, those would be quite disgusting for all.
>
>> Maybe I misunderstand - But if both sides have merit then wouldn't a
>> compromise be appropriate?
>>
>> Maybe the sensical compromise is to use sysfs code to create a new
>> namespace that would fit this purpose? It seems that I am also
>> hearing that the alternatives to sysfs aren't always adequate - so
>> why not use sysfs, but have a place where it's appropriate to use it?
>
> This is exactly what we are proposing: to use SYSFS in additional
> namespace /sys/kernel/scst_tgt.
>
> As far as I can see, only Greg is against it. Greg keeps his reasons
> private, so I can only guess that Greg is against extending usage of
> SYSFS (note, _usage_, not implementation! Everything needed long ago
> implemented.) beyond the scope it was originally designed around 10
> years ago. But SYSFS is already widely used this way in the kernel, as
> Joe illustrated, hence there is demand for it. People need it. So, why
> not to just acknowledge this fact and go ahead the simplest and most
> useful for both users and developers way?
Since nobody objected, Greg, could you consider to ACK SCST SYSFS
management interface in /sys/kernel/scst_tgt/, please? Please find the
SCST SYSFS ABI documentation file you requested below.
We are also preparing a new patch to free our objects in kobjects.release()
without explicit x_initialized flags as you requested.
Thank you for your effort,
Vlad
Documentation/ABI/testing/sysfs-scst:
What: /sys/kernel/scst_tgt/
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains SCST management interface entries.
What: /sys/kernel/scst_tgt/devices/
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains subdirectories for all SCST devices
What: /sys/kernel/scst_tgt/devices/<device>/exported/exportX
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Links to LUNs in the LUNs group where <device> exported, e.g. to
/sys/kernel/scst_tgt/targets/iscsi/iqn.2006-10.net.vlnb:tgt/luns/11
What: /sys/kernel/scst_tgt/devices/<device>/handler
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Link to dev handler of this device, if assigned, e.g. to
/sys/kernel/scst_tgt/handlers/dev_disk
What: /sys/kernel/scst_tgt/devices/<device>/type
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
SCSI type of this device as define by SAM.
What: /sys/kernel/scst_tgt/handlers/
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains all SCST dev handlers.
What: /sys/kernel/scst_tgt/handlers/<handler>/<device>
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Links to <device> managed by this dev handler, e.g.
ext3_disk1_4K -> /sys/kernel/scst_tgt/devices/ext3_disk1_4K
What: /sys/kernel/scst_tgt/handlers/<handler>/mgmt
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Management entry, which allows to create and delete
devices for this dev handler. See SysfsRules file for more
info.
What: /sys/kernel/scst_tgt/handlers/<handler>/type
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
SCSI type of this dev handler as define by SAM.
What: /sys/kernel/scst_tgt/sgv/
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains SCST SGV caches statistics.
What: /sys/kernel/scst_tgt/sgv/global_stats
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains global SGV cache statistics.
What: /sys/kernel/scst_tgt/sgv/<cache>/stats
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains statistics for SGV cache <cache>
What: /sys/kernel/scst_tgt/targets/
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains all SCST target drivers
What: /sys/kernel/scst_tgt/targets/<target_driver>
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains all targets for <target_driver>
What: /sys/kernel/scst_tgt/targets/<target_driver>/mgmt
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Optional management entry, which allows to create and
delete targets for this target driver. See SysfsRules
file for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/enable
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Optional attribute to enable <target_driver> and make it serve
incoming connections from initiators. Possible values:
1 - enable
0 - disable
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Contains security groups for <target>
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/mgmt
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Allows to create and delete security groups for <target>.
See README.scst for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/initiators/mgmt
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Allows to add and delete initiators to/from <group>.
See README.scst for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/initiators/<initiator_name>
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
One or more initiators in <group>. Contains initiator's name.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/luns/mgmt
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Allows to add and delete LUNs to/from <group>. See README.scst
for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/luns/<lun>/device
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Link to device for <lun>, e.g. to /sys/kernel/scst_tgt/devices/ext3_disk1_4K
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/luns/<lun>/read_only
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Sets if this LUN should be read only
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/addr_method
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Sets SCSI addressing method for <group>. See README.scst
for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/cpu_mask
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Sets CPU mask for threads serving initiators in <group>.
See README.scst for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/io_grouping_type
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Sets IO grouping types for threads serving initiators
in <group>. See README.scst for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/luns/mgmt
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Allows to add and delete LUNs to/from the <target>'s default
set of LUNs. See README.scst for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/luns/<lun>/device
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Link to device for <lun>, e.g. to /sys/kernel/scst_tgt/devices/ext3_disk1_4K
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/luns/<lun>/read_only
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Sets if this LUN should be read only
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/addr_method
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Sets SCSI addressing method for the <target>'s default
set of LUNs. See README.scst for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/cpu_mask
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Sets CPU mask for threads serving initiators in the
<target>'s default set of LUNs. See README.scst for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/io_grouping_type
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Sets IO grouping types for threads serving initiators
in the <target>'s default set of LUNs. See README.scst
for more info.
What: /sys/kernel/scst_tgt/targets/<target_driver>/<target>/enable
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Optional attribute to enable <target> and make it serve
incoming connections from initiators. Possible values:
1 - enable
0 - disable
What: /sys/kernel/scst_tgt/last_sysfs_mgmt_res
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Returning completion status of the last management
command. See README.scst for more info.
What: /sys/kernel/scst_tgt/setup_id
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Allows to read and write SCST setup ID. This ID can be
used in cases, when the same SCST configuration should
be installed on several targets, but exported from those
targets devices should have different IDs and SNs. For
instance, VDISK dev handler uses this ID to generate T10
vendor specific identifier and SN of the devices.
What: /sys/kernel/scst_tgt/threads
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Allows to read and set number of global SCST I/O
threads. Those threads used with async. dev handlers,
for instance, vdisk BLOCKIO or NULLIO
What: /sys/kernel/scst_tgt/version
Date: November 2010
Contact: Vladislav Bolkhovitin <[email protected]>
Description:
Allows to see version of SCST and enabled optional features.
On Fri, Nov 19, 2010 at 12:02:58AM +0300, Vladislav Bolkhovitin wrote:
> Since nobody objected, Greg, could you consider to ACK SCST SYSFS
> management interface in /sys/kernel/scst_tgt/, please? Please find the
> SCST SYSFS ABI documentation file you requested below.
No, sorry, again, you should not be using kobjects, and do not polute
the main /sys/kernel/ namespace with this.
Use 'struct device' please, that is what it is there for, and is what
the rest of the kernel is using. And use the rest of the
driver/bus/device infrastructure as your model will work with it just
fine.
Yes, I know you said you didn't think you could use it, and that your
code was different than everyone elses, but I still do not believe it,
sorry.
good luck,
greg k-h
Greg KH, on 11/19/2010 12:46 AM wrote:
> On Fri, Nov 19, 2010 at 12:02:58AM +0300, Vladislav Bolkhovitin wrote:
>> Since nobody objected, Greg, could you consider to ACK SCST SYSFS
>> management interface in /sys/kernel/scst_tgt/, please? Please find the
>> SCST SYSFS ABI documentation file you requested below.
>
> No, sorry, again, you should not be using kobjects, and do not polute
> the main /sys/kernel/ namespace with this.
Which other namespace should we "polute" then?
> Use 'struct device' please, that is what it is there for, and is what
> the rest of the kernel is using. And use the rest of the
> driver/bus/device infrastructure as your model will work with it just
> fine.
Greg, sorry, I don't understand your requirements and, because of this,
we can't go ahead implementing them. Could you explain your position,
please?
None of the SCST objects are Linux devices. None of them has entries in
/dev, none of them needs to send any events to udev and none of them
sends or receives data from DMA, hence has any DMA parameters or
restrictions. So, how can them fit into the driver/bus/device model you
are enforcing?
For instance:
- struct sgv_pool (/sys/kernel/scst_tgt/sgv/<cache>) is an SG vectors
cache. Isn't it a nonsense to make it device?
- struct scst_acg
(/sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/)
is an access control group to define which initiators see which LUNs
from target <target>. Same, isn't it a nonsense to make it device?
- struct scst_acn
(/sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/initiators/<initiator_name>)
is an entry in the access control group <group> to define which
initiators should use group <group>. Again, isn't it a nonsense to make
it device?
Etc.
How could they fit in the driver/bus/device model?
I guess, your confusion comes from word "device" you see in the SCST
SYSFS tree and SCST supposed to work with "devices" and "dev handlers"?
But those devices are not Linux devices, they are just objects to couple
initiators' requests and backstorage which stores data.
In other words, SCST devices are just redirection points passing
incoming requests to Linux files and devices.
SCST devices are the same as NFS exported directories. Do you believe,
that struct device must be created for each NFS export?
In the cases when we need to present SCST devices as Linux devices we
use scst_local driver [1], which performs it exactly as you requesting,
i.e. doing:
root_device_register()
bus_register()
driver_register()
...
> Yes, I know you said you didn't think you could use it, and that your
> code was different than everyone elses, but I still do not believe it,
> sorry.
What can we do to make you believe it? Shall we write a document
describing all SCST objects and their relationship between each other?
(Although looks like you haven't read even detailed docs we already
written...)
Our position is based on careful study of all possible alternatives and
10+ years of Linux kernel development.
SCST is being developed for 7+ years and, despite of not being mainline,
has proved those years being the right way to go, not the mainline STGT
which was chosen 5 years ago as the right way, but now we are
considering to replace it.
Sorry, but we have an impression that you are judging without seeing the
full picture. Isn't it a duty of a subsystem's maintainer to see full
picture before deciding if it's good or bad?
We appreciate your effort reviewing SCST and willing to make all the
needed actions to help you.
The world of SCSI targets is pretty complex (I have been studying it for
7+ years), hence SCST is very big (20K LOC only core) and hard to
understand. But is it a sufficient reason to force to convert elegant,
nice and carefully thought design into something ugly, hardly usable and
hardly maintainable? If we do it, that wouldn't make SCST simpler to
review. Opposite, it would make it HARDER to review.
Thanks,
Vlad
[1] Patch for scst_local you can find in http://lkml.org/lkml/2010/10/1/131)
On Thu, Nov 18, 2010 at 10:46 PM, Greg KH <[email protected]> wrote:
> On Fri, Nov 19, 2010 at 12:02:58AM +0300, Vladislav Bolkhovitin wrote:
>> Since nobody objected, Greg, could you consider to ACK SCST SYSFS
>> management interface in /sys/kernel/scst_tgt/, please? Please find the
>> SCST SYSFS ABI documentation file you requested below.
>
> No, sorry, again, you should not be using kobjects, and do not pollute
> the main /sys/kernel/ namespace with this.
>
> Use 'struct device' please, that is what it is there for, and is what
> the rest of the kernel is using. ?And use the rest of the
> driver/bus/device infrastructure as your model will work with it just
> fine.
>
> Yes, I know you said you didn't think you could use it, and that your
> code was different than everyone elses, but I still do not believe it,
> sorry.
Hello Greg,
As you can see in recent messages (e.g.
http://lkml.org/lkml/2010/11/18/578 or
http://lkml.org/lkml/2010/11/15/296), the abstractions represented in
SCST are:
* Target templates (scst_tgt_template), e.g. scst_local or ib_srpt.
These are drivers that implement a storage protocol.
* Target ports (scst_tgt), e.g. ib_srpt_target_0, which represent a
communication interface controlled by a storage protocol driver.
* Sessions (scst_session), e.g. 0x00000000000000000002c9030005f34b. A
session corresponds to a single initiator-target nexus.
* Device handlers (scst_dev_type), e.g. scst_disk or scst_vdisk, which
are drivers that allow SCST to export storage.
* Target devices (scst_device), e.g. disk01, which represent exported
storage. Each instance is controlled by a single device handler. This
concept includes e.g. block devices and files. Some but not all target
devices have a corresponding device node in /dev.
* Access control groups (scst_acg), which allow e.g. to implement LUN masking.
* Device-specific information such as the SCSI LUN (scst_acg_dev).
As we all know the driver, bus and device abstractions were invented
to model how peripheral devices are connected to a system. What a
storage target does is the converse - define devices and make it
possible for other systems to use these devices. As a result,
unfortunately, the driver, bus and device abstractions do not map
one-to-one on all of the concepts used in a storage target. So I'm
still not sure whether it is a good idea to use the driver, bus and
or/device concepts for all of the above concepts.
Also, using the driver, bus or device concept for one or more of the
storage target concepts would open up a potential for naming
conflicts. There is already e.g. the well-known null device. SCST e.g.
defines a vdisk_nullio target template. Having these all as device
nodes under /dev might not only confuse users but also creates a huge
potential for naming conflicts. That's why we prefer separate
namespaces instead of reusing one of the existing concepts.
In case you are still convinced that we should use the existing
driver, bus and device abstractions, which concept should we map to
which abstraction ?
Bart.
On Fri, Nov 19, 2010 at 09:00:42PM +0300, Vladislav Bolkhovitin wrote:
> Greg KH, on 11/19/2010 12:46 AM wrote:
> > On Fri, Nov 19, 2010 at 12:02:58AM +0300, Vladislav Bolkhovitin wrote:
> >> Since nobody objected, Greg, could you consider to ACK SCST SYSFS
> >> management interface in /sys/kernel/scst_tgt/, please? Please find the
> >> SCST SYSFS ABI documentation file you requested below.
> >
> > No, sorry, again, you should not be using kobjects, and do not polute
> > the main /sys/kernel/ namespace with this.
>
> Which other namespace should we "polute" then?
>
> > Use 'struct device' please, that is what it is there for, and is what
> > the rest of the kernel is using. And use the rest of the
> > driver/bus/device infrastructure as your model will work with it just
> > fine.
>
> Greg, sorry, I don't understand your requirements and, because of this,
> we can't go ahead implementing them. Could you explain your position,
> please?
>
> None of the SCST objects are Linux devices. None of them has entries in
> /dev, none of them needs to send any events to udev and none of them
> sends or receives data from DMA, hence has any DMA parameters or
> restrictions. So, how can them fit into the driver/bus/device model you
> are enforcing?
>
Note that the entities in /sys/devices/... tree and not necessarily
physical devices bit rather interface abstractionss. Consider, for
example, /sys/class/input/*. None of the "devices" there talk directly
to hardware, do DMA or other things. Some of them don't even talk to
usrespace directly but rather through additional interfaces (evdev.
mousedev, ect). Still they are represented there and even have suspend
and resume methods (because even for logical devices it makes sense to
save and restore some state).
> For instance:
>
> - struct sgv_pool (/sys/kernel/scst_tgt/sgv/<cache>) is an SG vectors
> cache. Isn't it a nonsense to make it device?
>
> - struct scst_acg
> (/sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/)
> is an access control group to define which initiators see which LUNs
> from target <target>. Same, isn't it a nonsense to make it device?
>
> - struct scst_acn
> (/sys/kernel/scst_tgt/targets/<target_driver>/<target>/ini_groups/<group>/initiators/<initiator_name>)
> is an entry in the access control group <group> to define which
> initiators should use group <group>. Again, isn't it a nonsense to make
> it device?
>
> Etc.
>
> How could they fit in the driver/bus/device model?
Maybe not all of them are. Some of them could probably be represented by
attributes of other devices. And some of them are, fitting into the
overall /sys/devices hierarchy and describing physical and logical
relations between them.
--
Dmitry
Dmitry Torokhov, on 11/19/2010 11:22 PM wrote:
>> None of the SCST objects are Linux devices. None of them has entries in
>> /dev, none of them needs to send any events to udev and none of them
>> sends or receives data from DMA, hence has any DMA parameters or
>> restrictions. So, how can them fit into the driver/bus/device model you
>> are enforcing?
>
> Note that the entities in /sys/devices/... tree and not necessarily
> physical devices bit rather interface abstractionss. Consider, for
> example, /sys/class/input/*. None of the "devices" there talk directly
> to hardware, do DMA or other things. Some of them don't even talk to
> usrespace directly but rather through additional interfaces (evdev.
> mousedev, ect). Still they are represented there and even have suspend
> and resume methods (because even for logical devices it makes sense to
> save and restore some state).
But all of them still place from where to events received and where
requests from Linux sent, aren't them?
SCST devices are not even logical devices. As I wrote, "devices" word is
misleading. SCST devices are converse of what Linux means under this
word. SCST devices are like NFS exports: a place where those events
generated and those requests received.
Think of SCST device as if it sits on the opposite side of the PCI bus
of the corresponding SCSI device Linux sees in /sys/class and /sys/bus.
So, if we need Linux devices for SCST devices, we create them using
scst_local driver. And then, of course, all them have their place in
/sys/class/ and /sys/bus. Although more common use of them from remote
systems via iSCSI, Fibre Channel, InfiniBand, etc. The remote systems
create devices in /sys/class/ and /sys/bus (if they are Linux), we serve
them.
Vlad
On Fri, Nov 19, 2010 at 11:50:43PM +0300, Vladislav Bolkhovitin wrote:
> Dmitry Torokhov, on 11/19/2010 11:22 PM wrote:
> >> None of the SCST objects are Linux devices. None of them has entries in
> >> /dev, none of them needs to send any events to udev and none of them
> >> sends or receives data from DMA, hence has any DMA parameters or
> >> restrictions. So, how can them fit into the driver/bus/device model you
> >> are enforcing?
> >
> > Note that the entities in /sys/devices/... tree and not necessarily
> > physical devices bit rather interface abstractionss. Consider, for
> > example, /sys/class/input/*. None of the "devices" there talk directly
> > to hardware, do DMA or other things. Some of them don't even talk to
> > usrespace directly but rather through additional interfaces (evdev.
> > mousedev, ect). Still they are represented there and even have suspend
> > and resume methods (because even for logical devices it makes sense to
> > save and restore some state).
This is correct.
> SCST devices are not even logical devices. As I wrote, "devices" word is
> misleading. SCST devices are converse of what Linux means under this
> word. SCST devices are like NFS exports: a place where those events
> generated and those requests received.
No, that's fine.
> Think of SCST device as if it sits on the opposite side of the PCI bus
> of the corresponding SCSI device Linux sees in /sys/class and /sys/bus.
Again, that's fine, look at usb gadgets, it's the same thing.
> So, if we need Linux devices for SCST devices, we create them using
> scst_local driver. And then, of course, all them have their place in
> /sys/class/ and /sys/bus.
No, just /sys/bus/ which will cause them to become part of the big
device tree in /sys/devices/
Again, use this interface, it is what it is there for, to not use it is
just wrong.
good luck,
greg k-h
On Fri, Nov 19, 2010 at 09:00:42PM +0300, Vladislav Bolkhovitin wrote:
> Greg KH, on 11/19/2010 12:46 AM wrote:
> > On Fri, Nov 19, 2010 at 12:02:58AM +0300, Vladislav Bolkhovitin wrote:
> >> Since nobody objected, Greg, could you consider to ACK SCST SYSFS
> >> management interface in /sys/kernel/scst_tgt/, please? Please find the
> >> SCST SYSFS ABI documentation file you requested below.
> >
> > No, sorry, again, you should not be using kobjects, and do not polute
> > the main /sys/kernel/ namespace with this.
>
> Which other namespace should we "polute" then?
None. Use 'struct device'
> > Use 'struct device' please, that is what it is there for, and is what
> > the rest of the kernel is using. And use the rest of the
> > driver/bus/device infrastructure as your model will work with it just
> > fine.
>
> Greg, sorry, I don't understand your requirements and, because of this,
> we can't go ahead implementing them. Could you explain your position,
> please?
I have multiple times.
> None of the SCST objects are Linux devices. None of them has entries in
> /dev, none of them needs to send any events to udev and none of them
> sends or receives data from DMA, hence has any DMA parameters or
> restrictions. So, how can them fit into the driver/bus/device model you
> are enforcing?
That doesn't matter. They are still "devices" that the kernel knows
about and as such, fit into the device tree of everything in the kernel.
> Sorry, but we have an impression that you are judging without seeing the
> full picture. Isn't it a duty of a subsystem's maintainer to see full
> picture before deciding if it's good or bad?
It's the duty of a subsystem's maintainer to enforce the correct model
of the kernel, and that is what I am doing.
Again, this is the last email I'm writing on this topic, as none of the
previous ones seem to be sinking in.
good luck,
greg k-h
Greg KH, on 11/20/2010 12:16 AM wrote:
>>>> None of the SCST objects are Linux devices. None of them has entries in
>>>> /dev, none of them needs to send any events to udev and none of them
>>>> sends or receives data from DMA, hence has any DMA parameters or
>>>> restrictions. So, how can them fit into the driver/bus/device model you
>>>> are enforcing?
>>>
>>> Note that the entities in /sys/devices/... tree and not necessarily
>>> physical devices bit rather interface abstractionss. Consider, for
>>> example, /sys/class/input/*. None of the "devices" there talk directly
>>> to hardware, do DMA or other things. Some of them don't even talk to
>>> usrespace directly but rather through additional interfaces (evdev.
>>> mousedev, ect). Still they are represented there and even have suspend
>>> and resume methods (because even for logical devices it makes sense to
>>> save and restore some state).
>
> This is correct.
>
>> SCST devices are not even logical devices. As I wrote, "devices" word is
>> misleading. SCST devices are converse of what Linux means under this
>> word. SCST devices are like NFS exports: a place where those events
>> generated and those requests received.
>
> No, that's fine.
I'm surprised you would make NFS exports devices
>> Think of SCST device as if it sits on the opposite side of the PCI bus
>> of the corresponding SCSI device Linux sees in /sys/class and /sys/bus.
>
> Again, that's fine, look at usb gadgets, it's the same thing.
USB gadgets are a good example, but not quite.
I'm objecting not the possibility to implement SCST objects as devices.
We are creating software, so all what hardware allows can be implemented.
I'm arguing usage of devices view for something which fundamentally not
devices.
USB gadgets are subset of what SCST allows. On the external side they
are tightly coupled to hardware, on the backend side they don't need any
management interface, because (at least for storage) all devices
configuration they have is static via module parameters. The only
attributes file_storage has are to change FUA and file path, which can
be placed anywhere, including /sys/module/g_file_storage/parameters.
So, for USB gadgets the config interface and place in /sys doesn't
matter much. I guess, making its LUNs as Linux devices were just part of
the ritual to get accepted into the kernel. On the USB gadgets
developers place I wouldn't mind too to go this way.
But SCST is a _big_, _complete_, _new_ Linux subsystem. Actually, USB
storage gadgets should be SCST targets to use full power of all possible
virtual and real backends SCST provides, hence stay inside SCST subtree.
For SCST it is very important that all its functionality concentrated in
one place and not confused with other client side devices Linux has.
Important for clearness, easy to use, easy to understand, flexibility,
maintainability, etc.
It's like as you created rules to keep and train dogs. Then you need to
keep and train cats as well. Would it be wise to keep cats in the same
place together with dogs and train them the same tricks using the same
rules as dogs?
>> > None of the SCST objects are Linux devices. None of them has entries in
>> > /dev, none of them needs to send any events to udev and none of them
>> > sends or receives data from DMA, hence has any DMA parameters or
>> > restrictions. So, how can them fit into the driver/bus/device model you
>> > are enforcing?
>
> That doesn't matter. They are still "devices" that the kernel knows
> about and as such, fit into the device tree of everything in the kernel.
If you have such wide devices definition, why file systems have separate
/sys/fs namespace? Or ksm place in /sys/kernel/mm/ksm? Or hugepages in
/sys/kernel/mm/hugepages?
If NFS exports are devices, file systems must also be devices, correct?
Like /sys/fs/ext4/sda11. Isn't it a full analogy to NFS export
/home/user/shared_dir?
>> > Sorry, but we have an impression that you are judging without seeing the
>> > full picture. Isn't it a duty of a subsystem's maintainer to see full
>> > picture before deciding if it's good or bad?
> It's the duty of a subsystem's maintainer to enforce the correct model
> of the kernel, and that is what I am doing.
>
> Again, this is the last email I'm writing on this topic, as none of the
> previous ones seem to be sinking in.
Well, if I don't agree with you it doesn't mean I'm not listening to
you. I do. Very carefully.
Thanks,
Vlad
On Fri, Nov 19, 2010 at 10:19 PM, Greg KH <[email protected]> wrote:
> [ ... ]
>
> None. ?Use 'struct device'
How about using 'struct device' as follows ?
* Move /sys/kernel/scst_tgt/targets to /sys/class/target_driver.
* Move /sys/kernel/scst_tgt/handlers to /sys/class/device_driver.
* Move /sys/kernel/scst_tgt/devices to /sys/class/target_device.
* Move the attributes of /sys/kernel/scst_tgt to /sys/devices/scst.
* Move /sys/kernel/scst_tgt/sgv to /sys/devices/scst/sgv
Some quickly hacked up code (that needs further polishing) that
implements this scheme can be found here:
https://scst.svn.sourceforge.net/svnroot/scst/branches/sysfs-tree-changes.
An example of an SCST configuration file and the corresponding sysfs hierarchy:
# uname -r
2.6.37-rc5-scst+
# cat /etc/scst.conf
HANDLER vdisk_blockio {
DEVICE disk01 {
filename /dev/ram0
nv_cache 1
}
DEVICE disk02 {
filename /dev/ram1
nv_cache 1
}
}
HANDLER vdisk_fileio {
DEVICE disk03 {
filename /dev/vdisk
nv_cache 1
}
DEVICE disk04 {
filename /dev/vdisk
nv_cache 1
}
}
HANDLER vdisk_nullio {
DEVICE disk05
DEVICE disk06
}
TARGET_DRIVER scst_local {
TARGET local {
session_name local
LUN 0 disk01
LUN 1 disk02
LUN 2 disk03
LUN 3 disk04
LUN 4 disk05
LUN 5 disk06
}
}
TARGET_DRIVER ib_srpt {
TARGET ib_srpt_target_0 {
rel_tgt_id 1
enabled 1
LUN 0 disk01
LUN 1 disk02
LUN 2 disk03
LUN 3 disk04
LUN 4 disk05
LUN 5 disk06
}
TARGET ib_srpt_target_1 {
rel_tgt_id 2
enabled 1
LUN 0 disk01
LUN 1 disk02
LUN 2 disk03
LUN 3 disk04
LUN 4 disk05
LUN 5 disk06
}
}
TARGET_DRIVER iscsi {
enabled 1
TARGET iqn.2005-03.org.open-iscsi:dbc01e1792b:storage {
rel_tgt_id 4
LUN 0 disk01
LUN 1 disk02
LUN 2 disk03
LUN 3 disk04
LUN 4 disk05
LUN 5 disk06
enabled 1
}
}
# find /sys/{class,devices/virtual}/{target_driver,device_driver,target_device}
/sys/devices/scst | while read f; do if [ -h $f ]; then echo "$f ->
$(readlink $f)"; else echo "$f"; fi; done
/sys/class/target_driver
/sys/class/target_driver/iscsi -> ../../devices/virtual/target_driver/iscsi
/sys/class/target_driver/ib_srpt -> ../../devices/virtual/target_driver/ib_srpt
/sys/class/target_driver/scst_local ->
../../devices/virtual/target_driver/scst_local
/sys/class/device_driver
/sys/class/device_driver/dev_disk_perf ->
../../devices/virtual/device_driver/dev_disk_perf
/sys/class/device_driver/dev_disk ->
../../devices/virtual/device_driver/dev_disk
/sys/class/device_driver/vdisk_fileio ->
../../devices/virtual/device_driver/vdisk_fileio
/sys/class/device_driver/vdisk_blockio ->
../../devices/virtual/device_driver/vdisk_blockio
/sys/class/device_driver/vdisk_nullio ->
../../devices/virtual/device_driver/vdisk_nullio
/sys/class/device_driver/vcdrom -> ../../devices/virtual/device_driver/vcdrom
/sys/class/target_device
/sys/class/target_device/disk01 -> ../../devices/virtual/target_device/disk01
/sys/class/target_device/2:0:0:0 -> ../../devices/virtual/target_device/2:0:0:0
/sys/class/target_device/2:0:1:0 -> ../../devices/virtual/target_device/2:0:1:0
/sys/class/target_device/3:0:0:0 -> ../../devices/virtual/target_device/3:0:0:0
/sys/class/target_device/disk02 -> ../../devices/virtual/target_device/disk02
/sys/class/target_device/disk03 -> ../../devices/virtual/target_device/disk03
/sys/class/target_device/disk04 -> ../../devices/virtual/target_device/disk04
/sys/class/target_device/disk05 -> ../../devices/virtual/target_device/disk05
/sys/class/target_device/disk06 -> ../../devices/virtual/target_device/disk06
/sys/devices/virtual/target_driver
/sys/devices/virtual/target_driver/iscsi
/sys/devices/virtual/target_driver/iscsi/iSNSServer
/sys/devices/virtual/target_driver/iscsi/enabled
/sys/devices/virtual/target_driver/iscsi/uevent
/sys/devices/virtual/target_driver/iscsi/subsystem ->
../../../../class/target_driver
/sys/devices/virtual/target_driver/iscsi/power
/sys/devices/virtual/target_driver/iscsi/power/wakeup
/sys/devices/virtual/target_driver/iscsi/power/wakeup_count
/sys/devices/virtual/target_driver/iscsi/power/wakeup_active_count
/sys/devices/virtual/target_driver/iscsi/power/wakeup_hit_count
/sys/devices/virtual/target_driver/iscsi/power/wakeup_active
/sys/devices/virtual/target_driver/iscsi/power/wakeup_total_time_ms
/sys/devices/virtual/target_driver/iscsi/power/wakeup_max_time_ms
/sys/devices/virtual/target_driver/iscsi/power/wakeup_last_time_ms
/sys/devices/virtual/target_driver/iscsi/power/runtime_status
/sys/devices/virtual/target_driver/iscsi/power/control
/sys/devices/virtual/target_driver/iscsi/power/runtime_suspended_time
/sys/devices/virtual/target_driver/iscsi/power/runtime_active_time
/sys/devices/virtual/target_driver/iscsi/power/autosuspend_delay_ms
/sys/devices/virtual/target_driver/iscsi/add_target_parameters
/sys/devices/virtual/target_driver/iscsi/driver_attributes
/sys/devices/virtual/target_driver/iscsi/target_attributes
/sys/devices/virtual/target_driver/iscsi/version
/sys/devices/virtual/target_driver/iscsi/open_state
/sys/devices/virtual/target_driver/iscsi/trace_level
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/uevent
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/subsystem
-> ../../../../../class/target_instance
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/device
-> ../../iscsi
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/wakeup
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/wakeup_count
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/wakeup_active_count
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/wakeup_hit_count
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/wakeup_active
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/wakeup_total_time_ms
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/wakeup_max_time_ms
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/wakeup_last_time_ms
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/runtime_status
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/control
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/runtime_suspended_time
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/runtime_active_time
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/power/autosuspend_delay_ms
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/enabled
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/commands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/active_commands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/initiator_name
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/InitialR2T
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/ImmediateData
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/MaxRecvDataSegmentLength
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/MaxXmitDataSegmentLength
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/MaxBurstLength
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/FirstBurstLength
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/MaxOutstandingR2T
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/HeaderDigest
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/DataDigest
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/sid
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/reinstating
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/force_close
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/luns
-> ../../luns
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun0
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun0/active_commands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun1
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun1/active_commands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun2
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun2/active_commands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun3
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun3/active_commands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun4
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun4/active_commands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun5
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/lun5/active_commands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/192.168.2.6
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/192.168.2.6/state
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/192.168.2.6/cid
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/sessions/iqn.1996-04.de.suse:01:e4bde122139a/192.168.2.6/ip
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/parameters
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/0
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/0/read_only
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/0/device
-> ../../../../../target_device/disk01
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/1
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/1/read_only
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/1/device
-> ../../../../../target_device/disk02
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/2
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/2/read_only
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/2/device
-> ../../../../../target_device/disk03
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/3
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/3/read_only
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/3/device
-> ../../../../../target_device/disk04
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/4
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/4/read_only
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/4/device
-> ../../../../../target_device/disk05
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/5
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/5/read_only
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/5/device
-> ../../../../../target_device/disk06
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/ini_groups
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/rel_tgt_id
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/addr_method
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/io_grouping_type
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/cpu_mask
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/tid
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/per_portal_acl
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/redirect
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/InitialR2T
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/ImmediateData
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxRecvDataSegmentLength
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxXmitDataSegmentLength
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxBurstLength
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/FirstBurstLength
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxOutstandingR2T
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/HeaderDigest
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/DataDigest
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/QueuedCommands
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/RspTimeout
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/NopInInterval
/sys/devices/virtual/target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/MaxSessions
/sys/devices/virtual/target_driver/ib_srpt
/sys/devices/virtual/target_driver/ib_srpt/uevent
/sys/devices/virtual/target_driver/ib_srpt/subsystem ->
../../../../class/target_driver
/sys/devices/virtual/target_driver/ib_srpt/power
/sys/devices/virtual/target_driver/ib_srpt/power/wakeup
/sys/devices/virtual/target_driver/ib_srpt/power/wakeup_count
/sys/devices/virtual/target_driver/ib_srpt/power/wakeup_active_count
/sys/devices/virtual/target_driver/ib_srpt/power/wakeup_hit_count
/sys/devices/virtual/target_driver/ib_srpt/power/wakeup_active
/sys/devices/virtual/target_driver/ib_srpt/power/wakeup_total_time_ms
/sys/devices/virtual/target_driver/ib_srpt/power/wakeup_max_time_ms
/sys/devices/virtual/target_driver/ib_srpt/power/wakeup_last_time_ms
/sys/devices/virtual/target_driver/ib_srpt/power/runtime_status
/sys/devices/virtual/target_driver/ib_srpt/power/control
/sys/devices/virtual/target_driver/ib_srpt/power/runtime_suspended_time
/sys/devices/virtual/target_driver/ib_srpt/power/runtime_active_time
/sys/devices/virtual/target_driver/ib_srpt/power/autosuspend_delay_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/uevent
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/subsystem
-> ../../../../../class/target_instance
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/device ->
../../ib_srpt
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/wakeup
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/wakeup_count
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/wakeup_active_count
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/wakeup_hit_count
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/wakeup_active
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/wakeup_total_time_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/wakeup_max_time_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/wakeup_last_time_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/runtime_status
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/control
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/runtime_suspended_time
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/runtime_active_time
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/power/autosuspend_delay_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/enabled
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/commands
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/active_commands
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/initiator_name
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/req_lim
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/req_lim_delta
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/luns
-> ../../luns
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun0
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun0/active_commands
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun1
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun1/active_commands
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun2
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun2/active_commands
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun3
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun3/active_commands
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun4
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun4/active_commands
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun5
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun5/active_commands
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/parameters
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/0
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/0/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/0/device
-> ../../../../../target_device/disk01
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/1
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/1/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/1/device
-> ../../../../../target_device/disk02
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/2
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/2/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/2/device
-> ../../../../../target_device/disk03
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/3
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/3/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/3/device
-> ../../../../../target_device/disk04
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/4
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/4/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/4/device
-> ../../../../../target_device/disk05
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/5
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/5/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/luns/5/device
-> ../../../../../target_device/disk06
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/ini_groups
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/rel_tgt_id
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/addr_method
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/io_grouping_type
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/cpu_mask
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_0/login_info
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/uevent
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/subsystem
-> ../../../../../class/target_instance
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/device ->
../../ib_srpt
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/wakeup
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/wakeup_count
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/wakeup_active_count
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/wakeup_hit_count
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/wakeup_active
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/wakeup_total_time_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/wakeup_max_time_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/wakeup_last_time_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/runtime_status
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/control
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/runtime_suspended_time
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/runtime_active_time
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/power/autosuspend_delay_ms
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/enabled
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/sessions
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/parameters
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/0
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/0/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/0/device
-> ../../../../../target_device/disk01
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/1
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/1/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/1/device
-> ../../../../../target_device/disk02
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/2
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/2/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/2/device
-> ../../../../../target_device/disk03
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/3
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/3/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/3/device
-> ../../../../../target_device/disk04
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/4
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/4/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/4/device
-> ../../../../../target_device/disk05
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/5
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/5/read_only
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/luns/5/device
-> ../../../../../target_device/disk06
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/ini_groups
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/rel_tgt_id
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/addr_method
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/io_grouping_type
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/cpu_mask
/sys/devices/virtual/target_driver/ib_srpt/ib_srpt_target_1/login_info
/sys/devices/virtual/target_driver/scst_local
/sys/devices/virtual/target_driver/scst_local/uevent
/sys/devices/virtual/target_driver/scst_local/subsystem ->
../../../../class/target_driver
/sys/devices/virtual/target_driver/scst_local/power
/sys/devices/virtual/target_driver/scst_local/power/wakeup
/sys/devices/virtual/target_driver/scst_local/power/wakeup_count
/sys/devices/virtual/target_driver/scst_local/power/wakeup_active_count
/sys/devices/virtual/target_driver/scst_local/power/wakeup_hit_count
/sys/devices/virtual/target_driver/scst_local/power/wakeup_active
/sys/devices/virtual/target_driver/scst_local/power/wakeup_total_time_ms
/sys/devices/virtual/target_driver/scst_local/power/wakeup_max_time_ms
/sys/devices/virtual/target_driver/scst_local/power/wakeup_last_time_ms
/sys/devices/virtual/target_driver/scst_local/power/runtime_status
/sys/devices/virtual/target_driver/scst_local/power/control
/sys/devices/virtual/target_driver/scst_local/power/runtime_suspended_time
/sys/devices/virtual/target_driver/scst_local/power/runtime_active_time
/sys/devices/virtual/target_driver/scst_local/power/autosuspend_delay_ms
/sys/devices/virtual/target_driver/scst_local/add_target_parameters
/sys/devices/virtual/target_driver/scst_local/version
/sys/devices/virtual/target_driver/scst_local/stats
/sys/devices/virtual/target_driver/scst_local/trace_level
/sys/devices/virtual/target_driver/scst_local/local
/sys/devices/virtual/target_driver/scst_local/local/uevent
/sys/devices/virtual/target_driver/scst_local/local/subsystem ->
../../../../../class/target_instance
/sys/devices/virtual/target_driver/scst_local/local/device -> ../../scst_local
/sys/devices/virtual/target_driver/scst_local/local/power
/sys/devices/virtual/target_driver/scst_local/local/power/wakeup
/sys/devices/virtual/target_driver/scst_local/local/power/wakeup_count
/sys/devices/virtual/target_driver/scst_local/local/power/wakeup_active_count
/sys/devices/virtual/target_driver/scst_local/local/power/wakeup_hit_count
/sys/devices/virtual/target_driver/scst_local/local/power/wakeup_active
/sys/devices/virtual/target_driver/scst_local/local/power/wakeup_total_time_ms
/sys/devices/virtual/target_driver/scst_local/local/power/wakeup_max_time_ms
/sys/devices/virtual/target_driver/scst_local/local/power/wakeup_last_time_ms
/sys/devices/virtual/target_driver/scst_local/local/power/runtime_status
/sys/devices/virtual/target_driver/scst_local/local/power/control
/sys/devices/virtual/target_driver/scst_local/local/power/runtime_suspended_time
/sys/devices/virtual/target_driver/scst_local/local/power/runtime_active_time
/sys/devices/virtual/target_driver/scst_local/local/power/autosuspend_delay_ms
/sys/devices/virtual/target_driver/scst_local/local/sessions
/sys/devices/virtual/target_driver/scst_local/local/sessions/local
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/commands
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/active_commands
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/initiator_name
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/transport_id
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/luns
-> ../../luns
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/host
-> ../../../../../../scst_local/local/host8/scsi_host/host8
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun0
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun0/active_commands
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun1
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun1/active_commands
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun2
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun2/active_commands
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun3
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun3/active_commands
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun4
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun4/active_commands
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun5
/sys/devices/virtual/target_driver/scst_local/local/sessions/local/lun5/active_commands
/sys/devices/virtual/target_driver/scst_local/local/luns
/sys/devices/virtual/target_driver/scst_local/local/luns/parameters
/sys/devices/virtual/target_driver/scst_local/local/luns/0
/sys/devices/virtual/target_driver/scst_local/local/luns/0/read_only
/sys/devices/virtual/target_driver/scst_local/local/luns/0/device ->
../../../../../target_device/disk01
/sys/devices/virtual/target_driver/scst_local/local/luns/1
/sys/devices/virtual/target_driver/scst_local/local/luns/1/read_only
/sys/devices/virtual/target_driver/scst_local/local/luns/1/device ->
../../../../../target_device/disk02
/sys/devices/virtual/target_driver/scst_local/local/luns/2
/sys/devices/virtual/target_driver/scst_local/local/luns/2/read_only
/sys/devices/virtual/target_driver/scst_local/local/luns/2/device ->
../../../../../target_device/disk03
/sys/devices/virtual/target_driver/scst_local/local/luns/3
/sys/devices/virtual/target_driver/scst_local/local/luns/3/read_only
/sys/devices/virtual/target_driver/scst_local/local/luns/3/device ->
../../../../../target_device/disk04
/sys/devices/virtual/target_driver/scst_local/local/luns/4
/sys/devices/virtual/target_driver/scst_local/local/luns/4/read_only
/sys/devices/virtual/target_driver/scst_local/local/luns/4/device ->
../../../../../target_device/disk05
/sys/devices/virtual/target_driver/scst_local/local/luns/5
/sys/devices/virtual/target_driver/scst_local/local/luns/5/read_only
/sys/devices/virtual/target_driver/scst_local/local/luns/5/device ->
../../../../../target_device/disk06
/sys/devices/virtual/target_driver/scst_local/local/ini_groups
/sys/devices/virtual/target_driver/scst_local/local/rel_tgt_id
/sys/devices/virtual/target_driver/scst_local/local/addr_method
/sys/devices/virtual/target_driver/scst_local/local/io_grouping_type
/sys/devices/virtual/target_driver/scst_local/local/cpu_mask
/sys/devices/virtual/target_driver/scst_local/local/scsi_transport_version
/sys/devices/virtual/target_driver/scst_local/local/phys_transport_version
/sys/devices/virtual/device_driver
/sys/devices/virtual/device_driver/dev_disk_perf
/sys/devices/virtual/device_driver/dev_disk_perf/uevent
/sys/devices/virtual/device_driver/dev_disk_perf/subsystem ->
../../../../class/device_driver
/sys/devices/virtual/device_driver/dev_disk_perf/type
/sys/devices/virtual/device_driver/dev_disk_perf/power
/sys/devices/virtual/device_driver/dev_disk_perf/power/wakeup
/sys/devices/virtual/device_driver/dev_disk_perf/power/wakeup_count
/sys/devices/virtual/device_driver/dev_disk_perf/power/wakeup_active_count
/sys/devices/virtual/device_driver/dev_disk_perf/power/wakeup_hit_count
/sys/devices/virtual/device_driver/dev_disk_perf/power/wakeup_active
/sys/devices/virtual/device_driver/dev_disk_perf/power/wakeup_total_time_ms
/sys/devices/virtual/device_driver/dev_disk_perf/power/wakeup_max_time_ms
/sys/devices/virtual/device_driver/dev_disk_perf/power/wakeup_last_time_ms
/sys/devices/virtual/device_driver/dev_disk_perf/power/runtime_status
/sys/devices/virtual/device_driver/dev_disk_perf/power/control
/sys/devices/virtual/device_driver/dev_disk_perf/power/runtime_suspended_time
/sys/devices/virtual/device_driver/dev_disk_perf/power/runtime_active_time
/sys/devices/virtual/device_driver/dev_disk_perf/power/autosuspend_delay_ms
/sys/devices/virtual/device_driver/dev_disk_perf/trace_level
/sys/devices/virtual/device_driver/dev_disk
/sys/devices/virtual/device_driver/dev_disk/trace_level
/sys/devices/virtual/device_driver/dev_disk/uevent
/sys/devices/virtual/device_driver/dev_disk/subsystem ->
../../../../class/device_driver
/sys/devices/virtual/device_driver/dev_disk/type
/sys/devices/virtual/device_driver/dev_disk/power
/sys/devices/virtual/device_driver/dev_disk/power/wakeup
/sys/devices/virtual/device_driver/dev_disk/power/wakeup_count
/sys/devices/virtual/device_driver/dev_disk/power/wakeup_active_count
/sys/devices/virtual/device_driver/dev_disk/power/wakeup_hit_count
/sys/devices/virtual/device_driver/dev_disk/power/wakeup_active
/sys/devices/virtual/device_driver/dev_disk/power/wakeup_total_time_ms
/sys/devices/virtual/device_driver/dev_disk/power/wakeup_max_time_ms
/sys/devices/virtual/device_driver/dev_disk/power/wakeup_last_time_ms
/sys/devices/virtual/device_driver/dev_disk/power/runtime_status
/sys/devices/virtual/device_driver/dev_disk/power/control
/sys/devices/virtual/device_driver/dev_disk/power/runtime_suspended_time
/sys/devices/virtual/device_driver/dev_disk/power/runtime_active_time
/sys/devices/virtual/device_driver/dev_disk/power/autosuspend_delay_ms
/sys/devices/virtual/device_driver/vdisk_fileio
/sys/devices/virtual/device_driver/vdisk_fileio/uevent
/sys/devices/virtual/device_driver/vdisk_fileio/subsystem ->
../../../../class/device_driver
/sys/devices/virtual/device_driver/vdisk_fileio/type
/sys/devices/virtual/device_driver/vdisk_fileio/power
/sys/devices/virtual/device_driver/vdisk_fileio/power/wakeup
/sys/devices/virtual/device_driver/vdisk_fileio/power/wakeup_count
/sys/devices/virtual/device_driver/vdisk_fileio/power/wakeup_active_count
/sys/devices/virtual/device_driver/vdisk_fileio/power/wakeup_hit_count
/sys/devices/virtual/device_driver/vdisk_fileio/power/wakeup_active
/sys/devices/virtual/device_driver/vdisk_fileio/power/wakeup_total_time_ms
/sys/devices/virtual/device_driver/vdisk_fileio/power/wakeup_max_time_ms
/sys/devices/virtual/device_driver/vdisk_fileio/power/wakeup_last_time_ms
/sys/devices/virtual/device_driver/vdisk_fileio/power/runtime_status
/sys/devices/virtual/device_driver/vdisk_fileio/power/control
/sys/devices/virtual/device_driver/vdisk_fileio/power/runtime_suspended_time
/sys/devices/virtual/device_driver/vdisk_fileio/power/runtime_active_time
/sys/devices/virtual/device_driver/vdisk_fileio/power/autosuspend_delay_ms
/sys/devices/virtual/device_driver/vdisk_fileio/add_device_parameters
/sys/devices/virtual/device_driver/vdisk_fileio/trace_level
/sys/devices/virtual/device_driver/vdisk_fileio/disk03 ->
../../target_device/disk03
/sys/devices/virtual/device_driver/vdisk_fileio/disk04 ->
../../target_device/disk04
/sys/devices/virtual/device_driver/vdisk_blockio
/sys/devices/virtual/device_driver/vdisk_blockio/uevent
/sys/devices/virtual/device_driver/vdisk_blockio/subsystem ->
../../../../class/device_driver
/sys/devices/virtual/device_driver/vdisk_blockio/type
/sys/devices/virtual/device_driver/vdisk_blockio/power
/sys/devices/virtual/device_driver/vdisk_blockio/power/wakeup
/sys/devices/virtual/device_driver/vdisk_blockio/power/wakeup_count
/sys/devices/virtual/device_driver/vdisk_blockio/power/wakeup_active_count
/sys/devices/virtual/device_driver/vdisk_blockio/power/wakeup_hit_count
/sys/devices/virtual/device_driver/vdisk_blockio/power/wakeup_active
/sys/devices/virtual/device_driver/vdisk_blockio/power/wakeup_total_time_ms
/sys/devices/virtual/device_driver/vdisk_blockio/power/wakeup_max_time_ms
/sys/devices/virtual/device_driver/vdisk_blockio/power/wakeup_last_time_ms
/sys/devices/virtual/device_driver/vdisk_blockio/power/runtime_status
/sys/devices/virtual/device_driver/vdisk_blockio/power/control
/sys/devices/virtual/device_driver/vdisk_blockio/power/runtime_suspended_time
/sys/devices/virtual/device_driver/vdisk_blockio/power/runtime_active_time
/sys/devices/virtual/device_driver/vdisk_blockio/power/autosuspend_delay_ms
/sys/devices/virtual/device_driver/vdisk_blockio/add_device_parameters
/sys/devices/virtual/device_driver/vdisk_blockio/trace_level
/sys/devices/virtual/device_driver/vdisk_blockio/disk01 ->
../../target_device/disk01
/sys/devices/virtual/device_driver/vdisk_blockio/disk02 ->
../../target_device/disk02
/sys/devices/virtual/device_driver/vdisk_nullio
/sys/devices/virtual/device_driver/vdisk_nullio/uevent
/sys/devices/virtual/device_driver/vdisk_nullio/subsystem ->
../../../../class/device_driver
/sys/devices/virtual/device_driver/vdisk_nullio/type
/sys/devices/virtual/device_driver/vdisk_nullio/power
/sys/devices/virtual/device_driver/vdisk_nullio/power/wakeup
/sys/devices/virtual/device_driver/vdisk_nullio/power/wakeup_count
/sys/devices/virtual/device_driver/vdisk_nullio/power/wakeup_active_count
/sys/devices/virtual/device_driver/vdisk_nullio/power/wakeup_hit_count
/sys/devices/virtual/device_driver/vdisk_nullio/power/wakeup_active
/sys/devices/virtual/device_driver/vdisk_nullio/power/wakeup_total_time_ms
/sys/devices/virtual/device_driver/vdisk_nullio/power/wakeup_max_time_ms
/sys/devices/virtual/device_driver/vdisk_nullio/power/wakeup_last_time_ms
/sys/devices/virtual/device_driver/vdisk_nullio/power/runtime_status
/sys/devices/virtual/device_driver/vdisk_nullio/power/control
/sys/devices/virtual/device_driver/vdisk_nullio/power/runtime_suspended_time
/sys/devices/virtual/device_driver/vdisk_nullio/power/runtime_active_time
/sys/devices/virtual/device_driver/vdisk_nullio/power/autosuspend_delay_ms
/sys/devices/virtual/device_driver/vdisk_nullio/add_device_parameters
/sys/devices/virtual/device_driver/vdisk_nullio/trace_level
/sys/devices/virtual/device_driver/vdisk_nullio/disk05 ->
../../target_device/disk05
/sys/devices/virtual/device_driver/vdisk_nullio/disk06 ->
../../target_device/disk06
/sys/devices/virtual/device_driver/vcdrom
/sys/devices/virtual/device_driver/vcdrom/uevent
/sys/devices/virtual/device_driver/vcdrom/subsystem ->
../../../../class/device_driver
/sys/devices/virtual/device_driver/vcdrom/type
/sys/devices/virtual/device_driver/vcdrom/power
/sys/devices/virtual/device_driver/vcdrom/power/wakeup
/sys/devices/virtual/device_driver/vcdrom/power/wakeup_count
/sys/devices/virtual/device_driver/vcdrom/power/wakeup_active_count
/sys/devices/virtual/device_driver/vcdrom/power/wakeup_hit_count
/sys/devices/virtual/device_driver/vcdrom/power/wakeup_active
/sys/devices/virtual/device_driver/vcdrom/power/wakeup_total_time_ms
/sys/devices/virtual/device_driver/vcdrom/power/wakeup_max_time_ms
/sys/devices/virtual/device_driver/vcdrom/power/wakeup_last_time_ms
/sys/devices/virtual/device_driver/vcdrom/power/runtime_status
/sys/devices/virtual/device_driver/vcdrom/power/control
/sys/devices/virtual/device_driver/vcdrom/power/runtime_suspended_time
/sys/devices/virtual/device_driver/vcdrom/power/runtime_active_time
/sys/devices/virtual/device_driver/vcdrom/power/autosuspend_delay_ms
/sys/devices/virtual/device_driver/vcdrom/trace_level
/sys/devices/virtual/target_device
/sys/devices/virtual/target_device/disk01
/sys/devices/virtual/target_device/disk01/uevent
/sys/devices/virtual/target_device/disk01/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/disk01/type
/sys/devices/virtual/target_device/disk01/power
/sys/devices/virtual/target_device/disk01/power/wakeup
/sys/devices/virtual/target_device/disk01/power/wakeup_count
/sys/devices/virtual/target_device/disk01/power/wakeup_active_count
/sys/devices/virtual/target_device/disk01/power/wakeup_hit_count
/sys/devices/virtual/target_device/disk01/power/wakeup_active
/sys/devices/virtual/target_device/disk01/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/disk01/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/disk01/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/disk01/power/runtime_status
/sys/devices/virtual/target_device/disk01/power/control
/sys/devices/virtual/target_device/disk01/power/runtime_suspended_time
/sys/devices/virtual/target_device/disk01/power/runtime_active_time
/sys/devices/virtual/target_device/disk01/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/disk01/exported
/sys/devices/virtual/target_device/disk01/exported/export0 ->
../../../target_driver/ib_srpt/ib_srpt_target_0/luns/0
/sys/devices/virtual/target_device/disk01/exported/export1 ->
../../../target_driver/ib_srpt/ib_srpt_target_1/luns/0
/sys/devices/virtual/target_device/disk01/exported/export2 ->
../../../target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/0
/sys/devices/virtual/target_device/disk01/exported/export3 ->
../../../target_driver/scst_local/local/luns/0
/sys/devices/virtual/target_device/disk01/dump_prs
/sys/devices/virtual/target_device/disk01/handler ->
../../device_driver/vdisk_blockio
/sys/devices/virtual/target_device/disk01/threads_num
/sys/devices/virtual/target_device/disk01/threads_pool_type
/sys/devices/virtual/target_device/disk01/size_mb
/sys/devices/virtual/target_device/disk01/blocksize
/sys/devices/virtual/target_device/disk01/read_only
/sys/devices/virtual/target_device/disk01/nv_cache
/sys/devices/virtual/target_device/disk01/removable
/sys/devices/virtual/target_device/disk01/filename
/sys/devices/virtual/target_device/disk01/resync_size
/sys/devices/virtual/target_device/disk01/t10_dev_id
/sys/devices/virtual/target_device/disk01/usn
/sys/devices/virtual/target_device/disk01/thin_provisioned
/sys/devices/virtual/target_device/2:0:0:0
/sys/devices/virtual/target_device/2:0:0:0/exported
/sys/devices/virtual/target_device/2:0:0:0/scsi_device ->
../../../pci0000:00/0000:00:1f.2/host2/target2:0:0/2:0:0:0/scsi_device/2:0:0:0
/sys/devices/virtual/target_device/2:0:0:0/uevent
/sys/devices/virtual/target_device/2:0:0:0/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/2:0:0:0/type
/sys/devices/virtual/target_device/2:0:0:0/power
/sys/devices/virtual/target_device/2:0:0:0/power/wakeup
/sys/devices/virtual/target_device/2:0:0:0/power/wakeup_count
/sys/devices/virtual/target_device/2:0:0:0/power/wakeup_active_count
/sys/devices/virtual/target_device/2:0:0:0/power/wakeup_hit_count
/sys/devices/virtual/target_device/2:0:0:0/power/wakeup_active
/sys/devices/virtual/target_device/2:0:0:0/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/2:0:0:0/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/2:0:0:0/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/2:0:0:0/power/runtime_status
/sys/devices/virtual/target_device/2:0:0:0/power/control
/sys/devices/virtual/target_device/2:0:0:0/power/runtime_suspended_time
/sys/devices/virtual/target_device/2:0:0:0/power/runtime_active_time
/sys/devices/virtual/target_device/2:0:0:0/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/2:0:1:0
/sys/devices/virtual/target_device/2:0:1:0/uevent
/sys/devices/virtual/target_device/2:0:1:0/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/2:0:1:0/type
/sys/devices/virtual/target_device/2:0:1:0/power
/sys/devices/virtual/target_device/2:0:1:0/power/wakeup
/sys/devices/virtual/target_device/2:0:1:0/power/wakeup_count
/sys/devices/virtual/target_device/2:0:1:0/power/wakeup_active_count
/sys/devices/virtual/target_device/2:0:1:0/power/wakeup_hit_count
/sys/devices/virtual/target_device/2:0:1:0/power/wakeup_active
/sys/devices/virtual/target_device/2:0:1:0/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/2:0:1:0/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/2:0:1:0/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/2:0:1:0/power/runtime_status
/sys/devices/virtual/target_device/2:0:1:0/power/control
/sys/devices/virtual/target_device/2:0:1:0/power/runtime_suspended_time
/sys/devices/virtual/target_device/2:0:1:0/power/runtime_active_time
/sys/devices/virtual/target_device/2:0:1:0/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/2:0:1:0/exported
/sys/devices/virtual/target_device/2:0:1:0/scsi_device ->
../../../pci0000:00/0000:00:1f.2/host2/target2:0:1/2:0:1:0/scsi_device/2:0:1:0
/sys/devices/virtual/target_device/3:0:0:0
/sys/devices/virtual/target_device/3:0:0:0/uevent
/sys/devices/virtual/target_device/3:0:0:0/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/3:0:0:0/type
/sys/devices/virtual/target_device/3:0:0:0/power
/sys/devices/virtual/target_device/3:0:0:0/power/wakeup
/sys/devices/virtual/target_device/3:0:0:0/power/wakeup_count
/sys/devices/virtual/target_device/3:0:0:0/power/wakeup_active_count
/sys/devices/virtual/target_device/3:0:0:0/power/wakeup_hit_count
/sys/devices/virtual/target_device/3:0:0:0/power/wakeup_active
/sys/devices/virtual/target_device/3:0:0:0/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/3:0:0:0/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/3:0:0:0/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/3:0:0:0/power/runtime_status
/sys/devices/virtual/target_device/3:0:0:0/power/control
/sys/devices/virtual/target_device/3:0:0:0/power/runtime_suspended_time
/sys/devices/virtual/target_device/3:0:0:0/power/runtime_active_time
/sys/devices/virtual/target_device/3:0:0:0/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/3:0:0:0/exported
/sys/devices/virtual/target_device/3:0:0:0/scsi_device ->
../../../pci0000:00/0000:00:1f.2/host3/target3:0:0/3:0:0:0/scsi_device/3:0:0:0
/sys/devices/virtual/target_device/disk02
/sys/devices/virtual/target_device/disk02/uevent
/sys/devices/virtual/target_device/disk02/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/disk02/type
/sys/devices/virtual/target_device/disk02/power
/sys/devices/virtual/target_device/disk02/power/wakeup
/sys/devices/virtual/target_device/disk02/power/wakeup_count
/sys/devices/virtual/target_device/disk02/power/wakeup_active_count
/sys/devices/virtual/target_device/disk02/power/wakeup_hit_count
/sys/devices/virtual/target_device/disk02/power/wakeup_active
/sys/devices/virtual/target_device/disk02/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/disk02/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/disk02/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/disk02/power/runtime_status
/sys/devices/virtual/target_device/disk02/power/control
/sys/devices/virtual/target_device/disk02/power/runtime_suspended_time
/sys/devices/virtual/target_device/disk02/power/runtime_active_time
/sys/devices/virtual/target_device/disk02/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/disk02/exported
/sys/devices/virtual/target_device/disk02/exported/export0 ->
../../../target_driver/ib_srpt/ib_srpt_target_0/luns/1
/sys/devices/virtual/target_device/disk02/exported/export1 ->
../../../target_driver/ib_srpt/ib_srpt_target_1/luns/1
/sys/devices/virtual/target_device/disk02/exported/export2 ->
../../../target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/1
/sys/devices/virtual/target_device/disk02/exported/export3 ->
../../../target_driver/scst_local/local/luns/1
/sys/devices/virtual/target_device/disk02/dump_prs
/sys/devices/virtual/target_device/disk02/handler ->
../../device_driver/vdisk_blockio
/sys/devices/virtual/target_device/disk02/threads_num
/sys/devices/virtual/target_device/disk02/threads_pool_type
/sys/devices/virtual/target_device/disk02/size_mb
/sys/devices/virtual/target_device/disk02/blocksize
/sys/devices/virtual/target_device/disk02/read_only
/sys/devices/virtual/target_device/disk02/nv_cache
/sys/devices/virtual/target_device/disk02/removable
/sys/devices/virtual/target_device/disk02/filename
/sys/devices/virtual/target_device/disk02/resync_size
/sys/devices/virtual/target_device/disk02/t10_dev_id
/sys/devices/virtual/target_device/disk02/usn
/sys/devices/virtual/target_device/disk02/thin_provisioned
/sys/devices/virtual/target_device/disk03
/sys/devices/virtual/target_device/disk03/uevent
/sys/devices/virtual/target_device/disk03/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/disk03/type
/sys/devices/virtual/target_device/disk03/power
/sys/devices/virtual/target_device/disk03/power/wakeup
/sys/devices/virtual/target_device/disk03/power/wakeup_count
/sys/devices/virtual/target_device/disk03/power/wakeup_active_count
/sys/devices/virtual/target_device/disk03/power/wakeup_hit_count
/sys/devices/virtual/target_device/disk03/power/wakeup_active
/sys/devices/virtual/target_device/disk03/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/disk03/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/disk03/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/disk03/power/runtime_status
/sys/devices/virtual/target_device/disk03/power/control
/sys/devices/virtual/target_device/disk03/power/runtime_suspended_time
/sys/devices/virtual/target_device/disk03/power/runtime_active_time
/sys/devices/virtual/target_device/disk03/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/disk03/exported
/sys/devices/virtual/target_device/disk03/exported/export0 ->
../../../target_driver/ib_srpt/ib_srpt_target_0/luns/2
/sys/devices/virtual/target_device/disk03/exported/export1 ->
../../../target_driver/ib_srpt/ib_srpt_target_1/luns/2
/sys/devices/virtual/target_device/disk03/exported/export2 ->
../../../target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/2
/sys/devices/virtual/target_device/disk03/exported/export3 ->
../../../target_driver/scst_local/local/luns/2
/sys/devices/virtual/target_device/disk03/dump_prs
/sys/devices/virtual/target_device/disk03/handler ->
../../device_driver/vdisk_fileio
/sys/devices/virtual/target_device/disk03/threads_num
/sys/devices/virtual/target_device/disk03/threads_pool_type
/sys/devices/virtual/target_device/disk03/size_mb
/sys/devices/virtual/target_device/disk03/blocksize
/sys/devices/virtual/target_device/disk03/read_only
/sys/devices/virtual/target_device/disk03/write_through
/sys/devices/virtual/target_device/disk03/thin_provisioned
/sys/devices/virtual/target_device/disk03/nv_cache
/sys/devices/virtual/target_device/disk03/o_direct
/sys/devices/virtual/target_device/disk03/removable
/sys/devices/virtual/target_device/disk03/filename
/sys/devices/virtual/target_device/disk03/resync_size
/sys/devices/virtual/target_device/disk03/t10_dev_id
/sys/devices/virtual/target_device/disk03/usn
/sys/devices/virtual/target_device/disk04
/sys/devices/virtual/target_device/disk04/uevent
/sys/devices/virtual/target_device/disk04/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/disk04/type
/sys/devices/virtual/target_device/disk04/power
/sys/devices/virtual/target_device/disk04/power/wakeup
/sys/devices/virtual/target_device/disk04/power/wakeup_count
/sys/devices/virtual/target_device/disk04/power/wakeup_active_count
/sys/devices/virtual/target_device/disk04/power/wakeup_hit_count
/sys/devices/virtual/target_device/disk04/power/wakeup_active
/sys/devices/virtual/target_device/disk04/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/disk04/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/disk04/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/disk04/power/runtime_status
/sys/devices/virtual/target_device/disk04/power/control
/sys/devices/virtual/target_device/disk04/power/runtime_suspended_time
/sys/devices/virtual/target_device/disk04/power/runtime_active_time
/sys/devices/virtual/target_device/disk04/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/disk04/exported
/sys/devices/virtual/target_device/disk04/exported/export0 ->
../../../target_driver/ib_srpt/ib_srpt_target_0/luns/3
/sys/devices/virtual/target_device/disk04/exported/export1 ->
../../../target_driver/ib_srpt/ib_srpt_target_1/luns/3
/sys/devices/virtual/target_device/disk04/exported/export2 ->
../../../target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/3
/sys/devices/virtual/target_device/disk04/exported/export3 ->
../../../target_driver/scst_local/local/luns/3
/sys/devices/virtual/target_device/disk04/dump_prs
/sys/devices/virtual/target_device/disk04/handler ->
../../device_driver/vdisk_fileio
/sys/devices/virtual/target_device/disk04/threads_num
/sys/devices/virtual/target_device/disk04/threads_pool_type
/sys/devices/virtual/target_device/disk04/size_mb
/sys/devices/virtual/target_device/disk04/blocksize
/sys/devices/virtual/target_device/disk04/read_only
/sys/devices/virtual/target_device/disk04/write_through
/sys/devices/virtual/target_device/disk04/thin_provisioned
/sys/devices/virtual/target_device/disk04/nv_cache
/sys/devices/virtual/target_device/disk04/o_direct
/sys/devices/virtual/target_device/disk04/removable
/sys/devices/virtual/target_device/disk04/filename
/sys/devices/virtual/target_device/disk04/resync_size
/sys/devices/virtual/target_device/disk04/t10_dev_id
/sys/devices/virtual/target_device/disk04/usn
/sys/devices/virtual/target_device/disk05
/sys/devices/virtual/target_device/disk05/uevent
/sys/devices/virtual/target_device/disk05/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/disk05/type
/sys/devices/virtual/target_device/disk05/power
/sys/devices/virtual/target_device/disk05/power/wakeup
/sys/devices/virtual/target_device/disk05/power/wakeup_count
/sys/devices/virtual/target_device/disk05/power/wakeup_active_count
/sys/devices/virtual/target_device/disk05/power/wakeup_hit_count
/sys/devices/virtual/target_device/disk05/power/wakeup_active
/sys/devices/virtual/target_device/disk05/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/disk05/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/disk05/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/disk05/power/runtime_status
/sys/devices/virtual/target_device/disk05/power/control
/sys/devices/virtual/target_device/disk05/power/runtime_suspended_time
/sys/devices/virtual/target_device/disk05/power/runtime_active_time
/sys/devices/virtual/target_device/disk05/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/disk05/exported
/sys/devices/virtual/target_device/disk05/exported/export0 ->
../../../target_driver/ib_srpt/ib_srpt_target_0/luns/4
/sys/devices/virtual/target_device/disk05/exported/export1 ->
../../../target_driver/ib_srpt/ib_srpt_target_1/luns/4
/sys/devices/virtual/target_device/disk05/exported/export2 ->
../../../target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/4
/sys/devices/virtual/target_device/disk05/exported/export3 ->
../../../target_driver/scst_local/local/luns/4
/sys/devices/virtual/target_device/disk05/dump_prs
/sys/devices/virtual/target_device/disk05/handler ->
../../device_driver/vdisk_nullio
/sys/devices/virtual/target_device/disk05/threads_num
/sys/devices/virtual/target_device/disk05/threads_pool_type
/sys/devices/virtual/target_device/disk05/size_mb
/sys/devices/virtual/target_device/disk05/blocksize
/sys/devices/virtual/target_device/disk05/read_only
/sys/devices/virtual/target_device/disk05/removable
/sys/devices/virtual/target_device/disk05/t10_dev_id
/sys/devices/virtual/target_device/disk05/usn
/sys/devices/virtual/target_device/disk06
/sys/devices/virtual/target_device/disk06/uevent
/sys/devices/virtual/target_device/disk06/subsystem ->
../../../../class/target_device
/sys/devices/virtual/target_device/disk06/type
/sys/devices/virtual/target_device/disk06/power
/sys/devices/virtual/target_device/disk06/power/wakeup
/sys/devices/virtual/target_device/disk06/power/wakeup_count
/sys/devices/virtual/target_device/disk06/power/wakeup_active_count
/sys/devices/virtual/target_device/disk06/power/wakeup_hit_count
/sys/devices/virtual/target_device/disk06/power/wakeup_active
/sys/devices/virtual/target_device/disk06/power/wakeup_total_time_ms
/sys/devices/virtual/target_device/disk06/power/wakeup_max_time_ms
/sys/devices/virtual/target_device/disk06/power/wakeup_last_time_ms
/sys/devices/virtual/target_device/disk06/power/runtime_status
/sys/devices/virtual/target_device/disk06/power/control
/sys/devices/virtual/target_device/disk06/power/runtime_suspended_time
/sys/devices/virtual/target_device/disk06/power/runtime_active_time
/sys/devices/virtual/target_device/disk06/power/autosuspend_delay_ms
/sys/devices/virtual/target_device/disk06/exported
/sys/devices/virtual/target_device/disk06/exported/export0 ->
../../../target_driver/ib_srpt/ib_srpt_target_0/luns/5
/sys/devices/virtual/target_device/disk06/exported/export1 ->
../../../target_driver/ib_srpt/ib_srpt_target_1/luns/5
/sys/devices/virtual/target_device/disk06/exported/export2 ->
../../../target_driver/iscsi/iqn.2005-03.org.open-iscsi:dbc01e1792b:storage/luns/5
/sys/devices/virtual/target_device/disk06/exported/export3 ->
../../../target_driver/scst_local/local/luns/5
/sys/devices/virtual/target_device/disk06/dump_prs
/sys/devices/virtual/target_device/disk06/handler ->
../../device_driver/vdisk_nullio
/sys/devices/virtual/target_device/disk06/threads_num
/sys/devices/virtual/target_device/disk06/threads_pool_type
/sys/devices/virtual/target_device/disk06/size_mb
/sys/devices/virtual/target_device/disk06/blocksize
/sys/devices/virtual/target_device/disk06/read_only
/sys/devices/virtual/target_device/disk06/removable
/sys/devices/virtual/target_device/disk06/t10_dev_id
/sys/devices/virtual/target_device/disk06/usn
/sys/devices/scst
/sys/devices/scst/uevent
/sys/devices/scst/power
/sys/devices/scst/power/control
/sys/devices/scst/power/wakeup
/sys/devices/scst/power/wakeup_count
/sys/devices/scst/power/wakeup_active_count
/sys/devices/scst/power/wakeup_hit_count
/sys/devices/scst/power/wakeup_active
/sys/devices/scst/power/wakeup_total_time_ms
/sys/devices/scst/power/wakeup_max_time_ms
/sys/devices/scst/power/wakeup_last_time_ms
/sys/devices/scst/power/runtime_status
/sys/devices/scst/power/runtime_suspended_time
/sys/devices/scst/power/runtime_active_time
/sys/devices/scst/power/autosuspend_delay_ms
/sys/devices/scst/mgmt
/sys/devices/scst/threads
/sys/devices/scst/setup_id
/sys/devices/scst/trace_level
/sys/devices/scst/version
/sys/devices/scst/sgv
/sys/devices/scst/sgv/global_stats
/sys/devices/scst/sgv/sgv-dma
/sys/devices/scst/sgv/sgv-dma/stats
/sys/devices/scst/sgv/sgv
/sys/devices/scst/sgv/sgv/stats
/sys/devices/scst/sgv/sgv-clust
/sys/devices/scst/sgv/sgv-clust/stats
On Fri, Dec 10, 2010 at 01:06:06PM +0100, Bart Van Assche wrote:
> On Fri, Nov 19, 2010 at 10:19 PM, Greg KH <[email protected]> wrote:
> > [ ... ]
> >
> > None. ?Use 'struct device'
>
> How about using 'struct device' as follows ?
> * Move /sys/kernel/scst_tgt/targets to /sys/class/target_driver.
Please never create a new class, use 'struct bus_type' instead and
create a bus and have drivers and devices on it.
thanks,
greg k-h
On Fri, Dec 10, 2010 at 8:36 PM, Greg KH <[email protected]> wrote:
>
> On Fri, Dec 10, 2010 at 01:06:06PM +0100, Bart Van Assche wrote:
> > On Fri, Nov 19, 2010 at 10:19 PM, Greg KH <[email protected]> wrote:
> > > [ ... ]
> > >
> > > None. ?Use 'struct device'
> >
> > How about using 'struct device' as follows ?
> > * Move /sys/kernel/scst_tgt/targets to /sys/class/target_driver.
>
> Please never create a new class, use 'struct bus_type' instead and
> create a bus and have drivers and devices on it.
How about the hierarchy illustrated below, using the two new bus types
scsi_tgt_dev and scsi_tgt_port ? Note: this time I neither have loaded
the scst_local kernel module nor iscsi-scst in order to keep the
output short.
# find /sys/bus/scsi_tgt_* /sys/devices/scst | while read f; do if [
-h $f ]; then echo "$f -> $(readlink $f)"; else echo $f; fi; done
/sys/bus/scsi_tgt_dev
/sys/bus/scsi_tgt_dev/uevent
/sys/bus/scsi_tgt_dev/devices
/sys/bus/scsi_tgt_dev/devices/2:0:0:0 -> ../../../devices/2:0:0:0
/sys/bus/scsi_tgt_dev/devices/2:0:1:0 -> ../../../devices/2:0:1:0
/sys/bus/scsi_tgt_dev/devices/3:0:0:0 -> ../../../devices/3:0:0:0
/sys/bus/scsi_tgt_dev/devices/disk01 -> ../../../devices/disk01
/sys/bus/scsi_tgt_dev/devices/disk02 -> ../../../devices/disk02
/sys/bus/scsi_tgt_dev/devices/disk03 -> ../../../devices/disk03
/sys/bus/scsi_tgt_dev/devices/disk04 -> ../../../devices/disk04
/sys/bus/scsi_tgt_dev/devices/disk05 -> ../../../devices/disk05
/sys/bus/scsi_tgt_dev/devices/disk06 -> ../../../devices/disk06
/sys/bus/scsi_tgt_dev/drivers
/sys/bus/scsi_tgt_dev/drivers/dev_disk
/sys/bus/scsi_tgt_dev/drivers/dev_disk/module -> ../../../../module/scst_disk
/sys/bus/scsi_tgt_dev/drivers/dev_disk/uevent
/sys/bus/scsi_tgt_dev/drivers/dev_disk/type
/sys/bus/scsi_tgt_dev/drivers/dev_disk/trace_level
/sys/bus/scsi_tgt_dev/drivers/dev_disk_perf
/sys/bus/scsi_tgt_dev/drivers/dev_disk_perf/module ->
../../../../module/scst_disk
/sys/bus/scsi_tgt_dev/drivers/dev_disk_perf/uevent
/sys/bus/scsi_tgt_dev/drivers/dev_disk_perf/type
/sys/bus/scsi_tgt_dev/drivers/dev_disk_perf/trace_level
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/module ->
../../../../module/scst_vdisk
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/uevent
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/type
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/add_device_parameters
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/trace_level
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/disk01 -> ../../../../devices/disk01
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/disk02 -> ../../../../devices/disk02
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/disk03 -> ../../../../devices/disk03
/sys/bus/scsi_tgt_dev/drivers/vdisk_fileio/disk04 -> ../../../../devices/disk04
/sys/bus/scsi_tgt_dev/drivers/vdisk_blockio
/sys/bus/scsi_tgt_dev/drivers/vdisk_blockio/module ->
../../../../module/scst_vdisk
/sys/bus/scsi_tgt_dev/drivers/vdisk_blockio/uevent
/sys/bus/scsi_tgt_dev/drivers/vdisk_blockio/type
/sys/bus/scsi_tgt_dev/drivers/vdisk_blockio/add_device_parameters
/sys/bus/scsi_tgt_dev/drivers/vdisk_blockio/trace_level
/sys/bus/scsi_tgt_dev/drivers/vdisk_nullio
/sys/bus/scsi_tgt_dev/drivers/vdisk_nullio/module ->
../../../../module/scst_vdisk
/sys/bus/scsi_tgt_dev/drivers/vdisk_nullio/uevent
/sys/bus/scsi_tgt_dev/drivers/vdisk_nullio/type
/sys/bus/scsi_tgt_dev/drivers/vdisk_nullio/add_device_parameters
/sys/bus/scsi_tgt_dev/drivers/vdisk_nullio/trace_level
/sys/bus/scsi_tgt_dev/drivers/vdisk_nullio/disk05 -> ../../../../devices/disk05
/sys/bus/scsi_tgt_dev/drivers/vdisk_nullio/disk06 -> ../../../../devices/disk06
/sys/bus/scsi_tgt_dev/drivers/vcdrom
/sys/bus/scsi_tgt_dev/drivers/vcdrom/module -> ../../../../module/scst_vdisk
/sys/bus/scsi_tgt_dev/drivers/vcdrom/uevent
/sys/bus/scsi_tgt_dev/drivers/vcdrom/type
/sys/bus/scsi_tgt_dev/drivers/vcdrom/trace_level
/sys/bus/scsi_tgt_dev/drivers_probe
/sys/bus/scsi_tgt_dev/drivers_autoprobe
/sys/bus/scsi_tgt_port
/sys/bus/scsi_tgt_port/uevent
/sys/bus/scsi_tgt_port/devices
/sys/bus/scsi_tgt_port/devices/ib_srpt_target_0 ->
../../../devices/ib_srpt_target_0
/sys/bus/scsi_tgt_port/devices/ib_srpt_target_1 ->
../../../devices/ib_srpt_target_1
/sys/bus/scsi_tgt_port/drivers
/sys/bus/scsi_tgt_port/drivers/ib_srpt
/sys/bus/scsi_tgt_port/drivers/ib_srpt/module -> ../../../../module/ib_srpt
/sys/bus/scsi_tgt_port/drivers/ib_srpt/uevent
/sys/bus/scsi_tgt_port/drivers/ib_srpt/add_target
/sys/bus/scsi_tgt_port/drivers/ib_srpt/ib_srpt_target_0 ->
../../../../devices/ib_srpt_target_0
/sys/bus/scsi_tgt_port/drivers/ib_srpt/ib_srpt_target_1 ->
../../../../devices/ib_srpt_target_1
/sys/bus/scsi_tgt_port/drivers_probe
/sys/bus/scsi_tgt_port/drivers_autoprobe
/sys/devices/2:0:0:0
/sys/devices/2:0:0:0/uevent
/sys/devices/2:0:0:0/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/2:0:0:0/power
/sys/devices/2:0:0:0/power/wakeup
/sys/devices/2:0:0:0/power/wakeup_count
/sys/devices/2:0:0:0/power/wakeup_active_count
/sys/devices/2:0:0:0/power/wakeup_hit_count
/sys/devices/2:0:0:0/power/wakeup_active
/sys/devices/2:0:0:0/power/wakeup_total_time_ms
/sys/devices/2:0:0:0/power/wakeup_max_time_ms
/sys/devices/2:0:0:0/power/wakeup_last_time_ms
/sys/devices/2:0:0:0/power/runtime_status
/sys/devices/2:0:0:0/power/control
/sys/devices/2:0:0:0/power/runtime_suspended_time
/sys/devices/2:0:0:0/power/runtime_active_time
/sys/devices/2:0:0:0/power/autosuspend_delay_ms
/sys/devices/2:0:0:0/scsi_device
/sys/devices/2:0:1:0
/sys/devices/2:0:1:0/uevent
/sys/devices/2:0:1:0/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/2:0:1:0/power
/sys/devices/2:0:1:0/power/wakeup
/sys/devices/2:0:1:0/power/wakeup_count
/sys/devices/2:0:1:0/power/wakeup_active_count
/sys/devices/2:0:1:0/power/wakeup_hit_count
/sys/devices/2:0:1:0/power/wakeup_active
/sys/devices/2:0:1:0/power/wakeup_total_time_ms
/sys/devices/2:0:1:0/power/wakeup_max_time_ms
/sys/devices/2:0:1:0/power/wakeup_last_time_ms
/sys/devices/2:0:1:0/power/runtime_status
/sys/devices/2:0:1:0/power/control
/sys/devices/2:0:1:0/power/runtime_suspended_time
/sys/devices/2:0:1:0/power/runtime_active_time
/sys/devices/2:0:1:0/power/autosuspend_delay_ms
/sys/devices/2:0:1:0/scsi_device
/sys/devices/3:0:0:0
/sys/devices/3:0:0:0/uevent
/sys/devices/3:0:0:0/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/3:0:0:0/power
/sys/devices/3:0:0:0/power/wakeup
/sys/devices/3:0:0:0/power/wakeup_count
/sys/devices/3:0:0:0/power/wakeup_active_count
/sys/devices/3:0:0:0/power/wakeup_hit_count
/sys/devices/3:0:0:0/power/wakeup_active
/sys/devices/3:0:0:0/power/wakeup_total_time_ms
/sys/devices/3:0:0:0/power/wakeup_max_time_ms
/sys/devices/3:0:0:0/power/wakeup_last_time_ms
/sys/devices/3:0:0:0/power/runtime_status
/sys/devices/3:0:0:0/power/control
/sys/devices/3:0:0:0/power/runtime_suspended_time
/sys/devices/3:0:0:0/power/runtime_active_time
/sys/devices/3:0:0:0/power/autosuspend_delay_ms
/sys/devices/3:0:0:0/scsi_device
/sys/devices/disk01
/sys/devices/disk01/uevent
/sys/devices/disk01/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/disk01/power
/sys/devices/disk01/power/wakeup
/sys/devices/disk01/power/wakeup_count
/sys/devices/disk01/power/wakeup_active_count
/sys/devices/disk01/power/wakeup_hit_count
/sys/devices/disk01/power/wakeup_active
/sys/devices/disk01/power/wakeup_total_time_ms
/sys/devices/disk01/power/wakeup_max_time_ms
/sys/devices/disk01/power/wakeup_last_time_ms
/sys/devices/disk01/power/runtime_status
/sys/devices/disk01/power/control
/sys/devices/disk01/power/runtime_suspended_time
/sys/devices/disk01/power/runtime_active_time
/sys/devices/disk01/power/autosuspend_delay_ms
/sys/devices/disk01/driver -> ../../bus/scsi_tgt_dev/drivers/vdisk_fileio
/sys/devices/disk01/exported
/sys/devices/disk01/exported/export0 -> ../../ib_srpt_target_0/luns/0
/sys/devices/disk01/exported/export1 -> ../../ib_srpt_target_1/luns/0
/sys/devices/disk01/type
/sys/devices/disk01/threads_num
/sys/devices/disk01/threads_pool_type
/sys/devices/disk01/size_mb
/sys/devices/disk01/blocksize
/sys/devices/disk01/read_only
/sys/devices/disk01/write_through
/sys/devices/disk01/thin_provisioned
/sys/devices/disk01/nv_cache
/sys/devices/disk01/o_direct
/sys/devices/disk01/removable
/sys/devices/disk01/filename
/sys/devices/disk01/resync_size
/sys/devices/disk01/t10_dev_id
/sys/devices/disk01/usn
/sys/devices/disk02
/sys/devices/disk02/uevent
/sys/devices/disk02/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/disk02/power
/sys/devices/disk02/power/wakeup
/sys/devices/disk02/power/wakeup_count
/sys/devices/disk02/power/wakeup_active_count
/sys/devices/disk02/power/wakeup_hit_count
/sys/devices/disk02/power/wakeup_active
/sys/devices/disk02/power/wakeup_total_time_ms
/sys/devices/disk02/power/wakeup_max_time_ms
/sys/devices/disk02/power/wakeup_last_time_ms
/sys/devices/disk02/power/runtime_status
/sys/devices/disk02/power/control
/sys/devices/disk02/power/runtime_suspended_time
/sys/devices/disk02/power/runtime_active_time
/sys/devices/disk02/power/autosuspend_delay_ms
/sys/devices/disk02/driver -> ../../bus/scsi_tgt_dev/drivers/vdisk_fileio
/sys/devices/disk02/exported
/sys/devices/disk02/exported/export0 -> ../../ib_srpt_target_0/luns/1
/sys/devices/disk02/exported/export1 -> ../../ib_srpt_target_1/luns/1
/sys/devices/disk02/type
/sys/devices/disk02/threads_num
/sys/devices/disk02/threads_pool_type
/sys/devices/disk02/size_mb
/sys/devices/disk02/blocksize
/sys/devices/disk02/read_only
/sys/devices/disk02/write_through
/sys/devices/disk02/thin_provisioned
/sys/devices/disk02/nv_cache
/sys/devices/disk02/o_direct
/sys/devices/disk02/removable
/sys/devices/disk02/filename
/sys/devices/disk02/resync_size
/sys/devices/disk02/t10_dev_id
/sys/devices/disk02/usn
/sys/devices/disk03
/sys/devices/disk03/uevent
/sys/devices/disk03/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/disk03/power
/sys/devices/disk03/power/wakeup
/sys/devices/disk03/power/wakeup_count
/sys/devices/disk03/power/wakeup_active_count
/sys/devices/disk03/power/wakeup_hit_count
/sys/devices/disk03/power/wakeup_active
/sys/devices/disk03/power/wakeup_total_time_ms
/sys/devices/disk03/power/wakeup_max_time_ms
/sys/devices/disk03/power/wakeup_last_time_ms
/sys/devices/disk03/power/runtime_status
/sys/devices/disk03/power/control
/sys/devices/disk03/power/runtime_suspended_time
/sys/devices/disk03/power/runtime_active_time
/sys/devices/disk03/power/autosuspend_delay_ms
/sys/devices/disk03/driver -> ../../bus/scsi_tgt_dev/drivers/vdisk_fileio
/sys/devices/disk03/exported
/sys/devices/disk03/exported/export0 -> ../../ib_srpt_target_0/luns/2
/sys/devices/disk03/exported/export1 -> ../../ib_srpt_target_1/luns/2
/sys/devices/disk03/type
/sys/devices/disk03/threads_num
/sys/devices/disk03/threads_pool_type
/sys/devices/disk03/size_mb
/sys/devices/disk03/blocksize
/sys/devices/disk03/read_only
/sys/devices/disk03/write_through
/sys/devices/disk03/thin_provisioned
/sys/devices/disk03/nv_cache
/sys/devices/disk03/o_direct
/sys/devices/disk03/removable
/sys/devices/disk03/filename
/sys/devices/disk03/resync_size
/sys/devices/disk03/t10_dev_id
/sys/devices/disk03/usn
/sys/devices/disk04
/sys/devices/disk04/uevent
/sys/devices/disk04/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/disk04/power
/sys/devices/disk04/power/wakeup
/sys/devices/disk04/power/wakeup_count
/sys/devices/disk04/power/wakeup_active_count
/sys/devices/disk04/power/wakeup_hit_count
/sys/devices/disk04/power/wakeup_active
/sys/devices/disk04/power/wakeup_total_time_ms
/sys/devices/disk04/power/wakeup_max_time_ms
/sys/devices/disk04/power/wakeup_last_time_ms
/sys/devices/disk04/power/runtime_status
/sys/devices/disk04/power/control
/sys/devices/disk04/power/runtime_suspended_time
/sys/devices/disk04/power/runtime_active_time
/sys/devices/disk04/power/autosuspend_delay_ms
/sys/devices/disk04/driver -> ../../bus/scsi_tgt_dev/drivers/vdisk_fileio
/sys/devices/disk04/exported
/sys/devices/disk04/exported/export0 -> ../../ib_srpt_target_0/luns/3
/sys/devices/disk04/exported/export1 -> ../../ib_srpt_target_1/luns/3
/sys/devices/disk04/type
/sys/devices/disk04/threads_num
/sys/devices/disk04/threads_pool_type
/sys/devices/disk04/size_mb
/sys/devices/disk04/blocksize
/sys/devices/disk04/read_only
/sys/devices/disk04/write_through
/sys/devices/disk04/thin_provisioned
/sys/devices/disk04/nv_cache
/sys/devices/disk04/o_direct
/sys/devices/disk04/removable
/sys/devices/disk04/filename
/sys/devices/disk04/resync_size
/sys/devices/disk04/t10_dev_id
/sys/devices/disk04/usn
/sys/devices/disk05
/sys/devices/disk05/uevent
/sys/devices/disk05/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/disk05/power
/sys/devices/disk05/power/wakeup
/sys/devices/disk05/power/wakeup_count
/sys/devices/disk05/power/wakeup_active_count
/sys/devices/disk05/power/wakeup_hit_count
/sys/devices/disk05/power/wakeup_active
/sys/devices/disk05/power/wakeup_total_time_ms
/sys/devices/disk05/power/wakeup_max_time_ms
/sys/devices/disk05/power/wakeup_last_time_ms
/sys/devices/disk05/power/runtime_status
/sys/devices/disk05/power/control
/sys/devices/disk05/power/runtime_suspended_time
/sys/devices/disk05/power/runtime_active_time
/sys/devices/disk05/power/autosuspend_delay_ms
/sys/devices/disk05/driver -> ../../bus/scsi_tgt_dev/drivers/vdisk_nullio
/sys/devices/disk05/exported
/sys/devices/disk05/exported/export0 -> ../../ib_srpt_target_0/luns/4
/sys/devices/disk05/exported/export1 -> ../../ib_srpt_target_1/luns/4
/sys/devices/disk05/type
/sys/devices/disk05/threads_num
/sys/devices/disk05/threads_pool_type
/sys/devices/disk05/size_mb
/sys/devices/disk05/blocksize
/sys/devices/disk05/read_only
/sys/devices/disk05/removable
/sys/devices/disk05/t10_dev_id
/sys/devices/disk05/usn
/sys/devices/disk06
/sys/devices/disk06/uevent
/sys/devices/disk06/subsystem -> ../../bus/scsi_tgt_dev
/sys/devices/disk06/power
/sys/devices/disk06/power/wakeup
/sys/devices/disk06/power/wakeup_count
/sys/devices/disk06/power/wakeup_active_count
/sys/devices/disk06/power/wakeup_hit_count
/sys/devices/disk06/power/wakeup_active
/sys/devices/disk06/power/wakeup_total_time_ms
/sys/devices/disk06/power/wakeup_max_time_ms
/sys/devices/disk06/power/wakeup_last_time_ms
/sys/devices/disk06/power/runtime_status
/sys/devices/disk06/power/control
/sys/devices/disk06/power/runtime_suspended_time
/sys/devices/disk06/power/runtime_active_time
/sys/devices/disk06/power/autosuspend_delay_ms
/sys/devices/disk06/driver -> ../../bus/scsi_tgt_dev/drivers/vdisk_nullio
/sys/devices/disk06/exported
/sys/devices/disk06/exported/export0 -> ../../ib_srpt_target_0/luns/5
/sys/devices/disk06/exported/export1 -> ../../ib_srpt_target_1/luns/5
/sys/devices/disk06/type
/sys/devices/disk06/threads_num
/sys/devices/disk06/threads_pool_type
/sys/devices/disk06/size_mb
/sys/devices/disk06/blocksize
/sys/devices/disk06/read_only
/sys/devices/disk06/removable
/sys/devices/disk06/t10_dev_id
/sys/devices/disk06/usn
/sys/devices/ib_srpt_target_0
/sys/devices/ib_srpt_target_0/uevent
/sys/devices/ib_srpt_target_0/subsystem -> ../../bus/scsi_tgt_port
/sys/devices/ib_srpt_target_0/power
/sys/devices/ib_srpt_target_0/power/wakeup
/sys/devices/ib_srpt_target_0/power/wakeup_count
/sys/devices/ib_srpt_target_0/power/wakeup_active_count
/sys/devices/ib_srpt_target_0/power/wakeup_hit_count
/sys/devices/ib_srpt_target_0/power/wakeup_active
/sys/devices/ib_srpt_target_0/power/wakeup_total_time_ms
/sys/devices/ib_srpt_target_0/power/wakeup_max_time_ms
/sys/devices/ib_srpt_target_0/power/wakeup_last_time_ms
/sys/devices/ib_srpt_target_0/power/runtime_status
/sys/devices/ib_srpt_target_0/power/control
/sys/devices/ib_srpt_target_0/power/runtime_suspended_time
/sys/devices/ib_srpt_target_0/power/runtime_active_time
/sys/devices/ib_srpt_target_0/power/autosuspend_delay_ms
/sys/devices/ib_srpt_target_0/driver -> ../../bus/scsi_tgt_port/drivers/ib_srpt
/sys/devices/ib_srpt_target_0/enabled
/sys/devices/ib_srpt_target_0/sessions
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/commands
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/active_commands
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/initiator_name
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/req_lim
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/req_lim_delta
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/luns
-> ../../luns
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun0
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun0/active_commands
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun1
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun1/active_commands
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun2
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun2/active_commands
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun3
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun3/active_commands
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun4
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun4/active_commands
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun5
/sys/devices/ib_srpt_target_0/sessions/0x00000000000000000002c9030005f34b/lun5/active_commands
/sys/devices/ib_srpt_target_0/luns
/sys/devices/ib_srpt_target_0/luns/parameters
/sys/devices/ib_srpt_target_0/luns/0
/sys/devices/ib_srpt_target_0/luns/0/read_only
/sys/devices/ib_srpt_target_0/luns/0/device -> ../../../disk01
/sys/devices/ib_srpt_target_0/luns/1
/sys/devices/ib_srpt_target_0/luns/1/read_only
/sys/devices/ib_srpt_target_0/luns/1/device -> ../../../disk02
/sys/devices/ib_srpt_target_0/luns/2
/sys/devices/ib_srpt_target_0/luns/2/read_only
/sys/devices/ib_srpt_target_0/luns/2/device -> ../../../disk03
/sys/devices/ib_srpt_target_0/luns/3
/sys/devices/ib_srpt_target_0/luns/3/read_only
/sys/devices/ib_srpt_target_0/luns/3/device -> ../../../disk04
/sys/devices/ib_srpt_target_0/luns/4
/sys/devices/ib_srpt_target_0/luns/4/read_only
/sys/devices/ib_srpt_target_0/luns/4/device -> ../../../disk05
/sys/devices/ib_srpt_target_0/luns/5
/sys/devices/ib_srpt_target_0/luns/5/read_only
/sys/devices/ib_srpt_target_0/luns/5/device -> ../../../disk06
/sys/devices/ib_srpt_target_0/ini_groups
/sys/devices/ib_srpt_target_0/rel_tgt_id
/sys/devices/ib_srpt_target_0/addr_method
/sys/devices/ib_srpt_target_0/io_grouping_type
/sys/devices/ib_srpt_target_0/cpu_mask
/sys/devices/ib_srpt_target_0/login_info
/sys/devices/ib_srpt_target_1
/sys/devices/ib_srpt_target_1/uevent
/sys/devices/ib_srpt_target_1/subsystem -> ../../bus/scsi_tgt_port
/sys/devices/ib_srpt_target_1/power
/sys/devices/ib_srpt_target_1/power/wakeup
/sys/devices/ib_srpt_target_1/power/wakeup_count
/sys/devices/ib_srpt_target_1/power/wakeup_active_count
/sys/devices/ib_srpt_target_1/power/wakeup_hit_count
/sys/devices/ib_srpt_target_1/power/wakeup_active
/sys/devices/ib_srpt_target_1/power/wakeup_total_time_ms
/sys/devices/ib_srpt_target_1/power/wakeup_max_time_ms
/sys/devices/ib_srpt_target_1/power/wakeup_last_time_ms
/sys/devices/ib_srpt_target_1/power/runtime_status
/sys/devices/ib_srpt_target_1/power/control
/sys/devices/ib_srpt_target_1/power/runtime_suspended_time
/sys/devices/ib_srpt_target_1/power/runtime_active_time
/sys/devices/ib_srpt_target_1/power/autosuspend_delay_ms
/sys/devices/ib_srpt_target_1/driver -> ../../bus/scsi_tgt_port/drivers/ib_srpt
/sys/devices/ib_srpt_target_1/enabled
/sys/devices/ib_srpt_target_1/sessions
/sys/devices/ib_srpt_target_1/luns
/sys/devices/ib_srpt_target_1/luns/parameters
/sys/devices/ib_srpt_target_1/luns/0
/sys/devices/ib_srpt_target_1/luns/0/read_only
/sys/devices/ib_srpt_target_1/luns/0/device -> ../../../disk01
/sys/devices/ib_srpt_target_1/luns/1
/sys/devices/ib_srpt_target_1/luns/1/read_only
/sys/devices/ib_srpt_target_1/luns/1/device -> ../../../disk02
/sys/devices/ib_srpt_target_1/luns/2
/sys/devices/ib_srpt_target_1/luns/2/read_only
/sys/devices/ib_srpt_target_1/luns/2/device -> ../../../disk03
/sys/devices/ib_srpt_target_1/luns/3
/sys/devices/ib_srpt_target_1/luns/3/read_only
/sys/devices/ib_srpt_target_1/luns/3/device -> ../../../disk04
/sys/devices/ib_srpt_target_1/luns/4
/sys/devices/ib_srpt_target_1/luns/4/read_only
/sys/devices/ib_srpt_target_1/luns/4/device -> ../../../disk05
/sys/devices/ib_srpt_target_1/luns/5
/sys/devices/ib_srpt_target_1/luns/5/read_only
/sys/devices/ib_srpt_target_1/luns/5/device -> ../../../disk06
/sys/devices/ib_srpt_target_1/ini_groups
/sys/devices/ib_srpt_target_1/rel_tgt_id
/sys/devices/ib_srpt_target_1/addr_method
/sys/devices/ib_srpt_target_1/io_grouping_type
/sys/devices/ib_srpt_target_1/cpu_mask
/sys/devices/ib_srpt_target_1/login_info
/sys/devices/scst
/sys/devices/scst/uevent
/sys/devices/scst/power
/sys/devices/scst/power/wakeup
/sys/devices/scst/power/wakeup_count
/sys/devices/scst/power/wakeup_active_count
/sys/devices/scst/power/wakeup_hit_count
/sys/devices/scst/power/wakeup_active
/sys/devices/scst/power/wakeup_total_time_ms
/sys/devices/scst/power/wakeup_max_time_ms
/sys/devices/scst/power/wakeup_last_time_ms
/sys/devices/scst/power/runtime_status
/sys/devices/scst/power/control
/sys/devices/scst/power/runtime_suspended_time
/sys/devices/scst/power/runtime_active_time
/sys/devices/scst/power/autosuspend_delay_ms
/sys/devices/scst/mgmt
/sys/devices/scst/threads
/sys/devices/scst/setup_id
/sys/devices/scst/trace_level
/sys/devices/scst/version
/sys/devices/scst/sgv
/sys/devices/scst/sgv/global_stats
/sys/devices/scst/sgv/sgv
/sys/devices/scst/sgv/sgv/stats
/sys/devices/scst/sgv/sgv-clust
/sys/devices/scst/sgv/sgv-clust/stats
/sys/devices/scst/sgv/sgv-dma
/sys/devices/scst/sgv/sgv-dma/stats
Bart.