Hi,
i try to set up the object-based layout with pnfs.
I am using the OSC-OSD as target. It works well and i can login.
Trying the scripts "do-*" from git://git.open-osd.org/open-osd.git the do-osd script works for me. The do-exofs doesnt, here i have some question:
1. Is the UUID in the do-exofs script optional?
2. If no how can i get it for my device?
3. It is mandatory to use raid-driver (raid456) in the script? Why?
The error is:
mount -t exofs -o osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000,_netdev /dev/osd0 /mnt/osd0/
mount: wrong fs type, bad option, bad superblock on /dev/osd0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
dmesg says:
exofs: Unable to mount exofs on (null) pid=0x0 err=-22
Thanks for answering my questions.
Regards
Johannes
--
NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
On 05/15/2012 03:19 PM, Johannes Schild wrote:
> Hi,
>
> thanks for your fast reply.
>
> But now i run into a nullpointer exception while mounting /dev/osd0
>
>
> [ 329.919112] Modules linked in: raid456 async_raid6_recov async_pq
> raid6_pq async_memcpy bnx2i cnic cxgb4i cxgb3i iscsi_tcp be2iscsi
> iscsi_boot_sysfs uio cxgb4 libcxgbi cxgb3 mdio libiscsi_tcp libiscsi
> scsi_transport_iscsi fuse ip6t_REJECT nf_conntrack_ipv6
> nf_defrag_ipv6 xt_state nf_conntrack 8021q ip6table_filter ip6_tables
> garp stp llc fcoe libfcoe libfc scsi_transport_fc scsi_tgt
> snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus joydev
> snd_seq snd_seq_device snd_pcm ppdev vmw_balloon floppy parport_pc
> snd_timer snd pcnet32 microcode parport pcspkr mii snd_page_alloc
> i2c_piix4 i2c_core shpchp uinput mptspi mptscsih mptbase
> scsi_transport_spi [last unloaded: iscsi_tcp] [ 329.919133]
One thing that worries me here is this: [last unloaded: iscsi_tcp]
Actually it should be in the loaded bunch not unloaded. I have no idea how can that happen.
Do you see any prints in dmsg regarding iscsi, before the crash?
> [ 329.919134] Pid: 2234, comm: mount Not tainted 3.3.0-pnfs #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
> [ 329.919137] RIP: 0010:[<ffffffff81259799>] [<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
> [ 329.919139] RSP: 0018:ffff88003c4e5bc8 EFLAGS: 00010246
> [ 329.919141] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000001795
> [ 329.919142] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88003c4f5000
> [ 329.919143] RBP: ffff88003c4e5bd8 R08: 0000000000000000 R09: 0000000000000000
> [ 329.919144] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88003c4f5000
> [ 329.919145] R13: ffff88003c4e5d58 R14: 00000000fffffffb R15: ffff88003bd4e360
> [ 329.919146] FS: 00007f68ce81d800(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
> [ 329.919148] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 329.919149] CR2: 0000000000000000 CR3: 000000003cbfd000 CR4: 00000000000406f0
> [ 329.919152] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 329.919155] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 329.919156] Process mount (pid: 2234, threadinfo ffff88003c4e4000, task ffff88003aa42e40)
> [ 329.919157] Stack:
> [ 329.919158] ffff88003c4f5000 ffff88003c4f5800 ffff88003c4e5d18 ffffffff8125a780
> [ 329.919160] ffff88003c4e5c84 0000000081f27618 ffffffff81f27600 ffff88003f80c288
> [ 329.919162] ffff88003ddbeee0 0000000000000000 ffff88003c4e5c48 ffffffff812c8433
> [ 329.919164] Call Trace:
> [ 329.919166] [<ffffffff8125a780>] exofs_fill_super+0x440/0x6e0
> [ 329.919171] [<ffffffff812c8433>] ? idr_pre_get+0x53/0x90
> [ 329.919173] [<ffffffff8125a340>] ? exofs_read_lookup_dev_table+0x480/0x480
> [ 329.919176] [<ffffffff81180e18>] mount_nodev+0x58/0xb0
> [ 329.919178] [<ffffffff812594ff>] exofs_mount+0x4f/0x80
> [ 329.919179] [<ffffffff81181e13>] mount_fs+0x43/0x1b0
> [ 329.919182] [<ffffffff8119b56a>] vfs_kern_mount+0x6a/0xf0
> [ 329.919183] [<ffffffff8119bd44>] do_kern_mount+0x54/0x110
> [ 329.919185] [<ffffffff8119d4b4>] do_mount+0x1a4/0x830
> [ 329.919188] [<ffffffff8115da33>] ? alloc_pages_current+0xa3/0x110
> [ 329.919191] [<ffffffff811229e4>] ? __get_free_pages+0x14/0x50
> [ 329.919193] [<ffffffff8119d18a>] ? copy_mount_options+0x3a/0x170
> [ 329.919194] [<ffffffff8119dc80>] sys_mount+0x90/0xe0
> [ 329.919199] [<ffffffff81605aa9>] system_call_fastpath+0x16/0x1b
> [ 329.919200] Code: 66 90 41 8b 84 24 70 03 00 00 83 eb 01 39 c3 72 50 41 8b 94 24 74 03 00 00 01 c2 39 d3 73 42 89 da 29 c2 49 8b 84 24 88 03 00 00 <48> 8b 04 d0 48 8b 38 48 85 ff 74 0c 48 c7 00 00 00 00 00 e8 6f
> [ 329.919214] RIP [<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
> [ 329.919216] RSP <ffff88003c4e5bc8>
> [ 329.919217] CR2: 0000000000000000
> [ 329.919220] ---[ end trace e5a3b1124e42d9e3 ]---
>
>
could you please do:
[]$ gdb fs/exofs/exofs.ko
Inside gdb
> list *(exofs_free_sbi+0x59)
and also
> list *(exofs_fill_super+0x440)
> Is there any possibility to avoid these failure?
>
I've never had it lets try to resolve it.
Could you enable CONFIG_EXOFS_DEBUG it's under:
miscellaneous-filesystems/exofs in make xconfig
Then re-run everything send me the output
[]$ ./do-osd stop
[]$ ls /dev/osd*
<1>
[]$ ./do-osd
[]$ ls /dev/osd*
[]$ ./do-exofs format
Send me the output of that
[]$ ./do-exofs start
Send me the dmesg output of this stage, or if not too big
the dmesg output of from before ./do-osd <1>
> Just now i am using the 3.3.0 kernel from the linux-pnfs repository.
>
When compiling the Kernel, Did you enable CONFIG_PNFSD ?
(That is the pNFSD Server Kernel Support)
What platform are you using? Distro + ARCH ?
This should be a good tree. Though usually I use an
open-osd.org/linux-open-osd.git based tree. But I don't see much
difference. Let me have a look if there is something grossly
missing.
>
> Thanks in adavance
>
Sorry for your trouble I have neglected the public trees for a
while. Let me see if I can push a more tested tree up on
open-osd.org.
> Cheers
> Johannes
>
Thanks
Boaz
> -------- Original-Nachricht --------
>> Datum: Tue, 15 May 2012 12:48:08 +0300
>> Von: Boaz Harrosh <[email protected]>
>> An: Johannes Schild <[email protected]>
>> CC: [email protected], open-osd <[email protected]>
>> Betreff: Re: Questions about Exofs
>
>> On 05/15/2012 12:03 PM, Johannes Schild wrote:
>>
>>> Hi,
>>>
>>> i try to set up the object-based layout with pnfs.
>>> I am using the OSC-OSD as target. It works well and i can login.
>>> Trying the scripts "do-*" from git://git.open-osd.org/open-osd.git the
>> do-osd script works for me. The do-exofs doesnt, here i have some question:
>>>
>>> 1. Is the UUID in the do-exofs script optional?
>>>
>>
>>
>> No it is not optional
>>
>>> 2. If no how can i get it for my device?
>>>
>>
>>
>> Exactly! the ./do-exofs format command will set this for you, as well as
>> mkfs.exofs the
>> FS for you.
>>
>>> 3. It is mandatory to use raid-driver (raid456) in the script? Why?
>>>
>>
>>
>> What? where? Rrrr you are right. this is a very old tree. Let me see
>> if I have something newer to push. It should all load automatically now.
>>
>> But yes the dependency on raid456.ko is built in. Though if you use
>> raid=0 on the format command line it will not be used in run time.
>>
>>>
>>> The error is:
>>> mount -t exofs -o
>> osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000,_netdev /dev/osd0 /mnt/osd0/
>>> mount: wrong fs type, bad option, bad superblock on /dev/osd0,
>>> missing codepage or helper program, or other error
>>> In some cases useful info is found in syslog - try
>>> dmesg | tail or so
>>>
>>> dmesg says:
>>> exofs: Unable to mount exofs on (null) pid=0x0 err=-22
>>>
>>
>>
>> Probably you forgot the "./do-exofs format" stage. Please read the script
>> before, you might need to edit it. It is the stage that calls mkfs.exofs
>> to make a new filesystem for you. (An OSD is a raw device that can host
>> many filesystems each in it's own osd-partition.
>>
>>>
>>> Thanks for answering my questions.
>>
>>
>>> Regards
>>> Johannes
>>>
>>
>>
>> Cheers
>> Boaz
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
On Tue, May 15, 2012 at 3:19 PM, Johannes Schild <[email protected]> wrote:
> Hi,
>
> thanks for your fast reply.
>
> But now i run into a nullpointer exception while mounting /dev/osd0
>
>
> [ ?329.919112] Modules linked in: raid456 async_raid6_recov async_pq raid6_pq async_memcpy bnx2i cnic cxgb4i cxgb3i iscsi_tcp be2iscsi iscsi_boot_sysfs uio cxgb4 libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi fuse ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack 8021q ip6table_filter ip6_tables garp stp llc fcoe libfcoe libfc scsi_transport_fc scsi_tgt snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus joydev snd_seq snd_seq_device snd_pcm ppdev vmw_balloon floppy parport_pc snd_timer snd pcnet32 microcode parport pcspkr mii snd_page_alloc i2c_piix4 i2c_core shpchp uinput mptspi mptscsih mptbase scsi_transport_spi [last unloaded: iscsi_tcp]
> [ ?329.919133]
> [ ?329.919134] Pid: 2234, comm: mount Not tainted 3.3.0-pnfs #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
> [ ?329.919137] RIP: 0010:[<ffffffff81259799>] ?[<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
> [ ?329.919139] RSP: 0018:ffff88003c4e5bc8 ?EFLAGS: 00010246
> [ ?329.919141] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000001795
> [ ?329.919142] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88003c4f5000
> [ ?329.919143] RBP: ffff88003c4e5bd8 R08: 0000000000000000 R09: 0000000000000000
> [ ?329.919144] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88003c4f5000
> [ ?329.919145] R13: ffff88003c4e5d58 R14: 00000000fffffffb R15: ffff88003bd4e360
> [ ?329.919146] FS: ?00007f68ce81d800(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
> [ ?329.919148] CS: ?0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ ?329.919149] CR2: 0000000000000000 CR3: 000000003cbfd000 CR4: 00000000000406f0
> [ ?329.919152] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ ?329.919155] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ ?329.919156] Process mount (pid: 2234, threadinfo ffff88003c4e4000, task ffff88003aa42e40)
> [ ?329.919157] Stack:
> [ ?329.919158] ?ffff88003c4f5000 ffff88003c4f5800 ffff88003c4e5d18 ffffffff8125a780
> [ ?329.919160] ?ffff88003c4e5c84 0000000081f27618 ffffffff81f27600 ffff88003f80c288
> [ ?329.919162] ?ffff88003ddbeee0 0000000000000000 ffff88003c4e5c48 ffffffff812c8433
> [ ?329.919164] Call Trace:
> [ ?329.919166] ?[<ffffffff8125a780>] exofs_fill_super+0x440/0x6e0
> [ ?329.919171] ?[<ffffffff812c8433>] ? idr_pre_get+0x53/0x90
> [ ?329.919173] ?[<ffffffff8125a340>] ? exofs_read_lookup_dev_table+0x480/0x480
> [ ?329.919176] ?[<ffffffff81180e18>] mount_nodev+0x58/0xb0
> [ ?329.919178] ?[<ffffffff812594ff>] exofs_mount+0x4f/0x80
> [ ?329.919179] ?[<ffffffff81181e13>] mount_fs+0x43/0x1b0
> [ ?329.919182] ?[<ffffffff8119b56a>] vfs_kern_mount+0x6a/0xf0
> [ ?329.919183] ?[<ffffffff8119bd44>] do_kern_mount+0x54/0x110
> [ ?329.919185] ?[<ffffffff8119d4b4>] do_mount+0x1a4/0x830
> [ ?329.919188] ?[<ffffffff8115da33>] ? alloc_pages_current+0xa3/0x110
> [ ?329.919191] ?[<ffffffff811229e4>] ? __get_free_pages+0x14/0x50
> [ ?329.919193] ?[<ffffffff8119d18a>] ? copy_mount_options+0x3a/0x170
> [ ?329.919194] ?[<ffffffff8119dc80>] sys_mount+0x90/0xe0
> [ ?329.919199] ?[<ffffffff81605aa9>] system_call_fastpath+0x16/0x1b
> [ ?329.919200] Code: 66 90 41 8b 84 24 70 03 00 00 83 eb 01 39 c3 72 50 41 8b 94 24 74 03 00 00 01 c2 39 d3 73 42 89 da 29 c2 49 8b 84 24 88 03 00 00 <48> 8b 04 d0 48 8b 38 48 85 ff 74 0c 48 c7 00 00 00 00 00 e8 6f
> [ ?329.919214] RIP ?[<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
> [ ?329.919216] ?RSP <ffff88003c4e5bc8>
> [ ?329.919217] CR2: 0000000000000000
> [ ?329.919220] ---[ end trace e5a3b1124e42d9e3 ]---
>
>
> Is there any possibility to avoid these failure?
>
> Just now i am using the 3.3.0 kernel from the linux-pnfs repository.
>
>
> Thanks in adavance
>
> Cheers
> Johannes
>
> -------- Original-Nachricht --------
>> Datum: Tue, 15 May 2012 12:48:08 +0300
>> Von: Boaz Harrosh <[email protected]>
>> An: Johannes Schild <[email protected]>
>> CC: [email protected], open-osd <[email protected]>
>> Betreff: Re: Questions about Exofs
>
>> On 05/15/2012 12:03 PM, Johannes Schild wrote:
>>
>> > Hi,
>> >
>> > i try to set up the object-based layout with pnfs.
>> > I am using the OSC-OSD as target. It works well and i can login.
>> > Trying the scripts "do-*" from git://git.open-osd.org/open-osd.git the
>> do-osd script works for me. The do-exofs doesnt, here i have some question:
>> >
>> > 1. Is the UUID in the do-exofs script optional?
>> >
>>
>>
>> No it is not optional
>>
>> > 2. If no how can i get it for my device?
>> >
>>
>>
>> Exactly! the ./do-exofs format command will set this for you, as well as
>> mkfs.exofs the
>> FS for you.
>>
>> > 3. It is mandatory to use raid-driver (raid456) in the script? Why?
>> >
>>
>>
>> What? where? Rrrr you are right. this is a very old tree. Let me see
>> if I have something newer to push. It should all load automatically now.
>>
>> But yes the dependency on raid456.ko is built in. Though if you use
>> raid=0 on the format command line it will not be used in run time.
>>
>> >
>> > The error is:
>> > mount -t exofs -o
>> osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000,_netdev /dev/osd0 /mnt/osd0/
>> > mount: wrong fs type, bad option, bad superblock on /dev/osd0,
>> > ? ? ? ?missing codepage or helper program, or other error
>> > ? ? ? ?In some cases useful info is found in syslog - try
>> > ? ? ? ?dmesg | tail ?or so
>> >
>> > dmesg says:
>> > exofs: Unable to mount exofs on (null) pid=0x0 err=-22
>> >
>>
>>
>> Probably you forgot the "./do-exofs format" stage. Please read the script
>> before, you might need to edit it. It is the stage that calls mkfs.exofs
>> to make a new filesystem for you. (An OSD is a raw device that can host
>> many filesystems each in it's own osd-partition.
>>
>> >
>> > Thanks for answering my questions.
>>
>>
>> > Regards
>> > Johannes
>> >
>>
>>
>> Cheers
>> Boaz
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
>
> --
> NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
> Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
Hi,
This crash has happened to me as well. I was not able to explicitly
resolve it, but I have managed to set up a working pNFS object layout
environment. I'll try to give the instructions I use for setting up
the MDS, without using the do-* scripts, assuming you managed to set
up the data server. (I'm assuming a fedora distro):
setup the iscsi initiator:
# yum -y install iscsi-initiator-utils
# service iscsid start
# modprobe osd
# iscsiadm -m discovery -t sendtargets -p <your DS> --login
on my setup, in some (most) cases, the MDS machine hangs when you
power it off. I don't know why this happens.
format and mount the OSD device:
# modprobe raid456
# modprobe exofs
# cd /your/open-osd/project/directory
# LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 --dev=/dev/osd0
to me, this works without UUID. I had trouble in setting up exofs with
RAID, but if you're not interested in RAID that will do. note that
this setup is from the old tree (commit
df9cc23ca0e15f2fc7f1d16a21acb87c63038f50), I don't know if and what
have been pushed to it recently.
note that if you use the --format flag you would have to supply a UUID.
now you can mount the object file system. /dev/osd0 is seen as a char
device and not a block device, so the system can't read its superblock
and tell which file system is it. you have to explicitly specify exofs
as the file system, and exofs will do the hard work.:
# mount -t exofs -o pid=0x10000 /dev/osd0 /mnt/pnfs
--
Idan Kedar
Tonian
On Tue, May 15, 2012 at 6:06 PM, Boaz Harrosh <[email protected]> wrote:
> On 05/15/2012 05:22 PM, Idan Kedar wrote:
>
>> On Tue, May 15, 2012 at 4:42 PM, Boaz Harrosh <[email protected]> wrote:
> <snip>
>
>>> This should not be needed exofs is dependent on ore, dependent on raid456
>>
>
>> True, but when setting up the environment my goal was to have a
>> working environment. If a bug is introduced to this dependence system
>> and the module wouldn't be loaded, I would have to reluctantly spend
>> time finding the problem, when I can actually live with such a bug if
>> the workaround is simply loading the module manually. so as a policy,
>> I always explicitly load the modules I need when setting up a pNFS
>> environment.
>
>
> I'm sorry about that, that was my mess. My intention was to make all these
> dependencies totally transparent. As an implementation de-jur detail, not
> user visible.
>
> The reason I had it in script for a while was because in UML raid456.ko
> had a bug, which probing it manually gave a 20% better chance of success
> (Tree below has a patch that fixes that, on UML)
>
> Please note that some versions of exofs need it and some don't. I have
> removed it from my scripts, now.
>
> <snip>
>
>>> Are you sure you have an exofs FS at /mnt/pnfs ? please do an
>
>>> # df -h /mnt/pnfs I want to see ?
>
>>
>
>> # df -hT /mnt/pnfs/
>> Filesystem ? ?Type ? ?Size ?Used Avail Use% Mounted on
>> /dev/osd0 ? ?exofs ? ? 55G ? 12G ? 44G ?21% /mnt/pnfs
>>
>
>
> OK I understand that now, the weird single device case. I don't
> promise it will continue to work in Future. As a rule all devices
> most have a network-unique OSD_NAME.
No problem, I will change my scripts accordingly.
>
>
>>>
>>> Does it actually work? OK you know maybe it does. I can see this now.
>>> If you have a single device it might work, I didn't realize this. I thought
>>> the mkfs.exofs would not let you.
>>>
>>> For sure if you have more then one device (pnfs right?) then it will not
>>> let you, because the devices have strict order in the device table. Device
>>> names are not reliable and may change from login to login. You need some
>>> kind of device-id
>> Indeed it is a pNFS setup. And indeed I've had trouble when using more
>> than one OSD.
>
>
> I use a better script system for the cluster case, now, that I never pushed
> that makes all this easier.
>
> For one not setting osdname= at --format would explain the above.
>
>>
>> By the way, several weeks I have tried setting up a RAID 5 environment
>> with 8 OSDs, 1 mirror and RAID nesting. I then tried cloning and
>> compiling the kernel tree over this pNFS-OSD-RAID. The result was that
>> otgtd died, and I don't know why. It didn't dump core anywhere I could
>> find and the only "log" it has - stdout - didn't give any useful info.
>> I was going to inquire about this in a couple of weeks when I need to
>> get this environment working, but since this issue came up, maybe we
>> can somehow resolve it sooner.
>
>
> 8 OSDs with a mirror ? what was the mkfs.exofs command line you used?
Something along the lines of
# LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 --format
--mirrors=1 --group_width=2 --group_depth=2 --dev=/dev/osd0
--osdname=$(uuid) --dev=/dev/osd1 --osdname=$(uuid) --dev=/dev/osd2
--osdname=$(uuid) ...
I don't remember exactly at the moment, but I will bump this thread
when I'll start using RAID again.
>
> And did you use one otgtd with 8 targets, or 8 targets (8 IP addresses)
> with one target each, or a combination?
one target with 8 LUNs
>
> What is the otgtd platform? what file system? what HW and HD environment?
osc-osd over ext4, 64 bit VirtualBox VM over x86_64.
>
> And yes otgtd has some instabilities.
>
> There are two I can think off:
> * Over xfs the --format command crashes the otgtd (aborted exit no
> ?crash dump) Debugging welcome.
>
> * When lots of pnfs clients do heavy writing to the same otgtd, it
> ?times-out and disconnects.
it was a single client performing git-clone of the kernel tree.
> ?At Panasas we have a watch-dog that reloads it in a loop.
> ?I have only seen this on FreeBSD, in Linux it never happened
> ?to me.
>
> Please give me more details on what you did before it exited
> like that.
Nothing special, just git-clone. at some point it hanged (at a
different place every time), and when investigated a bit I saw that
otgtd is dead.
>
>
> In anyway I pushed a tree I tested with at:
> ? ? ? ?git://git.open-osd.org/linux-open-osd.git
>
> checkout the *merge_and_compile-3.3* branch. But in principal they are the
> same:
> ? ? ? ?fs/exofs ? ? ? ? ? ? ? ?- Added autologin support
> ? ? ? ?fs/nfs/objlayout ? ? ? ?- Added autologin support
> ? ? ? ?fs/nfsd ? ? ? ? ? ? ? ? - Same
> ? ? ? ?fs/nfs ? ? ? ? ? ? ? ? ?- Few fixes that are in benny's tree are not in linux-open-osd
Thanks, I will try it soon.
>
> So it should all be the same. For a proper cluster setup you will probably
> need my do-ect scripts which take a cluster descriptor file and does
> generic loops on everything.
Please note that I didn't try a cluster setup, just a single DS with 8
LUNs, single MDS, and single pNFS client, all 3 different VMs on the
same host.
>
> Thanks
> Boaz
--
Idan Kedar
Tonian
On 05/15/2012 07:21 PM, Idan Kedar wrote:
<snip>
>> 8 OSDs with a mirror ? what was the mkfs.exofs command line you used?
> Something along the lines of
> # LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 --format
> --mirrors=1 --group_width=2 --group_depth=2 --dev=/dev/osd0
> --osdname=$(uuid) --dev=/dev/osd1 --osdname=$(uuid) --dev=/dev/osd2
> --osdname=$(uuid) ...
>
> I don't remember exactly at the moment, but I will bump this thread
> when I'll start using RAID again.
Certainly missing the --format thing. --osdname= without an --format
is ignored.
Again if you never set an OSD_NAME on the devices in the past this will
not work.
But perhaps you did --format and forgot
>>
>> And did you use one otgtd with 8 targets, or 8 targets (8 IP addresses)
>> with one target each, or a combination?
> one target with 8 LUNs
>>
>> What is the otgtd platform? what file system? what HW and HD environment?
> osc-osd over ext4, 64 bit VirtualBox VM over x86_64.
OK that's a fishy setup.
The otgtd is sensitive to timeouts which I never investigated properly.
It looks like the OSD_VM-to-host, probably a single link, is very slow
and imagine 8 initiators actually banging on the same single slow link.
One of the commands times out, probably just a guess
The best for a VM setup, that I have is:
VM1 - exofs+MDS
VM2 - pnfs client.
(On my dev machine I have VM2+VM1 combined, and mount on localhost)
HOST - Run otgtd naively on host for best results.
Best is to spread all targets on as many physical HD devices.
Note that multiple targets from the same OSD-host only makes
"preformance" sense if they each serve a different spindle.
Unless in a simulated test environment as yourself
Which reminds me that upstream tgtd as a few fixes in this area I should
integrate.
>>
>> And yes otgtd has some instabilities.
>>
>> There are two I can think off:
>> * Over xfs the --format command crashes the otgtd (aborted exit no
>> crash dump) Debugging welcome.
>>
>> * When lots of pnfs clients do heavy writing to the same otgtd, it
>> times-out and disconnects.
> it was a single client performing git-clone of the kernel tree.
>> At Panasas we have a watch-dog that reloads it in a loop.
>> I have only seen this on FreeBSD, in Linux it never happened
>> to me.
>>
>> Please give me more details on what you did before it exited
>> like that.
> Nothing special, just git-clone. at some point it hanged (at a
> different place every time), and when investigated a bit I saw that
> otgtd is dead.
>>
>>
>> In anyway I pushed a tree I tested with at:
>> git://git.open-osd.org/linux-open-osd.git
>>
>> checkout the *merge_and_compile-3.3* branch. But in principal they are the
>> same:
>> fs/exofs - Added autologin support
>> fs/nfs/objlayout - Added autologin support
>> fs/nfsd - Same
>> fs/nfs - Few fixes that are in benny's tree are not in linux-open-osd
> Thanks, I will try it soon.
>>
>> So it should all be the same. For a proper cluster setup you will probably
>> need my do-ect scripts which take a cluster descriptor file and does
>> generic loops on everything.
> Please note that I didn't try a cluster setup, just a single DS with 8
> LUNs, single MDS, and single pNFS client, all 3 different VMs on the
> same host.
Just semantics so we speak the same language. Yes you do have a cluster.
In pnfs-objects world there are no such thing as DSs there is MDS and
there are OSDs. (objects). The OSDs are the equivalent of DSs in "files"
and LUNs in "blocks"
A none cluster is when you have a single OSD. (No striping, no multiple
devices, what I call the trivial layout)
>>
>> Thanks
>> Boaz
>
If you want there is a new tree at:
git://git.open-osd.org/tests.git
There is one script that does everything. ./do-ect (exofs cluster test)
You edit the ect.conf file (or run ./do-ect -f alternate.conf file)
In turn inside ect.conf you edit your topology and setup. It also points
to a device-table file (osds_list=XXX.olst) see lots of *.olst example
files.
list of operations.
* Read the scripts and study what they do. They are just a convenience
not a black-box application.
* Edit a new XXX.olst file
* Edit ect.conf with your setup.
[On MDS]
* ./do-ect login2 - login to all devices specified by osds_list=
* ./do-ect format2 - Set up an FS as specified by ect.conf
* ./do-ect mount2 - mount the exofs file system
* ./do-ect seturi - If you have the autologin version and want an autologin support.
[On pnfs client]
* ./do-ect login2 - Only if autologin is not enabled. You will need the
/sbin/osd_login script, which is part of the newest nfs-utils
(Tell me if you can't find it)
* ./do-ect pnfs_start - mount the pnfs server on pnfs_dir as specified in ect.conf
And there are other facilities as well. In principal the commands that end with "2"
are those that preform an action on the XXX.olst file and receive an optional parameter
as the OSDs list. For example:
./do-ect login2
- login to default list specified in ect.conf
./do-ect -f clusterXYZ.conf format2 device_table7.olst
- Format according to setup in clusterXYZ.conf file but override the OSDs list
instead use device_table7.olst.
Again please read the scripts before using. the .conf and .olst files are yours, I
just have them in git as an history of the tests I conducted.
But if you have any changes to the do-ect and fn-osd.sh please send me a patch.
There are other interesting scripts in there for example the target/ dir as a
way of controlling lots of OSD hosts in a single command using the closh script (CLuster Output SHell)
which also operate on the .olst and .clst files. Have fun
Cheers
Boaz
On Wed, May 16, 2012 at 3:15 PM, Boaz Harrosh <[email protected]> wrote:
> On 05/15/2012 07:21 PM, Idan Kedar wrote:
>
>>>> By the way, several weeks I have tried setting up a RAID 5 environment
>>>> >> with 8 OSDs, 1 mirror and RAID nesting. I then tried cloning and
>>>> >> compiling the kernel tree over this pNFS-OSD-RAID. The result was that
>>>> >> otgtd died, and I don't know why. It didn't dump core anywhere I could
>>>> >> find and the only "log" it has - stdout - didn't give any useful info.
>>>> >> I was going to inquire about this in a couple of weeks when I need to
>>>> >> get this environment working, but since this issue came up, maybe we
>>>> >> can somehow resolve it sooner.
>>> >
>>> >
>>> > 8 OSDs with a mirror ? what was the mkfs.exofs command line you used?
>> Something along the lines of
>> # LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 --format
>> --mirrors=1 --group_width=2 --group_depth=2 --dev=/dev/osd0
>> --osdname=$(uuid) --dev=/dev/osd1 --osdname=$(uuid) --dev=/dev/osd2
>> --osdname=$(uuid) ...
>>
>> I don't remember exactly at the moment, but I will bump this thread
>> when I'll start using RAID again.
>>> >
>
>
> BTW I don't see a --raid=5 above but a raid5 of --group_width=2 is just
> a mirror. The minimum for --raid=5 is 3. (I'm not sure if
> the system checks and optimizes for this)
>
> So the above should probably be:
>
> # LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 ? ?\
> ?--mirrors=2 --group_width=2 --group_depth=1000 ? ? ? ?\
> ?--dev=/dev/osd0 --format --osdname=$(uuid) ? ? ? ? ? ?\
> ?--dev=/dev/osd1 --format --osdname=$(uuid) ? ? ? ? ? ?\
> ?--dev=/dev/osd2 --format --osdname=$(uuid) ? ? ? ? ? ?\
> ?...
>
> group_depth is how many times to repeat this group until
> advancing to the next group.
>
> The group_count is:
> ? ? ? ?group_count = int( (total_devices / mirrors+1) / group_width )
>
> an interesting raid5 configuration with 8 devices mirrors and groups
> can be:
>
> # LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 ? ?\
> ?--mirrors=1 --raid=5 --group_width=3 --group_depth=1000 ? ? ? \
> ?--dev=/dev/osd0 --format --osdname=$(uuid) ? ? ? ? ? ?\
> ?--dev=/dev/osd1 --format --osdname=$(uuid) ? ? ? ? ? ?\
> ?--dev=/dev/osd2 --format --osdname=$(uuid) ? ? ? ? ? ?\
>
> Which means each file will have only a single group ?- 3-out-of-4.
> but the next file will use another set of 3 devices out of the 4.
>
> And I think you need a --stripe_pages= right?
>
> Cheers
> Boaz
As I said, I don't remember the exact line, so you are probably right.
I will be testing the object RAID functionality, so the more
"interesting" - the better. I'll use the last one you suggested, and I
would appreciate any input about which topologies give the RAID code a
hard time.
thanks,
idank
Hi Boaz,
sorry for my late reply. It was a public holiday in Germany. So i wasn't at work.
> Datum: Wed, 16 May 2012 13:30:34 +0300
> Von: Boaz Harrosh <[email protected]>
> An: Johannes Schild <[email protected]>
> CC: [email protected], [email protected]
> Betreff: Re: Questions about Exofs
> On 05/16/2012 12:00 PM, Johannes Schild wrote:
>
> > Hi Boaz,
>
> <>
>
> >> Do you see any prints in dmsg regarding iscsi, before the crash?
> >
> > I see output like this. Always "registered" no unloading execpt after
> the crash.
> >
> > [ 4.713107] iscsi: registered transport (tcp)
> > #<some output removed>
> > [ 4.739465] iscsi: registered transport (cxgb3i)
> > #<some output removed>
> > [ 4.750756] iscsi: registered transport (cxgb4i)
> > #<some output removed>
> > [ 4.771300] iscsi: registered transport (bnx2i)
> > [ 4.781045] iscsi: registered transport (be2iscsi)
> >
>
> <>
>
> >> could you please do:
> >> []$ gdb fs/exofs/exofs.ko
> >
> > [root@ExB osd-repo]# gdb /root/pnfs-repo/fs/exofs/exofs.ko
> > GNU gdb (GDB) Fedora (7.3.50.20110722-13.fc16)
> > Copyright (C) 2011 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-redhat-linux-gnu".
> > For bug reporting instructions, please see:
> > <http://www.gnu.org/software/gdb/bugs/>...
> > Reading symbols from /root/pnfs-repo/fs/exofs/exofs.ko...done.
> >
> >> Inside gdb
> >>> list *(exofs_free_sbi+0x59)
> >
> > (gdb) list *(exofs_free_sbi+0x59)
> > 0x47a9 is in exofs_free_sbi (include/scsi/osd_ore.h:83).
> > 78 /* ore_comp_dev Recievies a logical device index */
> > 79 static inline struct osd_dev *ore_comp_dev(
> > 80 const struct ore_components *oc, unsigned i)
> > 81 {
> > 82 BUG_ON((i < oc->first_dev) || (oc->first_dev + oc->numdevs <= i));
> > 83 return oc->ods[i - oc->first_dev]->od;
> > 84 }
> > 85
> > 86 static inline void ore_comp_set_dev(
> > 87 struct ore_components *oc, unsigned i, struct osd_dev *od)
> >
> >> and also
> >>> list *(exofs_fill_super+0x440)
> >
> > (gdb) list *(exofs_fill_super+0x440)
> > 0x5850 is in exofs_fill_super (fs/exofs/super.c:847).
> > 842 dput(sb->s_root);
> > 843 sb->s_root = NULL;
> > 844 goto free_sbi;
> > 845 }
> > 846
> > 847 _exofs_print_device("Mounting", opts->dev_name,
> > 848 ore_comp_dev(&sbi->oc, 0),
> > 849 sbi->one_comp.obj.partition);
> > 850 return 0;
> > 851
> > (gdb)
> >
>
>
> OK I understand we are _exofs_print_device an array that does
> not exists yet.
>
> >>
> >> Could you enable CONFIG_EXOFS_DEBUG it's under:
> >> miscellaneous-filesystems/exofs in make xconfig
> >
> > I enabled it.
> >
> >> Then re-run everything send me the output
> >> []$ ./do-osd stop
> >
> > [root@ExB osd-repo]# ./do-osd stop
> > /dev/osd0
> > FATAL: Module osd is builtin
> >
> > Should it be a modul or doesn't matter?
> >
>
>
> It should be fine. scripts expect it as a module.
>
> >> []$ ls /dev/osd*
> >
> > [root@ExB osd-repo]# ls /dev/osd*
> > ls: cannot access /dev/osd*: No such file or directory
> >
> >> []$ ./do-osd
> >
> > [root@ExB osd-repo]# ./do-osd
> > iscsid.service - LSB: Starts and stops login iSCSI daemon.
> > Loaded: loaded (/etc/rc.d/init.d/iscsid)
> > Active: inactive (dead) since Wed, 16 May 2012 10:46:23 +0200; 3min
> 11s ago
> > Process: 2287 ExecStop=/etc/rc.d/init.d/iscsid stop (code=exited,
> status=0/SUCCESS)
> > Process: 1168 ExecStart=/etc/rc.d/init.d/iscsid start (code=exited,
> status=0/SUCCESS)
> > Main PID: 1213 (code=exited, status=0/SUCCESS)
> > CGroup: name=systemd:/system/iscsid.service
> > 18446744072101122080
> > login into: 192.168.0.1:3260
> > 192.168.0.1:3260,1 .root.var.osd-tgt.tgt-1.ExA
> >
> >> []$ ls /dev/osd*
> >
> > [root@ExB server]# ls /dev/os*
> > /dev/osd1
> >
>
>
> /dev/osd1 interesting. make sure your scripts are using /dev/osd1.
> I suspect this is an artifact of the last games. On a clean reboot
> a single device should be /dev/osd0. The scripts expect that.
>
This was my fault. I rebooted and it works fine.
> >> []$ ./do-exofs format
> >> Send me the output of that
> >
> > ./do-exofs format
> > mkexofs_format >>>
>
>
> No output from the format command? that is not good. mkfs.exofs is
> very bad in not saying anything when failing.
>
> Probably because it was formatting /dev/osd0 and we have /dev/osd1 only
>
> > osd stop? >>>
> > FATAL: Module osd is builtin
> > osd start? >>>
> > iscsid.service - LSB: Starts and stops login iSCSI daemon.
> > Loaded: loaded (/etc/rc.d/init.d/iscsid)
> > Active: inactive (dead) since Wed, 16 May 2012 10:46:23 +0200; 6min
> ago
> > Process: 2287 ExecStop=/etc/rc.d/init.d/iscsid stop (code=exited,
> status=0/SUCCESS)
> > Process: 1168 ExecStart=/etc/rc.d/init.d/iscsid start (code=exited,
> status=0/SUCCESS)
> > Main PID: 1213 (code=exited, status=0/SUCCESS)
> > CGroup: name=systemd:/system/iscsid.service
> > 18446744072101122080
> > login into: 192.168.0.1:3260
> > 192.168.0.1:3260,1 .root.var.osd-tgt.tgt-1.ExA
> > Logging in to [iface: default, target: .root.var.osd-tgt.tgt-1.ExA,
> portal: 192.168.0.1,3260] (multiple)
> > Login to [iface: default, target: .root.var.osd-tgt.tgt-1.ExA, portal:
> 192.168.0.1,3260] successful.
> >
> >> []$ ./do-exofs start
> >> Send me the dmesg output of this stage, or if not too big
> >> the dmesg output of from before ./do-osd <1>
> >
> > I pushed it on nopaste:
> > http://nopaste.info/cd3c6f9141.html
> >
>
>
> in the dmesg I see:
>
> [ 2516.994781] exofs @parse_options:88: parse_options
> osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000
> [ 2516.994808] osd @_mach_odi:261: found device sysid_len=0 osdname=36
> [ 2516.994816] osd @_osdv2_req_encode_common:617: OSDv2 execute opcode
> 0x8885
> [ 2516.994831] osd @_init_blk_request:1616: or=ffff880020d7ec00 has_in=1
> has_out=0 => 0, ffff88003bbf8a10
>
> the very first read below fails. This is the first read from super-block
> object.
> Here it gets an -5 (-EIO) if it was an osd-target error you would have
> a scsi-sense printout so it means it is a communication problem.
>
> [ 2516.996034] exofs @exofs_read_kern:245: osd_execute_request() => -5
> [ 2516.996041] exofs: Unable to mount exofs on (null) pid=0x10000 err=-5
>
> This crash below I should fix. Code is not dealing properly with the IO
> error
> and continues to try and dmesg-print an array that does not exist yet.
> I will fix that.
>
> [ 2516.996106] BUG: unable to handle kernel NULL pointer dereference at
> (null)
> [ 2516.996111] IP: [<ffffffffa033c779>] exofs_free_sbi+0x59/0xa0 [exofs]
>
> But the problem still remains why do we get IO errors from iscsi?
>
> Later we have:
> [ 3241.802074] connection1:0: detected conn error (1020)
>
> disconnect. Do you see some prints at the otgtd side.
> If you use the ./up script it might rederect these to a log file
> do "./up log"
http://nopaste.info/04c87daf8b.html
Thats the output from ./up.
In the up-script what i use is no optione „log“ maybe its too old?
>
> [ 3398.831629] Chelsio T3 iSCSI Driver cxgb3i v2.0.0 (Jun. 2010)
> [ 3398.831919] iscsi: registered transport (cxgb3i)
> [ 3398.836776] Chelsio T4 iSCSI Driver cxgb4i v0.9.1 (Aug. 2010)
> [ 3398.836996] iscsi: registered transport (cxgb4i)
> [ 3398.841397] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.8 (Jan 3,
> 2012)
> [ 3398.845267] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.0.3 (Jun 15,
> 2011)
> [ 3398.845475] iscsi: registered transport (bnx2i)
> [ 3400.201828] scsi4 : iSCSI Initiator over TCP/IP
> [ 3400.715101] scsi 4:0:0:0: Object storage IET OSD
> 0001 PQ: 0 ANSI: 5
> [ 3400.718038] osd @__detect_osd:359: start scsi_test_unit_ready
> ffff880020db3800 ffff880020dfa000 ffff88003974aca0
>
> Right after the crash. So iscsi unloaded and loaded. There was a
> disconnect.
> We must investigate why iscsi has communication problems?
>
> the "192.168.0.1:3260" above is that your host's IP? You are running the
> otgtd on
> the host and exofs in VM? That's good that's what I use all the time.
I have for every Server (DS, MDS, Client) a VM running.
Its only to test.
>
> If you have time you should do two experiments.
>
> 1. Please run the "./do-osd test" test. send me the output.
> It runs a user mode test of the osd device and does some
> very basic communications.
> Note that it will wipe your OSD and you will need to ./do-exofs format
> again
> after it.
[root@ExM osd-repo]# ./do-osd test
libosd: Detected OSD2 device
libosd: VENDOR_IDENTIFICATION [OSC]
libosd: PRODUCT_IDENTIFICATION [OSDEMU]
libosd: PRODUCT_MODEL [OSD2r05]
libosd: PRODUCT_REVISION_LEVEL [117]
libosd: PRODUCT_SERIAL_NUMBER [2]
libosd: OSD_NAME [d2683732-c906-4ee1-9dbd-c10c27bb40df]
libosd: TOTAL_CAPACITY [0xffffffffffffffff]
libosd: USED_CAPACITY [0xffffffffffffffff]
libosd: NUMBER_OF_PARTITIONS [17]
libosd: CLOCK [0x000000000000]
libosd: OSD_SYSTEM_ID(20)
[f181000e4f534320202020204f5344454d550000][....OSC OSDEMU..]
libosd: format
libosd: create_partition
libosd: create_object
libosd: create_object
libosd: write
libosd: write
libosd: read
libosd: read
libosd: !!! Failed osd_req_write_sg_kern
do_test_17 returned 12: Cannot allocate memory
> 2. on the osd-target side you probably ran ./up. the otgtd also supports
> none-osd regular disk-devices. Could you set up a regular disk
> backbend as well. Look into "man tgtadm" on how to add a second
> disk target.
I hope thats right what i did:
[root@ExA server]# tgtadm --lld iscsi --mode target --op new --tid=1 --targetname iqn.2012-04.ExA
[root@ExA server]# tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 -b /dev/sdb
[root@ExA server]# tgtadm --lld iscsi --mode target --op bind --tid 1 -I ALL
[root@ExA server]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2012-04.ExA
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Readonly: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 2149 MB, Block size: 512
Online: Yes
Removable media: No
Readonly: No
Backing store type: rdwr
Backing store path: /dev/sdb
Backing store flags:
Account information:
ACL information:
ALL
[root@ExA server]#
> Once you login to the target you will see a new /dev/sdX device
> try to dd into it, and also mkfs and mount an ext FS on it.
Yes /dev/sdd on my system.
I did if=/dev/zero of=/dev/sdd
worked well.
[root@ExM server]# /root/osd-repo/usr/mkfs.exofs –pid=0x10000 –raid=0 –mirrors=0 –stripe_apges=4 –dev=/dev/sdd
doesnt work. I got:
exofs_mkfs –pid=0x10000 returned -60: Unknown error -60
Maybe i do something wrong with the device?
> Or else investigate why there are iscsi communication problems.
>
> >
> >
> >>
> >>> Just now i am using the 3.3.0 kernel from the linux-pnfs repository.
> >>>
>
>
> That's perfect it should have everything.
>
> >>
> >>
> >> When compiling the Kernel, Did you enable CONFIG_PNFSD ?
> >> (That is the pNFSD Server Kernel Support)
> >
> > No pNFSD Server support wasn't enabled, i recompiled and activate it
> >
>
>
> It's fine for this stage you don't need it
>
> >
> >
> >
> >> What platform are you using? Distro + ARCH ?
> >
> > Iam experimenting with Fedora 16 (3.3.0 pnfs kernel) and arch x86_64
> >
>
>
> I use that here too
>
> >
> > Thanks for your efforts
> > Johannes
>
>
> Hope that helps. Thanks for the report we got a bug fix
> Boaz
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
I have some additional questions to object-layout and to the storage:
How do i add a second physical storage server to my test configuration?
The script „do-osd“ contains only one $IP, so how can i add a second one?
I read the rfc5664 (Object-Based Parallel NFS (pNFS) Operations) but iam not sure if i understand it correctly:
The client retrievs a Layout (in my case object-layout) from the MDS. Then i searched with Wireshark for GETDEVICELIST/GETDEVICEINFO but i cant find them.
Normaly the client gets a GETDEVICELIST/GETDEVICEINFO so he can determine which OSDs are available from MDS right? But how it works after receiving the list?
Hope you can answer my (dump) questions...
Cheers
Johannes
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
On Tue, May 15, 2012 at 8:20 PM, Boaz Harrosh <[email protected]> wrote:
> On 05/15/2012 07:21 PM, Idan Kedar wrote:
> <snip>
>
>>> 8 OSDs with a mirror ? what was the mkfs.exofs command line you used?
>> Something along the lines of
>> # LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 --format
>> --mirrors=1 --group_width=2 --group_depth=2 --dev=/dev/osd0
>> --osdname=$(uuid) --dev=/dev/osd1 --osdname=$(uuid) --dev=/dev/osd2
>> --osdname=$(uuid) ...
>>
>> I don't remember exactly at the moment, but I will bump this thread
>> when I'll start using RAID again.
>
>
> Certainly missing the --format thing. ?--osdname= without an --format
> is ignored.
>
> Again if you never set an OSD_NAME on the devices in the past this will
> not work.
>
> But perhaps you did --format and forgot
>
>>>
>>> And did you use one otgtd with 8 targets, or 8 targets (8 IP addresses)
>>> with one target each, or a combination?
>> one target with 8 LUNs
>>>
>>> What is the otgtd platform? what file system? what HW and HD environment?
>> osc-osd over ext4, 64 bit VirtualBox VM over x86_64.
>
>
> OK that's a fishy setup.
>
> The otgtd is sensitive to timeouts which I never investigated properly.
> It looks like the OSD_VM-to-host, probably a single link, is very slow
> and imagine 8 initiators actually banging on the same single slow link.
> One of the commands times out, probably just a guess
>
> The best for a VM setup, that I have is:
>
> VM1 ? ? - exofs+MDS
> VM2 ? ? - pnfs client.
> ? (On my dev machine I have VM2+VM1 combined, and mount on localhost)
>
> HOST ? ?- Run otgtd naively on host for best results.
> ? ? ? ? ?Best is to spread all targets on as many physical HD devices.
> ? ? ? ? ?Note that multiple targets from the same OSD-host only makes
> ? ? ? ? ?"preformance" sense if they each serve a different spindle.
> ? ? ? ? ?Unless in a simulated test environment as yourself
>
> Which reminds me that upstream tgtd as a few fixes in this area I should
> integrate.
>
>>>
>>> And yes otgtd has some instabilities.
>>>
>>> There are two I can think off:
>>> * Over xfs the --format command crashes the otgtd (aborted exit no
>>> ?crash dump) Debugging welcome.
>>>
>>> * When lots of pnfs clients do heavy writing to the same otgtd, it
>>> ?times-out and disconnects.
>> it was a single client performing git-clone of the kernel tree.
>>> ?At Panasas we have a watch-dog that reloads it in a loop.
>>> ?I have only seen this on FreeBSD, in Linux it never happened
>>> ?to me.
>>>
>>> Please give me more details on what you did before it exited
>>> like that.
>> Nothing special, just git-clone. at some point it hanged (at a
>> different place every time), and when investigated a bit I saw that
>> otgtd is dead.
>>>
>>>
>>> In anyway I pushed a tree I tested with at:
>>> ? ? ? ?git://git.open-osd.org/linux-open-osd.git
>>>
>>> checkout the *merge_and_compile-3.3* branch. But in principal they are the
>>> same:
>>> ? ? ? ?fs/exofs ? ? ? ? ? ? ? ?- Added autologin support
>>> ? ? ? ?fs/nfs/objlayout ? ? ? ?- Added autologin support
>>> ? ? ? ?fs/nfsd ? ? ? ? ? ? ? ? - Same
>>> ? ? ? ?fs/nfs ? ? ? ? ? ? ? ? ?- Few fixes that are in benny's tree are not in linux-open-osd
>> Thanks, I will try it soon.
>>>
>>> So it should all be the same. For a proper cluster setup you will probably
>>> need my do-ect scripts which take a cluster descriptor file and does
>>> generic loops on everything.
>> Please note that I didn't try a cluster setup, just a single DS with 8
>> LUNs, single MDS, and single pNFS client, all 3 different VMs on the
>> same host.
>
>
> Just semantics so we speak the same language. Yes you do have a cluster.
>
> In pnfs-objects world there are no such thing as DSs there is MDS and
> there are OSDs. (objects). The OSDs are the equivalent of DSs in "files"
> and LUNs in "blocks"
>
> A none cluster is when you have a single OSD. (No striping, no multiple
> devices, what I call the trivial layout)
>
>>>
>>> Thanks
>>> Boaz
>>
>
>
> If you want there is a new tree at:
> ? ? ? ?git://git.open-osd.org/tests.git
>
> There is one script that does everything. ./do-ect (exofs cluster test)
>
> You edit the ect.conf file (or run ./do-ect -f alternate.conf file)
>
> In turn inside ect.conf you edit your topology and setup. It also points
> to a device-table file (osds_list=XXX.olst) see lots of *.olst example
> files.
>
> list of operations.
> * Read the scripts and study what they do. They are just a convenience
> ?not a black-box application.
> * Edit a new XXX.olst file
> * Edit ect.conf with your setup.
> [On MDS]
> * ./do-ect login2 ? ? ? - login to all devices specified by osds_list=
> * ./do-ect format2 ? ? ?- Set up an FS as specified by ect.conf
> * ./do-ect mount2 ? ? ? - mount the exofs file system
> * ./do-ect seturi ? ? ? - If you have the autologin version and want an autologin support.
>
> [On pnfs client]
> * ./do-ect login2 ? ? ? - Only if autologin is not enabled. You will need the
> ? ? ? ? ? ? ? ? ? ? ? ? ?/sbin/osd_login script, which is part of the newest nfs-utils
> ? ? ? ? ? ? ? ? ? ? ? ? ?(Tell me if you can't find it)
>
> * ./do-ect pnfs_start ?- mount the pnfs server on pnfs_dir as specified in ect.conf
>
> And there are other facilities as well. In principal the commands that end with "2"
> are those that preform an action on the XXX.olst file and receive an optional parameter
> as the OSDs list. For example:
>
> ./do-ect login2
> ? ? ? ?- login to default list specified in ect.conf
>
> ./do-ect -f clusterXYZ.conf format2 device_table7.olst
> ? ? ? ?- Format according to setup in clusterXYZ.conf file but override the OSDs list
> ? ? ? ? ?instead use device_table7.olst.
>
> Again please read the scripts before using. the .conf and .olst files are yours, I
> just have them in git as an history of the tests I conducted.
>
> But if you have any changes to the do-ect and fn-osd.sh please send me a patch.
>
> There are other interesting scripts in there for example the target/ dir as a
> way of controlling lots of OSD hosts in a single command using the closh script (CLuster Output SHell)
> which also operate on the .olst and .clst files. Have fun
>
> Cheers
> Boaz
Thank you for the instructions, I will get to that soon.
Cheers,
idank
On 05/15/2012 12:03 PM, Johannes Schild wrote:
> Hi,
>
> i try to set up the object-based layout with pnfs.
> I am using the OSC-OSD as target. It works well and i can login.
> Trying the scripts "do-*" from git://git.open-osd.org/open-osd.git the do-osd script works for me. The do-exofs doesnt, here i have some question:
>
> 1. Is the UUID in the do-exofs script optional?
>
No it is not optional
> 2. If no how can i get it for my device?
>
Exactly! the ./do-exofs format command will set this for you, as well as mkfs.exofs the
FS for you.
> 3. It is mandatory to use raid-driver (raid456) in the script? Why?
>
What? where? Rrrr you are right. this is a very old tree. Let me see
if I have something newer to push. It should all load automatically now.
But yes the dependency on raid456.ko is built in. Though if you use
raid=0 on the format command line it will not be used in run time.
>
> The error is:
> mount -t exofs -o osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000,_netdev /dev/osd0 /mnt/osd0/
> mount: wrong fs type, bad option, bad superblock on /dev/osd0,
> missing codepage or helper program, or other error
> In some cases useful info is found in syslog - try
> dmesg | tail or so
>
> dmesg says:
> exofs: Unable to mount exofs on (null) pid=0x0 err=-22
>
Probably you forgot the "./do-exofs format" stage. Please read the script
before, you might need to edit it. It is the stage that calls mkfs.exofs
to make a new filesystem for you. (An OSD is a raw device that can host
many filesystems each in it's own osd-partition.
>
> Thanks for answering my questions.
> Regards
> Johannes
>
Cheers
Boaz
On 05/15/2012 07:21 PM, Idan Kedar wrote:
>>> By the way, several weeks I have tried setting up a RAID 5 environment
>>> >> with 8 OSDs, 1 mirror and RAID nesting. I then tried cloning and
>>> >> compiling the kernel tree over this pNFS-OSD-RAID. The result was that
>>> >> otgtd died, and I don't know why. It didn't dump core anywhere I could
>>> >> find and the only "log" it has - stdout - didn't give any useful info.
>>> >> I was going to inquire about this in a couple of weeks when I need to
>>> >> get this environment working, but since this issue came up, maybe we
>>> >> can somehow resolve it sooner.
>> >
>> >
>> > 8 OSDs with a mirror ? what was the mkfs.exofs command line you used?
> Something along the lines of
> # LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 --format
> --mirrors=1 --group_width=2 --group_depth=2 --dev=/dev/osd0
> --osdname=$(uuid) --dev=/dev/osd1 --osdname=$(uuid) --dev=/dev/osd2
> --osdname=$(uuid) ...
>
> I don't remember exactly at the moment, but I will bump this thread
> when I'll start using RAID again.
>> >
BTW I don't see a --raid=5 above but a raid5 of --group_width=2 is just
a mirror. The minimum for --raid=5 is 3. (I'm not sure if
the system checks and optimizes for this)
So the above should probably be:
# LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 \
--mirrors=2 --group_width=2 --group_depth=1000 \
--dev=/dev/osd0 --format --osdname=$(uuid) \
--dev=/dev/osd1 --format --osdname=$(uuid) \
--dev=/dev/osd2 --format --osdname=$(uuid) \
...
group_depth is how many times to repeat this group until
advancing to the next group.
The group_count is:
group_count = int( (total_devices / mirrors+1) / group_width )
an interesting raid5 configuration with 8 devices mirrors and groups
can be:
# LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 \
--mirrors=1 --raid=5 --group_width=3 --group_depth=1000 \
--dev=/dev/osd0 --format --osdname=$(uuid) \
--dev=/dev/osd1 --format --osdname=$(uuid) \
--dev=/dev/osd2 --format --osdname=$(uuid) \
Which means each file will have only a single group - 3-out-of-4.
but the next file will use another set of 3 devices out of the 4.
And I think you need a --stripe_pages= right?
Cheers
Boaz
On 05/15/2012 04:09 PM, Idan Kedar wrote:
> On Tue, May 15, 2012 at 3:19 PM, Johannes Schild <[email protected]> wrote:
>> Hi,
>>
>> thanks for your fast reply.
>>
>> But now i run into a nullpointer exception while mounting /dev/osd0
>>
>>
>> [ 329.919112] Modules linked in: raid456 async_raid6_recov async_pq raid6_pq async_memcpy bnx2i cnic cxgb4i cxgb3i iscsi_tcp be2iscsi iscsi_boot_sysfs uio cxgb4 libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi fuse ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack 8021q ip6table_filter ip6_tables garp stp llc fcoe libfcoe libfc scsi_transport_fc scsi_tgt snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus joydev snd_seq snd_seq_device snd_pcm ppdev vmw_balloon floppy parport_pc snd_timer snd pcnet32 microcode parport pcspkr mii snd_page_alloc i2c_piix4 i2c_core shpchp uinput mptspi mptscsih mptbase scsi_transport_spi [last unloaded: iscsi_tcp]
>> [ 329.919133]
>> [ 329.919134] Pid: 2234, comm: mount Not tainted 3.3.0-pnfs #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
>> [ 329.919137] RIP: 0010:[<ffffffff81259799>] [<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
>> [ 329.919139] RSP: 0018:ffff88003c4e5bc8 EFLAGS: 00010246
>> [ 329.919141] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000001795
>> [ 329.919142] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88003c4f5000
>> [ 329.919143] RBP: ffff88003c4e5bd8 R08: 0000000000000000 R09: 0000000000000000
>> [ 329.919144] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88003c4f5000
>> [ 329.919145] R13: ffff88003c4e5d58 R14: 00000000fffffffb R15: ffff88003bd4e360
>> [ 329.919146] FS: 00007f68ce81d800(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
>> [ 329.919148] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 329.919149] CR2: 0000000000000000 CR3: 000000003cbfd000 CR4: 00000000000406f0
>> [ 329.919152] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 329.919155] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [ 329.919156] Process mount (pid: 2234, threadinfo ffff88003c4e4000, task ffff88003aa42e40)
>> [ 329.919157] Stack:
>> [ 329.919158] ffff88003c4f5000 ffff88003c4f5800 ffff88003c4e5d18 ffffffff8125a780
>> [ 329.919160] ffff88003c4e5c84 0000000081f27618 ffffffff81f27600 ffff88003f80c288
>> [ 329.919162] ffff88003ddbeee0 0000000000000000 ffff88003c4e5c48 ffffffff812c8433
>> [ 329.919164] Call Trace:
>> [ 329.919166] [<ffffffff8125a780>] exofs_fill_super+0x440/0x6e0
>> [ 329.919171] [<ffffffff812c8433>] ? idr_pre_get+0x53/0x90
>> [ 329.919173] [<ffffffff8125a340>] ? exofs_read_lookup_dev_table+0x480/0x480
>> [ 329.919176] [<ffffffff81180e18>] mount_nodev+0x58/0xb0
>> [ 329.919178] [<ffffffff812594ff>] exofs_mount+0x4f/0x80
>> [ 329.919179] [<ffffffff81181e13>] mount_fs+0x43/0x1b0
>> [ 329.919182] [<ffffffff8119b56a>] vfs_kern_mount+0x6a/0xf0
>> [ 329.919183] [<ffffffff8119bd44>] do_kern_mount+0x54/0x110
>> [ 329.919185] [<ffffffff8119d4b4>] do_mount+0x1a4/0x830
>> [ 329.919188] [<ffffffff8115da33>] ? alloc_pages_current+0xa3/0x110
>> [ 329.919191] [<ffffffff811229e4>] ? __get_free_pages+0x14/0x50
>> [ 329.919193] [<ffffffff8119d18a>] ? copy_mount_options+0x3a/0x170
>> [ 329.919194] [<ffffffff8119dc80>] sys_mount+0x90/0xe0
>> [ 329.919199] [<ffffffff81605aa9>] system_call_fastpath+0x16/0x1b
>> [ 329.919200] Code: 66 90 41 8b 84 24 70 03 00 00 83 eb 01 39 c3 72 50 41 8b 94 24 74 03 00 00 01 c2 39 d3 73 42 89 da 29 c2 49 8b 84 24 88 03 00 00 <48> 8b 04 d0 48 8b 38 48 85 ff 74 0c 48 c7 00 00 00 00 00 e8 6f
>> [ 329.919214] RIP [<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
>> [ 329.919216] RSP <ffff88003c4e5bc8>
>> [ 329.919217] CR2: 0000000000000000
>> [ 329.919220] ---[ end trace e5a3b1124e42d9e3 ]---
>>
>>
>> Is there any possibility to avoid these failure?
>>
>> Just now i am using the 3.3.0 kernel from the linux-pnfs repository.
>>
>>
>> Thanks in adavance
>>
>> Cheers
>> Johannes
>>
>> -------- Original-Nachricht --------
>>> Datum: Tue, 15 May 2012 12:48:08 +0300
>>> Von: Boaz Harrosh <[email protected]>
>>> An: Johannes Schild <[email protected]>
>>> CC: [email protected], open-osd <[email protected]>
>>> Betreff: Re: Questions about Exofs
>>
>>> On 05/15/2012 12:03 PM, Johannes Schild wrote:
>>>
>>>> Hi,
>>>>
>>>> i try to set up the object-based layout with pnfs.
>>>> I am using the OSC-OSD as target. It works well and i can login.
>>>> Trying the scripts "do-*" from git://git.open-osd.org/open-osd.git the
>>> do-osd script works for me. The do-exofs doesnt, here i have some question:
>>>>
>>>> 1. Is the UUID in the do-exofs script optional?
>>>>
>>>
>>>
>>> No it is not optional
>>>
>>>> 2. If no how can i get it for my device?
>>>>
>>>
>>>
>>> Exactly! the ./do-exofs format command will set this for you, as well as
>>> mkfs.exofs the
>>> FS for you.
>>>
>>>> 3. It is mandatory to use raid-driver (raid456) in the script? Why?
>>>>
>>>
>>>
>>> What? where? Rrrr you are right. this is a very old tree. Let me see
>>> if I have something newer to push. It should all load automatically now.
>>>
>>> But yes the dependency on raid456.ko is built in. Though if you use
>>> raid=0 on the format command line it will not be used in run time.
>>>
>>>>
>>>> The error is:
>>>> mount -t exofs -o
>>> osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000,_netdev /dev/osd0 /mnt/osd0/
>>>> mount: wrong fs type, bad option, bad superblock on /dev/osd0,
>>>> missing codepage or helper program, or other error
>>>> In some cases useful info is found in syslog - try
>>>> dmesg | tail or so
>>>>
>>>> dmesg says:
>>>> exofs: Unable to mount exofs on (null) pid=0x0 err=-22
>>>>
>>>
>>>
>>> Probably you forgot the "./do-exofs format" stage. Please read the script
>>> before, you might need to edit it. It is the stage that calls mkfs.exofs
>>> to make a new filesystem for you. (An OSD is a raw device that can host
>>> many filesystems each in it's own osd-partition.
>>>
>>>>
>>>> Thanks for answering my questions.
>>>
>>>
>>>> Regards
>>>> Johannes
>>>>
>>>
>>>
>>> Cheers
>>> Boaz
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to [email protected]
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> --
>> NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
>> Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to [email protected]
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> Hi,
> This crash has happened to me as well. I was not able to explicitly
> resolve it, but I have managed to set up a working pNFS object layout
> environment. I'll try to give the instructions I use for setting up
> the MDS, without using the do-* scripts, assuming you managed to set
> up the data server. (I'm assuming a fedora distro):
>
> setup the iscsi initiator:
>
> # yum -y install iscsi-initiator-utils
> # service iscsid start
> # modprobe osd
> # iscsiadm -m discovery -t sendtargets -p <your DS> --login
>
> on my setup, in some (most) cases, the MDS machine hangs when you
> power it off. I don't know why this happens.
>
> format and mount the OSD device:
>
> # modprobe raid456
This should not be needed exofs is dependent on ore, dependent on raid456
> # modprobe exofs
> # cd /your/open-osd/project/directory
> # LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 --dev=/dev/osd0
>
NO you must --format with a UUID at least once.
After the iscsiadm --login stage, there is an osd.ko print in dmesg
Do you see a print of the form:
OSD_NAME [%s]
where %s might be empty or it might have something
If there is something (not []) then maybe it was specified on the
osd-target (otgtd) command line and all is well. But if not then
it can never work.
> to me, this works without UUID. I had trouble in setting up exofs with
> RAID, but if you're not interested in RAID that will do. note that
> this setup is from the old tree (commit
> df9cc23ca0e15f2fc7f1d16a21acb87c63038f50), I don't know if and what
> have been pushed to it recently.
>
> note that if you use the --format flag you would have to supply a UUID.
>
You must do that at least once in the lifetime of the device or set
an osd_name at the otgtd command line.
> now you can mount the object file system. /dev/osd0 is seen as a char
> device and not a block device, so the system can't read its superblock
> and tell which file system is it. you have to explicitly specify exofs
> as the file system, and exofs will do the hard work.:
Whaoo I've been using ./do-exofs for ever and did not realize this. I should
fix that. Thanks It's now on my todo list.
>
> # mount -t exofs -o pid=0x10000 /dev/osd0 /mnt/pnfs
>
Are you sure you have an exofs FS at /mnt/pnfs ? please do an
# df -h /mnt/pnfs I want to see ?
Does it actually work? OK you know maybe it does. I can see this now.
If you have a single device it might work, I didn't realize this. I thought
the mkfs.exofs would not let you.
For sure if you have more then one device (pnfs right?) then it will not
let you, because the devices have strict order in the device table. Device
names are not reliable and may change from login to login. You need some
kind of device-id
>
Sorry guys for the lack of DOCs and proper trees. I neglected
that for a while now. No one cared.
Let me push some better tested trees.
Thanks
Boaz
On 05/15/2012 05:22 PM, Idan Kedar wrote:
> On Tue, May 15, 2012 at 4:42 PM, Boaz Harrosh <[email protected]> wrote:
<snip>
>> This should not be needed exofs is dependent on ore, dependent on raid456
>
> True, but when setting up the environment my goal was to have a
> working environment. If a bug is introduced to this dependence system
> and the module wouldn't be loaded, I would have to reluctantly spend
> time finding the problem, when I can actually live with such a bug if
> the workaround is simply loading the module manually. so as a policy,
> I always explicitly load the modules I need when setting up a pNFS
> environment.
I'm sorry about that, that was my mess. My intention was to make all these
dependencies totally transparent. As an implementation de-jur detail, not
user visible.
The reason I had it in script for a while was because in UML raid456.ko
had a bug, which probing it manually gave a 20% better chance of success
(Tree below has a patch that fixes that, on UML)
Please note that some versions of exofs need it and some don't. I have
removed it from my scripts, now.
<snip>
>> Are you sure you have an exofs FS at /mnt/pnfs ? please do an
>> # df -h /mnt/pnfs I want to see ?
>
> # df -hT /mnt/pnfs/
> Filesystem Type Size Used Avail Use% Mounted on
> /dev/osd0 exofs 55G 12G 44G 21% /mnt/pnfs
>
OK I understand that now, the weird single device case. I don't
promise it will continue to work in Future. As a rule all devices
most have a network-unique OSD_NAME.
>>
>> Does it actually work? OK you know maybe it does. I can see this now.
>> If you have a single device it might work, I didn't realize this. I thought
>> the mkfs.exofs would not let you.
>>
>> For sure if you have more then one device (pnfs right?) then it will not
>> let you, because the devices have strict order in the device table. Device
>> names are not reliable and may change from login to login. You need some
>> kind of device-id
> Indeed it is a pNFS setup. And indeed I've had trouble when using more
> than one OSD.
I use a better script system for the cluster case, now, that I never pushed
that makes all this easier.
For one not setting osdname= at --format would explain the above.
>
> By the way, several weeks I have tried setting up a RAID 5 environment
> with 8 OSDs, 1 mirror and RAID nesting. I then tried cloning and
> compiling the kernel tree over this pNFS-OSD-RAID. The result was that
> otgtd died, and I don't know why. It didn't dump core anywhere I could
> find and the only "log" it has - stdout - didn't give any useful info.
> I was going to inquire about this in a couple of weeks when I need to
> get this environment working, but since this issue came up, maybe we
> can somehow resolve it sooner.
8 OSDs with a mirror ? what was the mkfs.exofs command line you used?
And did you use one otgtd with 8 targets, or 8 targets (8 IP addresses)
with one target each, or a combination?
What is the otgtd platform? what file system? what HW and HD environment?
And yes otgtd has some instabilities.
There are two I can think off:
* Over xfs the --format command crashes the otgtd (aborted exit no
crash dump) Debugging welcome.
* When lots of pnfs clients do heavy writing to the same otgtd, it
times-out and disconnects.
At Panasas we have a watch-dog that reloads it in a loop.
I have only seen this on FreeBSD, in Linux it never happened
to me.
Please give me more details on what you did before it exited
like that.
In anyway I pushed a tree I tested with at:
git://git.open-osd.org/linux-open-osd.git
checkout the *merge_and_compile-3.3* branch. But in principal they are the
same:
fs/exofs - Added autologin support
fs/nfs/objlayout - Added autologin support
fs/nfsd - Same
fs/nfs - Few fixes that are in benny's tree are not in linux-open-osd
So it should all be the same. For a proper cluster setup you will probably
need my do-ect scripts which take a cluster descriptor file and does
generic loops on everything.
Thanks
Boaz
Hi Boaz,
> Datum: Tue, 15 May 2012 16:18:17 +0300
> Von: Boaz Harrosh <[email protected]>
> An: Johannes Schild <[email protected]>
> CC: [email protected], [email protected]
> Betreff: Re: Questions about Exofs
> On 05/15/2012 03:19 PM, Johannes Schild wrote:
>
> > Hi,
> >
> > thanks for your fast reply.
> >
> > But now i run into a nullpointer exception while mounting /dev/osd0
> >
> >
>
> > [ 329.919112] Modules linked in: raid456 async_raid6_recov async_pq
> > raid6_pq async_memcpy bnx2i cnic cxgb4i cxgb3i iscsi_tcp be2iscsi
> > iscsi_boot_sysfs uio cxgb4 libcxgbi cxgb3 mdio libiscsi_tcp libiscsi
> > scsi_transport_iscsi fuse ip6t_REJECT nf_conntrack_ipv6
> > nf_defrag_ipv6 xt_state nf_conntrack 8021q ip6table_filter ip6_tables
> > garp stp llc fcoe libfcoe libfc scsi_transport_fc scsi_tgt
> > snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus joydev
> > snd_seq snd_seq_device snd_pcm ppdev vmw_balloon floppy parport_pc
> > snd_timer snd pcnet32 microcode parport pcspkr mii snd_page_alloc
> > i2c_piix4 i2c_core shpchp uinput mptspi mptscsih mptbase
> > scsi_transport_spi [last unloaded: iscsi_tcp] [ 329.919133]
>
>
> One thing that worries me here is this: [last unloaded: iscsi_tcp]
>
> Actually it should be in the loaded bunch not unloaded. I have no idea how
> can that happen.
> Do you see any prints in dmsg regarding iscsi, before the crash?
I see output like this. Always "registered" no unloading execpt after the crash.
[ 4.713107] iscsi: registered transport (tcp)
#<some output removed>
[ 4.739465] iscsi: registered transport (cxgb3i)
#<some output removed>
[ 4.750756] iscsi: registered transport (cxgb4i)
#<some output removed>
[ 4.771300] iscsi: registered transport (bnx2i)
[ 4.781045] iscsi: registered transport (be2iscsi)
>
> > [ 329.919134] Pid: 2234, comm: mount Not tainted 3.3.0-pnfs #1 VMware,
> Inc. VMware Virtual Platform/440BX Desktop Reference Platform
> > [ 329.919137] RIP: 0010:[<ffffffff81259799>] [<ffffffff81259799>]
> exofs_free_sbi+0x59/0xa0
> > [ 329.919139] RSP: 0018:ffff88003c4e5bc8 EFLAGS: 00010246
> > [ 329.919141] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> 0000000000001795
> > [ 329.919142] RDX: 0000000000000000 RSI: 0000000000000046 RDI:
> ffff88003c4f5000
> > [ 329.919143] RBP: ffff88003c4e5bd8 R08: 0000000000000000 R09:
> 0000000000000000
> > [ 329.919144] R10: 0000000000000000 R11: 0000000000000001 R12:
> ffff88003c4f5000
> > [ 329.919145] R13: ffff88003c4e5d58 R14: 00000000fffffffb R15:
> ffff88003bd4e360
> > [ 329.919146] FS: 00007f68ce81d800(0000) GS:ffff88003fc00000(0000)
> knlGS:0000000000000000
> > [ 329.919148] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [ 329.919149] CR2: 0000000000000000 CR3: 000000003cbfd000 CR4:
> 00000000000406f0
> > [ 329.919152] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> > [ 329.919155] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> > [ 329.919156] Process mount (pid: 2234, threadinfo ffff88003c4e4000,
> task ffff88003aa42e40)
> > [ 329.919157] Stack:
> > [ 329.919158] ffff88003c4f5000 ffff88003c4f5800 ffff88003c4e5d18
> ffffffff8125a780
> > [ 329.919160] ffff88003c4e5c84 0000000081f27618 ffffffff81f27600
> ffff88003f80c288
> > [ 329.919162] ffff88003ddbeee0 0000000000000000 ffff88003c4e5c48
> ffffffff812c8433
> > [ 329.919164] Call Trace:
> > [ 329.919166] [<ffffffff8125a780>] exofs_fill_super+0x440/0x6e0
> > [ 329.919171] [<ffffffff812c8433>] ? idr_pre_get+0x53/0x90
> > [ 329.919173] [<ffffffff8125a340>] ?
> exofs_read_lookup_dev_table+0x480/0x480
> > [ 329.919176] [<ffffffff81180e18>] mount_nodev+0x58/0xb0
> > [ 329.919178] [<ffffffff812594ff>] exofs_mount+0x4f/0x80
> > [ 329.919179] [<ffffffff81181e13>] mount_fs+0x43/0x1b0
> > [ 329.919182] [<ffffffff8119b56a>] vfs_kern_mount+0x6a/0xf0
> > [ 329.919183] [<ffffffff8119bd44>] do_kern_mount+0x54/0x110
> > [ 329.919185] [<ffffffff8119d4b4>] do_mount+0x1a4/0x830
> > [ 329.919188] [<ffffffff8115da33>] ? alloc_pages_current+0xa3/0x110
> > [ 329.919191] [<ffffffff811229e4>] ? __get_free_pages+0x14/0x50
> > [ 329.919193] [<ffffffff8119d18a>] ? copy_mount_options+0x3a/0x170
> > [ 329.919194] [<ffffffff8119dc80>] sys_mount+0x90/0xe0
> > [ 329.919199] [<ffffffff81605aa9>] system_call_fastpath+0x16/0x1b
> > [ 329.919200] Code: 66 90 41 8b 84 24 70 03 00 00 83 eb 01 39 c3 72 50
> 41 8b 94 24 74 03 00 00 01 c2 39 d3 73 42 89 da 29 c2 49 8b 84 24 88 03 00
> 00 <48> 8b 04 d0 48 8b 38 48 85 ff 74 0c 48 c7 00 00 00 00 00 e8 6f
> > [ 329.919214] RIP [<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
> > [ 329.919216] RSP <ffff88003c4e5bc8>
> > [ 329.919217] CR2: 0000000000000000
> > [ 329.919220] ---[ end trace e5a3b1124e42d9e3 ]---
> >
> >
>
>
> could you please do:
> []$ gdb fs/exofs/exofs.ko
[root@ExB osd-repo]# gdb /root/pnfs-repo/fs/exofs/exofs.ko
GNU gdb (GDB) Fedora (7.3.50.20110722-13.fc16)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/pnfs-repo/fs/exofs/exofs.ko...done.
> Inside gdb
> > list *(exofs_free_sbi+0x59)
(gdb) list *(exofs_free_sbi+0x59)
0x47a9 is in exofs_free_sbi (include/scsi/osd_ore.h:83).
78 /* ore_comp_dev Recievies a logical device index */
79 static inline struct osd_dev *ore_comp_dev(
80 const struct ore_components *oc, unsigned i)
81 {
82 BUG_ON((i < oc->first_dev) || (oc->first_dev + oc->numdevs <= i));
83 return oc->ods[i - oc->first_dev]->od;
84 }
85
86 static inline void ore_comp_set_dev(
87 struct ore_components *oc, unsigned i, struct osd_dev *od)
> and also
> > list *(exofs_fill_super+0x440)
(gdb) list *(exofs_fill_super+0x440)
0x5850 is in exofs_fill_super (fs/exofs/super.c:847).
842 dput(sb->s_root);
843 sb->s_root = NULL;
844 goto free_sbi;
845 }
846
847 _exofs_print_device("Mounting", opts->dev_name,
848 ore_comp_dev(&sbi->oc, 0),
849 sbi->one_comp.obj.partition);
850 return 0;
851
(gdb)
i build exofs into the kernel, not as a module.
So i recompiled the kernel and got the output seen above
>
> > Is there any possibility to avoid these failure?
> >
>
>
> I've never had it lets try to resolve it.
>
> Could you enable CONFIG_EXOFS_DEBUG it's under:
> miscellaneous-filesystems/exofs in make xconfig
I enabled it.
> Then re-run everything send me the output
> []$ ./do-osd stop
[root@ExB osd-repo]# ./do-osd stop
/dev/osd0
FATAL: Module osd is builtin
Should it be a modul or doesn't matter?
> []$ ls /dev/osd*
[root@ExB osd-repo]# ls /dev/osd*
ls: cannot access /dev/osd*: No such file or directory
> []$ ./do-osd
[root@ExB osd-repo]# ./do-osd
iscsid.service - LSB: Starts and stops login iSCSI daemon.
Loaded: loaded (/etc/rc.d/init.d/iscsid)
Active: inactive (dead) since Wed, 16 May 2012 10:46:23 +0200; 3min 11s ago
Process: 2287 ExecStop=/etc/rc.d/init.d/iscsid stop (code=exited, status=0/SUCCESS)
Process: 1168 ExecStart=/etc/rc.d/init.d/iscsid start (code=exited, status=0/SUCCESS)
Main PID: 1213 (code=exited, status=0/SUCCESS)
CGroup: name=systemd:/system/iscsid.service
18446744072101122080
login into: 192.168.0.1:3260
192.168.0.1:3260,1 .root.var.osd-tgt.tgt-1.ExA
> []$ ls /dev/osd*
[root@ExB server]# ls /dev/os*
/dev/osd1
> []$ ./do-exofs format
> Send me the output of that
./do-exofs format
mkexofs_format >>>
osd stop? >>>
FATAL: Module osd is builtin
osd start? >>>
iscsid.service - LSB: Starts and stops login iSCSI daemon.
Loaded: loaded (/etc/rc.d/init.d/iscsid)
Active: inactive (dead) since Wed, 16 May 2012 10:46:23 +0200; 6min ago
Process: 2287 ExecStop=/etc/rc.d/init.d/iscsid stop (code=exited, status=0/SUCCESS)
Process: 1168 ExecStart=/etc/rc.d/init.d/iscsid start (code=exited, status=0/SUCCESS)
Main PID: 1213 (code=exited, status=0/SUCCESS)
CGroup: name=systemd:/system/iscsid.service
18446744072101122080
login into: 192.168.0.1:3260
192.168.0.1:3260,1 .root.var.osd-tgt.tgt-1.ExA
Logging in to [iface: default, target: .root.var.osd-tgt.tgt-1.ExA, portal: 192.168.0.1,3260] (multiple)
Login to [iface: default, target: .root.var.osd-tgt.tgt-1.ExA, portal: 192.168.0.1,3260] successful.
> []$ ./do-exofs start
> Send me the dmesg output of this stage, or if not too big
> the dmesg output of from before ./do-osd <1>
I pushed it on nopaste:
http://nopaste.info/cd3c6f9141.html
>
> > Just now i am using the 3.3.0 kernel from the linux-pnfs repository.
> >
>
>
> When compiling the Kernel, Did you enable CONFIG_PNFSD ?
> (That is the pNFSD Server Kernel Support)
No pNFSD Server support wasn't enabled, i recompiled and activate it
> What platform are you using? Distro + ARCH ?
Iam experimenting with Fedora 16 (3.3.0 pnfs kernel) and arch x86_64
>
> This should be a good tree. Though usually I use an
> open-osd.org/linux-open-osd.git based tree. But I don't see much
> difference. Let me have a look if there is something grossly
> missing.
>
> >
> > Thanks in adavance
> >
>
>
> Sorry for your trouble I have neglected the public trees for a
> while. Let me see if I can push a more tested tree up on
> open-osd.org.
>
> > Cheers
> > Johannes
> >
>
>
> Thanks
> Boaz
>
> > -------- Original-Nachricht --------
> >> Datum: Tue, 15 May 2012 12:48:08 +0300
> >> Von: Boaz Harrosh <[email protected]>
> >> An: Johannes Schild <[email protected]>
> >> CC: [email protected], open-osd <[email protected]>
> >> Betreff: Re: Questions about Exofs
> >
> >> On 05/15/2012 12:03 PM, Johannes Schild wrote:
> >>
> >>> Hi,
> >>>
> >>> i try to set up the object-based layout with pnfs.
> >>> I am using the OSC-OSD as target. It works well and i can login.
> >>> Trying the scripts "do-*" from git://git.open-osd.org/open-osd.git the
> >> do-osd script works for me. The do-exofs doesnt, here i have some
> question:
> >>>
> >>> 1. Is the UUID in the do-exofs script optional?
> >>>
> >>
> >>
> >> No it is not optional
> >>
> >>> 2. If no how can i get it for my device?
> >>>
> >>
> >>
> >> Exactly! the ./do-exofs format command will set this for you, as well
> as
> >> mkfs.exofs the
> >> FS for you.
> >>
> >>> 3. It is mandatory to use raid-driver (raid456) in the script? Why?
> >>>
> >>
> >>
> >> What? where? Rrrr you are right. this is a very old tree. Let me see
> >> if I have something newer to push. It should all load automatically
> now.
> >>
> >> But yes the dependency on raid456.ko is built in. Though if you use
> >> raid=0 on the format command line it will not be used in run time.
> >>
> >>>
> >>> The error is:
> >>> mount -t exofs -o
> >> osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000,_netdev
> /dev/osd0 /mnt/osd0/
> >>> mount: wrong fs type, bad option, bad superblock on /dev/osd0,
> >>> missing codepage or helper program, or other error
> >>> In some cases useful info is found in syslog - try
> >>> dmesg | tail or so
> >>>
> >>> dmesg says:
> >>> exofs: Unable to mount exofs on (null) pid=0x0 err=-22
> >>>
> >>
> >>
> >> Probably you forgot the "./do-exofs format" stage. Please read the
> script
> >> before, you might need to edit it. It is the stage that calls
> mkfs.exofs
> >> to make a new filesystem for you. (An OSD is a raw device that can host
> >> many filesystems each in it's own osd-partition.
> >>
> >>>
> >>> Thanks for answering my questions.
> >>
> >>
> >>> Regards
> >>> Johannes
> >>>
> >>
> >>
> >> Cheers
> >> Boaz
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >> the body of a message to [email protected]
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
>
Thanks for your efforts
Johannes
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
On Tue, May 15, 2012 at 4:42 PM, Boaz Harrosh <[email protected]> wrote:
> On 05/15/2012 04:09 PM, Idan Kedar wrote:
>
>> On Tue, May 15, 2012 at 3:19 PM, Johannes Schild <[email protected]> wrote:
>>> Hi,
>>>
>>> thanks for your fast reply.
>>>
>>> But now i run into a nullpointer exception while mounting /dev/osd0
>>>
>>>
>>> [ ?329.919112] Modules linked in: raid456 async_raid6_recov async_pq raid6_pq async_memcpy bnx2i cnic cxgb4i cxgb3i iscsi_tcp be2iscsi iscsi_boot_sysfs uio cxgb4 libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi fuse ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack 8021q ip6table_filter ip6_tables garp stp llc fcoe libfcoe libfc scsi_transport_fc scsi_tgt snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus joydev snd_seq snd_seq_device snd_pcm ppdev vmw_balloon floppy parport_pc snd_timer snd pcnet32 microcode parport pcspkr mii snd_page_alloc i2c_piix4 i2c_core shpchp uinput mptspi mptscsih mptbase scsi_transport_spi [last unloaded: iscsi_tcp]
>>> [ ?329.919133]
>>> [ ?329.919134] Pid: 2234, comm: mount Not tainted 3.3.0-pnfs #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
>>> [ ?329.919137] RIP: 0010:[<ffffffff81259799>] ?[<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
>>> [ ?329.919139] RSP: 0018:ffff88003c4e5bc8 ?EFLAGS: 00010246
>>> [ ?329.919141] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000001795
>>> [ ?329.919142] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88003c4f5000
>>> [ ?329.919143] RBP: ffff88003c4e5bd8 R08: 0000000000000000 R09: 0000000000000000
>>> [ ?329.919144] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88003c4f5000
>>> [ ?329.919145] R13: ffff88003c4e5d58 R14: 00000000fffffffb R15: ffff88003bd4e360
>>> [ ?329.919146] FS: ?00007f68ce81d800(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
>>> [ ?329.919148] CS: ?0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [ ?329.919149] CR2: 0000000000000000 CR3: 000000003cbfd000 CR4: 00000000000406f0
>>> [ ?329.919152] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> [ ?329.919155] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> [ ?329.919156] Process mount (pid: 2234, threadinfo ffff88003c4e4000, task ffff88003aa42e40)
>>> [ ?329.919157] Stack:
>>> [ ?329.919158] ?ffff88003c4f5000 ffff88003c4f5800 ffff88003c4e5d18 ffffffff8125a780
>>> [ ?329.919160] ?ffff88003c4e5c84 0000000081f27618 ffffffff81f27600 ffff88003f80c288
>>> [ ?329.919162] ?ffff88003ddbeee0 0000000000000000 ffff88003c4e5c48 ffffffff812c8433
>>> [ ?329.919164] Call Trace:
>>> [ ?329.919166] ?[<ffffffff8125a780>] exofs_fill_super+0x440/0x6e0
>>> [ ?329.919171] ?[<ffffffff812c8433>] ? idr_pre_get+0x53/0x90
>>> [ ?329.919173] ?[<ffffffff8125a340>] ? exofs_read_lookup_dev_table+0x480/0x480
>>> [ ?329.919176] ?[<ffffffff81180e18>] mount_nodev+0x58/0xb0
>>> [ ?329.919178] ?[<ffffffff812594ff>] exofs_mount+0x4f/0x80
>>> [ ?329.919179] ?[<ffffffff81181e13>] mount_fs+0x43/0x1b0
>>> [ ?329.919182] ?[<ffffffff8119b56a>] vfs_kern_mount+0x6a/0xf0
>>> [ ?329.919183] ?[<ffffffff8119bd44>] do_kern_mount+0x54/0x110
>>> [ ?329.919185] ?[<ffffffff8119d4b4>] do_mount+0x1a4/0x830
>>> [ ?329.919188] ?[<ffffffff8115da33>] ? alloc_pages_current+0xa3/0x110
>>> [ ?329.919191] ?[<ffffffff811229e4>] ? __get_free_pages+0x14/0x50
>>> [ ?329.919193] ?[<ffffffff8119d18a>] ? copy_mount_options+0x3a/0x170
>>> [ ?329.919194] ?[<ffffffff8119dc80>] sys_mount+0x90/0xe0
>>> [ ?329.919199] ?[<ffffffff81605aa9>] system_call_fastpath+0x16/0x1b
>>> [ ?329.919200] Code: 66 90 41 8b 84 24 70 03 00 00 83 eb 01 39 c3 72 50 41 8b 94 24 74 03 00 00 01 c2 39 d3 73 42 89 da 29 c2 49 8b 84 24 88 03 00 00 <48> 8b 04 d0 48 8b 38 48 85 ff 74 0c 48 c7 00 00 00 00 00 e8 6f
>>> [ ?329.919214] RIP ?[<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
>>> [ ?329.919216] ?RSP <ffff88003c4e5bc8>
>>> [ ?329.919217] CR2: 0000000000000000
>>> [ ?329.919220] ---[ end trace e5a3b1124e42d9e3 ]---
>>>
>>>
>>> Is there any possibility to avoid these failure?
>>>
>>> Just now i am using the 3.3.0 kernel from the linux-pnfs repository.
>>>
>>>
>>> Thanks in adavance
>>>
>>> Cheers
>>> Johannes
>>>
>>> -------- Original-Nachricht --------
>>>> Datum: Tue, 15 May 2012 12:48:08 +0300
>>>> Von: Boaz Harrosh <[email protected]>
>>>> An: Johannes Schild <[email protected]>
>>>> CC: [email protected], open-osd <[email protected]>
>>>> Betreff: Re: Questions about Exofs
>>>
>>>> On 05/15/2012 12:03 PM, Johannes Schild wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> i try to set up the object-based layout with pnfs.
>>>>> I am using the OSC-OSD as target. It works well and i can login.
>>>>> Trying the scripts "do-*" from git://git.open-osd.org/open-osd.git the
>>>> do-osd script works for me. The do-exofs doesnt, here i have some question:
>>>>>
>>>>> 1. Is the UUID in the do-exofs script optional?
>>>>>
>>>>
>>>>
>>>> No it is not optional
>>>>
>>>>> 2. If no how can i get it for my device?
>>>>>
>>>>
>>>>
>>>> Exactly! the ./do-exofs format command will set this for you, as well as
>>>> mkfs.exofs the
>>>> FS for you.
>>>>
>>>>> 3. It is mandatory to use raid-driver (raid456) in the script? Why?
>>>>>
>>>>
>>>>
>>>> What? where? Rrrr you are right. this is a very old tree. Let me see
>>>> if I have something newer to push. It should all load automatically now.
>>>>
>>>> But yes the dependency on raid456.ko is built in. Though if you use
>>>> raid=0 on the format command line it will not be used in run time.
>>>>
>>>>>
>>>>> The error is:
>>>>> mount -t exofs -o
>>>> osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000,_netdev /dev/osd0 /mnt/osd0/
>>>>> mount: wrong fs type, bad option, bad superblock on /dev/osd0,
>>>>> ? ? ? ?missing codepage or helper program, or other error
>>>>> ? ? ? ?In some cases useful info is found in syslog - try
>>>>> ? ? ? ?dmesg | tail ?or so
>>>>>
>>>>> dmesg says:
>>>>> exofs: Unable to mount exofs on (null) pid=0x0 err=-22
>>>>>
>>>>
>>>>
>>>> Probably you forgot the "./do-exofs format" stage. Please read the script
>>>> before, you might need to edit it. It is the stage that calls mkfs.exofs
>>>> to make a new filesystem for you. (An OSD is a raw device that can host
>>>> many filesystems each in it's own osd-partition.
>>>>
>>>>>
>>>>> Thanks for answering my questions.
>>>>
>>>>
>>>>> Regards
>>>>> Johannes
>>>>>
>>>>
>>>>
>>>> Cheers
>>>> Boaz
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>> the body of a message to [email protected]
>>>> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
>>> Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to [email protected]
>>> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
>>
>> Hi,
>> This crash has happened to me as well. I was not able to explicitly
>> resolve it, but I have managed to set up a working pNFS object layout
>> environment. I'll try to give the instructions I use for setting up
>> the MDS, without using the do-* scripts, assuming you managed to set
>> up the data server. (I'm assuming a fedora distro):
>>
>> setup the iscsi initiator:
>>
>> # yum -y install iscsi-initiator-utils
>> # service iscsid start
>> # modprobe osd
>> # iscsiadm -m discovery -t sendtargets -p <your DS> --login
>>
>> on my setup, in some (most) cases, the MDS machine hangs when you
>> power it off. I don't know why this happens.
>>
>> format and mount the OSD device:
>>
>> # modprobe raid456
>
>
> This should not be needed exofs is dependent on ore, dependent on raid456
True, but when setting up the environment my goal was to have a
working environment. If a bug is introduced to this dependence system
and the module wouldn't be loaded, I would have to reluctantly spend
time finding the problem, when I can actually live with such a bug if
the workaround is simply loading the module manually. so as a policy,
I always explicitly load the modules I need when setting up a pNFS
environment.
>
>> # modprobe exofs
>
>> # cd /your/open-osd/project/directory
>> # LD_LIBRARY_PATH=lib ./usr/mkfs.exofs --pid=0x10000 --dev=/dev/osd0
>>
>
>
> NO you must --format with a UUID at least once.
>
> After the iscsiadm --login stage, there is an osd.ko print in dmesg
> Do you see a print of the form:
> ? ? ? ?OSD_NAME ? ? ? ? ? ? ? [%s]
> where %s might be empty or it might have something
> If there is something (not []) then maybe it was specified on the
> osd-target (otgtd) command line and all is well. But if not then
> it can never work.
>
>> to me, this works without UUID. I had trouble in setting up exofs with
>> RAID, but if you're not interested in RAID that will do. note that
>> this setup is from the old tree (commit
>> df9cc23ca0e15f2fc7f1d16a21acb87c63038f50), I don't know if and what
>> have been pushed to it recently.
>>
>> note that if you use the --format flag you would have to supply a UUID.
>>
>
>
> You must do that at least once in the lifetime of the device or set
> an osd_name at the otgtd command line.
>
>> now you can mount the object file system. /dev/osd0 is seen as a char
>> device and not a block device, so the system can't read its superblock
>> and tell which file system is it. you have to explicitly specify exofs
>> as the file system, and exofs will do the hard work.:
>
>
> Whaoo I've been using ./do-exofs for ever and did not realize this. I should
> fix that. Thanks It's now on my todo list.
You're welcome :)
>
>>
>> # mount -t exofs -o pid=0x10000 /dev/osd0 /mnt/pnfs
>>
>
>
> Are you sure you have an exofs FS at /mnt/pnfs ? please do an
> # df -h /mnt/pnfs I want to see ?
# df -hT /mnt/pnfs/
Filesystem Type Size Used Avail Use% Mounted on
/dev/osd0 exofs 55G 12G 44G 21% /mnt/pnfs
>
> Does it actually work? OK you know maybe it does. I can see this now.
> If you have a single device it might work, I didn't realize this. I thought
> the mkfs.exofs would not let you.
>
> For sure if you have more then one device (pnfs right?) then it will not
> let you, because the devices have strict order in the device table. Device
> names are not reliable and may change from login to login. You need some
> kind of device-id
Indeed it is a pNFS setup. And indeed I've had trouble when using more
than one OSD.
By the way, several weeks I have tried setting up a RAID 5 environment
with 8 OSDs, 1 mirror and RAID nesting. I then tried cloning and
compiling the kernel tree over this pNFS-OSD-RAID. The result was that
otgtd died, and I don't know why. It didn't dump core anywhere I could
find and the only "log" it has - stdout - didn't give any useful info.
I was going to inquire about this in a couple of weeks when I need to
get this environment working, but since this issue came up, maybe we
can somehow resolve it sooner.
>
>>
>
>
> Sorry guys for the lack of DOCs and proper trees. I neglected
> that for a while now. No one cared.
>
> Let me push some better tested trees.
>
> Thanks
> Boaz
Thanks,
IdanK
On 05/16/2012 12:00 PM, Johannes Schild wrote:
> Hi Boaz,
<>
>> Do you see any prints in dmsg regarding iscsi, before the crash?
>
> I see output like this. Always "registered" no unloading execpt after the crash.
>
> [ 4.713107] iscsi: registered transport (tcp)
> #<some output removed>
> [ 4.739465] iscsi: registered transport (cxgb3i)
> #<some output removed>
> [ 4.750756] iscsi: registered transport (cxgb4i)
> #<some output removed>
> [ 4.771300] iscsi: registered transport (bnx2i)
> [ 4.781045] iscsi: registered transport (be2iscsi)
>
<>
>> could you please do:
>> []$ gdb fs/exofs/exofs.ko
>
> [root@ExB osd-repo]# gdb /root/pnfs-repo/fs/exofs/exofs.ko
> GNU gdb (GDB) Fedora (7.3.50.20110722-13.fc16)
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /root/pnfs-repo/fs/exofs/exofs.ko...done.
>
>> Inside gdb
>>> list *(exofs_free_sbi+0x59)
>
> (gdb) list *(exofs_free_sbi+0x59)
> 0x47a9 is in exofs_free_sbi (include/scsi/osd_ore.h:83).
> 78 /* ore_comp_dev Recievies a logical device index */
> 79 static inline struct osd_dev *ore_comp_dev(
> 80 const struct ore_components *oc, unsigned i)
> 81 {
> 82 BUG_ON((i < oc->first_dev) || (oc->first_dev + oc->numdevs <= i));
> 83 return oc->ods[i - oc->first_dev]->od;
> 84 }
> 85
> 86 static inline void ore_comp_set_dev(
> 87 struct ore_components *oc, unsigned i, struct osd_dev *od)
>
>> and also
>>> list *(exofs_fill_super+0x440)
>
> (gdb) list *(exofs_fill_super+0x440)
> 0x5850 is in exofs_fill_super (fs/exofs/super.c:847).
> 842 dput(sb->s_root);
> 843 sb->s_root = NULL;
> 844 goto free_sbi;
> 845 }
> 846
> 847 _exofs_print_device("Mounting", opts->dev_name,
> 848 ore_comp_dev(&sbi->oc, 0),
> 849 sbi->one_comp.obj.partition);
> 850 return 0;
> 851
> (gdb)
>
OK I understand we are _exofs_print_device an array that does
not exists yet.
>>
>> Could you enable CONFIG_EXOFS_DEBUG it's under:
>> miscellaneous-filesystems/exofs in make xconfig
>
> I enabled it.
>
>> Then re-run everything send me the output
>> []$ ./do-osd stop
>
> [root@ExB osd-repo]# ./do-osd stop
> /dev/osd0
> FATAL: Module osd is builtin
>
> Should it be a modul or doesn't matter?
>
It should be fine. scripts expect it as a module.
>> []$ ls /dev/osd*
>
> [root@ExB osd-repo]# ls /dev/osd*
> ls: cannot access /dev/osd*: No such file or directory
>
>> []$ ./do-osd
>
> [root@ExB osd-repo]# ./do-osd
> iscsid.service - LSB: Starts and stops login iSCSI daemon.
> Loaded: loaded (/etc/rc.d/init.d/iscsid)
> Active: inactive (dead) since Wed, 16 May 2012 10:46:23 +0200; 3min 11s ago
> Process: 2287 ExecStop=/etc/rc.d/init.d/iscsid stop (code=exited, status=0/SUCCESS)
> Process: 1168 ExecStart=/etc/rc.d/init.d/iscsid start (code=exited, status=0/SUCCESS)
> Main PID: 1213 (code=exited, status=0/SUCCESS)
> CGroup: name=systemd:/system/iscsid.service
> 18446744072101122080
> login into: 192.168.0.1:3260
> 192.168.0.1:3260,1 .root.var.osd-tgt.tgt-1.ExA
>
>> []$ ls /dev/osd*
>
> [root@ExB server]# ls /dev/os*
> /dev/osd1
>
/dev/osd1 interesting. make sure your scripts are using /dev/osd1.
I suspect this is an artifact of the last games. On a clean reboot
a single device should be /dev/osd0. The scripts expect that.
>> []$ ./do-exofs format
>> Send me the output of that
>
> ./do-exofs format
> mkexofs_format >>>
No output from the format command? that is not good. mkfs.exofs is
very bad in not saying anything when failing.
Probably because it was formatting /dev/osd0 and we have /dev/osd1 only
> osd stop? >>>
> FATAL: Module osd is builtin
> osd start? >>>
> iscsid.service - LSB: Starts and stops login iSCSI daemon.
> Loaded: loaded (/etc/rc.d/init.d/iscsid)
> Active: inactive (dead) since Wed, 16 May 2012 10:46:23 +0200; 6min ago
> Process: 2287 ExecStop=/etc/rc.d/init.d/iscsid stop (code=exited, status=0/SUCCESS)
> Process: 1168 ExecStart=/etc/rc.d/init.d/iscsid start (code=exited, status=0/SUCCESS)
> Main PID: 1213 (code=exited, status=0/SUCCESS)
> CGroup: name=systemd:/system/iscsid.service
> 18446744072101122080
> login into: 192.168.0.1:3260
> 192.168.0.1:3260,1 .root.var.osd-tgt.tgt-1.ExA
> Logging in to [iface: default, target: .root.var.osd-tgt.tgt-1.ExA, portal: 192.168.0.1,3260] (multiple)
> Login to [iface: default, target: .root.var.osd-tgt.tgt-1.ExA, portal: 192.168.0.1,3260] successful.
>
>> []$ ./do-exofs start
>> Send me the dmesg output of this stage, or if not too big
>> the dmesg output of from before ./do-osd <1>
>
> I pushed it on nopaste:
> http://nopaste.info/cd3c6f9141.html
>
in the dmesg I see:
[ 2516.994781] exofs @parse_options:88: parse_options osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000
[ 2516.994808] osd @_mach_odi:261: found device sysid_len=0 osdname=36
[ 2516.994816] osd @_osdv2_req_encode_common:617: OSDv2 execute opcode 0x8885
[ 2516.994831] osd @_init_blk_request:1616: or=ffff880020d7ec00 has_in=1 has_out=0 => 0, ffff88003bbf8a10
the very first read below fails. This is the first read from super-block object.
Here it gets an -5 (-EIO) if it was an osd-target error you would have
a scsi-sense printout so it means it is a communication problem.
[ 2516.996034] exofs @exofs_read_kern:245: osd_execute_request() => -5
[ 2516.996041] exofs: Unable to mount exofs on (null) pid=0x10000 err=-5
This crash below I should fix. Code is not dealing properly with the IO error
and continues to try and dmesg-print an array that does not exist yet.
I will fix that.
[ 2516.996106] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 2516.996111] IP: [<ffffffffa033c779>] exofs_free_sbi+0x59/0xa0 [exofs]
But the problem still remains why do we get IO errors from iscsi?
Later we have:
[ 3241.802074] connection1:0: detected conn error (1020)
disconnect. Do you see some prints at the otgtd side.
If you use the ./up script it might rederect these to a log file
do "./up log"
[ 3398.831629] Chelsio T3 iSCSI Driver cxgb3i v2.0.0 (Jun. 2010)
[ 3398.831919] iscsi: registered transport (cxgb3i)
[ 3398.836776] Chelsio T4 iSCSI Driver cxgb4i v0.9.1 (Aug. 2010)
[ 3398.836996] iscsi: registered transport (cxgb4i)
[ 3398.841397] cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.8 (Jan 3, 2012)
[ 3398.845267] Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.0.3 (Jun 15, 2011)
[ 3398.845475] iscsi: registered transport (bnx2i)
[ 3400.201828] scsi4 : iSCSI Initiator over TCP/IP
[ 3400.715101] scsi 4:0:0:0: Object storage IET OSD 0001 PQ: 0 ANSI: 5
[ 3400.718038] osd @__detect_osd:359: start scsi_test_unit_ready ffff880020db3800 ffff880020dfa000 ffff88003974aca0
Right after the crash. So iscsi unloaded and loaded. There was a disconnect.
We must investigate why iscsi has communication problems?
the "192.168.0.1:3260" above is that your host's IP? You are running the otgtd on
the host and exofs in VM? That's good that's what I use all the time.
If you have time you should do two experiments.
1. Please run the "./do-osd test" test. send me the output.
It runs a user mode test of the osd device and does some
very basic communications.
Note that it will wipe your OSD and you will need to ./do-exofs format again
after it.
2. on the osd-target side you probably ran ./up. the otgtd also supports
none-osd regular disk-devices. Could you set up a regular disk
backbend as well. Look into "man tgtadm" on how to add a second
disk target.
Once you login to the target you will see a new /dev/sdX device
try to dd into it, and also mkfs and mount an ext FS on it.
Or else investigate why there are iscsi communication problems.
>
>
>>
>>> Just now i am using the 3.3.0 kernel from the linux-pnfs repository.
>>>
That's perfect it should have everything.
>>
>>
>> When compiling the Kernel, Did you enable CONFIG_PNFSD ?
>> (That is the pNFSD Server Kernel Support)
>
> No pNFSD Server support wasn't enabled, i recompiled and activate it
>
It's fine for this stage you don't need it
>
>
>
>> What platform are you using? Distro + ARCH ?
>
> Iam experimenting with Fedora 16 (3.3.0 pnfs kernel) and arch x86_64
>
I use that here too
>
> Thanks for your efforts
> Johannes
Hope that helps. Thanks for the report we got a bug fix
Boaz
Hi,
thanks for your fast reply.
But now i run into a nullpointer exception while mounting /dev/osd0
[ 329.919112] Modules linked in: raid456 async_raid6_recov async_pq raid6_pq async_memcpy bnx2i cnic cxgb4i cxgb3i iscsi_tcp be2iscsi iscsi_boot_sysfs uio cxgb4 libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi fuse ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack 8021q ip6table_filter ip6_tables garp stp llc fcoe libfcoe libfc scsi_transport_fc scsi_tgt snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus joydev snd_seq snd_seq_device snd_pcm ppdev vmw_balloon floppy parport_pc snd_timer snd pcnet32 microcode parport pcspkr mii snd_page_alloc i2c_piix4 i2c_core shpchp uinput mptspi mptscsih mptbase scsi_transport_spi [last unloaded: iscsi_tcp]
[ 329.919133]
[ 329.919134] Pid: 2234, comm: mount Not tainted 3.3.0-pnfs #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[ 329.919137] RIP: 0010:[<ffffffff81259799>] [<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
[ 329.919139] RSP: 0018:ffff88003c4e5bc8 EFLAGS: 00010246
[ 329.919141] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000001795
[ 329.919142] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88003c4f5000
[ 329.919143] RBP: ffff88003c4e5bd8 R08: 0000000000000000 R09: 0000000000000000
[ 329.919144] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88003c4f5000
[ 329.919145] R13: ffff88003c4e5d58 R14: 00000000fffffffb R15: ffff88003bd4e360
[ 329.919146] FS: 00007f68ce81d800(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[ 329.919148] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 329.919149] CR2: 0000000000000000 CR3: 000000003cbfd000 CR4: 00000000000406f0
[ 329.919152] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 329.919155] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 329.919156] Process mount (pid: 2234, threadinfo ffff88003c4e4000, task ffff88003aa42e40)
[ 329.919157] Stack:
[ 329.919158] ffff88003c4f5000 ffff88003c4f5800 ffff88003c4e5d18 ffffffff8125a780
[ 329.919160] ffff88003c4e5c84 0000000081f27618 ffffffff81f27600 ffff88003f80c288
[ 329.919162] ffff88003ddbeee0 0000000000000000 ffff88003c4e5c48 ffffffff812c8433
[ 329.919164] Call Trace:
[ 329.919166] [<ffffffff8125a780>] exofs_fill_super+0x440/0x6e0
[ 329.919171] [<ffffffff812c8433>] ? idr_pre_get+0x53/0x90
[ 329.919173] [<ffffffff8125a340>] ? exofs_read_lookup_dev_table+0x480/0x480
[ 329.919176] [<ffffffff81180e18>] mount_nodev+0x58/0xb0
[ 329.919178] [<ffffffff812594ff>] exofs_mount+0x4f/0x80
[ 329.919179] [<ffffffff81181e13>] mount_fs+0x43/0x1b0
[ 329.919182] [<ffffffff8119b56a>] vfs_kern_mount+0x6a/0xf0
[ 329.919183] [<ffffffff8119bd44>] do_kern_mount+0x54/0x110
[ 329.919185] [<ffffffff8119d4b4>] do_mount+0x1a4/0x830
[ 329.919188] [<ffffffff8115da33>] ? alloc_pages_current+0xa3/0x110
[ 329.919191] [<ffffffff811229e4>] ? __get_free_pages+0x14/0x50
[ 329.919193] [<ffffffff8119d18a>] ? copy_mount_options+0x3a/0x170
[ 329.919194] [<ffffffff8119dc80>] sys_mount+0x90/0xe0
[ 329.919199] [<ffffffff81605aa9>] system_call_fastpath+0x16/0x1b
[ 329.919200] Code: 66 90 41 8b 84 24 70 03 00 00 83 eb 01 39 c3 72 50 41 8b 94 24 74 03 00 00 01 c2 39 d3 73 42 89 da 29 c2 49 8b 84 24 88 03 00 00 <48> 8b 04 d0 48 8b 38 48 85 ff 74 0c 48 c7 00 00 00 00 00 e8 6f
[ 329.919214] RIP [<ffffffff81259799>] exofs_free_sbi+0x59/0xa0
[ 329.919216] RSP <ffff88003c4e5bc8>
[ 329.919217] CR2: 0000000000000000
[ 329.919220] ---[ end trace e5a3b1124e42d9e3 ]---
Is there any possibility to avoid these failure?
Just now i am using the 3.3.0 kernel from the linux-pnfs repository.
Thanks in adavance
Cheers
Johannes
-------- Original-Nachricht --------
> Datum: Tue, 15 May 2012 12:48:08 +0300
> Von: Boaz Harrosh <[email protected]>
> An: Johannes Schild <[email protected]>
> CC: [email protected], open-osd <[email protected]>
> Betreff: Re: Questions about Exofs
> On 05/15/2012 12:03 PM, Johannes Schild wrote:
>
> > Hi,
> >
> > i try to set up the object-based layout with pnfs.
> > I am using the OSC-OSD as target. It works well and i can login.
> > Trying the scripts "do-*" from git://git.open-osd.org/open-osd.git the
> do-osd script works for me. The do-exofs doesnt, here i have some question:
> >
> > 1. Is the UUID in the do-exofs script optional?
> >
>
>
> No it is not optional
>
> > 2. If no how can i get it for my device?
> >
>
>
> Exactly! the ./do-exofs format command will set this for you, as well as
> mkfs.exofs the
> FS for you.
>
> > 3. It is mandatory to use raid-driver (raid456) in the script? Why?
> >
>
>
> What? where? Rrrr you are right. this is a very old tree. Let me see
> if I have something newer to push. It should all load automatically now.
>
> But yes the dependency on raid456.ko is built in. Though if you use
> raid=0 on the format command line it will not be used in run time.
>
> >
> > The error is:
> > mount -t exofs -o
> osdname=d2683732-c906-4ee1-9dbd-c10c27bb40df,pid=0x10000,_netdev /dev/osd0 /mnt/osd0/
> > mount: wrong fs type, bad option, bad superblock on /dev/osd0,
> > missing codepage or helper program, or other error
> > In some cases useful info is found in syslog - try
> > dmesg | tail or so
> >
> > dmesg says:
> > exofs: Unable to mount exofs on (null) pid=0x0 err=-22
> >
>
>
> Probably you forgot the "./do-exofs format" stage. Please read the script
> before, you might need to edit it. It is the stage that calls mkfs.exofs
> to make a new filesystem for you. (An OSD is a raw device that can host
> many filesystems each in it's own osd-partition.
>
> >
> > Thanks for answering my questions.
>
>
> > Regards
> > Johannes
> >
>
>
> Cheers
> Boaz
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a