Hi,
I am trying to host KVM machines on an NFSoRDMA mount.
This works:
-drive file=/mnt/vm001.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
This Doesn't!
-drive file=/mnt/vm001.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
Any ideas why this could be? I have tried various kernel / ofed combinations.
Thanks,
Andrew
dmesg:
vda:
vda: vda1
vda: vda1
vda: vda1 vda2
vda: vda1 vda2
vda: vda1 vda2 vda3
vda: vda1 vda2 vda3
vda: vda1 vda2 vda3 vda4 < >
vda: vda1 vda2 vda3 vda4 < >
vda: vda1 vda2 vda3 vda4 < vda5 >
vda: vda1 vda2 vda3 vda4 < vda5 >
vda: vda1 vda2 vda3 vda4 < vda5 vda6 >
vda: vda1 vda2 vda3 vda4 < vda5 vda6 >
Adding 15998968k swap on /dev/vda5. Priority:-1 extents:1 across:15998968k
EXT3-fs (vda1): error: can't find ext3 filesystem on dev vda1.
/var/log/messages
Aug 31 18:53:46 (none) node-installer: Mounting disks.
Aug 31 18:53:46 (none) node-installer: Updating device status: mounting disks
Aug 31 18:53:46 (none) node-installer: Detecting device '/dev/sda': not found
Aug 31 18:53:46 (none) node-installer: Detecting device '/dev/hda': not found
Aug 31 18:53:46 (none) node-installer: Detecting device '/dev/vda': found
Aug 31 18:53:46 (none) node-installer: swapon /dev/vda5
Aug 31 18:53:46 (none) node-installer: mkdir -p /localdisk/
Aug 31 18:53:46 (none) kernel: Adding 15998968k swap on /dev/vda5. Priority:-1 extents:1 across:15998968k
Aug 31 18:53:46 (none) node-installer: Mounting /dev/vda1 on /localdisk/
Aug 31 18:53:46 (none) node-installer: mount -t ext3 -o defaults,noatime,nodiratime /dev/vda1 /localdisk/
Aug 31 18:53:46 (none) node-installer: mount: wrong fs type, bad option, bad superblock on /dev/vda1,
Aug 31 18:53:46 (none) node-installer: missing codepage or helper program, or other error
Aug 31 18:53:46 (none) node-installer: In some cases useful info is found in syslog - try
Aug 31 18:53:46 (none) node-installer: dmesg | tail or so
Aug 31 18:53:46 (none) node-installer:
Aug 31 18:53:46 (none) node-installer: Command failed.
Aug 31 18:53:46 (none) node-installer: Running: "mount -t ext3 -o defaults,noatime,nodiratime /dev/vda1 /localdisk/" failed:
Aug 31 18:53:46 (none) node-installer: Non zero exit code: 32
Aug 31 18:53:46 (none) kernel: EXT3-fs (vda1): error: can't find ext3 filesystem on dev vda1.
Aug 31 18:53:46 (none) node-installer: Failed to mount disks. (Exit code 12, signal 0)
Aug 31 18:53:46 (none) node-installer: There was a fatal problem. This node can not be installed until the problem is corrected.
Aug 31 18:53:46 (none) node-installer: The error was: failed to mount disks
Aug 31 18:53:46 (none) node-installer: Updating device status: failed to mount disks
Aug 31 18:53:28 (none) node-installer: Creating primary partition /dev/vda3.
Aug 31 18:53:28 (none) node-installer: parted -s -- /dev/vda mkpart primary ext3 22530 24578
Aug 31 18:53:28 (none) kernel: vda: vda1 vda2 vda3
Aug 31 18:53:28 (none) node-installer: dd if=/dev/zero of=/dev/vda3 bs=1k count=4
Aug 31 18:53:28 (none) node-installer: 4+0 records in
Aug 31 18:53:28 (none) node-installer: 4+0 records out
Aug 31 18:53:28 (none) node-installer: 4096 bytes (4.1 kB) copied, 0.000842261 s, 4.9 MB/s
Aug 31 18:53:28 (none) node-installer: Running partprobe on device: /dev/vda
Aug 31 18:53:28 (none) kernel: vda: vda1 vda2 vda3
Aug 31 18:53:45 (none) node-installer: Creating ext3 filesystem on /dev/vda3
Aug 31 18:53:45 (none) node-installer: mke2fs -j /dev/vda3
Aug 31 18:53:45 (none) node-installer: mke2fs 1.41.12 (17-May-2010)
Aug 31 18:53:45 (none) node-installer: Filesystem label=
Aug 31 18:53:45 (none) node-installer: OS type: Linux
Aug 31 18:53:45 (none) node-installer: Block size=4096 (log=2)
Aug 31 18:53:45 (none) node-installer: Fragment size=4096 (log=2)
Aug 31 18:53:45 (none) node-installer: Stride=0 blocks, Stripe width=0 blocks
Aug 31 18:53:45 (none) node-installer: 125184 inodes, 499968 blocks
Aug 31 18:53:45 (none) node-installer: 24998 blocks (5.00%) reserved for the super user
Aug 31 18:53:45 (none) node-installer: First data block=0
Aug 31 18:53:45 (none) node-installer: Maximum filesystem blocks=515899392
Aug 31 18:53:45 (none) node-installer: 16 block groups
Aug 31 18:53:45 (none) node-installer: 32768 blocks per group, 32768 fragments per group
Aug 31 18:53:45 (none) node-installer: 7824 inodes per group
Aug 31 18:53:45 (none) node-installer: Superblock backups stored on blocks:
Aug 31 18:53:45 (none) node-installer: #01132768, 98304, 163840, 229376, 294912
Aug 31 18:53:45 (none) node-installer:
Aug 31 18:53:45 (none) node-installer: Writing inode tables: 0/16#010#010#010#010#010 1/16#010#010#010#010#010 2/16#010#010#010#010#010 3/16#010#010#010#010#010 4/16#010#010#010#010#010 5/16#010#010#010#010#010 6/16#010#010#010#010#010 7/16#010#010#010#010#010 8/16#010#010#010#010#010 9/16#010#010#010#010#01010/16#010#010#010#010#01011/16#010#010#010#010#01012/16#010#010#010#010#01013/16#010#010#010#010#01014/16#010#010#010#010#01015/16#010#010#010#010#010done
Aug 31 18:53:45 (none) node-installer: Creating journal (8192 blocks): done
Aug 31 18:53:45 (none) node-installer: Writing superblocks and filesystem accounting information: done
Aug 31 18:53:45 (none) node-installer:
Aug 31 18:53:45 (none) node-installer: This filesystem will be automatically checked every 24 mounts or
Aug 31 18:53:45 (none) node-installer: 180 days, whichever comes first. Use tune2fs -c or -i to override.
T24gU3VuLCAyMDEyLTA5LTAyIGF0IDEyOjQ2ICswMjAwLCBBbmRyZXcgSG9sd2F5IHdyb3RlOg0K
PiBIaSwNCj4gDQo+IEkgYW0gdHJ5aW5nIHRvIGhvc3QgS1ZNIG1hY2hpbmVzIG9uIGFuIE5GU29S
RE1BIG1vdW50Lg0KPiANCj4gVGhpcyB3b3JrczoNCj4gDQo+IC1kcml2ZSBmaWxlPS9tbnQvdm0w
MDEuaW1nLGlmPW5vbmUsaWQ9ZHJpdmUtdmlydGlvLWRpc2swLGZvcm1hdD1yYXcgLWRldmljZSB2
aXJ0aW8tYmxrLXBjaSxzY3NpPW9mZixidXM9cGNpLjAsYWRkcj0weDUsZHJpdmU9ZHJpdmUtdmly
dGlvLWRpc2swLGlkPXZpcnRpby1kaXNrMA0KPiANCj4gVGhpcyBEb2Vzbid0IQ0KPiANCj4gLWRy
aXZlIGZpbGU9L21udC92bTAwMS5pbWcsaWY9bm9uZSxpZD1kcml2ZS12aXJ0aW8tZGlzazAsZm9y
bWF0PXJhdyxjYWNoZT1ub25lIC1kZXZpY2UgdmlydGlvLWJsay1wY2ksc2NzaT1vZmYsYnVzPXBj
aS4wLGFkZHI9MHg1LGRyaXZlPWRyaXZlLXZpcnRpby1kaXNrMCxpZD12aXJ0aW8tZGlzazAsYm9v
dGluZGV4PTINCj4gDQo+IEFueSBpZGVhcyB3aHkgdGhpcyBjb3VsZCBiZT8gSSBoYXZlIHRyaWVk
IHZhcmlvdXMga2VybmVsIC8gb2ZlZCBjb21iaW5hdGlvbnMuDQoNCkknbSBndWVzc2luZyB0aGF0
IHRoZSBORlNvUkRNQSBjb2RlIGlzIGJyb2tlbiBmb3IgdGhlIGNhc2Ugb2YgT19ESVJFQ1QsDQph
bmQgaGFzIGJlZW4gZm9yIGEgd2hpbGUuIFVuZm9ydHVuYXRlbHkgdGhlIFJETUEgY29kZSBnZXRz
IGxpdHRsZQ0KdGVzdGluZyB0aGVzZSBkYXlzLg0KDQpEbyBhbGwgdGhlIG90aGVyICdjYWNoZT0n
IG9wdGlvbnMgd29yaz8NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQg
bWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29tDQp3d3cubmV0
YXBwLmNvbQ0KDQo=
On Sep 3, 2012, at 1:35 AM, Myklebust, Trond wrote:
> On Sun, 2012-09-02 at 12:46 +0200, Andrew Holway wrote:
>> Hi,
>>
>> I am trying to host KVM machines on an NFSoRDMA mount.
>>
>> This works:
>>
>> -drive file=/mnt/vm001.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0
>>
>> This Doesn't!
>>
>> -drive file=/mnt/vm001.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
>>
>> Any ideas why this could be? I have tried various kernel / ofed combinations.
>
> I'm guessing that the NFSoRDMA code is broken for the case of O_DIRECT,
> and has been for a while. Unfortunately the RDMA code gets little
> testing these days.
>
> Do all the other 'cache=' options work?
Yes, Writeback cache also works.
When I test writethrough and do lots of heavy DIRECT_IO with iozone this seems to work correctly.
ta
Andrew
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
>