It seems one of these commits in
8357422d4bf33bc2c35884d4016c3fc9efbbc1d2 is causing LVM2
(LVM2.2.02.72) to fail.
Previously good logical volumes cannot be mounted anymore, and new one
cannot be created. All failed with the error ...
"device-mapper: version ioctl failed: Inappropriate ioctl for device"
Is there a new LVM2 for this new dm ioctl or the dm is not right?
Thanks,
Jeff.
On Sat, Aug 14, 2010 at 02:01:47AM +0800, Jeff Chua wrote:
> It seems one of these commits in
> 8357422d4bf33bc2c35884d4016c3fc9efbbc1d2 is causing LVM2
> (LVM2.2.02.72) to fail.
Did you (partially) bisect that or are you just guessing?
The only one there possibly relevant is 7e507eb6432afdd798d4c6dccf949b8c43ef151c.
> Previously good logical volumes cannot be mounted anymore, and new one
> cannot be created. All failed with the error ...
> "device-mapper: version ioctl failed: Inappropriate ioctl for device"
What distribution? What version of udev package?
'lvm version' output?
Provide long listing of /dev/mapper and any dm/lvm devices in /dev.
Add -vvvv to your lvm command for better clues.
> Is there a new LVM2 for this new dm ioctl or the dm is not right?
There's no new dm ioctl.
Alasdair
On Sat, Aug 14, 2010 at 04:54:01PM +0100, Alasdair G Kergon wrote:
> The only one there possibly relevant is 7e507eb6432afdd798d4c6dccf949b8c43ef151c.
I'm guessing you could have a configuration not covered by the latest userspace
libdevmapper - not handling some combination of kernel+userspace+initrd packages
correctly. We already dealt with one udev+initrd combination that failed last
week.
Alasdair
On Sat, Aug 14, 2010 at 11:54 PM, Alasdair G Kergon <[email protected]> wrote:
> On Sat, Aug 14, 2010 at 02:01:47AM +0800, Jeff Chua wrote:
>> It seems one of these commits in
>> 8357422d4bf33bc2c35884d4016c3fc9efbbc1d2 is causing LVM2
>> (LVM2.2.02.72) to fail.
>
> Did you (partially) bisect that or are you just guessing?
> The only one there possibly relevant is 7e507eb6432afdd798d4c6dccf949b8c43ef151c.
Guessing only. Not bisecting yet. Prior to the git pull, everything
was working fine. And double confirmed by going to prior to the pull,
everything still worked.
>> Previously good logical volumes cannot be mounted anymore, and new one
>> cannot be created. All failed with the error ...
>> "device-mapper: version ioctl failed: Inappropriate ioctl for device"
>
> What distribution? ?What version of udev package?
Just a vanilla linux. No udev.
> 'lvm version' output?
LVM version: 2.02.72(2) (2010-07-28)
Library version: 1.02.53 (2010-07-28)
Driver version: 4.17.0
> Provide long listing of /dev/mapper and any dm/lvm devices in /dev.
> Add -vvvv to your lvm command for better clues.
Attached. good.vg01 is from prior to the patch. bad.vg01 is the bad one.
>> Is there a new LVM2 for this new dm ioctl or the dm is not right?
> There's no new dm ioctl.
> I'm guessing you could have a configuration not covered by the latest userspace
> libdevmapper - not handling some combination of kernel+userspace+initrd
> packages correctly. We already dealt with one udev+initrd combination that
> failed last week.
Could be. Please help.
Thanks,
Jeff
On Sun, Aug 15, 2010 at 9:03 PM, Alasdair G Kergon <[email protected]> wrote:
>> Just a vanilla linux. No udev.
>
> So likely a userspace problem where udev is now assumed to be present.
>
>> > Provide long listing of /dev/mapper and any dm/lvm devices in /dev.
>
> You can do that by running:
> ?ls -l /dev/mapper /dev/vg01
> Also:
> ?cat /proc/devices
253 device-mapper
> ?cat /proc/misc
236 device-mapper <== was at 60 before the new kernel
Got it! Sorry, it's me being _dumb_!!!
With 2.6.35++, /dev/mapper/control has moved from 60 to 236!
So I fix it by recreating the control file with "mknod
/var/mapper/control c 10 253".
I got this "internal error" when I activate/deactivate the volume
group, but it seems ok. I could mount the volumes and so far it's all
good.
# vgchange -a n vg01
Internal error: Maps lock 14217216 < unlock 14221312
Internal error: Maps lock 14221312 < unlock 14225408
Internal error: Maps lock 14225408 < unlock 14229504
Internal error: Maps lock 14229504 < unlock 14233600
Internal error: Maps lock 14233600 < unlock 14237696
Internal error: Maps lock 14237696 < unlock 14241792
0 logical volume(s) in volume group "vg01" now active
# vgchange -a y vg01
Internal error: Maps lock 14192640 < unlock 14196736
Internal error: Maps lock 14196736 < unlock 14200832
Internal error: Maps lock 14200832 < unlock 14204928
Internal error: Maps lock 14204928 < unlock 14209024
Internal error: Maps lock 14209024 < unlock 14213120
Internal error: Maps lock 14213120 < unlock 14217216
6 logical volume(s) in volume group "vg01" now active
Hopefully those errors above means no harm.
Thanks,
Jeff
> Just a vanilla linux. No udev.
So likely a userspace problem where udev is now assumed to be present.
> > Provide long listing of /dev/mapper and any dm/lvm devices in /dev.
You can do that by running:
ls -l /dev/mapper /dev/vg01
Also:
cat /proc/devices
cat /proc/misc
Alasdair
Dne 15.8.2010 16:13, Jeff Chua napsal(a):
> On Sun, Aug 15, 2010 at 9:03 PM, Alasdair G Kergon <[email protected]> wrote:
>>> Just a vanilla linux. No udev.
>>
>> So likely a userspace problem where udev is now assumed to be present.
>>
>>>> Provide long listing of /dev/mapper and any dm/lvm devices in /dev.
>>
>> You can do that by running:
>> ls -l /dev/mapper /dev/vg01
>> Also:
>> cat /proc/devices
>
> 253 device-mapper
>
>> cat /proc/misc
>
> 236 device-mapper <== was at 60 before the new kernel
>
>
> Got it! Sorry, it's me being _dumb_!!!
>
> With 2.6.35++, /dev/mapper/control has moved from 60 to 236!
>
> So I fix it by recreating the control file with "mknod
> /var/mapper/control c 10 253".
>
> I got this "internal error" when I activate/deactivate the volume
> group, but it seems ok. I could mount the volumes and so far it's all
> good.
>
> # vgchange -a n vg01
> Internal error: Maps lock 14217216 < unlock 14221312
> Internal error: Maps lock 14221312 < unlock 14225408
> Internal error: Maps lock 14225408 < unlock 14229504
> Internal error: Maps lock 14229504 < unlock 14233600
> Internal error: Maps lock 14233600 < unlock 14237696
> Internal error: Maps lock 14237696 < unlock 14241792
> 0 logical volume(s) in volume group "vg01" now active
>
> # vgchange -a y vg01
> Internal error: Maps lock 14192640 < unlock 14196736
> Internal error: Maps lock 14196736 < unlock 14200832
> Internal error: Maps lock 14200832 < unlock 14204928
> Internal error: Maps lock 14204928 < unlock 14209024
> Internal error: Maps lock 14209024 < unlock 14213120
> Internal error: Maps lock 14213120 < unlock 14217216
> 6 logical volume(s) in volume group "vg01" now active
>
> Hopefully those errors above means no harm.
>
Well error is usually error - so some harm under certain circumstances could
happen (low memory situation)...
But in this case I assume it's related to this kernel commit which should fix
this problem ?
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=d7824370e26325c881b665350ce64fb0a4fde24a
Zdenek
On Mon, Aug 16, 2010 at 03:56:48PM +0200, Zdenek Kabelac wrote:
> Dne 15.8.2010 16:13, Jeff Chua napsal(a):
> > With 2.6.35++, /dev/mapper/control has moved from 60 to 236!
The latest LVM package made an assumption that it would be set up
correctly by udev (or manually in your case). We are updating
the userspace libdevmapper code to handle your circumstances
automatically.
> > # vgchange -a n vg01
> > Internal error: Maps lock 14217216 < unlock 14221312
> > Internal error: Maps lock 14221312 < unlock 14225408
> > Internal error: Maps lock 14225408 < unlock 14229504
> > Internal error: Maps lock 14229504 < unlock 14233600
> > Internal error: Maps lock 14233600 < unlock 14237696
> > Internal error: Maps lock 14237696 < unlock 14241792
> > 0 logical volume(s) in volume group "vg01" now active
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=d7824370e26325c881b665350ce64fb0a4fde24a
For those interested, originally we used mlockall() but in non-C locales
on some distributions glibc is configured to map 80MB+ of locale data
into memory without offering any way to unmap it. We don't need
that data locked and it increased the minimum memory required to install
a distribution as well as slowing down the LVM tools! So we had to
write a customised version that tries to skip locking unnecessary pages
like those.
Alasdair
On Mon, Aug 16, 2010 at 10:22 PM, Alasdair G Kergon <[email protected]> wrote:
> On Mon, Aug 16, 2010 at 03:56:48PM +0200, Zdenek Kabelac wrote:
>> Dne 15.8.2010 16:13, Jeff Chua napsal(a):
>> > With 2.6.35++, /dev/mapper/control has moved from 60 to 236!
>
> The latest LVM package made an assumption that it would be set up
> correctly by udev (or manually in your case). ?We are updating
> the userspace libdevmapper code to handle your circumstances
> automatically.
It'll be nice if lvm returns a more meanful message like "ioctl
failed. Check /proc/misc to ensure the device is mapped correctly."
To take care of it automatically with udev, I'm doing this in rc.S
if [ -f /sys/devices/virtual/misc/device-mapper/dev ]
then
NODE=$(</sys/devices/virtual/misc/device-mapper/dev)
rm -f /dev/mapper/control
mknod /dev/mapper/control c ${NODE/:/ }
fi
>> > # vgchange -a n vg01
>> > ? Internal error: Maps lock 14217216 < unlock 14221312
>> > ? Internal error: Maps lock 14221312 < unlock 14225408
>> > ? Internal error: Maps lock 14225408 < unlock 14229504
>> > ? Internal error: Maps lock 14229504 < unlock 14233600
>> > ? Internal error: Maps lock 14233600 < unlock 14237696
>> > ? Internal error: Maps lock 14237696 < unlock 14241792
>> > ? 0 logical volume(s) in volume group "vg01" now active
>
>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=d7824370e26325c881b665350ce64fb0a4fde24a
>
> For those interested, originally we used mlockall() but in non-C locales
> on some distributions glibc is configured to map 80MB+ of locale data
> into memory without offering any way to unmap it. ?We don't need
> that data locked and it increased the minimum memory required to install
> a distribution as well as slowing down the LVM tools! ?So we had to
> write a customised version that tries to skip locking unnecessary pages
> like those.
I see that now it's fixed with that commit.
Thanks,
Jeff
On Mon, Aug 16, 2010 at 11:32:15PM +0800, Jeff Chua wrote:
> It'll be nice if lvm returns a more meanful message like "ioctl
> failed. Check /proc/misc to ensure the device is mapped correctly."
No need - LVM should be quite capable of detecting/dealing with the problem.
> To take care of it automatically with udev, I'm doing this in rc.S
> if [ -f /sys/devices/virtual/misc/device-mapper/dev ]
> then
> NODE=$(</sys/devices/virtual/misc/device-mapper/dev)
> rm -f /dev/mapper/control
> mknod /dev/mapper/control c ${NODE/:/ }
The whole point of this change was to trigger the loading of dm-mod
automatically for people using it as a module, based on using a known
major/minor number instead of a dynamic one only known after the module
has been loaded. That needs to happen regardless of whether someone is
using udev or not.
This can all be fixed easily enough in userspace.
Alasdair
On Mon, Aug 16, 2010 at 11:55 PM, Alasdair G Kergon <[email protected]> wrote:
> On Mon, Aug 16, 2010 at 11:32:15PM +0800, Jeff Chua wrote:
>> It'll be nice if lvm returns a more meanful message like "ioctl
>> failed. Check /proc/misc to ensure the device is mapped correctly."
>
> No need - LVM should be quite capable of detecting/dealing with the problem.
Ok, cool.
>> To take care of it automatically with udev, I'm doing this in rc.S
>> if [ -f /sys/devices/virtual/misc/device-mapper/dev ]
>> then
>> ? ? ? ? NODE=$(</sys/devices/virtual/misc/device-mapper/dev)
>> ? ? ? ? rm -f /dev/mapper/control
>> ? ? ? ? mknod /dev/mapper/control c ${NODE/:/ }
>
> The whole point of this change was to trigger the loading of dm-mod
> automatically for people using it as a module, based on using a known
> major/minor number instead of a dynamic one only known after the module
> has been loaded. ?That needs to happen regardless of whether someone is
> using udev or not.
>
> This can all be fixed easily enough in userspace.
I'm not using module. It's all built-in to the kernel. Hopefully this
would be taken care of as well. That's why I had it in rc.S just
before the lvm filesystems are mounted.
Thanks,
Jeff.