Hi,
I am unmounting an ext4 filesystem (Ubuntu 16.04), fuser shows that no
user-space processes are holding a handle to it:
> root@kakhan-ubuntu:/# fuser -vm /oldroot
> USER PID ACCESS COMMAND
> /oldroot: root kernel mount /oldroot
lsof gives this:
> root@kakhan-ubuntu:/# lsof | grep sda
> jbd2/sda1 368 root cwd DIR 0,20 340 2 /
> jbd2/sda1 368 root rtd DIR 0,20 340 2 /
> jbd2/sda1 368 root txt unknown /proc/368/exe
If I look into /proc/../mounts I see lots of kernel specific processes
which have /dev/sda1 listed in their mounts:
> root@kakhan-ubuntu:/# find /proc -name mounts | xargs grep /dev/sda1
> /proc/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/1/task/1/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/1/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/2/task/2/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/2/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/3/task/3/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/3/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/5/task/5/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/5/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> /proc/7/task/7/mounts:/dev/sda1 /oldroot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
> # and whole bunch of others ...)
I am unable to unmount the filesystem:
> root@kakhan-ubuntu:/# umount /oldroot
> umount: /oldroot: target is busy
> (In some cases useful info about processes that
> use the device is found by lsof(8) or fuser(1).)
Can anyone please provide some pointers on how should I proceed with
unmounting the filesystem?
(Context: I copied binaries into a tmpfs and did a pivot_root to it,
i.e., the opposite of what happens from initramfs during boot. I
killed all the userspace processes holding a handle to the oldroot.
This issue is different than the one I faced a couple of months ago
during the same exercise on CentOS [1], where systemd-udevd needed a
restart to release the handle to the old filesystem. In this case
there's no userspace procs accessing the mountpoint.)
[1] http://permalink.gmane.org/gmane.comp.file-systems.ext4/53314
--
Kamran.
http://inspirated.com/
On Thu, Jul 07, 2016 at 04:38:39PM -0700, Kamran Khan wrote:
> Hi,
>
> I am unmounting an ext4 filesystem (Ubuntu 16.04), fuser shows that no
> user-space processes are holding a handle to it:
>
> > root@kakhan-ubuntu:/# fuser -vm /oldroot
> > USER PID ACCESS COMMAND
> > /oldroot: root kernel mount /oldroot
>
> lsof gives this:
>
> > root@kakhan-ubuntu:/# lsof | grep sda
> > jbd2/sda1 368 root cwd DIR 0,20 340 2 /
> > jbd2/sda1 368 root rtd DIR 0,20 340 2 /
> > jbd2/sda1 368 root txt unknown /proc/368/exe
This is the jbd2 kernel thread for /dev/sda1; there is one of these
for each mounted file system, and the thread will automatically exit
with the relevant file system is umounted: unmount /dev/sda1.
For example:
# ps augxww | grep jbd2
root 319 0.0 0.0 0 0 ? S Jul02 0:45 [jbd2/sda3-8]
root 1284 0.0 0.0 0 0 ? S Jul02 0:00 [jbd2/dm-9-8]
root 1286 0.0 0.0 0 0 ? S Jul02 0:03 [jbd2/dm-8-8]
root 1303 0.0 0.0 0 0 ? S Jul02 0:01 [jbd2/dm-1-8]
root 1314 0.0 0.0 0 0 ? S Jul02 0:00 [jbd2/dm-10-8]
root 1334 0.0 0.0 0 0 ? S Jul02 0:00 [jbd2/dm-0-8]
root 7338 0.0 0.0 12748 2180 pts/3 S+ 22:32 0:00 grep jbd2
# lsof -p 319
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
jbd2/sda3 319 root cwd DIR 8,3 4096 2 /
jbd2/sda3 319 root rtd DIR 8,3 4096 2 /
jbd2/sda3 319 root txt unknown /proc/319/exe
# lsof -p 1284
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
jbd2/dm-9 1284 root cwd DIR 8,3 4096 2 /
jbd2/dm-9 1284 root rtd DIR 8,3 4096 2 /
jbd2/dm-9 1284 root txt unknown /proc/1284/exe
# ls -l /dev/mapper/lambda-android
0 lrwxrwxrwx. 1 root root 7 Jul 2 23:02 /dev/mapper/lambda-android -> ../dm-9
# umount /dev/mapper/lambda-android
# ps augxww | grep jbd2
root 319 0.0 0.0 0 0 ? S Jul02 0:45 [jbd2/sda3-8]
root 1286 0.0 0.0 0 0 ? S Jul02 0:03 [jbd2/dm-8-8]
root 1303 0.0 0.0 0 0 ? S Jul02 0:01 [jbd2/dm-1-8]
root 1314 0.0 0.0 0 0 ? S Jul02 0:00 [jbd2/dm-10-8]
root 1334 0.0 0.0 0 0 ? S Jul02 0:00 [jbd2/dm-0-8]
root 7469 0.0 0.0 12748 2184 pts/3 S+ 22:35 0:00 grep jbd2
So there's nothing wrong with the fact that you see a jbd2 thread for
/dev/sda1, and that's not the reason why you are having trouble unmounting /oldroot.
> I am unable to unmount the filesystem:
>
> > root@kakhan-ubuntu:/# umount /oldroot
> > umount: /oldroot: target is busy
> > (In some cases useful info about processes that
> > use the device is found by lsof(8) or fuser(1).)
>
> Can anyone please provide some pointers on how should I proceed with
> unmounting the filesystem?
This is more of a VFS question than an ext4 question, and
unfortunately fuser and lsof are not the only reliable ways if a file
system is still busy.
You might want to try searching /proc/.../mounts for any potential
mountpoints on top of /oldroot that might be left over from a process
which created a separate namespace, and hence wasn't affected by the
umount of /oldroot/proc, or /oldroot/sys, etc.
Cheers,
- Ted