From: Jan Kara Subject: Re: jbd2: don't wake kjournald unnecessarily Date: Mon, 21 Jan 2013 15:07:38 +0100 Message-ID: <20130121140738.GI5588@quack.suse.cz> References: <20130121104733.GE5588@quack.suse.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Jan Kara , Eric Sandeen , linux-fsdevel , Ext4 Developers List , Theodore Ts'o , LKML , linux-next , mszeredi@suse.cz To: Sedat Dilek Return-path: Received: from cantor2.suse.de ([195.135.220.15]:57243 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752621Ab3AUOHk (ORCPT ); Mon, 21 Jan 2013 09:07:40 -0500 Content-Disposition: inline In-Reply-To: Sender: linux-ext4-owner@vger.kernel.org List-ID: On Mon 21-01-13 13:30:26, Sedat Dilek wrote: > On Mon, Jan 21, 2013 at 12:40 PM, Sedat Dilek wrote: > > On Mon, Jan 21, 2013 at 11:47 AM, Jan Kara wrote: > >> The traces don't suggest an ext4/jbd2 problem. What is happening is that > >> jbd2 is waiting for IO to finish and that never happens. Seeing that you > >> loop device I'd think it's some interaction of the loop device and > >> freezing. Can you reproduce the issue without the loop device (i.e. with > >> the filesystem directly on e.g. scsi disk)? I suspect the reason is > >> something like that the backing filesystem is already frozen so filesystem > >> on top of it cannot write all the data and hangs waiting for IO -> suspend > >> doesn't happen. Contents of /proc/mounts and losetup -l would help us > >> understand what's going on. > >> > > > > As said I am here in a very uncommon WUBI environment means my > > Ubuntu/precise rootfs-image lays on the Win7-partition (NTFS). > > Your explanation sounds reasonable to me as this line from my attached > > testcase causes the troubles. > > > > echo mem > /sys/power/state && sleep 1 > > > > So, /sys/ is not writable immediately after freezer ends > > > > I checked again and again my logs and have seen "starving" lines > > reported by rtkit-daemon, but did not really get wiser what they want > > to tell me. Stopping rtkit-daemon or resetting all or all-known > > threads before running my pm_test/freezer did not help, too. > > > > /usr/sbin/rtkitctl --help > > rtkitctl [options] > > > > -h, --help Show this help > > --version Show version > > > > --reset-known Reset real-time status of known threads > > --reset-all Reset real-time status of all threads > > --start Start RealtimeKit if it is not running already > > -k, --exit Terminate running RealtimeKit daemon > > > > Here are the outputs you wanted with some more (fstab, grub-config) etc. > > I have here no -l option for losetup command. > > > > > > - Sedat - > > > > P.S.: Outputs for Honza... > > > > $ sudo cat /proc/mounts > > rootfs / rootfs rw 0 0 > > sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 > > proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 > > udev /dev devtmpfs rw,relatime,size=1966948k,nr_inodes=491737,mode=755 0 0 > > devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 > > tmpfs /run tmpfs rw,nosuid,relatime,size=788076k,mode=755 0 0 > > /dev/sda2 /host fuseblk > > rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,blksize=4096 > > 0 0 > > /dev/loop0 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 > > none /sys/fs/fuse/connections fusectl rw,relatime 0 0 > > none /sys/kernel/debug debugfs rw,relatime 0 0 > > none /sys/kernel/security securityfs rw,relatime 0 0 > > none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 > > none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 > > gvfs-fuse-daemon /home/wearefam/.gvfs fuse.gvfs-fuse-daemon > > rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0 > > > > $ sudo losetup --all --verbose > > /dev/loop0: [0802]:17982 (/host/ubuntu/disks/root.disk) > > > > $ sudo losetup --find --verbose > > Loop device is /dev/loop1 > > /dev/loop1 > > > > [ /etc/fstab ] > > # /etc/fstab: static file system information. > > > > # Use 'blkid' to print the universally unique identifier for a > > # device; this may be used with UUID= as a more robust way to name devices > > # that works even if disks are added and removed. See fstab(5). > > > > # > > > > proc /proc proc > > nodev,noexec,nosuid 0 0 > > /host/ubuntu/disks/root.disk / ext4 > > loop,errors=remount-ro 0 1 > > /host/ubuntu/disks/swap.disk none swap loop,sw > > 0 0 > > - EOF - > > > > [ /boot/grub/grub.cfg ] > > ... > > menuentry 'Ubuntu, mit Linux 3.8.0-rc4-next20130121-1-iniza-generic' > > --class ubuntu --class gnu-linux --class gnu --class os { > > set gfxpayload=$linux_gfx_mode > > insmod part_msdos > > insmod ntfs > > set root='(hd0,msdos2)' > > search --no-floppy --fs-uuid --set=root 001AADA61AAD9964 > > loopback loop0 /ubuntu/disks/root.disk > > set root=(loop0) > > linux /boot/vmlinuz-3.8.0-rc4-next20130121-1-iniza-generic > > root=UUID=001AADA61AAD9964 loop=/ubuntu/disks/root.disk ro > > initrd /boot/initrd.img-3.8.0-rc4-next20130121-1-iniza-generic > > } > > ... > > > > Here some more useful outputs: > > $ LC_ALL=C df -h -T > Filesystem Type Size Used Avail Use% Mounted on > rootfs rootfs 17G 15G 1.5G 92% / > udev devtmpfs 1.9G 12K 1.9G 1% /dev > tmpfs tmpfs 770M 892K 769M 1% /run > /dev/sda2 fuseblk 444G 81G 364G 19% /host > /dev/loop0 ext4 17G 15G 1.5G 92% / > none tmpfs 5.0M 0 5.0M 0% /run/lock > none tmpfs 1.9G 260K 1.9G 1% /run/shm > > $ sudo LC_ALL=C fdisk -l /dev/sda > > Disk /dev/sda: 500.1 GB, 500107862016 bytes > 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 4096 bytes > I/O size (minimum/optimal): 4096 bytes / 4096 bytes > Disk identifier: 0xcb9885ab > > Device Boot Start End Blocks Id System > /dev/sda1 * 2048 206847 102400 7 HPFS/NTFS/exFAT > /dev/sda2 206848 931299327 465546240 7 HPFS/NTFS/exFAT > /dev/sda3 931299328 976773119 22736896 27 Hidden NTFS WinRE > > I am still reflecting on any shitty userspace app is causing all the > trouble, but I have zero clue how to dig that... Well, I think the outputs make it pretty clear. /dev/loop0 is a mounted image from fuse filesystem. Fuse daemon making filesystem accessible gets frozen before /dev/loop0 gets fully written out and so jbd2 journal thread hangs. Maybe Miklos (added to CC) could fill in some details / ideas but I think the setup like you have never really worked... Honza -- Jan Kara SUSE Labs, CR