Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752124AbZGYQjH (ORCPT ); Sat, 25 Jul 2009 12:39:07 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751803AbZGYQjG (ORCPT ); Sat, 25 Jul 2009 12:39:06 -0400 Received: from atreides.gradator.net ([212.85.155.42]:38389 "EHLO atreides.gradator.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751236AbZGYQjF (ORCPT ); Sat, 25 Jul 2009 12:39:05 -0400 X-Greylist: delayed 4867 seconds by postgrey-1.27 at vger.kernel.org; Sat, 25 Jul 2009 12:39:05 EDT Date: Sat, 25 Jul 2009 17:17:52 +0200 From: Sylvain Rochet To: Jan Kara Cc: linux-kernel@vger.kernel.org Message-ID: <20090725151751.GA6419@gradator.net> References: <20090420162017.GA28079@gradator.net> <20090716172749.GC3740@atrey.karlin.mff.cuni.cz> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MW5yreqqjyrRcusr" Content-Disposition: inline In-Reply-To: <20090716172749.GC3740@atrey.karlin.mff.cuni.cz> User-Agent: Mutt/1.5.13 (2006-08-11) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: gradator@atreides.gradator.net Subject: Re: 2.6.28.9: EXT3/NFS inodes corruption X-SA-Exim-Version: 4.2.1 (built Tue, 09 Jan 2007 17:51:29 +0000) X-SA-Exim-Scanned: Yes (on atreides.gradator.net) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 11188 Lines: 271 --MW5yreqqjyrRcusr Content-Type: multipart/mixed; boundary="3V7upXqbjpZ4EhLz" Content-Disposition: inline --3V7upXqbjpZ4EhLz Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, Sorry for the late answer, waiting for the problem to happen again ;) On Thu, Jul 16, 2009 at 07:27:49PM +0200, Jan Kara wrote: > Hi, >=20 > > We(TuxFamily) are having some inodes corruptions on a NFS server. > >=20 > > So, let's start with the facts. > >=20 > >=20 > > =3D=3D=3D=3D NFS Server > >=20 > > Linux bazooka 2.6.28.9 #1 SMP Mon Mar 30 12:58:22 CEST 2009 x86_64 GNU/= Linux >=20 > Can you still see the corruption with 2.6.30 kernel? Not upgraded yet, we'll give a try. > If you can still see this problem, could you run: debugfs /dev/md10 > and send output of the command: > stat <40420228> > (or whatever the corrupted inode number will be) > and also: > dump <40420228> /tmp/corrupted_dir One inode get corrupted recently, here is the output: root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# ls -l= ai total 64 88539836 drwxr-sr-x 2 18804 23084 4096 2009-07-25 07:53 . 88539821 drwxr-sr-x 20 18804 23084 4096 2008-08-20 10:14 .. 88541578 -rw-rw-rw- 1 18804 23084 471 2009-07-25 04:55 -inc_forum-10-wa.= 3cb1921f 88541465 -rw-rw-rw- 1 18804 23084 6693 2009-07-25 07:53 -inc_rss_item-32-= wa.23d91cc2 88541471 -rw-rw-rw- 1 18804 23084 1625 2009-07-25 07:53 -inc_rubriques-17= -wa.f2f152f0 88541549 -rw-rw-rw- 1 18804 23084 2813 2009-07-25 03:04 INDEX-.edfac52c 88541366 -rw-rw-rw- 1 18804 23084 0 2008-08-17 20:44 .ok ? ?--------- ? ? ? ? ? spip%3Farticle19.= f8740dca 88541671 -rw-rw-rw- 1 18804 23084 5619 2009-07-24 21:07 spip%3Fauteur1.c6= 4f7f7e 88541460 -rw-rw-rw- 1 18804 23084 5636 2009-07-24 19:30 spip%3Fmot5.f3e9a= dda 88540284 -rw-rw-rw- 1 18804 23084 3802 2009-07-25 16:10 spip%3Fpage%3Dfor= um-30.63b2c1b1 88541539 -rw-rw-rw- 1 18804 23084 12972 2009-07-25 11:14 spip%3Fpage%3Djqu= ery.cce608b6.gz root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cat s= pip%3Farticle19.f8740dca cat: spip%3Farticle19.f8740dca: Stale NFS file handle root@bazooka:~# debugfs /dev/md10 debugfs 1.40-WIP (14-Nov-2006) debugfs: stat <88539836> Inode: 88539836 Type: directory Mode: 0755 Flags: 0x0 Generation:= 791796957 User: 18804 Group: 23084 Size: 4096 File ACL: 0 Directory ACL: 0 Links: 2 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4a6a9dd5 -- Sat Jul 25 07:53:25 2009 atime: 0x4a0de585 -- Fri May 15 23:58:29 2009 mtime: 0x4a6a9dd5 -- Sat Jul 25 07:53:25 2009 Size of extra inode fields: 4 BLOCKS: (0):177096928 TOTAL: 1 debugfs: ls <88539836> 88539836 (12) . 88539821 (32) .. 88541366 (12) .ok 88541465 (56) -inc_rss_item-32-wa.23d91cc2 88541539 (40) spip%3Fpage%3Djquery.cce608b6.gz 88540284 (40) spip%3Fpage%3Dforum-30.63b2c1b1 88541460 (28) spip%3Fmot5.f3e9adda 88541471 (160) -inc_rubriques-17-wa.f2f152f0 88541549 (24) INDEX-.edfac52c 88541578 (284) -inc_forum-10-wa.3cb1921f 88541562 (36) spip%3Farticle19.f8740dca 88541671 (3372) spip%3Fauteur1.c64f7f7e debugfs: stat <88541562> Inode: 88541562 Type: regular Mode: 0666 Flags: 0x0 Generation: 8= 60068541 User: 18804 Group: 23084 Size: 0 File ACL: 0 Directory ACL: 0 Links: 0 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4a6a8fac -- Sat Jul 25 06:53:00 2009 atime: 0x4a6a612f -- Sat Jul 25 03:34:39 2009 mtime: 0x4a6a8fac -- Sat Jul 25 06:53:00 2009 dtime: 0x4a6a8fac -- Sat Jul 25 06:53:00 2009 Size of extra inode fields: 4 BLOCKS: debugfs: dump <88539836> /tmp/corrupted_dir (file attached) > You might want to try disabling the DIR_INDEX feature and see whether > the corruption still occurs... We'll try. > > Keeping inodes into servers' cache seems to prevent the problem to happ= en. > > ( yeah, # while true ; do ionice -c3 find /data -size +0 > /dev/null ; = done ) >=20 > I'd guess just because they don't have to be read from disk where they > get corrupted. Exactly. > Interesting, but it may well be just by the way how these files get > created / updated. Yes, this is only because of that. Additional data that may help, we replaced the storage server to=20 something slower (less number of CPU, less number of cores, ...). We are=20 still getting some corruption but with non-common sense with the former=20 server. The data are stored on two storage arrays of disks. The primary one is=20 made of fiber-channel disks used through a simple fiber-channel card,=20 RAID soft with md, raid6. The secondary one is made of SCSI disks used=20 through a RAID-hard card. We got corruption on both, depending on the one currently used into production. Sylvain --3V7upXqbjpZ4EhLz Content-Type: application/octet-stream Content-Disposition: attachment; filename=corrupted_dir Content-Transfer-Encoding: base64 vAJHBQwAAQIuAAAArQJHBSAAAgIuLgAAtghHBRQACwEuLm9rLmNoTU55WQC2CEcFDAADAS5v awAZCUcFOAAcAS1pbmNfcnNzX2l0ZW0tMzItd2EuMjNkOTFjYzIwAA8BSU5ERVgtLmVkZmFj NTJjAGMJRwUoACABc3BpcCUzRnBhZ2UlM0RqcXVlcnkuY2NlNjA4YjYuZ3p8BEcFKAAfAXNw aXAlM0ZwYWdlJTNEZm9ydW0tMzAuNjNiMmMxYjFhFAlHBRwAFAFzcGlwJTNGbW90NS5mM2U5 YWRkYR8JRwWgAB0BLWluY19ydWJyaXF1ZXMtMTctd2EuZjJmMTUyZjA1MmYwNWYzTglHBXQA HwFsb2ctbG9nLWxvZy1lY3ItLXB0LXB0LjkyMTE3ZGY3BVsJRwVMAB8BMjMyNTk5MSUzQi1v cHQtY3BmX2hhdC4zYmI4Y2JhYQFnCUcFJAAZAXNwaXAlM0ZydWJyaXF1ZTQuZjUyOGFlMzBp bmNtCUcFGAAPAUlOREVYLS5lZGZhYzUyYwWKCUcFHAEZAS1pbmNfZm9ydW0tMTAtd2EuM2Ni MTkyMWYJRwXgABUBc3BpcCUzRnNpdGU0LjM2MzU3MjEwLXdhaAlHBSQAGQEtaW5jX2ZvcnVt LTEwLXdhLjNjYjE5MjFmdWU0dwlHBSQAGQFzcGlwJTNGcnVicmlxdWU0LmY1MjhhZTMwLXdh jQlHBXgAIAFzcGlwJTNGcGFnZSUzRGZvcnVtLTgtMy5kYzQ3ZmJiN5oJRwVQACABc3BpcCUz RnBhZ2UlM0Rmb3J1bS04LTIuZWFjMDliMzOdCUcFKAAfAS0lMjMyNTk5MSUzQi1vcHQtcHRf YnIuNDlhMjIzMGIFoQlHBRwAFAFzcGlwJTNGbW90NS5mM2U5YWRkYXoJRwUkABkBc3BpcCUz RmFydGljbGUxOS5mODc0MGRjYQAYAecJRwUsDRcBc3BpcCUzRmF1dGV1cjEuYzY0ZjdmN2UB 6QlHBQwNHwFzcGlwJTNGcGFnZSUzRGZvcnVtLTQxLjEyMGUyYTgwAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA== --3V7upXqbjpZ4EhLz-- --MW5yreqqjyrRcusr Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFKayIfDFub3qtEsS8RAhbqAJ9neupmj7mRsdcX6B1p9eJaoMp8kQCfeD3t EwpJ909UiI/LEZbOGpqPXU0= =6moM -----END PGP SIGNATURE----- --MW5yreqqjyrRcusr-- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/