Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750944AbWCBNGU (ORCPT ); Thu, 2 Mar 2006 08:06:20 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751146AbWCBNGU (ORCPT ); Thu, 2 Mar 2006 08:06:20 -0500 Received: from duempel.org ([81.209.165.42]:30152 "HELO duempel.org") by vger.kernel.org with SMTP id S1750944AbWCBNGU (ORCPT ); Thu, 2 Mar 2006 08:06:20 -0500 Date: Thu, 2 Mar 2006 14:05:12 +0100 From: Max Kellermann To: linux-kernel@vger.kernel.org Cc: trond.myklebust@fys.uio.no Subject: 2.6.15.4: NFS-related BUG in radix_tree_tag_set() Message-ID: <20060302130512.GA23671@roonstrasse.net> Mail-Followup-To: linux-kernel@vger.kernel.org, trond.myklebust@fys.uio.no Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="r5Pyd7+fXNt84Ff3" Content-Disposition: inline User-Agent: Mutt/1.5.9i Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 11329 Lines: 183 --r5Pyd7+fXNt84Ff3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, after a kernel upgrade from 2.6.11 to 2.6.15.4, we were experiencing crashes on all four web servers. These web servers obtain their data from NFSv3 from a NetApp server. The servers were under heavy load - mostly reading, but also a lot of writing to NFS. Hardware: Compaq ProLiant with two (physical) Xeon 2.4 CPUs, 4 GB memory, Broadcom Tigon3 network interfaces. Kernel config is appended to this mail. After one of the crashes, an administrator made a screenshot (http://www.duempel.org/~max/linux/nfs_radix_tree_crash.png) and rebooted. Unfortunately, part of the stack trace is missing (25 lines console only), and I had no access to the KDB console. I am currently waiting for the next crash to happen so I can provide more information. The BUG_ON() failed in lib/radix-tree.c:372 : slot = slot->slots[offset]; BUG_ON(slot == NULL); I believe the missing stack trace calls are nfs_mark_request_dirty(), nfs_flush_one(), nfs_flush_list(), nfs_flush_inode(). That would mean that req->wb_index was somehow removed from nfsi->nfs_page_tree, maybe in another thread on another CPU? I see the spinlock nfsi->req_lock is only held for very short timespans - is it possible that another CPU tries to flush the same NFS write request which is currently in the middle of being handled by the first CPU? Any other explanation? Max --r5Pyd7+fXNt84Ff3 Content-Type: application/octet-stream Content-Disposition: attachment; filename=".config.gz" Content-Transfer-Encoding: base64 H4sICJvtBkQAAy5jb25maWcAlDzbcts4su/7FayZh02qJoklO448dXyqIBCUMCJIBAB1mReW YjOJTmTJq8tM/PfbIMULSICe82Bb6m4ADaDvAPzrv3710Pm0f1qfNg/r7fbF+5btssP6lD16 T+sfmfew333dfPvde9zv/n3yssfNCVqEm935p/cjO+yyrfdXdjhu9rvfveH72/eDj+9vgIDt d16QffGGV97g+vfh6PfhLXy+uv3Xr//CcRTQSboc3abXw/uX8rskDPFpLEgqQ0I4EbLGAW39 hbGk/pJQf9DATUhEBMUplSj1GbIgYhilBiOBpylDq3SK5iTlOA18DFjg8lcP7x8zWILT+bA5 vXjb7C+Y6v75BDM91rMgS+CUMhIpFNbd4pCgKMUx4zQkNTiM8SydERGRBi2NqEpJNAdegIIy qu6vhwUHk3wntt4xO52f6zGhGxTOYYFoHN3/8osNnKJExY21XTQnLVdyTjmuATyWdJmyzwlJ NLcw9QI+ln7KRYyJlCnCWHmbo7fbnzQ/jb6wCpuNUOJTG+U0VjxMJvWos3j8B8EqTcgclq/Z BZ0VH5q9VEjME0mUtIygV1IgFshUxonARK9N3QynMVewvH+SNIhFKuGDtX/CxsT3iW8ZYIbC UK5YQzBLSAp/mzOo4GQJLKUcSRvDXNBIzerexs3lGSMJnCZhQ1SCRJFl/ZXwuImVU0ZYQwYx MEUnEbSKsAKZkPdXHVyIxiS0IuKY2+B/JCyHVzNVNFoVQ1smmM9BMr04V4VIh/v14/rLFhRr /3iGP8fz8/P+cKqFm8V+EpLGEheANInCGPnNNb4gYDNxibawEI9lHBJFNDlHgrV6uOiLtIrC ZQQpcKVWYWiTCyBs7IOKOdgUPKURKW3JeLt/+OFt1y/ZoaHHY7/E09iTD98zvSSHhnWhscRT 4qcR7EbDYlygSHZhPkF+WAzcwuDgcw30SYCSUBVdVPMtoWUn1kUpiaC/Xrzm2bJWJfrC1v0v 6x04ns3z+rQ/vPxSrAY/7B+y43F/8E4vz5m33j16XzNtiLOj6T9yI1aNrCEkRJGVLY2cxys0 IcKJjxKG7JPSWJkwZlo2Az2mE8m4e2wqF3Yp09iLe9LuyElD5Kerqyu7oF6Pbu2IGxfiYw9C SezEMba0425dHXKw7jRhlL6C7sfbzEuJuzGUeubgY/bJAR/Z4VgkMrbrACNBQDGJ7aLGFjQC 7efYwcgFPezFXvt29ITEPpksBw6eV4IunUs5pwhfp/ZxG1JoWWmNxYwv8bThoTRwiXzfhISD FIPxA7M5pYG6/1TixAIivFT3AE3AkE5iQdW04bDKEA2iGDoWCEy2D9q8Mntf8HQRi5lM45mJ oNE85C3mxmbck1uMmCO/0/gytdsbEzyJY+CUU9weSpEwhShE4Ji3+ANoyiFmSmEF8AxMhokG zaoBU04U+E9GRAtGWBLq+QvVoAbjUn+JRB4H3Y8qbG7jJVONqE4Qwrg2xJER1ZXweRwmELaK lVUeLlR9uHQ8s3nDXAh0LGpbutgCZJh0AJrrABlBdYnhN2pKBENGwKViELcxsrJLRzOnyAsy jmMV0GXCHTEAxRABg845u2DS7VFgk6ihyLl/CzaHp7/Xh8zzD5u/Codfh5++XfFBF8JUjBOb S8U+BFr3TzVxFE/ppB2TVdgL7mZi6euCu71pqNIF1lIk1Ng0SUIdyQNRLFY6bGomOAHIMmAg JIoSc9N8KuGTopMabWVYQuAGxsNKZA5ijgrr4JO0aMebA9cdSoWU1eJJHkJexlWes+W6dtNc D64MnWJITS9qS02vUBIoIZoNSEAtVFR8HiOIYJoKIcikHbdKgnVuad/cP9OBGSXUiOHHq2Y3 Bamd9h5oK0MyXUmq9RkWS6j7q5+61VUjB5iRJcEdKef7v7MDJNC79bfsKdudyuTZe4Mwp795 iLO3dTjHG3vHWRqSCcIrw2wx0AHIkDrj6N6gz8e/1ruH7BHSdl2vOB/WerA8diwYobtTdvi6 fsjeerKddOguGkUB+Aa/jKRWg8ZIKWKaSwOdKAUJudnNnPokbsEgE56RVQsYoHbTS+Ydixbc Yv0KjmViN2E5lo7txqDosms9jZmFCM9CKlW6Ikg0s78c3dmWJpLg9rTiBWlPSa6k6hgYVvlG czhtMhBkJ6IrCZw1BKHYdlbJ31tvDElHY/Prbjnr9AUq7wWH7D/nbPfw4h0f1tvN7lstMYBO A0EaKVUJSRUah4ZxqDCuhaoItDWyttQIaA7zDm2FhMYIRXY1iecpJwJyY3axJjZualodzIBR xKSv826nVgq9vxLNXfhqKAc+jnwC/ftWnjUBwKCLOVjaObFtm94177lKIR8rH9sQrkKQgLbY QsNn5xOI4kXqSCRMGntSYdLYE4w8wlxCWik5eEx32skJ8UE9eIoh2xE0iv8BKe3JI2sqyWxO KOf9JsXa8xWFjXZIk29FlBde3MlEGEcTkbiTcY2fggB2tvD4HUKjx0ap1WjW3LncROpwPHBM o54qpBRVLeZ8rF0Rx+CJOGaYot88QiX8Zhh+waemc8LUcEWYgiTm1sQaHudoxoqvtrpfTuBT AUFTt18U2VyMxllbEB4LNU6s9UVow2SH9cK35ovn5N9lqi6FdR3FGxVbaQ+9YS3tER3+OXTU Mi51Yr3H3ZACYyR8vWt6wz7g9eERdvNto4rYmERO2u2BetP96Xl7/mbzBOXgmqzdlPzMHs6n vHj5daN/7Q9P61PDsIxpFDDI3sLAKKIXUBQntqr4BcuorM4douz09/7ww3A3EVFddPc0Aszq jDRTwPx7yowzjySijSLyMhDM/JZb6EaqCR0Y4QqNmkNQnkLYCzE0kiYU+XPtJfxUwMwNjy9n Gh/QcTpFcmqIUQEGK2dzRGaj1vg8JIXnlQYuHzwNFgyJmQVRNIXgvcVFC1tUNGzl5Yp2TsQ4 lqTVD4/sUbpeVMppH3Ii7CmnnnA+pD2cE9zGplzp46h4Ro3l0QOhaQtAJG9BKNfZRguokqh1 jAVAn6JJmw7zElwn6wCDj5NKRiwcVzRjWmkG5b97883hdF5vPZkdwK2b0X5Tk2GZ5nbzTPn8 1jIgMBvQUJkBZwV05B54D1k8aCQYhJOFm04v8Cmk0czQLhOVlqdwToIwnjQX5NXRYSWiQK8l RA9gDwwt0AiVn+11wCV92lEdA5sf3smW2BsUF75dwlywAM5IxeAWlO3MoKAKFO8OQ4XdhxVY GFp76dRxvlMQqUDZ9bBAIx2MOgpLOQHnve3z/XROinKL0QI4QwpPL8fBVhTlAkUTYkcyhO0I PlNqxZ2tuht9weSGEOIMOxrSRzsCwpXiaNeCIziyI3yJuR2Dpi3r1FwqEk3U1MGfCh0IzJl0 8D4lIW+6rSZOp2OORXTqWYGOF5GrU+T7wr05gqCQdeX/wg/uFcHLumLsVq6SecacG6Zn5hYR jR2vlEuStdMupdk+BZ3aWCPOQvlL02yqLhIT8DGC/EG6FuyCLMylDZO4UXbBuCCjoGWjL8YG dViIdA0SUhDiO7piSEJfAvmd3qqpVUG/lRWijNjOQErE2vHIhScZMa4P562Vz5qssLitzYom oYtZix24YCzKfsHYtL1anK6wXVA4RFLSYOVipKvwZUOHEFf9JpAuCtphSaCFawviUt3bjkNc bPurmqnrXH0O6jIp6NBx/myKTDs4zEOFXPvN2uib5uWlt63gqW0tcnRuZ/4fnTjtElL2auRY UH9iX4t5iKJ0dDUcuK4dYFAGKyoMsb1MQbn9LFufOzmOjIYf7UMgPnbG8j6F5MDOGoG/Dq4X MN0idepsw+e91IXtD/uD93W9OXj/OWfnrMgXjYGlI2mJckmXRfetvNI7ZceTpS+IHCbEXs2Z IgYWjNrrUlQ4gqexQ5QJIXqXB10Jzv7aPDTPzOo7cpuHC9iL2ykxuGoI38K4eSWGizxqBaci 2AIJko4TGjZMdLBI9W0iMxXIE4DUF3ozLXnAbpc9nGBD3nnn3ebrJnv0zkdg83kNLP/Pu/+9 3JUsvm83ux/Ni0DaKIGNb1b688pWFVvnY7DsaX948VT28H233+6/vVxW5Oi9Yco3VA++dysf 68N6u822nq55WCsmYF9iU06LhrpWkp+jbNcv1oZR11IUF54eCwYbJZJwBgs5TwOjrKihmH9O fdsJRInEVEqgMMrCGiSxpGD6uN1qlAP6CN/d2gtOJUniOi8tCXC8yCMk6wFfSRQWV7W6jcWK qzhsXYrqkEVjeyW4xMulvZhc4gWy3sOrsTCDBCK8wW1jGX0RM63g2J/bRweDmMYg9ylR085W A/ID/HD6gQXsgwjD7qU+2oxvSmYK4EXEsvUxgy5BvfcPZ31ak3uXD5vH7P3p50nX2rzv2fb5 w2b3de+B24HGRWXfKBA3uoboRzkMz4Vo6mu6ntUCrE+l6dkLUJEQ5Qfd/e1xR84viFcFFmiC MObcfjujQaXl325JfV0R08W2GKtumV3P/eH75hkA5YZ9+HL+9nXz01xT3Y3lYlBXwJl/e2M7 TG7wWtTR6451pVlOtQWm4rNtpeIgGMetOm6H6J9wp+/+3g7tt6Yq7fjTcRzeFAiG2qcBLWx+ LGItFVaty9vZhlgBKo7ClRavXi4RwbfDpT1wqWhCOvi4vO6nYf6nm9f6UZQu+w1Wvu39vUBm F4SknwavRkN8e9fPMpYfPw77jbgmuf4HJPYwrjINXF2/MilNcms/IKxsNR64DjlKEg7r26/g ajQc9JNEcvTpZtA/Ie7j4RVITRqH/bpUEUZk0aMJcr6YSZsSSEoZckTwNQ3swKB/q2WI767I KwusBBve9S/wnCIQrKVDzrVtQ6LPYxaaadFVOh+7dbyt37XLsaRUYL6LOKnrNwWiPiihar6F 0Q3Mb0WwGFSBYt7lpa/iBvWbx83xx2/eaf2c/eZh/x34+7fdoEyaZ+5TUUDtcXqJjqW0nmuV fQqrlIgUQls/FpaW1bgTKze4G4DI/VPWXEcIiLP3397DRL3/O//Ivux/VieE3tN5e9o8bzMv TKJmlqCXsXDqgGgtL3zWWYRZ5c4xYTyZtKpW9Q6ow3p3zAdFp9Nh8+V8ytojSn1tp7u9aYCt YJr/LjEmKxLJAmNnZrv/+13xhOmxe+ew6Ffhfkt/vUhBj5a5UDoOdaEXoLpzqVvBfvtctoVG uH8ARPGn/gEKAqepq4ju+nrxuUrp0J7TMjJBuWqCkXQlxRVNcdreTwOb5+bEGcjm2HEiQQod 0U8xE7a8HtwNehaD9I4QJCqBCM2PGaL2ueZkE1/Z75wUouu4XlsgI6oc5YMSjwYOT1os0Yp9 vMYj2HR7oefCgv2ebo78nC8iKN6rJIPhyBYjXkgQ+M5lRzk1fGBKWxsN7e6fOtDrq6vmvd4a PuyZKBDcXg/6CYau0K+wa7xvIXx8fffxZz/+yu41cnwk+XUPd/aTVqYN9zvTUXpvclOhaxPh vOnRmN9NNwH2VONT/b4JCQOkO7vqQAZdSJfoYwdya0DqewaN3QR4fgPPdt2nqluwuh+fFVUE AyIjxOU0NoGMChELA/QnEXFzcE1VMtVZ7OCsH/R6+v5/JzSpeggSSR0vYAqUdm99aIeMlY1R 15Xp4qA3uL678d4Em0O2gB/r5R9Nl5N1OgCL7p5Ry94bZVFXiBYRBftEcfPgy08YM+4uj+PI b8UJdQn4cwLp2p/EFgupPBYpqiwC7yBTrytqdXFGtCvfRU1lunIyDbjGhTj9NKtB3bkwyZCY 69cHQGMURBFMWxeOpnbv9lkSRtvYCy5cqnZ3mGLkozZ9cfPq9F1XT0HfB1fe/uCBL2BfNqe3 xi7o2pR+Wt04KoOJNUeYIs5XjLgeGSTRxFEAxPqwKbLLsh65iGXTa1iNDu/qvN08e1/XT5vt i7dzCZPRnUpCx82gKbff7c8L9O07igB0mFlI/UeDwaBdzavxPuKK4Py+f0Ad95DGNzd29c3r H66u/YmwBwKEQBZjnx0ZmF4wgC2M7N4rQkpLnWOnhrP2xb4KOQJz4Yh/NUrFjvCEyjtHYEI4 xc6gJYl8ffHJilSuJ6yQy6Zi2nqX21EQGLJUjnpDMYkcIaIfDu31JtIuhtULKUfXI0cdZory l89W3AqMSLwIHEGkGA1u71yLPHCk+XLmqDXI2d0odIyk1xEMWoyp1fUqOomj66bAjaPl8JU1 tyw6npIQPFmq7FVHupzYjwzlkHbdkNr/yHae0Lc+LS5AdU+ktG/cZsejp6XpzW6/e/d9/XRY P272b9v2pnN4V3Sw3nmb8s2MMdrCIZ+B79s3fkq5IxLg3K5z0mUBNb+u2BTsu+NxN7SCT/of AbjQ+l8pOAfUyPzZiYAPltM/Kv0IjPmX48vxlD2ZmbUfdfcSNub5+373YruBzKexRcHp7vl8 crpzGvGkuh+cHLPDVofExuY1KVMWJxJM9NwoaxmYlEuU2PKVFpnEgpAoXd4ProY3/TSr+0+3 o/Z4f8QrILHrR06gZD+ezF/D25KJYjk7B8pGyxlZ5UcPdThRQlKkZmOjVlZhwLDPHCd4FU04 e5VkqV4lichCOaLvxvr34GHppaLYcS+zWPw4wdNi+/oGorL7+G+6PjzmT1zphzg/cW6Wvoig zX9Oo7+mdHR1MzQS5xwMv9uH0i0KrEZD/Gng8A85CTh9LocWaS7QEAoDuju2QAtrpxPEiPWk HH9fH9YP+ipu5+B73ghL5/k1JG2Nath00YAZfKBQv4UurjFYHjTI7LBZb7tB+6XpqHjz2QX2 DJej81eV9hUrSSKRJkgo/TLW2gVZKgiMSZfnCLyRpgBIznzrRoLZFY4F6cxAA3tm8Ie01fT1 w4e7UcrVyiifls9hANw1FJxR804TpDRT2AqbG1isTw/fH/ffPP0upeUxFZ76sfXB9QLkDNIH 425nNLef4xf/EKCO3ZTjmpK4vru1R+eQAoW0lafUqxdHK959+BIU5wYQ73hft/vn55f8IMFM F41oc2L33b6wD8sWaG63VKCBlhs3VTJEjNuk+b9OgBXSZ8kO8vIlTCNcjSb5f2oo3pp2XTVn 1ngLww+3z0aRMH86Z6k9YHvRoWs9/fV2uz7+++gN3v0NcZz35WyKU/dyVGVv2X63Oe0POlDs aNR0wfIHyvX0NUC/UXYYT9AvIdMppIP2pK1o/9/GrmS5cRyG/koql77MVMZbYk9VDlottbVF i9vJReW2PWlV99gu2znk75sgRYkL6PYhiwCIm0gIJIAnFhkFoCX6qV1z3mADGNpxbRX6bpme 8v2/2zZrsoiO6+/Nr+bS7M53GWiIrRz5JcgiNdDE61oxDtjgNu/NhWidZbPdHe7s02G93axp zBsPqBLLcWU/H4s5O62PP5rNGT0Vw4161hyGk6C/Nw778+EXWUrN+QgxVqyj+vNbzi39zUEP mwRy6xL72G/FZhUp2XJqFVeFrddCiAwXL7DIs3cEA0jipIF84ABlaYo+Lb1/oRYAxcrBL7zb QwLdmTql/qInt1+Y5645/+RN+cKfMBQ5X2/fySYLsZSh2LnlzpGzrzh2HgqX6mHsGRE2YhL+ 1+wbG1qELVLyOwltCxnDIhGmDdOXkB3I9gISUmA5ZE7afovNSPUKHHf4JpxLhFjoPuGOWJEy gRWokxnOn+VESiMos/CcKsd3xF9la5dcGvNDSUGxTbPl+upzLyTvZcKRe9+RqQLG9T8XaRN9 /RSpUihe7bjIQjsvCmAD0PeZyuAdFktuqSveW+HaW7UZNF2hQH2pUoPzbXX9oaxMfQKGwWKO ySReKf3oxyKNTZ0MyUJm0fb96a7abq6OdFGIhtBKVrhjE9sHRCVft8xYFMMDRCzCetOWW1ik s8fHf6Sn8DWNQtlmeCNihoor18fqddPiwbfKh6TE6/Uhb0Ka53FB7sHHddlJC3d3AHup62VE Zz6PR08YP0ydAGBsyuf75nyYTiezvwcC4GVSamPO/ALn3cf2QFOYtcb3gSQiYaGY4K+FKFLG mdzfoCIqObINw9py60yJEuosvlgsPAcEML6WOotiR6yj/e7wcVZ6IRxgX5lvvpkXXGVlUWVk 2575VtvMunKXQ4cCNyZWV9qZmXkvyWps5gIsrIlX4XOJO8foG69QZ1OiKEG4Xo7Ua1lxUtpY vmYZUOKRAaG68lW9HGoFuWptLladq9bnQiRMTwFb3lUupVuYXlc7SqqpiyqTkyaBwSLIheVU JbkIxsuu67mYfkYI5E0AtHqR2xOUUWSLeCTthWPbOLdCAyNxMlxTEWVkKXqtJUGcDcBBGBab cbrN1Jp4lsKloYlF5edRtsQzKy8hTiS5ljbOlG0n2iW3rC/EpL6L1vv3j/X7TkeASiLx+UUF 17LP95sj0b/3Iosr5powpF27yHsa4cgwstDTBOmDJDKVMbsUHu7fU4Tw0FBF6IbWTg0JFIoQ 7u5QhG5p+CMeIqoI4eccitAtQ2AINlWEcBeVJDQb3VDSbHLDYM4MDlxZaHxDm6ZP5nEiNhBY DzWeWSIVMxje0mwiNTDMa6twwlBeabz6gTrPOcM8BlzCPFG4xJ97b54iXML8VLmEeRFxCfOj 6obhz50ZjA2j2wlM1LFcpOG0NjjHOBvDlgRmVfpTrkjD/fly6hN0UA9WnvphZAp3WRC27emK f8G+JfBjvfmpZB8ylHQtSZbvN+jhkwqqH1sQqkcsmvxFfHUDxLrfAtMOJn0RJaBXkp0ARXus ZKhIyKGEfH/8LcaqLyILP/Zp2VmYqEeCBhH28YMrgi1q/hUJBPpT4pOfHqxG4sAGW095NLy/ GXOJ1eMT09Ujm1MAypXiv3OnqktiEUE8aemhgeOelUevPTK+Okyl5SwgE82PUtw5I8jVVWFK RWBSoKysKEqxLPjxgpYhf4Si9sMEUHYpNA2hycw4o7CigsPQtSUvoWtzeHt8aRB+6mNYZsAh /5ZhUnm1Y5VWUeZpFoQOh9Uvdhv2mQoENW3hvRoMfuRwgd14+jxeDu/slFNHmWKpjH032XUd SKgjLTGRPmDQEmN3jNAmGq0IrAFGHE4eMfJkMNTI3zJGFbZVlF7O88EMcwm2fFeM52tpNo1e EcGneGHfUpQOzksJTqOlW15RT6Z6HwBDSx8FoOqypWfp5eaOPrCLwHqzXF02qexQ7yNkHUrY JvzxhE5geRH81RuYO6Ohgw2yssRlT8WGzjLs7L5ry5IsURd1pkTN99P69Hl3Onxcmr16u1M7 TlgaEldyZ4DhP9FeCC+L0O761ZLeCA10pDxAlKoNGwczD6ycbF9fCp1DqDV9F+osMmkgXLRN ihQ0jIQBDoSglK8p2hHDtJYZ8GGULG2/DPEbfNCTT/RnAAA= --r5Pyd7+fXNt84Ff3-- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/