Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757124AbYLPVWs (ORCPT ); Tue, 16 Dec 2008 16:22:48 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751527AbYLPVWh (ORCPT ); Tue, 16 Dec 2008 16:22:37 -0500 Received: from 1wt.eu ([62.212.114.60]:1090 "EHLO 1wt.eu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750998AbYLPVWg (ORCPT ); Tue, 16 Dec 2008 16:22:36 -0500 Date: Tue, 16 Dec 2008 22:21:55 +0100 From: Willy Tarreau To: Trond Myklebust Cc: Harry Edmon , Max Kellermann , linux-kernel@vger.kernel.org, linux-nfs@vger.kernel.org, stable@kernel.org Subject: Re: High load in 2.6.27, NFS / rpcauth_lookup_credcache()? Message-ID: <20081216212155.GA581@1wt.eu> References: <20081022091207.GA12996@squirrel.roonstrasse.net> <20081023123628.GA18549@squirrel.roonstrasse.net> <1224773745.7625.4.camel@localhost> <4946EBFA.60700@atmos.washington.edu> <1229432553.7257.4.camel@heimdal.trondhjem.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1229432553.7257.4.camel@heimdal.trondhjem.org> User-Agent: Mutt/1.5.11 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4456 Lines: 105 On Tue, Dec 16, 2008 at 08:02:33AM -0500, Trond Myklebust wrote: > On Mon, 2008-12-15 at 15:44 -0800, Harry Edmon wrote: > > Trond Myklebust wrote: > > > On Thu, 2008-10-23 at 14:36 +0200, Max Kellermann wrote: > > > > > >> On 2008/10/22 11:12, Max Kellermann wrote: > > >> > > >>> after I was able to fix http://lkml.org/lkml/2008/10/17/147, the > > >>> server which was already upgraded to 2.6.27.2 still gets very high > > >>> load. It is a web server with NFS file storage (NetApp), and while > > >>> the others in the cluster (kernel 2.6.25) have a load of 1-3, 2.6.27.2 > > >>> gets 30-50. > > >>> > > >>> I did an oprofile, with the following results (server just started, > > >>> load "only" 5-10): > > >>> > > >>> 87593 56.1116 (no location information) vmlinux > > >>> vmlinux rpcauth_lookup_credcache > > >>> 16037 10.2732 auth_generic.c:0 vmlinux > > >>> vmlinux generic_match > > >>> 6460 4.1382 (no location information) php4 > > >>> php4 (no symbols) > > >>> 2478 1.5874 (no location information) libc-2.7.so > > >>> libc-2.7.so (no symbols) > > >>> [...] > > >>> > > >>> We havn't configured any special authentication method. It is a NFSv3 > > >>> over UDP mount, but the kernel has NFSv4 and therefore KRB5 enabled. > > >>> > > >>> Any ideas why rpcauth_lookup_credcache() goes overboard with CPU > > >>> usage? > > >>> > > >> I have bisected the problem: 98a8e323 is the result ("SUNRPC: Add a > > >> helper rpcauth_lookup_generic_cred()"). 5c691044 is ok. > > >> > > >> See the attached oprofile annotation data for both commits. I guess > > >> that the function rpcauth_lookup_credcache() is waiting for a spinlock > > >> too often and too long. Trond, any idea? > > >> > > > > > > Can you add a '-v' to the rpc.gssd daemon startup line? I'd like to see > > > how often you are creating new gss contexts. > > > > > > > > >> Harry: added you to Cc because your problem sounds similar. > > >> > > > > > > Harry's problem is should be unrelated. afaik, he is seeing a problem > > > with userland RPC code, not kernel rpc code. > > > > > > Trond > > > > > > > > I am finally getting some time to look at my problem that I originally > > reported in October (SUNRPC problem with 2.6.26 and beyond), and I am > > seeing the same behavior as Max Kellermann when my machine slows as I > > described earlier. The system in question is currently running > > 2.6.27.7. Here is what I see when it is misbehaving: > > > > samples % image name app name > > symbol name > > 11380517 57.4191 sunrpc.ko sunrpc > > rpcauth_lookup_credcache > > 3263657 16.4664 sunrpc.ko sunrpc > > generic_match > > 1081287 5.4555 vmlinux vmlinux > > copy_user_generic_string > > 499407 2.5197 vmlinux vmlinux > > __posix_lock_file > > [...] > > > > And here is what I see when I stop the programs that are chewing up all > > the system time, and then starting them up again: > > > > samples % image name app name > > symbol name > > 6372650 21.7978 vmlinux vmlinux > > copy_user_generic_string > > 5401386 18.4755 sunrpc.ko sunrpc > > rpcauth_lookup_credcache > > 3018753 10.3257 vmlinux vmlinux > > __posix_lock_file > > 1050095 3.5919 sunrpc.ko sunrpc > > generic_match > > > > > > and I am not using Kerberos with NFSv4 (i.e. no rpc.gssd). Did you ever > > find a solution for this problem with rpcauth_lookup_credcache? > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=23918b03060f6e572168fdde1798a905679d2e06 Trond, this should be included into next stable, right ? It's fortunate because I know someone else who recently described me the same problem under the same circumstances when migrating from 2.6.22 to 2.6.27. Regards, Willy -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/