Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754088AbYHGQXW (ORCPT ); Thu, 7 Aug 2008 12:23:22 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1752668AbYHGQXI (ORCPT ); Thu, 7 Aug 2008 12:23:08 -0400 Received: from mailin.studentenwerk.mhn.de ([141.84.225.229]:45188 "EHLO email.studentenwerk.mhn.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751653AbYHGQXG convert rfc822-to-8bit (ORCPT ); Thu, 7 Aug 2008 12:23:06 -0400 From: Wolfgang Walter Organization: Studentenwerk =?iso-8859-1?q?M=FCnchen?= To: Suresh Siddha Subject: Re: Kernel oops with 2.6.26, padlock and ipsec: probably problem with fpu state changes Date: Thu, 7 Aug 2008 18:23:01 +0200 User-Agent: KMail/1.9.5 Cc: Herbert Xu , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , Ingo Molnar References: <200807171653.59177.wolfgang.walter@stwm.de> <200808061933.25631.wolfgang.walter@stwm.de> <20080806201401.GA607@linux-os.sc.intel.com> In-Reply-To: <20080806201401.GA607@linux-os.sc.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8BIT Content-Disposition: inline Message-Id: <200808071823.02364.wolfgang.walter@stwm.de> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 18667 Lines: 465 Am Mittwoch, 6. August 2008 22:14 schrieb Suresh Siddha: > On Wed, Aug 06, 2008 at 10:33:25AM -0700, Wolfgang Walter wrote: > > Hello Herbert, > > > > I think I finally found the problem. > > > > Here a short description again: all our routers with a via C3 using > > padlock for AES-encryption are crashing with 2.6.26 while they work fine > > with 2.6.25. Not using padlock (i.e. using the i386 assembler version of > > AES) they just work fine. > > Both the padlock version or asm version don't use FP/math registers, right? > It is interesting that you don't see the problem with asm version > but see the problem with padlock version. > > Does disabling CONFIG_PREEMPT in 2.6.26 change anything? And also, Didn't check that yet as I'm still running my modified 2.6.26. It now runs almost one day flawlessly. But yesterday I tried the following patch on top of a vanilla 2.6.26: ======================================================= diff -ur ../linux-2.6.26/drivers/crypto/padlock-aes.c ./drivers/crypto/padlock-aes.c --- ../linux-2.6.26/drivers/crypto/padlock-aes.c 2008-07-15 11:29:32.000000000 +0200 +++ ./drivers/crypto/padlock-aes.c 2008-08-07 17:46:55.000000000 +0200 @@ -16,6 +16,7 @@ #include #include #include +#include #include "padlock.h" /* Control word. */ @@ -144,9 +145,11 @@ static inline void padlock_xcrypt(const u8 *input, u8 *output, void *key, void *control_word) { + kernel_fpu_begin(); asm volatile (".byte 0xf3,0x0f,0xa7,0xc8" /* rep xcryptecb */ : "+S"(input), "+D"(output) : "d"(control_word), "b"(key), "c"(1)); + kernel_fpu_end(); } static void aes_crypt_copy(const u8 *in, u8 *out, u32 *key, struct cword *cword) @@ -179,6 +182,7 @@ return; } + kernel_fpu_begin(); asm volatile ("test $1, %%cl;" "je 1f;" "lea -1(%%ecx), %%eax;" @@ -190,15 +194,18 @@ : "+S"(input), "+D"(output) : "d"(control_word), "b"(key), "c"(count) : "ax"); + kernel_fpu_end(); } static inline u8 *padlock_xcrypt_cbc(const u8 *input, u8 *output, void *key, u8 *iv, void *control_word, u32 count) { /* rep xcryptcbc */ + kernel_fpu_begin(); asm volatile (".byte 0xf3,0x0f,0xa7,0xd0" : "+S" (input), "+D" (output), "+a" (iv) : "d" (control_word), "b" (key), "c" (count)); + kernel_fpu_end(); return iv; } ============================================================= I found that kernel_fpu_begin(); kernel_fpu_begin(); is used with MMX and/or SSE: include/asm/xor_32.h drivers/md/raid6mmx.c drivers/md/raid6sse1.c drivers/md/raid6sse2.c With this change I its a little bit more stable, I needed more then 5 minutes to crash the kernel (repeated it several times). If I read the code correctly this disables preemption for the time the padlock cmd is executing. > can you provide the complete kernel log till the point of failure(oops > that you sent doesn't have the call trace info) > Here are some of the oopses (all with unmodified 2.6.26): ################################################################## BUG: unable to handle kernel NULL pointer dereference at 000001f0 IP: [] __switch_to+0x23/0x103 *pde = 00000000 Oops: 0002 [#1] PREEMPT Modules linked in: Pid: 2014, comm: date Not tainted (2.6.26 #3) EIP: 0060:[] EFLAGS: 00010002 CPU: 0 EIP is at __switch_to+0x23/0x103 EAX: 00000000 EBX: dc4e2dc0 ECX: 0000015f EDX: dc4e2dc0 ESI: de9634a0 EDI: dc4e2fe8 EBP: de9636c8 ESP: cee9deec DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 Process date (pid: 2014, ti=cee9c000 task=de9634a0 task.ti=d13c6000) Stack: dc4e2dc0 00000000 cee81300 de9634a0 c039bf41 dc4e2dc0 00000082 de9634a0 00000000 dc4e2f2c 00000046 00000000 ffffffea 00000001 dc4e2db8 00000000 c011e766 0000000e 00000000 00000003 dc4e2dc0 00000001 00000010 dc4e2dc0 Call Trace: [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= Code: 00 90 5d 5b 5e 5f 5d c3 55 57 56 89 c6 53 89 d3 8d a8 28 02 00 00 8b 40 04 8d ba 28 02 00 00 f6 40 0c 01 74 31 8b 86 7c 02 00 00 <0f> ae 00 0f ba 60 02 07 73 02 db e2 8d 76 00 90 8d b4 26 00 00 EIP: [] __switch_to+0x23/0x103 SS:ESP 0068:cee9deec ---[ end trace 55bd4c8258a0a1eb ]--- Fixing recursive fault but reboot is needed! BUG: scheduling while atomic: bash/1363/0x00000003 Pid: 2014, comm: date Tainted: G D 2.6.26 #3 [] schedule+0x58/0x2bf [] printk+0x14/0x18 [] do_exit+0x9f/0x54a [] printk+0x14/0x18 [] die+0xfd/0x102 [] do_page_fault+0x488/0x53c [] do_page_fault+0x0/0x53c [] error_code+0x6a/0x70 [] __switch_to+0x23/0x103 [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= note: bash[1363] exited with preempt_count 2 BUG: unable to handle kernel NULL pointer dereference at 00000028 IP: [] mm_release+0x39/0x64 *pde = 00000000 Oops: 0000 [#2] PREEMPT Modules linked in: Pid: 1363, comm: bash Tainted: G D (2.6.26 #3) EIP: 0060:[] EFLAGS: 00010246 CPU: 0 EIP is at mm_release+0x39/0x64 EAX: 00000000 EBX: de9634a0 ECX: b7e306f8 EDX: 00000000 ESI: 00000000 EDI: de9634a0 EBP: c041e382 ESP: cee9de34 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 Process bash (pid: 1363, ti=cee9c000 task=dc4e2dc0 task.ti=cee9c000) Stack: 00000000 de9634a0 c011d7d1 00000002 de9634a0 0000000b c011eb01 00000000 c041e382 c011cea8 c041ebdc cee9de70 cee9de70 cee9deb4 00000006 00000002 c041e382 c010500f 00000000 000001f0 de9634a0 00000038 c01123d8 00000002 Call Trace: [] exit_mm+0x12/0xb4 [] do_exit+0x198/0x54a [] printk+0x14/0x18 [] die+0xfd/0x102 [] do_page_fault+0x488/0x53c [] do_page_fault+0x0/0x53c [] error_code+0x6a/0x70 [] __switch_to+0x23/0x103 [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= Code: 8e e8 85 d2 74 11 c7 83 40 01 00 00 00 00 00 00 89 d0 e8 42 f2 ff ff 8b 8b 48 01 00 00 85 c9 74 32 8b 43 0c 25 00 04 00 00 75 28 <83> 7e 28 01 7e 22 c7 83 48 01 00 00 00 00 00 00 e8 4b 41 0e 00 EIP: [] mm_release+0x39/0x64 SS:ESP 0068:cee9de34 ---[ end trace 55bd4c8258a0a1eb ]--- ################################################################## BUG: unable to handle kernel NULL pointer dereference at 000001f0 IP: [] __switch_to+0x23/0x103 *pde = 00000000 Oops: 0002 [#1] PREEMPT Modules linked in: Pid: 1569, comm: date Not tainted (2.6.26 #7) EIP: 0060:[] EFLAGS: 00010002 CPU: 0 EIP is at __switch_to+0x23/0x103 EAX: 00000000 EBX: de978000 ECX: 00000091 EDX: de978000 ESI: de979810 EDI: de978228 EBP: de979a38 ESP: d13ddeec DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 Process date (pid: 1569, ti=d13dc000 task=de979810 task.ti=d185e000) Stack: de978000 00000000 cfdbe780 de979810 c039c229 de978000 00000082 de979810 00000000 de97816c 00000046 00000000 ffffffea 00000001 de977ff8 00000000 c011e766 0000000e 00000000 00000003 de978000 00000001 00000010 de978000 Call Trace: [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= Code: 00 90 5d 5b 5e 5f 5d c3 55 57 56 89 c6 53 89 d3 8d a8 28 02 00 00 8b 40 04 8d ba 28 02 00 00 f6 40 0c 01 74 31 8b 86 7c 02 00 00 <0f> ae 00 0f ba 60 02 07 73 02 db e2 8d 76 00 90 8d b4 26 00 00 EIP: [] __switch_to+0x23/0x103 SS:ESP 0068:d13ddeec ---[ end trace 8061cb3e89ac8fe5 ]--- Fixing recursive fault but reboot is needed! BUG: scheduling while atomic: bash/1374/0x00000003 Pid: 1569, comm: date Tainted: G D 2.6.26 #7 [] schedule+0x58/0x2bf [] printk+0x14/0x18 [] do_exit+0x9f/0x54a [] printk+0x14/0x18 [] die+0xfd/0x102 [] do_page_fault+0x488/0x53c [] do_page_fault+0x0/0x53c [] error_code+0x6a/0x70 [] __switch_to+0x23/0x103 [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= note: bash[1374] exited with preempt_count 2 BUG: unable to handle kernel NULL pointer dereference at 00000028 IP: [] mm_release+0x39/0x64 *pde = 00000000 Oops: 0000 [#2] PREEMPT Modules linked in: Pid: 1374, comm: bash Tainted: G D (2.6.26 #7) EIP: 0060:[] EFLAGS: 00010246 CPU: 0 EIP is at mm_release+0x39/0x64 EAX: 00000000 EBX: de979810 ECX: b7e316f8 EDX: 00000000 ESI: 00000000 EDI: de979810 EBP: c041e382 ESP: d13dde34 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 Process bash (pid: 1374, ti=d13dc000 task=de978000 task.ti=d13dc000) Stack: 00000000 de979810 c011d7d1 00000002 de979810 0000000b c011eb01 00000000 c041e382 c011cea8 c041ebdc d13dde70 d13dde70 d13ddeb4 00000006 00000002 c041e382 c010500f 00000000 000001f0 de979810 00000038 c01123d8 00000002 Call Trace: [] exit_mm+0x12/0xb4 [] do_exit+0x198/0x54a [] printk+0x14/0x18 [] die+0xfd/0x102 [] do_page_fault+0x488/0x53c [] do_page_fault+0x0/0x53c [] error_code+0x6a/0x70 [] __switch_to+0x23/0x103 [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= Code: 8e e8 85 d2 74 11 c7 83 40 01 00 00 00 00 00 00 89 d0 e8 42 f2 ff ff 8b 8b 48 01 00 00 85 c9 74 32 8b 43 0c 25 00 04 00 00 75 28 <83> 7e 28 01 7e 22 c7 83 48 01 00 00 00 00 00 00 e8 4b 41 0e 00 EIP: [] mm_release+0x39/0x64 SS:ESP 0068:d13dde34 ---[ end trace 8061cb3e89ac8fe5 ]--- note: bash[1374] exited with preempt_count 2 ################################################################## BUG: unable to handle kernel NULL pointer dereference at 000001f0 IP: [] __switch_to+0x23/0x103 *pde = 00000000 Oops: 0002 [#1] PREEMPT Modules linked in: Pid: 1591, comm: date Not tainted (2.6.26 #8) EIP: 0060:[] EFLAGS: 00010002 CPU: 0 EIP is at __switch_to+0x23/0x103 EAX: 00000000 EBX: dc4e3b80 ECX: 000000e6 EDX: dc4e3b80 ESI: dc42bb80 EDI: dc4e3da8 EBP: dc42bda8 ESP: cc8d5eec DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 Process date (pid: 1591, ti=cc8d4000 task=dc42bb80 task.ti=cf542000) Stack: dc4e3b80 00000000 dc4b8000 dc42bb80 c039bbc9 dc4e3b80 00000082 dc42bb80 00000000 dc4e3cec 00000046 00000000 ffffffea 00000001 dc4e3b78 00000000 c011e766 0000000e 00000000 00000003 dc4e3b80 00000001 00000010 dc4e3b80 Call Trace: [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= Code: 00 90 5d 5b 5e 5f 5d c3 55 57 56 89 c6 53 89 d3 8d a8 28 02 00 00 8b 40 04 8d ba 28 02 00 00 f6 40 0c 01 74 31 8b 86 7c 02 00 00 <0f> ae 00 0f ba 60 02 07 73 02 db e2 8d 76 00 90 8d b4 26 00 00 EIP: [] __switch_to+0x23/0x103 SS:ESP 0068:cc8d5eec ---[ end trace fa9f688d2faab2aa ]--- Fixing recursive fault but reboot is needed! BUG: scheduling while atomic: bash/1374/0x00000003 Pid: 1591, comm: date Tainted: G D 2.6.26 #8 [] schedule+0x58/0x2bf [] printk+0x14/0x18 [] do_exit+0x9f/0x54a [] printk+0x14/0x18 [] die+0xfd/0x102 [] do_page_fault+0x488/0x53c [] do_page_fault+0x0/0x53c [] error_code+0x6a/0x70 [] __switch_to+0x23/0x103 [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= note: bash[1374] exited with preempt_count 2 BUG: unable to handle kernel NULL pointer dereference at 00000028 IP: [] mm_release+0x39/0x64 *pde = 00000000 Oops: 0000 [#2] PREEMPT Modules linked in: Pid: 1374, comm: bash Tainted: G D (2.6.26 #8) EIP: 0060:[] EFLAGS: 00010246 CPU: 0 EIP is at mm_release+0x39/0x64 EAX: 00000000 EBX: dc42bb80 ECX: b7e4a6f8 EDX: 00000000 ESI: 00000000 EDI: dc42bb80 EBP: c041e216 ESP: cc8d5e34 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 Process bash (pid: 1374, ti=cc8d4000 task=dc4e3b80 task.ti=cc8d4000) Stack: 00000000 dc42bb80 c011d7d1 00000002 dc42bb80 0000000b c011eb01 00000000 c041e216 c011cea8 c041ea70 cc8d5e70 cc8d5e70 cc8d5eb4 00000006 00000002 c041e216 c010500f 00000000 000001f0 dc42bb80 00000038 c01123d8 00000002 Call Trace: [] exit_mm+0x12/0xb4 [] do_exit+0x198/0x54a [] printk+0x14/0x18 [] die+0xfd/0x102 [] do_page_fault+0x488/0x53c [] do_page_fault+0x0/0x53c [] error_code+0x6a/0x70 [] __switch_to+0x23/0x103 [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= Code: 8e e8 85 d2 74 11 c7 83 40 01 00 00 00 00 00 00 89 d0 e8 42 f2 ff ff 8b 8b 48 01 00 00 85 c9 74 32 8b 43 0c 25 00 04 00 00 75 28 <83> 7e 28 01 7e 22 c7 83 48 01 00 00 00 00 00 00 e8 9b 3d 0e 00 EIP: [] mm_release+0x39/0x64 SS:ESP 0068:cc8d5e34 ---[ end trace fa9f688d2faab2aa ]--- note: bash[1374] exited with preempt_count 2 BUG: scheduling while atomic: bash/1374/0x10000003 Pid: 1374, comm: bash Tainted: G D 2.6.26 #8 [] schedule+0x58/0x2bf [] dput+0x15/0xfc [] _atomic_dec_and_lock+0x30/0x38 [] mntput_no_expire+0x11/0xd2 [] __cond_resched+0x13/0x2f [] _cond_resched+0x21/0x2a [] put_files_struct+0x63/0xa6 [] do_exit+0x1a6/0x54a [] printk+0x14/0x18 [] die+0xfd/0x102 [] do_page_fault+0x488/0x53c [] do_page_fault+0x0/0x53c [] error_code+0x6a/0x70 [] mm_release+0x39/0x64 [] exit_mm+0x12/0xb4 [] do_exit+0x198/0x54a [] printk+0x14/0x18 [] die+0xfd/0x102 [] do_page_fault+0x488/0x53c [] do_page_fault+0x0/0x53c [] error_code+0x6a/0x70 [] __switch_to+0x23/0x103 [] schedule+0x257/0x2bf [] do_wait+0x88e/0x963 [] handle_mm_fault+0x441/0x4aa [] tty_ioctl+0x0/0x877 [] default_wake_function+0x0/0x8 [] sys_wait4+0x7f/0x92 [] sys_waitpid+0x13/0x17 [] syscall_call+0x7/0xb ======================= ################################################################## BUG: unable to handle kernel NULL pointer dereference at 000001f0 IP: [] __switch_to+0x30/0x117 *pde = 00000000 Oops: 0002 [#1] PREEMPT Modules linked in: Pid: 2071, comm: sleep Not tainted (2.6.26 #11) EIP: 0060:[] EFLAGS: 00010002 CPU: 0 EIP is at __switch_to+0x30/0x117 EAX: 00000000 EBX: c0493300 ECX: dc48dd00 EDX: c0493300 ESI: dc48dd00 EDI: c0493530 EBP: c04cff8c ESP: c04cff7c DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068 Process sleep (pid: 2071, ti=c04ce000 task=dc48dd00 task.ti=d2fe6000) Stack: dc48df30 c0493300 00000000 00000000 d2fe7f44 c03b5b43 c04cffc8 00000046 c0131856 0000005a dc472d3c c0493300 c0493470 d983ae00 00002696 00000000 c0239f54 00000000 c04c4000 c04cffd8 c01025fe c04f3740 00049800 c04cffe0 Call Trace: [] ? schedule+0x285/0x2ff [] ? pm_qos_requirement+0x3c/0x53 [] ? acpi_processor_idle+0x0/0x434 [] ? cpu_idle+0x73/0x7f [] ? rest_init+0x61/0x63 ======================= Code: 89 c6 53 89 d3 83 ec 04 8d 80 30 02 00 00 89 45 f0 8d ba 30 02 00 00 e8 a2 bd 10 00 8b 46 04 f6 40 0c 01 74 31 8b 86 84 02 00 00 <0f> ae 00 0f ba 60 02 07 73 02 db e2 8d 76 00 90 8d b4 26 00 00 EIP: [] __switch_to+0x30/0x117 SS:ESP 0068:c04cff7c ---[ end trace 042daf7e67838617 ]--- note: sleep[2071] exited with preempt_count 2 BUG: scheduling while atomic: swapper/0/0x00000004 Pid: 2071, comm: sleep Tainted: G D 2.6.26 #11 [] __schedule_bug+0x42/0x47 [] schedule+0x65/0x2ff [] ? put_fs_struct+0x39/0x3c [] do_exit+0x54e/0x55d [] ? print_oops_end_marker+0x1e/0x23 [] die+0x104/0x10c [] do_page_fault+0x477/0x52c [] ? do_page_fault+0x0/0x52c [] error_code+0x6a/0x70 [] ? __switch_to+0x30/0x117 [] schedule+0x285/0x2ff [] ? pm_qos_requirement+0x3c/0x53 [] ? acpi_processor_idle+0x0/0x434 [] ? cpu_idle+0x73/0x7f [] ? rest_init+0x61/0x63 ################################################################## Regards, -- Wolfgang Walter Studentenwerk M?nchen Anstalt des ?ffentlichen Rechts -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/