Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752890AbaDDMZY (ORCPT ); Fri, 4 Apr 2014 08:25:24 -0400 Received: from mail-lb0-f179.google.com ([209.85.217.179]:41470 "EHLO mail-lb0-f179.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752815AbaDDMZV (ORCPT ); Fri, 4 Apr 2014 08:25:21 -0400 MIME-Version: 1.0 X-Originating-IP: [5.150.199.142] In-Reply-To: References:

Date: Fri, 4 Apr 2014 14:25:19 +0200 Message-ID: Subject: Re: Crash in rbd, need advice From: Hannes Landeholm To: Ilya Dryomov Cc: Linux Kernel Mailing List , Ceph Development , Sage Weil , Sage Weil , Alex Elder , Yehuda Sadeh , Thorwald Lundqvist Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Apr 4, 2014 at 1:08 PM, Ilya Dryomov wrote: > On Wed, Apr 2, 2014 at 12:18 AM, Hannes Landeholm wrote: >> Hello, >> >> We're running a couple of Arch Linux servers of version 3.13.5-1 in >> production and suddenly one of them had a strange problem after >> running for a few days. One process (pid 319) was running with a few >> threads, one of those threads (pid 322) was eating 100% cpu. I assumed >> it was stuck in an infinite loop (this was our own software so I >> assumed we had a bug) so I sent a SIGKILL to 319 which caused all >> other threads to exit and it turning into a zombie, but thread 322 was >> still running. After trying to stop some other services and failing I >> realized that sending any signals to any process now didn't work at >> all in the system. >> >> This was the process stack output: >> >> $ cat /proc/319/stack >> [] do_exit+0x73a/0xa80 >> [] do_group_exit+0x3f/0xa0 >> [] get_signal_to_deliver+0x295/0x5f0 >> [] do_signal+0x48/0x950 >> [] do_notify_resume+0x68/0xa0 >> [] int_signal+0x12/0x17 >> [] 0xffffffffffffffff >> $ cat /proc/319/task/322/stack >> [] error_exit+0x2a/0x60 >> [] 0xffffffffffffffff >> >> We're using ceph + rbd and this happened right after doing a rbd >> mapping (mounting it) or during the mapping itself, so we suspected >> rbd. >> >> A few days later (today) we had a server crash in another server, same >> version+distro and it had also just been running a few days as well. >> After starting it again we found the following in the system log: >> >> hostname kernel: BUG: unable to handle kernel paging request at ffff87fff75ad450 >> hostname kernel: IP: [] rbd_img_request_fill+0x126/0x930 [rbd] > > Can you try gdb'ing that exact rbd.ko and > > (gdb) list *rbd_img_request_fill+0x126 > > Also, the entire stack trace pertaining to rbd_img_request_fill would > help. > > Thanks, > > Ilya Sorry, rbd was not built with any symbols and there was no other output in the journal. We're building a debug version of Linux 3.14 (with symbols) now that is currently being tested and we hope that we can roll it out today. I saw that at least one commit (638c323 rbd: drop an unsafe assertion) looks like something that could fix a crash that has been happening regularly on one of our development servers: http://i.imgur.com/pEKsmql.jpg (rbd_img_obj_callback in the stack trace). It might be related to the production crash we had. If we continue to get kernel panics we should at least be able to provide a proper stack trace in the future. Thank you for your time, Hannes -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/