Date: Wed, 11 Jun 2014 14:12:14 +0200
From: Jan Kara <jack@suse.cz>
To: Weng Meiling <wengmeiling.weng@huawei.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        jack@suse.cz, akpm@linux-foundation.org, adilger.kernel@dilger.ca,
        Jens Axboe <axboe@kernel.dk>, Li Zefan <lizefan@huawei.com>,
        Huang Qiang <h.huangqiang@huawei.com>,
        Zhao Hongjiang <zhaohongjiang@huawei.com>
Subject: Re: [linux 3.4 question] reboot command stall when vdbench test
Message-ID: <20140611121214.GD3661@quack.suse.cz>
References: <5397FC71.8020403@huawei.com>
 <53981100.7030404@huawei.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <53981100.7030404@huawei.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org

  Hello,

On Wed 11-06-14 16:19:12, Weng Meiling wrote:
> We run vdbench test in our suse system with kernel 3.4, the vdbench test
> is about different block size seq and rand read/write. Before the vdbench
  Hum, this looks like some relatively old (not supported anymore)
openSUSE, right?

> test, we had did some test about: disk message lookup, raid rebuild(note
> we use hard raid: SAS2008 RAID).
> 
> we used nohup to run the vdbench test script:
> 
> #nohup ./vdbench_batch_test &
> 
> During test,  we cat the result file:
> 
> #cat nohup.out
> 
> at this time, the cat command stalled, then try to reboot, but the system
> didn't reboot, and the reboot also stalled, shutdown gone to uninterruptible
> sleep:
  Yeah, looking at the logs from sysrq, the machine seems to be waiting for
IO to complete which never happened. Most often I've seen this happening
because of a bug in driver for the hardware raid sometimes also because of
a bug in the firmware of the card itself. So I'd update the card firmware
to the latest version and check changes to the driver since the kernel
version you run.

> root     21716  0.0  0.0   4276   556 ?        D    18:31   0:00 cat nohup.out
> root     21726  0.0  0.0  17880  2876 ?        Ds   18:33   0:00 -bash
> root     21868  0.0  0.0   8224   740 ?        D    19:03   0:00 shutdown -r 0 w
> root     21892  0.0  0.0  17880  2884 ?        Ds   19:11   0:00 -bash
> root     21967  0.0  0.0   8224   740 ?        D    19:19   0:00 shutdown -r 0 w
> root     21970  0.0  0.0  86044  3680 ?        Ss   19:19   0:00 sshd: root@pts/4
> root     21975  0.0  0.0  17880  2880 pts/4    Ss   19:19   0:00 -bash
> root     22000  0.0  0.0  12932  1280 pts/4    T    19:20   0:00 top
> 
> after several hours the system gone to dead, all the ssh connect stalled,
> we can't connect to this server any more. The status kept for a week,
> finally we had to reboot the system by power key. After system reboot, we
> done the same steps to try to reproduce the problem for more than a
> month, but it didn't happen again.
> 
> We had analysed the code and lock information according the call trace,
> also review linux 3.4+ mainline patch to find similar problem fix, but no
> result.
> 
> Many others met the similar problem because use SAN/NFS/multipath devices, but we don't use none of these.
> 
> The attachments are our test program and dmesg information we get by
> sysrq before system dead.  Does anyone met the problem before? Any
> suggestion is appreciative. Thanks!

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/