Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1754917AbbDJQn6 (ORCPT <rfc822;w@1wt.eu>);
	Fri, 10 Apr 2015 12:43:58 -0400
Received: from mail-wg0-f42.google.com ([74.125.82.42]:35552 "EHLO
	mail-wg0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1751554AbbDJQny (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Fri, 10 Apr 2015 12:43:54 -0400
From: Sathya Prakash <sathya.prakash@avagotech.com>
References: <1428650094-12750-1-git-send-email-calvinowens@fb.com> <1428676329.2178.2.camel@Odin.com>
In-Reply-To: <1428676329.2178.2.camel@Odin.com>
MIME-Version: 1.0
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQG6XdxqAPvicjuKpNZf7irnD9BgLQHNJZP1nWQ0QdA=
Date: Fri, 10 Apr 2015 11:43:53 -0500
Message-ID: <5e187e92fcbd75e07e7caf9cd9a87e12@mail.gmail.com>
Subject: RE: [PATCH] mpt2sas: mpt3sas: Fix memory corruption during initialization
To: James Bottomley <jbottomley@odin.com>, calvinowens@fb.com
Cc: linux-kernel@vger.kernel.org,
        PDL-MPT-FUSIONLINUX <MPT-FusionLinux.pdl@avagotech.com>,
        kernel-team@fb.com, stable@vger.kernel.org,
        praveen.krishnamoorthy@avagotech.com, abhijit.mahajan@avagotech.com,
        nagalakshmi.nandigama@avagotech.com,
        Sreekanth Reddy <sreekanth.reddy@avagotech.com>
Content-Type: text/plain; charset=UTF-8
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 4296
Lines: 93

James & Calvin,
Noted this,  we will review and ACK/revert back with further questions in
next couple of weeks.

Thanks
Sathya

-----Original Message-----
From: James Bottomley [mailto:jbottomley@odin.com]
Sent: Friday, April 10, 2015 9:31 AM
To: calvinowens@fb.com
Cc: linux-kernel@vger.kernel.org; MPT-FusionLinux.pdl@avagotech.com;
kernel-team@fb.com; stable@vger.kernel.org;
praveen.krishnamoorthy@avagotech.com; abhijit.mahajan@avagotech.com;
nagalakshmi.nandigama@avagotech.com; sreekanth.reddy@avagotech.com
Subject: Re: [PATCH] mpt2sas: mpt3sas: Fix memory corruption during
initialization

On Fri, 2015-04-10 at 00:14 -0700, Calvin Owens wrote:
> While _scsih_probe_sas() is initializing devices, the hardware can
> trigger a MPI2_EVENT_SAS_TOPO_RC_TARG_NOT_RESPONDING interrupt.
>
> The handler for TARG_NOT_RESPONDING calls
> _scsih_device_remove_by_handle(), which deletes the device in question
> from either ioc->sas_device_list or
> ioc->sas_device_init_list. Since _scsih_probe_sas() uses no exclusion
> ioc->when
> iterating over ioc->sas_device_init_list, this results in a
> use-after-free in _scsih_probe_sas(), and also corrupts the list:
>
>   mpt2sas1: removing handle(0x0020), sas_addr(0x5f80f418573360e0)
>   mpt2sas1: log_info(0x31111000): originator(PL), code(0x11),
> sub_code(0x1000)
>   ------------[ cut here ]------------
>   WARNING: at lib/list_debug.c:56 __list_del_entry+0xc3/0xd0()
>   list_del corruption, ffff88240012fa00->prev is LIST_POISON2
> (dead000000200200)
>   <snip>
>   Workqueue: events work_for_cpu_fn
>    ffffffff810c4f17 ffff881214825b38 0000000000000009 ffff881214825ae8
>    ffffffff8169b61e ffff881214825b28 ffffffff8104a990 0000000000000002
>    ffff88240012f900 ffff88240012fa00 ffff881217595af8 0000000000000282
>   Call Trace:
>    [<ffffffff810c4f17>] ? print_modules+0xd7/0x120
>    [<ffffffff8169b61e>] dump_stack+0x19/0x1b
>    [<ffffffff8104a990>] warn_slowpath_common+0x70/0xa0
>    [<ffffffff8104aa76>] warn_slowpath_fmt+0x46/0x50
>    [<ffffffff816a2234>] ? _raw_spin_lock_irqsave+0x84/0xa0
>    [<ffffffffa0010e8e>] ? _scsih_probe_sas+0x8e/0x110 [mpt2sas]
>    [<ffffffff8132a5a3>] __list_del_entry+0xc3/0xd0
>    [<ffffffffa0010e99>] _scsih_probe_sas+0x99/0x110 [mpt2sas]
>    [<ffffffffa0011d5f>] _scsih_scan_finished+0x19f/0x2c0 [mpt2sas]
>    [<ffffffff81429d67>] do_scsi_scan_host+0x77/0xa0
>    [<ffffffff81429f20>] scsi_scan_host+0x190/0x1c0
>    [<ffffffffa0011402>] _scsih_probe+0x452/0x640 [mpt2sas]
>    [<ffffffff813444eb>] local_pci_probe+0x4b/0x80
>    [<ffffffff8106b848>] work_for_cpu_fn+0x18/0x30
>    [<ffffffff81070012>] process_one_work+0x212/0x6e0
>    [<ffffffff8106ffa6>] ? process_one_work+0x1a6/0x6e0
>    [<ffffffff8108ed1f>] ? local_clock+0x4f/0x60
>    [<ffffffff8107050c>] process_scheduled_works+0x2c/0x40
>    [<ffffffff81070ae2>] worker_thread+0x262/0x370
>    [<ffffffff81070880>] ? rescuer_thread+0x360/0x360
>    [<ffffffff81078f3b>] kthread+0xdb/0xe0
>    [<ffffffff810b5e8d>] ? trace_hardirqs_on+0xd/0x10
>    [<ffffffff81078e60>] ? kthread_create_on_node+0x140/0x140
>    [<ffffffff816ac01c>] ret_from_fork+0x7c/0xb0
>    [<ffffffff81078e60>] ? kthread_create_on_node+0x140/0x140
>   ---[ end trace 41352a0bd2d0d61b ]---
>
> This either results in an immediate panic, or corrupts random memory
> and causes nasty problems later in the box's uptime.
>
> This patch splices the discovered devices out of the global list while
> holding the lock, since _scsih_probe_sas() always removes them from
> that global list anyway (either deleting them if initialization fails,
> or moving them onto ioc->sas_device_list if it succeeds). The
> interrupt that caused this bug will no longer cause the device to be
> removed during initialization, since it won't exist on the global
> lists, but
> _scsih_probe_sas() will remove it anyway when it fails to initialize.

Hopefully the avago team will curate this, but just in case they don't, the
correct list to make sure it gets the attention of storage people should be

linux-scsi@vger.kernel.org

James
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/