Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S964804AbcLWAQo (ORCPT ); Thu, 22 Dec 2016 19:16:44 -0500 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:53849 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1761600AbcLWAQl (ORCPT ); Thu, 22 Dec 2016 19:16:41 -0500 Subject: Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0 To: Linus Torvalds , Dave Chinner References: <20161214222411.GH4326@dastard> <20161214222953.GI4326@dastard> <20161216185906.t2wmrr6wqjdsrduw@straylight.hirudinean.org> <20161221221638.GD4758@dastard> <20161222001303.nvrtm22szn3hgxar@straylight.hirudinean.org> <20161222051322.GF4758@dastard> <20161222062858.GG4758@dastard> <20161222204240.GJ4758@dastard> <20161222210607.GK4758@dastard> CC: Thomas Gleixner , Ingo Molnar , Peter Anvin , Linux Kernel Mailing List , the arch/x86 maintainers From: Jens Axboe Message-ID: <60385463-d6e6-3880-029e-93965f1fc940@fb.com> Date: Thu, 22 Dec 2016 17:16:23 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [66.29.164.166] X-ClientProxiedBy: BN6PR1401CA0019.namprd14.prod.outlook.com (10.174.237.157) To BN6PR15MB1186.namprd15.prod.outlook.com (10.172.205.140) X-MS-Office365-Filtering-Correlation-Id: 419c7512-083e-43d1-3e1e-08d42ac8f0d0 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:BN6PR15MB1186; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;3:LP+6epEEqZPj00TyG9eK6otjuxTbCSOUOiJK6j7cbj68PE8WQBIIrkysRoACevI/xvk4G1iFIxqWtW4dMHkOj8dHKe3THL5CGIBcANkDzu4b2QJpkdJ0EGZqU9k+j5eVrI2bm75MC8t4SSNW6v/aXWPLwnvROyxBGP/vG7wBwTkgaxsiUjhSnxOCqHsJvkkQecAb/5Tz6gqzuHR/IeIqPxZwDb/HDUypRa597wU/kOfBJwSF3lRZVqGOwZabpZr/wlHPmvIllLlCN1rA3gaqUw==;25:N65Z5m3e/nz2kCuzDrZBB3tfNXIwlMh270prE7Ibc+4AdKcKlydXDbqckE3kvzaVBmXFf0SBrHtUmXj3xt9fHFSU9IJQERZD+YsBeDrm3T1hO1bmJL9HpKOkdDpIP5D1CvuFtKm7iSQL8lQlopBbRDexsylXBHgt22B6SebENxzJHhZWgXG7MHtGnbXH6E2BXcVtxHLqQ3GmnNkO0i6PnmBF9yKIKR4WgNrhCDGhZP0C+oBMAJEovrtnZfeOI7liHOj2G42RQXuhSQjczGaedb/5hWf0dRe9SlJ7mbbAI9n4uI4Np2tOqpyYvRECNdFAKUQLegDZzVEKfdT3H0oXorQMlhozclHnzEVVH8QL2yiHteRl49aYHp8IcEXyxejQp9r875cStFgEicvaQprEVBm8iLJ3PTeeAWsYPSUPvKe8Wd0h/wd+bpwFvV91iCvO0zTJ+AmEIST/LYJDbBIPBQ== X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;31:w8QUQ2VNo/O6oKZSVho41cDWj2Xc7QR6fOeuQSZxYHt4wVuRzd5wYwVAEakATXok3KlvpxfLw6vUC+ggJsRWYEEB/Al+L8kCRe5cEAKRbkAtB3461w3toqNfc19oeHO8KuKKkQRXy15EkOVgdeOmAYWyMNT4vQFwG5mcjGL8/zLOntLAi1XH1OieJZ8BSzaxp+ral0Dyfeae6//q4fPAV5hIdu/IJMR9q1O4GgwPSomKHZbvrnR3m0S+2bQr4VBKqpcQ8WIQ+igMaMszQu0opQ==;20:qKJLBuoTw5HbXFRELQe5ELMXBVXKrH55kO0fcK/jLpeUNNdPHxnifH96JeUPesLJfOt9WOpXRFy8kzJ9cd9gKA8w1MogyeiGLLkc5r3AP0JmLpRVVDgtX4ulgRZAXn04cCT+x2gC1XOuiPICxRjM1fDwP34452Mw6L2BiSFNZr1wgx5ygK0EDf20xFvwkgX2lJfbl0iMu+Hwdv40ogs+UR1HGBeqz4B80Smcm5dL6BnyVtqB10Nk4u9cVkTMzsJ0b1UkGl603osQFUOVg0nSWVBICvl7+Wy/c4PKnchmd1WVH9C2FTSzpCOtxpB6+VzRZxCR7rz5FmU9AhrOxm6QYighcp/vL/b1puxGrnGOrUY2WYIobj6oyvoGXj3ilIMLrA8hDvvQVEAPs8ne5rXiypaYGLcc9n3l8t0BbGrW7G3O7pLaF3xONlC0yt4u6Wg6cTsh+FtUO9+LkjD+QrCUFlhBpkimOAYQnvhRla6wwzwy5bFDBJeYH9HGV45vxObL X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(81227570615382); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6041248)(20161123562025)(20161123560025)(20161123564025)(20161123555025)(6072148);SRVR:BN6PR15MB1186;BCL:0;PCL:0;RULEID:;SRVR:BN6PR15MB1186; X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;4:ke0lqsKmge7INv/8O8NlY7aUXdWE5bai+BSDQiCWw1vAxllDXnkJeKaF8t3P+CaT4cs8a8cxpAvYREHwBlJi0ytq51DMQUnSDxaHxCfzqI1X822VR8wlXhMP02Tont2MNPyJP29ZJ5ONTGEBFOOi42TJWH3qPSeOWmVFwePhebxvbEVKPuvFTz8PvWIOlIRwHeA2gDHe6Len6QF9Ub2MrHIqc1CZFTJCHMp9bfwkx8Dfgc2yYMUJ5zuevitiXVof0iPhCVCbh9qlukTfimHP44xoGbox60BoIW7WQ1XCFqASCk17Fsj7sqTZJlYAD0YKKO01CAnfZ6mOX/iiNk31VSHErG+tz2rNj2hReThPRgF7+ScNUOowna10HZFRDcJGxodWCr6x/w+V02FLLNWr3Wp5SlSrY9xzqFfqbJvRWgh+3kFfGeB9BsQ2wMGjmz4I5+QKwwLg2br5cBSJfji0TgxBGq+SZ1YQHeI0JtXzIWlqqlQsUm6WAJrQ0gOVPYRrynieOQyTWyOtD6AYMu2zyljKu89kN3Vk29Hhj/74hyCJCPaEJ2A4g0jlp+OJ0rYAIhMUnUJzbnNyYiSNKIBeshgU75mpjk1Kyh6HWFQlC/w= X-Forefront-PRVS: 016572D96D X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6049001)(6009001)(7916002)(39830400002)(39410400002)(39450400003)(24454002)(189002)(377454003)(199003)(23676002)(117156001)(3846002)(38730400001)(8676002)(6116002)(42186005)(6486002)(230700001)(2906002)(81166006)(93886004)(66066001)(81156014)(4326007)(68736007)(90366009)(50466002)(7736002)(36756003)(31696002)(64126003)(33646002)(25786008)(86362001)(305945005)(77096006)(83506001)(5001770100001)(97736004)(4001350100001)(47776003)(229853002)(65806001)(92566002)(189998001)(5660300001)(6666003)(101416001)(105586002)(106356001)(50986999)(2950100002)(31686004)(65956001)(76176999)(65826007)(54356999);DIR:OUT;SFP:1102;SCL:1;SRVR:BN6PR15MB1186;H:[192.168.1.176];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjZQUjE1TUIxMTg2OzIzOkhxNjdSSDYwNWZQN0FSRXU1c3NHTUpoUG9O?= =?utf-8?B?enJ4V0Q0eGljdVQ0Mko2YjEwbEV3YnhvVUJHdldYOGdHTk45d2pnVTRoRnFp?= =?utf-8?B?dlJsbS9MazJWQTJYY0RlTnozSFNQUWZuV1Q0Q2d3Sk5OeFVCNXZONUMvUnF0?= =?utf-8?B?ekhKcEtySFFVUTdaTFBiWVpQOHBmMjBBQm5BL3FqSU1kY0VmZ29QRXhHSk5x?= =?utf-8?B?a3I2SXlGU2lGUTFQbCt1Tzdha1o3VlovaWNKYUNjTU1sOUtOSWt5VDFENU5i?= =?utf-8?B?TUdmc3g3SVRmeHdWcXgybDFkK3NHQklmd0NNRkcxMFUxd3BOUHZoZ0FPMFhD?= =?utf-8?B?eUl4ZWdoL21URlFZYkl1MFZLRmErQzIyTXVrYWU0aFc2RlphU2FPS2syNExB?= =?utf-8?B?anpxNVdPaitVZUt3MzVtUmlGRjFyT2V6WVFiTmhjblVyem45ejhPUDBXR28r?= =?utf-8?B?aUNGRzF2UmlIeFJoVzN4NnR5RzR6ZmRjWHZicjhJKzJEazRTZmFiRVlVUU0z?= =?utf-8?B?ZkNMY0VuRE52ck16UGs3QjJ1UWNCTWlBdDd5NW9nV3ZQb05tamRlM2UwYWx6?= =?utf-8?B?R1k1WnFPbDBCL1NRejNaVDc4aGwvcVQ2YjVLd0ZHSi9iaXVQS21GV0hwYWxP?= =?utf-8?B?RXBOdFRuTGVuMTcvRVpGa2wzaFVMYkpEOVQ0MU0weG82cVdpRGFFanpIY1Vp?= =?utf-8?B?ZlUxWnVJZU8vUkhQNnBmbHRGVnZDcnhEUFZYNE8zUzZRam9EZ1FZRitNZWxv?= =?utf-8?B?eDlPTUtIVHF2M3F4dW81NVVuWnIycW4xUDZsNjduSFNKcHozRXNxMFRRUVNh?= =?utf-8?B?M1dETG5YTkVTWmhKNWJDTlZNSUdaQ3RlNy9Vc05CTDNPTnhyZDkxWmdaNVZE?= =?utf-8?B?dDAzMFdmQ0VrUm5YUml2OFRtVUV4VXZ1TlM5RWZ2RHMySGowRldEWDhhMXor?= =?utf-8?B?WVFIU2pqS3grZDl5ZjMxZHdRMUd6dlpiRVJzOXRaL2NmNk9lS2lPa1FFRDBR?= =?utf-8?B?Q0lNSGZuTnFwajRycjU3dTRhTjlVNVdGOGtGc0tSOFFuV3Rzdm5idWFua0VE?= =?utf-8?B?VlhQSXdvKzdSNW1IM2V2d2h1NDZVVXhoN1dRdW9rNktwbXIrNXROMkx2M25k?= =?utf-8?B?WFNnZEUxWHRnR0V5a2VCa1E2eVpiNkpVZjZoNWxXQnhjMVphRGhNZUQ1TzJu?= =?utf-8?B?dlVaSXlxTnUwalB1alppRmNMc0tVUVpnc0NwZ01pVmNES2JNQU9RS1FLblZn?= =?utf-8?B?d09ydVpuMzJmenp1dkFRNzQyMk5DYmtaMzJ4cEwzMmlFbGEzSlkrT3IrYzk2?= =?utf-8?B?QjAySTRlMmEwTnRrRWxzajNhNXpJbllvOTRoTTNzdGJVZTF6SC9jMEpleHY0?= =?utf-8?B?RENOWEhNaksyV3FKSW1VdDM1WmpBNWNIR1I1cHJXZ3pMSStCR0pHYzFrVm9m?= =?utf-8?B?ei9xcWhkOVlBdkhyYzNOSkxoaTU3a3RDTVN6ZjVWKzJkZ2FTOGIxaWtTWmE0?= =?utf-8?B?SW05enBsUGNkS0Z5aWN2MlR5QjZvMnJTQ1dLR2lVdG9Dek82VGJlUFluL2RK?= =?utf-8?B?dDAxM1E5SnFvYTNjdytSbmNTMDdwZjlDZnJpZ2NZTE9pRThvdEwvYzVaa0ZQ?= =?utf-8?B?WVUwVnF5MmdMUnV3YWxHdEtYR2w1RG5LcjM0OWhUTTlML0RlWTdJL0tValVU?= =?utf-8?B?Nnl6WHEwRDBtYVZyUGtUZ2pXYTNMVVg5SzhUaVZrNmlkRjM5NnVYSVJDNXg4?= =?utf-8?B?UENnMWZVL2RkVFJmTDBJNTNsVlRZM1RBc2VCb1NXN2xJZXFMWVFpUnJtSk84?= =?utf-8?B?YzlIRzhpKy9tREpDdSsrVElDeVdXTys0M1FXWTVuVVJpUk9JbVNFajR1NmVQ?= =?utf-8?B?K2xWUjRaZGlSRE9xdnVVZFBid21SeklhYmZybnYyY1ZhR2g0bFlwZ2hGL0NI?= =?utf-8?B?VWU2Vmx4OXhRPT0=?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;6:oe0zmClNs8D2XbF2J8tkCGx1N/oR46v+6VYXk4XpChp1TLisHNZdZag2qLhwT+za5l6Merp9ELjo6vTg+ZdOt/sL7YIdYAZVmr7biMNWw49jjd+igjrFn6BUmC5M1eL1Rvpd3GMZkWgDAW2KEl5DDl9HW5bmKeijJ9B47KFlM/xcdRsQD8jDjI3xuxkc4H9qTmkHF89A9FZGmlLsEWP/zKNd0Paykbs3xEAp6Zvp+XT74k+DTNN4bKumjf/E6ILQrsmH6Wp9ApBVb21MM6in4vmlsb9j1NAz7CH+g6k1eV1nT3i0yDNBiP3Hb+kjuMtVlykWEarzM3SQbTCGpnLe3dpv6tClpy95WOIRqrFW4xLz8+NwiOFw/TvX6e/00sE3BVPusbQY9tHZndfkeKmu1kjDQPqGQHhXtrxqKtDwi8c=;5:sN+py8tEScKp1Qs8Ow/Tdnjn++1ZyVKGmEYB29LbSVNs7YBaQxrCV43kIxTJDigTZRTUfXXllOX+XR3MEX1SkWqkUojQkDChIHX63elQcBK3cFJX1D7DUfq2jKJCLPRxCCi0sVMsqUOAoVXhmnfY5A==;24:/sSWxdILn2DZax30ObTIrXd28rsEknRFs1LSMGfzGbsBqUz+jSwUkZq0eCL8BoxwVZIulvgFyO3RvK7NRMwzyiUzu9ROlhbYJkn6mPRkjOM= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR15MB1186;7:v2uNn/jvniydAfMs6yvzli8V1onIxJ66lyyvdXJg38E7ltIHd5N0N0Bz47ZZ9qYlklqZFbaHd1A8BF5KMdi52JgOP4fy3FpZm2Bfsm1BNyTjojvnlsBxh56BHSOiZdsZ2H810J3o2hNysUjhpk5WdlSI/R4vcGNNVUUWdnJssIR+yQvej6yNw74S7IgqaNf9nnDb35N/ElM2Vx6UYslCmEDmrLIE05FyX2Y89NOqmV6pzE9EeCy6KPWUgDSgmcHJD9rwUE3QyicR6hE0djwINOEYNAlVFYA6GnS4onhwnkGRYGxQotpcmnkNI7A0ikp0B7Vu7bX+W30PcYlCHMINEayFSNb3yQNxs2mQeVS/Xpmvsesvo7yw3t9WyyIE7ZuEUc7PccQhjwc9ADFHM6KH/pvyonFeE1wHiIuoo2kgBZCrI5ATcl1sV35DZfh0F4+ZCUik8uKWIWUlSP7NaXMxMw==;20:yIMg9s4Z+M+mN5MAi8/VObI3l+nOTzjoRQmxDUi+7RUBjVweC9e992Af8+Go557Ote4umVg+cGM11e6Z7wE/uRJIgQHwh4WbfOqXi8BabHcD7cxqrKQiKK/6U89SxxOAdmnnqiMY2X4XTUhqbc3vKSRTYqj3SR2LqHSQ+4IK0m0= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Dec 2016 00:16:27.2334 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1186 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-12-22_20:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 823 Lines: 21 On 12/22/2016 02:10 PM, Linus Torvalds wrote: > Ok, so the numa issue was a red herring. With that fixed: > > On Thu, Dec 22, 2016 at 1:06 PM, Dave Chinner wrote: >> >> Better, but still bad. average files/s is not up to 200k files/s, >> so still a good 10-15% off where it should be. xfs_repair is back >> down to 10-15% off where it should be, too. bulkstat still fires off >> a bad page reference count warning, iscsi still panics immediately. > > Do you have CONFIG_BLK_WBT enabled, perhaps? > > It's new to this merge window, and I'm not convinced it's been tuned. > Particularly for your kinds of fairly extreme IO loads. We have it running on thousands of machines at Facebook, and I've used it in all of my testing, haven't seen any issues. It's been run pretty damn hard. -- Jens Axboe