Received: by 10.223.176.5 with SMTP id f5csp2575347wra; Thu, 1 Feb 2018 02:38:04 -0800 (PST) X-Google-Smtp-Source: AH8x227+zfEYuaalroKyJKsLK067Cj9LRyUj8lmn97WVIteQRpiCgVBi9EruoORn22n8dBXJOf93 X-Received: by 10.99.97.208 with SMTP id v199mr27927394pgb.387.1517481483961; Thu, 01 Feb 2018 02:38:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517481483; cv=none; d=google.com; s=arc-20160816; b=Jrl/Li6o5upH5IEJaSMgiIBCjr5gKqf9Ga8OpZJNwlDL1kJozLroBZmqQKNl6v2/g6 CodQ0rTr41DNP6k0KEpj9XsRauXpU1eVk8v0upW2jYjuWfINtOsKNs679eryhqLwkSYL robWHI7UkqY0xEGm+Ovz38iFYCI5vv2rVA91V8Oa6A2OpvXKl8lEkWQh5moWjSOPshf8 LlZYDFa5obbGnB5zJSn5jTwKjBpSASmKdwCf/DLnq9WBnEFtRN7aAp6zKk7pVMegRb6/ 6kvhjv0itOdP7COkJTWrYchN1/08rTNtlfDCoUdOiVSrAK5byKgdX+1RrzpPmxzlymM4 gHzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-transfer-encoding:content-disposition:mime-version :references:message-id:subject:cc:to:from:date :arc-authentication-results; bh=AEg9FyxNEUsRhFnPbtl+neHFAt99QUsjKxpwodssCtA=; b=iri8+O8rJkxlA7vnD3YAL8m7DC1yrbweYcTIgwpphScVjZ+gFwhKQ++koKwmgVNISv tJ6EmAosMPo6ydWSsmP/IrLtln2wBZLp7I5FDibCYmpvaYavTFWUikKQwI6fUstFg4Rt YnYyUU8PzcbtxpaQfk9KFTMIznlzovWym+lr0TVHFP8MHFyhfxV6M1vhv+TV92fkMVgh 1G2xrrTVBhRorWrag3sMJbwBOJ7lBvxQxgccfRuaRyRBVYSYGwxIP93/5MxlnFyEIKC4 A94sRYXPsUWWRzFaEZEgqPAdKWRA2bsfZTjboW/momCa1stH4XwWNzZzwpHHRme6K4dE 0vkw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s124si289609pgc.386.2018.02.01.02.37.49; Thu, 01 Feb 2018 02:38:03 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752030AbeBAKhX (ORCPT + 99 others); Thu, 1 Feb 2018 05:37:23 -0500 Received: from mx1.redhat.com ([209.132.183.28]:50754 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751520AbeBAKhT (ORCPT ); Thu, 1 Feb 2018 05:37:19 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 86DB719CF98; Thu, 1 Feb 2018 10:37:19 +0000 (UTC) Received: from ming.t460p (ovpn-12-90.pek2.redhat.com [10.72.12.90]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3CAE4620DF; Thu, 1 Feb 2018 10:37:00 +0000 (UTC) Date: Thu, 1 Feb 2018 18:36:57 +0800 From: Ming Lei To: Don Brace Cc: Laurence Oberman , Thomas Gleixner , Christoph Hellwig , Jens Axboe , "linux-block@vger.kernel.org" , "linux-kernel@vger.kernel.org" , Mike Snitzer Subject: Re: [PATCH 0/2] genirq/affinity: try to make sure online CPU is assgined to irq vector Message-ID: <20180201103651.GA29263@ming.t460p> References: <20180115160345.2611-1-ming.lei@redhat.com> <20180115174036.GA20191@infradead.org> <20180116013043.GA3213@ming.t460p> <1516109317.9574.1.camel@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.9.1 (2017-09-22) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Thu, 01 Feb 2018 10:37:19 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jan 16, 2018 at 03:22:18PM +0000, Don Brace wrote: > > -----Original Message----- > > From: Laurence Oberman [mailto:loberman@redhat.com] > > Sent: Tuesday, January 16, 2018 7:29 AM > > To: Thomas Gleixner ; Ming Lei > > Cc: Christoph Hellwig ; Jens Axboe ; > > linux-block@vger.kernel.org; linux-kernel@vger.kernel.org; Mike Snitzer > > ; Don Brace > > Subject: Re: [PATCH 0/2] genirq/affinity: try to make sure online CPU is assgined > > to irq vector > > > > > > It is because of irq_create_affinity_masks(). > > > > > > That still does not answer the question. If the interrupt for a queue > > > is > > > assigned to an offline CPU, then the queue should not be used and > > > never > > > raise an interrupt. That's how managed interrupts have been designed. > > > > > > Thanks, > > > > > > tglx > > > > > > > > > > > > > > > > I captured a full boot log for this issue for Microsemi, I will send it > > to Don Brace. > > I enabled all the HPSA debug and here is snippet > > > > > > .. > > .. > > .. > > 246.751135] INFO: task systemd-udevd:413 blocked for more than 120 > > seconds. > > [??246.788008]???????Tainted: G I 4.15.0-rc4.noming+ #1 > > [??246.822380] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" > > disables this message. > > [??246.865594] systemd-udevd D 0 413 411 0x80000004 > > [??246.895519] Call Trace: > > [??246.909713]??? __schedule+0x340/0xc20 > > [??246.930236]??schedule+0x32/0x80 > > [??246.947905]??schedule_timeout+0x23d/0x450 > > [ 246.970047]??? find_held_lock+0x2d/0x90 > > [??246.991774]??? wait_for_completion_io+0x108/0x170 > > [??247.018172]??io_schedule_timeout+0x19/0x40 > > [??247.041208]??wait_for_completion_io+0x110/0x170 > > [??247.067326]??? wake_up_q+0x70/0x70 > > [??247.086801]??hpsa_scsi_do_simple_cmd+0xc6/0x100 [hpsa] > > [??247.114315]??hpsa_scsi_do_simple_cmd_with_retry+0xb7/0x1c0 [hpsa] > > [??247.146629]??hpsa_scsi_do_inquiry+0x73/0xd0 [hpsa] > > [??247.174118]??hpsa_init_one+0x12cb/0x1a59 [hpsa] > > This trace comes from internally generated discovery commands. No SCSI devices have > been presented to the SML yet. > > At this point we should be running on only one CPU. These commands are meant to use > reply queue 0 which are tied to CPU 0. It's interesting that the patch helps. > > However, I was wondering if you could inspect the iLo IML logs and send the > AHS logs for inspection. Hello Don, Now the patch has been merged to linus tree as: 84676c1f21e8ff54b ("genirq/affinity: assign vectors to all possible CPUs") and it breaks Laurence's machine completely, :-( I just take a look at HPSA's code, and found that reply queue is chosen in the following way in most of code path: if (likely(reply_queue == DEFAULT_REPLY_QUEUE)) cp->ReplyQueue = smp_processor_id() % h->nreply_queues; h->nreply_queues is the msix vector number which is returned from pci_alloc_irq_vectors(), and now some of vectors may be mapped to all offline CPUs, for example, one processor isn't plugged to socket. If I understand correctly, 'cp->ReplyQueue' is aligned to one irq vector, and the command is expected by handled via that irq vector, is it right? If yes, now I guess this way can't work any more if number of online CPUs is >= h->nreply_queues, and you may need to check the cpu affinity of one vector before choosing the reply queue, and block/blk-mq-pci.c may be helpful for you. Thanks, Ming