Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 19 Feb 2018 13:12:10 -0800
From:   Jayachandran C <jnair@caviumnetworks.com>
To:     Marc Zyngier <marc.zyngier@arm.com>
Cc:     Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>,
        linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-arm-kernel@lists.infradead.org, tglx@linutronix.de,
        jason@lakedaemon.net, catalin.marinas@arm.com, will.deacon@arm.com,
        corbet@lwn.net, Robert.Richter@cavium.com, Jan.Glauber@cavium.com,
        Vadim.Lomovtsev@cavium.com, gklkml16@gmail.com
Subject: Re: [PATCH v2] irqchip/gic-v3-its: Add workaround for ThunderX2
 erratum #174
Message-ID: <20180219211151.GA158953@jc-sabre>
References: <20180118052820.30286-1-ganapatrao.kulkarni@cavium.com>
 <20180121070038.GA4450@jc-sabre>
 <86po635trt.wl-marc.zyngier@arm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <86po635trt.wl-marc.zyngier@arm.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Received-SPF: None (protection.outlook.com: cavium.com does not designate
 permitted sender hosts)
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2018 21:12:20.4033 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 1ba1c3a7-a58d-41e9-8d20-08d577dd77ab
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR07MB659
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Sun, Jan 21, 2018 at 11:35:34AM +0000, Marc Zyngier wrote:
> On Sun, 21 Jan 2018 07:00:48 +0000,
> Jayachandran C wrote:
> > 
> > On Thu, Jan 18, 2018 at 10:58:20AM +0530, Ganapatrao Kulkarni wrote:
> > > This erratum is observed on the ThunderX2 GICv3 ITS. When a
> > > MOVI command is used to change affinity of a LPI to a collection/cpu
> > > on another node, the LPI is not delivered to the cpu.
> > > An additional INV command is required after the MOVI to deliver
> > > the LPI to the new destination.
> > > 
> > > If we add INV after MOVI, there is a chance that we lose LPIs which
> > > are raised when the affinity is changed. So for now, adding workaround fix
> > > to disable inter node affinity change.
> > > 
> > > Signed-off-by: Ganapatrao Kulkarni <ganapatrao.kulkarni@cavium.com>
> > > ---
> > > 
> > > v2: Added workaround to avoid inter node affinity change.
> > > 
> > > v1: Initial patch
> > > 
> > >  Documentation/arm64/silicon-errata.txt |  1 +
> > >  arch/arm64/Kconfig                     | 10 ++++++++++
> > >  drivers/irqchip/irq-gic-v3-its.c       | 21 ++++++++++++++++++++-
> > >  3 files changed, 31 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/Documentation/arm64/silicon-errata.txt b/Documentation/arm64/silicon-errata.txt
> > > index fc1c884..fb27cb5 100644
> > > --- a/Documentation/arm64/silicon-errata.txt
> > > +++ b/Documentation/arm64/silicon-errata.txt
> > > @@ -63,6 +63,7 @@ stable kernels.
> > >  | Cavium         | ThunderX Core   | #27456          | CAVIUM_ERRATUM_27456        |
> > >  | Cavium         | ThunderX Core   | #30115          | CAVIUM_ERRATUM_30115        |
> > >  | Cavium         | ThunderX SMMUv2 | #27704          | N/A                         |
> > > +| Cavium         | ThunderX2 ITS   | #174            | CAVIUM_ERRATUM_174          |
> > >  | Cavium         | ThunderX2 SMMUv3| #74             | N/A                         |
> > >  | Cavium         | ThunderX2 SMMUv3| #126            | N/A                         |
> > >  |                |                 |                 |                             |
> > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> > > index c9a7e9e..0dbf3bd 100644
> > > --- a/arch/arm64/Kconfig
> > > +++ b/arch/arm64/Kconfig
> > > @@ -461,6 +461,16 @@ config ARM64_ERRATUM_843419
> > >  
> > >  	  If unsure, say Y.
> > >  
> > > +config CAVIUM_ERRATUM_174
> > > +	bool "Cavium ThunderX2 erratum 174"
> > > +	default y
> > > +	help
> > > +	  Cavium ThunderX2 dual socket systems may loose interrupts
> > > +	  on affinity change to a cpu on other node.
> > > +	  This workaround fix avoids inter node affinity change.
> > 
> > This has to be fixed up to match the commit message (and for spelling).
> > I have seen some questions offlist about how important this fix is,
> > and how it can affect users - so that would be useful to have in the
> > description as well.
> > 
> > To clarify, this errata comes into play only when the irq affinity is
> > forced from the node given by the device (and ITS) affinity to another
> > node.  This should not happen in normal, useful configurations.
> 
> Define normal. That's all under control of userspace, and the kernel
> doesn't really have a say. irqbalance will happily move interrupts
> around. Disable all CPUs from node at runtime (again, from userspace),
> and you'll get the exact same thing. I can't see what's so "abnormal"
> about any of that.
> 
> > Also, we will hold further posting of this errata until we do another
> > round of investigation with the hardware team for a better solution.
> > If we can handle the pending interrupts for the small window of MOVI/INV
> > in first workaround, we will not need this restriction at all.
> 
> What do you mean by "If we can handle the pending interrupts for the
> small window of MOVI/INV"? Taking the interrupt on the source CPU?
> Sure, that would be fine. But that's assuming that the souce CPU is in
> a position to actually handle this, and is not simply going down.
> 
> If there is only a slight possibility that you may loose an interrupt
> in the MOVI/INV window (which is not that small, since that's a 4
> command sequence), your only other solution is to inject a spurious
> interrupt to replace the one you may have lost in that window.
> 
> In the meantime, and until I see a patch fixing this (or a decent
> explanation of why this isn't a problem), I'll consider it broken.

After reviewing the issue with our hardware team, we decided to
tweak the redistributor cache configuration from firmware rather
than go with this errata workaournd in Linux (and other OSes).

So, with the new firmware MOVI will work across nodes as expected,
and this patch is no longer neeeded.

Thanks,
JC.