Message-ID: <c61f63b6-662e-9c85-9135-50710fec79fc@nvidia.com>
Date:   Sat, 18 Mar 2023 10:29:01 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.2.2
Subject: Re: [PATCH v4] irqchip/gicv3: Workaround for NVIDIA erratum
 T241-FABRIC-4
Content-Language: en-US
To:     Marc Zyngier <maz@kernel.org>
Cc:     Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        Jonathan Corbet <corbet@lwn.net>,
        Mark Rutland <mark.rutland@arm.com>,
        Lorenzo Pieralisi <lpieralisi@kernel.org>,
        Sudeep Holla <sudeep.holla@arm.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org,
        linux-kernel@vger.kernel.org, Vikram Sethi <vsethi@nvidia.com>,
        Thierry Reding <treding@nvidia.com>
References: <20230318045812.2043117-1-sdonthineni@nvidia.com>
 <87wn3egzr8.wl-maz@kernel.org>
From:   Shanker Donthineni <sdonthineni@nvidia.com>
In-Reply-To: <87wn3egzr8.wl-maz@kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dmYzLys3ODlIM3R6Ky8xVnN2V1dyRlFTSFcwTHF2QkpqcW52UENMb000ak9S?=
 =?utf-8?B?aVN4UEhKeHk4N2xqVTc4dE01NFR2K3RhVG1Yb2xObmJsWkg5L0czOEFGUzVH?=
 =?utf-8?B?c0N4VS9XSHBJeEtIakdHdFdpZUEzK2tncDRYTmxJRTQ1Vk8wNVZDTjJERmhP?=
 =?utf-8?B?VE9Fekw3cnBiQllXOGtPMTI4dUtqQXVuNVFXSHQxRXYyc2NLWFpFOW1UYUhW?=
 =?utf-8?B?ZHdnN2wvdk1kUUxHWEV0LzkvbHRaMGFVUXhlUkJMYVpNc1dnYkwvV0VUVVpV?=
 =?utf-8?B?L0JmMi9sZUowdUhFQTUxVGVweWg1NGRwQnQwOHZ5dGQwbHpTN01vbDlWdkVW?=
 =?utf-8?B?dUxQcHVCV0k2emlzVThEc2ttb2FCaDQvSWxsUFUyVmlZVVozQjlUZUhkNFho?=
 =?utf-8?B?U1RFRWxlbjBKem1DaDRpUHY1YWY1cFRPVlJKVWhzSlF1V3d4bS83NGdaNEpU?=
 =?utf-8?B?Z3RybXpwbjNKeVpTbUNTNXBKSjNIZkFRbWUvL3FBa0lNZHozWUR1enNTRkhR?=
 =?utf-8?B?QWoybmVOSlNkU21VcHNEOXZKbGpENUJnREIvN1J0eWJpVjdxUnc2VktFQjl3?=
 =?utf-8?B?R2FUc1IraEo1MU5mZmMxLzNrd2JtMGJ3V2xwWGNMWW1adTV5M0FRZnRhdSt3?=
 =?utf-8?B?bE0rMnZXMGhlWTliOUtVTm5OS0R4b1Vjc0szZXBNeWc0R1VqZGg4K3RwWlhy?=
 =?utf-8?B?RHpGU0UveUVOVElWTlZnaWVJWDBQQ0FVUzZSaFA2elNFdnNEMlZoOFNLYVJK?=
 =?utf-8?B?U0lvejNXM04yRWN4dkNnbnRYR3ZuU3ZCQ0RteTFxdXE5QWQ4dWZQbjByQWVS?=
 =?utf-8?B?SEx2VlpEbHlBUHJyMk85aE5TeDZKbzZzRnVTd1JBRmhWdXI5Ny9OcGlZQkhs?=
 =?utf-8?B?RUVjMTRKWHlvZjFOWEZKdU10Yzd5VXp1ZVdnaDc4OUdMN1R1T1hRWXAxajJu?=
 =?utf-8?B?MXlqdFE1S0lqdk1ROTk1YlZhaVlkbnZHL1Zyc2pqcnp0ejNnZmpxZkpzVERo?=
 =?utf-8?B?Z2dmSGw2d0EzTy9wek9SNnlEVVgvbjVZeE1hV01HbDh3UXkxZGxnNXRRa2VL?=
 =?utf-8?B?eXE2clVad0lRa2Qxc0ppUEJ2V1o3S1VFTHdJUEZqaEhFWDFFQm1lREsxWEF6?=
 =?utf-8?B?QWpybHBWa0xXWVhMTDNrMk5idVJBZ3VBd0NoM1FOelVXRTlWQU9UNjlBczBz?=
 =?utf-8?B?SHd3WXBNdDJUc2NHMkdic0hoVytWd3dZN1R4cGRwOUJWNWtoN3dnNFVtZmRo?=
 =?utf-8?B?WU9XOFErd1d6MXFXQXlyRkZJK0xISVNnNlZOeTEvNzBhMjBxUExJMFloTkE1?=
 =?utf-8?B?bVlWR0hZQWpoaWc3YkhnT3JhcE9PaWQrRVpLWWY4cTBiTzJacW5YK09tMzFt?=
 =?utf-8?B?UW80cnQ2SmQxQ1I2TUNkL3M2OEZCa2RVN01OSG9LU0NoZmlERHN0MjA3K2Js?=
 =?utf-8?B?RXRrOTR2VG1RMVRGQ3VEZ1Fpdkp1U0REVkk2WGJqLytjT21VOHMwT3ZkTjJG?=
 =?utf-8?B?THhPUDc1ZW9GRFlzdUhsL2hYdnk5cEJScWxUZU1TTjVJQWRkV2ZTc2FSLzV4?=
 =?utf-8?B?R3hIQmg5a0Z5ZWxsK0J4V1dLZ01BVkRyWGtqaXpNdGZabmlUWjRpRlFCcGlC?=
 =?utf-8?B?T1d2amdSUi9YUmdWTGJQQU5TR3ZVZlNsSUdTbFY2dUpRUFpEVWxrS21YdGgv?=
 =?utf-8?B?OVc1a1k5c0JHbDVodjlENzFHZ0lvMWxaanVXWENOS2pITXVVK281dkZEc2JU?=
 =?utf-8?B?Wkx1MmlYWFk4WTJvS0FQMnVsTFJpeERqTTA1ZjAwWmk5S25vdmJBbExtblVH?=
 =?utf-8?B?cThBOFY5UDhqRVYrcjlUQm9uWEIvV0FVSUtrV1FpczFqVm4zckpxWkFhajBZ?=
 =?utf-8?B?UkQxdlZwd3NnV25zQUJPbEhXTWR3UTRlQ0FQdU5JY3paYXZTWHQxcVhid1BT?=
 =?utf-8?B?S2c2YXVBRC9FcWJlVE9CWU9ad0hscTVmNWhZU2pFMGloeWM5L21TR0RmK2hG?=
 =?utf-8?B?V0VES21yODU2ZUhCY1RoWWpRNDJPaGRTaEs5bytWZzFFQWdvdGFmOVpJV21F?=
 =?utf-8?B?MlRhQXZQSkVlaE5PTTl2RGhXRU00OHBrS2crUVpBY2NvbzRsWDdnTi8zbDhU?=
 =?utf-8?B?ZmNodVk4Y3BoeHdSbGJkRHJSNGdSSFoveGpuK0c3NUIrcUVGZzRDbVF6WHBv?=
 =?utf-8?B?WGc9PQ==?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0298b11e-f246-4aac-3fc6-08db27c582fc
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB5762.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Mar 2023 15:29:06.5179
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DdZevC+6yNIdKaA3JQY3ZoEcC58oK8cvio3B0nFdpKOrmeu52U4CLdtH1v6XHWcLX50rLtSXtatarkR12/gzRg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4257
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Marc,

On 3/18/23 04:44, Marc Zyngier wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Sat, 18 Mar 2023 04:58:12 +0000,
> Shanker Donthineni <sdonthineni@nvidia.com> wrote:
>>
>> The T241 platform suffers from the T241-FABRIC-4 erratum which causes
>> unexpected behavior in the GIC when multiple transactions are received
>> simultaneously from different sources. This hardware issue impacts
>> NVIDIA server platforms that use more than two T241 chips
>> interconnected. Each chip has support for 320 {E}SPIs.
>>
>> This issue occurs when multiple packets from different GICs are
>> incorrectly interleaved at the target chip. The erratum text below
>> specifies exactly what can cause multiple transfer packets susceptible
>> to interleaving and GIC state corruption. GIC state corruption can
>> lead to a range of problems, including kernel panics, and unexpected
>> behavior.
>>
>>  From the erratum text:
>>    "In some cases, inter-socket AXI4 Stream packets with multiple
>>    transfers, may be interleaved by the fabric when presented to ARM
>>    Generic Interrupt Controller. GIC expects all transfers of a packet
>>    to be delivered without any interleaving.
>>
>>    The following GICv3 commands may result in multiple transfer packets
>>    over inter-socket AXI4 Stream interface:
>>     - Register reads from GICD_I* and GICD_N*
>>     - Register writes to 64-bit GICD registers other than GICD_IROUTERn*
>>     - ITS command MOVALL
>>
>>    Multiple commands in GICv4+ utilize multiple transfer packets,
>>    including VMOVP, VMOVI, VMAPP, and 64-bit register accesses."
>>
>>    This issue impacts system configurations with more than 2 sockets,
>>    that require multi-transfer packets to be sent over inter-socket
>>    AXI4 Stream interface between GIC instances on different sockets.
>>    GICv4 cannot be supported. GICv3 SW model can only be supported
>>    with the workaround. Single and Dual socket configurations are not
>>    impacted by this issue and support GICv3 and GICv4."
>>
>> Link: https://developer.nvidia.com/docs/t241-fabric-4/nvidia-t241-fabric-4-errata.pdf
>>
>> Writing to the chip alias region of the GICD_In{E} registers except
>> GICD_ICENABLERn has an equivalent effect as writing to the global
>> distributor. The SPI interrupt deactivate path is not impacted by
>> the erratum.
>>
>> To fix this problem, implement a workaround that ensures read accesses
>> to the GICD_In{E} registers are directed to the chip that owns the
>> SPI, and disables GICv4.x features for KVM. To simplify code changes,
>> the gic_configure_irq() function uses the same alias region for both
>> read and write operations to GICD_ICFGR.
>>
>> Co-developed-by: Vikram Sethi <vsethi@nvidia.com>
>> Signed-off-by: Vikram Sethi <vsethi@nvidia.com>
>> Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
>> ---
>> Changes since v2:
>>   - Fix the build issue for the 32bit arch
>> Changes since v2:
>>   - Add accessors for the SOC-ID version & revision
>>   - Include "linux/bitfield.h" and "linux/bits.h" in irq-gic-v3.c
>> Changes since v1:
>>   - Use SMCCC SOC-ID API for detecting the T241 chip
>>   - Implement Marc's suggestions
>>   - Edit commit text
> 
> You seem to have ignored most of my comments on v2[1] apart from the
> SOC_ID stuff. I guess I'll wait for v5...
> 
>          M.
> 
> [1] https://lore.kernel.org/all/871qlqif9v.wl-maz@kernel.org/
> 

Sorry, I did not intentionally ignore your input, but unfortunately, lost
this specific email in my outlook. Your feedback is valuable, and we will
ensure that all of your review comments are addressed in the v5.

-Shanker