Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Message-ID: <766f1321-ea01-40c1-832a-3989677e7d4b@intel.com>
Date:   Mon, 30 Oct 2023 14:52:30 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v4 2/5] r8169: Coalesce RTL8411b PHY power-down recovery
 calls to reduce spinlock stalls
Content-Language: en-US
To:     Mirsad Goran Todorovac <mirsad.todorovac@alu.unizg.hr>,
        Jason Gunthorpe <jgg@ziepe.ca>, Joerg Roedel <jroedel@suse.de>,
        Lu Baolu <baolu.lu@linux.intel.com>, <iommu@lists.linux.dev>,
        <linux-kernel@vger.kernel.org>, <netdev@vger.kernel.org>
CC:     Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
        "Robin Murphy" <robin.murphy@arm.com>,
        Heiner Kallweit <hkallweit1@gmail.com>, <nic_swsd@realtek.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>, Marco Elver <elver@google.com>
References: <20231029110442.347448-1-mirsad.todorovac@alu.unizg.hr>
 <20231029110442.347448-2-mirsad.todorovac@alu.unizg.hr>
From:   Jacob Keller <jacob.e.keller@intel.com>
In-Reply-To: <20231029110442.347448-2-mirsad.todorovac@alu.unizg.hr>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SzBlUE5oajFHVmlGSGZnSDRWcUNKR1JCN2lZWCsvQ0ZBSjBzcEJrSXo2endv?=
 =?utf-8?B?T1BaRkZrT0FyaGVTdkxaZWlFWVdiYUNlNDFyY1pVM3VBMGZhTTdaS2RZcmlJ?=
 =?utf-8?B?OCtkVkQwYW5NNjdWS0xyOWVicUtFcnRRTjJ0QXdCdFBBbFloT1dXNk1QMW42?=
 =?utf-8?B?dnIvNjhPSGR6Q2pMdzJNRlNWOFNhQzl4TjFkMys3TG5CY1lwYUZSclRiNTRW?=
 =?utf-8?B?eHhBK0htbGUyOGhHWmNkZWFSWkl6RHVjL1lYS3lFU3owUy9mWmppQUNwejN6?=
 =?utf-8?B?TG1nN0tCN2pjdTdSN0xNdVpRMEtkNVdhS2dFd1Jjc1VodERURTRoMjgydnFu?=
 =?utf-8?B?UnErcXc3eTVoVDNZOEVmb2NUdTBhK2pkZW9UOWc0VmRjVkMySnpMSE8ySHhl?=
 =?utf-8?B?cm1IaTZxd0JPd3NWQ1J6clRSQXZ4anBsT0RiQXBsb0dRTE1HNVpMUUpDaWNG?=
 =?utf-8?B?V05JZlI5b1l0dDI3TFJ0VjAra25RaW4xbzFWOWttZlJaOUNpRm5kU2hFUnVC?=
 =?utf-8?B?K2JuUmhBMWNDWGFUdmtvc0dpNGZITFdvdHVrc0I2N0VHRlp5a0d5R3NGcXY3?=
 =?utf-8?B?YklIMjgzSm0za1JPNWt0T2hrcFE4anN6VkxNUmJYK2c4czNPMEZZcDRVbkdY?=
 =?utf-8?B?emIrUTZPdlY1RFVGZHhjUjk5bkM1UVlvZm53b2F3eXlkTWl4Ky9WYVVmUFZm?=
 =?utf-8?B?NkVSZktPVkRMWUdvaEpPVEt0eE5nNTg3TWRQeXBXZlZ6WTVBdHA4ZWVSdTNv?=
 =?utf-8?B?VmZMUDR3NXB2KzFzOG5PVlZBcVBuMmlTc29PcGxXMW9RREsrODRyczZ6T2lB?=
 =?utf-8?B?eU9GcDNZTGZlZUtiQWZkMWQxM0djaUtFZGRnNGltQm9WVzZHNXpYbi9KbWZw?=
 =?utf-8?B?azVoOGN3Qy83YjZSRkM4eDlrUUZMeFVERDZNWEI3ejBTbEVEWTBMV28yeXRo?=
 =?utf-8?B?T2VET01nQzBsNEphOHltZVA0WjlyTnkydUcvV2JKT2I3WnRtOVAxQ3VmdUp0?=
 =?utf-8?B?MmpCZWVJazVTR0ZvWk1JRk5oV2xHcnFJUTFVWWdHVURROU9rM0p3Zm9HLzMv?=
 =?utf-8?B?cy9acFpEM1paTHIyZSs3SWZFMU5oSStsRzZPT3VqTjM0RjZVZlZia0hhdHZN?=
 =?utf-8?B?Yk9zZ1dDOUtHZkVUcWlGRnhucU9BdkRvV1ovaXVGckI2TkFkK09QL1l0Qi9n?=
 =?utf-8?B?Nld4ajZXYTlONDg3dkJ0ODJDd2pLRlkzcDM2bzVxVnZydDdMVjVNamdaMGVs?=
 =?utf-8?B?ZjljZnpxYzk5cDloU2tNSjRjeEw3RGZQVVpId0xrVFJad3htaWVmL2kzT0R1?=
 =?utf-8?B?ME1sR3NKbjEzQWc4eWF0emNTN1hKdFlEVEJGcDZHMUlLM0tBYXRDeURvajA1?=
 =?utf-8?B?ZWtZTDlsdnZ6cXpCcmYyY0hLRUV2UnZXc2Nja1VNRkJ4Z3lnSCtQZnVxRzFr?=
 =?utf-8?B?SHJvakY2cTM3VHBGaDgrbVZXTHZSQThNbTZtUDZLR3BCcXBueGgxM2RrMC9n?=
 =?utf-8?B?WWFLYUhNaFNOQUo0ek14dW13Z0E0RTZrTSthQTFnQkt3c0phM1o5WXlUUThq?=
 =?utf-8?B?THl0Z2F1Z25zL0cxQkRaamh2WXJwV2hYckQ4TUsrRHhWVFhWNE9mcXd1RXc0?=
 =?utf-8?B?bHE2TUJzWlh5UkE5aFM1bXpxODhLOE5wdExyOFJkMzBhQUNkaWUyck5mSkNr?=
 =?utf-8?B?aGtkQXpNOW9MbGUvRHRQc1NDVzZEcWlidHV6VE1zMnd1blhaRkIzclRnRzdN?=
 =?utf-8?B?ZDB1SWhqSlMrMWx0MHhPcVVWMGNWWGhBaU9MWDh3NnRTZ1FhVjRxRUJWeWpx?=
 =?utf-8?B?b0VueGw2bUpDdWJDdTNzKzJERjV6RSt6Ym9KY0JaeWtEVGpnWGlNOHk2VTZH?=
 =?utf-8?B?UFk2ek0ybDYyRnYvNkl2M05TRnZyUnBDNXNzYnpXSGRsaVVnTWZ2R2xjbFhB?=
 =?utf-8?B?WFUyamVjcGhidkRta1pPRW1SeUdWblJqdGJ0K0x3bHd5YTB3a0hjSUJkbDdM?=
 =?utf-8?B?cEtYaiszZTZDaGc3VXMvTmROMW4zVXhkcE1VUG5WOVNEK2dyMjRHYmR6NVVm?=
 =?utf-8?B?NjlHMjNqZHJZRnR1cWhoM0Jnck1PRGQ2RU05c3l1dUZQNUU0clkya3d6azRw?=
 =?utf-8?B?YjdtdGd0c2ZTdXRzakJLbDVHVWRvaDZ6K2hXRkliUEE3Y0FsMVpoakNYOTl5?=
 =?utf-8?B?bFE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 78b6b7f7-342a-4b06-f061-08dbd992854f
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5095.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Oct 2023 21:52:32.9466
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iMs58hD8f0YHIttAMdD8MQrF6Yuo9Dva+61ui/B/QOL8IVB3SA9034uJy1+PjkjEIYwb8yFuYgBLk79KFNW7JihvBjHXCcKEh36FWXuRmbQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5549
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 30 Oct 2023 14:53:01 -0700 (PDT)


On 10/29/2023 4:04 AM, Mirsad Goran Todorovac wrote:
> On RTL8411b the RX unit gets confused if the PHY is powered-down.
> This was reported in [0] and confirmed by Realtek. Realtek provided
> a sequence to fix the RX unit after PHY wakeup.
> 
> A series of about 130 r8168_mac_ocp_write() calls is performed to
> program the RTL registers for recovery.
> 
> With about 130 of these sequential calls to r8168_mac_ocp_write() this looks like
> a lock storm that will stall all of the cores and CPUs on the same memory controller
> for certain time I/O takes to finish.
> 
> In a sequential case of RTL register programming, a sequence of writes to the RTL
> registers can be coalesced under a same raw spinlock. This can dramatically decrease
> the number of bus stalls in a multicore or multi-CPU system:
> 
>     static void rtl_hw_start_8411_2(struct rtl8169_private *tp)
>     {
> 
> 	...
> 
> 	/* The following Realtek-provided magic fixes an issue with the RX unit
> 	 * getting confused after the PHY having been powered-down.
> 	 */
> 
> 	static const struct recover_8411b_info init_zero_seq[] = {
> 	    { 0xFC28, 0x0000 }, { 0xFC2A, 0x0000 }, { 0xFC2C, 0x0000 },
> 	    ...
> 	};
> 
> 	static const struct recover_8411b_info recover_seq[] = {
> 	    { 0xF800, 0xE008 }, { 0xF802, 0xE00A }, { 0xF804, 0xE00C },
> 	    ...
> 	};
> 
> 	static const struct recover_8411b_info final_seq[] = {
> 	    { 0xFC2A, 0x0743 }, { 0xFC2C, 0x0801 }, { 0xFC2E, 0x0BE9 },
> 	    ...
> 	};
> 
> 	r8168_mac_ocp_write_seq(tp, init_zero_seq);
> 	mdelay(3);
> 	r8168_mac_ocp_write(tp, 0xFC26, 0x0000);
> 	r8168_mac_ocp_write_seq(tp, recover_seq);
> 	r8168_mac_ocp_write(tp, 0xFC26, 0x8000);
> 	r8168_mac_ocp_write_seq(tp, final_seq);
>     }
> 
> The hex data is preserved intact through s/r8168_mac_ocp_write[(]tp,/{ / and s/[)];/ },/
> functions that only changed the function names and the ending of the line, so the actual
> hex data is unchanged.
> 
> Note that the reason for the introduction of the original commit
> was to enable recovery of the RX unit on the RTL8411b which was confused by the
> powered-down PHY. This sequence of r8168_mac_ocp_write() calls amplifies the problem
> into a series of about 500+ memory bus locks, most waiting for the main MMIO memory
> read-modify-write under a LOCK. The memory barrier in RTL_W32 should suffice for
> the programming sequence to reach RTL NIC registers.
> 
> [0] https://bugzilla.redhat.com/show_bug.cgi?id=1692075
> 
> Fixes: fe4e8db0392a6 ("r8169: fix issue with confused RX unit after PHY power-down on RTL8411b")
> Cc: Heiner Kallweit <hkallweit1@gmail.com>
> Cc: Marco Elver <elver@google.com>
> Cc: nic_swsd@realtek.com
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: Eric Dumazet <edumazet@google.com>
> Cc: Jakub Kicinski <kuba@kernel.org>
> Cc: Paolo Abeni <pabeni@redhat.com>
> Cc: netdev@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Link: https://lore.kernel.org/lkml/20231028005153.2180411-1-mirsad.todorovac@alu.unizg.hr/
> Link: https://lore.kernel.org/lkml/20231028110459.2644926-1-mirsad.todorovac@alu.unizg.hr/
> Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac@alu.unizg.hr>
> ---

Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>