Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Message-ID: <e7a6b0c1-9fc6-480c-a135-7e142514d0e7@intel.com>
Date:   Mon, 30 Oct 2023 14:50:39 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v4 1/5] r8169: Coalesce r8169_mac_ocp_write/modify calls
 to reduce spinlock stalls
Content-Language: en-US
To:     Mirsad Goran Todorovac <mirsad.todorovac@alu.unizg.hr>,
        Jason Gunthorpe <jgg@ziepe.ca>, Joerg Roedel <jroedel@suse.de>,
        Lu Baolu <baolu.lu@linux.intel.com>, <iommu@lists.linux.dev>,
        <linux-kernel@vger.kernel.org>, <netdev@vger.kernel.org>
CC:     Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
        "Robin Murphy" <robin.murphy@arm.com>,
        Heiner Kallweit <hkallweit1@gmail.com>, <nic_swsd@realtek.com>,
        "David S. Miller" <davem@davemloft.net>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>,
        Paolo Abeni <pabeni@redhat.com>, Marco Elver <elver@google.com>
References: <20231029110442.347448-1-mirsad.todorovac@alu.unizg.hr>
From:   Jacob Keller <jacob.e.keller@intel.com>
In-Reply-To: <20231029110442.347448-1-mirsad.todorovac@alu.unizg.hr>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZDBmY2VQVUkzMG9USnNqV1FwejMrckFmRGdHRE50emJYdjV6T1BzNTM3c2JP?=
 =?utf-8?B?cWQ5M2Z3M2JMM2w2dHY3Mm9oaWtpTjBDb3dwWUlBczVYVlVXV0tCYXdtWWpI?=
 =?utf-8?B?ZGdYSVdJbXl3TnR2Uzl2cDVqMDFzbEZUdmJ4UjBuTk1QQ09lNHAxMDZodk9E?=
 =?utf-8?B?Ym81VUo4cm8wSTJmMHl3VmVQbHBaaVZPS0FFUXI0MVViUmtBMkpoR2EvejZC?=
 =?utf-8?B?aDM0Wm1jc2U3bEJmZnM2VE5ObENwa0dOcUlLN0JrSVZDNUptWllPMWxZaVFr?=
 =?utf-8?B?dTFYdUw4TjAvNndJQ28wUnVRNUo5cStvOVNGTTlWUk1qYnpEbjkwYWl4eWZN?=
 =?utf-8?B?ZjNENUcyVEZ6ZW91aFEyZnJSQkZPVUd5cTdyYkZGVThCVm81YWEzQ2w5N05o?=
 =?utf-8?B?clFYY0U5SkR1V2lXa2VmRUpsQ3VjNmdKR1dwcEY4dStiSHZPb1NRQWYzSFpR?=
 =?utf-8?B?V3dqRXpMOHk5Q3JFQllRKzhNMitSVFoyc3A0Zm04Y0NjcHNEbnFXT0Vxc1VI?=
 =?utf-8?B?NytucmZQOURDcFpCYnRtTUV1SExYQnJTWitRMEQwUlFpalNuZWtDTko2SWpF?=
 =?utf-8?B?U3k3OUduYUdRVDgxbDBJQW9OOTRlOG9paHg0eWhnZEpVSU1PUU9JUDUxTEdX?=
 =?utf-8?B?andxeGNTYjlVcmNPL0t6NGUxdVF0N0Q2UjFOVGpWVkM1OGFGZ00xL1JxS2RN?=
 =?utf-8?B?T1YyNEZWRlVGZVNaZFB1YncycGdJcElJbHpvbW5MMWpyWDNRdkZLK2lCOUhH?=
 =?utf-8?B?K3ZRRHkzUnBjM3M2TGlaeDlYR3pBRnJ4UWE0VFpQM2ZoTUp2OTlDb29ualNJ?=
 =?utf-8?B?aVFzRmpNQVBVdHpHT2pERnhFS2dwWFNTdG0rbHkwWExZY2FzV3VnSUpXWXlW?=
 =?utf-8?B?d2JLSDdvVmFVL1FpTkk1eGR0cGE5RE1KUE0xdGx1UGl0dC9WN2JrdXVEUFBx?=
 =?utf-8?B?UVJnKzRyV2N1SEYzVVlaRy9ydGJ3bHROeS9GNzNnbEM3TytCMDByRmEzeWx5?=
 =?utf-8?B?WU1yck1CSFp5djVqZ2xYMFdWeTNRbkdlSVNaTnNYNGM1N3UzV3lLYWhuazZ0?=
 =?utf-8?B?RjZ1RWczakJyWjcyTTRBRkdHSVhMSXNlRFN5TkJNMWYrWThqNWZoamZ5ZjV2?=
 =?utf-8?B?RWhyRDVJWGlKcWZCRFVtTU1FdDJkc0M1YU9LaVhRNXh6WDRzNHFORXdDcTIz?=
 =?utf-8?B?WU1GK29rcmljUEF3aStCdXIyWThmc0VRZWFOSmFGTlFiL0FNM2NmYzVGL2Vl?=
 =?utf-8?B?NkxaNy9IU1NsQ1U2ZDhaaERoMTBpWlR2UGZ3dzN5elAzUEp6cHBjcm9SM2tk?=
 =?utf-8?B?ektQTzZ1dzkrZ0Q2cGs1bm9UOVZ3Nm1kQzlIdHNieDdLTGJNTDRQcXJ2cWYx?=
 =?utf-8?B?RUJtRTNMOEx2NVVVRWxFOVNjRVJ3aWE1Z291ODdLNFc3OS93OVBUR0tRb2Er?=
 =?utf-8?B?MHBDL3M3Y25CVDB1VUdMaTZ4Q0dTUkhPYlBjblhoYnJmNFNuWVhnenhwY0gr?=
 =?utf-8?B?K01NYURoUmNqNkI3dGZQeG1WZnRxdDAycllrL1hwdGZRRzdKWGNoaVhpU0pm?=
 =?utf-8?B?Vk1KK3p4Z1ROZWZYY2oxMjcyNFE1NUczRkdpWGZiYmhqVkVtWiszb3gzY3Nm?=
 =?utf-8?B?cTgzTnNHWjFmSmx0RHR6VG5CMnMzZXBzSloxVnFUYWxhamsvRVBpRGU3UlJo?=
 =?utf-8?B?ZWoxWlJRUmJLTWM4M3oyTE9oQnBCMHpiNy82SUh6ZUZYYVpDZXQwWk9NS1pq?=
 =?utf-8?B?UFlaa3orbHpHdWFzUEdJdFpQVGVDeDlxcUs1UjJ2d3ozdDBFdk92K3hvQmNI?=
 =?utf-8?B?RnNLNkNHeWpUK0JYQ3UrY2hzTUF1NytDby9HdUl4L1RPb0gzd3IwWkJ1bzRx?=
 =?utf-8?B?N0RWa0FZVWtBSlNHZ3ZNNWZWQ2FBR3VoL1hMc0FTWmd1YSs3OGNOSGdSWXFr?=
 =?utf-8?B?b3c5dTdjT1FyQU9KUzg4N2NmYVJ2OVNEemJGUlZ1VWVaKzFKdFB4WFhLM1dX?=
 =?utf-8?B?U2tGUzZhSlZqM2o1RGhybDkvb2RXN1hMMjNkVi90TDZZZ2JISjU3K25qZjBX?=
 =?utf-8?B?TzQ0NEtvOHBCakpCamNjTytYdHJtUU50TTl6TjVGK1ltSGRjN0ozVFlXSi9k?=
 =?utf-8?B?eG4wZ1RGS1hpSW9kZFBoVkJOL0FBZGRScHYvWkdtaHlremkrM05OWUhmRVZI?=
 =?utf-8?B?eGc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 8dc5fa9e-99c0-4d29-0a64-08dbd99243bf
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5095.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Oct 2023 21:50:43.0404
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zmNgeGwNGM4wQTgmvu0vzo+1PLlJdH5+3pOKKQ4Ut9h2zYMGyyWVxRgCvL4vQlh5BUlqTQHj/mrTWdrTWWdIVo+G7RXbGnwuqiUSf7aURNc=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB8072
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 30 Oct 2023 14:51:18 -0700 (PDT)


On 10/29/2023 4:04 AM, Mirsad Goran Todorovac wrote:> A pair of new
helpers r8168_mac_ocp_write_seq() and r8168_mac_ocp_modify_seq()
> are introduced.
> 
> The motivation for these helpers was the locking overhead of 130 consecutive
> r8168_mac_ocp_write() calls in the RTL8411b reset after the NIC gets confused
> if the PHY is powered-down.
> 
> To quote Heiner:
> 
>     On RTL8411b the RX unit gets confused if the PHY is powered-down.
>     This was reported in [0] and confirmed by Realtek. Realtek provided
>     a sequence to fix the RX unit after PHY wakeup.
> 
> A series of about 130 r8168_mac_ocp_write() calls is performed to program the
> RTL registers for recovery, each doing an expensive spin_lock_irqsave() and
> spin_unlock_irqrestore().
> 
> Each mac ocp write is made of:
> 
>     static void __r8168_mac_ocp_write(struct rtl8169_private *tp, u32 reg,
>                       u32 data)
>     {
>         if (rtl_ocp_reg_failure(reg))
>             return;
> 
>         RTL_W32(tp, OCPDR, OCPAR_FLAG | (reg << 15) | data);
>     }
> 
>     static void r8168_mac_ocp_write(struct rtl8169_private *tp, u32 reg,
>                     u32 data)
>     {
>         unsigned long flags;
> 
>         raw_spin_lock_irqsave(&tp->mac_ocp_lock, flags);
>         __r8168_mac_ocp_write(tp, reg, data);
>         raw_spin_unlock_irqrestore(&tp->mac_ocp_lock, flags);
>     }
> 
> Register programming is done through RTL_W32() macro which expands into
> 
>     #define RTL_W32(tp, reg, val32) writel((val32), tp->mmio_addr + (reg))
> 
> which is further (on Alpha):
> 
>     extern inline void writel(u32 b, volatile void __iomem *addr)
>     {
>         mb();
>         __raw_writel(b, addr);
>     }
> 
> or on i386/x86_64:
> 
>     #define build_mmio_write(name, size, type, reg, barrier) \
>     static inline void name(type val, volatile void __iomem *addr) \
>     { asm volatile("mov" size " %0,%1": :reg (val), \
>     "m" (*(volatile type __force *)addr) barrier); }
> 
>     build_mmio_write(writel, "l", unsigned int, "r", :"memory")
> 
> This obviously involves iat least a compiler barrier.
> 
> mb() expands into something like this i.e. on x86_64:
> 
>     #define mb()    asm volatile("lock; addl $0,0(%%esp)" ::: "memory")
> 
> This means a whole lot of memory bus stalls: for spin_lock_irqsave(),
> memory barrier, writel(), and spin_unlock_irqrestore().
> 
> With about 130 of these sequential calls to r8168_mac_ocp_write() this looks like
> a lock storm that will stall all of the cores and CPUs on the same memory controller
> for certain time I/O takes to finish.
> 
> In a sequential case of RTL register programming, the writes to RTL registers
> can be coalesced under a same raw spinlock. This can dramatically decrease the
> number of bus stalls in a multicore or multi-CPU system.
> 
> Macro helpers r8168_mac_ocp_write_seq() and r8168_mac_ocp_modify_seq() are
> provided to reduce lock contention:
> 
>     static void rtl_hw_start_8411_2(struct rtl8169_private *tp)
>     {
> 
>         ...
> 
>         /* The following Realtek-provided magic fixes an issue with the RX unit
>          * getting confused after the PHY having been powered-down.
>          */
> 
>         static const struct recover_8411b_info init_zero_seq[] = {
>             { 0xFC28, 0x0000 }, { 0xFC2A, 0x0000 }, { 0xFC2C, 0x0000 },
>             ...
>         };
> 
>         ...
> 
>         r8168_mac_ocp_write_seq(tp, init_zero_seq);
> 
>         ...
> 
>     }
> 
> The hex data is preserved intact through s/r8168_mac_ocp_write[(]tp,/{ / and s/[)];/ },/
> functions that only changed the function names and the ending of the line, so the actual
> hex data is unchanged.
> 
> To repeat, the reason for the introduction of the original commit
> was to enable recovery of the RX unit on the RTL8411b which was confused by the
> powered-down PHY. This sequence of r8168_mac_ocp_write() calls amplifies the problem
> into a series of about 500+ memory bus locks, most waiting for the main memory read,
> modify and write under a LOCK. The memory barrier in RTL_W32 should suffice for
> the programming sequence to reach RTL NIC registers.
> 
> [0] https://bugzilla.redhat.com/show_bug.cgi?id=1692075
> 


I might have chosen to send some of this information as the cover letter
for the series instead of just as part of the commit message for [1/5],
but either way:

Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>