Received: by 2002:a05:7412:d8a:b0:e2:908c:2ebd with SMTP id b10csp735495rdg; Thu, 12 Oct 2023 22:02:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEi2Gw3NzmPM/LbJfOpViT/pPdBf0/zg43L9lzGRmzUOz8mNdvk/deHPR+IG+E1LuGcaABN X-Received: by 2002:a17:902:da84:b0:1c1:ed61:e058 with SMTP id j4-20020a170902da8400b001c1ed61e058mr31536628plx.16.1697173355516; Thu, 12 Oct 2023 22:02:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697173355; cv=none; d=google.com; s=arc-20160816; b=UIxNcNpSyKp14Tgt0Pme7QUz1u413ELOq9/27qcGC2ckZbBeNj3xdDOZP3yWniaf62 f2cyriAea3mZuO2KZ3bNB+TtdJy8hDSMgeUFoLFqScKhzvgMyBBBKc8U3ALlKdr9T64h CR78pEgl9TUaG8p2t7ek3Ye12i8QVYy+2Ti4vvhprFbu3mu5FmSjCIo1Bh7RjIC0z9Tr HQAW4qsSWnEN/flLTjLiD9rxYZf3H3Ck2bpgMVnRk7MZiFNw084aU6t4pR1yzhELAmX/ ury2ec6y+HYFLvFfgufnmqid+o3AkMsiZt7mrY0yWU7BvubyHGoZcpa6ZltQ2fOHZIMP 6axg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:in-reply-to :organization:autocrypt:from:references:cc:to:content-language :subject:user-agent:mime-version:date:message-id:dkim-signature :dkim-signature; bh=/dUnXa4bMa4kVD2/LGJ0UV0zuyHRm/6vpx8CyqJg3jc=; fh=RX7v26HnI82S2hZf6DMWM91XtrC8BicMZoJZrRQa98c=; b=D8vTuErJt1NbwPUEOcGup/bPbcPzBK7nrjtjY6zB7bRVOkTUXIlySKIpz0rF/uqIOe h7FxljdPivcQUBBw5T7Kj2CHC6xtw6a76yC2YUHDtnEKHCgIbqD9npLti2cxq798PaAd LhaVIqGiuvSJlpJMVXyXeGRozOU3nkSkeAmf0cumwWnez7+xIPwqFxIITuyIfOlcH2At bEWpc7ej5QbAqTUyr2fyF4tOQQCAAvW+FH3km8g1ttnfPpMbi4DdhOTgQVd6rwmXHIRJ p3fVCDbJVd2sKJeKdzRf7eehzPk40OurlS51gDyMEie+Foiihrb1R1jVDpI5hplsm6pK He9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@alu.unizg.hr header.s=mail header.b=e2m9wx9U; dkim=fail header.i=@alu.unizg.hr header.s=mail header.b=QLRdvd1m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alu.unizg.hr Return-Path: Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id q11-20020a170902bd8b00b001bbf293f45esi3538523pls.625.2023.10.12.22.02.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Oct 2023 22:02:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=fail header.i=@alu.unizg.hr header.s=mail header.b=e2m9wx9U; dkim=fail header.i=@alu.unizg.hr header.s=mail header.b=QLRdvd1m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alu.unizg.hr Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 4DEF58092D86; Thu, 12 Oct 2023 22:01:38 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229504AbjJMFB2 (ORCPT + 99 others); Fri, 13 Oct 2023 01:01:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229436AbjJMFB0 (ORCPT ); Fri, 13 Oct 2023 01:01:26 -0400 Received: from domac.alu.hr (domac.alu.unizg.hr [IPv6:2001:b68:2:2800::3]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 851F9B7; Thu, 12 Oct 2023 22:01:23 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by domac.alu.hr (Postfix) with ESMTP id 284FE60152; Fri, 13 Oct 2023 07:01:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=alu.unizg.hr; s=mail; t=1697173280; bh=mhSRRMLz3jAmea0c/jfOxmFHe6AXoaopsEh0rw/b3gA=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=e2m9wx9UCdqvrvXBRAgbBNTu1S0TAKErc7FwON5uwKHePP5AZYJGh/2EECcoyzPEc 2IjM4OEYLJ9HcRogs/Qkqx2Mj9WU7OWrHrFeXXVj9QLZZisHlAfkFbMDa81jBGzwuT BhB3SNCvlzEFCOMjNWagaxIxglkF70w5cMUOFzhc9VDsrOBX3eoxxU9xNCimN9E4zw B5tJ2wm0ngk9iNpK1Dlc1+VvtPincPYwB5etcgwuaLC4qux+RCHy6OGuzWtYcwVGnW 6D934I1w3mQA/tgSq3uMuwh4YeGZEQ+Dw7eHDXGCe6FGfRGPijGeP+eJnTpIgXR2bM xuVRL+bmrpiwQ== X-Virus-Scanned: Debian amavisd-new at domac.alu.hr Received: from domac.alu.hr ([127.0.0.1]) by localhost (domac.alu.hr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w7y9ULEQZ4SK; Fri, 13 Oct 2023 07:01:16 +0200 (CEST) Received: from [192.168.1.3] (78-2-88-84.adsl.net.t-com.hr [78.2.88.84]) by domac.alu.hr (Postfix) with ESMTPSA id 8A61C6013C; Fri, 13 Oct 2023 07:01:15 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=alu.unizg.hr; s=mail; t=1697173276; bh=mhSRRMLz3jAmea0c/jfOxmFHe6AXoaopsEh0rw/b3gA=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=QLRdvd1mJ2L8o0XO/ipNMd7s6g386TnpZTuZD6sNeHErNiQ1lhiAM8uy9NOix1n3V BukV8V8L+hO4EyRo4fGKfdtFwiCvRDFwhNHreZZRogrxg/AIc/aAmPBYhEHjTKovqs kL7gxua+s6P6YO+lf8TUSvDMGfAlG0CXrcN8lpzZWeJfOQaZvMovh/y/XRU34GiYu4 inN7I6IE3cs5PHX8t9Dp9kgV8QuvjX56I1nv7iLcMjJ2Rori0dWUU5AW0tLevevqrR TI9xozBmwkyBh+b1Mjw04G44l3HwWAozmzxHVUd5JWe3cXYGqz06gS4uO8jKMDC8vR KCVeioZdR28aw== Message-ID: <250140d1-d592-45f4-aa27-691c1d68a528@alu.unizg.hr> Date: Fri, 13 Oct 2023 07:01:15 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC v1 1/3] r8169: fix the KCSAN reported data-race in rtl_tx() while reading tp->cur_tx Content-Language: en-US To: Marco Elver , Heiner Kallweit Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, nic_swsd@realtek.com, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni References: <20230927184158.243575-1-mirsad.todorovac@alu.unizg.hr> <0a201a6f-90dd-403c-97d0-94372be1e3e6@gmail.com> From: Mirsad Goran Todorovac Autocrypt: addr=mirsad.todorovac@alu.unizg.hr; keydata= xjMEYp0QmBYJKwYBBAHaRw8BAQdAI14D1/OE3jLBYycg8HaOJOYrvEaox0abFZtJf3vagyLN Nk1pcnNhZCBHb3JhbiBUb2Rvcm92YWMgPG1pcnNhZC50b2Rvcm92YWNAYWx1LnVuaXpnLmhy PsKPBBMWCAA3FiEEdCs8n09L2Xwp/ytk6p9/SWOJhIAFAmKdEJgFCQ0oaIACGwMECwkIBwUV CAkKCwUWAgMBAAAKCRDqn39JY4mEgIf/AP9hx09nve6VH6D/F3m5jRT5m1lzt5YzSMpxLGGU vGlI4QEAvOvGI6gPCQMhuQQrOfRr1CnnTXeaXHhlp9GaZEW45QzOOARinRCZEgorBgEEAZdV AQUBAQdAqJ1CxZGdTsiS0cqW3AvoufnWUIC/h3W2rpJ+HUxm61QDAQgHwn4EGBYIACYWIQR0 KzyfT0vZfCn/K2Tqn39JY4mEgAUCYp0QmQUJDShogAIbDAAKCRDqn39JY4mEgIMnAQDPKMJJ fs8+QnWS2xx299NkVTRsZwfg54z9NIvH5L3HiAD9FT3zfHfvQxIViWEzcj0q+FLWoRkOh02P Ny0lWTyFlgc= Organization: Academy of Fine Arts, University of Zagreb In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 12 Oct 2023 22:01:38 -0700 (PDT) On 9/28/2023 8:02 AM, Marco Elver wrote: > On Wed, 27 Sept 2023 at 21:52, Heiner Kallweit wrote: >> >> On 27.09.2023 20:41, Mirsad Goran Todorovac wrote: >>> KCSAN reported the following data-race: >>> >>> ================================================================== >>> BUG: KCSAN: data-race in rtl8169_poll [r8169] / rtl8169_start_xmit [r8169] >>> >>> write (marked) to 0xffff888102474b74 of 4 bytes by task 5358 on cpu 29: >>> rtl8169_start_xmit (drivers/net/ethernet/realtek/r8169_main.c:4254) r8169 >>> dev_hard_start_xmit (./include/linux/netdevice.h:4889 ./include/linux/netdevice.h:4903 net/core/dev.c:3544 net/core/dev.c:3560) >>> sch_direct_xmit (net/sched/sch_generic.c:342) >>> __dev_queue_xmit (net/core/dev.c:3817 net/core/dev.c:4306) >>> ip_finish_output2 (./include/linux/netdevice.h:3082 ./include/net/neighbour.h:526 ./include/net/neighbour.h:540 net/ipv4/ip_output.c:233) >>> __ip_finish_output (net/ipv4/ip_output.c:311 net/ipv4/ip_output.c:293) >>> ip_finish_output (net/ipv4/ip_output.c:328) >>> ip_output (net/ipv4/ip_output.c:435) >>> ip_send_skb (./include/net/dst.h:458 net/ipv4/ip_output.c:127 net/ipv4/ip_output.c:1486) >>> udp_send_skb (net/ipv4/udp.c:963) >>> udp_sendmsg (net/ipv4/udp.c:1246) >>> inet_sendmsg (net/ipv4/af_inet.c:840 (discriminator 4)) >>> sock_sendmsg (net/socket.c:730 net/socket.c:753) >>> __sys_sendto (net/socket.c:2177) >>> __x64_sys_sendto (net/socket.c:2185) >>> do_syscall_64 (arch/x86/entry/common.c:50 arch/x86/entry/common.c:80) >>> entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:120) >>> >>> read to 0xffff888102474b74 of 4 bytes by interrupt on cpu 21: >>> rtl8169_poll (drivers/net/ethernet/realtek/r8169_main.c:4397 drivers/net/ethernet/realtek/r8169_main.c:4581) r8169 >>> __napi_poll (net/core/dev.c:6527) >>> net_rx_action (net/core/dev.c:6596 net/core/dev.c:6727) >>> __do_softirq (kernel/softirq.c:553) >>> __irq_exit_rcu (kernel/softirq.c:427 kernel/softirq.c:632) >>> irq_exit_rcu (kernel/softirq.c:647) >>> common_interrupt (arch/x86/kernel/irq.c:247 (discriminator 14)) >>> asm_common_interrupt (./arch/x86/include/asm/idtentry.h:636) >>> cpuidle_enter_state (drivers/cpuidle/cpuidle.c:291) >>> cpuidle_enter (drivers/cpuidle/cpuidle.c:390) >>> call_cpuidle (kernel/sched/idle.c:135) >>> do_idle (kernel/sched/idle.c:219 kernel/sched/idle.c:282) >>> cpu_startup_entry (kernel/sched/idle.c:378 (discriminator 1)) >>> start_secondary (arch/x86/kernel/smpboot.c:210 arch/x86/kernel/smpboot.c:294) >>> secondary_startup_64_no_verify (arch/x86/kernel/head_64.S:433) >>> >>> value changed: 0x002f4815 -> 0x002f4816 >>> >>> Reported by Kernel Concurrency Sanitizer on: >>> CPU: 21 PID: 0 Comm: swapper/21 Tainted: G L 6.6.0-rc2-kcsan-00143-gb5cbe7c00aa0 #41 >>> Hardware name: ASRock X670E PG Lightning/X670E PG Lightning, BIOS 1.21 04/26/2023 >>> ================================================================== >>> >>> The write side of drivers/net/ethernet/realtek/r8169_main.c is: >>> ================== >>> 4251 /* rtl_tx needs to see descriptor changes before updated tp->cur_tx */ >>> 4252 smp_wmb(); >>> 4253 >>> → 4254 WRITE_ONCE(tp->cur_tx, tp->cur_tx + frags + 1); >>> 4255 >>> 4256 stop_queue = !netif_subqueue_maybe_stop(dev, 0, rtl_tx_slots_avail(tp), >>> 4257 R8169_TX_STOP_THRS, >>> 4258 R8169_TX_START_THRS); >>> >>> The read side is the function rtl_tx(): >>> >>> 4355 static void rtl_tx(struct net_device *dev, struct rtl8169_private *tp, >>> 4356 int budget) >>> 4357 { >>> 4358 unsigned int dirty_tx, bytes_compl = 0, pkts_compl = 0; >>> 4359 struct sk_buff *skb; >>> 4360 >>> 4361 dirty_tx = tp->dirty_tx; >>> 4362 >>> 4363 while (READ_ONCE(tp->cur_tx) != dirty_tx) { >>> 4364 unsigned int entry = dirty_tx % NUM_TX_DESC; >>> 4365 u32 status; >>> 4366 >>> 4367 status = le32_to_cpu(tp->TxDescArray[entry].opts1); >>> 4368 if (status & DescOwn) >>> 4369 break; >>> 4370 >>> 4371 skb = tp->tx_skb[entry].skb; >>> 4372 rtl8169_unmap_tx_skb(tp, entry); >>> 4373 >>> 4374 if (skb) { >>> 4375 pkts_compl++; >>> 4376 bytes_compl += skb->len; >>> 4377 napi_consume_skb(skb, budget); >>> 4378 } >>> 4379 dirty_tx++; >>> 4380 } >>> 4381 >>> 4382 if (tp->dirty_tx != dirty_tx) { >>> 4383 dev_sw_netstats_tx_add(dev, pkts_compl, bytes_compl); >>> 4384 WRITE_ONCE(tp->dirty_tx, dirty_tx); >>> 4385 >>> 4386 netif_subqueue_completed_wake(dev, 0, pkts_compl, bytes_compl, >>> 4387 rtl_tx_slots_avail(tp), >>> 4388 R8169_TX_START_THRS); >>> 4389 /* >>> 4390 * 8168 hack: TxPoll requests are lost when the Tx packets are >>> 4391 * too close. Let's kick an extra TxPoll request when a burst >>> 4392 * of start_xmit activity is detected (if it is not detected, >>> 4393 * it is slow enough). -- FR >>> 4394 * If skb is NULL then we come here again once a tx irq is >>> 4395 * triggered after the last fragment is marked transmitted. >>> 4396 */ >>> → 4397 if (tp->cur_tx != dirty_tx && skb) >>> 4398 rtl8169_doorbell(tp); >>> 4399 } >>> 4400 } >>> >>> Obviously from the code, an earlier detected data-race for tp->cur_tx was fixed in the >>> line 4363: >>> >>> 4363 while (READ_ONCE(tp->cur_tx) != dirty_tx) { >>> >>> but the same solution is required for protecting the other access to tp->cur_tx: >>> >>> → 4397 if (READ_ONCE(tp->cur_tx) != dirty_tx && skb) >>> 4398 rtl8169_doorbell(tp); >>> >>> The write in the line 4254 is protected with WRITE_ONCE(), but the read in the line 4397 >>> might have suffered read tearing under some compiler optimisations. >>> >>> The fix eliminated the KCSAN data-race report for this bug. >>> >>> It is yet to be evaluated what happens if tp->cur_tx changes between the test in line 4363 >>> and line 4397. This test should certainly not be cached by the compiler in some register >>> for such a long time, while asynchronous writes to tp->cur_tx might have occurred in line >>> 4254 in the meantime. >>> >> >> netif_subqueue_completed_wake() has barriers ensuring that no cached value for tp->cur_tx >> is used in line 4397. I'm not aware of any reported issues with an obvious link to the >> potentential issue you describe. >> I don't have a strong opinion on these patches. They shouldn't hurt, and if they make >> KCSAN happy, why not. > > Barries don't protect unmarked accesses from being miscompiled. So the > use of barriers and marked accesses like READ_ONCE() is correct: > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/memory-model/Documentation/access-marking.txt > That said, actually encountering a miscompilation depends on > architecture and compiler. Using the right marked accesses just > ensures things don't suddenly break because the compiler decided to be > a little more clever. > >>> Fixes: 94d8a98e6235c ("r8169: reduce number of workaround doorbell rings") >>> Cc: Heiner Kallweit >>> Cc: nic_swsd@realtek.com >>> Cc: "David S. Miller" >>> Cc: Eric Dumazet >>> Cc: Jakub Kicinski >>> Cc: Paolo Abeni >>> Cc: Marco Elver >>> Cc: netdev@vger.kernel.org >>> Link: https://lore.kernel.org/lkml/dc7fc8fa-4ea4-e9a9-30a6-7c83e6b53188@alu.unizg.hr/ >>> Signed-off-by: Mirsad Goran Todorovac > > Acked-by: Marco Elver Hi, Marco, Does this Acked-by: cover all of the [123]/3 in the patch series? I guess I should resubmit the patches as the formal ones as patchwork will not pick up a PATH RFC? Thanks, Mirsad Todorovac >>> --- >>> v1: >>> the initial patch proposal. fixes the KCSAN warning. >>> >>> drivers/net/ethernet/realtek/r8169_main.c | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-) >>> >>> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c >>> index 6351a2dc13bc..281aaa851847 100644 >>> --- a/drivers/net/ethernet/realtek/r8169_main.c >>> +++ b/drivers/net/ethernet/realtek/r8169_main.c >>> @@ -4394,7 +4394,7 @@ static void rtl_tx(struct net_device *dev, struct rtl8169_private *tp, >>> * If skb is NULL then we come here again once a tx irq is >>> * triggered after the last fragment is marked transmitted. >>> */ >>> - if (tp->cur_tx != dirty_tx && skb) >>> + if (READ_ONCE(tp->cur_tx) != dirty_tx && skb) >>> rtl8169_doorbell(tp); >>> } >>> } >> -- Mirsad Todorovac Sistem inženjer Grafički fakultet | Akademija likovnih umjetnosti Sveučilište u Zagrebu System engineer Faculty of Graphic Arts | Academy of Fine Arts University of Zagreb, Republic of Croatia tel. +385 (0)1 3711 451 mob. +385 91 57 88 355