Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 19 Aug 2021 16:26:02 +0100
Message-ID: <87y28xqwvp.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Matteo Croce <mcroce@linux.microsoft.com>
Cc:     Jakub Kicinski <kuba@kernel.org>,
        Eric Dumazet <eric.dumazet@gmail.com>,
        Thierry Reding <thierry.reding@gmail.com>,
        netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-riscv@lists.infradead.org,
        Giuseppe Cavallaro <peppe.cavallaro@st.com>,
        Alexandre Torgue <alexandre.torgue@foss.st.com>,
        "David S. Miller" <davem@davemloft.net>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Drew Fustini <drew@beagleboard.org>,
        Emil Renner Berthing <kernel@esmil.dk>,
        Jon Hunter <jonathanh@nvidia.com>,
        Will Deacon <will@kernel.org>
Subject: Re: [PATCH net-next] stmmac: align RX buffers
In-Reply-To: <20210817020157.3b9d015e@linux.microsoft.com>
References: <20210614022504.24458-1-mcroce@linux.microsoft.com>
        <871r71azjw.wl-maz@kernel.org>
        <YROmOQ+4Kqukgd6z@orome.fritz.box>
        <202417ef-f8ae-895d-4d07-1f9f3d89b4a4@gmail.com>
        <87o8a49idp.wl-maz@kernel.org>
        <fe5f99c8-5655-7fbb-a64e-b5f067c3273c@gmail.com>
        <20210812121835.405d2e37@linux.microsoft.com>
        <874kbuapod.wl-maz@kernel.org>
        <20210816081208.522ac47c@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
        <20210817020157.3b9d015e@linux.microsoft.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Tue, 17 Aug 2021 01:01:57 +0100,
Matteo Croce <mcroce@linux.microsoft.com> wrote:
> 
> On Mon, 16 Aug 2021 08:12:08 -0700
> Jakub Kicinski <kuba@kernel.org> wrote:
> 
> > On Thu, 12 Aug 2021 12:05:38 +0100 Marc Zyngier wrote:
> > > > A possible fix, which takes in account also the XDP headroom for
> > > > stmmac_rx_buf1_len() only could be (only compile tested, I don't
> > > > have the hardware now):  
> > > 
> > > However, this doesn't fix my issue. I still get all sort of
> > > corruption. Probably stmmac_rx_buf2_len() also need adjusting (it
> > > has a similar logic as its buf1 counterpart...)
> > > 
> > > Unless you can fix it very quickly, and given that we're towards the
> > > end of the cycle, I'd be more comfortable if we reverted this patch.
> > 
> > Any luck investigating this one? The rc6 announcement sounds like
> > there may not be that many more rc releases for 5.14.
> 
> Hi Jackub.
> 
> Unfortunately I have only a device with stmmac, and it works fine with
> the patch. It seems that not all hardware suffers from this issue.
> 
> Also, using NET_IP_ALIGN on RX is a common pattern, I think that any
> ethernet device is doing the same to align the IPv4 header.
> 
> Anyway, I asked for two tests on the affected device:
> 1. Change NET_IP_ALIGN with 8, to see if the DMA has problems in
>    receiving to a non word aligned address
> 2. load a nop XDP program (I provided one), to see if the problem is
>    already there when XDP is used

Catching up on email, as I was away earlier this week. I'm yet to test
these two things (hopefully by the end of the day), but I just gave
the patch below a go, just in case...

> 
> I doubt that changing also stmmac_rx_buf2_len would help,
> but it's worth a try, here is a patch:
> 
> diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
> index 7b8404a21544..73d1f0ec66ff 100644
> --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
> +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
> @@ -93,7 +93,7 @@ static int tc = TC_DEFAULT;
>  module_param(tc, int, 0644);
>  MODULE_PARM_DESC(tc, "DMA threshold control value");
>  
> -#define	DEFAULT_BUFSIZE	1536
> +#define	DEFAULT_BUFSIZE	1536 + XDP_PACKET_HEADROOM + NET_IP_ALIGN
>  static int buf_sz = DEFAULT_BUFSIZE;
>  module_param(buf_sz, int, 0644);
>  MODULE_PARM_DESC(buf_sz, "DMA buffer size");
> @@ -4508,12 +4508,12 @@ static unsigned int stmmac_rx_buf1_len(struct stmmac_priv *priv,
>  
>  	/* First descriptor, not last descriptor and not split header */
>  	if (status & rx_not_ls)
> -		return priv->dma_buf_sz;
> +		return priv->dma_buf_sz - stmmac_rx_offset(priv);
>  
>  	plen = stmmac_get_rx_frame_len(priv, p, coe);
>  
>  	/* First descriptor and last descriptor and not split header */
> -	return min_t(unsigned int, priv->dma_buf_sz, plen);
> +	return min_t(unsigned int, priv->dma_buf_sz - stmmac_rx_offset(priv), plen);
>  }
>  
>  static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
> @@ -4529,12 +4529,12 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
>  
>  	/* Not last descriptor */
>  	if (status & rx_not_ls)
> -		return priv->dma_buf_sz;
> +		return priv->dma_buf_sz - stmmac_rx_offset(priv);
>  
>  	plen = stmmac_get_rx_frame_len(priv, p, coe);
>  
>  	/* Last descriptor */
> -	return plen - len;
> +	return plen - len - stmmac_rx_offset(priv);
>  }
>  
>  static int stmmac_xdp_xmit_xdpf(struct stmmac_priv *priv, int queue,

This patch results in a cosmic explosion, as it tries to invalidate a
range of VAs that cannot possibly be allocated to the DMA. Not really
what was expected.

Thanks,

	M.

[   19.587499] Unable to handle kernel write to read-only memory at virtual address ffff000088f89000
[   19.596375] Mem abort info:
[   19.599165]   ESR = 0x9600014f
[   19.602215]   EC = 0x25: DABT (current EL), IL = 32 bits
[   19.607519]   SET = 0, FnV = 0
[   19.610568]   EA = 0, S1PTW = 0
[   19.613703]   FSC = 0x0f: level 3 permission fault
[   19.618487] Data abort info:
[   19.621362]   ISV = 0, ISS = 0x0000014f
[   19.625192]   CM = 1, WnR = 1
[   19.628154] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000fa7b2000
[   19.634844] [ffff000088f89000] pgd=18000002771f7003, p4d=18000002771f7003, pud=1800000275ffd003, pmd=1800000275fb5003, pte=0060000108f89f87
[   19.647358] Internal error: Oops: 9600014f [#1] PREEMPT SMP
[   19.652920] Modules linked in: nls_ascii(E) nls_cp437(E) vfat(E) fat(E) tegra_drm(E) cec(E) drm_kms_helper(E) evdev(E) snd_hda_codec_hdmi(E) snd_hda_tegra(E) snd_hda_codec(E) aes_ce_blk(E) crypto_simd(E) snd_hda_core(E) cryptd(E) snd_hwdep(E) at24(E) aes_ce_cipher(E) snd_pcm(E) ghash_ce(E) gf128mul(E) snd_timer(E) sha2_ce(E) sha256_arm64(E) sha1_ce(E) snd(E) soundcore(E) tegra_bpmp_thermal(E) efi_pstore(E) tegra_xudc(E) udc_core(E) host1x(E) ina3221(E) fuse(E) drm(E) configfs(E) efivarfs(E) ip_tables(E) x_tables(E) autofs4(E) nvme(E) nvme_core(E) t10_pi(E) broadcom(E) bcm_phy_lib(E) ahci_tegra(E) libahci_platform(E) gpio_keys(E) libahci(E) sdhci_tegra(E) dwmac_dwc_qos_eth(E) stmmac_platform(E) libata(E) stmmac(E) sdhci_pltfm(E) pcs_xpcs(E) max77620_regulator(E) xhci_tegra(E) phylink(E) of_mdio(E) cqhci(E) scsi_mod(E) fixed_phy(E) sdhci(E) fwnode_mdio(E) libphy(E)
[   19.729311] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G S          E     5.14.0-rc6-dirty #4155
[   19.737734] Hardware name:  , BIOS 2021.04-rc2-00042-g2dddc1bb29 02/18/2021
[   19.744679] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
[   19.750673] pc : dcache_inval_poc+0x40/0x58
[   19.754852] lr : arch_sync_dma_for_cpu+0x2c/0x40
[   19.759460] sp : ffff800010003be0
[   19.762765] x29: ffff800010003be0 x28: ffff000085338000 x27: 0000000000000028
[   19.769889] x26: ffff000080a8fc00 x25: ffff000085338028 x24: 0000000000000000
[   19.777012] x23: 0000000000000002 x22: 0000000000000002 x21: 00000000ffffffbc
[   19.784135] x20: 0000000108f43000 x19: ffff000080223010 x18: 0000000000000000
[   19.791258] x17: ffff8001e3815000 x16: ffff800010004000 x15: 0000000000004000
[   19.798380] x14: 0000000000000000 x13: 0000000080000000 x12: 0000000000000001
[   19.805504] x11: 0000000000000004 x10: 0000000000000008 x9 : ffff8000109b9a2c
[   19.812626] x8 : 0000000000000000 x7 : 0000000000000001 x6 : ffff000088246500
[   19.819748] x5 : fffffffffffff000 x4 : 0003000108f40000 x3 : 000000000000003f
[   19.826869] x2 : 0000000000000040 x1 : ffff000188f42f80 x0 : ffff000088f89000
[   19.833994] Call trace:
[   19.836434]  dcache_inval_poc+0x40/0x58
[   19.840262]  iommu_dma_sync_single_for_cpu+0xdc/0xe0
[   19.845219]  dma_sync_single_for_cpu+0x3c/0x124
[   19.849742]  stmmac_rx+0x448/0x930 [stmmac]
[   19.853937]  stmmac_napi_poll_rx+0x4c/0xec [stmmac]
[   19.858818]  __napi_poll+0x40/0x210
[   19.862300]  net_rx_action+0x304/0x370
[   19.866041]  __do_softirq+0x130/0x3d8
[   19.869695]  __irq_exit_rcu+0x100/0x110
[   19.873525]  irq_exit+0x1c/0x30
[   19.876658]  handle_domain_irq+0x70/0x9c
[   19.880573]  gic_handle_irq+0x58/0xe0
[   19.884225]  call_on_irq_stack+0x2c/0x54
[   19.888138]  do_interrupt_handler+0x5c/0x70
[   19.892313]  el1_interrupt+0x30/0x70
[   19.895881]  el1h_64_irq_handler+0x18/0x24
[   19.899970]  el1h_64_irq+0x78/0x7c
[   19.903361]  arch_cpu_idle+0x18/0x3c
[   19.906930]  default_idle_call+0x4c/0x1d4
[   19.910931]  cpuidle_idle_call+0x168/0x1e0
[   19.915019]  do_idle+0xb8/0x110
[   19.918152]  cpu_startup_entry+0x30/0x8c
[   19.922065]  rest_init+0xf0/0x100
[   19.925372]  arch_call_rest_init+0x1c/0x28
[   19.929463]  start_kernel+0x4a0/0x4d8
[   19.933114]  __primary_switched+0xc0/0xc8
[   19.937118] Code: 8a230000 54000060 d50b7e20 14000002 (d5087620) 
[   19.943204] ---[ end trace f26fd0d14eea9a7d ]---
[   19.947811] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[   19.954670] SMP: stopping secondary CPUs
[   19.958592] Kernel Offset: 0x1a0000 from 0xffff800010000000
[   19.964151] PHYS_OFFSET: 0x80000000
[   19.967630] CPU features: 0x10000231,00000846
[   19.971976] Memory Limit: none
[   19.975026] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---


-- 
Without deviation from the norm, progress is not possible.