Received-SPF: pass (google.com: domain of linux-kernel+bounces-13137-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Fri, 29 Dec 2023 17:25:19 +0200
From: Vladimir Oltean <olteanv@gmail.com>
To: Jagan Teki <jagan@amarulasolutions.com>
Cc: Andrew Lunn <andrew@lunn.ch>, Heiner Kallweit <hkallweit1@gmail.com>,
	"Andrew F. Davis" <afd@ti.com>,
	Florian Fainelli <f.fainelli@gmail.com>,
	linux-kernel <linux-kernel@vger.kernel.org>, netdev@vger.kernel.org,
	Michael Nazzareno Trimarchi <michael@amarulasolutions.com>,
	Ioana Ciornei <ioana.ciornei@nxp.com>,
	Shawn Guo <shawnguo@kernel.org>,
	linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
	Fabio Estevam <festevam@gmail.com>
Subject: Re: PHY issue with SJA1105Q/DP84849I Design
Message-ID: <20231229152519.2jxrwaeltp4pxlms@skbuf>
References: <CAMty3ZCn+yGr2MG3WYg+i4DsZWk5b-xEw0SDvNbeGzs6pMwjfQ@mail.gmail.com>
 <20231222145100.sfcuux7ayxtxgogo@skbuf>
 <CAMty3ZBZNugYmKMjDdZnY0kFMeEb86uzSg2XL9Tn6Yb4t-TXKQ@mail.gmail.com>
 <20231226153055.4yihsmu6kiak6hkf@skbuf>
 <CAMty3ZCn+yGr2MG3WYg+i4DsZWk5b-xEw0SDvNbeGzs6pMwjfQ@mail.gmail.com>
 <20231222145100.sfcuux7ayxtxgogo@skbuf>
 <CAMty3ZBZNugYmKMjDdZnY0kFMeEb86uzSg2XL9Tn6Yb4t-TXKQ@mail.gmail.com>
 <20231226153055.4yihsmu6kiak6hkf@skbuf>
 <CAMty3ZDnAFR9a1BM89mx3bmrQzGC7nvazt42_v4JF_QpwPkS4w@mail.gmail.com>
 <CAMty3ZDnAFR9a1BM89mx3bmrQzGC7nvazt42_v4JF_QpwPkS4w@mail.gmail.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAMty3ZDnAFR9a1BM89mx3bmrQzGC7nvazt42_v4JF_QpwPkS4w@mail.gmail.com>
 <CAMty3ZDnAFR9a1BM89mx3bmrQzGC7nvazt42_v4JF_QpwPkS4w@mail.gmail.com>

On Fri, Dec 29, 2023 at 05:12:39PM +0530, Jagan Teki wrote:
> With fec0 fixed-link and 3 different switch port configurations, the
> result of the link seems to be up but the ping not working and even
> the packets are not transmitted via eth0.
> 
> DT Combinations:
> 
> - Port0 is ethphy0, Port1 is ethphy1, Port2 is disabled, Port3 is
> disabled, Port4 is FEC
> - Port0 is disabled, Port1 is ethphy0, Port2 is ethphy1, Port3 is
> disabled, Port4 is FEC
> - Port0 is disabled, Port1 is disabled, Port2 is ethphy0, Port3 is
> ethphy1, Port4 as FEC

Why all these combinations? You don't know which switch port is which?

> DT: (with Port0 is ethphy0, Port1 is ethphy1, Port2 is disabled, Port3
> is disabled, Port4 is FEC)
> 
> &ecspi2 {
>     cs-gpios = <&gpio2 27 GPIO_ACTIVE_HIGH>;
>     pinctrl-names = "default";
>     pinctrl-0 = <&pinctrl_ecspi2>;
>     status = "okay";
> 
>     switch@0 {
>         compatible = "nxp,sja1105q";
>         reg = <0>;
>         spi-max-frequency = <4000000>;
>         spi-rx-delay-us = <1>;
>         spi-tx-delay-us = <1>;
>         spi-cpha;
> 
>        clocks = <&clk25m>;
> 
>        pinctrl-0 = <&pinctrl_sja1105_rst>;
>        pinctrl-names = "default";
>        reset-gpios = <&gpio6 5 GPIO_ACTIVE_LOW>;
> 
>        ports {
>               #address-cells = <1>;
>               #size-cells = <0>;
> 
>              port@0 {
>                   reg = <0>;
>                   label = "ethphy0";
>                   phy-handle = <&ethphy0>;
>                   phy-mode = "mii";
>             };
> 
>             port@1 {
>                  reg = <1>;
>                  label = "ethphy1";
>                  phy-handle = <&ethphy1>;
>                  phy-mode = "mii";
>            };
> 
>            port@2 {
>                 reg = <2>;
>                 status = "disabled";
>            };
> 
>             port@3 {
>                  reg = <3>;
>                  status = "disabled";
>            };
> 
>            port@4 {
>                 reg = <4>;
>                 label = "cpu";
>                 ethernet = <&fec>;
>                 phy-mode = "mii";
>                 rx-internal-delay-ps = <2000>;
>                 tx-internal-delay-ps = <2000>;

This looks suspicious. "rx-internal-delay-ps" and "tx-internal-delay-ps"
are only relevant for the RGMII modes, but you specify phy-mode = "mii".
Does the board schematic confirm that MII is the physical connection
being used from the switch to the FEC?

If you are truly using MII, then you should remove the RGMII delay
properties, and since you are using a 6.1 kernel - hence after kernel
commit 5d645df99ac6 ("net: dsa: sja1105: determine PHY/MAC role from PHY
interface type") - you should be using phy-mode = "rev-mii" to put this
port in MII PHY ("RevMII") mode - to interoperate with the FEC in MII
MAC mode.

> 
>                fixed-link {
>                    speed = <100>;
>                    full-duplex;
>                };
>             };
>           };
>      };
> };
> 
> &fec {
>        pinctrl-names = "default";
>        pinctrl-0 = <&pinctrl_enet>;
>        phy-mode = "mii";
>        status = "okay";
> 
>        fixed-link {
>           speed = <100>;
>           full-duplex;
>        };
> 
>        mdio {
>             #address-cells = <1>;
>             #size-cells = <0>;
> 
>             ethphy0: ethernet-phy@0 {
>                     compatible = "ethernet-phy-ieee802.3-c22";
>                     reg = <0>;
>             };
> 
>            ethphy1: ethernet-phy@1 {
>                compatible = "ethernet-phy-ieee802.3-c22";
>                 reg = <1>;
>             };
>     };
> };
> 
> root@ltts-imx6solo:~# bash /usr/phynew.sh
> ======= MDIO: PHY0 ========
> [  162.426515] mdio_netlink: loading out-of-tree module taints kernel.

Still, please refrain from involving out-of-tree modules when asking for
help upstream. Thanks.

> root@ltts-imx6solo:~# [  165.208656] sja1105 spi1.0 ethphy0: Link is
> Up - 100Mbps/Full - flow control off
> [  165.225788] sja1105 spi1.0 ethphy1: Link is Up - 100Mbps/Full -
> flow control off
> [  165.235925] IPv6: ADDRCONF(NETDEV_CHANGE): ethphy0: link becomes ready
> [  165.255777] IPv6: ADDRCONF(NETDEV_CHANGE): ethphy1: link becomes ready
> 
> root@ltts-imx6solo:~# ifconfig
> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1504
>         inet6 fe80::68fb:8ff:fedf:d377  prefixlen 64  scopeid 0x20<link>
>         ether 6a:fb:08:df:d3:77  txqueuelen 1000  (Ethernet)
>         RX packets 0  bytes 0 (0.0 B)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 0  bytes 0 (0.0 B)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> ethphy0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 169.254.178.1  netmask 255.255.0.0  broadcast 0.0.0.0
>         inet6 fe80::211:22ff:fe33:4455  prefixlen 64  scopeid 0x20<link>
>         ether 00:11:22:33:44:55  txqueuelen 1000  (Ethernet)
>         RX packets 0  bytes 0 (0.0 B)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 30  bytes 4071 (3.9 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> ethphy1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 169.253.178.2  netmask 255.255.0.0  broadcast 0.0.0.0
>         inet6 fe80::211:22ff:fe33:4466  prefixlen 64  scopeid 0x20<link>
>         ether 00:11:22:33:44:66  txqueuelen 1000  (Ethernet)
>         RX packets 0  bytes 0 (0.0 B)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 30  bytes 4071 (3.9 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
>         inet 127.0.0.1  netmask 255.0.0.0
>         inet6 ::1  prefixlen 128  scopeid 0x10<host>
>         loop  txqueuelen 1000  (Local Loopback)
>         RX packets 89  bytes 7675 (7.4 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 89  bytes 7675 (7.4 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ifconfig reports statistics counters from the /proc/net/dev interface,
which the sja1105 does not report very well (they don't come from hardware).
It's best to use "ethtool -S eth0 | grep -v ': 0'" for FEC and SJA1105
CPU port (named "p04_*") counters, and "ethtool -S ethphy0 | grep -v ': 0'"
to get hardware counters from the switch user ports.

You can also use the RX counters to determine which switch port is which
(but the phy-handle of each port to each PHY needs to be correct).

> mytsl02383@MYTSL02383:~$ ifconfig -a
> enp43s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 169.254.178.2  netmask 255.255.0.0  broadcast 169.254.255.255
>         inet6 fe80::d71b:4bdd:27bd:2a1a  prefixlen 64  scopeid 0x20<link>
>         ether 00:be:43:20:9a:26  txqueuelen 1000  (Ethernet)
>         RX packets 272356  bytes 27099064 (27.0 MB)
>         RX errors 0  dropped 19  overruns 0  frame 0
>         TX packets 862  bytes 300806 (300.8 KB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> mytsl02383@MYTSL02383:~$ ping 169.254.178.1
> PING 169.254.178.1 (169.254.178.1) 56(84) bytes of data.
> From 169.254.178.2 icmp_seq=1 Destination Host Unreachable
> 
> Let me know if you need any more details.

I'm not convinced packets are routed through ethphy0 or ethphy1, since
all interfaces have IPv4 link-local addresses only. You can use
"ip route get 169.254.178.1" to confirm what interface gets chosen.
This is not indicative of a device-level problem, just a setup one.
Please set up some IPv4 static addresses which are not link-local on the
DSA user ports and try to ping a link partner which has an IP address in
the same subnet.