Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Li Chen <lchen@ambarella.com>
To:     Keith Busch <kbusch@kernel.org>
CC:     Bjorn Helgaas <helgaas@kernel.org>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
        Rob Herring <robh@kernel.org>, "kw@linux.com" <kw@linux.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Tom Joseph <tjoseph@cadence.com>, Jens Axboe <axboe@fb.com>,
        Christoph Hellwig <hch@lst.de>,
        Sagi Grimberg <sagi@grimberg.me>,
        "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Subject: RE: [EXT] Re: nvme may get timeout from dd when using different
 non-prefetch mmio outbound/ranges
Thread-Topic: [EXT] Re: nvme may get timeout from dd when using different
 non-prefetch mmio outbound/ranges
Thread-Index: AdfHKUVD4pdAQATdSnyIoo960S9VyQCjl4+AAAEy0AAAFvQC8AAB+KwAAADI3xA=
Date:   Tue, 26 Oct 2021 04:44:32 +0000
Message-ID: <CH2PR19MB4024FEB7A178841BD7EB85F9A0849@CH2PR19MB4024.namprd19.prod.outlook.com>
References: <CH2PR19MB4024E04EBD0E4958F0BBB2ACA0809@CH2PR19MB4024.namprd19.prod.outlook.com>
 <20211025154739.GA4760@bhelgaas>
 <20211025162158.GA2335242@dhcp-10-100-145-180.wdc.com>
 <CH2PR19MB4024372120E0E74C8F2CB685A0849@CH2PR19MB4024.namprd19.prod.outlook.com>
 <20211026041538.GB2335242@dhcp-10-100-145-180.wdc.com>
In-Reply-To: <20211026041538.GB2335242@dhcp-10-100-145-180.wdc.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?mBbfSf/jjHIqPfz5wCG5HvJQ1nnM9cK/KpUZ5sWWgQDwO1Qd35DwN4PJbH+i?=
 =?us-ascii?Q?mEPHftsmYLJU3gAfEUl1MtC+n4r01fUOjUHohipfG+0iTm37jSOZN/4Bnhh7?=
 =?us-ascii?Q?ldFg57Ra8225rG9qkb2NwNRs0PSB0SLIQ5H1h7Qzu7gPvmOOl+UOToGl1u1n?=
 =?us-ascii?Q?VG1tnl+1OELhwwZSmdKur8q0sQwChRvYnLwruCHEih5uCTh3ByHcHnGgD8NI?=
 =?us-ascii?Q?vUHD/MHKe8rd0kpMa9VaZv/MvPuqMo802uDdI174KLysf/Z6iAM9GO17+M/K?=
 =?us-ascii?Q?5NR71UJINbh8tO+An0DIdwFCMerNyG2V+7I6y51vtsWBAbSs9u5iy8FNgpIO?=
 =?us-ascii?Q?maOZv0x9WH0RdzrqT60to3hJ2e2SgP2bFJcahHjnfrJ9zzIjGcjfgzrN89fs?=
 =?us-ascii?Q?PFhAoZWxoE4uCM08sWcgX3r9W4IZTJPq/SJx63RkAfMW4lCFp4Nn/odGb/it?=
 =?us-ascii?Q?szlCr8kKgVvihxLEyGLXiDYRAKjNLS558ihb8leHMdyX+PfmJjt5/3ZuYU3Z?=
 =?us-ascii?Q?2XwHcRWKpMXW0ki8O3h3LyhMm3+lK0OBDjJXDwiaGH4P7disEBIgTK0D8lha?=
 =?us-ascii?Q?+rpiOuv6Zy4tnFBMYQ5huvwqhoKRGvSYUJK3cDvPlp9SAJL4tGq9pDKwTxTA?=
 =?us-ascii?Q?AFcJkPYOqEXhD7N5gTOCM5nIrVU5/6mNrYevYimdowfO2eS/MlBeMdWNTjKz?=
 =?us-ascii?Q?VfgKFCScW4vc0FtST8vi0+/Ig9KORrSOiYw39aOmOvfx92AW785Fz+wpeBu5?=
 =?us-ascii?Q?BPWZSzj0RSnz+8j6HBYZpH1M9l3BKMJk/Ora4PXJfYICahakFSQwe2haDkPn?=
 =?us-ascii?Q?PJYu3uv5zjkVBU10KICQfdi5P2MDtxwiNPhHYaSSbNe6mQ0PbCrPyDVNx/qF?=
 =?us-ascii?Q?gFVKFMzdi+oY2qRWPnofAs3E7f62WFdLg0XlqdeXkoMivP5mVb82K85BxcTn?=
 =?us-ascii?Q?FMYCs/qUTJ3xzQ8wlVolWqu2MzNXOFSLSi7ugaYzJf9NyOFzq0oH7g7jBxPx?=
 =?us-ascii?Q?ERIb2hxvj0GAvEZsr8mkT6AGulNzy/8XdPWTR8MbiikB9As5KnVuQMw9lfds?=
 =?us-ascii?Q?7Z/FI6Hlxqpzem/gNmfF0P1dMCIu9/rj8PUOeuo19xDfdST9/Yfw9BvqTp5u?=
 =?us-ascii?Q?98sVo5hpzVJkDzzxuuSzvckxqtJRBCAU2ObRUOKj3mhxCCmFnHSdwYT2Mn3E?=
 =?us-ascii?Q?WiwToen5pENGmtEbQPNcnTDpOqvn0K9hLM2mbxJRywdi41wauAZ2SRfAUe7l?=
 =?us-ascii?Q?hkQrOeGzTkM6svw1E6vrew7XjsWinN4Rm6666O00IgxbMCzFlYuTZQEN4EX6?=
 =?us-ascii?Q?3C0crhCaiyqWoIKvw/rHxSVz/qkIZ6b3uanr8GjnKRIN+PNMKS4KiUCDpZhD?=
 =?us-ascii?Q?EnB/3rJrO8CFk8PEpA0AOMxndu+bs11L9m43Hr0K/IwSGr9rvmxqVBpXso0w?=
 =?us-ascii?Q?P49TbdgY1CRzhDxbD9nkZxPGO0XcfBJu6ar7XfWfQaZj/Z+9qG+D/smPgrhM?=
 =?us-ascii?Q?yZ6N0CmylUvJslaKO5vxH+TVseidXUdjLtNb5MIecwSf1TKLWC1UV6Uhmd+i?=
 =?us-ascii?Q?afZHH5uYSgBydMyL2acKBb7afzV+vWej8BgVkwv5EEe3d9XZItb1R6qcH/ei?=
 =?us-ascii?Q?1iltoyBL1aDJzZDJ/UJpz0k=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: ambarella.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: CH2PR19MB4024.namprd19.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5f3dcee5-6e78-4cd5-0245-08d9983b4dff
X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Oct 2021 04:44:32.6645
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 3ccd3c8d-5f7c-4eb4-ae6f-32d8c106402c
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: UbhUEGVS+xoKAkNQ4JXKoMOOIveZ8CtuG8Uz1GC38rNIDribws2l5SMb+nb1oe1Tjzm8r5x9NSghMRsFjryq8Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR19MB3672
X-Proofpoint-ORIG-GUID: HPJj5TW_nMghPZ21KRGKCoUmzB71p3U0
X-Proofpoint-GUID: HPJj5TW_nMghPZ21KRGKCoUmzB71p3U0
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.0.607.475
 definitions=2021-10-25_08,2021-10-25_02,2020-04-07_01
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 lowpriorityscore=0
 adultscore=0 clxscore=1015 bulkscore=0 mlxlogscore=999 malwarescore=0
 phishscore=0 suspectscore=0 impostorscore=0 spamscore=0 priorityscore=1501
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2110260025
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi, Keith

> -----Original Message-----
> From: Keith Busch [mailto:kbusch@kernel.org]
> Sent: Tuesday, October 26, 2021 12:16 PM
> To: Li Chen
> Cc: Bjorn Helgaas; linux-pci@vger.kernel.org; Lorenzo Pieralisi; Rob Herr=
ing;
> kw@linux.com; Bjorn Helgaas; linux-kernel@vger.kernel.org; Tom Joseph; Je=
ns
> Axboe; Christoph Hellwig; Sagi Grimberg; linux-nvme@lists.infradead.org
> Subject: Re: [EXT] Re: nvme may get timeout from dd when using different =
non-
> prefetch mmio outbound/ranges
>=20
> On Tue, Oct 26, 2021 at 03:40:54AM +0000, Li Chen wrote:
> > My nvme is " 05:00.0 Non-Volatile memory controller: Samsung Electronic=
s Co
> Ltd NVMe SSD Controller 980". From its datasheet,
> https://urldefense.com/v3/__https://s3.ap-northeast-
> 2.amazonaws.com/global.semi.static/Samsung_NVMe_SSD_980_Data_Sheet_R
> ev.1.1.pdf__;!!PeEy7nZLVv0!3MU3LdTWuzON9JMUkq29zwJM4d7g7wKtkiZszTu-
> PVepWchI_uLHpQGgdR_LEZM$ , it says nothing about CMB/SQEs, so I'm not sur=
e.
> Is there other ways/tools(like nvme-cli) to query?
>=20
> The driver will export a sysfs property for it if it is supported:
>=20
>   # cat /sys/class/nvme/nvme0/cmb
>=20
> If the file doesn't exist, then /dev/nvme0 doesn't have the capability.
>=20

# ls /sys/class/nvme/nvme0/
address            model              rescan_controller  subsysnqn
cntlid             numa_node          reset_controller   subsystem
dev                nvme0n1            serial             transport
device             power              sqsize             uevent
firmware_rev       queue_count        state

my nvme doesn't cmb.


> > > > I don't know how to interpret "ranges".  Can you supply the dmesg a=
nd
> > > > "lspci -vvs 0000:05:00.0" output both ways, e.g.,
> > > >
> > > >   pci_bus 0000:00: root bus resource [mem 0x7f800000-0xefffffff win=
dow]
> > > >   pci_bus 0000:00: root bus resource [mem 0xfd000000-0xfe7fffff win=
dow]
> > > >   pci 0000:05:00.0: [vvvv:dddd] type 00 class 0x...
> > > >   pci 0000:05:00.0: reg 0x10: [mem 0x.....000-0x.....fff ...]
> > > >
> > > > > Question:
> > > > > 1.  Why dd can cause nvme timeout? Is there more debug ways?
> > >
> > > That means the nvme controller didn't provide a response to a posted
> > > command within the driver's latency tolerance.
> >
> > FYI, with the help of pci bridger's vendor, they find something interes=
ting:
> "From catc log, I saw some memory read pkts sent from SSD card, but its m=
emory
> range is within the memory range of switch down port. So, switch down por=
t will
> replay UR pkt. It seems not normal." and "Why SSD card send out some memo=
ry
> pkts which memory address is within switch down port's memory range. If s=
o,
> switch will response UR pkts". I also don't understand how can this happe=
n?
>=20
> I think we can safely assume you're not attempting peer-to-peer, so that
> behavior as described shouldn't be happening. It sounds like the memory
> windows may be incorrect. The dmesg may help to show if something appears
> wrong.

Dmesg has been pasted in https://marc.info/?l=3Dlinux-pci&m=3D1635222810246=
80&w=3D3

Yes, peer-to-peer cannot happen, nvme is my only ep:

# lspci -i /usr/share/misc/pci.ids
00:00.0 PCI bridge: Cadence Design Systems, Inc. Device 0100
01:00.0 PCI bridge: Pericom Semiconductor Device c016 (rev 07)
02:02.0 PCI bridge: Pericom Semiconductor Device c016 (rev 06)
02:04.0 PCI bridge: Pericom Semiconductor Device c016 (rev 06)
02:06.0 PCI bridge: Pericom Semiconductor Device c016 (rev 06)
05:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD=
 Controller 980

**********************************************************************
This email and attachments contain Ambarella Proprietary and/or Confidentia=
l Information and is intended solely for the use of the individual(s) to wh=
om it is addressed. Any unauthorized review, use, disclosure, distribute, c=
opy, or print is prohibited. If you are not an intended recipient, please c=
ontact the sender by reply email and destroy all copies of the original mes=
sage. Thank you.