Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20221222072603.1175248-1-korantwork@gmail.com>
 <3d1834d9-7905-1225-741a-f298dd5b8a8e@linux.dev> <Y6TSgGdCTvkwPiVg@kbusch-mbp.dhcp.thefacebook.com>
 <CAEm4hYUWf+Fx3FV7vNTc8+O9NSb0iQp75MTC6gra6XapXK=cxw@mail.gmail.com>
 <d14ac29d-027a-08a7-c5c8-848a6920d4a2@linux.dev> <CAEm4hYXncuvL-Gk1aEZExrvkbx=N1aiOQNeNjFdB4443EbKNBA@mail.gmail.com>
 <f05ee82a-4532-b12b-490f-904b946ff7b0@linux.dev>
In-Reply-To: <f05ee82a-4532-b12b-490f-904b946ff7b0@linux.dev>
From:   Xinghui Li <korantwork@gmail.com>
Date:   Tue, 10 Jan 2023 20:28:54 +0800
Message-ID: <CAEm4hYXk1RuKEw41VukH2iGTo_9GmZjUfrESWK5vFtpFA_O_4A@mail.gmail.com>
Subject: Re: [PATCH] PCI: vmd: Do not disable MSI-X remapping in VMD 28C0 controller
To:     Jonathan Derrick <jonathan.derrick@linux.dev>
Cc:     Keith Busch <kbusch@kernel.org>, nirmal.patel@linux.intel.com,
        lpieralisi@kernel.org, linux-pci@vger.kernel.org,
        linux-kernel@vger.kernel.org, Xinghui Li <korantli@tencent.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

Jonathan Derrick <jonathan.derrick@linux.dev> =E4=BA=8E2023=E5=B9=B41=E6=9C=
=8810=E6=97=A5=E5=91=A8=E4=BA=8C 05:00=E5=86=99=E9=81=93=EF=BC=9A
>
> As the bypass mode seems to affect performance greatly depending on the s=
pecific configuration,
> it may make sense to use a moduleparam to control it
>
We found that each pcie port can mount four drives. If we only test 2
or 1 dirve of one pcie port,
the performance of the drive performance will be normal. Also, we
observed the interruptions in different modes.
bypass:
.....
2022-12-28-11-39-14: 1224       181665   IR-PCI-MSI 201850948-edge      nvm=
e0q68
2022-12-28-11-39-14: 1179       180115   IR-PCI-MSI 201850945-edge      nvm=
e0q65
2022-12-28-11-39-14:  RES        26743   Rescheduling interrupts
2022-12-28-11-39-17: irqtop - IRQ : 3029, TOTAL : 2100315228, CPU :
192, ACTIVE CPU : 192
disable:
......
2022-12-28-12-05-56: 1714       169797   IR-PCI-MSI 14155850-edge      nvme=
1q74
2022-12-28-12-05-56: 1701       168753   IR-PCI-MSI 14155849-edge      nvme=
1q73
2022-12-28-12-05-56:  LOC       163697   Local timer interrupts
2022-12-28-12-05-56:  TLB         5465   TLB shootdowns
2022-12-28-12-06-00: irqtop - IRQ : 3029, TOTAL : 2179022106, CPU :
192, ACTIVE CPU : 192
remapping:
022-12-28-11-25-38:  283       325568   IR-PCI-MSI 24651790-edge      vmd3
2022-12-28-11-25-38:  140       267899   IR-PCI-MSI 13117447-edge      vmd1
2022-12-28-11-25-38:  183       265978   IR-PCI-MSI 13117490-edge      vmd1
......
2022-12-28-11-25-42: irqtop - IRQ : 2109, TOTAL : 2377172002, CPU :
192, ACTIVE CPU : 192

From the result it is not difficult to find, in remapping mode the
interruptions come from vmd.
While in other modes, interrupts come from nvme devices. Besides, we
found the port mounting
4 dirves total interruptions is much fewer than the port mounting 2 or 1 dr=
ive.
NVME 8 and 9 mount in one port, other port mount 4 dirves.

2022-12-28-11-39-14: 2582       494635   IR-PCI-MSI 470810698-edge      nvm=
e9q74
2022-12-28-11-39-14: 2579       489972   IR-PCI-MSI 470810697-edge      nvm=
e9q73
2022-12-28-11-39-14: 2573       480024   IR-PCI-MSI 470810695-edge      nvm=
e9q71
2022-12-28-11-39-14: 2544       312967   IR-PCI-MSI 470286401-edge      nvm=
e8q65
2022-12-28-11-39-14: 2556       312229   IR-PCI-MSI 470286405-edge      nvm=
e8q69
2022-12-28-11-39-14: 2547       310013   IR-PCI-MSI 470286402-edge      nvm=
e8q66
2022-12-28-11-39-14: 2550       308993   IR-PCI-MSI 470286403-edge      nvm=
e8q67
2022-12-28-11-39-14: 2559       308794   IR-PCI-MSI 470286406-edge      nvm=
e8q70
......
2022-12-28-11-39-14: 1296       185773   IR-PCI-MSI 202375243-edge      nvm=
e1q75
2022-12-28-11-39-14: 1209       185646   IR-PCI-MSI 201850947-edge      nvm=
e0q67
2022-12-28-11-39-14: 1831       184151   IR-PCI-MSI 203423828-edge      nvm=
e3q84
2022-12-28-11-39-14: 1254       182313   IR-PCI-MSI 201850950-edge      nvm=
e0q70
2022-12-28-11-39-14: 1224       181665   IR-PCI-MSI 201850948-edge      nvm=
e0q68
2022-12-28-11-39-14: 1179       180115   IR-PCI-MSI 201850945-edge      nvm=
e0q65
> I'd vote for it being in VMD mode (non-bypass) by default.
I speculate that the vmd controller equalizes the interrupt load and
acts like a buffer,
which improves the performance of nvme. I am not sure about my
analysis. So, I'd like
to discuss it with the community.