Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 26 Oct 2022 14:49:55 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Pankaj Gupta <pankaj.gupta.linux@gmail.com>,
        Michael Sammler <sammler@google.com>
CC:     Dan Williams <dan.j.williams@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Ira Weiny <ira.weiny@intel.com>,
        Pasha Tatashin <pasha.tatashin@soleen.com>,
        <nvdimm@lists.linux.dev>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v1] virtio_pmem: populate numa information
Message-ID: <6359ab83d6e4d_4da3294d0@dwillia2-xfh.jf.intel.com.notmuch>
References: <20221017171118.1588820-1-sammler@google.com>
 <CAM9Jb+ggq5L9XZZHhfA98XDO+P=8y-mT+ct0JFAtXRbsCuORsA@mail.gmail.com>
 <CAFPP518-gU1M1XcHMHgpx=ZPPkSyjPmfOK6D+wM6t6vM6Ve6XQ@mail.gmail.com>
 <CAM9Jb+hk0ZRtXnF+WVj0LiRiO7uH-jDydJrnUQ_57yTEcs--Dw@mail.gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <CAM9Jb+hk0ZRtXnF+WVj0LiRiO7uH-jDydJrnUQ_57yTEcs--Dw@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?9hdz3jN44KQDz55zJKGgWkZKHo6q3uUNTv4Yivyn0yh8mRUQZxOt6brLtnsi?=
 =?us-ascii?Q?nD9T3DvrcryFgjx/vygYRyXe4lrMXBY9VP7pKpSXn7WtDEbGfPMY/ey0kRQG?=
 =?us-ascii?Q?l6nYSTeC3HXp45NTLHjDiTkyrok7aYTZG0YhYVKuDLqoNU+3jPBknb8H/vXW?=
 =?us-ascii?Q?5URrhZdxIkC5mniJCn3OmuUVc+0p7qnmpRAffSgR7lthU3kKx6SslDMpMKk9?=
 =?us-ascii?Q?0eHfl0nA43YVm83wMTaGOGjJvMqe2M3wncp89P9LoTuMpazQbYhAmnypZwN0?=
 =?us-ascii?Q?EuIK6vDV3Df4U65ts33xTLYWlh/Wa44pPafj/jyI8CJw4GM//GVV+SUuTrEp?=
 =?us-ascii?Q?i/Puc9zExWPRmUEqh3Ieq71ekXBsZHcG6RncrmKw1qeVW6eIgZ1E30+m1J/h?=
 =?us-ascii?Q?IaNXaF1XQqOonQs3Xnd8RVoqSRHvYxBCQpNAYWY/2RoPOQuaoI5C1leFaghM?=
 =?us-ascii?Q?447M8jJOiZmvKyeDbPrhTUCf6zrLJiqiwj4DBZuZQWmBwXgnVGWWC7keW1zd?=
 =?us-ascii?Q?J6MM8Dvk3cKKkeypwLJIXv1IUhMMJdKvs19QwyweC4TL9MQ0k8Y9k+GgxOn2?=
 =?us-ascii?Q?nOvDyarBkSyuiQIji9vYVfcMEPFaCHVtDEBZ/1ekaHUq4R1C56lBdx9wswih?=
 =?us-ascii?Q?+EbT+GZwRT+SmRpZltX4smUz4+EmKJ/hnpcfaAz6rIoO41Rc7BnCF36DENHq?=
 =?us-ascii?Q?ii4fqMOT9llXkANNi5Vcd+Tyac8CwkSA4xvEhLf3txwGaUvZgCaxiBGWC7eG?=
 =?us-ascii?Q?5zbeB5L08eQ9dekJNFavMhQGIkHvQ3EkZ3+hoK/el1VS+ahGE95Iyh7aDCDA?=
 =?us-ascii?Q?CtqGuPqJVh+BuriDLBXZ+I5jlalTLE5+f60v+Pgd2HR82DNWgFuME3ZvSEIq?=
 =?us-ascii?Q?JLq1UvPpi3h2ua1lTmPIaKveiw4sRhQzbV4pC7m+CJdjdzGssT1/U6Ws0mPk?=
 =?us-ascii?Q?cma2YI1FKVCHmnoyKMPqqMzPjw01vfnpupYnZ/VTtfZBZ2UibsEJDkrg/bgE?=
 =?us-ascii?Q?chCib19RPesJMZtbUsbPkmrV7uGE3xzc4QJEgg83mW900vLh8njo1dVo3WdT?=
 =?us-ascii?Q?yc2sC7SD1j1rgBWj0U0+yMEYE128CRTC2HVdFeNMW0BOcttxyvZVLXbYFQQR?=
 =?us-ascii?Q?e4c5Gh/WkYydJP/Ojzqm13GpisUF6AqhIHiW2v5vXN+JzhaE6lHUcvo7nYXD?=
 =?us-ascii?Q?P5zB+IbBb329whmDuwGvGeQFsnJ02lh4+FowMEngPu74r0tU3jS3+kk8NQSE?=
 =?us-ascii?Q?xYwXu8y6dcOX3aVhPm6ERAsLy/pmK+IE38DQMk7YoluAKsHeWnl7ttEUyD6E?=
 =?us-ascii?Q?AAdNWmCvFAu4qie1z6LKhZfo0Pc410DVZrgtClus/BAG45VrpxYKtJWaNFhA?=
 =?us-ascii?Q?x9Ul/QJDgtNOPhR7TiXwqphWkVmvpSTbQ1e4v6h3BO2zMFG1UzI9D90KEoqF?=
 =?us-ascii?Q?/CfLrcSDewHEQGBX/v5huSnxytwzSiJhKCmI4/v7Sf0bTI999K6B0Qk0Vmpj?=
 =?us-ascii?Q?GLMfoyODbZ/5ODSHooAbP3aSmtSol5tL0qBUu3exx38iwxt8c+ddEHbZrv+B?=
 =?us-ascii?Q?ag2uIlWHpN/LQ8caQ5ettwyr2LNBqfCWeeuWRMpQSHxW+nNyHCHiQgbI5VgV?=
 =?us-ascii?Q?QQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 40851ac4-c396-4eec-1a97-08dab79c06c4
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2022 21:49:58.4048
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jf8hKAjxtGBjHTcZ4si1Dey/b+19ETQ4HuaupDrIP7R5NzbAp+4t5VbNSiVSy66RLzviEmUFUl3ifYa4yYUYOpNQOYQNN81zipSVtIy97rQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4966
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Pankaj Gupta wrote:
> > > > Compute the numa information for a virtio_pmem device from the memory
> > > > range of the device. Previously, the target_node was always 0 since
> > > > the ndr_desc.target_node field was never explicitly set. The code for
> > > > computing the numa node is taken from cxl_pmem_region_probe in
> > > > drivers/cxl/pmem.c.
> > > >
> > > > Signed-off-by: Michael Sammler <sammler@google.com>
> > > > ---
> > > >  drivers/nvdimm/virtio_pmem.c | 11 +++++++++--
> > > >  1 file changed, 9 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/drivers/nvdimm/virtio_pmem.c b/drivers/nvdimm/virtio_pmem.c
> > > > index 20da455d2ef6..a92eb172f0e7 100644
> > > > --- a/drivers/nvdimm/virtio_pmem.c
> > > > +++ b/drivers/nvdimm/virtio_pmem.c
> > > > @@ -32,7 +32,6 @@ static int init_vq(struct virtio_pmem *vpmem)
> > > >  static int virtio_pmem_probe(struct virtio_device *vdev)
> > > >  {
> > > >         struct nd_region_desc ndr_desc = {};
> > > > -       int nid = dev_to_node(&vdev->dev);
> > > >         struct nd_region *nd_region;
> > > >         struct virtio_pmem *vpmem;
> > > >         struct resource res;
> > > > @@ -79,7 +78,15 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
> > > >         dev_set_drvdata(&vdev->dev, vpmem->nvdimm_bus);
> > > >
> > > >         ndr_desc.res = &res;
> > > > -       ndr_desc.numa_node = nid;
> > > > +
> > > > +       ndr_desc.numa_node = memory_add_physaddr_to_nid(res.start);
> > > > +       ndr_desc.target_node = phys_to_target_node(res.start);
> > > > +       if (ndr_desc.target_node == NUMA_NO_NODE) {
> > > > +               ndr_desc.target_node = ndr_desc.numa_node;
> > > > +               dev_dbg(&vdev->dev, "changing target node from %d to %d",
> > > > +                       NUMA_NO_NODE, ndr_desc.target_node);
> > > > +       }
> > >
> > > As this memory later gets hotplugged using "devm_memremap_pages". I don't
> > > see if 'target_node' is used for fsdax case?
> > >
> > > It seems to me "target_node" is used mainly for volatile range above
> > > persistent memory ( e.g kmem driver?).
> > >
> > I am not sure if 'target_node' is used in the fsdax case, but it is
> > indeed used by the devdax/kmem driver when hotplugging the memory (see
> > 'dev_dax_kmem_probe' and '__dax_pmem_probe').
> 
> Yes, but not currently for FS_DAX iiuc.

The target_node is only used by the dax_kmem driver. In the FSDAX case
the memory (persistent or otherwise) is mapped behind a block-device.
That block-device has affinity to a CPU initiator, but that memory does
not itself have any NUMA affinity or identity as a target.

So:

block-device NUMA node == closest CPU initiator node to the device

dax-device target node == memory only NUMA node target, after onlining