Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Date:   Tue, 19 Sep 2023 11:57:34 -0400
From:   Gregory Price <gregory.price@memverge.com>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     Gregory Price <gourry.memverge@gmail.com>,
        linux-mm@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-arch@vger.kernel.org, linux-api@vger.kernel.org,
        linux-cxl@vger.kernel.org, luto@kernel.org, mingo@redhat.com,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        arnd@arndb.de, akpm@linux-foundation.org, x86@kernel.org
Subject: Re: [RFC PATCH 3/3] mm/migrate: Create move_phys_pages syscall
Message-ID: <ZQnE7i7tYOPnrL3w@memverge.com>
References: <20230907075453.350554-1-gregory.price@memverge.com>
 <20230907075453.350554-4-gregory.price@memverge.com>
 <877conxbhw.ffs@tglx>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <877conxbhw.ffs@tglx>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?JPwpC6lstxOOSf32lyotMa7LTvjyJb+igCkO6HXVkjEoNi6kJ54vu/WwJEEl?=
 =?us-ascii?Q?R47lKA8BDSe2JqoGx3mTULW35qdy19Svcp7H7i37r6Nm0VVUN/32b6rYg2x/?=
 =?us-ascii?Q?snXM0pLNxE18Jp5+hc9+WYXA8YtdDNq1BbPEx32D38X/0KTlhWuX28skq8cb?=
 =?us-ascii?Q?n+P8WEfCSgOTnWth+u7egqVLWCN6pqgvgXK73O51x7OC3Fa86S+EnoSDOG3O?=
 =?us-ascii?Q?rTMc86P3FatV7IdxYZ4OsybrwZMNLwWlcWjMKzEpCEN8FlENIZvFyol9Ok95?=
 =?us-ascii?Q?0Vm5e1jCrPJZ3MAwiW06CDxaPjl6nyXhtSD/3BUDKnN7zBfgVsKIz5qtj4yU?=
 =?us-ascii?Q?BaTD0vN5WMvukP/iEcBI+X2rmyr6Yy9SLtQ/5KNq7+QKUkt2jY0amy8VmsMo?=
 =?us-ascii?Q?t8KAxHnTP7Y7stJ6Kcc+EaBZeDbFwnqXUVPjEP0AlK0xnODEWaCMomDg4eA0?=
 =?us-ascii?Q?eDn3egKVFEc++PnnhXiRE8/wvRvCk8gLCQ1EEXLP3tzAsAAmXvkO1TJN2heN?=
 =?us-ascii?Q?TeEir6bulze2q4fLPXw4fMTBTFe5X2DEQ6gpfDsGm2rxTJwIqON+Fs9jFw+K?=
 =?us-ascii?Q?p+eUIT1a04A5f6NLOVol6qpVL6W7pUfA32Tr1iClyr98r1KSE497Q4m4AruF?=
 =?us-ascii?Q?rDDg9jGvW+Ymvc3EzgD7LA1XNDbMI1xk3TlkWueR9/cTHDndk7MvXe2PdiYD?=
 =?us-ascii?Q?d5Fy8DskNj3v4J3E9RFoALbeEXpTmn1JmLODZRy2Zos2rwGH55E1dHmRDd2D?=
 =?us-ascii?Q?18lUR+HYG/EmuqEj5ROzhkvm0uq1ceCvjb3utl5DgH7XVFrx0lB/8tfpUZag?=
 =?us-ascii?Q?jsm09RBVu05FNSH24vtol6a3W4BKHhiTPiYYH3C3fgx7Hothtzp+XrjFMbq0?=
 =?us-ascii?Q?OiMEz4ssxZzAS/+QnFtiqtKqh2PMHU7b806QgFUh4WgCKTW/27Aq9PLNiss5?=
 =?us-ascii?Q?dmfhEvLdPudLwqppfBBJ62yTj39flrERMhd72IsSL81TXdFjm8MU2v1UWv7R?=
 =?us-ascii?Q?CMfDkq19XUyGNJhbZBsNULzPfO8mj1rXIL/sDohA4fhfFH7Z3sJ1+bXu5yfX?=
 =?us-ascii?Q?Q32FAYXhTK0P4unpGGBfcvu8BPGMNP8kdXB48s7JT24K5zI9UiEjfN62nmm1?=
 =?us-ascii?Q?xHFYRmXGelHHnvBGY9h6/zwZ2ZmkBWfSuCgOngrN5IIMRohOLmO1gczPMoax?=
 =?us-ascii?Q?uef+IY1po4Z001NrWJFMmYN9yaFwYSyVlPDPiHfM5CAzks7el9oeAvPEv4sy?=
 =?us-ascii?Q?pZlWOzUlhAjuJaF57G88XgSXSwBwA8oaXxpsScQByjJMsSaIfE8FLmn5kDSY?=
 =?us-ascii?Q?whcqUB7RShcTJomLLo03QXmDBRZ28sha69jDepGQt46glfUPeMpgmK3bXW8v?=
 =?us-ascii?Q?GzMaYblwbn01LW9h53x66BMHlL+oj57GyE91ZnE8zjD6gbBEXN/RaunX+fLp?=
 =?us-ascii?Q?9yE5V+5sAnFNe03HXL6VTF6VpaoEdxtjrjmyawrrazqHXyLa1CPAp78O9SJE?=
 =?us-ascii?Q?tBDl+SfPc7ilGTKJA5qk+nzC2qd/FsMKHyBmESC7UJBGiNK5SZLLBxDzN34i?=
 =?us-ascii?Q?fbjJMFSphMrG9CGRqINNml7kTv1YLwTfHP0bVuxpEc8HKzpx/n88uqA5XwYd?=
 =?us-ascii?Q?2w=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e1712260-8337-4309-d44e-08dbb9292838
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Sep 2023 15:57:42.5255
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Di0FeeRc/UupUgcBKZTf0IZ4y40bHaxujCDI4vYJI0onnB760mWQta69CtHMqeulE3ACzBWFr6dyBsvt8NuFWhdoU35t0ewJ+kjfGy9/k7k=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR17MB6522
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 19 Sep 2023 08:58:29 -0700 (PDT)

On Tue, Sep 19, 2023 at 02:17:15AM +0200, Thomas Gleixner wrote:
> On Thu, Sep 07 2023 at 03:54, Gregory Price wrote:
> > Similar to the move_pages system call, instead of taking a pid and
> > list of virtual addresses, this system call takes a list of physical
> > addresses.
> 
> Silly question. Where are these physical addresses coming from?
> 
> In my naive understanding user space deals with virtual addresses for a
> reason.
> 
> Exposing access to physical addresses is definitely helpful to write
> more powerful exploits, so what are the restriction applied to this?
> 

There are a variety of sources from which userspace can acquire physical
addresses, most require SYS_CAP_ADMIN.

I should probably add this list explicitly to the RFC for clarification:

1) /proc/pid/pagemap: can be used to do page table translation.
     This is only really useful for testing, and it requires
     CAP_SYS_ADMIN.

2) X86:  IBS (AMD) and PEBS (Intel) can be configured to return
     physical, virtual, or both physical and virtual adressing.
     This requires CAP_SYS_ADMIN.

3) zoneinfo:  /proc/zoneinfo exposes the start PFN of zones. Not largely
     useful in this context, but noteably it does expose a PFN does not
     require privilege.

4) /sys/kernel/mm/page_idle:  A way to query whether a PFN is idle.
     One tiering strategy can be to use idle information to move
     less-used pages to "lower tiers" of memory.

     With page_idle, you can query page_idle migrate based on PFN,
     without the need to know which process it belongs to.  Obviously
     migrations must still respect memcg restrictions, which is why
     iterating through each mapping VMA is required.

     https://docs.kernel.org/admin-guide/mm/idle_page_tracking.html

5) CXL (Proposed): a CXL memory device on the PCIe bus may provide
     hot/cold information about its memory.  If a heatmap is provided,
     for example, it would have to be a device address (0-based) or a
     physical address (some base defined by the kernel and programmed
     into device decoders such that it can convert them to 0-based).

     This is presently being proposed but has not been agreed upon yet.


> > +	if (flags & ~(MPOL_MF_MOVE|MPOL_MF_MOVE_ALL))
> > +		return -EINVAL;
> > +
> > +	if ((flags & MPOL_MF_MOVE_ALL) && !capable(CAP_SYS_NICE))
> > +		return -EPERM;
> 
> According to this logic here MPOL_MF_MOVE is unrestricted, right?
> 
> But how is an unpriviledged process knowing which physical address the
> pages have? Confused....
> 

Someone else pointed this out as well, I should have a SYS_CAP_ADMIN
check here that i neglected to add, I have a v2 of the RFC with a test
and fixes, and I am working on a sample man page.

This entire syscall should require SYS_CAP_ADMIN, though there may be an
argument for CAP_SYS_NICE.  I personally am of the opinion that ADMIN is
the right choice, because you're right that operations on physical
addresses are a major security issue.

> > +	/* All tasks mapping each page is checked in phys_page_migratable */
> > +	nodes_setall(target_nodes);
> 
> How is the comment related to nodes_setall() and why is nodes_setall()
> unconditional when target_nodes is only used in the @nodes != NULL case?
> 

Yeah this comment is confusing.  This is a bit of a wart that comes from
trying to re-use the existing do_pages_move to ensure correctness, and
I really should have added this information directly in the comments.

The relevant code is here:

static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,
                         unsigned long nr_pages,
                         const void __user * __user *pages,
                         const int __user *nodes,
                         int __user *status, int flags)
{
...
	err = -EACCES;
	if (!node_isset(node, task_nodes))
		goto out_flush;
...
	if (mm)
		err = add_virt_page_for_migration(mm, p, current_node, &pagelist,
					     flags & MPOL_MF_MOVE_ALL);
	else
		err = add_phys_page_for_migration(p, current_node, &pagelist,
				flags & MPOL_MF_MOVE_ALL);


do_pages_move was originally written to check that the memcg contained
the node as a valid destination for the entire request. This nodemask
is aquired from find_mm_struct in a call to cpuset_mems_allowed(task).

e.g. if memcg.valid_nodes=0,1 and you attempt a move_pages(task, 2), then
the entire syscall will error out with -EACCES.

The first chunk above checks this condition.

The second chunk (add_virt/phys...), these calls check that individual
pages are actually migratable, and if so removes them from the LRU and
places them onto the list of pages to migrate in-bulk at a later time.

In the physical addressing path, there is no pid/task provided by the
user, because physical addresses are not associated directly with task.
We must look up each VMA, and each owner of that VMA, and validate
the intersection of the allowed nodes defined by that set contain the
requested node.

This is implemented here:

static int add_phys_page_for_migration(const void __user *p, int node,
                                       struct list_head *pagelist,
                                       bool migrate_all)
...
        pfn = ((unsigned long)p) >> PAGE_SHIFT;
        page = pfn_to_online_page(pfn);
        if (!page || PageTail(page))
                return -ENOENT;

        folio = phys_migrate_get_folio(page);
        if (folio) {
                rmap_walk(folio, &rwc); <---- per-vma walk
                folio_put(folio);
        }


static bool phys_page_migratable(struct folio *folio,
                                 struct vm_area_struct *vma,
                                 unsigned long address,
                                 void *arg)
{
        struct rmap_page_ctxt *ctxt = (struct rmap_page_ctxt *)arg;
        struct task_struct *owner = vma->vm_mm->owner;
        nodemask_t task_nodes = cpuset_mems_allowed(owner);
... the actual node check here ...
        ctxt->node_allowed &= node_isset(ctxt->node, task_nodes);


I will add some better comments that lay this out.

~Gregory