Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Song Liu <songliubraving@fb.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Song Liu <song@kernel.org>, bpf <bpf@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "linux-modules@vger.kernel.org" <linux-modules@vger.kernel.org>,
        "mcgrof@kernel.org" <mcgrof@kernel.org>,
        "rostedt@goodmis.org" <rostedt@goodmis.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "mingo@redhat.com" <mingo@redhat.com>,
        "bp@alien8.de" <bp@alien8.de>,
        "mhiramat@kernel.org" <mhiramat@kernel.org>,
        "naveen.n.rao@linux.ibm.com" <naveen.n.rao@linux.ibm.com>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "anil.s.keshavamurthy@intel.com" <anil.s.keshavamurthy@intel.com>,
        "keescook@chromium.org" <keescook@chromium.org>,
        "hch@infradead.org" <hch@infradead.org>,
        "dave@stgolabs.net" <dave@stgolabs.net>,
        "daniel@iogearbox.net" <daniel@iogearbox.net>,
        Kernel Team <Kernel-team@fb.com>,
        "x86@kernel.org" <x86@kernel.org>,
        "dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
        "rick.p.edgecombe@intel.com" <rick.p.edgecombe@intel.com>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>
Subject: Re: [PATCH bpf-next 1/3] mm/vmalloc: introduce vmalloc_exec which
 allocates RO+X memory
Thread-Topic: [PATCH bpf-next 1/3] mm/vmalloc: introduce vmalloc_exec which
 allocates RO+X memory
Thread-Index: AQHYlpK25UY5kW1ND0S/krM8qG//OK18FxyAgCPUggA=
Date:   Fri, 5 Aug 2022 05:29:51 +0000
Message-ID: <DC201D0D-E51B-42D7-9B3F-194D929A5E9C@fb.com>
References: <20220713071846.3286727-1-song@kernel.org>
 <20220713071846.3286727-2-song@kernel.org>
 <Ys6cWUMHO8XwyYgr@hirez.programming.kicks-ass.net>
In-Reply-To: <Ys6cWUMHO8XwyYgr@hirez.programming.kicks-ass.net>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?lDKduxP+vvJGgaFUpFRvxmFlFvFrW3QKoC/A/BAWI/0WXnKcEBN68k6qqMMe?=
 =?us-ascii?Q?J7lD+vSNBivXAektDn+bufftpcbKaI5oAWBEbO5VUsGNo/KmP63nGbnmtjUn?=
 =?us-ascii?Q?KN6Ikre06lIMjpuJXOyG6LN35wF2QkAFm5McIg6IdsnBIIINDhSaEoZDS1FV?=
 =?us-ascii?Q?M/ytbue6kaH/wkpPmhpV9RiLo2pGb/qy/T2a9k21aYr5lcai1orRLtDpdO1f?=
 =?us-ascii?Q?bT9p74VXtZIt/sk4Nuqq5w3q9ewWg6rEcQIfL+t5o0wSLiB8IatwIvSUYZzW?=
 =?us-ascii?Q?sBFtpaWmshHpkLvhEiq7dIlKvkAwuGe6eZ63YzScewHgjGXDTRY6sq6iBB6P?=
 =?us-ascii?Q?GdrBJxanHg2l7lKEn8e+Oc3HHglsFa4ZJNFHFOXvQmQnZl9Kos3ZN64Y0ZUe?=
 =?us-ascii?Q?9j04MqmdauGubs8feml1uGCUrVLI2NqpcibrwEFbABYAlpHJx/DE+G7lwA4k?=
 =?us-ascii?Q?Sk/YzUjtjZI7GrV5Rbi+5oaFNHbgr0/d9snIS9GE/F3Y60h5hDJ6aICgVk4m?=
 =?us-ascii?Q?hDLI5+maWDO8FpdiENRwvLaCftx4zo/naD5SiVHXnnPeSgJ7h0FR+pZGYNnN?=
 =?us-ascii?Q?DNPfZgRVZJZ0rPS29d+k8vKN4LvIEOU27rBsdugAASgM6ZbbLD996V9pWWJa?=
 =?us-ascii?Q?mgZmAYcNjjFeDMM3OMbqpQCPZNoHOCRu5MP038vSRhbise1xwztL2TcwqQ6h?=
 =?us-ascii?Q?G9QnSiUVJJwGylk8rybdLmTtTky6A2IgNwWXpilvolbO6v+dXExKYKXA6ccL?=
 =?us-ascii?Q?scoWehROgWnn0A0lelC+sr1hp/RmtI+ZwCf2+/vx8TfYGsS5/kNYr/COzyqE?=
 =?us-ascii?Q?QauU/CbOp2UPCRrv72jkIhY2Q6uh9yWCGxVyGeaVxwCn5lfLwvf4O1IXEe3i?=
 =?us-ascii?Q?nTrKZKkIKoKfRXXMLdud9xd2Qw4Dmh9VJk6q8iDnZENmDR2HMOpgVU9v0aUI?=
 =?us-ascii?Q?2nIxq/LmObNqAgpPUZP6enAKF9xuco6ddE6dnI5dBS1tWeCDIezL9vXLimuV?=
 =?us-ascii?Q?rrs/Ts+C95EhLuIb2PqRezEUJl2i2zH0ckMcZ4HrILvvtB9NC4hhpelEWODQ?=
 =?us-ascii?Q?gBj61NVOczy0u1OSOnfPyVzHA7FKTWJw3dlNzVavdQQZ1YLi58uYRpq/OaVm?=
 =?us-ascii?Q?k5rrgfcEgYY+xPtx76FsrVdGRwv+NpUGeSKK4uoI9dSmSGe+4FLcxT9sf86F?=
 =?us-ascii?Q?ISKk73nm1BtBntSrjWV1fr0hizceUBwTlku3B6qz/3qEx8tVJRl0kx1LA4OI?=
 =?us-ascii?Q?bd8JuRwAi06U3JiyCxv91aV0O4+zpNISdB909nYDwk4eSbR6awgaSGEiFz0O?=
 =?us-ascii?Q?mUrm7ZQUfECstuTN99+Dz+g2JE7rZmS7qxHMIMAPlbLfrcYfXC0Or2Yh/hWa?=
 =?us-ascii?Q?6DRnD+CrmroNg2mAkLvwNxN3HSQpEZGbzX1n4kvxP0PDVoxqW8F+sKfvJ8Vi?=
 =?us-ascii?Q?YkC9p8uV5ZHPCLv+1rFISTJBJ+gR5Ea9jVukbVlXInJ8Rerue1QA763pi6B4?=
 =?us-ascii?Q?zSV/DT7efnRIcoZu3Kq2UtOGeZ8Q0emAU7cfZSTb0VR0fkozkNkGem2lOtEQ?=
 =?us-ascii?Q?LwOeCqhHzIWkOu4Ih6cBGQI8QXDMCQ9i+ALKHd6tN/EtYW/8uvnXoM5J9HOe?=
 =?us-ascii?Q?7SlXnlmk1lUUiJ2y2K+AoyI=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <2A718FF731656141B0DA383C714CD85C@namprd15.prod.outlook.com>
MIME-Version: 1.0
X-OriginatorOrg: fb.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SA1PR15MB5109.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5037842a-90ff-4a2a-1b64-08da76a385b0
X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Aug 2022 05:29:51.6786
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: wNpUAYDCfQYNa06Lq7FwsFJaGBFYgbiLe/xCSRJXPysBksSMW2IANEsIwF/KNVQwzA1ZoWnvihw1RQ4ppUFqFQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR15MB4056
X-Proofpoint-ORIG-GUID: iz_Y62ONZSD0FfmCBquEMNwmfNA-Zgc_
X-Proofpoint-GUID: iz_Y62ONZSD0FfmCBquEMNwmfNA-Zgc_
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-08-04_06,2022-08-04_02,2022-06-22_01
X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Peter,

> On Jul 13, 2022, at 3:20 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> 

[...]

> 
> So how about instead we separate them? Then much of the problem goes
> away, you don't need to track these 2M chunks at all.
> 
> Start by adding VM_TOPDOWN_VMAP, which instead of returning the lowest
> (leftmost) vmap_area that fits, picks the higests (rightmost).
> 
> Then add module_alloc_data() that uses VM_TOPDOWN_VMAP and make
> ARCH_WANTS_MODULE_DATA_IN_VMALLOC use that instead of vmalloc (with a
> weak function doing the vmalloc).
> 
> This gets you bottom of module range is RO+X only, top is shattered
> between different !X types.
> 
> Then track the boundary between X and !X and ensure module_alloc_data()
> and module_alloc() never cross over and stay strictly separated.
> 
> Then change all module_alloc() users to expect RO+X memory, instead of
> RW.
> 
> Then make sure any extention of the X range is 2M aligned.
> 
> And presto, *everybody* always uses 2M TLB for text, modules, bpf,
> ftrace, the lot and nobody is tracking chunks.
> 
> Maybe migration can be eased by instead providing module_alloc_text()
> and ARCH_WANTS_MODULE_ALLOC_TEXT.

I finally got some time to look into the code. A few questions:

1. AFAICT, vmap_area tree only works with PAGE_SIZE aligned addresses. 
   For the sharing to be more efficient, I think we need to go with
   smaller granularity. Will this work? Shall we pick a smaller 
   granularity, say 64 bytes? Or shall we go all the way to 1 byte?

2. I think we will need multiple vmap_area's sharing the same vm_struct. 
   Do we need to add refcount to vm_struct?

Thanks,
Song