Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 20 Jun 2022 19:11:45 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Song Liu <song@kernel.org>
CC:     <linux-kernel@vger.kernel.org>, <bpf@vger.kernel.org>,
        <linux-mm@kvack.org>, <ast@kernel.org>, <daniel@iogearbox.net>,
        <peterz@infradead.org>, <mcgrof@kernel.org>,
        <torvalds@linux-foundation.org>, <rick.p.edgecombe@intel.com>,
        <kernel-team@fb.com>
Subject: Re: [PATCH v4 bpf-next 0/8] bpf_prog_pack followup
Message-ID: <YrBV8darrlmUnrHR@ziqianlu-Dell-Optiplex7000>
References: <20220520235758.1858153-1-song@kernel.org>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220520235758.1858153-1-song@kernel.org>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?j4jXP+2ZCzo3K0zbkRAissDcIukQTCdtfbLzj1Qaufik26WDGgIxHvwZKnEn?=
 =?us-ascii?Q?GMAtFAPgVKBAm76xlnpYVUHYmm1pZQkI4zlqBOJLPPlC2JlRsR+eFgGrZQt4?=
 =?us-ascii?Q?vVDHHsNVu51vWCC/J3hVzI6DHiiP6LfDu5TDxxMKxFCf+tDXqadMmpUb6zDQ?=
 =?us-ascii?Q?O4DYwhW2AeRSJz5/D/s+ZOhCEayT6ynKUMq8CbRHGZ8KTeJBTdunZo+ghP/v?=
 =?us-ascii?Q?I68JqkZKurTkKgJiR8sK4jcxCR1rMOl4VZTwXqB53cdTZ6X0PdwWqOQoUiL3?=
 =?us-ascii?Q?Yw5IOjwW+vf1usM2Os6vocove9W+W7k1RGURR3+m0eGFb+pllS6FduCeGgsm?=
 =?us-ascii?Q?Hu4j5m1xo+MOv0PSxuGAWgLY4mJewRR6LlpPW9VaRTq66t0QjoONgapSTOUc?=
 =?us-ascii?Q?28OhwiVIlfF/c43MW62Cr3YCOTXtn+SbIBE35GAvXpzIMtWbGp5o66h0eZ0l?=
 =?us-ascii?Q?vWLepHjnwvinvpiIcy+xg2t0ekOdhZSpiThzDYMzWEQkueeR62rqIOSVjB9m?=
 =?us-ascii?Q?c4DsQ1c4hbwzVShl98hcqkWouyppumAp4YxlSnc4P286OLMvGiP13pvRQ5DT?=
 =?us-ascii?Q?326U9nAR1WyavkDLCRO5lmfKKAT3zMBu6D2z2xr2/SeBa8MV3+2hDCwmm/xz?=
 =?us-ascii?Q?JFRMw+9r06R9vdDjpsefH7V7rCcnZaJVh8qq9s5QYrztlWu9mq8GuOfam4oe?=
 =?us-ascii?Q?Fr6AZIfkMMVFelx4gr3TqhY9hE3QqZeAUcTZE69go4TBIdg/BVr5RbPFl8KX?=
 =?us-ascii?Q?XID2YX5/OaqD8IJby68Smq7nkGLnPzzgge5widwEptvc0bpU4EcxU0OzDGwZ?=
 =?us-ascii?Q?GqrlLUgZ9mUifp1Y5JTh2/1FF5IQR3xL82qSli9gvx8JvXQGxKBvHIVUuFxE?=
 =?us-ascii?Q?y1D1gyK5ZgwTFOkQbQ5Li6tlect5CQEIEyUmSwGFGSeQ0o0G6Njrfo5wb++0?=
 =?us-ascii?Q?aq/DtHcFHnitnpBG9MgLqbe3ZtkLxDOEjUyb22sSTFU9IzSCQcizV+zHD3Mn?=
 =?us-ascii?Q?4rVm2bWQgiNqs7nnLtTryfqgRj69lcJgyQMtqdeUqkLOyYWVw/j+w2jzsU/b?=
 =?us-ascii?Q?Cx4GijZz4H6mDG7LhGRgdv2V3vVGXrKQzg1KKJqMUdnQP3TOgHWE4aC0Jw6b?=
 =?us-ascii?Q?KugMP1I1HtyDlrVcKFbDnOUIygjulxcPXe90X12gQ6LezMRrDtMMs+XZ/isf?=
 =?us-ascii?Q?XI8CNcZeAXlBknCW7/HkHl4y7ZBKtVjbKavLgLCrpChEuEe4u1j3RmTpjUF+?=
 =?us-ascii?Q?8a72FQH+emSZQewFdKCIgsq4akt2XneR++sfehT7AMo7ur1lb8a1TYgyDLRs?=
 =?us-ascii?Q?NoTlc4k1Uusb0jjsUv/nqX4HRSM+tqInGFUhBw6zvqygFZpuBfxHQLHdpar5?=
 =?us-ascii?Q?2lFeUJ5E5Ikm+fIrChTIPRhcFDQVTBCA6hJnGbyVNYkuG3ELfME02Z0nhHmj?=
 =?us-ascii?Q?5LU4irLYShvWT6ALFVvld8TosHOcmCfdUt8Mva3ZOlTpABov5sjnAshWgdFH?=
 =?us-ascii?Q?/pPLB98U+NTkWV4amF0fXCLd/gL6oAc3bT8gFNXtNyCwGFv6GKJJ02G4jlbi?=
 =?us-ascii?Q?qIGqXPk3VPOCJiDurM6rwNVgMt5cZmaemwMRSSGq7V11DmA0oUt3m4O5XLI2?=
 =?us-ascii?Q?zanfNgrzFHnLWAHwNWzS37zjMiv5QLtkPEikEWS5X3Rrt7u9MnjV8EWWfCk4?=
 =?us-ascii?Q?V7P249rUV/vAxPOaA/lDDH7Hy+yXHjk8zAe9igyZDbtly6oC83F5+xMcIUfK?=
 =?us-ascii?Q?5PylOxtKeA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 6536740b-dd0a-4c82-5f77-08da52adb6d1
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2022 11:12:07.8997
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JIXUkuqxSadAnzk/FwkQVd3dCv/jtWhZSKjV754b2esi7nYjLku+0M23stCtL7DQgj2FwS1UOJItNhJAR9T/+Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR11MB1990
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Song,

On Fri, May 20, 2022 at 04:57:50PM -0700, Song Liu wrote:

... ...

> The primary goal of bpf_prog_pack is to reduce iTLB miss rate and reduce
> direct memory mapping fragmentation. This leads to non-trivial performance
> improvements.
>
> For our web service production benchmark, bpf_prog_pack on 4kB pages
> gives 0.5% to 0.7% more throughput than not using bpf_prog_pack.
> bpf_prog_pack on 2MB pages 0.6% to 0.9% more throughput than not using
> bpf_prog_pack. Note that 0.5% is a huge improvement for our fleet. I
> believe this is also significant for other companies with many thousand
> servers.
>

I'm evaluationg performance impact due to direct memory mapping
fragmentation and seeing the above, I wonder: is the performance improve
mostly due to prog pack and hugepage instead of less direct mapping
fragmentation?

I can understand that when progs are packed together, iTLB miss rate will
be reduced and thus, performance can be improved. But I don't see
immediately how direct mapping fragmentation can impact performance since
the bpf code are running from the module alias addresses, not the direct
mapping addresses IIUC?

I appreciate it if you can shed some light on performance impact direct
mapping fragmentation can cause, thanks.