Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c11ae803-cea7-8b7f-9992-2f640c90f104@intel.com>
Date:   Tue, 10 May 2022 11:43:16 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
Subject: Re: [mm/page_alloc] f26b3fa046: netperf.Throughput_Mbps -18.0%
 regression
Content-Language: en-US
To:     "ying.huang@intel.com" <ying.huang@intel.com>
CC:     Mel Gorman <mgorman@techsingularity.net>,
        kernel test robot <oliver.sang@intel.com>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        Vlastimil Babka <vbabka@suse.cz>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Jesper Dangaard Brouer <brouer@redhat.com>,
        Michal Hocko <mhocko@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        LKML <linux-kernel@vger.kernel.org>, <lkp@lists.01.org>,
        <lkp@intel.com>, <feng.tang@intel.com>,
        <zhengjun.xing@linux.intel.com>, <fengwei.yin@intel.com>
References: <20220420013526.GB14333@xsang-OptiPlex-9020>
 <YmvMDyx05UoPFtQy@ziqianlu-desk1>
 <bd3db4de223a010d1e06013e93b09879fc9b36a8.camel@intel.com>
 <YnURx04+hE0sQ3v3@ziqianlu-desk1>
 <7d20a9543f69523cfda280e3f5ab17d68db037ab.camel@intel.com>
 <YnXnLuYjmEWdVyBP@ziqianlu-desk1>
 <ae763d63e50d14650c5762103d113934412bef57.camel@intel.com>
 <ba83270a-4f37-7d5a-b37a-0b7a6df5f5b4@intel.com>
 <d13688d1483e9d87ec477292893f2916832b3bdc.camel@intel.com>
From:   Aaron Lu <aaron.lu@intel.com>
In-Reply-To: <d13688d1483e9d87ec477292893f2916832b3bdc.camel@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QUh5Sm9zS24rTkE3UWlpZW5aKzk5NlBxOC9SRmtRTTFRNTk1ZTkxUXJMTzdl?=
 =?utf-8?B?RXhMR3dlOThpZ0VKYXhpQ1EzUk5URXZwK0FwYjhMQy9PUTdiMFI4Z2QraHZs?=
 =?utf-8?B?S1M4aCtvSUFkYjBnTmlwb0h2Sm5Pd1J5aU9WajI4U05DUHpac0JnQWRZVVhQ?=
 =?utf-8?B?UnJCOGF5elhmbG1DNWpDS09DSG1vZ3dTVnhLa0JZQ1NNeWRUOFpKTG5wR3Ir?=
 =?utf-8?B?UHVXbVFMWFNzckZtckt0TVpWcUl1M1ZGSUdvZ0NSQnNDU1VHL1ZqTFFIZnk5?=
 =?utf-8?B?aDJkcU5XUGk5aThYUmxYZ0hYUkttRkl5OThOQk5zVU5vbUszNzJNaFdiRkFx?=
 =?utf-8?B?ZDA1NllrSWRlK09haFQva3BSenZOVEVKcWNlVW1xdHhRZ0hBK3FTQW80ZTZm?=
 =?utf-8?B?T1pXb1QxeXIrUWx5akNZN3hsdHFEUTByMUlwemx1MThzOWRPL2hQamFrT1dQ?=
 =?utf-8?B?czRCSmkzL2tQTDd1SGl5b3RwZDM0bDYvSGR6U21Sb0h5b0dZWXNXYlJVUTVw?=
 =?utf-8?B?Q1JyV3RxK3lEUmkzdXZXdDVybkFnNWRkS2tLRGNHNHBhR2YzdXZ6VkJLaTlF?=
 =?utf-8?B?V0lZcDJFNGVONFpxUW5qaFFuUlpKUE9iK0JSMmxYZXU3Q3R4d1NWWGFlQWlI?=
 =?utf-8?B?UlozVmZpY3F2N3ZRQjN5MmNEcEhYTVdjMmtteWhPSHZDU3VxZUUrbktDRTNG?=
 =?utf-8?B?eUFWYUxGaFIxMWVnWkJXbFFLQnkxT0FJeFRzSVloRnRSTEFibnVFdy8xV05V?=
 =?utf-8?B?bE1SQzRFRWIrOHRVMTA3VmFoVkdRK3FYRThaWlJReW0zOCs3aWxQcStNdCtM?=
 =?utf-8?B?RTFKSWVLdTY5ZkRaTlBabVFpVmJ4Q3hLSWpxOXNrMWU3MGpkemg3NDdlNGNU?=
 =?utf-8?B?QlRuMnVzYUFDOW8xZmVhU2I2djZ5U2VzQm9lbTkwQnlnWE9XS2xHU0FPbnNk?=
 =?utf-8?B?R1lFQ3R1Z2xEdm0zcVNOY3ZjQjlBbTNYVXpKc3FlTnNiN0J3K0JYQnFHeE5I?=
 =?utf-8?B?aVFES092Uy9LQTBYdy9rNDMvb014U3lucWozK0N0NzB0ZVlQTXV1R2V0cU5B?=
 =?utf-8?B?cWlLOWxsa04xMVQ1YWRHaGRHektDR3lIRy9mbVl1aXRseWpIVFR4WFhjMTNL?=
 =?utf-8?B?eDhCWitFNGdJdDhJN3cvQjVIbXZsSWdVUURyZ00vZ0Q5NXJYaXVhSElJaXI1?=
 =?utf-8?B?ZVN5WWZiY000Z3JRbUoyREdBeWdCQW1NWENBV05sdEprVzJIclhkZjFCK1NI?=
 =?utf-8?B?T3hhMGplSElpdWo2cXNkU1VDbGpKTWw1OHVRWjZ4dFFVRXRyWllBanhDSm1n?=
 =?utf-8?B?Sk9kbG5wZWsraCtvNkdIdDVIM2k5WmlKNmlvQ0hjM1M2UlY2S09kdU5Na1ps?=
 =?utf-8?B?aDdYV01vSXhtVk45Y1lUcm55bTZaNzhCbE5GSDRieEVYa0h1c3JOWGxWR0to?=
 =?utf-8?B?dG42a3dGK1V0VWpSSllLRXdvUDhwQXRNZE5HOHVuRW10b0JOZDN5V1Iya2Z0?=
 =?utf-8?B?SkhocWQra0RibTRqckpuN2ZJU2FEZ3hMcnVoOE94eWh4ZzZuc3dia2ZvSE90?=
 =?utf-8?B?NjhKcDVldTJmSW9iT01XMjF2ODJ4aUlhMTdRMWxSNE5vbXNIK3YwTUpHOW9N?=
 =?utf-8?B?ZEdFMDNtaDY1RUdxSmNtL2xJSERTSGdCZEs4TGlmOFBFeW5Fc1pZVHprZDFo?=
 =?utf-8?B?Y0lsZGZoT3ZSRzBrZWZsNjU3Qm5HUUVmWlNTekg5Zm9oRHhEUzVjUnVWRWVN?=
 =?utf-8?B?clg2azdqMEVuQW5PMGRyb2JMN3E1SjBUQy91WHJrRWkyUkx0N00zU1BLb1Bw?=
 =?utf-8?B?OUdyaGRVdFpCbm9MSXJ6dkFxODJkSThKWk1va1AyY2RuUVFJWEJwdS85UjVE?=
 =?utf-8?B?WC9pTmdvdWpwYkthQWRpTFp4UGhVS2lUYThocGlpcGF3WDBmejBVYmFJTXRU?=
 =?utf-8?B?RjdjcGRJdGp6NlAvWDZSaXhwRmJCZjVjTVhPQ0EzUVU2ZnB5R0V4a3U3Rks4?=
 =?utf-8?B?eFk3LzFkdU5kQk00K2xnc0hDbzA0MnpONE8vN3pZSU5XOUNja3BteE1EMGJv?=
 =?utf-8?B?eVp4bjVIWEVuenFjSTAzSmIyNjc2QnpCVFRvTWZ1cTNrYnpicDNiNDA5SkVW?=
 =?utf-8?B?cytwY2paWDBBcGF3MWlHczdpU2YyUmI5cDI0UXliQjc3a2pjcVoraXdNYXpj?=
 =?utf-8?B?QTkrVjNiQmZHaDFwdTRxWm5WY0dMSGU4THVublJ0UWJuQ21veHVJZ0wrSEJR?=
 =?utf-8?B?WXVPRXRBSGg3WGwxa2syUGl2VU1HbHhHclFpMEtzOWFlUG5qQWVkeXdXTjYv?=
 =?utf-8?B?M2JKWWpNVElZRXlsWGJLTjYvVHdtL0NNTnBGZWRSL2plU1BwR3Y5dz09?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2f91bc50-2830-4cb6-c2f5-08da32373cfd
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2022 03:43:25.7324
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ub3d+2ik++YTGVNe3BuG0ZwYvgrLumCtKroNP6MxIMPeSgDwsd3iA6P+AxuwHcnm39ahNYb0TMM+Iats1qMWLA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB3121
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-8.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_HI,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 5/7/2022 3:44 PM, ying.huang@intel.com wrote:
> On Sat, 2022-05-07 at 15:31 +0800, Aaron Lu wrote:

... ...

>>
>> I thought the overhead of changing the cache line from "shared" to
>> "own"/"modify" is pretty cheap.
> 
> This is the read/write pattern of cache ping-pong.  Although it should
> be cheaper than the write/write pattern of cache ping-pong in theory, we
> have gotten sevious regression for that before.
>

Can you point me to the regression report? I would like to take a look,
thanks.

>> Also, this is the same case as the Skylake desktop machine, why it is a
>> gain there but a loss here? 
> 
> I guess the reason is the private cache size.  The size of the private
> L2 cache of SKL server is much larger than that of SKL client (1MB vs.
> 256KB).  So there's much more core-2-core traffic on SKL server.
> 

It could be. The 256KiB L2 in Skylake desktop can only store 8 order-3
pages and that means the allocator side may have a higher chance of
reusing a page that is evicted from the free cpu's L2 cache than the
server machine, whose L2 can store 40 order-3 pages.

I can do more tests using different high for the two machines:
1) high=0, this is the case when page reuse is the extreme. core-2-core
transfer should be the most. This is the behavior of this bisected commit.
2) high=L2_size, this is the case when page reuse is fewer compared to
the above case, core-2-core should still be the majority.
3) high=2 times of L2_size and smaller than llc size, this is the case
when cache reuse is further reduced, and when the page is indeed reused,
it shouldn't cause core-2-core transfer but can benefit from llc.
4) high>llc_size, this is the case when page reuse is the least and when
page is indeed reused, it is likely not in the entire cache hierarchy.
This is the behavior of this bisected commit's parent commit for the
Skylake desktop machine.

I expect case 3) should give us the best performance and 1) or 4) is the
worst for this testcase.

case 4) is difficult to test on the server machine due to the cap of
pcp->high which is affected by the low watermark of the zone. The server
machine has 128 cpus but only 128G memory, which makes the pcp->high
capped at 421, while llc size is 40MiB and that translates to a page
number of 12288.

>> Is it that this "overhead" is much greater
>> in server machine to the extent that it is even better to use a totally
>> cold page than a hot one?
> 
> Yes.  And I think the private cache size matters here.  And after being
> evicted from the private cache (L1/L2), the cache lines of the reused
> pages will go to shared cache (L3), that will help performance.
>

Sounds reasonable.

>> If so, it seems to suggest we should avoid
>> cache reuse in server machine unless the two CPUs happens to be two
>> hyperthreads of the same core.
> 
> Yes.  I think so.