Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <ef901bdb-6117-bad4-2f26-c80df270b86e@nvidia.com>
Date:   Tue, 17 May 2022 11:12:37 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.0
Subject: Re: [PATCH v4] mm: fix is_pinnable_page against on cma page
Content-Language: en-US
To:     Jason Gunthorpe <jgg@ziepe.ca>, Minchan Kim <minchan@kernel.org>
Cc:     "Paul E. McKenney" <paulmck@kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        linux-mm <linux-mm@kvack.org>,
        LKML <linux-kernel@vger.kernel.org>,
        John Dias <joaodias@google.com>,
        David Hildenbrand <david@redhat.com>
References: <ea64b36c-ecc1-74db-dd2e-909e7e507ef8@nvidia.com>
 <0d90390c-3624-4f93-f8bd-fb29e92237d3@nvidia.com>
 <20220512002207.GJ1790663@paulmck-ThinkPad-P17-Gen-1>
 <YnxUTxnCJ6EsmjEi@google.com>
 <f7ff4186-e629-a5da-3f4d-ec629b0c8dd9@nvidia.com>
 <20220512004949.GK1790663@paulmck-ThinkPad-P17-Gen-1>
 <YnxczoehQJ+x6m9Y@google.com>
 <0accce46-fac6-cdfb-db7f-d08396bf9d35@nvidia.com>
 <aafc0954-80df-dceb-03f2-2caea29cbbda@nvidia.com>
 <YnyCqwY4DxcZ/NjM@google.com> <20220517140049.GF63055@ziepe.ca>
From:   John Hubbard <jhubbard@nvidia.com>
In-Reply-To: <20220517140049.GF63055@ziepe.ca>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NGZkRWpQWjJwcXhaTXNrZk0yOGxGU3J4c0JteUwva1puN2ZYNmVlT1ppRmRh?=
 =?utf-8?B?YnViaElnU0JleEZGbDBHdjc5TVhXN2wvMHRQYm1pMElWV1czUHJmeU1UODhR?=
 =?utf-8?B?eWJaZWMyRHhoa1dkdTBNcHNGUlFJbFZrREJmb0h5cXFpa2d2NlErRDhKNG1L?=
 =?utf-8?B?QUF4OGFVUXZLNDNCQ3Y3amd0bE5JSTZpMlZpNWZEbTZiUkxLMXV2WG1VRTA1?=
 =?utf-8?B?WkNHV3hjRWNJam85ZDlDY0Z4YTIxM1JjdStXSk82dXlMMW1aalhwNzl4WWZm?=
 =?utf-8?B?bHAwVTY5aDduVHdkSDRUcWlRQWF2TGdNLzBwQmZadm13ajlrSWtPRWtlakNS?=
 =?utf-8?B?U3JEZWxiTHIxMk0xdmt4SGRicmNkVW5zNzRyR05vUXFndGtWZUxwS0dycGdI?=
 =?utf-8?B?N0ZqVEZ6Yjh4S3Vrb1U4aEVWcmdjLzY1OG5UK284eFBURDg5eHVaM1ZtVE43?=
 =?utf-8?B?SnNHbmZIZS9iSTYwQ0QrVldxbEx2UnJxampJcVlxQ3NsbUhvOUw0ekE0ZnZB?=
 =?utf-8?B?YjhGMy9udDNqUC8rdG1yT1BEalNFMkc4NTFMNWVjTHpSc01qc2VjYjFjMzR0?=
 =?utf-8?B?K1pCc2FqZGlpQ3h6NFNRMkJ1UElzVTk0aDRYTk1kS2NyMDdyZWhyblJHSzY2?=
 =?utf-8?B?aWdUTHRhbXBNZUpVeWNUSUlwam9XeTcxQ2FkL0s5eXVnUURXazA1ZnVIbXN4?=
 =?utf-8?B?eEtacjhZcUdabHhTdjk3b3lhQkEvbUtnL1hhR1lzSEZDNWlhRW5NZ01tMGxJ?=
 =?utf-8?B?dG5teUtJeU1GMTAwTW9kMFl6cVlxbVJ2dVZQM1AyK1ZjcHBhbExURUlPUysy?=
 =?utf-8?B?ZHZPdmZDVC94UmNlRzJMZmZocUo3REh0RklpemU2ZUFSckIzclhEYWczTXZM?=
 =?utf-8?B?YzRBVmRMMnhYR2hCd2FzSGFYTjVMVHl5QkN3SHo1RU5oUndINGJzQW15dXZK?=
 =?utf-8?B?YnhHdEZEdjhhMEtNOWFWZndqUEFDNlI1S3JBY1F4Q0QyMkUyRUwzYkFKYzkw?=
 =?utf-8?B?YWdJWGpmcjFKQ2U3MjkwMVgrK0t0UjdCMEUwZHJwZmxIbERkOXg5SGFHRTRU?=
 =?utf-8?B?b0xpeXhmMVltQWlGUWxLcXkxeDhvbkhrVndQOW1NSjVVb3lGTzlVa0FEeXFC?=
 =?utf-8?B?U1BjSTJ5ckRuVkJQUWF0V2w4UTRYajdkNU91RW5LYndjcm1oY2xaWE95U2pM?=
 =?utf-8?B?SHU1VHhRazFxV1ZhTjQ1RXc3anZRQjU1TjMzZ2ZrZ2VVUGdKbDhWZWdzanF4?=
 =?utf-8?B?S3ZzNG13dWROY2tMU1ZGdGlyRXFudEpXZ0F3VGQ5TktSaEtjcUN2VGRRbGdo?=
 =?utf-8?B?Sm56ZlltK3JYUW1mZE9yQ0E3aWpRV0ZLNnBPV3ZHd05FOTFsbzhEUzgzZkE2?=
 =?utf-8?B?VFJ1bHMvbmhlSmhjM29aZnN0a2tRQnVMRjJxajAzMUtrV0tOQ05GdzRoaW0r?=
 =?utf-8?B?SW5Uck5YMzZjT09YNFhXYytDNEoyY3BnaVBjd0RZMDMxVWpwZU12ZTVJRFZG?=
 =?utf-8?B?NzA4N1doRUZ5dXhESjJYYUdwTnMwSXIvR1o1bDNPemhPWHpzaVBYaXYvaGRm?=
 =?utf-8?B?RVNId3BFYjVmZ3ZVeWpWZWV6OGpGaGxrQWo3R1RVcmxMWkZXSk5KaTNTc1FX?=
 =?utf-8?B?K2V2S3R5MEh3ZGJCNVhXMkRzdVZoZDVFZ0k0aytCdVZ1UitwdVJEa2R4ZTVm?=
 =?utf-8?B?Nm84eHBUb0FZSjNjNHRDUjFLcUgyVWVaUDZSbzcrcncrN05IK043cTc0VUcr?=
 =?utf-8?B?empSdU1EcGd2NEJ4ZERUZkxoVlkycGtuRU4xNFhLS2R1c1VWc3JZVFU5RXVL?=
 =?utf-8?B?Y2NOcHJCTy9kWE9KUjd0YzdiSjR2N1hsdnQyaHZnMnllMmhmLzBiVUhuWXpR?=
 =?utf-8?B?d1M1VEpLVE9sRUZpZWlPZm1WQytWQXVlMmliQnN3dzNlWkhSdE9JWWVpemtY?=
 =?utf-8?B?NXdkeFcrUEl3RjRYZEd2N0VwclJZRm9uNkx6TjdWNW5pVHpKaUNnSFJUdURM?=
 =?utf-8?B?RkVld0p6V0pUQWJoNGk1M0JYM01iOHVYVjQxRUEwRjJJcGI4cDJDcFQvNU1U?=
 =?utf-8?B?ZytQN2NIaHk5dGZEVlJqVDVKaGt5UWlLTkU1OUR2T0U4Y0YwZGVCNzhmMnhm?=
 =?utf-8?B?R3p5SWo4U1A2M2hCWk1HdmlKVVJpU1dqbVMweTc0M0lmNTlSS0MrcUtHM3p6?=
 =?utf-8?B?UHM5dzdYbHFPVFo0cm9pNWwxVkM1ZStJRDdacUpZTnJTaVpLR204eWdwTWQ0?=
 =?utf-8?B?RlRpblBuck96c3lMUHRtcW9ZY1paNXJPdEc3UUZrYmlYKyt2UUZyaFlLblZJ?=
 =?utf-8?B?bVB2by9rY2lEbVZNbWMwUmJFdXMzV3VSajRhMG8rUFE2UXVoOEMxZz09?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 14335d69-9d56-4aea-7972-08da3830d3e7
X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB4130.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 May 2022 18:12:39.2656
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: a7i+LU5vtRv8F5USnCoR8K4LhkKl2ghLGNsxYsT0PqVnTkoKUE7BmTqrTxpCvhEDQgplOSrjt+2fkVhURS6YPw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1399
X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 5/17/22 07:00, Jason Gunthorpe wrote:
>>> It does change the generated code slightly. I don't know if this will
>>> affect performance here or not. But just for completeness, here you go:
>>>
>>> free_one_page() originally has this (just showing the changed parts):
>>>
>>>      mov    0x8(%rdx,%rax,8),%rbx
>>>      and    $0x3f,%ecx
>>>      shr    %cl,%rbx
>>>      and    $0x7,
>>>
>>>
>>> And after applying this diff:
>>>
>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>>> index 0e42038382c1..df1f8e9a294f 100644
>>> +++ b/mm/page_alloc.c
>>> @@ -482,7 +482,7 @@ unsigned long __get_pfnblock_flags_mask(const struct
>>> page *page,
>>>          word_bitidx = bitidx / BITS_PER_LONG;
>>>          bitidx &= (BITS_PER_LONG-1);
>>>
>>> -       word = bitmap[word_bitidx];
>>> +       word = READ_ONCE(bitmap[word_bitidx]);
>>>          return (word >> bitidx) & mask;
>>>   }
>>>
>>>
>>> ...it now does an extra memory dereference:
>>>
>>>      lea    0x8(%rdx,%rax,8),%rax
>>>      and    $0x3f,%ecx
>>>      mov    (%rax),%rbx
>>>      shr    %cl,%rbx
>>>      and    $0x7,%ebx
> 
> Where is the extra memory reference? 'lea' is not a memory reference,
> it is just some maths?

If you compare this to the snippet above, you'll see that there is
an extra mov statement, and that one dereferences a pointer from
%rax:

     mov    (%rax),%rbx

> 
>> Thanks for checking, John.
>>
>> I don't want to have the READ_ONCE in __get_pfnblock_flags_mask
>> atm even though it's an extra memory dereference for specific
>> architecutre and specific compiler unless other callsites *do*
>> need it.
> 
> If a callpath is called under locking or not under locking then I
> would expect to have two call chains clearly marked what their locking
> conditions are. ie __get_pfn_block_flags_mask_unlocked() - and

__get_pfn_block_flags_mask_unlocked() would definitely clarify things,
and allow some clear documentation, good idea.

I haven't checked to see if some code could keep using the normal
__get_pfn_block_flags_mask(), but if it could, that would help with the
problem of keeping the fast path fast.

> obviously clearly document and check what the locking requirements are
> of the locked path.
> 
> IMHO putting a READ_ONCE on something that is not a memory load from
> shared data is nonsense - if a simple == has a stability risk then so
> does the '(word >> bitidx) & mask'.
> 
> Jason

Doing something like this:

     int __x = y();
     int x = READ_ONCE(__x);

is just awful! I agree. Really, y() should handle any barriers, because
otherwise it really does look pointless, and people reading the code
need something that is clearer. My first reaction was that this was
pointless and wrong, and it turns out that that's only about 80% true:
as long as LTO-of-the-future doesn't arrive, and as long as no one
refactors y() to be inline.


thanks,
-- 
John Hubbard
NVIDIA