Message-ID: <57939078-f292-305a-b16d-6ab497ef2edd@oracle.com>
Date:   Wed, 12 Jan 2022 10:25:42 -0600
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.4.0
Subject: Re: [PATCH v4 0/3] Handle warning of allocation failure on DMA zone
 w/o managed pages
Content-Language: en-US
To:     Baoquan He <bhe@redhat.com>, linux-kernel@vger.kernel.org
Cc:     linux-mm@kvack.org, akpm@linux-foundation.org, hch@lst.de,
        cl@linux.com, kexec@lists.infradead.org, 42.hyeyoo@gmail.com,
        penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com,
        vbabka@suse.cz, David.Laight@ACULAB.COM, david@redhat.com,
        x86@kernel.org, bp@alien8.de
References: <20211223094435.248523-1-bhe@redhat.com>
From:   john.p.donnelly@oracle.com
In-Reply-To: <20211223094435.248523-1-bhe@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YWt0MXFSOXNGd3BQRkRrVS9rN3ovaVlyOVV0alNhcDdta0JEV0dmRXlkZWR5?=
 =?utf-8?B?dzBPZlZaenpVeUViR1hENzlueFpHYy9MRDgxZTcvWkZkeVlJVUFEWWhKV1F4?=
 =?utf-8?B?anBUVVRUdGlhUUtORzh2dGIzdFRMRTNTZlQ3eUp0cUxadUlLM3FkdDd0MS8y?=
 =?utf-8?B?WDZNT25YT0NhVjc1WWNLVU9GU2Y4Zk9reFBzL0VIanMwS0VPTnFTSTVwNm9L?=
 =?utf-8?B?WGZPOGVTaFJCRTd1OVhZSTJGMTlwWmFKb0xBaUphWFdQU1liZ3ZEV2ZhdzJq?=
 =?utf-8?B?UFM5OWxneFgxK0xzQ3pqeE5HcXd0c3ZzbWd3aWY3MmdqbFNXbThwZXJZSHNG?=
 =?utf-8?B?NmZrUWZDNGdQdDlXUHVhYjlsME5GbHJCQmlsTGpGM2xaa05LWTcvbW5nMTBE?=
 =?utf-8?B?NlJ6NWRqOXdaU0J2aU4vaExyV212S09Ld3h0d2xzYW83ZHc5dXZXKzlZS21F?=
 =?utf-8?B?c3Z2d21KYUJIOVVvekxELy9GeW5ZUkNKczgwR2doMWlJMHRYTUN2THV0K3p4?=
 =?utf-8?B?WE9GUnNFM1NRMCt6RkoxYWczdUpoY1FPN3JFWmM1TnE2b051WUVwWkpsR3RQ?=
 =?utf-8?B?Wk5obWZkdjNpZnhucnV1cGxXb1J3eENZQlJiM29MRzlJMzJHTDJVbzBDMEtu?=
 =?utf-8?B?M0xwbmNIMWp6R3BsZEgvR1BSZlpNY1hKL2IzZFVVbkR2VERGa2tsL1owaFVW?=
 =?utf-8?B?V0lOcEVyWnZUMnVacTRFZFBZQi9DRkxRN2k4TEJJM0ZVTGZqZXEyVHpuT3li?=
 =?utf-8?B?WUg3TXB6Y05xc0xPenpSM0djNkp4KzByKy9Ba05wT3plSEw0MEtyRUtpRjg2?=
 =?utf-8?B?eFJ5MDNiREtUTnVrK0kyNjNNbDZyK1VzcFRncVpQLytvSDJ3YjJRZm0xN0p5?=
 =?utf-8?B?WlpIbkRiWEdFYVNOVGRsc1lXNnZ4S0YwdXdaakVqWlU1RmEvdVZoS1loeTFa?=
 =?utf-8?B?YVdoS1dxQ3JFaDhBazhvUUFRZURFZmNHM1hyRjJmV3NFRWo4NHZwdFFlNXZN?=
 =?utf-8?B?T1hkNEc4U0VDZER6SkllNlZmRDBLWVVPMFZkVXZ6djJLY3FtYzJBK2duVkFz?=
 =?utf-8?B?WG1VVTd1YXZTZGtCb0dVbEhlbDNxMm9ZQU9jb3c3empJWml3YUJTZWJYall0?=
 =?utf-8?B?d0ZKNXZlTCtMZldUUmRZWndWS2JMQjhpVldHdml4M01YeEtMcGJiNTFTVjhY?=
 =?utf-8?B?OTAvejJ3dWlvbVhadkJoSEVxVzA3b3MvMHdtQWhndXRKeTVEcXN6dUM1RG1Q?=
 =?utf-8?B?UTR3eSsvRVBBNkRuZHJpYzVmVzlCQjBydVdQbFZBY3gyYzcyMmRlMUJDT1V1?=
 =?utf-8?B?S2RoMmljMXpVVWhqRTd1M2dydjZlUGNBVmhDZmg3bDJHL0Q5Q05DVTFrUHBM?=
 =?utf-8?B?V1pnWDdVeU1zMFVBOEJRQ09XbTJRQkltR05DSjhGdnhzTStLOUdTU1c2bTI0?=
 =?utf-8?B?bng2ZEx1N2NkWG5Wa3pXVGdNc3pJci9wV3NmTXgxM01kVkFXbzFkcWxaeldq?=
 =?utf-8?B?azRPeWNXNEJDK3h5MDR4NlRaOCtoRzRqaG5CZlg0SEdHWGd4TUNTRGQycDlm?=
 =?utf-8?B?T3VXNmhoT1ZheGJTcTU0dXBCVGhVZUZNWDNoS3ZpMVlhQ1I0MWxpMlpNQlpJ?=
 =?utf-8?B?RzM4ZXQvNmxiY3BsYmtpV0RmRG1QUGlnNStLSG8zTk42UUl5dm9HZllLZW9m?=
 =?utf-8?B?MlQ3aWozeXRqckpzUnBYMVBMNHUrQVN2aGcrbW9zb0hFVGptS1c1WkxpdEdK?=
 =?utf-8?B?V200aDNlbzVMaFRKbXoyMGZ2WGlDb1dtWCtZeVB2cVJuVnp4eG5mcm40RUMz?=
 =?utf-8?B?dkkxcDhjK1ptdEVpM2drT1RFVEpUTzFGK3paZU1GVkxpOEUyc1FHc3AwTTRL?=
 =?utf-8?B?dC90UkkzR1EvRXpaSUVNZk5NZ08vSlluZ2xmdnV5dEZ2OXM5K01nbkh5MENJ?=
 =?utf-8?B?dHdRNFdKV3pCRzM0anNhRGR0VjFCV3hCM2pWMEhvSjBidTAzaEVQWjJUMzZp?=
 =?utf-8?B?QXphWjFUY3N1YVRyM21SNUhLbFd1U21rL3F1TVJwUnZoNjBUTkFCcUtOY0NP?=
 =?utf-8?B?ZFV2ZDlKdTJNMXBsamphSUpqNUtxYnBJaTJVN255UFhnT2tIakNxUUFtOExY?=
 =?utf-8?B?R29YdWJ2Tzh1V2VGVDFVZE9XMkNLeDBCVFBiNEdQN3piR1R0cDM5L3JkeCt6?=
 =?utf-8?B?Y2dyUFhqci9Pb2NpRnVZYkp4TXhsaDBHVEkyVzFRdDhxakhWQ1Y0RG9KbEZF?=
 =?utf-8?B?RHNVYjNKN2VNS29ZNTk5OHpCTnlRPT0=?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 45e4cd0b-f338-49d1-f9ad-08d9d5e835b0
X-MS-Exchange-CrossTenant-AuthSource: CO1PR10MB4722.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jan 2022 16:25:56.2679
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9i79Han8ZljEYuGp5MQj5FdY+COIo3Vg2ojOQUHQIaP0FqUWLtSL402caM1LSJEcjgSammg0K1jrUNRsYnXWkAEKO/yEZUcRcpE1Scxn11s=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR10MB4722
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10225 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 adultscore=0
 suspectscore=0 mlxlogscore=999 bulkscore=0 phishscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201120104
X-Proofpoint-ORIG-GUID: 6gc_rJzR0WTMgpHTXF-mMB3ktcmhWBVA
X-Proofpoint-GUID: 6gc_rJzR0WTMgpHTXF-mMB3ktcmhWBVA
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 12/23/21 3:44 AM, Baoquan He wrote:
> **Problem observed:
> On x86_64, when crash is triggered and entering into kdump kernel, page
> allocation failure can always be seen.
> 
>   ---------------------------------
>   DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
>   swapper/0: page allocation failure: order:5, mode:0xcc1(GFP_KERNEL|GFP_DMA), nodemask=(null),cpuset=/,mems_allowed=0
>   CPU: 0 PID: 1 Comm: swapper/0
>   Call Trace:
>    dump_stack+0x7f/0xa1
>    warn_alloc.cold+0x72/0xd6
>    ......
>    __alloc_pages+0x24d/0x2c0
>    ......
>    dma_atomic_pool_init+0xdb/0x176
>    do_one_initcall+0x67/0x320
>    ? rcu_read_lock_sched_held+0x3f/0x80
>    kernel_init_freeable+0x290/0x2dc
>    ? rest_init+0x24f/0x24f
>    kernel_init+0xa/0x111
>    ret_from_fork+0x22/0x30
>   Mem-Info:
>   ------------------------------------
> 
> ***Root cause:
> In the current kernel, it assumes that DMA zone must have managed pages
> and try to request pages if CONFIG_ZONE_DMA is enabled. While this is not
> always true. E.g in kdump kernel of x86_64, only low 1M is presented and
> locked down at very early stage of boot, so that this low 1M won't be
> added into buddy allocator to become managed pages of DMA zone. This
> exception will always cause page allocation failure if page is requested
> from DMA zone.
> 
> ***Investigation:
> This failure happens since below commit merged into linus's tree.
>    1a6a9044b967 x86/setup: Remove CONFIG_X86_RESERVE_LOW and reservelow= options
>    23721c8e92f7 x86/crash: Remove crash_reserve_low_1M()
>    f1d4d47c5851 x86/setup: Always reserve the first 1M of RAM
>    7c321eb2b843 x86/kdump: Remove the backup region handling
>    6f599d84231f x86/kdump: Always reserve the low 1M when the crashkernel option is specified
> 
> Before them, on x86_64, the low 640K area will be reused by kdump kernel.
> So in kdump kernel, the content of low 640K area is copied into a backup
> region for dumping before jumping into kdump. Then except of those firmware
> reserved region in [0, 640K], the left area will be added into buddy
> allocator to become available managed pages of DMA zone.
> 
> However, after above commits applied, in kdump kernel of x86_64, the low
> 1M is reserved by memblock, but not released to buddy allocator. So any
> later page allocation requested from DMA zone will fail.
> 
> At the beginning, if crashkernel is reserved, the low 1M need be locked
> down because AMD SME encrypts memory making the old backup region
> mechanims impossible when switching into kdump kernel.
> 
> Later, it was also observed that there are BIOSes corrupting memory
> under 1M. To solve this, in commit f1d4d47c5851, the entire region of
> low 1M is always reserved after the real mode trampoline is allocated.
> 
> Besides, recently, Intel engineer mentioned their TDX (Trusted domain
> extensions) which is under development in kernel also needs to lock down
> the low 1M. So we can't simply revert above commits to fix the page allocation
> failure from DMA zone as someone suggested.
> 
> ***Solution:
> Currently, only DMA atomic pool and dma-kmalloc will initialize and
> request page allocation with GFP_DMA during bootup.
> 
> So only initializ DMA atomic pool when DMA zone has available managed
> pages, otherwise just skip the initialization.
> 
> For dma-kmalloc(), for the time being, let's mute the warning of
> allocation failure if requesting pages from DMA zone while no manged
> pages. Meanwhile, change code to use dma_alloc_xx/dma_map_xx API to
> replace kmalloc(GFP_DMA), or do not use GFP_DMA when calling kmalloc()
> if not necessary. Christoph is posting patches to fix those under
> drivers/scsi/. Finally, we can remove the need of dma-kmalloc() as
> people suggested.
> 
> Changelog:
> v3->v4:
>   - Split the old v3 into two separate patchset. The first two clean
>     up/improvement patches in v3 have been sent out in a independent
>     patchset. The fixes patchs are adapted and sent in this patchset.
>   - Do not change dma-kmalloc(), mute the warning of allocation failure
>     instead if it's requesting page from DMA zone which has no managed
>     pages.
> 
> v2-Resend -> v3:
>   - Re-implement has_managed_dma() according to David's suggestion.
>   - Add Fixes tag and cc stable.
> 
> v2->v2 RESEND:
>   - John pinged to push the repost of this patchset. So fix one typo of
>     suject of patch 3/5; Fix a building error caused by mix declaration in
>     patch 5/5. Both of them are found by John from his testing.
>   - Rewrite cover letter to add more information.
> 
> v1->v2:
>   Change to check if managed DMA zone exists. If DMA zone has managed
>   pages, go further to request page from DMA zone to initialize. Otherwise,
>   just skip to initialize stuffs which need pages from DMA zone.
> 
> v3:
> https://urldefense.com/v3/__https://lore.kernel.org/all/20211213122712.23805-1-bhe@redhat.com/T/*u__;Iw!!ACWV5N9M2RV99hQ!e1KjpVuZycBkxdeNxcsRUQ7MH92KQQk7FfCZs5tzEcBVusUiph0w9zpxOgKpS2Y0ecPm$
> 
> V2 RESEND post:
> https://urldefense.com/v3/__https://lore.kernel.org/all/20211207030750.30824-1-bhe@redhat.com/T/*u__;Iw!!ACWV5N9M2RV99hQ!e1KjpVuZycBkxdeNxcsRUQ7MH92KQQk7FfCZs5tzEcBVusUiph0w9zpxOgKpSzZmH18k$
> 
> v2 post:
> https://urldefense.com/v3/__https://lore.kernel.org/all/20210810094835.13402-1-bhe@redhat.com/T/*u__;Iw!!ACWV5N9M2RV99hQ!e1KjpVuZycBkxdeNxcsRUQ7MH92KQQk7FfCZs5tzEcBVusUiph0w9zpxOgKpS0Fbih0f$
> 
> v1 post:
> https://urldefense.com/v3/__https://lore.kernel.org/all/20210624052010.5676-1-bhe@redhat.com/T/*u__;Iw!!ACWV5N9M2RV99hQ!e1KjpVuZycBkxdeNxcsRUQ7MH92KQQk7FfCZs5tzEcBVusUiph0w9zpxOgKpSwc3nQBz$
> 
> 
> 
> Baoquan He (3):
>    mm_zone: add function to check if managed dma zone exists
>    dma/pool: create dma atomic pool only if dma zone has managed pages
>    mm/page_alloc.c: do not warn allocation failure on zone DMA if no
>      managed pages
> 
>   include/linux/mmzone.h |  9 +++++++++
>   kernel/dma/pool.c      |  4 ++--
>   mm/page_alloc.c        | 18 +++++++++++++++++-
>   3 files changed, 28 insertions(+), 3 deletions(-)
> 


Tested-by: John Donnelly

I don't see GFP malloc failures when the CD-ROM is enumerated anymore 
either when kdump kernel is started.

tested on 5.15.13.