Received: by 2002:a05:6a10:6744:0:0:0:0 with SMTP id w4csp226879pxu; Wed, 7 Oct 2020 01:10:39 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzse/97RbWFNV3UgRwfxQ3SdDpTy7WUlTFwcoYRhHJ2gwT80Xw+tIZfcGVIVOSkSfmlILPf X-Received: by 2002:a17:906:7d7:: with SMTP id m23mr2191324ejc.47.1602058239527; Wed, 07 Oct 2020 01:10:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1602058239; cv=none; d=google.com; s=arc-20160816; b=d2mCtRrPP2MrOJXipiBWQVWq1L/BQEwRSHYng7Z1fPiYRGUjYOmKOhN7M47cMghKtc w6vXhfJwR4pGb+UkfK0b62bM7F6BC6fFrGqk9RENTeEweoBS4RNL00/B1Q0MPDTYbVbV 3PWJO/pXhQUPxCX+V7yPt9yu8HPZw29tYocjJLANgeXoi+qzsuoSVJ09C1MGU5xnh8Pu k1R5wLPRhVOqRlDgH35QXYM4G4N62ftTq6Kjm2un0Oo43Hfy7/KLSeEIan6n1KPOUllR wikBETTyozlUYsdvlbqVTcEyCYRyk4z06kVPlJVWUV0NOBUJBFlAunwCORkJ6RVx3xDZ yilA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:subject:message-id:date:from:in-reply-to :references:mime-version:dkim-signature; bh=2emVL8rNfBMhk//RxZehtElMhUSqx0aMuZNVdSt+Tbg=; b=WDq47HeMb7ULp+Of410BLY8AsaFcx7r8dNJNNBso1c0wUaMuJKdCbsKrOOdtWFlGq9 Aznerqq+aqoTsOgfwhh7BBsa2YgVqZFZ3S3H6m/0SuzNxhN7TnumkI28SphKF3FzqpM6 GlvyGaZv8g5kQZQqWc9sHx+V9Z5ire/+W+BAW8bAw4S3C1Zm4S1Jeb3I+aDu3xvYj9ps xY0O21nylVMnoYP2UR0DlbiKZEbWLUce+HBKqOMZCK8YBbpM7ghR+ugM8Xf6Nqazu/1b xEVCFGzJgugMXJUG92wRHI3rtKet34vAV5iEY98FoRlTDPlSeGNHXjf1qkaQ1j5ajMI/ atWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KJVxHGp3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id dp1si1220861ejc.307.2020.10.07.01.10.16; Wed, 07 Oct 2020 01:10:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KJVxHGp3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727557AbgJGHIH (ORCPT + 99 others); Wed, 7 Oct 2020 03:08:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:37139 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727033AbgJGHIH (ORCPT ); Wed, 7 Oct 2020 03:08:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1602054485; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=2emVL8rNfBMhk//RxZehtElMhUSqx0aMuZNVdSt+Tbg=; b=KJVxHGp3Pbos2n9/huRZxkhNaTHZqdMZTpECUD75EAeOtG2fqEG+3fA45aIy97R/KvkPFY UOx2NTT+j28R+3DfVyAe0GEwpL8w/UUs4XFN6x2KMKwLAZbX+On0QINEdACTBCPSuzJ7dc ndGweDuQMib/CmlU4eitNT6QIGmJZr0= Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-490-CBGm5rJXPr6Ts-3uDnn9ow-1; Wed, 07 Oct 2020 03:08:04 -0400 X-MC-Unique: CBGm5rJXPr6Ts-3uDnn9ow-1 Received: by mail-oi1-f198.google.com with SMTP id k7so497559oif.22 for ; Wed, 07 Oct 2020 00:08:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2emVL8rNfBMhk//RxZehtElMhUSqx0aMuZNVdSt+Tbg=; b=f341quV5G/jkTerVswPFCm9Y+FRxH58sRc/ZzCZr3o5lFusOMs1PkSabVSbXCvewa4 /Ywo/we41hGwB78zvEfIchMaZdEL+OfesxhAUpT0pJfwSv8yR6m6GwCWor19bYGcSn8J 4j3tfXNFEbYdJictwA36cFE0zz26LdgJvgCAPP/w7xGnejUe6bn2ei7TQDRk2JcA/vgw 6SP9beg9v094coOCrLGy2C0UO+D8gw0Tgjs0hzUutu1bvvWUmatGPbdSEKAfnh+tMxLZ s/9C50XtvuigmFcXJ73edxTOaREndghhs8vx2lQVkzct7ajVoNFknLteN72+eu/8XEnl Nvhw== X-Gm-Message-State: AOAM530IMaSLAgoriLQur6ABkGkcY3hAd9ErxAWaU7V00YwcTvhyCg8f pwF7tX46tbtnswyfcqtoYze6X6ksy/JWk0I8d20mlZVeJrGE8f92mPZjvDG83HlENDDWge4x7rr LsZkaPQaW1zqDd+kLIG/hxaTnsc2sMrWQRSxvSBj2 X-Received: by 2002:a9d:2c05:: with SMTP id f5mr1035021otb.183.1602054481586; Wed, 07 Oct 2020 00:08:01 -0700 (PDT) X-Received: by 2002:a9d:2c05:: with SMTP id f5mr1034994otb.183.1602054481228; Wed, 07 Oct 2020 00:08:01 -0700 (PDT) MIME-Version: 1.0 References: <20200907134745.25732-1-chenzhou10@huawei.com> <20201005170937.GA14576@gaia> <20201006180012.GB31946@C02TF0J2HF1T.local> In-Reply-To: <20201006180012.GB31946@C02TF0J2HF1T.local> From: Bhupesh Sharma Date: Wed, 7 Oct 2020 12:37:49 +0530 Message-ID: Subject: Re: [PATCH v12 0/9] support reserving crashkernel above 4G on arm64 kdump To: Catalin Marinas Cc: John Donnelly , Chen Zhou , Will Deacon , James Morse , Thomas Gleixner , Ingo Molnar , RuiRui Yang , Baoquan He , Jonathan Corbet , Prabhakar Kushwaha , Simon Horman , Rob Herring , Arnd Bergmann , nsaenzjulienne@suse.de, linux-arm-kernel , Linux Kernel Mailing List , kexec mailing list , Linux Doc Mailing List , guohanjun@huawei.com, xiexiuqi@huawei.com, huawei.libin@huawei.com, wangkefeng.wang@huawei.com Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Catalin, On Tue, Oct 6, 2020 at 11:30 PM Catalin Marinas wrote: > > On Mon, Oct 05, 2020 at 11:12:10PM +0530, Bhupesh Sharma wrote: > > I think my earlier email with the test results on this series bounced > > off the mailing list server (for some weird reason), but I still see > > several issues with this patchset. I will add specific issues in the > > review comments for each patch again, but overall, with a crashkernel > > size of say 786M, I see the following issue: > > > > # cat /proc/cmdline > > BOOT_IMAGE=(hd7,gpt2)/vmlinuz-5.9.0-rc7+ root=<..snip..> rd.lvm.lv=<..snip..> crashkernel=786M > > > > I see two regions of size 786M and 256M reserved in low and high > > regions respectively, So we reserve a total of 1042M of memory, which > > is an incorrect behaviour: > > > > # dmesg | grep -i crash > > [ 0.000000] Reserving 256MB of low memory at 2816MB for crashkernel (System low RAM: 768MB) > > [ 0.000000] Reserving 786MB of memory at 654158MB for crashkernel (System RAM: 130816MB) > > [ 0.000000] Kernel command line: BOOT_IMAGE=(hd2,gpt2)/vmlinuz-5.9.0-rc7+ root=/dev/mapper/rhel_ampere--hr330a--03-root ro rd.lvm.lv=rhel_ampere-hr330a-03/root rd.lvm.lv=rhel_ampere-hr330a-03/swap crashkernel=786M cma=1024M > > > > # cat /proc/iomem | grep -i crash > > b0000000-bfffffff : Crash kernel (low) > > bfcbe00000-bffcffffff : Crash kernel > > As Chen said, that's the intended behaviour and how x86 works. The > requested 768M goes in the high range if there's not enough low memory > and an additional buffer for swiotlb is allocated, hence the low 256M. I understand, but why 256M (as low) for arm64? x86_64 setups usually have more system memory available as compared to several commercially available arm64 setups. So is the intent, just to keep the behavior similar between arm64 and x86_64? Should we have a CONFIG option / bootarg to help one select the max 'low_size'? Currently the ' low_size' value is calculated as: /* * two parts from kernel/dma/swiotlb.c: * -swiotlb size: user-specified with swiotlb= or default. * * -swiotlb overflow buffer: now hardcoded to 32k. We round it * to 8M for other buffers that may need to stay low too. Also * make sure we allocate enough extra low memory so that we * don't run out of DMA buffers for 32-bit devices. */ low_size = max(swiotlb_size_or_default() + (8UL << 20), 256UL << 20); Since many arm64 boards ship with swiotlb=0 (turned off) via kernel bootargs, the low_size, still ends up being 256M in such cases, whereas this 256M can be used for some other purposes - so should we be limiting this to 64M and failing the crash kernel allocation request (gracefully) otherwise? > We could (as an additional patch), subtract the 256M from the high > allocation so that you'd get a low 256M and a high 512M, not sure it's > worth it. Note that with a "crashkernel=768M,high" option, you still get > the additional low 256M, otherwise the crashkernel won't be able to > boot as there's no memory in ZONE_DMA. In the explicit ",high" request > case, I'm not sure subtracted the 256M is more intuitive. > In 5.11, we also hope to fix the ZONE_DMA layout for non-RPi4 platforms > to cover the entire 32-bit address space (i.e. identical to the current > ZONE_DMA32). > > > IMO, we should test this feature more before including this in 5.11 > > Definitely. That's one of the reasons we haven't queued it yet. So any > help with testing here is appreciated. Sure, I am running more checks on this series. I will be soon back with more updates. Regards, Bhupesh