Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1;
MIME-Version: 1.0
References: <CAO7dBbVJ=ytRra_77VRZ8ud1wVkP9fub=Vj6cfTkx=CnYg5J2A@mail.gmail.com>
 <ZWVMUxmi66xLZPsr@MiWiFi-R3L-srv> <ZWWuBSiZZdF2W12j@tiehlicka>
 <ZWbyDx3TJ7zo3jCw@MiWiFi-R3L-srv> <91a31ce5-63d1-7470-18f7-92b039fda8e6@redhat.com>
 <ZWf64BowWrYqA2Rf@MiWiFi-R3L-srv> <ZWhg_b3O6piZtkQ-@tiehlicka>
 <ZWh6ax8YmkhxAzIf@MiWiFi-R3L-srv> <ZWiOO-KNJ82f6Gxu@tiehlicka>
 <CAF+s44QSJL5e6BVTAyyHR9Kzx7RJqZSkR=uXEypaouK_XuBbEw@mail.gmail.com> <ZWiRbLGdBMO2jFGs@tiehlicka>
In-Reply-To: <ZWiRbLGdBMO2jFGs@tiehlicka>
From:   Pingfan Liu <piliu@redhat.com>
Date:   Fri, 1 Dec 2023 08:54:20 +0800
Message-ID: <CAF+s44TQ2g6VTL4JSubvch5VkW7SSsePp-aBz+kigg563NijJg@mail.gmail.com>
Subject: Re: [PATCH 0/4] kdump: crashkernel reservation from CMA
To:     Michal Hocko <mhocko@suse.com>
Cc:     Baoquan He <bhe@redhat.com>, Donald Dutile <ddutile@redhat.com>,
        Jiri Bohac <jbohac@suse.cz>, Tao Liu <ltao@redhat.com>,
        Vivek Goyal <vgoyal@redhat.com>,
        Dave Young <dyoung@redhat.com>, kexec@lists.infradead.org,
        linux-kernel@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Thu, Nov 30, 2023 at 9:43=E2=80=AFPM Michal Hocko <mhocko@suse.com> wrot=
e:
>
> On Thu 30-11-23 21:33:04, Pingfan Liu wrote:
> > On Thu, Nov 30, 2023 at 9:29=E2=80=AFPM Michal Hocko <mhocko@suse.com> =
wrote:
> > >
> > > On Thu 30-11-23 20:04:59, Baoquan He wrote:
> > > > On 11/30/23 at 11:16am, Michal Hocko wrote:
> > > > > On Thu 30-11-23 11:00:48, Baoquan He wrote:
> > > > > [...]
> > > > > > Now, we are worried if there's risk if the CMA area is retaken =
into kdump
> > > > > > kernel as system RAM. E.g is it possible that 1st kernel's ongo=
ing RDMA
> > > > > > or DMA will interfere with kdump kernel's normal memory accessi=
ng?
> > > > > > Because kdump kernel usually only reset and initialize the need=
ed
> > > > > > device, e.g dump target. Those unneeded devices will be unshutd=
own and
> > > > > > let go.
> > > > >
> > > > > I do not really want to discount your concerns but I am bit confu=
sed why
> > > > > this matters so much. First of all, if there is a buggy RDMA driv=
er
> > > > > which doesn't use the proper pinning API (which would migrate awa=
y from
> > > > > the CMA) then what is the worst case? We will get crash kernel co=
rrupted
> > > > > potentially and fail to take a proper kernel crash, right? Is thi=
s
> > > > > worrisome? Yes. Is it a real roadblock? I do not think so. The pr=
oblem
> > > > > seems theoretical to me and it is not CMA usage at fault here IMH=
O. It
> > > > > is the said theoretical driver that needs fixing anyway.
> > > > >
> > > > > Now, it is really fair to mention that CMA backed crash kernel me=
mory
> > > > > has some limitations
> > > > >     - CMA reservation can only be used by the userspace in the
> > > > >       primary kernel. If the size is overshot this might have
> > > > >       negative impact on kernel allocations
> > > > >     - userspace memory dumping in the crash kernel is fundamental=
ly
> > > > >       incomplete.
> > > >
> > > > I am not sure if we are talking about the same thing. My concern is=
:
> > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> > > > 1) system corrutption happened, crash dumping is prepared, cpu and
> > > > interrupt controllers are shutdown;
> > > > 2) all pci devices are kept alive;
> > > > 3) kdump kernel boot up, initialization is only done on those devic=
es
> > > > which drivers are added into kdump kernel's initrd;
> > > > 4) those on-flight DMA engine could be still working if their kerne=
l
> > > > module is not loaded;
> > > >
> > > > In this case, if the DMA's destination is located in crashkernel=3D=
,cma
> > > > region, the DMA writting could continue even when kdump kernel has =
put
> > > > important kernel data into the area. Is this possible or absolutely=
 not
> > > > possible with DMA, RDMA, or any other stuff which could keep access=
ing
> > > > that area?
> > >
> > > I do nuderstand your concern. But as already stated if anybody uses
> > > movable memory (CMA including) as a target of {R}DMA then that memory
> > > should be properly pinned. That would mean that the memory will be
> > > migrated to somewhere outside of movable (CMA) memory before the
> > > transfer is configured. So modulo bugs this shouldn't really happen.
> > > Are there {R}DMA drivers that do not pin memory correctly? Possibly. =
Is
> > > that a road bloack to not using CMA to back crash kernel memory, I do
> > > not think so. Those drivers should be fixed instead.
> > >
> > I think that is our concern. Is there any method to guarantee that
                           ^^^ Sorry, to clarify, I am only speaking for my=
self.

> > will not happen instead of 'should be' ?
> > Any static analysis during compiling time or dynamic checking method?
>
> I am not aware of any method to detect a driver is going to configure a
> RDMA.
>

If there is a pattern, scripts/coccinelle may give some help. But I am
not sure about that.

> > If this can be resolved, I think this method is promising.
>
> Are you indicating this is a mandatory prerequisite?

IMHO, that should be mandatory. Otherwise for any unexpected kdump
kernel collapses,  it can not shake off its suspicion.

Thanks,

Pingfan