Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Shesha Sreenivasamurthy <sheshas@gmail.com>
Mime-Version: 1.0 (1.0)
Subject: Re: Hot ADD using CXL1.1 host
Message-Id: <0D3A81E2-C99A-491D-AB66-FC6005E38667@gmail.com>
Date:   Tue, 31 Jan 2023 07:19:58 -0800
Cc:     linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org
To:     Dan Williams <dan.j.williams@intel.com>
Precedence: bulk

=EF=BB=BF

On Mon, Jan 30, 2023 at 2:00 PM Dan Williams <dan.j.williams@intel.com> wrot=
e:
>=20
> Hi Shesha, Linux email expectations are to not top post, i.e. respond
> inline, like below:
>=20
> Shesha Sreenivasamurthy wrote:
>> The re-configuration does not reset the device. It does re-program the PC=
Ie
>> DVSEC for CXL Device register (Section 8.1.3 CXL 2.0 spec Pg 258), regist=
er
>> (DVSEC vendor ID 0x1E98, DCSEC ID 0x0).
>> =E2=80=9CSo you need to dynamically recreate the region, especially if yo=
ur step 10
>> above resets the device.=E2=80=9D
>> Do you mean the DAX region ?
>=20
> No, I mean the CXL region.
>=20
>> If so, I can if the system stays up. After a few seconds the system
>> crashes. Can the crash be because of a mismatch between DVSEC
>> information with what kernel was informed by BIOS during boot (Some
>> ACPI tables ?)
>=20
> My concern is that the platform memory decode configuration is not
> prepared for the CXL device to claim more than what was originally
> programmed in the CXL DVSEC range registers. One of the platform
> firmware updates for CXL 2.0 was the creation of the CFMWS (CXL Fixed
> Memory Window Structure) in the ACPI CEDT (CXL Early Discovery Table).
> That structure indicates which platform address ranges decode to which
> CXL host bridges. Those windows are defined in platform specific
> registersi (not enumerated to the OS). If the window is only 8GB then
> the endpoint device can not decode more. You would need to reboot to get
> the BIOS to allocate more host address space for CXL.
>=20
> The expectation for newer platforms is that platform firmware define
> CFMWS such that there is spare capacity in the address map for the OS to
> dynmaically map more CXL.

There seems to be some instability in using DAX. When the system is given al=
l the device memory using efi=3Dnosoftreserve, the stressapptest (https://gi=
thub.com/stressapptest/stressapptest) runs for an extended period of time. H=
owever, when the system is booted without efi=3Dnosoftreserve, and assigned t=
he special purpose memory to system-ram using daxctl, the system crashes aft=
er some time (20-30 mins). Is there any known instabilities when using DAX?