Date:   Mon, 30 Jan 2023 14:00:47 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     Shesha Sreenivasamurthy <sheshas@gmail.com>,
        Dan Williams <dan.j.williams@intel.com>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        <linux-cxl@vger.kernel.org>
Subject: Re: Hot ADD using CXL1.1 host
Message-ID: <63d83e0f9378_3a36e529468@dwillia2-xfh.jf.intel.com.notmuch>
References: <DM6PR18MB28441B6B542B2A59CF39BF6DAFCE9@DM6PR18MB2844.namprd18.prod.outlook.com>
 <DM6PR18MB2844505042F7EDCF69CE08DBAFD39@DM6PR18MB2844.namprd18.prod.outlook.com>
 <63d8292f70918_ea22229465@dwillia2-xfh.jf.intel.com.notmuch>
 <CABL7MgHENj0FcT+whjdskOmqPfNa0zo8gQobpZ0mrW3a3NY_pA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CABL7MgHENj0FcT+whjdskOmqPfNa0zo8gQobpZ0mrW3a3NY_pA@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TktqbzlpOUN4MDlJL1pUc0JVNmh3SWRVbXBYOHF3UktiTm8rZDhmRjl5dEFz?=
 =?utf-8?B?UjRjOXBnd212ZUpyTVBvVmszQ1NCcE1hWnRIMk9ybTJpd1o5eWJnK00yMWlL?=
 =?utf-8?B?Q1hsVDlpNXBldDk2aG4zZzZ1Tnd6NTFuajdwQnhQZGNZWXlLczlhSmt2THpt?=
 =?utf-8?B?ekNBNHp2Y3BObWV1SWJEeHVVZ29OVlhTTDh2SHVjUHFldTBxalBFaGhwTS92?=
 =?utf-8?B?WjJFc1V1TUpQK3VqdWtBOU5LYkk3dUhwdHF1bGQwRXkyLzJtcHlRYkRsbkhp?=
 =?utf-8?B?V2MvT2pXUEV0dkU5Q2ZuSmdEVTRBbC93RkJ3azRicUFJUGU5aXNjTS9FM0Qy?=
 =?utf-8?B?UlVQYVA2dlQ3SWtmd3hIVjlJVnVLQjVvN3BXS04rQ1hhWThnalh4SnhsS0Jx?=
 =?utf-8?B?YmtFaFBKeG5sajMzNE9ETmxSMTdRSzlweVoxWnFTSWlYblBXeDFwbURlOVlS?=
 =?utf-8?B?RjZ2dDdvdXRvaStOYVhmVlNpQ0NzOHhJWGE1SDJjcnZPOVBJL2FKVlVtRVRF?=
 =?utf-8?B?a0QxeW9rOHkxRE5QS2lna2E3WGxnenFuTDRHcGU2UGR5TlJwOEFqVUZ3cTdP?=
 =?utf-8?B?RkZCV3I5N29wRkoxNDRJUjhSRjNaMis0U2V2VlBFUmhueHJ4S3pnelk2WEha?=
 =?utf-8?B?MG5UQ01yTFFvekR3R3B3TDVhRnQxcHpIaVA2K1M2b0JsS1hiRWpadVZjNmlN?=
 =?utf-8?B?eWVHMnNna1p1bjBWeWh4RnNIV0FqaFhzbzhVeEtUK0Q3NlhOMjgzQ2dZa0FR?=
 =?utf-8?B?Zmd2WW9qSE5PME8xaWV4N0hqNFhTTllFWDhKb0tFWGRVcS83eStBd1R2TWxn?=
 =?utf-8?B?eGVrT1RZQkc1OW5FL2FkZnMzNmJ6QTl6UDMvT0d2SzF2bm9wMEtXR3hQeHVj?=
 =?utf-8?B?UTRUOEYzUmQ2ZTlzem5rM0JRRFJTZ3YzcG80Q0lzRUdVWlJONlZQZlcvRDBW?=
 =?utf-8?B?QzdlMjFBWTlwblNaQlkxb2tnTTRMaVdLZzFkUFdycmZNWWxpNjYzblBwNlBU?=
 =?utf-8?B?dDlmMjJLWnBvbTZQSjM5ajY2QVRRVkRBdk1WTlc1K1FZRTFKYjV0Zkdsa2Z1?=
 =?utf-8?B?SmpBWlNUYTF4OGpld0U4bmRwb01iMFI3Y1V3Q0toTjBHQXAxR3R0Vlg4NkZq?=
 =?utf-8?B?ZFcrdmNUT0hQQlFoNnRidTJ3cDl0bDZwWVRuaXJuL2w4aGM4bWh3bHhzb1Aw?=
 =?utf-8?B?ZGR5TDRiazJYMFlVN2NFSjk4T0lHdXRJQXIyai80QWI4bTVlb0x4VndkVVUx?=
 =?utf-8?B?VENVNjhiREdRNzZYNWVQeUkybUxBSE1yRVdBR1BVNTNyaGJmdDYzcElYa3lm?=
 =?utf-8?B?R1dsOHF4MFYvNkV5OW43ZmVlSFYwaGJuK1I1Y25CSDQxem5PMG1QeHhVK2Q3?=
 =?utf-8?B?QkFTTHJhQ1Vta0x1M2pncFcvdUJCWHgwWUpNZFBiU0Yvczd5UHVLRnpDdVlT?=
 =?utf-8?B?UEdrbFR0MDNXU3VYaENxOGVNODN4enMzWCtEdlhPS2ZjakI0ZmhSRXBoWE5t?=
 =?utf-8?B?R1I4RG92czYxTG9rbVhaelJNUUJPR3AyaUlCVFRLZVRLY3k0aDFEOFNxZDRp?=
 =?utf-8?B?Q2wwOGdMcHg2MUxWc2Fvd2RRTSsvM1kzcnhJT1lDMWRNRjh2UGgyeHY2dVJ1?=
 =?utf-8?B?YnFyaVZNd2xWejhLUDNFNHJ3VVFwNmh6bG9YQjQrSy9CMHord1NwRXk1anFC?=
 =?utf-8?B?QlovbGF5aXNvY0duVWhzNTZ6cWZTcWNvODhrOVpWcFRKRTVpakZKTnF2T0c4?=
 =?utf-8?B?am5IaVdZa1N2aHpiaFplVG9EdktXeDFIc2RiQ0VoQ0xrdFIzajRBTy9WOThV?=
 =?utf-8?B?Zm1aRllrNnJOdytpaFowdFNGY0krb01BeTdwditFQjlCZ3JRT1Q3czdSY0I4?=
 =?utf-8?B?TXg5Z2xIdmxaL2ZUL0kyRkM5bEI2Z3FWaGJtaklhRk5BeWJzbk5ITXNVSTlq?=
 =?utf-8?B?Tjhmemd5SUJ4dWsySGNraG5yWU1Galc4Smo5QWFjZ0diV24zaVk5TEFMSFpq?=
 =?utf-8?B?UEVrd1BMakpBaVo1Sm5odE5zSVZqRVdFbDVON0pEZWpacnFQRmJFVDdDL3dR?=
 =?utf-8?B?Z0EzMFpzL3UybDZETHZ1Zi9XaUJVWTJwejJwWGRJWHlPOFZaQVYyZVBoa0tn?=
 =?utf-8?B?OHdvalVhYlBEUjV0b2RqblVNMDY0dVdUTmRQQXVwelI2RlpocWo0WEtiaWRx?=
 =?utf-8?B?SEE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5e0296df-cd4c-4b67-8a62-08db030d72dd
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jan 2023 22:00:50.3087
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: dveYVl3PjZTBPZfCbntzkzd3BKm/K6Y32pTlvT3s/uxww/AySTPXnmJoa+GP5NPDADpSMyDQbQpa2HHJVLrb6FGQBF/cmJORB+ibR+YxItI=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5022
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Shesha, Linux email expectations are to not top post, i.e. respond
inline, like below:

Shesha Sreenivasamurthy wrote:
> The re-configuration does not reset the device. It does re-program the PCIe
> DVSEC for CXL Device register (Section 8.1.3 CXL 2.0 spec Pg 258), register
> (DVSEC vendor ID 0x1E98, DCSEC ID 0x0).
> 
> 
> 
> “So you need to dynamically recreate the region, especially if your step 10
> above resets the device.”
> 
> 
> 
> Do you mean the DAX region ?

No, I mean the CXL region.

> If so, I can if the system stays up. After a few seconds the system
> crashes. Can the crash be because of a mismatch between DVSEC
> information with what kernel was informed by BIOS during boot (Some
> ACPI tables ?)

My concern is that the platform memory decode configuration is not
prepared for the CXL device to claim more than what was originally
programmed in the CXL DVSEC range registers. One of the platform
firmware updates for CXL 2.0 was the creation of the CFMWS (CXL Fixed
Memory Window Structure) in the ACPI CEDT (CXL Early Discovery Table).
That structure indicates which platform address ranges decode to which
CXL host bridges. Those windows are defined in platform specific
registersi (not enumerated to the OS). If the window is only 8GB then
the endpoint device can not decode more. You would need to reboot to get
the BIOS to allocate more host address space for CXL.

The expectation for newer platforms is that platform firmware define
CFMWS such that there is spare capacity in the address map for the OS to
dynmaically map more CXL.