Received-SPF: pass (google.com: domain of linux-kernel+bounces-59922-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Fri, 9 Feb 2024 12:15:57 -0800
From: Dan Williams <dan.j.williams@intel.com>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>, Dan Williams
	<dan.j.williams@intel.com>
CC: linux-kernel <linux-kernel@vger.kernel.org>, Dave Hansen
	<dave.hansen@linux.intel.com>, <linux-cxl@vger.kernel.org>,
	<nvdimm@lists.linux.dev>, Vishal Verma <vishal.l.verma@intel.com>, Dave Jiang
	<dave.jiang@intel.com>, rostedt <rostedt@goodmis.org>, "Masami Hiramatsu
 (Google)" <mhiramat@kernel.org>
Subject: RE: CPU data cache across reboot/kexec for pmem/dax devices
Message-ID: <65c687fd30cf2_afa4294bc@dwillia2-xfh.jf.intel.com.notmuch>
References: <bc925b57-4814-4788-b369-cd2d135a7381@efficios.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <bc925b57-4814-4788-b369-cd2d135a7381@efficios.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?YZFW3Uu9/vUJYc+Ar089ivsESEGY3rGvoWenlpR42QiKViVJI0GUJ7vdEjOH?=
 =?us-ascii?Q?cxX/I2jWbVC2/PQu2NehtnKPorHFwKRlmV9YJwaxO/yP3c90Y1571QpsvKQB?=
 =?us-ascii?Q?yqBDu7W5BWWsz83lcuPSYZYHRdbGoXxuGzXX70LFVV8LSbLGbnx8gO3OeJya?=
 =?us-ascii?Q?rFrnEok4t78OflKLcXDdSdN9vTHNfMBL9BmDJCplw6GcgQijGNjoVxgZ0fhH?=
 =?us-ascii?Q?mzZIP7uSJC/ppD17kbDFcyIsNnQZI+2rmSA5Zxm8HTV98BjcL0nPnYIe10Gu?=
 =?us-ascii?Q?ZXkWpciZGjIgVERVb//JlleNcmdoLnLMmTIpVbcdgTNNV/vbGgKsxBmSL85f?=
 =?us-ascii?Q?A8p4T/wK/gL+txURzkxJFbvUuYLXMdXPnK2m5d4vw8AJeBWbAfIKexH/U/Uq?=
 =?us-ascii?Q?+B99zTYSSEy+slcBd+uHSSSVONg3xaDS9pGx8kKNcwdJwg50N2YzMkftbB0b?=
 =?us-ascii?Q?c+/MJtf3qv/7HTwiPEgPlUDZaOLkORMSWhE/yYZJo7XiUr9DLA/EQ21qHabA?=
 =?us-ascii?Q?ojJU9LzdM5FhITPClQa/MprghTaFHobQlLKApxdmAzNpJZm6tNZtRjMfpc+b?=
 =?us-ascii?Q?1hZPRbn/UGlp4yw3z7AiSA0zq/NZVkXfdFbZJQqOosbQqCCtQe9HNA/zsZoh?=
 =?us-ascii?Q?NyYVGk5JiuKkHEJcHj/Jd2deOWkGEeaHWhNpRQENG+fBZU3N/SnePrigo2ms?=
 =?us-ascii?Q?vi2Kapwl/d5rr+sYOQtGfTRYrZUBGoLg7kZeP5D1eHGTjRyxA2Z3jLXR2Ibq?=
 =?us-ascii?Q?fmLrkWto6sUtcvwj2a+hkX2yHg+/rfQi+tRH6gK+BZkBnL2D3S4U2WpzIsvv?=
 =?us-ascii?Q?9i72GlEt6VCCBQZdRslLYjgXmxLEm9/3MlCqFZu1S9V4D45vETSToDkPfAM5?=
 =?us-ascii?Q?cwYvynSrM991uxhJp/Ef9bYSNTlkuksE7WnN0U3Q05ZldiFpaohhyoYdF4ZQ?=
 =?us-ascii?Q?rzukfEt7+c9Bt5UPsA0ihP/fjf3Zsi7V3Ql3nteRIrb3NAs8baX9Xx3nzafU?=
 =?us-ascii?Q?seFzdK0F42srPphmyL+lkfhcPbwRosijsaSTjEx4/VberZ6O5HIJzoio6Ej4?=
 =?us-ascii?Q?xZOBp/d3tuhbvkTiXbaq5gv3VQAI9kD4+JP+wqbwoecz2q6ta3ov3OpOUPIY?=
 =?us-ascii?Q?4Cuaf/MdIBLTdDtKe64Ci3sTSSXJdFcuFs0mlE0sFNXlkkEabD7ucNkbhq/J?=
 =?us-ascii?Q?ocwhUaQ/h1sag683+OUTAbQlpRSLBMRL4MMKY9Mz16OIJWFahLBtEVjkEZrV?=
 =?us-ascii?Q?yEzX5mmF2WpYuLxEzUDsh93GFLi1Bb6+Hmy+zsCyx1b4HzI5A+zVRP9+TqDp?=
 =?us-ascii?Q?Iqse7CfKj/ASHspEdqIVucNK+LQ5DbmrD+0WoCwEq0XwtVUl1KRCVPtBo6bL?=
 =?us-ascii?Q?Ym3xgAYY2pBhZkd724bR7x2NIaUXHp6MPxRsWaw9VyZjuWatRZWV5H3ACwxY?=
 =?us-ascii?Q?qpx0JXi25CBFPq/tamcVElvKMOM6GCWYLQXVJwGY3raj5F+myr1WjA22FwBD?=
 =?us-ascii?Q?U47OeaobyFO9yDY+QUuxgHILDrN7m6CA3NP1B2iaUu6juO4zpK7LXoZ3mUuU?=
 =?us-ascii?Q?ht7R6qd6xMoHUriIL8M0ly7g2S3sI8XvFWOp+qqrE7O18GosIw7qpIgj58wJ?=
 =?us-ascii?Q?gg=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: af6dfbf2-6bf4-49fc-672f-08dc29abee1b
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2024 20:15:59.1862
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: XsSxVIbHg9bUqkgn+R0dnuFBlprkRTNfmyHSbKJkw8DnW+zI43AckTp9vYeTTBR5lWwIUJQQ7Ofv+URin54iojJ0zL7j3IpU1z9cfs1uJtE=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB7620
X-OriginatorOrg: intel.com

Mathieu Desnoyers wrote:
> Hi Dan,
> 
> In the context of extracting user-space trace data when the kernel crashes,
> the LTTng user-space tracer recommends using nvdimm/pmem to reserve an area
> of physical (volatile) RAM at boot (memmap=nn[KMG]!ss[KMG]), and use the
> resulting device to create/mount a dax-enabled fs (e.g. ext4).
> 
> We then use this filesystem to mmap() the shared memory files for the tracer.
> 
> I want to make sure that the very last events from the userspace tracer written
> to the memory mapped buffers (mmap()) by userspace are present after a
> warm-reboot (or kexec/kdump).
> 
> Note that the LTTng user-space tracer (LTTng-UST) does *not* issue any clflush
> (or equivalent pmem_persist() from libpmem) for performance reasons: ring buffer
> data is usually overwritten many times before the system actually crashes, and
> the only thing we really need to make sure is that the cache lines are not
> invalidated without write back.
> 
> So I understand that the main use-case for pmem is nvdimm, and that in order to
> guarantee persistence of the data on power off an explicit pmem_persist() is
> needed after each "transaction", but for the needs of tracing, is there some
> kind of architectural guarantee that the data present in the cpu data cache
> is not invalidated prior to write back in each of those scenarios ?
>
> - reboot with bios explicitly not clearing memory,

This one gives me pause, because a trip through the BIOS typically means
lots of resets and other low level magic, so this would likely require
pushing dirty data out of CPU caches prior to entering the BIOS code
paths.

So this either needs explicit cache flushing or mapping the memory with
write-through semantics. That latter one is not supported in the stack
today.

> - kexec/kdump.

This should maintain the state of CPU caches. As far as the CPU is
concerned it is just long jumping into a new kernel in memory without
resetting any CPU cache state.