Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933643AbbLOU0f (ORCPT ); Tue, 15 Dec 2015 15:26:35 -0500 Received: from g9t5008.houston.hp.com ([15.240.92.66]:51619 "EHLO g9t5008.houston.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932456AbbLOU0d (ORCPT ); Tue, 15 Dec 2015 15:26:33 -0500 From: "Elliott, Robert (Persistent Memory)" To: Borislav Petkov CC: Dan Williams , "Luck, Tony" , linux-nvdimm , X86 ML , "linux-kernel@vger.kernel.org" , Linux MM , "Andy Lutomirski" , Andrew Morton , Ingo Molnar Subject: RE: [PATCHV2 3/3] x86, ras: Add mcsafe_memcpy() function to recover from machine checks Thread-Topic: [PATCHV2 3/3] x86, ras: Add mcsafe_memcpy() function to recover from machine checks Thread-Index: AQHRNEq4ghk1LvWIwU2qH4H+2NJLyJ7MC6mAgABMagCAAAJcgIAACYCAgAACUYCAAAEBAIAAATVQgAAMi4CAAAtmAA== Date: Tue, 15 Dec 2015 20:25:37 +0000 Message-ID: <94D0CD8314A33A4D9D801C0FE68B40295BE9F3D5@G4W3202.americas.hpqcorp.net> References: <23b2515da9d06b198044ad83ca0a15ba38c24e6e.1449861203.git.tony.luck@intel.com> <20151215131135.GE25973@pd.tnic> <3908561D78D1C84285E8C5FCA982C28F39F8566E@ORSMSX114.amr.corp.intel.com> <20151215183924.GJ25973@pd.tnic> <94D0CD8314A33A4D9D801C0FE68B40295BE9F290@G4W3202.americas.hpqcorp.net> <20151215192837.GL25973@pd.tnic> In-Reply-To: <20151215192837.GL25973@pd.tnic> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [16.210.48.36] Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id tBFKQduP004374 Content-Length: 6337 Lines: 121 --- Robert Elliott, HPE Persistent Memory > -----Original Message----- > From: Borislav Petkov [mailto:bp@alien8.de] > Sent: Tuesday, December 15, 2015 1:29 PM > To: Elliott, Robert (Persistent Memory) > Cc: Dan Williams ; Luck, Tony > ; linux-nvdimm ; X86 ML > ; linux-kernel@vger.kernel.org; Linux MM mm@kvack.org>; Andy Lutomirski ; Andrew Morton > ; Ingo Molnar > Subject: Re: [PATCHV2 3/3] x86, ras: Add mcsafe_memcpy() function to > recover from machine checks > > On Tue, Dec 15, 2015 at 07:19:58PM +0000, Elliott, Robert (Persistent > Memory) wrote: > > ... > > > Due to the historic long latency of storage devices, > > applications don't re-read from storage again; they > > save the results. > > So, the streaming-load instructions are beneficial: > > That's the theory... > > Do you also have some actual performance numbers where non-temporal > operations are better than the REP; MOVSB and *actually* show > improvements? And no microbenchmarks please. > > Thanks. > This isn't exactly what you're looking for, but here is an example of fio doing reads from pmem devices (reading from NVDIMMs, writing to DIMMs) with various transfer sizes. At 256 KiB, all the main memory buffers fit in the CPU caches, so no write traffic appears on DDR (just the reads from the NVDIMMs). At 1 MiB, the data spills out of the caches, and writes to the DIMMs end up on DDR. Although DDR is busier, fio gets a lot less work done: * 256 KiB: 90 GiB/s by fio * 1 MiB: 49 GiB/s by fio We could try modifying pmem to use its own non-temporal memcpy functions (I've posted experimental patches before that did this) to see if that transition point shifts. We can also watch the CPU cache statistics while running. Here are statistics from Intel's pcm-memory.x (pardon the wide formatting): 256 KiB ======= pmem0: (groupid=0, jobs=40): err= 0: pid=20867: Tue Nov 24 18:20:08 2015 read : io=5219.1GB, bw=89079MB/s, iops=356314, runt= 60006msec cpu : usr=1.74%, sys=96.16%, ctx=49576, majf=0, minf=21997 Run status group 0 (all jobs): READ: io=5219.1GB, aggrb=89079MB/s, minb=89079MB/s, maxb=89079MB/s, mint=60006msec, maxt=60006msec |---------------------------------------||---------------------------------------| |-- Socket 0 --||-- Socket 1 --| |---------------------------------------||---------------------------------------| |-- Memory Channel Monitoring --||-- Memory Channel Monitoring --| |---------------------------------------||---------------------------------------| |-- Mem Ch 0: Reads (MB/s): 11778.11 --||-- Mem Ch 0: Reads (MB/s): 11743.99 --| |-- Writes(MB/s): 51.83 --||-- Writes(MB/s): 43.25 --| |-- Mem Ch 1: Reads (MB/s): 11779.90 --||-- Mem Ch 1: Reads (MB/s): 11736.06 --| |-- Writes(MB/s): 48.73 --||-- Writes(MB/s): 37.86 --| |-- Mem Ch 4: Reads (MB/s): 11784.79 --||-- Mem Ch 4: Reads (MB/s): 11746.94 --| |-- Writes(MB/s): 52.90 --||-- Writes(MB/s): 43.73 --| |-- Mem Ch 5: Reads (MB/s): 11778.48 --||-- Mem Ch 5: Reads (MB/s): 11741.55 --| |-- Writes(MB/s): 47.62 --||-- Writes(MB/s): 37.80 --| |-- NODE 0 Mem Read (MB/s) : 47121.27 --||-- NODE 1 Mem Read (MB/s) : 46968.53 --| |-- NODE 0 Mem Write(MB/s) : 201.08 --||-- NODE 1 Mem Write(MB/s) : 162.65 --| |-- NODE 0 P. Write (T/s): 190927 --||-- NODE 1 P. Write (T/s): 182961 --| |-- NODE 0 Memory (MB/s): 47322.36 --||-- NODE 1 Memory (MB/s): 47131.17 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- System Read Throughput(MB/s): 94089.80 --| |-- System Write Throughput(MB/s): 363.73 --| |-- System Memory Throughput(MB/s): 94453.52 --| |---------------------------------------||---------------------------------------| 1 MiB ===== |---------------------------------------||---------------------------------------| |-- Socket 0 --||-- Socket 1 --| |---------------------------------------||---------------------------------------| |-- Memory Channel Monitoring --||-- Memory Channel Monitoring --| |---------------------------------------||---------------------------------------| |-- Mem Ch 0: Reads (MB/s): 7227.83 --||-- Mem Ch 0: Reads (MB/s): 7047.45 --| |-- Writes(MB/s): 5894.47 --||-- Writes(MB/s): 6010.66 --| |-- Mem Ch 1: Reads (MB/s): 7229.32 --||-- Mem Ch 1: Reads (MB/s): 7041.79 --| |-- Writes(MB/s): 5891.38 --||-- Writes(MB/s): 6003.19 --| |-- Mem Ch 4: Reads (MB/s): 7230.70 --||-- Mem Ch 4: Reads (MB/s): 7052.44 --| |-- Writes(MB/s): 5888.63 --||-- Writes(MB/s): 6012.49 --| |-- Mem Ch 5: Reads (MB/s): 7229.16 --||-- Mem Ch 5: Reads (MB/s): 7047.19 --| |-- Writes(MB/s): 5882.45 --||-- Writes(MB/s): 6008.11 --| |-- NODE 0 Mem Read (MB/s) : 28917.01 --||-- NODE 1 Mem Read (MB/s) : 28188.87 --| |-- NODE 0 Mem Write(MB/s) : 23556.93 --||-- NODE 1 Mem Write(MB/s) : 24034.46 --| |-- NODE 0 P. Write (T/s): 238713 --||-- NODE 1 P. Write (T/s): 228040 --| |-- NODE 0 Memory (MB/s): 52473.94 --||-- NODE 1 Memory (MB/s): 52223.33 --| |---------------------------------------||---------------------------------------| |---------------------------------------||---------------------------------------| |-- System Read Throughput(MB/s): 57105.87 --| |-- System Write Throughput(MB/s): 47591.39 --| |-- System Memory Throughput(MB/s): 104697.27 --| |---------------------------------------||---------------------------------------| ????{.n?+???????+%?????ݶ??w??{.n?+????{??G?????{ay?ʇڙ?,j??f???h?????????z_??(?階?ݢj"???m??????G????????????&???~???iO???z??v?^?m????????????I?