Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Date:   Thu, 21 Sep 2023 16:24:57 +0800
From:   Yi Sun <yi.sun@intel.com>
To:     Ingo Molnar <mingo@kernel.org>
CC:     <dave.hansen@intel.com>, <linux-kernel@vger.kernel.org>,
        <x86@kernel.org>, <sohil.mehta@intel.com>, <ak@linux.intel.com>,
        <ilpo.jarvinen@linux.intel.com>, <heng.su@intel.com>,
        <tony.luck@intel.com>, <yi.sun@linux.intel.com>,
        <yu.c.chen@intel.com>
Subject: Re: [PATCH v7 1/3] x86/fpu: Measure the Latency of XSAVE and XRSTOR
Message-ID: <ZQv92QD7AtodfDPB@ysun46-mobl.ccr.corp.intel.com>
References: <20230921062900.864679-1-yi.sun@intel.com>
 <20230921062900.864679-2-yi.sun@intel.com>
 <ZQvqvpSbyub6gFZX@gmail.com>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Disposition: inline
In-Reply-To: <ZQvqvpSbyub6gFZX@gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SzVPcC9VcS9pSEE1Z0RwK3VjRk5kelVVQmRzdmhFTzZKazlzQmp6dDJHMklO?=
 =?utf-8?B?WFJMSFFpOVBuY1kxbWpwR3FJQWxFSHBtOTF2NTlDV2ovcmhjUEVONFB2SDRo?=
 =?utf-8?B?QXRtWVRQU2lCeTJEMUFiZTVwb2hoRXhaU2hVdi91bmRrK0pTekNNaUZxUUZS?=
 =?utf-8?B?MnZQTlFVWHBSVEt1UkxtSnRHbUJVMVNXalpxYWQ0NlBGSXNmQTRGVTU5dW1Z?=
 =?utf-8?B?cHZWZnVuOEc2MlFJamUrQ1l3Zkl5YjlaTFJEcTkxS2ZoYzRMQ0Fkb1hJWmFL?=
 =?utf-8?B?M1krVUJKVnQ4WndXSXhRRUFSYnRrcU5pS1RmU1BFM2o1cTY4Z09ZRGJGT1NR?=
 =?utf-8?B?WDdtUGwySGlsT0xWVU1NNXFsVkZVbHlWSUoxbjlaOXpVWFlMdGh2bTFnSUx4?=
 =?utf-8?B?Z1JxcFdja2JaOXg0MVdXMDJQRE96eVBUZkV1TzU0Nnh2dDRyT2VhcVZCVGpV?=
 =?utf-8?B?a2I5MWRZVCs4anEyVWpPaGVlYkFvYjczMzE1d2k2VTA4QTJXeFAyeVBJMUdI?=
 =?utf-8?B?RnNsMGxjNmw3ZE92dVRWMUhGbGtoMmxkM25yNFA1VUE3aGhwdlFzRXI1Vm5a?=
 =?utf-8?B?YVdtekdIc2F5MklFTy9wajIvYkZCUmtyRitYbW95RUdrNG0vZTdYZ3c2NHdY?=
 =?utf-8?B?VDdWazI5SGd3MTJkOVVIazNqejRJWkRxWWtEZlA2eXNDb1dJTmVtTkRiTHZK?=
 =?utf-8?B?c01Hb29jcVZzVjJiUzJ6UUUwUVF3ZTY1NHBhVEk4ZmcxK3NBSDhRaFFsT1g1?=
 =?utf-8?B?T1Y5bzgvellrc2kvZ25mN1Z5a25TMXowOUw5ZWNVZ0hiOVdEMlBVY0NidnZq?=
 =?utf-8?B?MERTNkZmM3N0Ulp3eUlwcEF4L2RyT2Rac0Z0YUFXOC9XdERZdWtEZHc1Szd2?=
 =?utf-8?B?azlsL2NZRGdpNUhua0xaVTBoYWJVSFdMRi9mbmlMa0ZtdTB4dWlwZWNGdXlj?=
 =?utf-8?B?Ym5RMEM4T2FLc2pjRnpYZ01xRWN1YitjMEp3TEZ2VDd6YXlSc2VaUmZNOWZJ?=
 =?utf-8?B?VTJjK1hyZktPbWlzNFRHRVlZRzJxTkk3cHJsUDF1ejE4emJMTzgvS0pJZmtJ?=
 =?utf-8?B?NFRsY3AxcElQMHU4a3hEYTVMUnl5c0gzbERTRGpjRlVxSzRhUzViVUZ1RXJi?=
 =?utf-8?B?d3dhY3MzbmhNQ0tkb1pGZnArSE1uQ3ZpOEpuYmYrbGVpTUduWmEydnI2UjNC?=
 =?utf-8?B?ZFBjYzVkWFdiOHlWeHBPVHNkTnFoVHpRQWU2a0JHQmxzekU5aXhCRDh6c1VD?=
 =?utf-8?B?UkZuOE5BUmN4eHVVSjNZR3FjUHBkQ1QzbE5GNUc1aEp2M2Y1SU5QQzlmamJG?=
 =?utf-8?B?cVJ3VWU1dUlxYUNMRlF5KzN2M3ZIQnFWUXAwZE9adDJQcVNScXBzQytFYjBw?=
 =?utf-8?B?Z0FPNEtiNjJTTFlEcWYzK1FFQkdOR0R2QlFTT0c4a0tBcjRKZXRNaDZMRjR6?=
 =?utf-8?B?WVB4SzdHZEhqb3kzdVFEdk9xVTd0MjFGaTRVVzJlTWRCL1Y4ZmlKRjFOQldk?=
 =?utf-8?B?UzlYd0dWcXp6ZmhUcHMvQmVFL3R3SXNqMnQ1c1JSLzhBUTNKSUZzdmg3eTkw?=
 =?utf-8?B?UVBBdnNrZjgvQUhraXM1Z2pFb3lrZGRsQlNPalF6YS9GWmVKVVBHWlNhczFl?=
 =?utf-8?B?QndaQk1POWVxeE5MS0Z3ZVFaVWlrSXBKbXl0VSszRDAvaU1tdnFYMnNTYlk0?=
 =?utf-8?B?VDUxei9wSE15MVdKbGJySUJrbWZ4NG0zQ3Q5UHVOSVZ3bU45SWltdnRWNEsv?=
 =?utf-8?B?V2hMR01reTVhd2JpNHF3cmJ4QTRNS3ZDell5Sk80bWNoblh0UWl1TU9rTUlT?=
 =?utf-8?B?a3dnV0NQVE5NTjlpLzduV0h1ZXpYQStNNU5nZWVaYWFYQkVoRm1EYWFVWVUr?=
 =?utf-8?B?NXVTM05HT0trMC9YSDByRlJWVjFzZU5CeW5XcHhuajBmRFVLQW5MMEJoM0Rw?=
 =?utf-8?B?WklmT0M3R0lZTzFJUFR6NFY4Wm5zR0RJMW82VUExSktsNGM5ZnpSODZIYU0v?=
 =?utf-8?B?WURTVEpTTGJ3VDFxUVB5RFdUNFFtekp1WkFTLzYwbGpKcCtqMXZEWWN3dFoy?=
 =?utf-8?B?NHhTWitaMHJhSGpLTEREeVZkaTRPQ1dZWVF4Z3JVZWw3b256NmlSeFlUdldk?=
 =?utf-8?Q?2lJHsyEvVOtyl1VxbL0Yir3Xy?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 438ee905-330d-421a-4f0e-08dbba7c4b5b
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3094.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Sep 2023 08:25:22.3534
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Sgl3ok5XLEFKbCyh3A2N7NshKNqeXYfAK3t78ufW24ZC7r4U2OpFaPhmD9TWr0KWqNtBcpOsmPGJd0BcBbrH1g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6666
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Thu, 21 Sep 2023 12:44:10 -0700 (PDT)

On 21.09.2023 09:03, Ingo Molnar wrote:
>
>* Yi Sun <yi.sun@intel.com> wrote:
>
>> -#define XSTATE_XRESTORE(st, lmask, hmask)				\
>> +#define __XSTATE_XRESTORE(st, lmask, hmask)				\
>>  	asm volatile(ALTERNATIVE(XRSTOR,				\
>>  				 XRSTORS, X86_FEATURE_XSAVES)		\
>>  		     "\n"						\
>> @@ -140,6 +143,35 @@ static inline u64 xfeatures_mask_independent(void)
>>  		     : "D" (st), "m" (*st), "a" (lmask), "d" (hmask)	\
>>  		     : "memory")
>>
>> +#if defined(CONFIG_X86_DEBUG_FPU)
>> +#define XSTATE_XSAVE(fps, lmask, hmask, err)				\
>> +	do {								\
>> +		struct fpstate *f = fps;				\
>> +		u64 tc = -1;						\
>> +		if (tracepoint_enabled(x86_fpu_latency_xsave))		\
>> +			tc = trace_clock();				\
>> +		__XSTATE_XSAVE(&f->regs.xsave, lmask, hmask, err);	\
>> +		if (tracepoint_enabled(x86_fpu_latency_xsave))		\
>> +			trace_x86_fpu_latency_xsave(f, trace_clock() - tc);\
>> +	} while (0)
>> +
>> +#define XSTATE_XRESTORE(fps, lmask, hmask)				\
>> +	do {								\
>> +		struct fpstate *f = fps;				\
>> +		u64 tc = -1;						\
>> +		if (tracepoint_enabled(x86_fpu_latency_xrstor))		\
>> +			tc = trace_clock();				\
>> +		__XSTATE_XRESTORE(&f->regs.xsave, lmask, hmask);	\
>> +		if (tracepoint_enabled(x86_fpu_latency_xrstor))		\
>> +			trace_x86_fpu_latency_xrstor(f, trace_clock() - tc);\
>
>This v7 version does not adequately address the review feedback I gave for
>v6: it adds tracing overhead to potential hot paths, and putting it behind
>CONFIG_X86_DEBUG_FPU is not a solution either: it's default-y, so de-facto
>enabled on all major distros...
Hi Ingo,

Appreciate your comments!

What do you think if I were to add a new config here instead of
CONFIG_X86_DEBUG_FPU, and set it as the default value of n?
This way, all the overhead will be removed from the hot paths.
```
#else
#define XSTATE_XSAVE(fps, lmask, hmask, err)
                 __XSTATE_XSAVE(&(fps)->regs.xsave, lmask, hmask, err)
#define XSTATE_XRESTORE(fps, lmask, hmask)
                 __XSTATE_XRESTORE(&(fps)->regs.xsave, lmask, hmask)
#endif
```
>
>It seems unnecessarily complex: why does it have to measure latency
>directly? Tracepoints *by default* come with event timestamps. A latency
>measurement tool should be able to subtract two timestamps to extract the
>latency between two tracepoints...
>
>In fact, function tracing is enabled on all major Linux distros:
>
>  kepler:~/tip> grep FUNCTION_TRACER /boot/config-6.2.0-33-generic
>  CONFIG_HAVE_FUNCTION_TRACER=y
>  CONFIG_FUNCTION_TRACER=y
>
>Why not just enable function tracing for the affected FPU context switching
>functions?
>
>The relevant functions are already standalone in a typical kernel:
>
>   xsave:                # ffffffff8103cfe0 T save_fpregs_to_fpstate
>   xrstor:               # ffffffff8103d160 T restore_fpregs_from_fpstate
>   xrstor_supervisor:    # ffffffff8103dc50 T fpu__clear_user_states
>
>... and FPU context switching overhead dominates the cost of these
>functions.
>
I am aware of these trace points, but we add new ones for two reasons:
1. As I described in the patch comments. It would be more accurate to
calculate the latency in a single trace event, which will NOT include the
latency of the trace functions themselves. The cost of the trace
functions is much longer than the latency of xsaves, by our experiments.

Calculating the latency by subtracting two timestamps has to
contain the latency of trace points, that would make the actual latency
data less prominent.

2. We want to measure with finer granularity. The new added trace points
dump XINUSE and RFBM each time they are called. By doing so, we can
collect the latency and analyze the data grouped according to each FPU
instruction indicated in the XCR0 bits.

Thanks
    --Sun, Yi