Received: by 2002:ac0:a679:0:0:0:0:0 with SMTP id p54csp1481046imp; Fri, 22 Feb 2019 04:32:02 -0800 (PST) X-Google-Smtp-Source: AHgI3Ia4mo6GKKPAEKGTGcCb5R8F2G5SsldUHLPwVseZXt33OAKKHBfuYh5UuPqtAVTF6k8V6WNY X-Received: by 2002:a17:902:2ba8:: with SMTP id l37mr3959021plb.243.1550838722595; Fri, 22 Feb 2019 04:32:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1550838722; cv=none; d=google.com; s=arc-20160816; b=1FR2yvqBUOZT0zSdTKjLcgwM+ROxZgEfiOyUZq9tEYOq7udY6ymnbX2yS7R15+1Z2a puilRI0TJVSztW1nNfLPB+LNYB9qqrxqEe8t8uUbZwy3S3z8XcgZSbInKATuyq4SdXL7 66xFpz+BkQ2QGM97ANbWg6NJd3U3p9dOtQTXYX8JAETrs9PLNRqlERgmRPjehOPHRExS sf4FLNuc4w9h9aN1WNY7LShhEiO4c0YSuJs2sMbaBorDnqNPx/DhBDgzIsBpCoDzP9+m CoSeJyuqqQ42Kf1+S801bpe8FprBykd4H1hKUox8L8TRXHGMnM6uXlnKRBviGoU4+XA9 bKsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding :content-language:in-reply-to:mime-version:user-agent:date :message-id:autocrypt:openpgp:from:references:cc:to:subject; bh=yEuvqAXOl7yxSvkTzPkAF4NQIiHunP25rhzvDnHXOiU=; b=rY3fTMmnuJj68cOqtZeSKXI4YxvhXB97wLX83mQpRE7HhWRqkT1at+qtmXo3Sd/T10 fHDm5OtpUkWCFO0f1xVJnyA5F7pN3/YlJbFJ1JrNawvIP7AkwuKDIzx65O+4G41A5XFs kWqtkx66vu1ZDLjiKr7sLjPCHHv7dOKcOzAsCXWIR3VgN2HlqfoeZeAkY+JhahIvNBdD NdJeccFSTgOfv6z4GtCSn7K41bkc+M1rJrLiviw4ap+eRrsrNqwoZdTSoaIS5hOwF6rL ZRK/eIX85hMOecLvH1O9eKo+G1jX562TK7ZBNwiIpObPlKzES0lXne/VtuEhFN+9708G j6zw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f21si1261686plr.104.2019.02.22.04.31.47; Fri, 22 Feb 2019 04:32:02 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726765AbfBVMbY (ORCPT + 99 others); Fri, 22 Feb 2019 07:31:24 -0500 Received: from mx1.redhat.com ([209.132.183.28]:49694 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726285AbfBVMbX (ORCPT ); Fri, 22 Feb 2019 07:31:23 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CE7053093417; Fri, 22 Feb 2019 12:31:22 +0000 (UTC) Received: from [10.36.112.23] (ovpn-112-23.ams2.redhat.com [10.36.112.23]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7320C600CD; Fri, 22 Feb 2019 12:31:19 +0000 (UTC) Subject: Re: recalibrating x86 TSC during suspend/resume To: Thomas Gleixner , Olaf Hering Cc: John Stultz , Stephen Boyd , LKML , x86@kernel.org References: <20190222105302.GA26398@aepfle.de> From: Paolo Bonzini Openpgp: preference=signencrypt Autocrypt: addr=pbonzini@redhat.com; prefer-encrypt=mutual; keydata= mQHhBFRCcBIBDqDGsz4K0zZun3jh+U6Z9wNGLKQ0kSFyjN38gMqU1SfP+TUNQepFHb/Gc0E2 CxXPkIBTvYY+ZPkoTh5xF9oS1jqI8iRLzouzF8yXs3QjQIZ2SfuCxSVwlV65jotcjD2FTN04 hVopm9llFijNZpVIOGUTqzM4U55sdsCcZUluWM6x4HSOdw5F5Utxfp1wOjD/v92Lrax0hjiX DResHSt48q+8FrZzY+AUbkUS+Jm34qjswdrgsC5uxeVcLkBgWLmov2kMaMROT0YmFY6A3m1S P/kXmHDXxhe23gKb3dgwxUTpENDBGcfEzrzilWueOeUWiOcWuFOed/C3SyijBx3Av/lbCsHU Vx6pMycNTdzU1BuAroB+Y3mNEuW56Yd44jlInzG2UOwt9XjjdKkJZ1g0P9dwptwLEgTEd3Fo UdhAQyRXGYO8oROiuh+RZ1lXp6AQ4ZjoyH8WLfTLf5g1EKCTc4C1sy1vQSdzIRu3rBIjAvnC tGZADei1IExLqB3uzXKzZ1BZ+Z8hnt2og9hb7H0y8diYfEk2w3R7wEr+Ehk5NQsT2MPI2QBd wEv1/Aj1DgUHZAHzG1QN9S8wNWQ6K9DqHZTBnI1hUlkp22zCSHK/6FwUCuYp1zcAEQEAAbQj UGFvbG8gQm9uemluaSA8cGJvbnppbmlAcmVkaGF0LmNvbT6JAg0EEwECACMFAlRCcBICGwMH CwkIBwMCAQYVCAIJCgsEFgIDAQIeAQIXgAAKCRB+FRAMzTZpsbceDp9IIN6BIA0Ol7MoB15E 11kRz/ewzryFY54tQlMnd4xxfH8MTQ/mm9I482YoSwPMdcWFAKnUX6Yo30tbLiNB8hzaHeRj jx12K+ptqYbg+cevgOtbLAlL9kNgLLcsGqC2829jBCUTVeMSZDrzS97ole/YEez2qFpPnTV0 VrRWClWVfYh+JfzpXmgyhbkuwUxNFk421s4Ajp3d8nPPFUGgBG5HOxzkAm7xb1cjAuJ+oi/K CHfkuN+fLZl/u3E/fw7vvOESApLU5o0icVXeakfSz0LsygEnekDbxPnE5af/9FEkXJD5EoYG SEahaEtgNrR4qsyxyAGYgZlS70vkSSYJ+iT2rrwEiDlo31MzRo6Ba2FfHBSJ7lcYdPT7bbk9 AO3hlNMhNdUhoQv7M5HsnqZ6unvSHOKmReNaS9egAGdRN0/GPDWr9wroyJ65ZNQsHl9nXBqE AukZNr5oJO5vxrYiAuuTSd6UI/xFkjtkzltG3mw5ao2bBpk/V/YuePrJsnPFHG7NhizrxttB nTuOSCMo45pfHQ+XYd5K1+Cv/NzZFNWscm5htJ0HznY+oOsZvHTyGz3v91pn51dkRYN0otqr bQ4tlFFuVjArBZcapSIe6NV8C4cEiSS5AQ0EVEJxcwEIAK+nUrsUz3aP2aBjIrX3a1+C+39R nctpNIPcJjFJ/8WafRiwcEuLjbvJ/4kyM6K7pWUIQftl1P8Woxwb5nqL7zEFHh5I+hKS3haO 5pgco//V0tWBGMKinjqntpd4U4Dl299dMBZ4rRbPvmI8rr63sCENxTnHhTECyHdGFpqSzWzy 97rH68uqMpxbUeggVwYkYihZNd8xt1+lf7GWYNEO/QV8ar/qbRPG6PEfiPPHQd/sldGYavmd //o6TQLSJsvJyJDt7KxulnNT8Q2X/OdEuVQsRT5glLaSAeVAABcLAEnNgmCIGkX7TnQF8a6w gHGrZIR9ZCoKvDxAr7RP6mPeS9sAEQEAAYkDEgQYAQIACQUCVEJxcwIbAgEpCRB+FRAMzTZp scBdIAQZAQIABgUCVEJxcwAKCRC/+9JfeMeug/SlCACl7QjRnwHo/VzENWD9G2VpUOd9eRnS DZGQmPo6Mp3Wy8vL7snGFBfRseT9BevXBSkxvtOnUUV2YbyLmolAODqUGzUI8ViF339poOYN i6Ffek0E19IMQ5+CilqJJ2d5ZvRfaq70LA/Ly9jmIwwX4auvXrWl99/2wCkqnWZI+PAepkcX JRD4KY2fsvRi64/aoQmcxTiyyR7q3/52Sqd4EdMfj0niYJV0Xb9nt8G57Dp9v3Ox5JeWZKXS krFqy1qyEIypIrqcMbtXM7LSmiQ8aJRM4ZHYbvgjChJKR4PsKNQZQlMWGUJO4nVFSkrixc9R Z49uIqQK3b3ENB1QkcdMg9cxsB0Onih8zR+Wp1uDZXnz1ekto+EivLQLqvTjCCwLxxJafwKI bqhQ+hGR9jF34EFur5eWt9jJGloEPVv0GgQflQaE+rRGe+3f5ZDgRe5Y/EJVNhBhKcafcbP8 MzmLRh3UDnYDwaeguYmxuSlMdjFL96YfhRBXs8tUw6SO9jtCgBvoOIBDCxxAJjShY4KIvEpK b2hSNr8KxzelKKlSXMtB1bbHbQxiQcerAipYiChUHq1raFc3V0eOyCXK205rLtknJHhM5pfG 6taABGAMvJgm/MrVILIxvBuERj1FRgcgoXtiBmLEJSb7akcrRlqe3MoPTntSTNvNzAJmfWhd SvP0G1WDLolqvX0OtKMppI91AWVu72f1kolJg43wbaKpRJg1GMkKEI3H+jrrlTBrNl/8e20m TElPRDKzPiowmXeZqFSS1A6Azv0TJoo9as+lWF+P4zCXt40+Zhh5hdHO38EV7vFAVG3iuay6 7ToF8Uy7tgc3mdH98WQSmHcn/H5PFYk3xTP3KHB7b0FZPdFPQXBZb9+tJeZBi9gMqcjMch+Y R8dmTcQRQX14bm5nXlBF7VpSOPZMR392LY7wzAvRdhz7aeIUkdO7VelaspFk2nT7wOj1Y6uL nRxQlLkBDQRUQnHuAQgAx4dxXO6/Zun0eVYOnr5GRl76+2UrAAemVv9Yfn2PbDIbxXqLff7o yVJIkw4WdhQIIvvtu5zH24iYjmdfbg8iWpP7NqxUQRUZJEWbx2CRwkMHtOmzQiQ2tSLjKh/c HeyFH68xjeLcinR7jXMrHQK+UCEw6jqi1oeZzGvfmxarUmS0uRuffAb589AJW50kkQK9VD/9 QC2FJISSUDnRC0PawGSZDXhmvITJMdD4TjYrePYhSY4uuIV02v028TVAaYbIhxvDY0hUQE4r 8ZbGRLn52bEzaIPgl1p/adKfeOUeMReg/CkyzQpmyB1TSk8lDMxQzCYHXAzwnGi8WU9iuE1P 0wARAQABiQHzBBgBAgAJBQJUQnHuAhsMAAoJEH4VEAzNNmmxp1EOoJy0uZggJm7gZKeJ7iUp eX4eqUtqelUw6gU2daz2hE/jsxsTbC/w5piHmk1H1VWDKEM4bQBTuiJ0bfo55SWsUNN+c9hh IX+Y8LEe22izK3w7mRpvGcg+/ZRG4DEMHLP6JVsv5GMpoYwYOmHnplOzCXHvmdlW0i6SrMsB Dl9rw4AtIa6bRwWLim1lQ6EM3PWifPrWSUPrPcw4OLSwFk0CPqC4HYv/7ZnASVkR5EERFF3+ 6iaaVi5OgBd81F1TCvCX2BEyIDRZLJNvX3TOd5FEN+lIrl26xecz876SvcOb5SL5SKg9/rCB ufdPSjojkGFWGziHiFaYhbuI2E+NfWLJtd+ZvWAAV+O0d8vFFSvriy9enJ8kxJwhC0ECbSKF Y+W1eTIhMD3aeAKY90drozWEyHhENf4l/V+Ja5vOnW+gCDQkGt2Y1lJAPPSIqZKvHzGShdh8 DduC0U3xYkfbGAUvbxeepjgzp0uEnBXfPTy09JGpgWbg0w91GyfT/ujKaGd4vxG2Ei+MMNDm S1SMx7wu0evvQ5kT9NPzyq8R2GIhVSiAd2jioGuTjX6AZCFv3ToO53DliFMkVTecLptsXaes uUHgL9dKIfvpm+rNXRn9wAwGjk0X/A== Message-ID: <036fab04-0c1b-134d-a170-399b2dc6ab5f@redhat.com> Date: Fri, 22 Feb 2019 13:31:15 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Fri, 22 Feb 2019 12:31:22 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 22/02/19 12:44, Thomas Gleixner wrote: >> The specific usecase I have is a workload within VMs that makes heavy >> use of TSC. The kernel is booted with 'clocksource=tsc highres=off nohz=off' >> because only this clocksource gives enough granularity. The default >> paravirtualized clock will return the same values via >> clock_gettime(CLOCK_MONOTONIC) if the timespan between two calls is too >> short. This does not happen with 'clocksource=tsc'. This shouldn't happen. clock_gettime(CLOCK_MONOTONIC) should be monotonic increasing. Do you have a testcase? The KVM clocksource is high-resolution and also TSC-based, the difference is that it performs two multiplications instead of one. The first uses TSC parameters from the host. The second, which is the one in arch/x86/entry/vdso/vclock_gettime.c's do_hres function, will have a 1:1 multiplier (excluding adjtime shearing) because kvmclock already returns nanoseconds. > Newer Intels support TSC scaling for VMX, which could solve the problem. It > affects TSC readout by: > > TSC = (read(HWTSC) * multiplier) >> 48 > > So you can standarize on a TSC frequency accross a fleet. Not sure when > that was introduced and no idea whether it's available on AMD. It's Skylake (server parts only) or newer. AMD instead has had it (almost) forever. QEMU 2.6 or newer will use it automatically across live migration, if available. > For a software solution we could try the following: > > 1) Provide the raw TSC frequency of the host to the guest in some magic > software defined MSR or CPUID. If there is an existing mechanism, use > that. This shouldn't be needed for two reasons: 1) you could also use kvmclock's provided mult/shift 2) I am not convinced that kvmclock has the behavior that Olaf mentions, and if it does it would be a bug. Paolo