Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <41375c87-f367-3632-051f-64c8b283737c@meta.com>
Date:   Wed, 12 Jul 2023 13:21:52 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.13.0
Subject: Re: [PATCH v2] kallsyms: strip LTO-only suffixes from promoted global
 functions
From:   Yonghong Song <yhs@meta.com>
To:     Nick Desaulniers <ndesaulniers@google.com>,
        Yonghong Song <yhs@fb.com>, Kees Cook <keescook@chromium.org>
Cc:     Petr Mladek <pmladek@suse.com>, Song Liu <song@kernel.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        Fangrui Song <maskray@google.com>, kernel-team@fb.com,
        Leizhen <thunder.leizhen@huawei.com>,
        linux-kernel@vger.kernel.org, llvm@lists.linux.dev,
        Sami Tolvanen <samitolvanen@google.com>,
        "KE . LI" <like1@oppo.com>,
        Padmanabha Srinivasaiah <treasure4paddy@gmail.com>
References: <20230628181926.4102448-1-yhs@fb.com>
 <CAKwvOdm_Cp9dF1yseTXvOOtFMsOPzCBWhfszAt0k5SXXyFyjtw@mail.gmail.com>
 <233802b0-7404-1f79-39ae-385da567f657@meta.com>
Content-Language: en-US
In-Reply-To: <233802b0-7404-1f79-39ae-385da567f657@meta.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MFIvcnc0a0F2Mml2RVFhSlYxZWdiTDN3RmYzR0ZLM3VISUc4WHRRT0g3TGJW?=
 =?utf-8?B?eWVlY1JkT08xbG1PbkdiMDJwNFlVWnJtM3NaWDdUTzBqU1dyNHFNNjhscWZC?=
 =?utf-8?B?RGp1SnpXZGNQRE5OeUY2cFQwclMyUXdoSGVBTGhiRDF2L0xQQWJEQkdid01y?=
 =?utf-8?B?MEs4V2d3OFRrZ21zTmdwcUR6SEhHcTZzdkpXUGRvdWw2MDJFZFlLbTk2a1pq?=
 =?utf-8?B?WHlEUGl5Q3krT1NGMTVRSlZjeURPZGp3Y2FDVUNQWCtmZW5MQi8vcXc2RENH?=
 =?utf-8?B?YU9GWjNLTDUyZXZWdTdDQVF5azcvTTlHOEdMRVpEM25XcGhPWnhKbVFXZkIw?=
 =?utf-8?B?OFNzMENLeUYrSXROTE5yOE9aZ2JxT3QwV0FpOFhWMzU3S1YzZlRDUnBtRUZi?=
 =?utf-8?B?cGVKMlBmOWdaeVdwZ01DNkhHdFVRM2FRbFEyaElOZ3hzSG5iUzJkUFc2WGlR?=
 =?utf-8?B?cGRyRWN1Y2d3THFiUTlkLys5NE5TNjlNNDhreGdJa29GL09rZHBCM2hWUkt4?=
 =?utf-8?B?WVRyTEhxem5sbHh5eXdKVUhlQVlVT0xpMTE3UFU0bVplbFgxQm1zYkRXQlUz?=
 =?utf-8?B?SnYzdGsvcG9tdmlyYy9xVzFxOE1NeXB3Q2Jmekt6YUFwV0k1RFFQWlhkTHdP?=
 =?utf-8?B?YXdVVEtOaklrSmlYTENoUGprTW5TUmRQMlZPazljaUdWbnNFSWJOY1dHSm9G?=
 =?utf-8?B?RGJTOTBGcHg3MDU0Rmp3bFptSmgwV3M1YjlCbUNVU2R0Yk82ajRpZ0thbWJs?=
 =?utf-8?B?R25kQkRYWXRmVWNVbVY4SGsyTVVFdHQ0eHl3MFVoRUZONU5XSk14RWs1Q0Ey?=
 =?utf-8?B?V25BN0lRNWluUW91YysvbzZFNDgvdGpVVjN5Qkp0cXUxWXc2dHZmOSthTUJI?=
 =?utf-8?B?RTRaWGxZTEs5MUhDMmhGYVJ6L2JxTkVUMzRGRlZQVmNTLy9PNE03ZzVLd2hJ?=
 =?utf-8?B?c1B4aWhCYjk1VFlBbjBuVVJzVlNqMGNrWnd0NHNrMGxrQXZQVEZlK0ora2t4?=
 =?utf-8?B?YlpEM1FTaGtUTWJqTUR4K3lEZktPdGMreVZ2T05jd0tyOUtjNDhuTlh3VW5j?=
 =?utf-8?B?VXdsYTlveS9kV29DTDFOeGYvZk9DNi9aNng0d0dzNEplcitHcERQTHI4Y1Bx?=
 =?utf-8?B?dS9qeVRtZUJtZ2VaWlBCeWxWbUpQMm4ya3I0Q0JmR0pOajY3VnNIRVZ0TGpa?=
 =?utf-8?B?YUJXamtzMUtCeUYyVXh2TFg1TWg0K0FrcytHY0tkdHRUa2NHQ25hWEFjNVRs?=
 =?utf-8?B?eUtVWnFuaGlEUFpPOTk1TEMvTUlER2hqampPdTRUM2M3OUloQS82Vzg1WjJi?=
 =?utf-8?B?aHY0SDdmeUdDSit5RE9KaFIyeGUrMkk2WFBTWk43cWJoOFRwUnpUckltR2ov?=
 =?utf-8?B?RW1kckpadFpJZTBRb3hza0NKZTlGSjNXa3NFQmZOaFBNRzRyU2tVc3dlcDRl?=
 =?utf-8?B?QW1aTkpnSWhlcFB1a25lV2hodW1xbzZtREJCdzRsNXQ4dll4Z2RvNEk3ME1P?=
 =?utf-8?B?K0QwcytSc3FYZi9DS2tweTZPdENHL1kycmcwNC9BWmlCTFJVR2s5ckFxVHZi?=
 =?utf-8?B?NlEyOG9TNDhTZjN5cFRxSGpPVkdVTWlZcmNDUmRwelBtY2lqYStSbzR3aXg3?=
 =?utf-8?B?REo4OER5WWMrdkpYU2MySzZ3dzI1SkoyaW8vRW1SQjRZZmhCQlBGbXZwdU9E?=
 =?utf-8?B?aXd1b2FvaVVLZnVmN0ZublBsY001MTRMVXNBaXZqQ014YVoxTHV6R2h0cDhp?=
 =?utf-8?B?d2t6MXY5S0phbW5qdVpGQkJIZ1dTU1kxRGlGdkZFcXJXeXVzRkdjZ2xPWVhF?=
 =?utf-8?B?K3VBanFmQWJOWXhkcFp6SThkUGtKOUlzeUV0VUpmMmo0eU93QlNJdFNFZ3Ir?=
 =?utf-8?B?Zk1MUVR6TXE4SHZsc3pjaFVVY216d0lRa1ZkcGxJRnpTL3o2SFNkckY1bG9r?=
 =?utf-8?B?aUlWenMza0JPc1UzTllsV1NYcGpieGNKVDVuQlRITFU3Y0NjZkFER2V2VzBr?=
 =?utf-8?B?T1JLeXVEY1M2QTFqcXJzdWREdWtUQXBoNG54d1BLMjE5VGVYbUM0NWNuUW15?=
 =?utf-8?B?SmtTNXhMNHRQTHVpbktFaW5JSTVLVGp0cVFyR1hwSXJhZTBBeUVmMDBjT1gw?=
 =?utf-8?B?NWx0MnJGRGVoV2YrdHdNY0NGc2htaUh5WFlFRG4wWCtuT2NQRExWNzdUSFZR?=
 =?utf-8?B?akE9PQ==?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5edb8ecd-af66-47d3-8769-08db8315a34e
X-MS-Exchange-CrossTenant-AuthSource: SN6PR1501MB2064.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2023 20:21:56.2486
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mTUp7Jh4PbjBpKIl+f6zPwqRoXGtGOr3+Do6rt6CAHbfLmKm0ngtl9CkwbcwfV0W
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR15MB4341
X-Proofpoint-GUID: X49DNP_Ytk07ixz8e1fnHLWQ25l_fRFM
X-Proofpoint-ORIG-GUID: X49DNP_Ytk07ixz8e1fnHLWQ25l_fRFM
Content-Transfer-Encoding: 8bit
X-Proofpoint-UnRewURL: 0 URL was un-rewritten
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-07-12_15,2023-07-11_01,2023-05-22_02
X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 7/7/23 5:23 PM, Yonghong Song wrote:
> 
> 
> On 7/7/23 3:27 PM, Nick Desaulniers wrote:
>> On Wed, Jun 28, 2023 at 11:19 AM Yonghong Song <yhs@fb.com> wrote:
>>>
>>> Commit 6eb4bd92c1ce ("kallsyms: strip LTO suffixes from static 
>>> functions")
>>> stripped all function/variable suffixes started with '.' regardless
>>> of whether those suffixes are generated at LTO mode or not. In fact,
>>> as far as I know, in LTO mode, when a static function/variable is
>>> promoted to the global scope, '.llvm.<...>' suffix is added.
>>>
>>> The existing mechanism breaks live patch for a LTO kernel even if
>>> no <symbol>.llvm.<...> symbols are involved. For example, for the 
>>> following
>>> kernel symbols:
>>>    $ grep bpf_verifier_vlog /proc/kallsyms
>>>    ffffffff81549f60 t bpf_verifier_vlog
>>>    ffffffff8268b430 d bpf_verifier_vlog._entry
>>>    ffffffff8282a958 d bpf_verifier_vlog._entry_ptr
>>>    ffffffff82e12a1f d bpf_verifier_vlog.__already_done
>>> 'bpf_verifier_vlog' is a static function. '_entry', '_entry_ptr' and
>>> '__already_done' are static variables used inside 'bpf_verifier_vlog',
>>> so llvm promotes them to file-level static with prefix 
>>> 'bpf_verifier_vlog.'.
>>> Note that the func-level to file-level static function promotion also
>>> happens without LTO.
>>>
>>> Given a symbol name 'bpf_verifier_vlog', with LTO kernel, current 
>>> mechanism will
>>> return 4 symbols to live patch subsystem which current live patching
>>> subsystem cannot handle it. With non-LTO kernel, only one symbol
>>> is returned.
>>>
>>> In [1], we have a lengthy discussion, the suggestion is to separate two
>>> cases:
>>>    (1). new symbols with suffix which are generated regardless of 
>>> whether
>>>         LTO is enabled or not, and
>>>    (2). new symbols with suffix generated only when LTO is enabled.
>>>
>>> The cleanup_symbol_name() should only remove suffixes for case (2).
>>> Case (1) should not be changed so it can work uniformly with or 
>>> without LTO.
>>>
>>> This patch removed LTO-only suffix '.llvm.<...>' so live patching and
>>> tracing should work the same way for non-LTO kernel.
>>> The cleanup_symbol_name() in scripts/kallsyms.c is also changed to 
>>> have the same
>>> filtering pattern so both kernel and kallsyms tool have the same
>>> expectation on the order of symbols.
>>>
>>>   [1] 
>>> https://lore.kernel.org/live-patching/20230615170048.2382735-1-song@kernel.org/T/#u
>>>
>>> Fixes: 6eb4bd92c1ce ("kallsyms: strip LTO suffixes from static 
>>> functions")
>>> Reported-by: Song Liu <song@kernel.org>
>>> Signed-off-by: Yonghong Song <yhs@fb.com>
>>
>> Thanks for the patch and improving live patch with LTO.  Looking back
>> at the internal report that resulted in
>> commit 6eb4bd92c1ce ("kallsyms: strip LTO suffixes from static 
>> functions")
>> your version was what I originally had.  I did not leave a comment as
>> to why I changed it when I sent it 2 years ago, and no longer recall
>> the reason.
>>
>>
>> Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
> 
> Nick, thanks for the review. I am not sure how this patch could
> be merged into the mainline. I checked your patch
>     6eb4bd92c1ce ("kallsyms: strip LTO suffixes from static functions")
> and it looks like the patch was merged by Kees Cook.
> 
> I added Kees in the 'To' list. Kees, could you help merge
> this patch if you are okay with the change?

Looks like Kees might be on vacation. Nick, Petr, do you
have any suggestions who could take this patch? Thanks!

> 
> Thanks!
> 
> Yonghong
> 
>>
>>> ---
>>>   kernel/kallsyms.c  | 5 ++---
>>>   scripts/kallsyms.c | 6 +++---
>>>   2 files changed, 5 insertions(+), 6 deletions(-)
>>>
>>> Changelogs:
>>>    v1 -> v2:
>>>      . add 'Reported-by: Song Liu <song@kernel.org>'
>>>      . also fix in scripts/kallsyms.c.
>>>
>>> diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c
>>> index 77747391f49b..4874508bb950 100644
>>> --- a/kernel/kallsyms.c
>>> +++ b/kernel/kallsyms.c
>>> @@ -174,11 +174,10 @@ static bool cleanup_symbol_name(char *s)
>>>           * LLVM appends various suffixes for local functions and 
>>> variables that
>>>           * must be promoted to global scope as part of LTO.  This 
>>> can break
>>>           * hooking of static functions with kprobes. '.' is not a valid
>>> -        * character in an identifier in C. Suffixes observed:
>>> +        * character in an identifier in C. Suffixes only in LLVM LTO 
>>> observed:
>>>           * - foo.llvm.[0-9a-f]+
>>> -        * - foo.[0-9a-f]+
>>>           */
>>> -       res = strchr(s, '.');
>>> +       res = strstr(s, ".llvm.");
>>>          if (res) {
>>>                  *res = '\0';
>>>                  return true;
>>> diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c
>>> index 0d2db41177b2..13af6d0ff845 100644
>>> --- a/scripts/kallsyms.c
>>> +++ b/scripts/kallsyms.c
>>> @@ -346,10 +346,10 @@ static void cleanup_symbol_name(char *s)
>>>           * ASCII[_]   = 5f
>>>           * ASCII[a-z] = 61,7a
>>>           *
>>> -        * As above, replacing '.' with '\0' does not affect the main 
>>> sorting,
>>> -        * but it helps us with subsorting.
>>> +        * As above, replacing the first '.' in ".llvm." with '\0' 
>>> does not
>>> +        * affect the main sorting, but it helps us with subsorting.
>>>           */
>>> -       p = strchr(s, '.');
>>> +       p = strstr(s, ".llvm.");
>>>          if (p)
>>>                  *p = '\0';
>>>   }
>>> -- 
>>> 2.34.1
>>>
>>
>>