Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <e9212ccf-b02d-c2d0-f45f-a94ec2b82c5b@meta.com>
Date:   Mon, 1 May 2023 10:20:21 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.10.1
Subject: Re: [PATCH v2] libbpf: Improve version handling when attaching uprobe
Content-Language: en-US
To:     Espen Grindhaug <espen.grindhaug@gmail.com>
Cc:     Andrii Nakryiko <andrii@kernel.org>,
        Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Martin KaFai Lau <martin.lau@linux.dev>,
        Song Liu <song@kernel.org>, Yonghong Song <yhs@fb.com>,
        John Fastabend <john.fastabend@gmail.com>,
        KP Singh <kpsingh@kernel.org>,
        Stanislav Fomichev <sdf@google.com>,
        Hao Luo <haoluo@google.com>, Jiri Olsa <jolsa@kernel.org>,
        Mykola Lysenko <mykolal@fb.com>, Shuah Khan <shuah@kernel.org>,
        bpf@vger.kernel.org, linux-kernel@vger.kernel.org,
        linux-kselftest@vger.kernel.org
References: <ZEV/EzOM+TJomP66@eg>
 <07e05c57-feb7-5482-9c07-eb41f976f9fd@meta.com> <ZErK0M/qAi/VS8U8@eg>
 <0e7b884a-05c3-83c7-0de1-9bfe14ff77b3@meta.com> <ZE+4Ct7ZMecFy7YV@eg>
 <533437a4-a76d-96e0-b04a-ab8eb7b5fb7f@meta.com> <ZE/pIM/z7x+35KQo@eg>
From:   Yonghong Song <yhs@meta.com>
In-Reply-To: <ZE/pIM/z7x+35KQo@eg>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?T3ZiUVVQZ1gyNWxGYmxjRytnSEx6TVpHNStLd09pRWdIWENhdXkzSkR6OUdH?=
 =?utf-8?B?cXA4TCtseWxPWEc5T2pVdGVXUGdRWlFtT3FqcHhiaEYyYVFscnBNTHJCNk8r?=
 =?utf-8?B?SEFNZ2IraXFjOE1IQkQxSWtEYjRPaE1wSVE1QjFpWUJleFY1ajVjakxrNktK?=
 =?utf-8?B?WURUZDB5V2I0TVRKODg3OUtoaDlUazNFTDhLZmVjd3BPSVJ1V3lkcVAvVkJi?=
 =?utf-8?B?bXJwWlZjdml0SmN2aVFWdXVWZlJjOFROejBhNU0vb2M4Y1JGWVYvTTdHVDhw?=
 =?utf-8?B?Y0FqU3hDVXVLNU5NWit3ZlZycTJnakZTTGRLVXEzYVEwVWZkYW9pTy9PYVkr?=
 =?utf-8?B?aStYYytwQVlVQkxuVk9zZUJDVmh3azNRU2NMSDh1ekE2c1VudWJwWjNxcTVF?=
 =?utf-8?B?U09sMjkvRDFaYXVVTk9HSXJGMWVvd0VaaGx1bkRBVG5BdmtteHZWNUcxUWxJ?=
 =?utf-8?B?RzFTUUpXcmxMTVVhQThkM3VNVkI5RWpOT1VNYm5PUWF1ZnZkUkpHQVc5aUdt?=
 =?utf-8?B?OW5HcnQ1ZmpSQnpwbXZPZVRPVWhBVGtUSmdqWnhzWVYzcEVpbW5tZ1ZQbkJZ?=
 =?utf-8?B?ZUE4andQVmhvT0hOMUVDeHNSK2IzMTVoOVRPOW5zN0orMUZSVVdKaHJxczAz?=
 =?utf-8?B?VHAyejhCbjBZdExwR0l3L3hha2tYOWJ2RnBWOHZLeDJZTkJIc2pSVnRYbndR?=
 =?utf-8?B?QkF4d3B1Nk1xMnpuMDBCTTFyekFvNjFIeWF5Sy9hRDNJWVZTdmMzUFNNdXdw?=
 =?utf-8?B?U21iKzlkb1NHQlpHamtiWFFPampQTjZSM0l5WStPQ1g0T0RHenZ1N0ZmNlVp?=
 =?utf-8?B?SnZBKzZLTk1yaTlEZ2Vkb2ROTER4Yy9veXdLQmthTEdLem91V1B4NVVyS2Rv?=
 =?utf-8?B?b3NWNVl4T1lFQUw5dWlEMTZkczBUeTA0NTN0TzkvSjZMZ1JSKzQ5UUVDMlJm?=
 =?utf-8?B?a0FQN1Bkc3FXNHdrb21jRGs1UDF1d2lETzRXa0VMM0dvMGpPZ2xrOUZ0TUZT?=
 =?utf-8?B?MG84bGxQQ01KZkdwTmdiNUVQcVlDb2Y0VUl4bGYrZlJSVUhHV0tjRm1pNjMv?=
 =?utf-8?B?OWVTd2pIa2xjUnRTdFl6SXh6Z3lkRUVEYlJLZlhxSHZ1c1FXT1NWdjJ2dUZX?=
 =?utf-8?B?ZndOcXVmYzhKZGg2UzhkVmlENk5wTW5YRlc0RTJ1Q0NYamVhQWd6VWxBVVNF?=
 =?utf-8?B?UWozc3RrNXhrV0MrbURObUFmMlM1Wlh1ODdmc2JaZDN6cnZPQitiN2dDUzhl?=
 =?utf-8?B?TS9oZENLZ3IzSmd3TkdCZ1RoSitWenhrQno4SlhHZDR2OExXVjVXRC90N0xz?=
 =?utf-8?B?ZlBEQkZGVTh5MDR6N2I4dnJ0ZEpMbFdRRlJsOFFpYkJPMWZhQWFzSWtxTmZ3?=
 =?utf-8?B?UEFjUjB6QmROamN1WDl4ZDBIY0dBSm1tZDhsSVNqOW9hb1FDdTdoOFdmTEdF?=
 =?utf-8?B?SEZmYnJXWlhjYTBBMWtuUHJSTW5oTHpnZXBIYVI2cWhOSXVZM0xlbmRibkVm?=
 =?utf-8?B?M2NHQUxXTW02SzJObk9mMUZuR1VjdGliRk5NOXA0U1ZBYlpVRnMrbkxrTmVq?=
 =?utf-8?B?K1ZlbUNjcndXNkR6enEwOEZiMGQyZ0FOSTZHaFhFVnZ5WEVsQ1c1MU94emhJ?=
 =?utf-8?B?TnNBRmtXSWpkSDdKaC9SV045aU4zTDdzalc5RXZNckpyZ0EvbjB0VWxZU240?=
 =?utf-8?B?bEZIcDNUU1hxTzN0V1hia0hpb1BrcFNQcmVCRW11OXpnRmlGZGl2b3BNdGZ1?=
 =?utf-8?B?UzFXcTVzcW1DM2M4cVV1cjE1dDVrOTYrekRJSXljL3A5cnhMNjFYUHliRFRM?=
 =?utf-8?B?SlkrcDI3d01zbFV2THVSazh3TnNMdjRGWVlBbEZETlFwUTJWdVNWZmlJTjhG?=
 =?utf-8?B?cTd3aXkvZzhTbDlJRCswaFQrT00zUG5td2Y2Z1ltdng1M1NFMDE1c2VsT1lH?=
 =?utf-8?B?MUVreXFEalRVemkrbnFNMjNqQTFRS0NKamlkeS9saVBrWXZGUU5qNXpRSTN3?=
 =?utf-8?B?UjhvaU0wTXB2RVpSclRUZ3N2VXpoNUUrbGczYUhzbEkrQSt5Z1RwR2hZaXBT?=
 =?utf-8?B?TWluYUZ6dk9Cbmo3eVUwRmJDT011M1NJUSs2T3piR1FQL3pKR1FTMEVTV21m?=
 =?utf-8?B?Rjh1Tm1ka0swY3luTTFscEdTanBZN2s4a2lnN0JYT1dGVVR4cTdTaFRpeTh6?=
 =?utf-8?B?S1E9PQ==?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 273e0839-4d39-4606-7689-08db4a6859ca
X-MS-Exchange-CrossTenant-AuthSource: SN6PR1501MB2064.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 May 2023 17:20:24.7812
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Wcgd9VAIdS90cg3BDAlRgD116NN1H8zL/WSgaIM2CnNs6joS8/l17/ZoxmVcqPcv
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR15MB5791
X-Proofpoint-GUID: BDcqKu4TmLl4tdEnq9FZWvtqfqoQ8YAg
X-Proofpoint-ORIG-GUID: BDcqKu4TmLl4tdEnq9FZWvtqfqoQ8YAg
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22
 definitions=2023-05-01_10,2023-04-27_01,2023-02-09_01
X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_PASS,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 5/1/23 9:30 AM, Espen Grindhaug wrote:
> On Mon, May 01, 2023 at 08:23:35AM -0700, Yonghong Song wrote:
>>
>>
>> On 5/1/23 6:00 AM, Espen Grindhaug wrote:
>>> On Thu, Apr 27, 2023 at 06:19:29PM -0700, Yonghong Song wrote:
>>>>
>>>>
>>>> On 4/27/23 12:19 PM, Espen Grindhaug wrote:
>>>>> On Wed, Apr 26, 2023 at 02:47:27PM -0700, Yonghong Song wrote:
>>>>>>
>>>>>>
>>>>>> On 4/23/23 11:55 AM, Espen Grindhaug wrote:
>>>>>>> This change fixes the handling of versions in elf_find_func_offset.
>>>>>>> In the previous implementation, we incorrectly assumed that the
>>>>>>
>>>>>> Could you give more explanation/example in the commit message
>>>>>> what does 'incorrectly' mean here? In which situations the
>>>>>> current libbpf implementation will not be correct?
>>>>>>
>>>>>
>>>>> How about something like this?
>>>>>
>>>>>
>>>>> libbpf: Improve version handling when attaching uprobe
>>>>>
>>>>> This change fixes the handling of versions in elf_find_func_offset.
>>>>>
>>>>> For example, let's assume we are trying to attach an uprobe to pthread_create in
>>>>> glibc. Prior to this commit, it would fail with an error message saying 'elf:
>>>>> ambiguous match [...]', this is because there are two entries in the symbol
>>>>> table with that name.
>>>>>
>>>>> $ nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep pthread_create
>>>>> 0000000000094cc0 T pthread_create@GLIBC_2.2.5
>>>>> 0000000000094cc0 T pthread_create@@GLIBC_2.34
>>>>>
>>>>> So we go ahead and modify our code to attach to 'pthread_create@@GLIBC_2.34',
>>>>> and this also fails, but this time with the error 'elf: failed to find symbol
>>>>> [...]'. This fails because we incorrectly assumed that the version information
>>>>> would be present in the string found in the string table, but there is only the
>>>>> string 'pthread_create'.
>>>>
>>>> I tried one example with my centos8 libpthread library.
>>>>
>>>> $ llvm-readelf -s /lib64/libc-2.28.so | grep pthread_cond_signal
>>>>       39: 0000000000095f70    43 FUNC    GLOBAL DEFAULT    14
>>>> pthread_cond_signal@@GLIBC_2.3.2
>>>>       40: 0000000000096250    43 FUNC    GLOBAL DEFAULT    14
>>>> pthread_cond_signal@GLIBC_2.2.5
>>>>     3160: 0000000000096250    43 FUNC    LOCAL  DEFAULT    14
>>>> __pthread_cond_signal_2_0
>>>>     3589: 0000000000095f70    43 FUNC    LOCAL  DEFAULT    14
>>>> __pthread_cond_signal
>>>>     5522: 0000000000095f70    43 FUNC    GLOBAL DEFAULT    14
>>>> pthread_cond_signal@@GLIBC_2.3.2
>>>>     5545: 0000000000096250    43 FUNC    GLOBAL DEFAULT    14
>>>> pthread_cond_signal@GLIBC_2.2.5
>>>> $ nm -D /lib64/libc-2.28.so | grep pthread_cond_signal
>>>> 0000000000095f70 T pthread_cond_signal@@GLIBC_2.3.2
>>>> 0000000000096250 T pthread_cond_signal@GLIBC_2.2.5
>>>> $
>>>>
>>>> Note that two pthread_cond_signal functions have different addresses,
>>>> which is expected as they implemented for different versions.
>>>>
>>>> But in your case,
>>>>> $ nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep pthread_create
>>>>> 0000000000094cc0 T pthread_create@GLIBC_2.2.5
>>>>> 0000000000094cc0 T pthread_create@@GLIBC_2.34
>>>>
>>>> Two functions have the same address which is very weird and I suspect
>>>> some issues here at least needs some investigation.
>>>>
>>>
>>> I am no expert on this, but as far as I can tell, this is normal,
>>> although much more common on my Ubuntu machine than my Fedora machine.
>>>
>>> Script to find duplicates:
>>>
>>> nm -D /usr/lib64/libc-2.33.so | awk '
>>> {
>>>       addr = $1;
>>>       symbol = $3;
>>>       sub(/[@].*$/, "", symbol);
>>>
>>>       if (addr == prev_addr && symbol == prev_symbol) {
>>>           if (prev_symbol_printed == 0) {
>>>               print prev_line;
>>>               prev_symbol_printed = 1;
>>>           }
>>>           print;
>>>       } else {
>>>           prev_symbol_printed = 0;
>>>       }
>>>       prev_addr = addr;
>>>       prev_symbol = symbol;
>>>       prev_line = $0;
>>> }'
>>>
>>>
>>>> Second, for the symbol table, the following is ELF encoding,
>>>>
>>>> typedef struct {
>>>>           Elf64_Word      st_name;
>>>>           unsigned char   st_info;
>>>>           unsigned char   st_other;
>>>>           Elf64_Half      st_shndx;
>>>>           Elf64_Addr      st_value;
>>>>           Elf64_Xword     st_size;
>>>> } Elf64_Sym;
>>>>
>>>> where
>>>> st_name
>>>>
>>>>       An index into the object file's symbol string table, which holds the
>>>> character representations of the symbol names. If the value is nonzero, the
>>>> value represents a string table index that gives the symbol name. Otherwise,
>>>> the symbol table entry has no name.
>>>>
>>>> So, the function name (including @..., @@...) should be in string table
>>>> which is the same for the above two pthread_cond_signal symbols.
>>>>
>>>> I think it is worthwhile to debug why in your situation
>>>> pthread_create@GLIBC_2.2.5 and pthread_create@@GLIBC_2.34 do not
>>>> have them in the string table.
>>>>
>>>
>>> I think you are mistaken here; the strings in the strings table don't contain
>>> the version. Take a look at this partial dump of the strings table.
>>>
>>> 	$ readelf -W -p .dynstr /usr/lib64/libc-2.33.so
>>>
>>> 	String dump of section '.dynstr':
>>> 		[     1]  xdrmem_create
>>> 		[     f]  __wctomb_chk
>>> 		[    1c]  getmntent
>>> 		[    26]  __freelocale
>>> 		[    33]  __rawmemchr
>>> 		[    3f]  _IO_vsprintf
>>> 		[    4c]  getutent
>>> 		[    55]  __file_change_detection_for_path
>>> 	(...)
>>> 		[  350e]  memrchr
>>> 		[  3516]  pthread_cond_signal
>>> 		[  352a]  __close
>>> 	(...)
>>> 		[  61b6]  GLIBC_2.2.5
>>> 		[  61c2]  GLIBC_2.2.6
>>> 		[  61ce]  GLIBC_2.3
>>> 		[  61d8]  GLIBC_2.3.2
>>> 		[  61e4]  GLIBC_2.3.3
>>>
>>> As you can see, the strings have no versions, and the version strings
>>> themselves are also in this table as entries at the end of the table.
>>
>> I see you search .dynstr section. Do you think whether we should
>> search .strtab instead since it contains versioned symbols?
>>
> 
> I searched .dynstr since my libc files only have that section, but I do see
> your point. If const char *binary_path points to an executable and not an
> .so file, then we would find some versioned symbols in the .strtab section.
> However, since libbpf supports using the .so as binary_path, would we not
> need the functionality to build the complete name regardless?

Okay, so you do not have .strtab section, the section probably removed
with `llvm-strip --strip-all <binary>`. In this particular case, I think
your approach to search SHT_GNU_versym and SHT_GNU_verdef for versioned
symbols probably is the right choice. Please do add such information
in the commit message.

> 
> Adding a check to not build the full name if it already contains an '@' is
> probably a good idea, though.

If you search strtab, you will find name with '@', but this won't be the 
case if you using SHT_GNU_versym/SHT_GNU_verdef. Since both dynstr and 
strtab are searched, I guess adding this check is a good idea if the 
version in strtab case is not NULL.

> 
>>>
>>>>>
>>>>> This patch reworks how we compare the symbol name provided by the user if it is
>>>>> qualified with a version (using @ or @@). We now look up the correct version
>>>>> string in the version symbol table before constructing the full name, as also
>>>>> done above by nm, before comparing.
>>>>>
>>>>>>> version information would be present in the string found in the
>>>>>>> string table.
>>>>>>>
>>>>>>> We now look up the correct version string in the version symbol
>>>>>>> table before constructing the full name and then comparing.
>>>>>>>
>>>>>>> This patch adds support for both name@version and name@@version to
>>>>>>> match output of the various elf parsers.
>>>>>>>
>>>>>>> Signed-off-by: Espen Grindhaug <espen.grindhaug@gmail.com>
>>>>>>
>>>>>> [...]