Greeting,
FYI, we noticed the following commit (built with gcc-9):
commit: 719774377622bc4025d2a74f551b5dc2158c6c30 ("netfilter: conntrack: convert to refcount_t api")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: kernel-selftests
version: kernel-selftests-x86_64-db530529-1_20220114
with following parameters:
group: netfilter
ucode: 0xe2
test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel.
test-url: https://www.kernel.org/doc/Documentation/kselftest.txt
on test machine: 8 threads Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz with 32G memory
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <[email protected]>
[ 934.186149][ C2] ==================================================================
[ 934.194459][ C2] BUG: KASAN: use-after-free in nft_ct_set_zone_eval+0x183/0x500 [nft_ct]
[ 934.203087][ C2] Read of size 4 at addr ffff88810b4c2c00 by task ping/18706
[ 934.210516][ C2]
[ 934.212796][ C2] CPU: 2 PID: 18706 Comm: ping Tainted: G I 5.16.0-rc4-01319-g719774377622 #1
[ 934.223135][ C2] Hardware name: /NUC6i7KYB, BIOS KYSKLi70.86A.0041.2016.0817.1130 08/17/2016
[ 934.232199][ C2] Call Trace:
[ 934.235467][ C2] <IRQ>
[ 934.238280][ C2] dump_stack_lvl+0x45/0x59
[ 934.243211][ C2] print_address_description+0x21/0x140
[ 934.249863][ C2] ? nft_ct_set_zone_eval+0x183/0x500 [nft_ct]
[ 934.256060][ C2] ? nft_ct_set_zone_eval+0x183/0x500 [nft_ct]
[ 934.262242][ C2] kasan_report.cold+0x7f/0x11b
[ 934.267106][ C2] ? nft_ct_set_zone_eval+0x183/0x500 [nft_ct]
[ 934.273386][ C2] kasan_check_range+0x14d/0x200
[ 934.278311][ C2] nft_ct_set_zone_eval+0x183/0x500 [nft_ct]
[ 934.284344][ C2] ? nft_ct_get_dump+0x240/0x240 [nft_ct]
[ 934.290111][ C2] ? lockdep_hardirqs_on_prepare+0x129/0x400
[ 934.296154][ C2] ? __local_bh_enable_ip+0xa2/0x100
[ 934.301469][ C2] nft_do_chain+0x2dc/0x1200 [nf_tables]
[ 934.307176][ C2] ? __alloc_skb+0xd1/0x2c0
[ 934.311633][ C2] ? rcu_read_lock_sched_held+0xa1/0x100
[ 934.317248][ C2] ? nft_update_chain_stats+0x140/0x140 [nf_tables]
[ 934.324516][ C2] ? mark_lock+0xca/0x13c0
[ 934.329631][ C2] ? mark_lock_irq+0x1240/0x1240
[ 934.334605][ C2] ? mark_lock_irq+0x1240/0x1240
[ 934.339965][ C2] ? kmalloc_reserve+0xc0/0xc0
[ 934.344786][ C2] ? __lock_acquire+0xc06/0x3440
[ 934.349745][ C2] ? lockdep_hardirqs_on_prepare+0x400/0x400
[ 934.355781][ C2] ? lock_is_held_type+0x98/0x140
[ 934.360792][ C2] ? rcu_read_lock_sched_held+0xa1/0x100
[ 934.366493][ C2] nft_do_chain_ipv4+0x17e/0x200 [nf_tables]
[ 934.372633][ C2] ? nft_do_chain_arp+0xc0/0xc0 [nf_tables]
[ 934.378565][ C2] ? rcu_read_unlock+0x40/0x40
[ 934.383364][ C2] nf_hook_slow+0xa7/0x180
[ 934.387813][ C2] vrf_l3_out+0xb0f/0x1080
[ 934.392264][ C2] ? vrf_l3_rcv+0x1200/0x1200
[ 934.396942][ C2] ? lock_is_held_type+0x98/0x140
[ 934.401940][ C2] ? vrf_ip_local_out+0x780/0x780
[ 934.407024][ C2] ? netdev_lower_get_first_private_rcu+0x100/0x100
[ 934.413694][ C2] __ip_local_out+0x42e/0x6c0
[ 934.418336][ C2] ? dst_output+0x380/0x380
[ 934.422875][ C2] ? kfree+0x202/0x400
[ 934.426889][ C2] ? __ip_make_skb+0xf29/0x1f00
[ 934.431731][ C2] ip_local_out+0x21/0x3c0
[ 934.436170][ C2] ip_send_skb+0x37/0xc0
[ 934.440473][ C2] icmp_reply+0x8b1/0xa00
[ 934.444866][ C2] ? icmpv4_xrlim_allow+0x3c0/0x3c0
[ 934.450721][ C2] ? lock_release+0x1df/0x680
[ 934.455480][ C2] ? lock_is_held_type+0x98/0x140
[ 934.460520][ C2] ? raw_local_deliver+0x7a7/0xd40
[ 934.465639][ C2] ? find_held_lock+0x2c/0x140
[ 934.470413][ C2] ? raw_local_deliver+0x7a7/0xd40
[ 934.475561][ C2] ? icmp_echo+0x174/0x200
[ 934.480601][ C2] icmp_echo+0x174/0x200
[ 934.485508][ C2] ? icmp_timestamp+0x1c0/0x1c0
[ 934.490377][ C2] ? rcu_read_lock_held+0xa1/0xc0
[ 934.495393][ C2] ? lock_is_held_type+0x98/0x140
[ 934.500578][ C2] ? rcu_read_lock_held+0xa1/0xc0
[ 934.505736][ C2] ? rcu_read_lock_sched_held+0x100/0x100
[ 934.511457][ C2] ? rcu_read_lock_held+0xa1/0xc0
[ 934.516500][ C2] ? rcu_read_lock_sched_held+0x100/0x100
[ 934.522274][ C2] icmp_echo+0xb9/0x180
[ 934.527003][ C2] icmp_rcv+0xae1/0x1a40
[ 934.531163][ C2] ? rcu_read_lock_sched_held+0x100/0x100
[ 934.536981][ C2] ? lock_release+0x1df/0x680
[ 934.541768][ C2] ip_protocol_deliver_rcu+0x7a5/0xb00
[ 934.547268][ C2] ip_local_deliver_finish+0x1df/0x300
[ 934.552809][ C2] ip_local_deliver+0x2da/0x440
[ 934.557746][ C2] ? ip_local_deliver_finish+0x300/0x300
[ 934.563524][ C2] ? rcu_read_lock_held+0xa1/0xc0
[ 934.568522][ C2] ? ip_protocol_deliver_rcu+0xb00/0xb00
[ 934.574251][ C2] ip_rcv+0x1fb/0x300
[ 934.578217][ C2] ? ip_sublist_rcv+0x800/0x800
[ 934.583044][ C2] ? rcu_read_lock_sched_held+0xa1/0x100
[ 934.588754][ C2] ? ip_local_deliver+0x440/0x440
[ 934.593818][ C2] ? lock_acquire+0x194/0x4c0
[ 934.598554][ C2] ? ip_sublist_rcv+0x800/0x800
[ 934.603454][ C2] __netif_receive_skb_one_core+0x162/0x1c0
[ 934.609466][ C2] ? __netif_receive_skb_core+0x3580/0x3580
[ 934.615505][ C2] ? mark_held_locks+0x9e/0x100
[ 934.620388][ C2] process_backlog+0x1cc/0x700
[ 934.625142][ C2] ? mark_held_locks+0x9e/0x100
[ 934.630030][ C2] __napi_poll+0xa1/0x500
[ 934.634394][ C2] net_rx_action+0x6f9/0xa80
[ 934.638951][ C2] ? napi_threaded_poll+0x480/0x480
[ 934.644169][ C2] ? lock_is_held_type+0x98/0x140
[ 934.649225][ C2] ? rcu_read_lock_bh_held+0xc0/0xc0
[ 934.654571][ C2] __do_softirq+0x1cb/0x860
[ 934.659072][ C2] ? ip_finish_output2+0x632/0x1d80
[ 934.664220][ C2] do_softirq+0x72/0xc0
[ 934.668398][ C2] </IRQ>
[ 934.671277][ C2] <TASK>
[ 934.674220][ C2] __local_bh_enable_ip+0xd8/0x100
[ 934.679376][ C2] ip_finish_output2+0x656/0x1d80
[ 934.684381][ C2] ? ip_frag_next+0x9c0/0x9c0
[ 934.689082][ C2] ? ip_output+0x35e/0x4c0
[ 934.693548][ C2] __ip_finish_output+0x821/0x1380
[ 934.698644][ C2] ? ip_output+0x4c0/0x4c0
[ 934.703050][ C2] ip_output+0x1c9/0x4c0
[ 934.707312][ C2] ? ip_fragment+0x240/0x240
[ 934.713006][ C2] ? __ip_make_skb+0xf29/0x1f00
[ 934.717945][ C2] ? ip_local_out+0x21/0x3c0
[ 934.722564][ C2] ip_send_skb+0x37/0xc0
[ 934.726736][ C2] raw_sendmsg+0xee0/0x16c0
[ 934.731183][ C2] ? dst_output+0x380/0x380
[ 934.735645][ C2] ? lockdep_hardirqs_on_prepare+0x400/0x400
[ 934.741652][ C2] ? __might_fault+0xb8/0x180
[ 934.746369][ C2] ? find_held_lock+0x2c/0x140
[ 934.751126][ C2] ? gup_pgd_range+0xcc8/0x1400
[ 934.755986][ C2] ? lock_release+0x1df/0x680
[ 934.760721][ C2] ? lock_downgrade+0x700/0x700
[ 934.765611][ C2] ? inet_send_prepare+0x3c0/0x3c0
[ 934.770690][ C2] ? sock_sendmsg+0xdd/0x140
[ 934.775381][ C2] ? dst_output+0x380/0x380
[ 934.779937][ C2] sock_sendmsg+0xdd/0x140
[ 934.784400][ C2] __sys_sendto+0x1a6/0x280
[ 934.788979][ C2] ? __ia32_sys_getpeername+0xc0/0xc0
[ 934.794364][ C2] ? __x64_sys_rt_sigsuspend+0x140/0x140
[ 934.800069][ C2] ? syscall_exit_to_user_mode+0x1e/0x80
[ 934.805734][ C2] ? syscall_exit_to_user_mode+0x1e/0x80
[ 934.811379][ C2] ? syscall_enter_from_user_mode+0x1c/0x80
[ 934.817325][ C2] ? rcu_read_lock_sched_held+0xa1/0x100
[ 934.823036][ C2] ? rcu_read_lock_bh_held+0xc0/0xc0
[ 934.828390][ C2] __x64_sys_sendto+0xdd/0x1c0
[ 934.833165][ C2] ? syscall_enter_from_user_mode+0x21/0x80
[ 934.839149][ C2] do_syscall_64+0x5c/0x80
[ 934.843553][ C2] ? do_syscall_64+0x69/0x80
[ 934.848172][ C2] ? lockdep_hardirqs_on_prepare+0x273/0x400
[ 934.854195][ C2] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 934.860179][ C2] RIP: 0033:0x7f7662be49b7
[ 934.864680][ C2] Code: 64 89 02 48 c7 c0 ff ff ff ff eb b6 0f 1f 80 00 00 00 00 48 8d 05 49 5d 0c 00 41 89 ca 8b 00 85 c0 75 10 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 71 c3 41 57 4d 89 c7 41 56 41 89 ce 41 55 49
[ 934.885487][ C2] RSP: 002b:00007ffc60016c18 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[ 934.894237][ C2] RAX: ffffffffffffffda RBX: 0000556977276a20 RCX: 00007f7662be49b7
[ 934.902495][ C2] RDX: 0000000000000040 RSI: 0000556977276a20 RDI: 0000000000000005
[ 934.911820][ C2] RBP: 0000000000000040 R08: 0000556977273800 R09: 0000000000000010
[ 934.920071][ C2] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc60018340
[ 934.928373][ C2] R13: 00007ffc60016c20 R14: 00007ffc60016d10 R15: 00005569772723a0
[ 934.936587][ C2] </TASK>
[ 934.939625][ C2]
[ 934.941972][ C2] Allocated by task 18705:
[ 934.946441][ C2] kasan_save_stack+0x1e/0x80
[ 934.951248][ C2] __kasan_kmalloc+0x81/0xc0
[ 934.955875][ C2] nf_ct_tmpl_alloc+0x8c/0x300 [nf_conntrack]
[ 934.962075][ C2] nft_ct_set_init+0x435/0x680 [nft_ct]
[ 934.967741][ C2] nf_tables_newrule+0xb95/0x2780 [nf_tables]
[ 934.973968][ C2] nfnetlink_rcv_batch+0xd08/0x1980 [nfnetlink]
[ 934.980406][ C2] nfnetlink_rcv+0x2c1/0x340 [nfnetlink]
[ 934.986167][ C2] netlink_unicast+0x430/0x680
[ 934.990982][ C2] netlink_sendmsg+0x77a/0xc40
[ 934.995899][ C2] sock_sendmsg+0xe4/0x140
[ 935.000340][ C2] ____sys_sendmsg+0x54f/0x7c0
[ 935.005237][ C2] ___sys_sendmsg+0xe9/0x180
[ 935.009882][ C2] __sys_sendmsg+0xb7/0x140
[ 935.014434][ C2] do_syscall_64+0x5c/0x80
[ 935.018948][ C2] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 935.024933][ C2]
[ 935.027243][ C2] Freed by task 18706:
[ 935.031347][ C2] kasan_save_stack+0x1e/0x80
[ 935.036154][ C2] kasan_set_track+0x21/0x40
[ 935.040787][ C2] kasan_set_free_info+0x20/0x40
[ 935.045809][ C2] __kasan_slab_free+0xea/0x140
[ 935.050790][ C2] kfree+0xd0/0x400
[ 935.054599][ C2] nf_conntrack_destroy+0x83/0x180
[ 935.056340][T18663] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[ 935.059834][ C2] nf_conntrack_in+0x1023/0x1440 [nf_conntrack]
[ 935.073886][ C2] nf_hook_slow+0xa7/0x180
[ 935.078545][ C2] ip_rcv+0x1be/0x300
[ 935.082691][ C2] __netif_receive_skb_one_core+0x162/0x1c0
[ 935.088698][ C2] process_backlog+0x1cc/0x700
[ 935.093454][ C2] __napi_poll+0xa1/0x500
[ 935.097804][ C2] net_rx_action+0x6f9/0xa80
[ 935.102399][ C2] __do_softirq+0x1cb/0x860
[ 935.106979][ C2]
[ 935.109295][ C2] The buggy address belongs to the object at ffff88810b4c2c00
[ 935.109295][ C2] which belongs to the cache kmalloc-512 of size 512
[ 935.123767][ C2] The buggy address is located 0 bytes inside of
[ 935.123767][ C2] 512-byte region [ffff88810b4c2c00, ffff88810b4c2e00)
[ 935.137280][ C2] The buggy address belongs to the page:
[ 935.143076][ C2] page:00000000196a0aed refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x10b4c0
[ 935.153730][ C2] head:00000000196a0aed order:3 compound_mapcount:0 compound_pincount:0
[ 935.162199][ C2] flags: 0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
[ 935.170718][ C2] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff888100042c80
[ 935.179551][ C2] raw: 0000000000000000 0000000000200020 00000001ffffffff 0000000000000000
[ 935.188262][ C2] page dumped because: kasan: bad access detected
[ 935.194823][ C2]
[ 935.197145][ C2] Memory state around the buggy address:
[ 935.202878][ C2] ffff88810b4c2b00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 935.211159][ C2] ffff88810b4c2b80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 935.219417][ C2] >ffff88810b4c2c00: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 935.227736][ C2] ^
[ 935.231898][ C2] ffff88810b4c2c80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 935.240179][ C2] ffff88810b4c2d00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 935.248437][ C2] ==================================================================
[ 935.256767][ C2] Disabling lock debugging due to kernel taint
[ 935.300943][ T429] # PASS: entry found in conntrack zone 1
[ 935.300956][ T429]
[ 935.333644][T12286] ------------[ cut here ]------------
[ 935.339289][T12286] refcount_t: underflow; use-after-free.
[ 935.345231][T12286] WARNING: CPU: 4 PID: 12286 at lib/refcount.c:28 refcount_warn_saturate+0xaa/0x140
[ 935.355000][T12286] Modules linked in: xt_state xt_conntrack xt_nat nft_compat nft_limit nfnetlink_queue nft_queue nf_conntrack_ftp ip_vs_rr ip_vs authenc echainiv esp4 nft_flow_offload nf_flow_table_inet nf_flow_table ebtable_filter ebt_redirect ebt_ip ebtable_broute ebtables nft_ct nf_conntrack_netlink nft_redir nft_masq nft_nat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_objref nf_log_syslog nft_log nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nf_tables nfnetlink netconsole btrfs blake2b_generic intel_rapl_msr xor intel_rapl_common raid6_pq zstd_compress libcrc32c sd_mod t10_pi sg ipmi_devintf i915 ipmi_msghandler x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul intel_wmi_thunderbolt crc32_pclmul intel_gtt crc32c_intel ttm ghash_clmulni_intel drm_kms_helper sdhci_pci rapl cqhci ahci syscopyarea intel_cstate sysfillrect sdhci libahci mei_me sysimgblt mmc_core i2c_i801 intel_uncore mei i2c_smbus libata intel_pch_thermal fb_sys_fops
[ 935.355219][T12286] ir_rc6_decoder wmi rc_rc6_mce nuvoton_cir rc_core video acpi_pad intel_pmc_core ip_tables
[ 935.456890][T12286] CPU: 4 PID: 12286 Comm: kworker/4:1 Tainted: G B I 5.16.0-rc4-01319-g719774377622 #1
[ 935.468165][T12286] Hardware name: /NUC6i7KYB, BIOS KYSKLi70.86A.0041.2016.0817.1130 08/17/2016
[ 935.477576][T12286] Workqueue: events nf_tables_trans_destroy_work [nf_tables]
[ 935.485327][T12286] RIP: 0010:refcount_warn_saturate+0xaa/0x140
[ 935.491632][T12286] Code: fa be bc 03 01 e8 59 4a 6c 01 0f 0b eb d5 80 3d e8 be bc 03 00 75 cc 48 c7 c7 a0 f5 27 84 c6 05 d8 be bc 03 01 e8 39 4a 6c 01 <0f> 0b eb b5 80 3d c6 be bc 03 00 75 ac 48 c7 c7 60 f6 27 84 c6 05
[ 935.512113][T12286] RSP: 0018:ffffc90007cefba8 EFLAGS: 00010286
[ 935.518393][T12286] RAX: 0000000000000000 RBX: ffff88810b4c2c00 RCX: 0000000000000000
[ 935.526773][T12286] RDX: 0000000000000004 RSI: 0000000000000008 RDI: fffff52000f9df67
[ 935.534989][T12286] RBP: 0000000000000003 R08: 0000000000000001 R09: ffffed1101f868e1
[ 935.543360][T12286] R10: ffff88880fc34707 R11: ffffed1101f868e0 R12: ffffffff8492b8b0
[ 935.551694][T12286] R13: 000000000003b150 R14: 0000000000000002 R15: 0000000000000002
[ 935.560086][T12286] FS: 0000000000000000(0000) GS:ffff88880fc00000(0000) knlGS:0000000000000000
[ 935.569373][T12286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 935.576078][T12286] CR2: 00007f07dbc083b8 CR3: 00000008bc42a001 CR4: 00000000003706e0
[ 935.584172][T12286] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 935.592284][T12286] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 935.600458][T12286] Call Trace:
[ 935.603802][T12286] <TASK>
[ 935.606757][T12286] nft_ct_tmpl_put_pcpu+0x158/0x200 [nft_ct]
[ 935.612990][T12286] __nft_ct_set_destroy+0x2d/0x40 [nft_ct]
[ 935.620176][T12286] nft_ct_set_destroy+0x4c/0x100 [nft_ct]
[ 935.626035][T12286] nf_tables_expr_destroy+0x7d/0x100 [nf_tables]
[ 935.632544][T12286] ? module_put+0x137/0x1c0
[ 935.637807][T12286] nf_tables_rule_destroy+0xa7/0x140 [nf_tables]
[ 935.644375][T12286] nf_tables_trans_destroy_work+0x335/0x680 [nf_tables]
[ 935.651419][T12286] ? lock_downgrade+0x700/0x700
[ 935.656386][T12286] ? nf_tables_destroy_set+0xc0/0xc0 [nf_tables]
[ 935.662824][T12286] ? rcu_read_lock_bh_held+0xc0/0xc0
[ 935.668279][T12286] process_one_work+0x817/0x1380
[ 935.673415][T12286] ? rcu_read_unlock+0x40/0x40
[ 935.678204][T12286] ? pwq_dec_nr_in_flight+0x280/0x280
[ 935.683678][T12286] ? rwlock_bug+0xc0/0xc0
[ 935.688781][T12286] worker_thread+0x8b/0xd80
[ 935.693375][T12286] ? __kthread_parkme+0xd9/0x200
[ 935.698514][T12286] ? schedule+0xf5/0x280
[ 935.702904][T12286] ? process_one_work+0x1380/0x1380
[ 935.708386][T12286] ? process_one_work+0x1380/0x1380
[ 935.713715][T12286] kthread+0x3a4/0x480
[ 935.717813][T12286] ? _raw_spin_unlock_irq+0x24/0x40
[ 935.723075][T12286] ? set_kthread_struct+0x100/0x100
[ 935.728294][T12286] ret_from_fork+0x22/0x30
[ 935.732760][T12286] </TASK>
[ 935.735812][T12286] irq event stamp: 34520
[ 935.740077][T12286] hardirqs last enabled at (34519): [<ffffffff83bb9fa4>] _raw_spin_unlock_irq+0x24/0x40
[ 935.750267][T12286] hardirqs last disabled at (34520): [<ffffffff83ba1f45>] __schedule+0x1405/0x2500
[ 935.759910][T12286] softirqs last enabled at (33626): [<ffffffff83e00527>] __do_softirq+0x527/0x860
[ 935.769465][T12286] softirqs last disabled at (33621): [<ffffffff813ce6e3>] irq_exit_rcu+0x163/0x1c0
[ 935.779030][T12286] ---[ end trace 6838ac6b830f7e2f ]---
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
sudo bin/lkp install job.yaml # job file is attached in this email
bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run
sudo bin/lkp run generated-yaml-file
# if come across any failure that blocks the test,
# please remove ~/.lkp and /lkp dir to run from a clean state.
---
0DAY/LKP+ Test Infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/[email protected] Intel Corporation
Thanks,
Oliver Sang