[ 185.671655][ C3]
[ 185.671855][ C3] ======================================================
[ 185.672132][ C3] WARNING: possible circular locking dependency detected
[ 185.672393][ C3] 6.9.0-rc2-virtme #1 Not tainted
[ 185.672606][ C3] ------------------------------------------------------
[ 185.672868][ C3] tls/2801 is trying to acquire lock:
[ 185.673081][ C3] ffff888003b8b818 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}, at: tw_timer_handler+0xe9/0x130
[ 185.673511][ C3]
[ 185.673511][ C3] but task is already holding lock:
[ 185.673823][ C3] ffffc90000288d68 ((&tw->tw_timer)){+.-.}-{0:0}, at: call_timer_fn+0xe8/0x230
[ 185.674203][ C3]
[ 185.674203][ C3] which lock already depends on the new lock.
[ 185.674203][ C3]
[ 185.674618][ C3]
[ 185.674618][ C3] the existing dependency chain (in reverse order) is:
[ 185.674984][ C3]
[ 185.674984][ C3] -> #1 ((&tw->tw_timer)){+.-.}-{0:0}:
[ 185.675302][ C3] __lock_acquire+0xaf0/0x1570
[ 185.675519][ C3] lock_acquire.part.0+0xeb/0x330
[ 185.675732][ C3] __timer_delete_sync+0xe1/0x110
[ 185.675950][ C3] inet_twsk_deschedule_put+0xec/0x150
[ 185.676222][ C3] inet_twsk_purge+0x4c8/0x660
[ 185.676434][ C3] tcp_twsk_purge+0x112/0x160
[ 185.676645][ C3] tcp_sk_exit_batch+0x17/0x120
[ 185.676854][ C3] cleanup_net+0x4f3/0xb60
[ 185.677064][ C3] process_one_work+0xe2c/0x1730
[ 185.677277][ C3] worker_thread+0x587/0xd30
[ 185.677486][ C3] kthread+0x28a/0x350
[ 185.677698][ C3] ret_from_fork+0x31/0x70
[ 185.677910][ C3] ret_from_fork_asm+0x1a/0x30
[ 185.678122][ C3]
[ 185.678122][ C3] -> #0 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}:
[ 185.678440][ C3] check_prev_add+0x182/0xce0
[ 185.678654][ C3] validate_chain+0x71d/0x9b0
[ 185.678864][ C3] __lock_acquire+0xaf0/0x1570
[ 185.679073][ C3] lock_acquire.part.0+0xeb/0x330
[ 185.679282][ C3] _raw_spin_lock+0x30/0x40
[ 185.679496][ C3] tw_timer_handler+0xe9/0x130
[ 185.679706][ C3] call_timer_fn+0x13b/0x230
[ 185.679915][ C3] __run_timers+0x545/0x810
[ 185.680125][ C3] run_timer_softirq+0x7a/0x1b0
[ 185.680338][ C3] __do_softirq+0x1f8/0x5df
[ 185.680545][ C3] irq_exit_rcu+0x97/0xc0
[ 185.680757][ C3] sysvec_apic_timer_interrupt+0x75/0x80
[ 185.681025][ C3] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 185.681288][ C3] lock_acquire.part.0+0x119/0x330
[ 185.681498][ C3] is_bpf_text_address+0x37/0x120
[ 185.681712][ C3] kernel_text_address+0xce/0xe0
[ 185.681922][ C3] __kernel_text_address+0x12/0x40
[ 185.682131][ C3] unwind_get_return_address+0x5e/0xa0
[ 185.682392][ C3] arch_stack_walk+0xa2/0xf0
[ 185.682601][ C3] stack_trace_save+0x94/0xd0
[ 185.682811][ C3] kasan_save_stack+0x24/0x50
[ 185.683024][ C3] kasan_save_track+0x14/0x30
[ 185.683233][ C3] __kasan_slab_alloc+0x59/0x70
[ 185.683449][ C3] kmem_cache_alloc+0xef/0x270
[ 185.683663][ C3] __alloc_object+0x39/0x6f0
[ 185.683875][ C3] __create_object+0x22/0xb0
[ 185.684084][ C3] kmem_cache_alloc_node+0x230/0x2a0
[ 185.684347][ C3] __alloc_skb+0x215/0x2e0
[ 185.684562][ C3] tcp_stream_alloc_skb+0x30/0x520
[ 185.684774][ C3] tcp_sendmsg_locked+0x9ef/0x30f0
[ 185.684980][ C3] tls_push_sg+0x225/0x6d0 [tls]
[ 185.685223][ C3] tls_tx_records+0x2dd/0x750 [tls]
[ 185.685439][ C3] tls_push_record+0xc26/0x1d20 [tls]
[ 185.685705][ C3] bpf_exec_tx_verdict+0xb6c/0x1160 [tls]
[ 185.685973][ C3] tls_sw_sendmsg_locked.constprop.0+0x8cf/0x1870 [tls]
[ 185.686302][ C3] tls_sw_sendmsg+0x9f/0xe0 [tls]
[ 185.686519][ C3] sock_sendmsg+0x263/0x370
[ 185.686730][ C3] splice_to_socket+0x790/0xea0
[ 185.686945][ C3] direct_splice_actor+0x175/0x5c0
[ 185.687155][ C3] splice_direct_to_actor+0x2c7/0x850
[ 185.687415][ C3] do_splice_direct+0x137/0x200
[ 185.687623][ C3] do_sendfile+0x94a/0xea0
[ 185.687835][ C3] __x64_sys_sendfile64+0x192/0x1d0
[ 185.688044][ C3] do_syscall_64+0xc6/0x1e0
[ 185.688253][ C3] entry_SYSCALL_64_after_hwframe+0x72/0x7a
[ 185.688516][ C3]
[ 185.688516][ C3] other info that might help us debug this:
[ 185.688516][ C3]
[ 185.688925][ C3] Possible unsafe locking scenario:
[ 185.688925][ C3]
[ 185.689232][ C3] CPU0 CPU1
[ 185.689438][ C3] ---- ----
[ 185.689644][ C3] lock((&tw->tw_timer));
[ 185.689855][ C3] lock(&hashinfo->ehash_locks[i]);
[ 185.690165][ C3] lock((&tw->tw_timer));
[ 185.690425][ C3] lock(&hashinfo->ehash_locks[i]);
[ 185.690634][ C3]
[ 185.690634][ C3] *** DEADLOCK ***
[ 185.690634][ C3]
[ 185.691061][ C3] 4 locks held by tls/2801:
[ 185.691273][ C3] #0: ffff888005a7a6f0 (&ctx->tx_lock){+.+.}-{3:3}, at: tls_sw_sendmsg+0x86/0xe0 [tls]
[ 185.691777][ C3] #1: ffff888014794258 (sk_lock-AF_INET){+.+.}-{0:0}, at: tls_sw_sendmsg+0x94/0xe0 [tls]
[ 185.692156][ C3] #2: ffffffffae1663a0 (rcu_read_lock){....}-{1:2}, at: is_bpf_text_address+0x23/0x120
[ 185.692645][ C3] #3: ffffc90000288d68 ((&tw->tw_timer)){+.-.}-{0:0}, at: call_timer_fn+0xe8/0x230
[ 185.693013][ C3]
[ 185.693013][ C3] stack backtrace:
[ 185.693266][ C3] CPU: 3 PID: 2801 Comm: tls Not tainted 6.9.0-rc2-virtme #1
[ 185.693575][ C3] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 185.694274][ C3] Call Trace:
[ 185.694432][ C3]
[ 185.694539][ C3] dump_stack_lvl+0x82/0xd0
[ 185.694749][ C3] check_noncircular+0x2f9/0x3e0
[ 185.695076][ C3] ? __pfx_check_noncircular+0x10/0x10
[ 185.695283][ C3] ? lock_release+0x13/0x110
[ 185.695490][ C3] ? is_bpf_text_address+0x71/0x120
[ 185.695697][ C3] ? alloc_chain_hlocks+0x4e0/0x720
[ 185.696044][ C3] check_prev_add+0x182/0xce0
[ 185.696252][ C3] validate_chain+0x71d/0x9b0
[ 185.696460][ C3] ? __pfx_validate_chain+0x10/0x10
[ 185.696666][ C3] ? hlock_class+0x4e/0x130
[ 185.696991][ C3] ? mark_lock+0x38/0x3e0
[ 185.697149][ C3] __lock_acquire+0xaf0/0x1570
[ 185.697358][ C3] ? __pfx_tw_timer_handler+0x10/0x10
[ 185.697563][ C3] lock_acquire.part.0+0xeb/0x330
[ 185.697772][ C3] ? tw_timer_handler+0xe9/0x130
[ 185.698097][ C3] ? __pfx_tw_timer_handler+0x10/0x10
[ 185.698303][ C3] ? lock_acquire.part.0+0xeb/0x330
[ 185.698510][ C3] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 185.698715][ C3] ? trace_lock_acquire+0x135/0x1c0
[ 185.699040][ C3] ? tw_timer_handler+0xe9/0x130
[ 185.699244][ C3] ? lock_acquire+0x32/0xc0
[ 185.699450][ C3] ? tw_timer_handler+0xe9/0x130
[ 185.699658][ C3] _raw_spin_lock+0x30/0x40
[ 185.699984][ C3] ? tw_timer_handler+0xe9/0x130
[ 185.700190][ C3] tw_timer_handler+0xe9/0x130
[ 185.700396][ C3] call_timer_fn+0x13b/0x230
[ 185.700601][ C3] ? call_timer_fn+0xe8/0x230
[ 185.700806][ C3] ? call_timer_fn+0xe8/0x230
[ 185.701132][ C3] ? __pfx_call_timer_fn+0x10/0x10
[ 185.701338][ C3] ? hlock_class+0x4e/0x130
[ 185.701542][ C3] ? mark_held_locks+0x9e/0xe0
[ 185.701753][ C3] __run_timers+0x545/0x810
[ 185.702079][ C3] ? __pfx_tw_timer_handler+0x10/0x10
[ 185.702287][ C3] ? __pfx___run_timers+0x10/0x10
[ 185.702495][ C3] ? do_raw_spin_lock+0x131/0x270
[ 185.702701][ C3] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 185.703025][ C3] ? lock_acquire+0x32/0xc0
[ 185.703231][ C3] ? run_timer_softirq+0x72/0x1b0
[ 185.703438][ C3] run_timer_softirq+0x7a/0x1b0
[ 185.703644][ C3] __do_softirq+0x1f8/0x5df
[ 185.703972][ C3] irq_exit_rcu+0x97/0xc0
[ 185.704129][ C3] sysvec_apic_timer_interrupt+0x75/0x80
[ 185.704334][ C3]
[ 185.704446][ C3]
[ 185.704549][ C3] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 185.704809][ C3] RIP: 0010:lock_acquire.part.0+0x119/0x330
[ 185.705189][ C3] Code: ff 48 83 c4 28 65 0f c1 05 44 18 93 55 83 f8 01 0f 85 b5 01 00 00 9c 58 f6 c4 02 0f 85 be 01 00 00 48 85 ed 0f 85 9b 01 00 00 <48> b8 00 00 00 00 00 fc ff df 48 01 c3 48 c7 03 00 00 00 00 48 c7
[ 185.706039][ C3] RSP: 0018:ffffc9000126e608 EFLAGS: 00000206
[ 185.706304][ C3] RAX: 0000000000000046 RBX: 1ffff9200024dcc4 RCX: 0000000000000001
[ 185.706611][ C3] RDX: 1ffff1100241614d RSI: ffff8880120b0ac0 RDI: ffff8880120b0ae2
[ 185.707040][ C3] RBP: 0000000000000200 R08: 0000000000000000 R09: ffff8880120b0ac0
[ 185.707352][ C3] R10: 1ffff9200024dc96 R11: ffffc9000126e789 R12: ffffffffae1663a0
[ 185.707660][ C3] R13: 0000000000000000 R14: ffff8880120b0040 R15: 0000000000092cc0
[ 185.708093][ C3] ? __pfx___lock_release+0x10/0x10
[ 185.708300][ C3] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 185.708505][ C3] ? trace_lock_acquire+0x135/0x1c0
[ 185.708712][ C3] ? is_bpf_text_address+0x23/0x120
[ 185.709040][ C3] ? lock_acquire+0x32/0xc0
[ 185.709246][ C3] ? is_bpf_text_address+0x23/0x120
[ 185.709453][ C3] is_bpf_text_address+0x37/0x120
[ 185.709658][ C3] ? is_bpf_text_address+0x23/0x120
[ 185.710101][ C3] kernel_text_address+0xce/0xe0
[ 185.710308][ C3] __kernel_text_address+0x12/0x40
[ 185.710512][ C3] unwind_get_return_address+0x5e/0xa0
[ 185.710718][ C3] ? __pfx_stack_trace_consume_entry+0x10/0x10
[ 185.711096][ C3] arch_stack_walk+0xa2/0xf0
[ 185.711307][ C3] stack_trace_save+0x94/0xd0
[ 185.711511][ C3] ? __pfx_stack_trace_save+0x10/0x10
[ 185.711721][ C3] ? find_held_lock+0x2c/0x110
[ 185.711928][ C3] kasan_save_stack+0x24/0x50
[ 185.712250][ C3] ? kasan_save_stack+0x24/0x50
[ 185.712455][ C3] ? kasan_save_track+0x14/0x30
[ 185.712661][ C3] ? __kasan_slab_alloc+0x59/0x70
[ 185.712866][ C3] ? kmem_cache_alloc+0xef/0x270
[ 185.713194][ C3] ? __alloc_object+0x39/0x6f0
[ 185.713401][ C3] ? __create_object+0x22/0xb0
[ 185.713607][ C3] ? kmem_cache_alloc_node+0x230/0x2a0
[ 185.713814][ C3] ? __alloc_skb+0x215/0x2e0
[ 185.714140][ C3] ? tcp_stream_alloc_skb+0x30/0x520
[ 185.714343][ C3] ? tcp_sendmsg_locked+0x9ef/0x30f0
[ 185.714548][ C3] ? tls_push_sg+0x225/0x6d0 [tls]
[ 185.714762][ C3] ? tls_tx_records+0x2dd/0x750 [tls]
[ 185.715097][ C3] ? tls_push_record+0xc26/0x1d20 [tls]
[ 185.715310][ C3] ? bpf_exec_tx_verdict+0xb6c/0x1160 [tls]
[ 185.715572][ C3] ? tls_sw_sendmsg_locked.constprop.0+0x8cf/0x1870 [tls]
[ 185.715836][ C3] ? tls_sw_sendmsg+0x9f/0xe0 [tls]
[ 185.716177][ C3] ? sock_sendmsg+0x263/0x370
[ 185.716384][ C3] ? splice_to_socket+0x790/0xea0
[ 185.716589][ C3] ? direct_splice_actor+0x175/0x5c0
[ 185.716797][ C3] ? splice_direct_to_actor+0x2c7/0x850
[ 185.717120][ C3] ? do_splice_direct+0x137/0x200
[ 185.717326][ C3] ? do_sendfile+0x94a/0xea0
[ 185.717533][ C3] ? __x64_sys_sendfile64+0x192/0x1d0
[ 185.717737][ C3] ? do_syscall_64+0xc6/0x1e0
[ 185.717944][ C3] ? entry_SYSCALL_64_after_hwframe+0x72/0x7a
[ 185.718319][ C3] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370
[ 185.718579][ C3] ? get_partial_node.part.0+0x1c5/0x3a0
[ 185.718787][ C3] ? ___slab_alloc+0xb70/0x10a0
[ 185.719112][ C3] ? kmem_cache_alloc+0x42/0x270
[ 185.719318][ C3] ? __alloc_object+0x39/0x6f0
[ 185.719523][ C3] ? __pfx___lock_release+0x10/0x10
[ 185.719732][ C3] kasan_save_track+0x14/0x30
[ 185.719939][ C3] __kasan_slab_alloc+0x59/0x70
[ 185.720264][ C3] kmem_cache_alloc+0xef/0x270
[ 185.720469][ C3] __alloc_object+0x39/0x6f0
[ 185.720679][ C3] __create_object+0x22/0xb0
[ 185.720885][ C3] kmem_cache_alloc_node+0x230/0x2a0
[ 185.721209][ C3] ? __alloc_skb+0x150/0x2e0
[ 185.721418][ C3] __alloc_skb+0x215/0x2e0
[ 185.721624][ C3] ? __pfx___alloc_skb+0x10/0x10
[ 185.721829][ C3] ? __sk_mem_raise_allocated+0x5e2/0x11e0
[ 185.722206][ C3] tcp_stream_alloc_skb+0x30/0x520
[ 185.722411][ C3] tcp_sendmsg_locked+0x9ef/0x30f0
[ 185.722623][ C3] ? chacha_stream_xor+0x48e/0x620 [chacha_generic]
[ 185.722883][ C3] ? __pfx_tcp_sendmsg_locked+0x10/0x10
[ 185.723209][ C3] ? __pfx_scatterwalk_map_and_copy+0x10/0x10
[ 185.723467][ C3] tls_push_sg+0x225/0x6d0 [tls]
[ 185.723681][ C3] ? __pfx_tls_push_sg+0x10/0x10 [tls]
[ 185.723897][ C3] tls_tx_records+0x2dd/0x750 [tls]
[ 185.724232][ C3] ? sk_psock_get+0xee/0x310 [tls]
[ 185.724449][ C3] tls_push_record+0xc26/0x1d20 [tls]
[ 185.724664][ C3] ? __pfx_tls_push_record+0x10/0x10 [tls]
[ 185.724927][ C3] ? sk_psock_get+0xf3/0x310 [tls]
[ 185.725261][ C3] ? __pfx_sk_psock_get+0x10/0x10 [tls]
[ 185.725476][ C3] ? is_bpf_text_address+0x71/0x120
[ 185.725683][ C3] bpf_exec_tx_verdict+0xb6c/0x1160 [tls]
[ 185.725896][ C3] ? __pfx_stack_trace_consume_entry+0x10/0x10
[ 185.726400][ C3] ? __pfx_bpf_exec_tx_verdict+0x10/0x10 [tls]
[ 185.726679][ C3] ? __pfx_tls_sw_sendmsg_splice+0x10/0x10 [tls]
[ 185.726943][ C3] ? sk_msg_alloc+0x388/0xb80
[ 185.727152][ C3] ? tls_get_rec+0x338/0x580 [tls]
[ 185.727484][ C3] tls_sw_sendmsg_locked.constprop.0+0x8cf/0x1870 [tls]
[ 185.727750][ C3] ? __lock_release+0x103/0x460
[ 185.727956][ C3] ? tls_sw_sendmsg+0x94/0xe0 [tls]
[ 185.728174][ C3] ? __pfx_tls_sw_sendmsg_locked.constprop.0+0x10/0x10 [tls]
[ 185.728610][ C3] ? hlock_class+0x4e/0x130
[ 185.728815][ C3] ? mark_lock+0x38/0x3e0
[ 185.728975][ C3] tls_sw_sendmsg+0x9f/0xe0 [tls]
[ 185.729189][ C3] sock_sendmsg+0x263/0x370
[ 185.729513][ C3] ? __pfx_validate_chain+0x10/0x10
[ 185.729719][ C3] ? __pfx_sock_sendmsg+0x10/0x10
[ 185.729923][ C3] ? mark_lock+0x38/0x3e0
[ 185.730081][ C3] ? __lock_acquire+0xaf0/0x1570
[ 185.730411][ C3] splice_to_socket+0x790/0xea0
[ 185.730621][ C3] ? __pfx_splice_to_socket+0x10/0x10
[ 185.730827][ C3] ? __pfx_timestamp_truncate+0x10/0x10
[ 185.731037][ C3] ? atime_needs_update+0x306/0x680
[ 185.731361][ C3] ? shmem_get_folio_gfp+0x213/0x970
[ 185.731574][ C3] direct_splice_actor+0x175/0x5c0
[ 185.731779][ C3] ? __pfx_validate_chain+0x10/0x10
[ 185.731988][ C3] splice_direct_to_actor+0x2c7/0x850
[ 185.732193][ C3] ? __pfx_direct_splice_actor+0x10/0x10
[ 185.732519][ C3] ? __pfx_splice_direct_to_actor+0x10/0x10
[ 185.732774][ C3] do_splice_direct+0x137/0x200
[ 185.732978][ C3] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 185.733185][ C3] ? __pfx_do_splice_direct+0x10/0x10
[ 185.733507][ C3] ? __mutex_trylock_common+0xfa/0x260
[ 185.733717][ C3] ? __pfx_direct_file_splice_eof+0x10/0x10
[ 185.733973][ C3] ? __fget_light+0x53/0x1e0
[ 185.734182][ C3] do_sendfile+0x94a/0xea0
[ 185.734513][ C3] ? __pfx_do_sendfile+0x10/0x10
[ 185.734718][ C3] ? __mutex_unlock_slowpath+0x145/0x3b0
[ 185.734924][ C3] __x64_sys_sendfile64+0x192/0x1d0
[ 185.735130][ C3] ? __pfx___x64_sys_sendfile64+0x10/0x10
[ 185.735455][ C3] do_syscall_64+0xc6/0x1e0
[ 185.735665][ C3] entry_SYSCALL_64_after_hwframe+0x72/0x7a
[ 185.735921][ C3] RIP: 0033:0x7ff4ca841c0e
[ 185.736142][ C3] Code: 48 8b 0d f5 a1 1b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c2 a1 1b 00 f7 d8 64 89 01 48
[ 185.736974][ C3] RSP: 002b:00007ffe3d72b728 EFLAGS: 00000246 ORIG_RAX: 0000000000000028
[ 185.737405][ C3] RAX: ffffffffffffffda RBX: 00000000000668a0 RCX: 00007ff4ca841c0e
[ 185.737715][ C3] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000005
[ 185.738022][ C3] RBP: 00007ffe3d73c140 R08: 0000000000000000 R09: 0000000000000000
[ 185.738447][ C3] R10: 000000000000ea60 R11: 0000000000000246 R12: 0000000000422ac0
[ 185.738754][ C3] R13: 0000000000000006 R14: 00007ffe3d73c174 R15: 000000000000ea60
[ 185.739067][ C3]