[ 263.248045][ C3] TCP: AO hash mismatch for [2001:db8:1::1].54454->[2001:db8:254::1].7013 [FP.]L3index: 0
[ 263.248644][ C3] TCP: AO hash mismatch for [2001:db8:1::1].54454->[2001:db8:254::1].7013 [.]L3index: 0
[ 264.270833][ C3]
[ 264.270955][ C3] ======================================================
[ 264.271172][ C3] WARNING: possible circular locking dependency detected
[ 264.271238][ C2] TCP: AO hash mismatch for [2001:db8:254::1].7014->[2001:db8:1::1].51840 [FP.]L3index: 0
[ 264.271411][ C3] 6.9.0-rc2-virtme #1 Not tainted
[ 264.271904][ C3] ------------------------------------------------------
[ 264.272003][ C0] TCP: AO hash mismatch for [2001:db8:254::1].7014->[2001:db8:1::1].51840 [.]L3index: 0
[ 264.272136][ C3] swapper/3/0 is trying to acquire lock:
[ 264.272138][ C3] ffff888003a33958 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}, at: tw_timer_handler+0xe9/0x130
[ 264.273112][ C3]
[ 264.273112][ C3] but task is already holding lock:
[ 264.273381][ C3] ffffc90000288d68 ((&tw->tw_timer)){+.-.}-{0:0}, at: call_timer_fn+0xe8/0x230
[ 264.273747][ C3]
[ 264.273747][ C3] which lock already depends on the new lock.
[ 264.273747][ C3]
[ 264.274115][ C3]
[ 264.274115][ C3] the existing dependency chain (in reverse order) is:
[ 264.274413][ C3]
[ 264.274413][ C3] -> #1 ((&tw->tw_timer)){+.-.}-{0:0}:
[ 264.274662][ C3] __lock_acquire+0xaf0/0x1570
[ 264.274863][ C3] lock_acquire.part.0+0xeb/0x330
[ 264.275012][ C3] __timer_delete_sync+0xe1/0x110
[ 264.275190][ C3] inet_twsk_deschedule_put+0xec/0x150
[ 264.275422][ C3] inet_twsk_purge+0x4c8/0x660
[ 264.275577][ C3] tcp_twsk_purge+0x112/0x160
[ 264.275767][ C3] tcp_sk_exit_batch+0x17/0x120
[ 264.275919][ C3] cleanup_net+0x4f3/0xa20
[ 264.276104][ C3] process_one_work+0xe2c/0x1730
[ 264.276269][ C3] worker_thread+0x587/0xd30
[ 264.276419][ C3] kthread+0x28a/0x350
[ 264.276619][ C3] ret_from_fork+0x31/0x70
[ 264.276786][ C3] ret_from_fork_asm+0x1a/0x30
[ 264.276983][ C3]
[ 264.276983][ C3] -> #0 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}:
[ 264.277214][ C3] check_prev_add+0x182/0xce0
[ 264.277366][ C3] validate_chain+0x71d/0x9b0
[ 264.277547][ C3] __lock_acquire+0xaf0/0x1570
[ 264.277705][ C3] lock_acquire.part.0+0xeb/0x330
[ 264.277859][ C3] _raw_spin_lock+0x30/0x40
[ 264.278010][ C3] tw_timer_handler+0xe9/0x130
[ 264.278186][ C3] call_timer_fn+0x13b/0x230
[ 264.278319][ C3] __run_timers+0x545/0x810
[ 264.278468][ C3] run_timer_softirq+0x7a/0x1b0
[ 264.278644][ C3] __do_softirq+0x1f8/0x5df
[ 264.278796][ C3] irq_exit_rcu+0x97/0xc0
[ 264.278968][ C3] sysvec_apic_timer_interrupt+0x75/0x80
[ 264.279167][ C3] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 264.279351][ C3] default_idle+0xf/0x20
[ 264.279524][ C3] default_idle_call+0x6d/0xb0
[ 264.279676][ C3] cpuidle_idle_call+0x1f4/0x280
[ 264.279845][ C3] do_idle+0xf9/0x160
[ 264.279993][ C3] cpu_startup_entry+0x54/0x60
[ 264.280145][ C3] start_secondary+0x21c/0x2b0
[ 264.280301][ C3] common_startup_64+0x12c/0x138
[ 264.280451][ C3]
[ 264.280451][ C3] other info that might help us debug this:
[ 264.280451][ C3]
[ 264.280734][ C3] Possible unsafe locking scenario:
[ 264.280734][ C3]
[ 264.280968][ C3] CPU0 CPU1
[ 264.281117][ C3] ---- ----
[ 264.281263][ C3] lock((&tw->tw_timer));
[ 264.281427][ C3] lock(&hashinfo->ehash_locks[i]);
[ 264.281647][ C3] lock((&tw->tw_timer));
[ 264.281834][ C3] lock(&hashinfo->ehash_locks[i]);
[ 264.281992][ C3]
[ 264.281992][ C3] *** DEADLOCK ***
[ 264.281992][ C3]
[ 264.282220][ C3] 1 lock held by swapper/3/0:
[ 264.282375][ C3] #0: ffffc90000288d68 ((&tw->tw_timer)){+.-.}-{0:0}, at: call_timer_fn+0xe8/0x230
[ 264.282636][ C3]
[ 264.282636][ C3] stack backtrace:
[ 264.282821][ C3] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.9.0-rc2-virtme #1
[ 264.283054][ C3] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 264.283414][ C3] Call Trace:
[ 264.283547][ C3]
[ 264.283624][ C3] dump_stack_lvl+0x82/0xd0
[ 264.283791][ C3] check_noncircular+0x2f9/0x3e0
[ 264.283955][ C3] ? get_stack_info+0x35/0xc0
[ 264.284117][ C3] ? __pfx_check_noncircular+0x10/0x10
[ 264.284290][ C3] ? common_startup_64+0x12c/0x138
[ 264.284445][ C3] ? alloc_chain_hlocks+0x4e0/0x720
[ 264.284593][ C3] check_prev_add+0x182/0xce0
[ 264.284747][ C3] validate_chain+0x71d/0x9b0
[ 264.284892][ C3] ? __pfx_validate_chain+0x10/0x10
[ 264.285059][ C3] ? hlock_class+0x4e/0x130
[ 264.285214][ C3] ? mark_lock+0x38/0x3e0
[ 264.285334][ C3] __lock_acquire+0xaf0/0x1570
[ 264.285493][ C3] ? __pfx_tw_timer_handler+0x10/0x10
[ 264.285666][ C3] lock_acquire.part.0+0xeb/0x330
[ 264.285816][ C3] ? tw_timer_handler+0xe9/0x130
[ 264.285963][ C3] ? __pfx_tw_timer_handler+0x10/0x10
[ 264.286096][ C3] ? lock_acquire.part.0+0xeb/0x330
[ 264.286280][ C3] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 264.286414][ C3] ? trace_lock_acquire+0x135/0x1c0
[ 264.286569][ C3] ? tw_timer_handler+0xe9/0x130
[ 264.286745][ C3] ? lock_acquire+0x32/0xc0
[ 264.286933][ C3] ? tw_timer_handler+0xe9/0x130
[ 264.287084][ C3] _raw_spin_lock+0x30/0x40
[ 264.287248][ C3] ? tw_timer_handler+0xe9/0x130
[ 264.287392][ C3] tw_timer_handler+0xe9/0x130
[ 264.287554][ C3] call_timer_fn+0x13b/0x230
[ 264.287702][ C3] ? call_timer_fn+0xe8/0x230
[ 264.287860][ C3] ? call_timer_fn+0xe8/0x230
[ 264.288028][ C3] ? __pfx_call_timer_fn+0x10/0x10
[ 264.288282][ C3] ? mark_lock+0x38/0x3e0
[ 264.288403][ C3] __run_timers+0x545/0x810
[ 264.288536][ C3] ? __pfx_tw_timer_handler+0x10/0x10
[ 264.288670][ C3] ? __pfx___run_timers+0x10/0x10
[ 264.288831][ C3] ? do_raw_spin_lock+0x131/0x270
[ 264.288963][ C3] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 264.289090][ C3] ? lock_acquire+0x32/0xc0
[ 264.289252][ C3] ? run_timer_softirq+0x72/0x1b0
[ 264.289402][ C3] run_timer_softirq+0x7a/0x1b0
[ 264.289551][ C3] __do_softirq+0x1f8/0x5df
[ 264.289688][ C3] irq_exit_rcu+0x97/0xc0
[ 264.289817][ C3] sysvec_apic_timer_interrupt+0x75/0x80
[ 264.289952][ C3]
[ 264.290021][ C3]
[ 264.290111][ C3] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 264.290274][ C3] RIP: 0010:default_idle+0xf/0x20
[ 264.290450][ C3] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d 03 f0 39 00 fb f4 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
[ 264.290945][ C3] RSP: 0018:ffffc9000016fdf8 EFLAGS: 00000246
[ 264.291124][ C3] RAX: 0000000000347227 RBX: 1ffff9200002dfc1 RCX: ffffffff98cc2b75
[ 264.291331][ C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff966aa3b4
[ 264.291564][ C3] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1005f3709c
[ 264.291757][ C3] R10: ffff88802f9b84e3 R11: ffff88802f9bdc40 R12: 0000000000000000
[ 264.291957][ C3] R13: ffff888001b88040 R14: dffffc0000000000 R15: 0000000000000000
[ 264.292152][ C3] ? ct_kernel_exit.constprop.0+0xc5/0xf0
[ 264.292288][ C3] ? cpuidle_idle_call+0x1f4/0x280
[ 264.292425][ C3] default_idle_call+0x6d/0xb0
[ 264.292581][ C3] cpuidle_idle_call+0x1f4/0x280
[ 264.292722][ C3] ? __pfx_cpuidle_idle_call+0x10/0x10
[ 264.292869][ C3] ? tsc_verify_tsc_adjust+0x5e/0x2b0
[ 264.293013][ C3] do_idle+0xf9/0x160
[ 264.293120][ C3] cpu_startup_entry+0x54/0x60
[ 264.293261][ C3] start_secondary+0x21c/0x2b0
[ 264.293399][ C3] ? __pfx_start_secondary+0x10/0x10
[ 264.293551][ C3] common_startup_64+0x12c/0x138
[ 264.293702][ C3]