[ 30.575267][ T297] netem: version 1.3 [ 30.581823][ T297] modprobe (297) used greatest stack depth: 24192 bytes left [ 144.759589][ T73] [ 144.759697][ T73] ====================================================== [ 144.759866][ T73] WARNING: possible circular locking dependency detected [ 144.760022][ T73] 6.9.0-rc2-virtme #1 Not tainted [ 144.760147][ T73] ------------------------------------------------------ [ 144.760303][ T73] kworker/u16:1/73 is trying to acquire lock: [ 144.760459][ T73] ffff888006651588 ((&tw->tw_timer)){+.-.}-{0:0}, at: __timer_delete_sync+0xd3/0x110 [ 144.760690][ T73] [ 144.760690][ T73] but task is already holding lock: [ 144.760876][ T73] ffff888003e03958 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}, at: inet_twsk_deschedule_put+0xe0/0x150 [ 144.761130][ T73] [ 144.761130][ T73] which lock already depends on the new lock. [ 144.761130][ T73] [ 144.761373][ T73] [ 144.761373][ T73] the existing dependency chain (in reverse order) is: [ 144.761592][ T73] [ 144.761592][ T73] -> #1 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}: [ 144.761777][ T73] __lock_acquire+0xaf0/0x1570 [ 144.761909][ T73] lock_acquire.part.0+0xeb/0x330 [ 144.762037][ T73] _raw_spin_lock+0x30/0x40 [ 144.762165][ T73] tw_timer_handler+0xe9/0x130 [ 144.762281][ T73] call_timer_fn+0x13b/0x230 [ 144.762409][ T73] __run_timers+0x545/0x810 [ 144.762531][ T73] run_timer_softirq+0x7a/0x1b0 [ 144.762662][ T73] __do_softirq+0x1f8/0x5df [ 144.762786][ T73] irq_exit_rcu+0x97/0xc0 [ 144.762905][ T73] sysvec_apic_timer_interrupt+0x75/0x80 [ 144.763062][ T73] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 144.763225][ T73] default_idle+0xf/0x20 [ 144.763349][ T73] default_idle_call+0x6d/0xb0 [ 144.763473][ T73] cpuidle_idle_call+0x1f4/0x280 [ 144.763598][ T73] do_idle+0xf9/0x160 [ 144.763720][ T73] cpu_startup_entry+0x54/0x60 [ 144.763842][ T73] start_secondary+0x21c/0x2b0 [ 144.763967][ T73] common_startup_64+0x12c/0x138 [ 144.764111][ T73] [ 144.764111][ T73] -> #0 ((&tw->tw_timer)){+.-.}-{0:0}: [ 144.764300][ T73] check_prev_add+0x182/0xce0 [ 144.764424][ T73] validate_chain+0x71d/0x9b0 [ 144.764557][ T73] __lock_acquire+0xaf0/0x1570 [ 144.764679][ T73] lock_acquire.part.0+0xeb/0x330 [ 144.764804][ T73] __timer_delete_sync+0xe1/0x110 [ 144.764929][ T73] inet_twsk_deschedule_put+0xec/0x150 [ 144.765082][ T73] inet_twsk_purge+0x4c8/0x660 [ 144.765205][ T73] tcp_twsk_purge+0x112/0x160 [ 144.765329][ T73] tcp_sk_exit_batch+0x17/0x120 [ 144.765454][ T73] cleanup_net+0x4f3/0xa20 [ 144.765578][ T73] process_one_work+0xe2c/0x1730 [ 144.765703][ T73] worker_thread+0x587/0xd30 [ 144.765824][ T73] kthread+0x28a/0x350 [ 144.765949][ T73] ret_from_fork+0x31/0x70 [ 144.766075][ T73] ret_from_fork_asm+0x1a/0x30 [ 144.766201][ T73] [ 144.766201][ T73] other info that might help us debug this: [ 144.766201][ T73] [ 144.766434][ T73] Possible unsafe locking scenario: [ 144.766434][ T73] [ 144.766613][ T73] CPU0 CPU1 [ 144.766742][ T73] ---- ---- [ 144.766868][ T73] lock(&hashinfo->ehash_locks[i]); [ 144.766997][ T73] lock((&tw->tw_timer)); [ 144.767151][ T73] lock(&hashinfo->ehash_locks[i]); [ 144.767338][ T73] lock((&tw->tw_timer)); [ 144.767465][ T73] [ 144.767465][ T73] *** DEADLOCK *** [ 144.767465][ T73] [ 144.767640][ T73] 4 locks held by kworker/u16:1/73: [ 144.767766][ T73] #0: ffff8880018fa948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 144.768018][ T73] #1: ffffc900004efda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 144.768235][ T73] #2: ffffffffa0b9de50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xa20 [ 144.768454][ T73] #3: ffff888003e03958 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}, at: inet_twsk_deschedule_put+0xe0/0x150 [ 144.768705][ T73] [ 144.768705][ T73] stack backtrace: [ 144.768854][ T73] CPU: 1 PID: 73 Comm: kworker/u16:1 Not tainted 6.9.0-rc2-virtme #1 [ 144.769038][ T73] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 144.769303][ T73] Workqueue: netns cleanup_net [ 144.769429][ T73] Call Trace: [ 144.769524][ T73] [ 144.769595][ T73] dump_stack_lvl+0x82/0xd0 [ 144.769719][ T73] check_noncircular+0x2f9/0x3e0 [ 144.769841][ T73] ? __pfx_check_noncircular+0x10/0x10 [ 144.769965][ T73] ? __pfx_stack_trace_save+0x10/0x10 [ 144.770089][ T73] ? alloc_chain_hlocks+0x4e0/0x720 [ 144.770214][ T73] check_prev_add+0x182/0xce0 [ 144.770337][ T73] validate_chain+0x71d/0x9b0 [ 144.770460][ T73] ? __pfx_validate_chain+0x10/0x10 [ 144.770579][ T73] ? hlock_class+0x4e/0x130 [ 144.770698][ T73] ? mark_lock+0x38/0x3e0 [ 144.770792][ T73] __lock_acquire+0xaf0/0x1570 [ 144.770913][ T73] ? __lock_acquire+0xa40/0x1570 [ 144.771034][ T73] ? __timer_delete_sync+0xd3/0x110 [ 144.771155][ T73] lock_acquire.part.0+0xeb/0x330 [ 144.771279][ T73] ? __timer_delete_sync+0xd3/0x110 [ 144.771404][ T73] ? inet_twsk_deschedule_put+0xe0/0x150 [ 144.771525][ T73] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 144.771647][ T73] ? trace_lock_acquire+0x135/0x1c0 [ 144.771767][ T73] ? __timer_delete_sync+0xd3/0x110 [ 144.771887][ T73] ? lock_acquire+0x32/0xc0 [ 144.772008][ T73] ? __timer_delete_sync+0xd3/0x110 [ 144.772130][ T73] __timer_delete_sync+0xe1/0x110 [ 144.772251][ T73] ? __timer_delete_sync+0xd3/0x110 [ 144.772372][ T73] inet_twsk_deschedule_put+0xec/0x150 [ 144.772496][ T73] ? inet_twsk_purge+0x412/0x660 [ 144.772617][ T73] inet_twsk_purge+0x4c8/0x660 [ 144.772737][ T73] ? inet_twsk_purge+0x126/0x660 [ 144.772861][ T73] ? __pfx_inet_twsk_purge+0x10/0x10 [ 144.772985][ T73] ? mark_lock+0x38/0x3e0 [ 144.773080][ T73] tcp_twsk_purge+0x112/0x160 [ 144.773201][ T73] tcp_sk_exit_batch+0x17/0x120 [ 144.773328][ T73] cleanup_net+0x4f3/0xa20 [ 144.773453][ T73] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 144.773580][ T73] ? __pfx_cleanup_net+0x10/0x10 [ 144.773703][ T73] ? trace_lock_acquire+0x135/0x1c0 [ 144.773824][ T73] ? process_one_work+0xde2/0x1730 [ 144.773951][ T73] ? lock_acquire+0x32/0xc0 [ 144.774073][ T73] ? process_one_work+0xde2/0x1730 [ 144.774197][ T73] process_one_work+0xe2c/0x1730 [ 144.774324][ T73] ? __pfx___lock_release+0x10/0x10 [ 144.774451][ T73] ? __pfx_process_one_work+0x10/0x10 [ 144.774575][ T73] ? assign_work+0x16c/0x240 [ 144.774697][ T73] worker_thread+0x587/0xd30 [ 144.774818][ T73] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 144.774969][ T73] ? __pfx_worker_thread+0x10/0x10 [ 144.775156][ T73] ? __pfx_worker_thread+0x10/0x10 [ 144.775283][ T73] kthread+0x28a/0x350 [ 144.775379][ T73] ? __pfx_kthread+0x10/0x10 [ 144.775497][ T73] ret_from_fork+0x31/0x70 [ 144.775688][ T73] ? __pfx_kthread+0x10/0x10 [ 144.775809][ T73] ret_from_fork_asm+0x1a/0x30 [ 144.775933][ T73]