[ 25.932087][ T261] ip (261) used greatest stack depth: 23200 bytes left [ 44.051910][ T368] netem: version 1.3 [ 417.697326][ T10] [ 417.697421][ T10] ====================================================== [ 417.697581][ T10] WARNING: possible circular locking dependency detected [ 417.697738][ T10] 6.9.0-rc2-virtme #1 Not tainted [ 417.697862][ T10] ------------------------------------------------------ [ 417.698013][ T10] kworker/u16:0/10 is trying to acquire lock: [ 417.698167][ T10] ffff888015996268 ((&tw->tw_timer)){+.-.}-{0:0}, at: __timer_delete_sync+0xd3/0x110 [ 417.698398][ T10] [ 417.698398][ T10] but task is already holding lock: [ 417.698578][ T10] ffff888003f8b998 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}, at: inet_twsk_deschedule_put+0xe0/0x150 [ 417.698834][ T10] [ 417.698834][ T10] which lock already depends on the new lock. [ 417.698834][ T10] [ 417.699087][ T10] [ 417.699087][ T10] the existing dependency chain (in reverse order) is: [ 417.699296][ T10] [ 417.699296][ T10] -> #1 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}: [ 417.699488][ T10] __lock_acquire+0xaf0/0x1570 [ 417.699618][ T10] lock_acquire.part.0+0xeb/0x330 [ 417.699742][ T10] _raw_spin_lock+0x30/0x40 [ 417.699869][ T10] tw_timer_handler+0xe9/0x130 [ 417.699992][ T10] call_timer_fn+0x13b/0x230 [ 417.700115][ T10] __run_timers+0x545/0x810 [ 417.700241][ T10] run_timer_softirq+0x7a/0x1b0 [ 417.700367][ T10] __do_softirq+0x1f8/0x5df [ 417.700492][ T10] irq_exit_rcu+0x97/0xc0 [ 417.700619][ T10] sysvec_apic_timer_interrupt+0x75/0x80 [ 417.700774][ T10] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 417.700923][ T10] default_idle+0xf/0x20 [ 417.701047][ T10] default_idle_call+0x6d/0xb0 [ 417.701178][ T10] cpuidle_idle_call+0x1f4/0x280 [ 417.701307][ T10] do_idle+0xf9/0x160 [ 417.701429][ T10] cpu_startup_entry+0x54/0x60 [ 417.701551][ T10] rest_init+0x14f/0x260 [ 417.701670][ T10] start_kernel+0x318/0x3d0 [ 417.701795][ T10] x86_64_start_reservations+0x18/0x30 [ 417.701949][ T10] x86_64_start_kernel+0xba/0x110 [ 417.702078][ T10] common_startup_64+0x12c/0x138 [ 417.702206][ T10] [ 417.702206][ T10] -> #0 ((&tw->tw_timer)){+.-.}-{0:0}: [ 417.702398][ T10] check_prev_add+0x182/0xce0 [ 417.702522][ T10] validate_chain+0x71d/0x9b0 [ 417.702646][ T10] __lock_acquire+0xaf0/0x1570 [ 417.702773][ T10] lock_acquire.part.0+0xeb/0x330 [ 417.702897][ T10] __timer_delete_sync+0xe1/0x110 [ 417.703022][ T10] inet_twsk_deschedule_put+0xec/0x150 [ 417.703178][ T10] inet_twsk_purge+0x4c8/0x660 [ 417.703305][ T10] tcp_twsk_purge+0x112/0x160 [ 417.703430][ T10] tcp_sk_exit_batch+0x17/0x120 [ 417.703555][ T10] cleanup_net+0x4f3/0xa20 [ 417.703681][ T10] process_one_work+0xe2c/0x1730 [ 417.703807][ T10] worker_thread+0x587/0xd30 [ 417.703931][ T10] kthread+0x28a/0x350 [ 417.704056][ T10] ret_from_fork+0x31/0x70 [ 417.704181][ T10] ret_from_fork_asm+0x1a/0x30 [ 417.704309][ T10] [ 417.704309][ T10] other info that might help us debug this: [ 417.704309][ T10] [ 417.704547][ T10] Possible unsafe locking scenario: [ 417.704547][ T10] [ 417.704737][ T10] CPU0 CPU1 [ 417.704859][ T10] ---- ---- [ 417.704985][ T10] lock(&hashinfo->ehash_locks[i]); [ 417.705109][ T10] lock((&tw->tw_timer)); [ 417.705266][ T10] lock(&hashinfo->ehash_locks[i]); [ 417.705447][ T10] lock((&tw->tw_timer)); [ 417.705576][ T10] [ 417.705576][ T10] *** DEADLOCK *** [ 417.705576][ T10] [ 417.705763][ T10] 4 locks held by kworker/u16:0/10: [ 417.705884][ T10] #0: ffff8880018fa948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 417.706141][ T10] #1: ffffc900000afda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 417.706360][ T10] #2: ffffffff8d19de50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xa20 [ 417.706586][ T10] #3: ffff888003f8b998 (&hashinfo->ehash_locks[i]){+.-.}-{2:2}, at: inet_twsk_deschedule_put+0xe0/0x150 [ 417.706838][ T10] [ 417.706838][ T10] stack backtrace: [ 417.706990][ T10] CPU: 1 PID: 10 Comm: kworker/u16:0 Not tainted 6.9.0-rc2-virtme #1 [ 417.707175][ T10] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 417.707445][ T10] Workqueue: netns cleanup_net [ 417.707580][ T10] Call Trace: [ 417.707675][ T10] [ 417.707738][ T10] dump_stack_lvl+0x82/0xd0 [ 417.707862][ T10] check_noncircular+0x2f9/0x3e0 [ 417.707986][ T10] ? __pfx_check_noncircular+0x10/0x10 [ 417.708108][ T10] ? __pfx_stack_trace_save+0x10/0x10 [ 417.708231][ T10] ? alloc_chain_hlocks+0x4e0/0x720 [ 417.708356][ T10] check_prev_add+0x182/0xce0 [ 417.708479][ T10] validate_chain+0x71d/0x9b0 [ 417.708600][ T10] ? __pfx_validate_chain+0x10/0x10 [ 417.708722][ T10] ? hlock_class+0x4e/0x130 [ 417.708844][ T10] ? mark_lock+0x38/0x3e0 [ 417.708940][ T10] __lock_acquire+0xaf0/0x1570 [ 417.709062][ T10] ? __lock_acquire+0xa40/0x1570 [ 417.709196][ T10] ? __timer_delete_sync+0xd3/0x110 [ 417.709319][ T10] lock_acquire.part.0+0xeb/0x330 [ 417.709440][ T10] ? __timer_delete_sync+0xd3/0x110 [ 417.709561][ T10] ? inet_twsk_deschedule_put+0xe0/0x150 [ 417.709683][ T10] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 417.709808][ T10] ? trace_lock_acquire+0x135/0x1c0 [ 417.709928][ T10] ? __timer_delete_sync+0xd3/0x110 [ 417.710048][ T10] ? lock_acquire+0x32/0xc0 [ 417.710167][ T10] ? __timer_delete_sync+0xd3/0x110 [ 417.710288][ T10] __timer_delete_sync+0xe1/0x110 [ 417.710409][ T10] ? __timer_delete_sync+0xd3/0x110 [ 417.710529][ T10] inet_twsk_deschedule_put+0xec/0x150 [ 417.710648][ T10] ? inet_twsk_purge+0x412/0x660 [ 417.710768][ T10] inet_twsk_purge+0x4c8/0x660 [ 417.710891][ T10] ? inet_twsk_purge+0x126/0x660 [ 417.711011][ T10] ? __pfx_inet_twsk_purge+0x10/0x10 [ 417.711129][ T10] ? mark_lock+0x38/0x3e0 [ 417.711222][ T10] tcp_twsk_purge+0x112/0x160 [ 417.711348][ T10] tcp_sk_exit_batch+0x17/0x120 [ 417.711470][ T10] cleanup_net+0x4f3/0xa20 [ 417.711589][ T10] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 417.711710][ T10] ? __pfx_cleanup_net+0x10/0x10 [ 417.711834][ T10] ? trace_lock_acquire+0x135/0x1c0 [ 417.711957][ T10] ? process_one_work+0xde2/0x1730 [ 417.712083][ T10] ? lock_acquire+0x32/0xc0 [ 417.712205][ T10] ? process_one_work+0xde2/0x1730 [ 417.712327][ T10] process_one_work+0xe2c/0x1730 [ 417.712448][ T10] ? __pfx___lock_release+0x10/0x10 [ 417.712570][ T10] ? __pfx_process_one_work+0x10/0x10 [ 417.712696][ T10] ? assign_work+0x16c/0x240 [ 417.712821][ T10] worker_thread+0x587/0xd30 [ 417.712949][ T10] ? __pfx_worker_thread+0x10/0x10 [ 417.713069][ T10] kthread+0x28a/0x350 [ 417.713158][ T10] ? __pfx_kthread+0x10/0x10 [ 417.713279][ T10] ret_from_fork+0x31/0x70 [ 417.713401][ T10] ? __pfx_kthread+0x10/0x10 [ 417.713523][ T10] ret_from_fork_asm+0x1a/0x30 [ 417.713648][ T10]