[ 58.587321][ T71] [ 58.587417][ T71] ===================================================== [ 58.587579][ T71] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected [ 58.587772][ T71] 6.9.0-rc1-virtme #1 Not tainted [ 58.587896][ T71] ----------------------------------------------------- [ 58.588056][ T71] kworker/u16:1/71 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 58.588245][ T71] ffff88800552ac28 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70 [ 58.588487][ T71] [ 58.588487][ T71] and this task is already holding: [ 58.588680][ T71] ffff88803613f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 58.588912][ T71] which would create a new lock dependency: [ 58.589078][ T71] (&ul->lock#2){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2} [ 58.589275][ T71] [ 58.589275][ T71] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 58.589494][ T71] (&ul->lock#2){+.-.}-{2:2} [ 58.589499][ T71] [ 58.589499][ T71] ... which became SOFTIRQ-irq-safe at: [ 58.589808][ T71] __lock_acquire+0x797/0x1570 [ 58.589946][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.590071][ T71] _raw_spin_lock_bh+0x38/0x50 [ 58.590201][ T71] rt_add_uncached_list+0x55/0x1b0 [ 58.590341][ T71] rt_cache_route+0xfd/0x150 [ 58.590467][ T71] ip_route_input_slow+0x1001/0x2230 [ 58.590599][ T71] ip_route_input_noref+0xe6/0x230 [ 58.590741][ T71] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 58.590902][ T71] ip_rcv+0x199/0x740 [ 58.591001][ T71] __netif_receive_skb_one_core+0x166/0x1b0 [ 58.591166][ T71] process_backlog+0x103/0x670 [ 58.591294][ T71] __napi_poll.constprop.0+0xa2/0x460 [ 58.591421][ T71] net_rx_action+0x49f/0xc00 [ 58.591555][ T71] __do_softirq+0x1f8/0x5df [ 58.591686][ T71] do_softirq+0x4d/0xa0 [ 58.591784][ T71] __local_bh_enable_ip+0xf6/0x120 [ 58.591910][ T71] __dev_queue_xmit+0x6bc/0x16e0 [ 58.592040][ T71] ip6_finish_output2+0x512/0xf60 [ 58.592167][ T71] ip6_finish_output+0x553/0xdf0 [ 58.592293][ T71] ip6_output+0x1f3/0x770 [ 58.592423][ T71] NF_HOOK.constprop.0+0xe2/0x680 [ 58.592550][ T71] mld_sendpack+0x62d/0xbc0 [ 58.592676][ T71] mld_ifc_work+0x32/0x1f0 [ 58.592808][ T71] process_one_work+0xe2c/0x1730 [ 58.592935][ T71] worker_thread+0x587/0xd30 [ 58.593062][ T71] kthread+0x28a/0x350 [ 58.593159][ T71] ret_from_fork+0x31/0x70 [ 58.593287][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.593417][ T71] [ 58.593417][ T71] to a SOFTIRQ-irq-unsafe lock: [ 58.593580][ T71] (&p->alloc_lock){+.+.}-{2:2} [ 58.593584][ T71] [ 58.593584][ T71] ... which became SOFTIRQ-irq-unsafe at: [ 58.593896][ T71] ... [ 58.593897][ T71] __lock_acquire+0x797/0x1570 [ 58.594090][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.594218][ T71] _raw_spin_lock+0x30/0x40 [ 58.594346][ T71] set_mems_allowed+0x1d/0x210 [ 58.594476][ T71] kernel_init_freeable+0x72/0x310 [ 58.594600][ T71] kernel_init+0x20/0x200 [ 58.594726][ T71] ret_from_fork+0x31/0x70 [ 58.594850][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.594977][ T71] [ 58.594977][ T71] other info that might help us debug this: [ 58.594977][ T71] [ 58.595229][ T71] Possible interrupt unsafe locking scenario: [ 58.595229][ T71] [ 58.595418][ T71] CPU0 CPU1 [ 58.595549][ T71] ---- ---- [ 58.595674][ T71] lock(&p->alloc_lock); [ 58.595772][ T71] local_irq_disable(); [ 58.595932][ T71] lock(&ul->lock#2); [ 58.596092][ T71] lock(&p->alloc_lock); [ 58.596250][ T71] [ 58.596346][ T71] lock(&ul->lock#2); [ 58.596444][ T71] [ 58.596444][ T71] *** DEADLOCK *** [ 58.596444][ T71] [ 58.596631][ T71] 5 locks held by kworker/u16:1/71: [ 58.596765][ T71] #0: ffff88800190a948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 58.597024][ T71] #1: ffffc900004cfda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 58.597266][ T71] #2: ffffffff8d99bc50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xb60 [ 58.597494][ T71] #3: ffffffff8d9b7208 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0 [ 58.597752][ T71] #4: ffff88803613f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 58.597986][ T71] [ 58.597986][ T71] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 58.598231][ T71] -> (&ul->lock#2){+.-.}-{2:2} { [ 58.598366][ T71] HARDIRQ-ON-W at: [ 58.598464][ T71] __lock_acquire+0x797/0x1570 [ 58.598629][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.598790][ T71] _raw_spin_lock_bh+0x38/0x50 [ 58.598954][ T71] rt_add_uncached_list+0x55/0x1b0 [ 58.599117][ T71] rt_cache_route+0xfd/0x150 [ 58.599276][ T71] rt_set_nexthop.constprop.0+0x283/0xe90 [ 58.599470][ T71] __mkroute_output+0x4fa/0x1210 [ 58.599628][ T71] ip_route_output_key_hash+0x15d/0x2b0 [ 58.599822][ T71] ip_route_output_flow+0x23/0x140 [ 58.599987][ T71] udp_tunnel_dst_lookup+0x236/0x3b0 [ 58.600150][ T71] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 58.600357][ T71] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 58.600534][ T71] dev_hard_start_xmit+0x10e/0x360 [ 58.600704][ T71] __dev_queue_xmit+0xe79/0x16e0 [ 58.600865][ T71] ip6_finish_output2+0x512/0xf60 [ 58.601028][ T71] ip6_finish_output+0x553/0xdf0 [ 58.601188][ T71] ip6_output+0x1f3/0x770 [ 58.601347][ T71] NF_HOOK.constprop.0+0xe2/0x680 [ 58.601508][ T71] mld_sendpack+0x62d/0xbc0 [ 58.601671][ T71] mld_ifc_work+0x32/0x1f0 [ 58.601836][ T71] process_one_work+0xe2c/0x1730 [ 58.601996][ T71] worker_thread+0x587/0xd30 [ 58.602160][ T71] kthread+0x28a/0x350 [ 58.602295][ T71] ret_from_fork+0x31/0x70 [ 58.602457][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.602617][ T71] IN-SOFTIRQ-W at: [ 58.602720][ T71] __lock_acquire+0x797/0x1570 [ 58.602964][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.603129][ T71] _raw_spin_lock_bh+0x38/0x50 [ 58.603294][ T71] rt_add_uncached_list+0x55/0x1b0 [ 58.603455][ T71] rt_cache_route+0xfd/0x150 [ 58.603619][ T71] ip_route_input_slow+0x1001/0x2230 [ 58.603873][ T71] ip_route_input_noref+0xe6/0x230 [ 58.604039][ T71] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 58.604232][ T71] ip_rcv+0x199/0x740 [ 58.604439][ T71] __netif_receive_skb_one_core+0x166/0x1b0 [ 58.604631][ T71] process_backlog+0x103/0x670 [ 58.604793][ T71] __napi_poll.constprop.0+0xa2/0x460 [ 58.604953][ T71] net_rx_action+0x49f/0xc00 [ 58.605115][ T71] __do_softirq+0x1f8/0x5df [ 58.605353][ T71] do_softirq+0x4d/0xa0 [ 58.605511][ T71] __local_bh_enable_ip+0xf6/0x120 [ 58.605669][ T71] __dev_queue_xmit+0x6bc/0x16e0 [ 58.605829][ T71] ip6_finish_output2+0x512/0xf60 [ 58.605987][ T71] ip6_finish_output+0x553/0xdf0 [ 58.606146][ T71] ip6_output+0x1f3/0x770 [ 58.606304][ T71] NF_HOOK.constprop.0+0xe2/0x680 [ 58.606536][ T71] mld_sendpack+0x62d/0xbc0 [ 58.606694][ T71] mld_ifc_work+0x32/0x1f0 [ 58.606852][ T71] process_one_work+0xe2c/0x1730 [ 58.607085][ T71] worker_thread+0x587/0xd30 [ 58.607247][ T71] kthread+0x28a/0x350 [ 58.607377][ T71] ret_from_fork+0x31/0x70 [ 58.607540][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.607772][ T71] INITIAL USE at: [ 58.607867][ T71] __lock_acquire+0x797/0x1570 [ 58.608025][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.608185][ T71] _raw_spin_lock_bh+0x38/0x50 [ 58.608506][ T71] rt_add_uncached_list+0x55/0x1b0 [ 58.608664][ T71] rt_cache_route+0xfd/0x150 [ 58.608826][ T71] rt_set_nexthop.constprop.0+0x283/0xe90 [ 58.609019][ T71] __mkroute_output+0x4fa/0x1210 [ 58.609180][ T71] ip_route_output_key_hash+0x15d/0x2b0 [ 58.609342][ T71] ip_route_output_flow+0x23/0x140 [ 58.609499][ T71] udp_tunnel_dst_lookup+0x236/0x3b0 [ 58.609734][ T71] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 58.609909][ T71] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 58.610082][ T71] dev_hard_start_xmit+0x10e/0x360 [ 58.610333][ T71] __dev_queue_xmit+0xe79/0x16e0 [ 58.610495][ T71] ip6_finish_output2+0x512/0xf60 [ 58.610657][ T71] ip6_finish_output+0x553/0xdf0 [ 58.610814][ T71] ip6_output+0x1f3/0x770 [ 58.610973][ T71] NF_HOOK.constprop.0+0xe2/0x680 [ 58.611133][ T71] mld_sendpack+0x62d/0xbc0 [ 58.611295][ T71] mld_ifc_work+0x32/0x1f0 [ 58.611535][ T71] process_one_work+0xe2c/0x1730 [ 58.611694][ T71] worker_thread+0x587/0xd30 [ 58.611853][ T71] kthread+0x28a/0x350 [ 58.611984][ T71] ret_from_fork+0x31/0x70 [ 58.612149][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.612312][ T71] } [ 58.612379][ T71] ... key at: [] __key.7+0x0/0x40 [ 58.612689][ T71] [ 58.612689][ T71] the dependencies between the lock to be acquired [ 58.612691][ T71] and SOFTIRQ-irq-unsafe lock: [ 58.613029][ T71] -> (&p->alloc_lock){+.+.}-{2:2} { [ 58.613165][ T71] HARDIRQ-ON-W at: [ 58.613262][ T71] __lock_acquire+0x797/0x1570 [ 58.613504][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.613668][ T71] _raw_spin_lock+0x30/0x40 [ 58.613842][ T71] set_mems_allowed+0x1d/0x210 [ 58.614008][ T71] kernel_init_freeable+0x72/0x310 [ 58.614172][ T71] kernel_init+0x20/0x200 [ 58.614418][ T71] ret_from_fork+0x31/0x70 [ 58.614582][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.614743][ T71] SOFTIRQ-ON-W at: [ 58.614838][ T71] __lock_acquire+0x797/0x1570 [ 58.615004][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.615166][ T71] _raw_spin_lock+0x30/0x40 [ 58.615329][ T71] set_mems_allowed+0x1d/0x210 [ 58.615490][ T71] kernel_init_freeable+0x72/0x310 [ 58.615741][ T71] kernel_init+0x20/0x200 [ 58.615900][ T71] ret_from_fork+0x31/0x70 [ 58.616063][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.616225][ T71] INITIAL USE at: [ 58.616324][ T71] __lock_acquire+0x797/0x1570 [ 58.616561][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.616727][ T71] _raw_spin_lock+0x30/0x40 [ 58.616890][ T71] set_mems_allowed+0x1d/0x210 [ 58.617056][ T71] kernel_init_freeable+0x72/0x310 [ 58.617295][ T71] kernel_init+0x20/0x200 [ 58.617460][ T71] ret_from_fork+0x31/0x70 [ 58.617621][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.617784][ T71] } [ 58.617851][ T71] ... key at: [] __key.388+0x0/0x40 [ 58.618188][ T71] ... acquired at: [ 58.618284][ T71] check_prev_add+0x194/0xce0 [ 58.618414][ T71] validate_chain+0x71d/0x9b0 [ 58.618542][ T71] __lock_acquire+0xaf0/0x1570 [ 58.618673][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.618875][ T71] _raw_spin_lock+0x30/0x40 [ 58.619005][ T71] __get_task_comm+0x27/0x70 [ 58.619136][ T71] ref_tracker_alloc+0x2ee/0x490 [ 58.619268][ T71] rt_flush_dev+0x371/0x670 [ 58.619476][ T71] fib_netdev_event+0x248/0x5a0 [ 58.619605][ T71] notifier_call_chain+0xcd/0x150 [ 58.619735][ T71] unregister_netdevice_many_notify+0x548/0x1190 [ 58.619895][ T71] default_device_exit_batch+0x234/0x2d0 [ 58.620048][ T71] cleanup_net+0x4f3/0xb60 [ 58.620171][ T71] process_one_work+0xe2c/0x1730 [ 58.620463][ T71] worker_thread+0x587/0xd30 [ 58.620589][ T71] kthread+0x28a/0x350 [ 58.620686][ T71] ret_from_fork+0x31/0x70 [ 58.620812][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.620940][ T71] [ 58.621005][ T71] [ 58.621005][ T71] stack backtrace: [ 58.621160][ T71] CPU: 1 PID: 71 Comm: kworker/u16:1 Not tainted 6.9.0-rc1-virtme #1 [ 58.621348][ T71] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 58.621629][ T71] Workqueue: netns cleanup_net [ 58.621835][ T71] Call Trace: [ 58.621937][ T71] [ 58.622004][ T71] dump_stack_lvl+0x82/0xd0 [ 58.622136][ T71] print_bad_irq_dependency+0x957/0xd40 [ 58.622264][ T71] ? __pfx_print_bad_irq_dependency+0x10/0x10 [ 58.622495][ T71] ? __pfx_usage_skip+0x10/0x10 [ 58.622622][ T71] ? __bfs+0x246/0x650 [ 58.622717][ T71] ? __pfx_usage_match+0x10/0x10 [ 58.622844][ T71] check_irq_usage+0x596/0x850 [ 58.623044][ T71] ? arch_stack_walk+0x8c/0xf0 [ 58.623172][ T71] ? __pfx_check_irq_usage+0x10/0x10 [ 58.623298][ T71] ? __bfs+0x246/0x650 [ 58.623394][ T71] ? __pfx_hlock_conflict+0x10/0x10 [ 58.623522][ T71] ? check_path.constprop.0+0x24/0x50 [ 58.623722][ T71] ? check_noncircular+0x14c/0x3e0 [ 58.623855][ T71] ? __pfx_check_noncircular+0x10/0x10 [ 58.623981][ T71] ? find_held_lock+0x2c/0x110 [ 58.624109][ T71] ? check_prev_add+0x194/0xce0 [ 58.624307][ T71] check_prev_add+0x194/0xce0 [ 58.624435][ T71] ? validate_chain+0x130/0x9b0 [ 58.624562][ T71] validate_chain+0x71d/0x9b0 [ 58.624690][ T71] ? mark_lock+0x38/0x3e0 [ 58.624785][ T71] ? __pfx_validate_chain+0x10/0x10 [ 58.624985][ T71] ? hlock_class+0x4e/0x130 [ 58.625113][ T71] ? mark_lock+0x38/0x3e0 [ 58.625210][ T71] __lock_acquire+0xaf0/0x1570 [ 58.625340][ T71] lock_acquire.part.0+0xeb/0x330 [ 58.625538][ T71] ? __get_task_comm+0x27/0x70 [ 58.625663][ T71] ? mark_lock+0x38/0x3e0 [ 58.625759][ T71] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 58.625887][ T71] ? trace_lock_acquire+0x135/0x1c0 [ 58.626016][ T71] ? __get_task_comm+0x27/0x70 [ 58.626215][ T71] ? lock_acquire+0x32/0xc0 [ 58.626344][ T71] ? __get_task_comm+0x27/0x70 [ 58.626472][ T71] _raw_spin_lock+0x30/0x40 [ 58.626599][ T71] ? __get_task_comm+0x27/0x70 [ 58.626798][ T71] __get_task_comm+0x27/0x70 [ 58.626925][ T71] ref_tracker_alloc+0x2ee/0x490 [ 58.627054][ T71] ? __pfx_ref_tracker_alloc+0x10/0x10 [ 58.627180][ T71] ? rt_flush_dev+0x371/0x670 [ 58.627386][ T71] ? fib_netdev_event+0x248/0x5a0 [ 58.627512][ T71] ? notifier_call_chain+0xcd/0x150 [ 58.627638][ T71] ? unregister_netdevice_many_notify+0x548/0x1190 [ 58.627794][ T71] ? default_device_exit_batch+0x234/0x2d0 [ 58.627952][ T71] ? cleanup_net+0x4f3/0xb60 [ 58.628079][ T71] ? process_one_work+0xe2c/0x1730 [ 58.628206][ T71] ? worker_thread+0x587/0xd30 [ 58.628341][ T71] ? kthread+0x28a/0x350 [ 58.628439][ T71] ? ret_from_fork+0x31/0x70 [ 58.628565][ T71] ? ret_from_fork_asm+0x1a/0x30 [ 58.628694][ T71] rt_flush_dev+0x371/0x670 [ 58.628820][ T71] ? __timer_delete_sync+0xd3/0x110 [ 58.628948][ T71] ? __timer_delete_sync+0xa7/0x110 [ 58.629215][ T71] fib_netdev_event+0x248/0x5a0 [ 58.629340][ T71] notifier_call_chain+0xcd/0x150 [ 58.629468][ T71] unregister_netdevice_many_notify+0x548/0x1190 [ 58.629627][ T71] ? mutex_is_locked+0x17/0x50 [ 58.629825][ T71] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 58.629980][ T71] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 58.630139][ T71] default_device_exit_batch+0x234/0x2d0 [ 58.630350][ T71] ? __pfx_default_device_exit_batch+0x10/0x10 [ 58.630506][ T71] ? ops_exit_list+0xb4/0x170 [ 58.630634][ T71] cleanup_net+0x4f3/0xb60 [ 58.630760][ T71] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 58.630888][ T71] ? __pfx_cleanup_net+0x10/0x10 [ 58.631013][ T71] ? trace_lock_acquire+0x135/0x1c0 [ 58.631139][ T71] ? process_one_work+0xde2/0x1730 [ 58.631265][ T71] ? lock_acquire+0x32/0xc0 [ 58.631391][ T71] ? process_one_work+0xde2/0x1730 [ 58.631519][ T71] process_one_work+0xe2c/0x1730 [ 58.631647][ T71] ? __pfx___lock_release+0x10/0x10 [ 58.631773][ T71] ? __pfx_process_one_work+0x10/0x10 [ 58.631904][ T71] ? assign_work+0x16c/0x240 [ 58.632107][ T71] worker_thread+0x587/0xd30 [ 58.632234][ T71] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 58.632393][ T71] ? __pfx_worker_thread+0x10/0x10 [ 58.632519][ T71] ? __pfx_worker_thread+0x10/0x10 [ 58.632716][ T71] kthread+0x28a/0x350 [ 58.632813][ T71] ? __pfx_kthread+0x10/0x10 [ 58.632940][ T71] ret_from_fork+0x31/0x70 [ 58.633066][ T71] ? __pfx_kthread+0x10/0x10 [ 58.633261][ T71] ret_from_fork_asm+0x1a/0x30 [ 58.633397][ T71]