[ 57.962651][ T10] [ 57.962744][ T10] ===================================================== [ 57.962903][ T10] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected [ 57.963092][ T10] 6.9.0-rc1-virtme #1 Not tainted [ 57.963250][ T10] ----------------------------------------------------- [ 57.963405][ T10] kworker/u16:0/10 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 57.963598][ T10] ffff888001a70968 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70 [ 57.963828][ T10] [ 57.963828][ T10] and this task is already holding: [ 57.964021][ T10] ffff8880361bf198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 57.964257][ T10] which would create a new lock dependency: [ 57.964415][ T10] (&ul->lock#2){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2} [ 57.964609][ T10] [ 57.964609][ T10] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 57.964822][ T10] (&ul->lock#2){+.-.}-{2:2} [ 57.964826][ T10] [ 57.964826][ T10] ... which became SOFTIRQ-irq-safe at: [ 57.965141][ T10] __lock_acquire+0x797/0x1570 [ 57.965289][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.965420][ T10] _raw_spin_lock_bh+0x38/0x50 [ 57.965545][ T10] rt_add_uncached_list+0x55/0x1b0 [ 57.965671][ T10] rt_cache_route+0xfd/0x150 [ 57.965800][ T10] ip_route_input_slow+0x1001/0x2230 [ 57.965926][ T10] ip_route_input_noref+0xe6/0x230 [ 57.966052][ T10] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 57.966219][ T10] ip_rcv+0x199/0x740 [ 57.966317][ T10] __netif_receive_skb_one_core+0x166/0x1b0 [ 57.966481][ T10] process_backlog+0x103/0x670 [ 57.966607][ T10] __napi_poll.constprop.0+0xa2/0x460 [ 57.966732][ T10] net_rx_action+0x49f/0xc00 [ 57.966867][ T10] __do_softirq+0x1f8/0x5df [ 57.966999][ T10] do_softirq+0x4d/0xa0 [ 57.967096][ T10] __local_bh_enable_ip+0xf6/0x120 [ 57.967242][ T10] __dev_queue_xmit+0x6bc/0x16e0 [ 57.967371][ T10] ip6_finish_output2+0x512/0xf60 [ 57.967505][ T10] ip6_finish_output+0x553/0xdf0 [ 57.967629][ T10] ip6_output+0x1f3/0x770 [ 57.967752][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 57.967878][ T10] mld_sendpack+0x62d/0xbc0 [ 57.968004][ T10] mld_ifc_work+0x32/0x1f0 [ 57.968130][ T10] process_one_work+0xe2c/0x1730 [ 57.968265][ T10] worker_thread+0x587/0xd30 [ 57.968390][ T10] kthread+0x28a/0x350 [ 57.968485][ T10] ret_from_fork+0x31/0x70 [ 57.968612][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.968739][ T10] [ 57.968739][ T10] to a SOFTIRQ-irq-unsafe lock: [ 57.968898][ T10] (&p->alloc_lock){+.+.}-{2:2} [ 57.968903][ T10] [ 57.968903][ T10] ... which became SOFTIRQ-irq-unsafe at: [ 57.969206][ T10] ... [ 57.969207][ T10] __lock_acquire+0x797/0x1570 [ 57.969404][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.969529][ T10] _raw_spin_lock+0x30/0x40 [ 57.969665][ T10] set_mems_allowed+0x1d/0x210 [ 57.969792][ T10] kernel_init_freeable+0x72/0x310 [ 57.969919][ T10] kernel_init+0x20/0x200 [ 57.970044][ T10] ret_from_fork+0x31/0x70 [ 57.970173][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.970303][ T10] [ 57.970303][ T10] other info that might help us debug this: [ 57.970303][ T10] [ 57.970545][ T10] Possible interrupt unsafe locking scenario: [ 57.970545][ T10] [ 57.970731][ T10] CPU0 CPU1 [ 57.970854][ T10] ---- ---- [ 57.970977][ T10] lock(&p->alloc_lock); [ 57.971074][ T10] local_irq_disable(); [ 57.971235][ T10] lock(&ul->lock#2); [ 57.971417][ T10] lock(&p->alloc_lock); [ 57.971582][ T10] [ 57.971682][ T10] lock(&ul->lock#2); [ 57.971784][ T10] [ 57.971784][ T10] *** DEADLOCK *** [ 57.971784][ T10] [ 57.971967][ T10] 5 locks held by kworker/u16:0/10: [ 57.972092][ T10] #0: ffff88800190a948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 57.972368][ T10] #1: ffffc900000afda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 57.972598][ T10] #2: ffffffffa959bc50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xb60 [ 57.972824][ T10] #3: ffffffffa95b7208 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0 [ 57.973083][ T10] #4: ffff8880361bf198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 57.973326][ T10] [ 57.973326][ T10] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 57.973576][ T10] -> (&ul->lock#2){+.-.}-{2:2} { [ 57.973710][ T10] HARDIRQ-ON-W at: [ 57.973809][ T10] __lock_acquire+0x797/0x1570 [ 57.973976][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.974140][ T10] _raw_spin_lock_bh+0x38/0x50 [ 57.974312][ T10] rt_add_uncached_list+0x55/0x1b0 [ 57.974477][ T10] rt_cache_route+0xfd/0x150 [ 57.974648][ T10] rt_set_nexthop.constprop.0+0x283/0xe90 [ 57.974847][ T10] __mkroute_output+0x4fa/0x1210 [ 57.975009][ T10] ip_route_output_key_hash+0x15d/0x2b0 [ 57.975203][ T10] ip_route_output_flow+0x23/0x140 [ 57.975385][ T10] udp_tunnel_dst_lookup+0x236/0x3b0 [ 57.975543][ T10] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 57.975751][ T10] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 57.975925][ T10] dev_hard_start_xmit+0x10e/0x360 [ 57.976087][ T10] __dev_queue_xmit+0xe79/0x16e0 [ 57.976248][ T10] ip6_finish_output2+0x512/0xf60 [ 57.976424][ T10] ip6_finish_output+0x553/0xdf0 [ 57.976591][ T10] ip6_output+0x1f3/0x770 [ 57.976752][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 57.976917][ T10] mld_sendpack+0x62d/0xbc0 [ 57.977078][ T10] mld_ifc_work+0x32/0x1f0 [ 57.977244][ T10] process_one_work+0xe2c/0x1730 [ 57.977409][ T10] worker_thread+0x587/0xd30 [ 57.977569][ T10] kthread+0x28a/0x350 [ 57.977698][ T10] ret_from_fork+0x31/0x70 [ 57.977856][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.978020][ T10] IN-SOFTIRQ-W at: [ 57.978125][ T10] __lock_acquire+0x797/0x1570 [ 57.978289][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.978534][ T10] _raw_spin_lock_bh+0x38/0x50 [ 57.978784][ T10] rt_add_uncached_list+0x55/0x1b0 [ 57.978944][ T10] rt_cache_route+0xfd/0x150 [ 57.979102][ T10] ip_route_input_slow+0x1001/0x2230 [ 57.979366][ T10] ip_route_input_noref+0xe6/0x230 [ 57.979606][ T10] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 57.979800][ T10] ip_rcv+0x199/0x740 [ 57.979929][ T10] __netif_receive_skb_one_core+0x166/0x1b0 [ 57.980201][ T10] process_backlog+0x103/0x670 [ 57.980369][ T10] __napi_poll.constprop.0+0xa2/0x460 [ 57.980532][ T10] net_rx_action+0x49f/0xc00 [ 57.980701][ T10] __do_softirq+0x1f8/0x5df [ 57.980930][ T10] do_softirq+0x4d/0xa0 [ 57.981090][ T10] __local_bh_enable_ip+0xf6/0x120 [ 57.981250][ T10] __dev_queue_xmit+0x6bc/0x16e0 [ 57.981499][ T10] ip6_finish_output2+0x512/0xf60 [ 57.981662][ T10] ip6_finish_output+0x553/0xdf0 [ 57.981826][ T10] ip6_output+0x1f3/0x770 [ 57.982072][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 57.982232][ T10] mld_sendpack+0x62d/0xbc0 [ 57.982402][ T10] mld_ifc_work+0x32/0x1f0 [ 57.982641][ T10] process_one_work+0xe2c/0x1730 [ 57.982802][ T10] worker_thread+0x587/0xd30 [ 57.982958][ T10] kthread+0x28a/0x350 [ 57.983088][ T10] ret_from_fork+0x31/0x70 [ 57.983332][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.983509][ T10] INITIAL USE at: [ 57.983607][ T10] __lock_acquire+0x797/0x1570 [ 57.983766][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.984003][ T10] _raw_spin_lock_bh+0x38/0x50 [ 57.984166][ T10] rt_add_uncached_list+0x55/0x1b0 [ 57.984325][ T10] rt_cache_route+0xfd/0x150 [ 57.984660][ T10] rt_set_nexthop.constprop.0+0x283/0xe90 [ 57.984860][ T10] __mkroute_output+0x4fa/0x1210 [ 57.985020][ T10] ip_route_output_key_hash+0x15d/0x2b0 [ 57.985186][ T10] ip_route_output_flow+0x23/0x140 [ 57.985347][ T10] udp_tunnel_dst_lookup+0x236/0x3b0 [ 57.985514][ T10] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 57.985687][ T10] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 57.985933][ T10] dev_hard_start_xmit+0x10e/0x360 [ 57.986099][ T10] __dev_queue_xmit+0xe79/0x16e0 [ 57.986259][ T10] ip6_finish_output2+0x512/0xf60 [ 57.986505][ T10] ip6_finish_output+0x553/0xdf0 [ 57.986663][ T10] ip6_output+0x1f3/0x770 [ 57.986823][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 57.986983][ T10] mld_sendpack+0x62d/0xbc0 [ 57.987217][ T10] mld_ifc_work+0x32/0x1f0 [ 57.987375][ T10] process_one_work+0xe2c/0x1730 [ 57.987541][ T10] worker_thread+0x587/0xd30 [ 57.987774][ T10] kthread+0x28a/0x350 [ 57.987901][ T10] ret_from_fork+0x31/0x70 [ 57.988060][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.988218][ T10] } [ 57.988354][ T10] ... key at: [] __key.7+0x0/0x40 [ 57.988523][ T10] [ 57.988523][ T10] the dependencies between the lock to be acquired [ 57.988525][ T10] and SOFTIRQ-irq-unsafe lock: [ 57.988932][ T10] -> (&p->alloc_lock){+.+.}-{2:2} { [ 57.989065][ T10] HARDIRQ-ON-W at: [ 57.989167][ T10] __lock_acquire+0x797/0x1570 [ 57.989331][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.989586][ T10] _raw_spin_lock+0x30/0x40 [ 57.989748][ T10] set_mems_allowed+0x1d/0x210 [ 57.989906][ T10] kernel_init_freeable+0x72/0x310 [ 57.990066][ T10] kernel_init+0x20/0x200 [ 57.990229][ T10] ret_from_fork+0x31/0x70 [ 57.990394][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.990560][ T10] SOFTIRQ-ON-W at: [ 57.990675][ T10] __lock_acquire+0x797/0x1570 [ 57.990834][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.990993][ T10] _raw_spin_lock+0x30/0x40 [ 57.991154][ T10] set_mems_allowed+0x1d/0x210 [ 57.991312][ T10] kernel_init_freeable+0x72/0x310 [ 57.991480][ T10] kernel_init+0x20/0x200 [ 57.991654][ T10] ret_from_fork+0x31/0x70 [ 57.991902][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.992063][ T10] INITIAL USE at: [ 57.992160][ T10] __lock_acquire+0x797/0x1570 [ 57.992325][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.992662][ T10] _raw_spin_lock+0x30/0x40 [ 57.992827][ T10] set_mems_allowed+0x1d/0x210 [ 57.992988][ T10] kernel_init_freeable+0x72/0x310 [ 57.993158][ T10] kernel_init+0x20/0x200 [ 57.993321][ T10] ret_from_fork+0x31/0x70 [ 57.993486][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.993655][ T10] } [ 57.993723][ T10] ... key at: [] __key.388+0x0/0x40 [ 57.993992][ T10] ... acquired at: [ 57.994087][ T10] check_prev_add+0x194/0xce0 [ 57.994217][ T10] validate_chain+0x71d/0x9b0 [ 57.994345][ T10] __lock_acquire+0xaf0/0x1570 [ 57.994474][ T10] lock_acquire.part.0+0xeb/0x330 [ 57.994694][ T10] _raw_spin_lock+0x30/0x40 [ 57.994825][ T10] __get_task_comm+0x27/0x70 [ 57.994969][ T10] ref_tracker_alloc+0x2ee/0x490 [ 57.995103][ T10] rt_flush_dev+0x371/0x670 [ 57.995233][ T10] fib_netdev_event+0x248/0x5a0 [ 57.995440][ T10] notifier_call_chain+0xcd/0x150 [ 57.995589][ T10] unregister_netdevice_many_notify+0x548/0x1190 [ 57.995749][ T10] default_device_exit_batch+0x234/0x2d0 [ 57.995907][ T10] cleanup_net+0x4f3/0xb60 [ 57.996116][ T10] process_one_work+0xe2c/0x1730 [ 57.996247][ T10] worker_thread+0x587/0xd30 [ 57.996378][ T10] kthread+0x28a/0x350 [ 57.996559][ T10] ret_from_fork+0x31/0x70 [ 57.996686][ T10] ret_from_fork_asm+0x1a/0x30 [ 57.996815][ T10] [ 57.996880][ T10] [ 57.996880][ T10] stack backtrace: [ 57.997111][ T10] CPU: 0 PID: 10 Comm: kworker/u16:0 Not tainted 6.9.0-rc1-virtme #1 [ 57.997307][ T10] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 57.997687][ T10] Workqueue: netns cleanup_net [ 57.997819][ T10] Call Trace: [ 57.997919][ T10] [ 57.997987][ T10] dump_stack_lvl+0x82/0xd0 [ 57.998118][ T10] print_bad_irq_dependency+0x957/0xd40 [ 57.998253][ T10] ? __pfx_print_bad_irq_dependency+0x10/0x10 [ 57.998415][ T10] ? __pfx_usage_skip+0x10/0x10 [ 57.998549][ T10] ? __bfs+0x246/0x650 [ 57.998650][ T10] ? __pfx_usage_match+0x10/0x10 [ 57.998851][ T10] check_irq_usage+0x596/0x850 [ 57.998981][ T10] ? arch_stack_walk+0x8c/0xf0 [ 57.999110][ T10] ? __pfx_check_irq_usage+0x10/0x10 [ 57.999238][ T10] ? __bfs+0x246/0x650 [ 57.999337][ T10] ? __pfx_hlock_conflict+0x10/0x10 [ 57.999541][ T10] ? check_path.constprop.0+0x24/0x50 [ 57.999686][ T10] ? check_noncircular+0x14c/0x3e0 [ 57.999815][ T10] ? __pfx_check_noncircular+0x10/0x10 [ 57.999943][ T10] ? find_held_lock+0x2c/0x110 [ 58.000149][ T10] ? check_prev_add+0x194/0xce0 [ 58.000276][ T10] check_prev_add+0x194/0xce0 [ 58.000406][ T10] ? validate_chain+0x130/0x9b0 [ 58.000535][ T10] validate_chain+0x71d/0x9b0 [ 58.000815][ T10] ? mark_lock+0x38/0x3e0 [ 58.000917][ T10] ? __pfx_validate_chain+0x10/0x10 [ 58.001044][ T10] ? hlock_class+0x4e/0x130 [ 58.001168][ T10] ? mark_lock+0x38/0x3e0 [ 58.001266][ T10] __lock_acquire+0xaf0/0x1570 [ 58.001470][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.001609][ T10] ? __get_task_comm+0x27/0x70 [ 58.001739][ T10] ? mark_lock+0x38/0x3e0 [ 58.001837][ T10] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 58.001963][ T10] ? trace_lock_acquire+0x135/0x1c0 [ 58.002244][ T10] ? __get_task_comm+0x27/0x70 [ 58.002375][ T10] ? lock_acquire+0x32/0xc0 [ 58.002502][ T10] ? __get_task_comm+0x27/0x70 [ 58.002629][ T10] _raw_spin_lock+0x30/0x40 [ 58.002759][ T10] ? __get_task_comm+0x27/0x70 [ 58.002964][ T10] __get_task_comm+0x27/0x70 [ 58.003089][ T10] ref_tracker_alloc+0x2ee/0x490 [ 58.003217][ T10] ? lock_acquire.part.0+0xeb/0x330 [ 58.003345][ T10] ? __pfx_ref_tracker_alloc+0x10/0x10 [ 58.003474][ T10] ? rt_flush_dev+0x371/0x670 [ 58.003709][ T10] ? fib_netdev_event+0x248/0x5a0 [ 58.003841][ T10] ? notifier_call_chain+0xcd/0x150 [ 58.003975][ T10] ? unregister_netdevice_many_notify+0x548/0x1190 [ 58.004127][ T10] ? default_device_exit_batch+0x234/0x2d0 [ 58.004285][ T10] ? cleanup_net+0x4f3/0xb60 [ 58.004413][ T10] ? process_one_work+0xe2c/0x1730 [ 58.004619][ T10] ? worker_thread+0x587/0xd30 [ 58.004747][ T10] ? kthread+0x28a/0x350 [ 58.004849][ T10] ? ret_from_fork+0x31/0x70 [ 58.004982][ T10] ? ret_from_fork_asm+0x1a/0x30 [ 58.005114][ T10] ? do_raw_spin_lock+0x131/0x270 [ 58.005313][ T10] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 58.005439][ T10] ? lock_acquire+0x32/0xc0 [ 58.005564][ T10] ? rt_flush_dev+0x13b/0x670 [ 58.005698][ T10] rt_flush_dev+0x371/0x670 [ 58.005900][ T10] ? __timer_delete_sync+0xd3/0x110 [ 58.006028][ T10] ? __timer_delete_sync+0xa7/0x110 [ 58.006154][ T10] fib_netdev_event+0x248/0x5a0 [ 58.006279][ T10] notifier_call_chain+0xcd/0x150 [ 58.006487][ T10] unregister_netdevice_many_notify+0x548/0x1190 [ 58.006651][ T10] ? mutex_is_locked+0x17/0x50 [ 58.006779][ T10] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 58.006939][ T10] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 58.007168][ T10] default_device_exit_batch+0x234/0x2d0 [ 58.007297][ T10] ? __pfx_default_device_exit_batch+0x10/0x10 [ 58.007458][ T10] ? ops_exit_list+0xb4/0x170 [ 58.007587][ T10] cleanup_net+0x4f3/0xb60 [ 58.007822][ T10] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 58.007950][ T10] ? __pfx_cleanup_net+0x10/0x10 [ 58.008075][ T10] ? trace_lock_acquire+0x135/0x1c0 [ 58.008206][ T10] ? process_one_work+0xde2/0x1730 [ 58.008337][ T10] ? lock_acquire+0x32/0xc0 [ 58.008468][ T10] ? process_one_work+0xde2/0x1730 [ 58.008596][ T10] process_one_work+0xe2c/0x1730 [ 58.008731][ T10] ? __pfx___lock_release+0x10/0x10 [ 58.008932][ T10] ? __pfx_process_one_work+0x10/0x10 [ 58.009060][ T10] ? assign_work+0x16c/0x240 [ 58.009188][ T10] worker_thread+0x587/0xd30 [ 58.009318][ T10] ? __pfx_worker_thread+0x10/0x10 [ 58.009606][ T10] kthread+0x28a/0x350 [ 58.009711][ T10] ? __pfx_kthread+0x10/0x10 [ 58.009851][ T10] ret_from_fork+0x31/0x70 [ 58.009978][ T10] ? __pfx_kthread+0x10/0x10 [ 58.010106][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.010243][ T10]