[ 40.168345][ T404] ip (404) used greatest stack depth: 24016 bytes left [ 58.059882][ T10] [ 58.059985][ T10] ===================================================== [ 58.060163][ T10] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected [ 58.060380][ T10] 6.9.0-rc1-virtme #1 Not tainted [ 58.060525][ T10] ----------------------------------------------------- [ 58.060702][ T10] kworker/u16:0/10 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 58.060914][ T10] ffff888001a70968 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70 [ 58.061162][ T10] [ 58.061162][ T10] and this task is already holding: [ 58.061371][ T10] ffff88803613f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 58.061627][ T10] which would create a new lock dependency: [ 58.061805][ T10] (&ul->lock#2){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2} [ 58.062023][ T10] [ 58.062023][ T10] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 58.062280][ T10] (&ul->lock#2){+.-.}-{2:2} [ 58.062285][ T10] [ 58.062285][ T10] ... which became SOFTIRQ-irq-safe at: [ 58.062638][ T10] __lock_acquire+0x797/0x1570 [ 58.062788][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.062935][ T10] _raw_spin_lock_bh+0x38/0x50 [ 58.063079][ T10] rt_add_uncached_list+0x55/0x1b0 [ 58.063225][ T10] rt_cache_route+0xfd/0x150 [ 58.063366][ T10] ip_route_input_slow+0x1001/0x2230 [ 58.063503][ T10] ip_route_input_noref+0xe6/0x230 [ 58.063646][ T10] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 58.063823][ T10] ip_rcv+0x199/0x740 [ 58.063931][ T10] __netif_receive_skb_one_core+0x166/0x1b0 [ 58.064101][ T10] process_backlog+0x103/0x670 [ 58.064248][ T10] __napi_poll.constprop.0+0xa2/0x460 [ 58.064381][ T10] net_rx_action+0x49f/0xc00 [ 58.064524][ T10] __do_softirq+0x1f8/0x5df [ 58.064656][ T10] do_softirq+0x4d/0xa0 [ 58.064767][ T10] __local_bh_enable_ip+0xf6/0x120 [ 58.064904][ T10] __dev_queue_xmit+0x6bc/0x16e0 [ 58.065047][ T10] ip6_finish_output2+0x59b/0xf60 [ 58.065201][ T10] ip6_finish_output+0x553/0xdf0 [ 58.065343][ T10] ip6_output+0x1f3/0x770 [ 58.065492][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 58.065634][ T10] mld_sendpack+0x62d/0xbc0 [ 58.065776][ T10] mld_ifc_work+0x32/0x1f0 [ 58.065922][ T10] process_one_work+0xe2c/0x1730 [ 58.066067][ T10] worker_thread+0x587/0xd30 [ 58.066209][ T10] kthread+0x28a/0x350 [ 58.066320][ T10] ret_from_fork+0x31/0x70 [ 58.066466][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.066611][ T10] [ 58.066611][ T10] to a SOFTIRQ-irq-unsafe lock: [ 58.066783][ T10] (&p->alloc_lock){+.+.}-{2:2} [ 58.066787][ T10] [ 58.066787][ T10] ... which became SOFTIRQ-irq-unsafe at: [ 58.067122][ T10] ... [ 58.067124][ T10] __lock_acquire+0x797/0x1570 [ 58.067339][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.067475][ T10] _raw_spin_lock+0x30/0x40 [ 58.067612][ T10] set_mems_allowed+0x1d/0x210 [ 58.067754][ T10] kernel_init_freeable+0x72/0x310 [ 58.067899][ T10] kernel_init+0x20/0x200 [ 58.068038][ T10] ret_from_fork+0x31/0x70 [ 58.068180][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.068328][ T10] [ 58.068328][ T10] other info that might help us debug this: [ 58.068328][ T10] [ 58.068600][ T10] Possible interrupt unsafe locking scenario: [ 58.068600][ T10] [ 58.068815][ T10] CPU0 CPU1 [ 58.068958][ T10] ---- ---- [ 58.069097][ T10] lock(&p->alloc_lock); [ 58.069207][ T10] local_irq_disable(); [ 58.069394][ T10] lock(&ul->lock#2); [ 58.069575][ T10] lock(&p->alloc_lock); [ 58.069751][ T10] [ 58.069857][ T10] lock(&ul->lock#2); [ 58.069967][ T10] [ 58.069967][ T10] *** DEADLOCK *** [ 58.069967][ T10] [ 58.070172][ T10] 5 locks held by kworker/u16:0/10: [ 58.070316][ T10] #0: ffff88800190a948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 58.070613][ T10] #1: ffffc900000afda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 58.070862][ T10] #2: ffffffff95b9bc50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xb60 [ 58.071115][ T10] #3: ffffffff95bb7208 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0 [ 58.071398][ T10] #4: ffff88803613f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 58.071650][ T10] [ 58.071650][ T10] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 58.071923][ T10] -> (&ul->lock#2){+.-.}-{2:2} { [ 58.072073][ T10] HARDIRQ-ON-W at: [ 58.072182][ T10] __lock_acquire+0x797/0x1570 [ 58.072365][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.072557][ T10] _raw_spin_lock_bh+0x38/0x50 [ 58.072734][ T10] rt_add_uncached_list+0x55/0x1b0 [ 58.072911][ T10] rt_cache_route+0xfd/0x150 [ 58.073088][ T10] rt_set_nexthop.constprop.0+0x283/0xe90 [ 58.073303][ T10] __mkroute_output+0x4fa/0x1210 [ 58.073481][ T10] ip_route_output_key_hash+0x15d/0x2b0 [ 58.073692][ T10] ip_route_output_flow+0x23/0x140 [ 58.073872][ T10] udp_tunnel_dst_lookup+0x236/0x3b0 [ 58.074050][ T10] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 58.074279][ T10] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 58.074477][ T10] dev_hard_start_xmit+0x10e/0x360 [ 58.074660][ T10] __dev_queue_xmit+0xe79/0x16e0 [ 58.074845][ T10] ip6_finish_output2+0x59b/0xf60 [ 58.075026][ T10] ip6_finish_output+0x553/0xdf0 [ 58.075208][ T10] ip6_output+0x1f3/0x770 [ 58.075388][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 58.075566][ T10] mld_sendpack+0x62d/0xbc0 [ 58.075748][ T10] mld_ifc_work+0x32/0x1f0 [ 58.075925][ T10] process_one_work+0xe2c/0x1730 [ 58.076110][ T10] worker_thread+0x587/0xd30 [ 58.076285][ T10] kthread+0x28a/0x350 [ 58.076430][ T10] ret_from_fork+0x31/0x70 [ 58.076611][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.076789][ T10] IN-SOFTIRQ-W at: [ 58.076897][ T10] __lock_acquire+0x797/0x1570 [ 58.077073][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.077267][ T10] _raw_spin_lock_bh+0x38/0x50 [ 58.077448][ T10] rt_add_uncached_list+0x55/0x1b0 [ 58.077631][ T10] rt_cache_route+0xfd/0x150 [ 58.077812][ T10] ip_route_input_slow+0x1001/0x2230 [ 58.078073][ T10] ip_route_input_noref+0xe6/0x230 [ 58.078255][ T10] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 58.078472][ T10] ip_rcv+0x199/0x740 [ 58.078690][ T10] __netif_receive_skb_one_core+0x166/0x1b0 [ 58.078908][ T10] process_backlog+0x103/0x670 [ 58.079169][ T10] __napi_poll.constprop.0+0xa2/0x460 [ 58.079350][ T10] net_rx_action+0x49f/0xc00 [ 58.079530][ T10] __do_softirq+0x1f8/0x5df [ 58.079701][ T10] do_softirq+0x4d/0xa0 [ 58.079872][ T10] __local_bh_enable_ip+0xf6/0x120 [ 58.080206][ T10] __dev_queue_xmit+0x6bc/0x16e0 [ 58.080390][ T10] ip6_finish_output2+0x59b/0xf60 [ 58.080564][ T10] ip6_finish_output+0x553/0xdf0 [ 58.080823][ T10] ip6_output+0x1f3/0x770 [ 58.081005][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 58.081177][ T10] mld_sendpack+0x62d/0xbc0 [ 58.081367][ T10] mld_ifc_work+0x32/0x1f0 [ 58.081656][ T10] process_one_work+0xe2c/0x1730 [ 58.081836][ T10] worker_thread+0x587/0xd30 [ 58.082019][ T10] kthread+0x28a/0x350 [ 58.082164][ T10] ret_from_fork+0x31/0x70 [ 58.082344][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.082610][ T10] INITIAL USE at: [ 58.082720][ T10] __lock_acquire+0x797/0x1570 [ 58.082900][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.083081][ T10] _raw_spin_lock_bh+0x38/0x50 [ 58.083263][ T10] rt_add_uncached_list+0x55/0x1b0 [ 58.083444][ T10] rt_cache_route+0xfd/0x150 [ 58.083617][ T10] rt_set_nexthop.constprop.0+0x283/0xe90 [ 58.083843][ T10] __mkroute_output+0x4fa/0x1210 [ 58.084021][ T10] ip_route_output_key_hash+0x15d/0x2b0 [ 58.084274][ T10] ip_route_output_flow+0x23/0x140 [ 58.084458][ T10] udp_tunnel_dst_lookup+0x236/0x3b0 [ 58.084708][ T10] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 58.084901][ T10] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 58.085103][ T10] dev_hard_start_xmit+0x10e/0x360 [ 58.085284][ T10] __dev_queue_xmit+0xe79/0x16e0 [ 58.085464][ T10] ip6_finish_output2+0x59b/0xf60 [ 58.085755][ T10] ip6_finish_output+0x553/0xdf0 [ 58.085937][ T10] ip6_output+0x1f3/0x770 [ 58.086116][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 58.086298][ T10] mld_sendpack+0x62d/0xbc0 [ 58.086482][ T10] mld_ifc_work+0x32/0x1f0 [ 58.086658][ T10] process_one_work+0xe2c/0x1730 [ 58.086841][ T10] worker_thread+0x587/0xd30 [ 58.087099][ T10] kthread+0x28a/0x350 [ 58.087245][ T10] ret_from_fork+0x31/0x70 [ 58.087429][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.087771][ T10] } [ 58.087845][ T10] ... key at: [] __key.7+0x0/0x40 [ 58.088038][ T10] [ 58.088038][ T10] the dependencies between the lock to be acquired [ 58.088039][ T10] and SOFTIRQ-irq-unsafe lock: [ 58.088408][ T10] -> (&p->alloc_lock){+.+.}-{2:2} { [ 58.088637][ T10] HARDIRQ-ON-W at: [ 58.088749][ T10] __lock_acquire+0x797/0x1570 [ 58.088934][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.089106][ T10] _raw_spin_lock+0x30/0x40 [ 58.089281][ T10] set_mems_allowed+0x1d/0x210 [ 58.089463][ T10] kernel_init_freeable+0x72/0x310 [ 58.089725][ T10] kernel_init+0x20/0x200 [ 58.089902][ T10] ret_from_fork+0x31/0x70 [ 58.090083][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.090265][ T10] SOFTIRQ-ON-W at: [ 58.090447][ T10] __lock_acquire+0x797/0x1570 [ 58.090622][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.090802][ T10] _raw_spin_lock+0x30/0x40 [ 58.090988][ T10] set_mems_allowed+0x1d/0x210 [ 58.091166][ T10] kernel_init_freeable+0x72/0x310 [ 58.091422][ T10] kernel_init+0x20/0x200 [ 58.091599][ T10] ret_from_fork+0x31/0x70 [ 58.091775][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.091941][ T10] INITIAL USE at: [ 58.092052][ T10] __lock_acquire+0x797/0x1570 [ 58.092233][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.092490][ T10] _raw_spin_lock+0x30/0x40 [ 58.092678][ T10] set_mems_allowed+0x1d/0x210 [ 58.092856][ T10] kernel_init_freeable+0x72/0x310 [ 58.093109][ T10] kernel_init+0x20/0x200 [ 58.093288][ T10] ret_from_fork+0x31/0x70 [ 58.093469][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.093723][ T10] } [ 58.093795][ T10] ... key at: [] __key.388+0x0/0x40 [ 58.094007][ T10] ... acquired at: [ 58.094113][ T10] check_prev_add+0x194/0xce0 [ 58.094256][ T10] validate_chain+0x71d/0x9b0 [ 58.094469][ T10] __lock_acquire+0xaf0/0x1570 [ 58.094612][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.094756][ T10] _raw_spin_lock+0x30/0x40 [ 58.094900][ T10] __get_task_comm+0x27/0x70 [ 58.095112][ T10] ref_tracker_alloc+0x2ee/0x490 [ 58.095255][ T10] rt_flush_dev+0x371/0x670 [ 58.095396][ T10] fib_netdev_event+0x248/0x5a0 [ 58.095540][ T10] notifier_call_chain+0xcd/0x150 [ 58.095826][ T10] unregister_netdevice_many_notify+0x548/0x1190 [ 58.095999][ T10] default_device_exit_batch+0x234/0x2d0 [ 58.096179][ T10] cleanup_net+0x4f3/0xb60 [ 58.096321][ T10] process_one_work+0xe2c/0x1730 [ 58.096549][ T10] worker_thread+0x587/0xd30 [ 58.096692][ T10] kthread+0x28a/0x350 [ 58.096799][ T10] ret_from_fork+0x31/0x70 [ 58.096939][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.097081][ T10] [ 58.097153][ T10] [ 58.097153][ T10] stack backtrace: [ 58.097327][ T10] CPU: 0 PID: 10 Comm: kworker/u16:0 Not tainted 6.9.0-rc1-virtme #1 [ 58.097540][ T10] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 58.097853][ T10] Workqueue: netns cleanup_net [ 58.098000][ T10] Call Trace: [ 58.098111][ T10] [ 58.098185][ T10] dump_stack_lvl+0x82/0xd0 [ 58.098331][ T10] print_bad_irq_dependency+0x957/0xd40 [ 58.098474][ T10] ? __pfx_print_bad_irq_dependency+0x10/0x10 [ 58.098657][ T10] ? __pfx_usage_skip+0x10/0x10 [ 58.098796][ T10] ? __bfs+0x246/0x650 [ 58.098906][ T10] ? __pfx_usage_match+0x10/0x10 [ 58.099048][ T10] check_irq_usage+0x596/0x850 [ 58.099190][ T10] ? arch_stack_walk+0x8c/0xf0 [ 58.099333][ T10] ? __pfx_check_irq_usage+0x10/0x10 [ 58.099477][ T10] ? __bfs+0x246/0x650 [ 58.099584][ T10] ? __pfx_hlock_conflict+0x10/0x10 [ 58.099729][ T10] ? check_path.constprop.0+0x24/0x50 [ 58.099870][ T10] ? check_noncircular+0x14c/0x3e0 [ 58.100014][ T10] ? __pfx_check_noncircular+0x10/0x10 [ 58.100156][ T10] ? find_held_lock+0x2c/0x110 [ 58.100300][ T10] ? check_prev_add+0x194/0xce0 [ 58.100441][ T10] check_prev_add+0x194/0xce0 [ 58.100584][ T10] ? validate_chain+0x130/0x9b0 [ 58.100729][ T10] validate_chain+0x71d/0x9b0 [ 58.100874][ T10] ? mark_lock+0x38/0x3e0 [ 58.100980][ T10] ? __pfx_validate_chain+0x10/0x10 [ 58.101123][ T10] ? hlock_class+0x4e/0x130 [ 58.101267][ T10] ? mark_lock+0x38/0x3e0 [ 58.101376][ T10] __lock_acquire+0xaf0/0x1570 [ 58.101520][ T10] lock_acquire.part.0+0xeb/0x330 [ 58.101663][ T10] ? __get_task_comm+0x27/0x70 [ 58.101807][ T10] ? mark_lock+0x38/0x3e0 [ 58.101914][ T10] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 58.102057][ T10] ? trace_lock_acquire+0x135/0x1c0 [ 58.102201][ T10] ? __get_task_comm+0x27/0x70 [ 58.102345][ T10] ? lock_acquire+0x32/0xc0 [ 58.102495][ T10] ? __get_task_comm+0x27/0x70 [ 58.102710][ T10] _raw_spin_lock+0x30/0x40 [ 58.102853][ T10] ? __get_task_comm+0x27/0x70 [ 58.102995][ T10] __get_task_comm+0x27/0x70 [ 58.103136][ T10] ref_tracker_alloc+0x2ee/0x490 [ 58.103349][ T10] ? lock_acquire.part.0+0xeb/0x330 [ 58.103494][ T10] ? __pfx_ref_tracker_alloc+0x10/0x10 [ 58.103635][ T10] ? rt_flush_dev+0x371/0x670 [ 58.103777][ T10] ? fib_netdev_event+0x248/0x5a0 [ 58.103916][ T10] ? notifier_call_chain+0xcd/0x150 [ 58.104128][ T10] ? unregister_netdevice_many_notify+0x548/0x1190 [ 58.104301][ T10] ? default_device_exit_batch+0x234/0x2d0 [ 58.104475][ T10] ? cleanup_net+0x4f3/0xb60 [ 58.104685][ T10] ? process_one_work+0xe2c/0x1730 [ 58.104825][ T10] ? worker_thread+0x587/0xd30 [ 58.104968][ T10] ? kthread+0x28a/0x350 [ 58.105074][ T10] ? ret_from_fork+0x31/0x70 [ 58.105212][ T10] ? ret_from_fork_asm+0x1a/0x30 [ 58.105423][ T10] ? do_raw_spin_lock+0x131/0x270 [ 58.105564][ T10] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 58.105704][ T10] ? lock_acquire+0x32/0xc0 [ 58.105844][ T10] ? rt_flush_dev+0x13b/0x670 [ 58.106059][ T10] rt_flush_dev+0x371/0x670 [ 58.106200][ T10] ? __timer_delete_sync+0xd3/0x110 [ 58.106341][ T10] ? __timer_delete_sync+0xa7/0x110 [ 58.106482][ T10] fib_netdev_event+0x248/0x5a0 [ 58.106694][ T10] notifier_call_chain+0xcd/0x150 [ 58.106836][ T10] unregister_netdevice_many_notify+0x548/0x1190 [ 58.107009][ T10] ? mutex_is_locked+0x17/0x50 [ 58.107150][ T10] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 58.107399][ T10] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 58.107577][ T10] default_device_exit_batch+0x234/0x2d0 [ 58.107717][ T10] ? __pfx_default_device_exit_batch+0x10/0x10 [ 58.107891][ T10] ? ops_exit_list+0xb4/0x170 [ 58.108178][ T10] cleanup_net+0x4f3/0xb60 [ 58.108319][ T10] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 58.108460][ T10] ? __pfx_cleanup_net+0x10/0x10 [ 58.108602][ T10] ? trace_lock_acquire+0x135/0x1c0 [ 58.108819][ T10] ? process_one_work+0xde2/0x1730 [ 58.108961][ T10] ? lock_acquire+0x32/0xc0 [ 58.109102][ T10] ? process_one_work+0xde2/0x1730 [ 58.109245][ T10] process_one_work+0xe2c/0x1730 [ 58.109459][ T10] ? __pfx___lock_release+0x10/0x10 [ 58.109603][ T10] ? __pfx_process_one_work+0x10/0x10 [ 58.109748][ T10] ? assign_work+0x16c/0x240 [ 58.109894][ T10] worker_thread+0x587/0xd30 [ 58.110040][ T10] ? __pfx_worker_thread+0x10/0x10 [ 58.110252][ T10] kthread+0x28a/0x350 [ 58.110359][ T10] ? __pfx_kthread+0x10/0x10 [ 58.110501][ T10] ret_from_fork+0x31/0x70 [ 58.110643][ T10] ? __pfx_kthread+0x10/0x10 [ 58.110855][ T10] ret_from_fork_asm+0x1a/0x30 [ 58.111002][ T10] [ 124.112693][ T1092] udpgso_bench_tx (1092) used greatest stack depth: 23808 bytes left