[ 68.973023][ T70] [ 68.973146][ T70] ===================================================== [ 68.973389][ T70] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected [ 68.973593][ T70] 6.9.0-rc1-virtme #1 Not tainted [ 68.973733][ T70] ----------------------------------------------------- [ 68.973902][ T70] kworker/u16:1/70 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 68.974183][ T70] ffff888005330968 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70 [ 68.974434][ T70] [ 68.974434][ T70] and this task is already holding: [ 68.974627][ T70] ffff8880360bf198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 68.974867][ T70] which would create a new lock dependency: [ 68.975031][ T70] (&ul->lock#2){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2} [ 68.975240][ T70] [ 68.975240][ T70] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 68.975466][ T70] (&ul->lock#2){+.-.}-{2:2} [ 68.975471][ T70] [ 68.975471][ T70] ... which became SOFTIRQ-irq-safe at: [ 68.975868][ T70] __lock_acquire+0x797/0x1570 [ 68.976016][ T70] lock_acquire.part.0+0xeb/0x330 [ 68.976149][ T70] _raw_spin_lock_bh+0x38/0x50 [ 68.976298][ T70] rt_add_uncached_list+0x55/0x1b0 [ 68.976499][ T70] rt_cache_route+0xfd/0x150 [ 68.976634][ T70] ip_route_input_slow+0x1001/0x2230 [ 68.976767][ T70] ip_route_input_noref+0xe6/0x230 [ 68.976908][ T70] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 68.977144][ T70] ip_rcv+0x199/0x740 [ 68.977246][ T70] __netif_receive_skb_one_core+0x166/0x1b0 [ 68.977414][ T70] process_backlog+0x103/0x670 [ 68.977549][ T70] __napi_poll.constprop.0+0xa2/0x460 [ 68.977680][ T70] net_rx_action+0x49f/0xc00 [ 68.977959][ T70] __do_softirq+0x1f8/0x5df [ 68.978089][ T70] do_softirq+0x4d/0xa0 [ 68.978197][ T70] __local_bh_enable_ip+0xf6/0x120 [ 68.978330][ T70] __dev_queue_xmit+0x6bc/0x16e0 [ 68.978530][ T70] ip6_finish_output2+0x512/0xf60 [ 68.978663][ T70] ip6_finish_output+0x553/0xdf0 [ 68.978791][ T70] ip6_output+0x1f3/0x770 [ 68.978927][ T70] NF_HOOK.constprop.0+0xe2/0x680 [ 68.979138][ T70] mld_sendpack+0x62d/0xbc0 [ 68.979271][ T70] mld_ifc_work+0x32/0x1f0 [ 68.979401][ T70] process_one_work+0xe2c/0x1730 [ 68.979538][ T70] worker_thread+0x587/0xd30 [ 68.979669][ T70] kthread+0x28a/0x350 [ 68.979780][ T70] ret_from_fork+0x31/0x70 [ 68.979912][ T70] ret_from_fork_asm+0x1a/0x30 [ 68.980044][ T70] [ 68.980044][ T70] to a SOFTIRQ-irq-unsafe lock: [ 68.980204][ T70] (&p->alloc_lock){+.+.}-{2:2} [ 68.980209][ T70] [ 68.980209][ T70] ... which became SOFTIRQ-irq-unsafe at: [ 68.980600][ T70] ... [ 68.980602][ T70] __lock_acquire+0x797/0x1570 [ 68.980801][ T70] lock_acquire.part.0+0xeb/0x330 [ 68.980930][ T70] _raw_spin_lock+0x30/0x40 [ 68.981062][ T70] set_mems_allowed+0x1d/0x210 [ 68.981195][ T70] kernel_init_freeable+0x72/0x310 [ 68.981333][ T70] kernel_init+0x20/0x200 [ 68.981470][ T70] ret_from_fork+0x31/0x70 [ 68.981601][ T70] ret_from_fork_asm+0x1a/0x30 [ 68.981738][ T70] [ 68.981738][ T70] other info that might help us debug this: [ 68.981738][ T70] [ 68.982012][ T70] Possible interrupt unsafe locking scenario: [ 68.982012][ T70] [ 68.982205][ T70] CPU0 CPU1 [ 68.982337][ T70] ---- ---- [ 68.982470][ T70] lock(&p->alloc_lock); [ 68.982577][ T70] local_irq_disable(); [ 68.982752][ T70] lock(&ul->lock#2); [ 68.982921][ T70] lock(&p->alloc_lock); [ 68.983092][ T70] [ 68.983193][ T70] lock(&ul->lock#2); [ 68.983299][ T70] [ 68.983299][ T70] *** DEADLOCK *** [ 68.983299][ T70] [ 68.983499][ T70] 5 locks held by kworker/u16:1/70: [ 68.983638][ T70] #0: ffff88800190a948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 68.983979][ T70] #1: ffffc900004bfda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 68.984211][ T70] #2: ffffffff9e19bc50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xb60 [ 68.984517][ T70] #3: ffffffff9e1b7208 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0 [ 68.984787][ T70] #4: ffff8880360bf198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 68.985026][ T70] [ 68.985026][ T70] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 68.985357][ T70] -> (&ul->lock#2){+.-.}-{2:2} { [ 68.985498][ T70] HARDIRQ-ON-W at: [ 68.985598][ T70] __lock_acquire+0x797/0x1570 [ 68.985853][ T70] lock_acquire.part.0+0xeb/0x330 [ 68.986027][ T70] _raw_spin_lock_bh+0x38/0x50 [ 68.986204][ T70] rt_add_uncached_list+0x55/0x1b0 [ 68.986372][ T70] rt_cache_route+0xfd/0x150 [ 68.986613][ T70] rt_set_nexthop.constprop.0+0x283/0xe90 [ 68.986813][ T70] __mkroute_output+0x4fa/0x1210 [ 68.986981][ T70] ip_route_output_key_hash+0x15d/0x2b0 [ 68.987261][ T70] ip_route_output_flow+0x23/0x140 [ 68.987429][ T70] udp_tunnel_dst_lookup+0x236/0x3b0 [ 68.987600][ T70] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 68.987912][ T70] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 68.988091][ T70] dev_hard_start_xmit+0x10e/0x360 [ 68.988253][ T70] __dev_queue_xmit+0xe79/0x16e0 [ 68.988492][ T70] ip6_finish_output2+0x512/0xf60 [ 68.988660][ T70] ip6_finish_output+0x553/0xdf0 [ 68.988829][ T70] ip6_output+0x1f3/0x770 [ 68.989073][ T70] NF_HOOK.constprop.0+0xe2/0x680 [ 68.989234][ T70] mld_sendpack+0x62d/0xbc0 [ 68.989402][ T70] mld_ifc_work+0x32/0x1f0 [ 68.989568][ T70] process_one_work+0xe2c/0x1730 [ 68.989809][ T70] worker_thread+0x587/0xd30 [ 68.989988][ T70] kthread+0x28a/0x350 [ 68.990126][ T70] ret_from_fork+0x31/0x70 [ 68.990376][ T70] ret_from_fork_asm+0x1a/0x30 [ 68.990543][ T70] IN-SOFTIRQ-W at: [ 68.990640][ T70] __lock_acquire+0x797/0x1570 [ 68.990807][ T70] lock_acquire.part.0+0xeb/0x330 [ 68.991067][ T70] _raw_spin_lock_bh+0x38/0x50 [ 68.991238][ T70] rt_add_uncached_list+0x55/0x1b0 [ 68.991406][ T70] rt_cache_route+0xfd/0x150 [ 68.991576][ T70] ip_route_input_slow+0x1001/0x2230 [ 68.991825][ T70] ip_route_input_noref+0xe6/0x230 [ 68.991993][ T70] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 68.992208][ T70] ip_rcv+0x199/0x740 [ 68.992421][ T70] __netif_receive_skb_one_core+0x166/0x1b0 [ 68.992619][ T70] process_backlog+0x103/0x670 [ 68.992785][ T70] __napi_poll.constprop.0+0xa2/0x460 [ 68.993105][ T70] net_rx_action+0x49f/0xc00 [ 68.993279][ T70] __do_softirq+0x1f8/0x5df [ 68.993447][ T70] do_softirq+0x4d/0xa0 [ 68.993623][ T70] __local_bh_enable_ip+0xf6/0x120 [ 68.993865][ T70] __dev_queue_xmit+0x6bc/0x16e0 [ 68.994030][ T70] ip6_finish_output2+0x512/0xf60 [ 68.994278][ T70] ip6_finish_output+0x553/0xdf0 [ 68.994434][ T70] ip6_output+0x1f3/0x770 [ 68.994601][ T70] NF_HOOK.constprop.0+0xe2/0x680 [ 68.994761][ T70] mld_sendpack+0x62d/0xbc0 [ 68.995010][ T70] mld_ifc_work+0x32/0x1f0 [ 68.995174][ T70] process_one_work+0xe2c/0x1730 [ 68.995340][ T70] worker_thread+0x587/0xd30 [ 68.995512][ T70] kthread+0x28a/0x350 [ 68.995655][ T70] ret_from_fork+0x31/0x70 [ 68.995831][ T70] ret_from_fork_asm+0x1a/0x30 [ 68.996089][ T70] INITIAL USE at: [ 68.996205][ T70] __lock_acquire+0x797/0x1570 [ 68.996383][ T70] lock_acquire.part.0+0xeb/0x330 [ 68.996549][ T70] _raw_spin_lock_bh+0x38/0x50 [ 68.996716][ T70] rt_add_uncached_list+0x55/0x1b0 [ 68.996963][ T70] rt_cache_route+0xfd/0x150 [ 68.997135][ T70] rt_set_nexthop.constprop.0+0x283/0xe90 [ 68.997338][ T70] __mkroute_output+0x4fa/0x1210 [ 68.997577][ T70] ip_route_output_key_hash+0x15d/0x2b0 [ 68.997743][ T70] ip_route_output_flow+0x23/0x140 [ 68.997910][ T70] udp_tunnel_dst_lookup+0x236/0x3b0 [ 68.998075][ T70] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 68.998326][ T70] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 68.998500][ T70] dev_hard_start_xmit+0x10e/0x360 [ 68.998663][ T70] __dev_queue_xmit+0xe79/0x16e0 [ 68.998904][ T70] ip6_finish_output2+0x512/0xf60 [ 68.999067][ T70] ip6_finish_output+0x553/0xdf0 [ 68.999234][ T70] ip6_output+0x1f3/0x770 [ 68.999399][ T70] NF_HOOK.constprop.0+0xe2/0x680 [ 68.999636][ T70] mld_sendpack+0x62d/0xbc0 [ 68.999798][ T70] mld_ifc_work+0x32/0x1f0 [ 68.999962][ T70] process_one_work+0xe2c/0x1730 [ 69.000277][ T70] worker_thread+0x587/0xd30 [ 69.000440][ T70] kthread+0x28a/0x350 [ 69.000575][ T70] ret_from_fork+0x31/0x70 [ 69.000745][ T70] ret_from_fork_asm+0x1a/0x30 [ 69.000989][ T70] } [ 69.001059][ T70] ... key at: [] __key.7+0x0/0x40 [ 69.001232][ T70] [ 69.001232][ T70] the dependencies between the lock to be acquired [ 69.001234][ T70] and SOFTIRQ-irq-unsafe lock: [ 69.001653][ T70] -> (&p->alloc_lock){+.+.}-{2:2} { [ 69.001790][ T70] HARDIRQ-ON-W at: [ 69.001890][ T70] __lock_acquire+0x797/0x1570 [ 69.002153][ T70] lock_acquire.part.0+0xeb/0x330 [ 69.002319][ T70] _raw_spin_lock+0x30/0x40 [ 69.002485][ T70] set_mems_allowed+0x1d/0x210 [ 69.002647][ T70] kernel_init_freeable+0x72/0x310 [ 69.002817][ T70] kernel_init+0x20/0x200 [ 69.002992][ T70] ret_from_fork+0x31/0x70 [ 69.003161][ T70] ret_from_fork_asm+0x1a/0x30 [ 69.003412][ T70] SOFTIRQ-ON-W at: [ 69.003514][ T70] __lock_acquire+0x797/0x1570 [ 69.003688][ T70] lock_acquire.part.0+0xeb/0x330 [ 69.003861][ T70] _raw_spin_lock+0x30/0x40 [ 69.004120][ T70] set_mems_allowed+0x1d/0x210 [ 69.004290][ T70] kernel_init_freeable+0x72/0x310 [ 69.004461][ T70] kernel_init+0x20/0x200 [ 69.004712][ T70] ret_from_fork+0x31/0x70 [ 69.004877][ T70] ret_from_fork_asm+0x1a/0x30 [ 69.005042][ T70] INITIAL USE at: [ 69.005144][ T70] __lock_acquire+0x797/0x1570 [ 69.005315][ T70] lock_acquire.part.0+0xeb/0x330 [ 69.005572][ T70] _raw_spin_lock+0x30/0x40 [ 69.005738][ T70] set_mems_allowed+0x1d/0x210 [ 69.005908][ T70] kernel_init_freeable+0x72/0x310 [ 69.006082][ T70] kernel_init+0x20/0x200 [ 69.006258][ T70] ret_from_fork+0x31/0x70 [ 69.006503][ T70] ret_from_fork_asm+0x1a/0x30 [ 69.006669][ T70] } [ 69.006743][ T70] ... key at: [] __key.388+0x0/0x40 [ 69.006950][ T70] ... acquired at: [ 69.007055][ T70] check_prev_add+0x194/0xce0 [ 69.007353][ T70] validate_chain+0x71d/0x9b0 [ 69.007489][ T70] __lock_acquire+0xaf0/0x1570 [ 69.007626][ T70] lock_acquire.part.0+0xeb/0x330 [ 69.007759][ T70] _raw_spin_lock+0x30/0x40 [ 69.007903][ T70] __get_task_comm+0x27/0x70 [ 69.008125][ T70] ref_tracker_alloc+0x2ee/0x490 [ 69.008266][ T70] rt_flush_dev+0x371/0x670 [ 69.008403][ T70] fib_netdev_event+0x248/0x5a0 [ 69.008537][ T70] notifier_call_chain+0xcd/0x150 [ 69.008676][ T70] unregister_netdevice_many_notify+0x548/0x1190 [ 69.008844][ T70] default_device_exit_batch+0x234/0x2d0 [ 69.009006][ T70] cleanup_net+0x4f3/0xb60 [ 69.009146][ T70] process_one_work+0xe2c/0x1730 [ 69.009353][ T70] worker_thread+0x587/0xd30 [ 69.009483][ T70] kthread+0x28a/0x350 [ 69.009590][ T70] ret_from_fork+0x31/0x70 [ 69.009722][ T70] ret_from_fork_asm+0x1a/0x30 [ 69.009931][ T70] [ 69.010009][ T70] [ 69.010009][ T70] stack backtrace: [ 69.010176][ T70] CPU: 3 PID: 70 Comm: kworker/u16:1 Not tainted 6.9.0-rc1-virtme #1 [ 69.010387][ T70] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 69.010768][ T70] Workqueue: netns cleanup_net [ 69.010916][ T70] Call Trace: [ 69.011023][ T70] [ 69.011096][ T70] dump_stack_lvl+0x82/0xd0 [ 69.011325][ T70] print_bad_irq_dependency+0x957/0xd40 [ 69.011461][ T70] ? __pfx_print_bad_irq_dependency+0x10/0x10 [ 69.011633][ T70] ? __pfx_usage_skip+0x10/0x10 [ 69.011767][ T70] ? __bfs+0x246/0x650 [ 69.011942][ T70] ? __pfx_usage_match+0x10/0x10 [ 69.012087][ T70] check_irq_usage+0x596/0x850 [ 69.012218][ T70] ? arch_stack_walk+0x8c/0xf0 [ 69.012357][ T70] ? __pfx_check_irq_usage+0x10/0x10 [ 69.012494][ T70] ? __bfs+0x246/0x650 [ 69.012603][ T70] ? __pfx_hlock_conflict+0x10/0x10 [ 69.012737][ T70] ? check_path.constprop.0+0x24/0x50 [ 69.012871][ T70] ? check_noncircular+0x14c/0x3e0 [ 69.013004][ T70] ? __pfx_check_noncircular+0x10/0x10 [ 69.013142][ T70] ? find_held_lock+0x2c/0x110 [ 69.013282][ T70] ? check_prev_add+0x194/0xce0 [ 69.013419][ T70] check_prev_add+0x194/0xce0 [ 69.013554][ T70] ? validate_chain+0x130/0x9b0 [ 69.013690][ T70] validate_chain+0x71d/0x9b0 [ 69.013819][ T70] ? mark_lock+0x38/0x3e0 [ 69.013921][ T70] ? __pfx_validate_chain+0x10/0x10 [ 69.014052][ T70] ? hlock_class+0x4e/0x130 [ 69.014189][ T70] ? mark_lock+0x38/0x3e0 [ 69.014285][ T70] __lock_acquire+0xaf0/0x1570 [ 69.014427][ T70] lock_acquire.part.0+0xeb/0x330 [ 69.014564][ T70] ? __get_task_comm+0x27/0x70 [ 69.014709][ T70] ? mark_lock+0x38/0x3e0 [ 69.014812][ T70] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 69.014941][ T70] ? trace_lock_acquire+0x135/0x1c0 [ 69.015076][ T70] ? __get_task_comm+0x27/0x70 [ 69.015209][ T70] ? lock_acquire+0x32/0xc0 [ 69.015345][ T70] ? __get_task_comm+0x27/0x70 [ 69.015476][ T70] _raw_spin_lock+0x30/0x40 [ 69.015616][ T70] ? __get_task_comm+0x27/0x70 [ 69.015752][ T70] __get_task_comm+0x27/0x70 [ 69.015887][ T70] ref_tracker_alloc+0x2ee/0x490 [ 69.016027][ T70] ? lock_acquire.part.0+0xeb/0x330 [ 69.016167][ T70] ? __pfx_ref_tracker_alloc+0x10/0x10 [ 69.016303][ T70] ? rt_flush_dev+0x371/0x670 [ 69.016444][ T70] ? fib_netdev_event+0x248/0x5a0 [ 69.016586][ T70] ? notifier_call_chain+0xcd/0x150 [ 69.016729][ T70] ? unregister_netdevice_many_notify+0x548/0x1190 [ 69.016895][ T70] ? default_device_exit_batch+0x234/0x2d0 [ 69.017059][ T70] ? cleanup_net+0x4f3/0xb60 [ 69.017195][ T70] ? process_one_work+0xe2c/0x1730 [ 69.017326][ T70] ? worker_thread+0x587/0xd30 [ 69.017464][ T70] ? kthread+0x28a/0x350 [ 69.017564][ T70] ? ret_from_fork+0x31/0x70 [ 69.017700][ T70] ? ret_from_fork_asm+0x1a/0x30 [ 69.017837][ T70] ? do_raw_spin_lock+0x131/0x270 [ 69.017972][ T70] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 69.018102][ T70] ? lock_acquire+0x32/0xc0 [ 69.018235][ T70] ? rt_flush_dev+0x13b/0x670 [ 69.018372][ T70] rt_flush_dev+0x371/0x670 [ 69.018505][ T70] ? __timer_delete_sync+0xd3/0x110 [ 69.018642][ T70] ? __timer_delete_sync+0xa7/0x110 [ 69.018777][ T70] fib_netdev_event+0x248/0x5a0 [ 69.018911][ T70] notifier_call_chain+0xcd/0x150 [ 69.019045][ T70] unregister_netdevice_many_notify+0x548/0x1190 [ 69.019211][ T70] ? mutex_is_locked+0x17/0x50 [ 69.019345][ T70] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 69.019511][ T70] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 69.019679][ T70] default_device_exit_batch+0x234/0x2d0 [ 69.019810][ T70] ? __pfx_default_device_exit_batch+0x10/0x10 [ 69.019985][ T70] ? ops_exit_list+0xb4/0x170 [ 69.020121][ T70] cleanup_net+0x4f3/0xb60 [ 69.020253][ T70] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 69.020389][ T70] ? __pfx_cleanup_net+0x10/0x10 [ 69.020525][ T70] ? trace_lock_acquire+0x135/0x1c0 [ 69.020660][ T70] ? process_one_work+0xde2/0x1730 [ 69.020794][ T70] ? lock_acquire+0x32/0xc0 [ 69.020925][ T70] ? process_one_work+0xde2/0x1730 [ 69.021064][ T70] process_one_work+0xe2c/0x1730 [ 69.021205][ T70] ? __pfx___lock_release+0x10/0x10 [ 69.021340][ T70] ? __pfx_process_one_work+0x10/0x10 [ 69.021474][ T70] ? assign_work+0x16c/0x240 [ 69.021608][ T70] worker_thread+0x587/0xd30 [ 69.021746][ T70] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 69.021909][ T70] ? __pfx_worker_thread+0x10/0x10 [ 69.022042][ T70] ? __pfx_worker_thread+0x10/0x10 [ 69.022170][ T70] kthread+0x28a/0x350 [ 69.022273][ T70] ? __pfx_kthread+0x10/0x10 [ 69.022405][ T70] ret_from_fork+0x31/0x70 [ 69.022537][ T70] ? __pfx_kthread+0x10/0x10 [ 69.022665][ T70] ret_from_fork_asm+0x1a/0x30 [ 69.022799][ T70] [ 112.803674][ T943] udpgso_bench_tx (943) used greatest stack depth: 23336 bytes left