[ 59.212950][ T10] [ 59.213064][ T10] ===================================================== [ 59.213252][ T10] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected [ 59.213468][ T10] 6.9.0-rc1-virtme #1 Not tainted [ 59.213617][ T10] ----------------------------------------------------- [ 59.213812][ T10] kworker/u16:0/10 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 59.214036][ T10] ffff888001a70968 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70 [ 59.214307][ T10] [ 59.214307][ T10] and this task is already holding: [ 59.214534][ T10] ffff8880360bf198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 59.214804][ T10] which would create a new lock dependency: [ 59.214984][ T10] (&ul->lock#2){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2} [ 59.215205][ T10] [ 59.215205][ T10] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 59.215458][ T10] (&ul->lock#2){+.-.}-{2:2} [ 59.215463][ T10] [ 59.215463][ T10] ... which became SOFTIRQ-irq-safe at: [ 59.215831][ T10] __lock_acquire+0x797/0x1570 [ 59.216006][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.216157][ T10] _raw_spin_lock_bh+0x38/0x50 [ 59.216321][ T10] rt_add_uncached_list+0x55/0x1b0 [ 59.216480][ T10] rt_cache_route+0xfd/0x150 [ 59.216629][ T10] ip_route_input_slow+0x1001/0x2230 [ 59.216786][ T10] ip_route_input_noref+0xe6/0x230 [ 59.216937][ T10] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 59.217123][ T10] ip_rcv+0x199/0x740 [ 59.217234][ T10] __netif_receive_skb_one_core+0x166/0x1b0 [ 59.217422][ T10] process_backlog+0x103/0x670 [ 59.217574][ T10] __napi_poll.constprop.0+0xa2/0x460 [ 59.217721][ T10] net_rx_action+0x49f/0xc00 [ 59.217869][ T10] __do_softirq+0x1f8/0x5df [ 59.218035][ T10] do_softirq+0x4d/0xa0 [ 59.218149][ T10] __local_bh_enable_ip+0xf6/0x120 [ 59.218306][ T10] __dev_queue_xmit+0x6bc/0x16e0 [ 59.218452][ T10] ip6_finish_output2+0x512/0xf60 [ 59.218599][ T10] ip6_finish_output+0x553/0xdf0 [ 59.218751][ T10] ip6_output+0x1f3/0x770 [ 59.218900][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 59.219056][ T10] mld_sendpack+0x62d/0xbc0 [ 59.219196][ T10] mld_ifc_work+0x32/0x1f0 [ 59.219344][ T10] process_one_work+0xe2c/0x1730 [ 59.219485][ T10] worker_thread+0x587/0xd30 [ 59.219637][ T10] kthread+0x28a/0x350 [ 59.219749][ T10] ret_from_fork+0x31/0x70 [ 59.219909][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.220069][ T10] [ 59.220069][ T10] to a SOFTIRQ-irq-unsafe lock: [ 59.220254][ T10] (&p->alloc_lock){+.+.}-{2:2} [ 59.220259][ T10] [ 59.220259][ T10] ... which became SOFTIRQ-irq-unsafe at: [ 59.220622][ T10] ... [ 59.220623][ T10] __lock_acquire+0x797/0x1570 [ 59.220848][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.221001][ T10] _raw_spin_lock+0x30/0x40 [ 59.221151][ T10] set_mems_allowed+0x1d/0x210 [ 59.221305][ T10] kernel_init_freeable+0x72/0x310 [ 59.221449][ T10] kernel_init+0x20/0x200 [ 59.221608][ T10] ret_from_fork+0x31/0x70 [ 59.221756][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.221903][ T10] [ 59.221903][ T10] other info that might help us debug this: [ 59.221903][ T10] [ 59.222191][ T10] Possible interrupt unsafe locking scenario: [ 59.222191][ T10] [ 59.222430][ T10] CPU0 CPU1 [ 59.222569][ T10] ---- ---- [ 59.222714][ T10] lock(&p->alloc_lock); [ 59.222826][ T10] local_irq_disable(); [ 59.223005][ T10] lock(&ul->lock#2); [ 59.223185][ T10] lock(&p->alloc_lock); [ 59.223366][ T10] [ 59.223482][ T10] lock(&ul->lock#2); [ 59.223597][ T10] [ 59.223597][ T10] *** DEADLOCK *** [ 59.223597][ T10] [ 59.223816][ T10] 5 locks held by kworker/u16:0/10: [ 59.223974][ T10] #0: ffff88800190a948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 59.224349][ T10] #1: ffffc900000afda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 59.224606][ T10] #2: ffffffffaa59bc50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xb60 [ 59.224935][ T10] #3: ffffffffaa5b7208 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0 [ 59.225235][ T10] #4: ffff8880360bf198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 59.225494][ T10] [ 59.225494][ T10] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 59.225851][ T10] -> (&ul->lock#2){+.-.}-{2:2} { [ 59.226002][ T10] HARDIRQ-ON-W at: [ 59.226113][ T10] __lock_acquire+0x797/0x1570 [ 59.226370][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.226567][ T10] _raw_spin_lock_bh+0x38/0x50 [ 59.226745][ T10] rt_add_uncached_list+0x55/0x1b0 [ 59.226925][ T10] rt_cache_route+0xfd/0x150 [ 59.227180][ T10] rt_set_nexthop.constprop.0+0x283/0xe90 [ 59.227407][ T10] __mkroute_output+0x4fa/0x1210 [ 59.227595][ T10] ip_route_output_key_hash+0x15d/0x2b0 [ 59.227892][ T10] ip_route_output_flow+0x23/0x140 [ 59.228075][ T10] udp_tunnel_dst_lookup+0x236/0x3b0 [ 59.228274][ T10] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 59.228602][ T10] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 59.228803][ T10] dev_hard_start_xmit+0x10e/0x360 [ 59.228996][ T10] __dev_queue_xmit+0xe79/0x16e0 [ 59.229191][ T10] ip6_finish_output2+0x512/0xf60 [ 59.229473][ T10] ip6_finish_output+0x553/0xdf0 [ 59.229673][ T10] ip6_output+0x1f3/0x770 [ 59.229863][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 59.230061][ T10] mld_sendpack+0x62d/0xbc0 [ 59.230256][ T10] mld_ifc_work+0x32/0x1f0 [ 59.230531][ T10] process_one_work+0xe2c/0x1730 [ 59.230713][ T10] worker_thread+0x587/0xd30 [ 59.230899][ T10] kthread+0x28a/0x350 [ 59.231143][ T10] ret_from_fork+0x31/0x70 [ 59.231333][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.231522][ T10] IN-SOFTIRQ-W at: [ 59.231637][ T10] __lock_acquire+0x797/0x1570 [ 59.231825][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.232105][ T10] _raw_spin_lock_bh+0x38/0x50 [ 59.232296][ T10] rt_add_uncached_list+0x55/0x1b0 [ 59.232561][ T10] rt_cache_route+0xfd/0x150 [ 59.232749][ T10] ip_route_input_slow+0x1001/0x2230 [ 59.232937][ T10] ip_route_input_noref+0xe6/0x230 [ 59.233135][ T10] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 59.233454][ T10] ip_rcv+0x199/0x740 [ 59.233612][ T10] __netif_receive_skb_one_core+0x166/0x1b0 [ 59.233838][ T10] process_backlog+0x103/0x670 [ 59.234018][ T10] __napi_poll.constprop.0+0xa2/0x460 [ 59.234280][ T10] net_rx_action+0x49f/0xc00 [ 59.234548][ T10] __do_softirq+0x1f8/0x5df [ 59.234743][ T10] do_softirq+0x4d/0xa0 [ 59.234928][ T10] __local_bh_enable_ip+0xf6/0x120 [ 59.235114][ T10] __dev_queue_xmit+0x6bc/0x16e0 [ 59.235300][ T10] ip6_finish_output2+0x512/0xf60 [ 59.235489][ T10] ip6_finish_output+0x553/0xdf0 [ 59.235699][ T10] ip6_output+0x1f3/0x770 [ 59.235891][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 59.236078][ T10] mld_sendpack+0x62d/0xbc0 [ 59.236272][ T10] mld_ifc_work+0x32/0x1f0 [ 59.236457][ T10] process_one_work+0xe2c/0x1730 [ 59.236653][ T10] worker_thread+0x587/0xd30 [ 59.236838][ T10] kthread+0x28a/0x350 [ 59.236991][ T10] ret_from_fork+0x31/0x70 [ 59.237191][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.237385][ T10] INITIAL USE at: [ 59.237493][ T10] __lock_acquire+0x797/0x1570 [ 59.237680][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.237861][ T10] _raw_spin_lock_bh+0x38/0x50 [ 59.238049][ T10] rt_add_uncached_list+0x55/0x1b0 [ 59.238242][ T10] rt_cache_route+0xfd/0x150 [ 59.238439][ T10] rt_set_nexthop.constprop.0+0x283/0xe90 [ 59.238659][ T10] __mkroute_output+0x4fa/0x1210 [ 59.238851][ T10] ip_route_output_key_hash+0x15d/0x2b0 [ 59.239039][ T10] ip_route_output_flow+0x23/0x140 [ 59.239234][ T10] udp_tunnel_dst_lookup+0x236/0x3b0 [ 59.239419][ T10] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 59.239624][ T10] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 59.239818][ T10] dev_hard_start_xmit+0x10e/0x360 [ 59.240005][ T10] __dev_queue_xmit+0xe79/0x16e0 [ 59.240209][ T10] ip6_finish_output2+0x512/0xf60 [ 59.240391][ T10] ip6_finish_output+0x553/0xdf0 [ 59.240568][ T10] ip6_output+0x1f3/0x770 [ 59.240752][ T10] NF_HOOK.constprop.0+0xe2/0x680 [ 59.240938][ T10] mld_sendpack+0x62d/0xbc0 [ 59.241127][ T10] mld_ifc_work+0x32/0x1f0 [ 59.241324][ T10] process_one_work+0xe2c/0x1730 [ 59.241513][ T10] worker_thread+0x587/0xd30 [ 59.241698][ T10] kthread+0x28a/0x350 [ 59.241848][ T10] ret_from_fork+0x31/0x70 [ 59.242040][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.242229][ T10] } [ 59.242306][ T10] ... key at: [] __key.7+0x0/0x40 [ 59.242498][ T10] [ 59.242498][ T10] the dependencies between the lock to be acquired [ 59.242500][ T10] and SOFTIRQ-irq-unsafe lock: [ 59.242942][ T10] -> (&p->alloc_lock){+.+.}-{2:2} { [ 59.243102][ T10] HARDIRQ-ON-W at: [ 59.243218][ T10] __lock_acquire+0x797/0x1570 [ 59.243410][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.243597][ T10] _raw_spin_lock+0x30/0x40 [ 59.243781][ T10] set_mems_allowed+0x1d/0x210 [ 59.243972][ T10] kernel_init_freeable+0x72/0x310 [ 59.244161][ T10] kernel_init+0x20/0x200 [ 59.244348][ T10] ret_from_fork+0x31/0x70 [ 59.244540][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.244717][ T10] SOFTIRQ-ON-W at: [ 59.244827][ T10] __lock_acquire+0x797/0x1570 [ 59.245013][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.245203][ T10] _raw_spin_lock+0x30/0x40 [ 59.245401][ T10] set_mems_allowed+0x1d/0x210 [ 59.245588][ T10] kernel_init_freeable+0x72/0x310 [ 59.245771][ T10] kernel_init+0x20/0x200 [ 59.245966][ T10] ret_from_fork+0x31/0x70 [ 59.246157][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.246346][ T10] INITIAL USE at: [ 59.246458][ T10] __lock_acquire+0x797/0x1570 [ 59.246649][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.246836][ T10] _raw_spin_lock+0x30/0x40 [ 59.247025][ T10] set_mems_allowed+0x1d/0x210 [ 59.247216][ T10] kernel_init_freeable+0x72/0x310 [ 59.247402][ T10] kernel_init+0x20/0x200 [ 59.247587][ T10] ret_from_fork+0x31/0x70 [ 59.247772][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.247958][ T10] } [ 59.248035][ T10] ... key at: [] __key.388+0x0/0x40 [ 59.248257][ T10] ... acquired at: [ 59.248368][ T10] check_prev_add+0x194/0xce0 [ 59.248522][ T10] validate_chain+0x71d/0x9b0 [ 59.248671][ T10] __lock_acquire+0xaf0/0x1570 [ 59.248819][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.248964][ T10] _raw_spin_lock+0x30/0x40 [ 59.249110][ T10] __get_task_comm+0x27/0x70 [ 59.249259][ T10] ref_tracker_alloc+0x2ee/0x490 [ 59.249405][ T10] rt_flush_dev+0x371/0x670 [ 59.249554][ T10] fib_netdev_event+0x248/0x5a0 [ 59.249700][ T10] notifier_call_chain+0xcd/0x150 [ 59.249852][ T10] unregister_netdevice_many_notify+0x548/0x1190 [ 59.250044][ T10] default_device_exit_batch+0x234/0x2d0 [ 59.250231][ T10] cleanup_net+0x4f3/0xb60 [ 59.250378][ T10] process_one_work+0xe2c/0x1730 [ 59.250522][ T10] worker_thread+0x587/0xd30 [ 59.250667][ T10] kthread+0x28a/0x350 [ 59.250777][ T10] ret_from_fork+0x31/0x70 [ 59.250925][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.251144][ T10] [ 59.251219][ T10] [ 59.251219][ T10] stack backtrace: [ 59.251400][ T10] CPU: 3 PID: 10 Comm: kworker/u16:0 Not tainted 6.9.0-rc1-virtme #1 [ 59.251629][ T10] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 59.252118][ T10] Workqueue: netns cleanup_net [ 59.252287][ T10] Call Trace: [ 59.252402][ T10] [ 59.252478][ T10] dump_stack_lvl+0x82/0xd0 [ 59.252623][ T10] print_bad_irq_dependency+0x957/0xd40 [ 59.252797][ T10] ? __pfx_print_bad_irq_dependency+0x10/0x10 [ 59.252989][ T10] ? __pfx_usage_skip+0x10/0x10 [ 59.253221][ T10] ? __bfs+0x246/0x650 [ 59.253340][ T10] ? __pfx_usage_match+0x10/0x10 [ 59.253489][ T10] check_irq_usage+0x596/0x850 [ 59.253642][ T10] ? arch_stack_walk+0x8c/0xf0 [ 59.253871][ T10] ? __pfx_check_irq_usage+0x10/0x10 [ 59.254024][ T10] ? __bfs+0x246/0x650 [ 59.254138][ T10] ? __pfx_hlock_conflict+0x10/0x10 [ 59.254289][ T10] ? check_path.constprop.0+0x24/0x50 [ 59.254436][ T10] ? check_noncircular+0x14c/0x3e0 [ 59.254590][ T10] ? __pfx_check_noncircular+0x10/0x10 [ 59.254810][ T10] ? find_held_lock+0x2c/0x110 [ 59.254962][ T10] ? check_prev_add+0x194/0xce0 [ 59.255110][ T10] check_prev_add+0x194/0xce0 [ 59.255263][ T10] ? validate_chain+0x130/0x9b0 [ 59.255495][ T10] validate_chain+0x71d/0x9b0 [ 59.255727][ T10] ? mark_lock+0x38/0x3e0 [ 59.255850][ T10] ? __pfx_validate_chain+0x10/0x10 [ 59.256003][ T10] ? hlock_class+0x4e/0x130 [ 59.256151][ T10] ? mark_lock+0x38/0x3e0 [ 59.256264][ T10] __lock_acquire+0xaf0/0x1570 [ 59.256414][ T10] lock_acquire.part.0+0xeb/0x330 [ 59.256557][ T10] ? __get_task_comm+0x27/0x70 [ 59.256785][ T10] ? mark_lock+0x38/0x3e0 [ 59.256892][ T10] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 59.257035][ T10] ? trace_lock_acquire+0x135/0x1c0 [ 59.257178][ T10] ? __get_task_comm+0x27/0x70 [ 59.257321][ T10] ? lock_acquire+0x32/0xc0 [ 59.257473][ T10] ? __get_task_comm+0x27/0x70 [ 59.257698][ T10] _raw_spin_lock+0x30/0x40 [ 59.257842][ T10] ? __get_task_comm+0x27/0x70 [ 59.257998][ T10] __get_task_comm+0x27/0x70 [ 59.258140][ T10] ref_tracker_alloc+0x2ee/0x490 [ 59.258285][ T10] ? lock_acquire.part.0+0xeb/0x330 [ 59.258429][ T10] ? __pfx_ref_tracker_alloc+0x10/0x10 [ 59.258574][ T10] ? rt_flush_dev+0x371/0x670 [ 59.258717][ T10] ? fib_netdev_event+0x248/0x5a0 [ 59.258928][ T10] ? notifier_call_chain+0xcd/0x150 [ 59.259070][ T10] ? unregister_netdevice_many_notify+0x548/0x1190 [ 59.259247][ T10] ? default_device_exit_batch+0x234/0x2d0 [ 59.259429][ T10] ? cleanup_net+0x4f3/0xb60 [ 59.259639][ T10] ? process_one_work+0xe2c/0x1730 [ 59.259779][ T10] ? worker_thread+0x587/0xd30 [ 59.259922][ T10] ? kthread+0x28a/0x350 [ 59.260028][ T10] ? ret_from_fork+0x31/0x70 [ 59.260196][ T10] ? ret_from_fork_asm+0x1a/0x30 [ 59.260410][ T10] ? do_raw_spin_lock+0x131/0x270 [ 59.260558][ T10] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 59.260700][ T10] ? lock_acquire+0x32/0xc0 [ 59.260842][ T10] ? rt_flush_dev+0x13b/0x670 [ 59.261062][ T10] rt_flush_dev+0x371/0x670 [ 59.261207][ T10] ? __timer_delete_sync+0xd3/0x110 [ 59.261363][ T10] ? __timer_delete_sync+0xa7/0x110 [ 59.261507][ T10] fib_netdev_event+0x248/0x5a0 [ 59.261718][ T10] notifier_call_chain+0xcd/0x150 [ 59.261878][ T10] unregister_netdevice_many_notify+0x548/0x1190 [ 59.262055][ T10] ? mutex_is_locked+0x17/0x50 [ 59.262201][ T10] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 59.262457][ T10] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 59.262635][ T10] default_device_exit_batch+0x234/0x2d0 [ 59.262777][ T10] ? __pfx_default_device_exit_batch+0x10/0x10 [ 59.262952][ T10] ? ops_exit_list+0xb4/0x170 [ 59.263192][ T10] cleanup_net+0x4f3/0xb60 [ 59.263332][ T10] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 59.263475][ T10] ? __pfx_cleanup_net+0x10/0x10 [ 59.263616][ T10] ? trace_lock_acquire+0x135/0x1c0 [ 59.263827][ T10] ? process_one_work+0xde2/0x1730 [ 59.263970][ T10] ? lock_acquire+0x32/0xc0 [ 59.264111][ T10] ? process_one_work+0xde2/0x1730 [ 59.264261][ T10] process_one_work+0xe2c/0x1730 [ 59.264465][ T10] ? __pfx___lock_release+0x10/0x10 [ 59.264618][ T10] ? __pfx_process_one_work+0x10/0x10 [ 59.264764][ T10] ? assign_work+0x16c/0x240 [ 59.264907][ T10] worker_thread+0x587/0xd30 [ 59.265054][ T10] ? __pfx_worker_thread+0x10/0x10 [ 59.265197][ T10] kthread+0x28a/0x350 [ 59.265304][ T10] ? __pfx_kthread+0x10/0x10 [ 59.265446][ T10] ret_from_fork+0x31/0x70 [ 59.265589][ T10] ? __pfx_kthread+0x10/0x10 [ 59.265868][ T10] ret_from_fork_asm+0x1a/0x30 [ 59.266026][ T10]