[ 24.953951][ T243] ip (243) used greatest stack depth: 23168 bytes left [ 60.436014][ T72] [ 60.436117][ T72] ===================================================== [ 60.436294][ T72] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected [ 60.436512][ T72] 6.9.0-rc1-virtme #1 Not tainted [ 60.436655][ T72] ----------------------------------------------------- [ 60.436829][ T72] kworker/u16:1/72 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 60.437041][ T72] ffff888005514ee8 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70 [ 60.437306][ T72] [ 60.437306][ T72] and this task is already holding: [ 60.437514][ T72] ffff88803613f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 60.437763][ T72] which would create a new lock dependency: [ 60.437938][ T72] (&ul->lock#2){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2} [ 60.438157][ T72] [ 60.438157][ T72] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 60.438389][ T72] (&ul->lock#2){+.-.}-{2:2} [ 60.438393][ T72] [ 60.438393][ T72] ... which became SOFTIRQ-irq-safe at: [ 60.438746][ T72] __lock_acquire+0x797/0x1570 [ 60.438890][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.439033][ T72] _raw_spin_lock_bh+0x38/0x50 [ 60.439173][ T72] rt_add_uncached_list+0x55/0x1b0 [ 60.439312][ T72] rt_cache_route+0xfd/0x150 [ 60.439447][ T72] ip_route_input_slow+0x1001/0x2230 [ 60.439586][ T72] ip_route_input_noref+0xe6/0x230 [ 60.439730][ T72] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 60.439890][ T72] ip_rcv+0x199/0x740 [ 60.439995][ T72] __netif_receive_skb_one_core+0x166/0x1b0 [ 60.440170][ T72] process_backlog+0x103/0x670 [ 60.440317][ T72] __napi_poll.constprop.0+0xa2/0x460 [ 60.440458][ T72] net_rx_action+0x49f/0xc00 [ 60.440596][ T72] __do_softirq+0x1f8/0x5df [ 60.440732][ T72] do_softirq+0x4d/0xa0 [ 60.440840][ T72] __local_bh_enable_ip+0xf6/0x120 [ 60.440980][ T72] __dev_queue_xmit+0x6bc/0x16e0 [ 60.441121][ T72] ip6_finish_output2+0x512/0xf60 [ 60.441266][ T72] ip6_finish_output+0x553/0xdf0 [ 60.441411][ T72] ip6_output+0x1f3/0x770 [ 60.441628][ T72] NF_HOOK.constprop.0+0xe2/0x680 [ 60.441773][ T72] ndisc_send_skb+0x8e0/0x1230 [ 60.441917][ T72] ndisc_send_ns+0x9e/0x100 [ 60.442054][ T72] addrconf_dad_work+0x7dd/0xab0 [ 60.442284][ T72] process_one_work+0xe2c/0x1730 [ 60.442427][ T72] worker_thread+0x587/0xd30 [ 60.442570][ T72] kthread+0x28a/0x350 [ 60.442685][ T72] ret_from_fork+0x31/0x70 [ 60.442826][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.442968][ T72] [ 60.442968][ T72] to a SOFTIRQ-irq-unsafe lock: [ 60.443147][ T72] (&p->alloc_lock){+.+.}-{2:2} [ 60.443151][ T72] [ 60.443151][ T72] ... which became SOFTIRQ-irq-unsafe at: [ 60.443564][ T72] ... [ 60.443566][ T72] __lock_acquire+0x797/0x1570 [ 60.443777][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.443919][ T72] _raw_spin_lock+0x30/0x40 [ 60.444063][ T72] set_mems_allowed+0x1d/0x210 [ 60.444281][ T72] kernel_init_freeable+0x72/0x310 [ 60.444488][ T72] kernel_init+0x20/0x200 [ 60.444628][ T72] ret_from_fork+0x31/0x70 [ 60.444770][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.444911][ T72] [ 60.444911][ T72] other info that might help us debug this: [ 60.444911][ T72] [ 60.445178][ T72] Possible interrupt unsafe locking scenario: [ 60.445178][ T72] [ 60.445382][ T72] CPU0 CPU1 [ 60.445590][ T72] ---- ---- [ 60.445729][ T72] lock(&p->alloc_lock); [ 60.445836][ T72] local_irq_disable(); [ 60.446005][ T72] lock(&ul->lock#2); [ 60.446250][ T72] lock(&p->alloc_lock); [ 60.446424][ T72] [ 60.446528][ T72] lock(&ul->lock#2); [ 60.446634][ T72] [ 60.446634][ T72] *** DEADLOCK *** [ 60.446634][ T72] [ 60.446918][ T72] 5 locks held by kworker/u16:1/72: [ 60.447061][ T72] #0: ffff88800190a948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 60.447343][ T72] #1: ffffc900004dfda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 60.447663][ T72] #2: ffffffffb699bc50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xb60 [ 60.447907][ T72] #3: ffffffffb69b7208 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0 [ 60.448264][ T72] #4: ffff88803613f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 60.448515][ T72] [ 60.448515][ T72] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 60.448852][ T72] -> (&ul->lock#2){+.-.}-{2:2} { [ 60.448997][ T72] HARDIRQ-ON-W at: [ 60.449105][ T72] __lock_acquire+0x797/0x1570 [ 60.449279][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.449610][ T72] _raw_spin_lock_bh+0x38/0x50 [ 60.449786][ T72] rt_add_uncached_list+0x55/0x1b0 [ 60.449975][ T72] rt_cache_route+0xfd/0x150 [ 60.450154][ T72] rt_set_nexthop.constprop.0+0x283/0xe90 [ 60.450451][ T72] __mkroute_output+0x4fa/0x1210 [ 60.450627][ T72] ip_route_output_key_hash+0x15d/0x2b0 [ 60.450836][ T72] ip_route_output_flow+0x23/0x140 [ 60.451095][ T72] udp_tunnel_dst_lookup+0x236/0x3b0 [ 60.451270][ T72] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 60.451501][ T72] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 60.451689][ T72] dev_hard_start_xmit+0x10e/0x360 [ 60.451950][ T72] __dev_queue_xmit+0xe79/0x16e0 [ 60.452127][ T72] ip6_finish_output2+0x512/0xf60 [ 60.452305][ T72] ip6_finish_output+0x553/0xdf0 [ 60.452484][ T72] ip6_output+0x1f3/0x770 [ 60.452738][ T72] NF_HOOK.constprop.0+0xe2/0x680 [ 60.452915][ T72] ndisc_send_skb+0x8e0/0x1230 [ 60.453174][ T72] ndisc_send_ns+0x9e/0x100 [ 60.453353][ T72] addrconf_dad_work+0x7dd/0xab0 [ 60.453535][ T72] process_one_work+0xe2c/0x1730 [ 60.453799][ T72] worker_thread+0x587/0xd30 [ 60.453978][ T72] kthread+0x28a/0x350 [ 60.454122][ T72] ret_from_fork+0x31/0x70 [ 60.454297][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.454542][ T72] IN-SOFTIRQ-W at: [ 60.454651][ T72] __lock_acquire+0x797/0x1570 [ 60.454826][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.455001][ T72] _raw_spin_lock_bh+0x38/0x50 [ 60.455265][ T72] rt_add_uncached_list+0x55/0x1b0 [ 60.455450][ T72] rt_cache_route+0xfd/0x150 [ 60.455625][ T72] ip_route_input_slow+0x1001/0x2230 [ 60.455891][ T72] ip_route_input_noref+0xe6/0x230 [ 60.456066][ T72] ip_rcv_finish_core.constprop.0+0x461/0x1300 [ 60.456278][ T72] ip_rcv+0x199/0x740 [ 60.456418][ T72] __netif_receive_skb_one_core+0x166/0x1b0 [ 60.456703][ T72] process_backlog+0x103/0x670 [ 60.456877][ T72] __napi_poll.constprop.0+0xa2/0x460 [ 60.457055][ T72] net_rx_action+0x49f/0xc00 [ 60.457308][ T72] __do_softirq+0x1f8/0x5df [ 60.457484][ T72] do_softirq+0x4d/0xa0 [ 60.457663][ T72] __local_bh_enable_ip+0xf6/0x120 [ 60.457914][ T72] __dev_queue_xmit+0x6bc/0x16e0 [ 60.458171][ T72] ip6_finish_output2+0x512/0xf60 [ 60.458348][ T72] ip6_finish_output+0x553/0xdf0 [ 60.458528][ T72] ip6_output+0x1f3/0x770 [ 60.458702][ T72] NF_HOOK.constprop.0+0xe2/0x680 [ 60.458958][ T72] ndisc_send_skb+0x8e0/0x1230 [ 60.459133][ T72] ndisc_send_ns+0x9e/0x100 [ 60.459312][ T72] addrconf_dad_work+0x7dd/0xab0 [ 60.459488][ T72] process_one_work+0xe2c/0x1730 [ 60.459664][ T72] worker_thread+0x587/0xd30 [ 60.459925][ T72] kthread+0x28a/0x350 [ 60.460065][ T72] ret_from_fork+0x31/0x70 [ 60.460242][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.460423][ T72] INITIAL USE at: [ 60.460602][ T72] __lock_acquire+0x797/0x1570 [ 60.460781][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.461038][ T72] _raw_spin_lock_bh+0x38/0x50 [ 60.461218][ T72] rt_add_uncached_list+0x55/0x1b0 [ 60.461393][ T72] rt_cache_route+0xfd/0x150 [ 60.461659][ T72] rt_set_nexthop.constprop.0+0x283/0xe90 [ 60.461869][ T72] __mkroute_output+0x4fa/0x1210 [ 60.462051][ T72] ip_route_output_key_hash+0x15d/0x2b0 [ 60.462226][ T72] ip_route_output_flow+0x23/0x140 [ 60.462401][ T72] udp_tunnel_dst_lookup+0x236/0x3b0 [ 60.462576][ T72] vxlan_xmit_one+0x11c9/0x4570 [vxlan] [ 60.462778][ T72] vxlan_xmit+0x9b6/0xf40 [vxlan] [ 60.462964][ T72] dev_hard_start_xmit+0x10e/0x360 [ 60.463138][ T72] __dev_queue_xmit+0xe79/0x16e0 [ 60.463395][ T72] ip6_finish_output2+0x512/0xf60 [ 60.463569][ T72] ip6_finish_output+0x553/0xdf0 [ 60.463820][ T72] ip6_output+0x1f3/0x770 [ 60.463992][ T72] NF_HOOK.constprop.0+0xe2/0x680 [ 60.464168][ T72] ndisc_send_skb+0x8e0/0x1230 [ 60.464346][ T72] ndisc_send_ns+0x9e/0x100 [ 60.464599][ T72] addrconf_dad_work+0x7dd/0xab0 [ 60.464776][ T72] process_one_work+0xe2c/0x1730 [ 60.464956][ T72] worker_thread+0x587/0xd30 [ 60.465135][ T72] kthread+0x28a/0x350 [ 60.465352][ T72] ret_from_fork+0x31/0x70 [ 60.465609][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.465791][ T72] } [ 60.465863][ T72] ... key at: [] __key.7+0x0/0x40 [ 60.466045][ T72] [ 60.466045][ T72] the dependencies between the lock to be acquired [ 60.466046][ T72] and SOFTIRQ-irq-unsafe lock: [ 60.466487][ T72] -> (&p->alloc_lock){+.+.}-{2:2} { [ 60.466633][ T72] HARDIRQ-ON-W at: [ 60.466742][ T72] __lock_acquire+0x797/0x1570 [ 60.466917][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.467094][ T72] _raw_spin_lock+0x30/0x40 [ 60.467349][ T72] set_mems_allowed+0x1d/0x210 [ 60.467525][ T72] kernel_init_freeable+0x72/0x310 [ 60.467703][ T72] kernel_init+0x20/0x200 [ 60.467881][ T72] ret_from_fork+0x31/0x70 [ 60.468200][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.468375][ T72] SOFTIRQ-ON-W at: [ 60.468484][ T72] __lock_acquire+0x797/0x1570 [ 60.468743][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.468923][ T72] _raw_spin_lock+0x30/0x40 [ 60.469097][ T72] set_mems_allowed+0x1d/0x210 [ 60.469269][ T72] kernel_init_freeable+0x72/0x310 [ 60.469447][ T72] kernel_init+0x20/0x200 [ 60.469628][ T72] ret_from_fork+0x31/0x70 [ 60.469802][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.469975][ T72] INITIAL USE at: [ 60.470079][ T72] __lock_acquire+0x797/0x1570 [ 60.470256][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.470439][ T72] _raw_spin_lock+0x30/0x40 [ 60.470690][ T72] set_mems_allowed+0x1d/0x210 [ 60.470864][ T72] kernel_init_freeable+0x72/0x310 [ 60.471036][ T72] kernel_init+0x20/0x200 [ 60.471285][ T72] ret_from_fork+0x31/0x70 [ 60.471461][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.471636][ T72] } [ 60.471714][ T72] ... key at: [] __key.388+0x0/0x40 [ 60.472000][ T72] ... acquired at: [ 60.472107][ T72] check_prev_add+0x194/0xce0 [ 60.472264][ T72] validate_chain+0x71d/0x9b0 [ 60.472410][ T72] __lock_acquire+0xaf0/0x1570 [ 60.472556][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.472794][ T72] _raw_spin_lock+0x30/0x40 [ 60.472937][ T72] __get_task_comm+0x27/0x70 [ 60.473082][ T72] ref_tracker_alloc+0x2ee/0x490 [ 60.473225][ T72] rt_flush_dev+0x371/0x670 [ 60.473368][ T72] fib_netdev_event+0x248/0x5a0 [ 60.473511][ T72] notifier_call_chain+0xcd/0x150 [ 60.473648][ T72] unregister_netdevice_many_notify+0x548/0x1190 [ 60.473895][ T72] default_device_exit_batch+0x234/0x2d0 [ 60.474093][ T72] cleanup_net+0x4f3/0xb60 [ 60.474231][ T72] process_one_work+0xe2c/0x1730 [ 60.474370][ T72] worker_thread+0x587/0xd30 [ 60.474581][ T72] kthread+0x28a/0x350 [ 60.474770][ T72] ret_from_fork+0x31/0x70 [ 60.474909][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.475050][ T72] [ 60.475118][ T72] [ 60.475118][ T72] stack backtrace: [ 60.475293][ T72] CPU: 0 PID: 72 Comm: kworker/u16:1 Not tainted 6.9.0-rc1-virtme #1 [ 60.475503][ T72] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 60.475896][ T72] Workqueue: netns cleanup_net [ 60.476043][ T72] Call Trace: [ 60.476155][ T72] [ 60.476230][ T72] dump_stack_lvl+0x82/0xd0 [ 60.476464][ T72] print_bad_irq_dependency+0x957/0xd40 [ 60.476607][ T72] ? __pfx_print_bad_irq_dependency+0x10/0x10 [ 60.476796][ T72] ? __pfx_usage_skip+0x10/0x10 [ 60.476936][ T72] ? __bfs+0x246/0x650 [ 60.477042][ T72] ? __pfx_usage_match+0x10/0x10 [ 60.477183][ T72] check_irq_usage+0x596/0x850 [ 60.477403][ T72] ? arch_stack_walk+0x8c/0xf0 [ 60.477548][ T72] ? __pfx_check_irq_usage+0x10/0x10 [ 60.477692][ T72] ? __bfs+0x246/0x650 [ 60.477794][ T72] ? __pfx_hlock_conflict+0x10/0x10 [ 60.478016][ T72] ? check_path.constprop.0+0x24/0x50 [ 60.478248][ T72] ? check_noncircular+0x14c/0x3e0 [ 60.478389][ T72] ? __pfx_check_noncircular+0x10/0x10 [ 60.478532][ T72] ? find_held_lock+0x2c/0x110 [ 60.478672][ T72] ? check_prev_add+0x194/0xce0 [ 60.478897][ T72] check_prev_add+0x194/0xce0 [ 60.479051][ T72] ? validate_chain+0x130/0x9b0 [ 60.479196][ T72] validate_chain+0x71d/0x9b0 [ 60.479339][ T72] ? mark_lock+0x38/0x3e0 [ 60.479434][ T72] ? __pfx_validate_chain+0x10/0x10 [ 60.479646][ T72] ? hlock_class+0x4e/0x130 [ 60.479786][ T72] ? mark_lock+0x38/0x3e0 [ 60.479895][ T72] __lock_acquire+0xaf0/0x1570 [ 60.480113][ T72] lock_acquire.part.0+0xeb/0x330 [ 60.480327][ T72] ? __get_task_comm+0x27/0x70 [ 60.480464][ T72] ? mark_lock+0x38/0x3e0 [ 60.480574][ T72] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 60.480715][ T72] ? trace_lock_acquire+0x135/0x1c0 [ 60.480859][ T72] ? __get_task_comm+0x27/0x70 [ 60.481072][ T72] ? lock_acquire+0x32/0xc0 [ 60.481212][ T72] ? __get_task_comm+0x27/0x70 [ 60.481351][ T72] _raw_spin_lock+0x30/0x40 [ 60.481570][ T72] ? __get_task_comm+0x27/0x70 [ 60.481711][ T72] __get_task_comm+0x27/0x70 [ 60.481933][ T72] ref_tracker_alloc+0x2ee/0x490 [ 60.482072][ T72] ? lock_acquire.part.0+0xeb/0x330 [ 60.482214][ T72] ? __pfx_ref_tracker_alloc+0x10/0x10 [ 60.482357][ T72] ? rt_flush_dev+0x371/0x670 [ 60.482502][ T72] ? fib_netdev_event+0x248/0x5a0 [ 60.482736][ T72] ? notifier_call_chain+0xcd/0x150 [ 60.482876][ T72] ? unregister_netdevice_many_notify+0x548/0x1190 [ 60.483046][ T72] ? default_device_exit_batch+0x234/0x2d0 [ 60.483221][ T72] ? cleanup_net+0x4f3/0xb60 [ 60.483365][ T72] ? process_one_work+0xe2c/0x1730 [ 60.483505][ T72] ? worker_thread+0x587/0xd30 [ 60.483644][ T72] ? kthread+0x28a/0x350 [ 60.483749][ T72] ? ret_from_fork+0x31/0x70 [ 60.483886][ T72] ? ret_from_fork_asm+0x1a/0x30 [ 60.484025][ T72] ? do_raw_spin_lock+0x131/0x270 [ 60.484170][ T72] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 60.484393][ T72] ? lock_acquire+0x32/0xc0 [ 60.484533][ T72] ? rt_flush_dev+0x13b/0x670 [ 60.484676][ T72] rt_flush_dev+0x371/0x670 [ 60.484818][ T72] ? __timer_delete_sync+0xd3/0x110 [ 60.485044][ T72] ? __timer_delete_sync+0xa7/0x110 [ 60.485185][ T72] fib_netdev_event+0x248/0x5a0 [ 60.485324][ T72] notifier_call_chain+0xcd/0x150 [ 60.485465][ T72] unregister_netdevice_many_notify+0x548/0x1190 [ 60.485735][ T72] ? mutex_is_locked+0x17/0x50 [ 60.485876][ T72] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 60.486049][ T72] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 60.486224][ T72] default_device_exit_batch+0x234/0x2d0 [ 60.486362][ T72] ? __pfx_default_device_exit_batch+0x10/0x10 [ 60.486532][ T72] ? ops_exit_list+0xb4/0x170 [ 60.486672][ T72] cleanup_net+0x4f3/0xb60 [ 60.486837][ T72] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 60.487049][ T72] ? __pfx_cleanup_net+0x10/0x10 [ 60.487186][ T72] ? trace_lock_acquire+0x135/0x1c0 [ 60.487323][ T72] ? process_one_work+0xde2/0x1730 [ 60.487459][ T72] ? lock_acquire+0x32/0xc0 [ 60.487669][ T72] ? process_one_work+0xde2/0x1730 [ 60.487807][ T72] process_one_work+0xe2c/0x1730 [ 60.487957][ T72] ? __pfx___lock_release+0x10/0x10 [ 60.488099][ T72] ? __pfx_process_one_work+0x10/0x10 [ 60.488310][ T72] ? assign_work+0x16c/0x240 [ 60.488447][ T72] worker_thread+0x587/0xd30 [ 60.488585][ T72] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 60.488755][ T72] ? __pfx_worker_thread+0x10/0x10 [ 60.488963][ T72] ? __pfx_worker_thread+0x10/0x10 [ 60.489100][ T72] kthread+0x28a/0x350 [ 60.489205][ T72] ? __pfx_kthread+0x10/0x10 [ 60.489344][ T72] ret_from_fork+0x31/0x70 [ 60.489482][ T72] ? __pfx_kthread+0x10/0x10 [ 60.489687][ T72] ret_from_fork_asm+0x1a/0x30 [ 60.489826][ T72]