[ 23.675958][ T241] gre: GRE over IPv4 demultiplexor driver [ 25.449757][ T264] ip_gre: GRE over IPv4 tunneling driver [ 45.338503][ T504] ip (504) used greatest stack depth: 23760 bytes left [ 75.590811][ T836] ip6_gre: GRE over IPv6 tunneling driver [ 97.494105][ T50] ip6_tunnel: tep1 xmit: Local address not yet configured! [ 110.293971][ C0] ip6_tunnel: tep0 xmit: Local address not yet configured! [ 115.413959][ C2] ip6_tunnel: tep0 xmit: Local address not yet configured! [ 119.957988][ C2] ip6_tunnel: tep0 xmit: Local address not yet configured! [ 124.949960][ C2] ip6_tunnel: tep0 xmit: Local address not yet configured! [ 148.587821][ T72] [ 148.587923][ T72] ===================================================== [ 148.588088][ T72] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected [ 148.588276][ T72] 6.9.0-rc1-virtme #1 Not tainted [ 148.588402][ T72] ----------------------------------------------------- [ 148.588557][ T72] kworker/u16:1/72 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 148.588761][ T72] ffff8880055acee8 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70 [ 148.588993][ T72] [ 148.588993][ T72] and this task is already holding: [ 148.589180][ T72] ffff88803613f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 148.589452][ T72] which would create a new lock dependency: [ 148.589607][ T72] (&ul->lock#2){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2} [ 148.589801][ T72] [ 148.589801][ T72] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 148.590021][ T72] (&ul->lock#2){+.-.}-{2:2} [ 148.590025][ T72] [ 148.590025][ T72] ... which became SOFTIRQ-irq-safe at: [ 148.590339][ T72] __lock_acquire+0x797/0x1570 [ 148.590472][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.590598][ T72] _raw_spin_lock_bh+0x38/0x50 [ 148.590727][ T72] rt_del_uncached_list+0x77/0x200 [ 148.590859][ T72] dst_destroy+0xbc/0x370 [ 148.590988][ T72] rcu_do_batch+0x3c0/0xfb0 [ 148.591115][ T72] rcu_core+0x2be/0x500 [ 148.591210][ T72] __do_softirq+0x1f8/0x5df [ 148.591338][ T72] irq_exit_rcu+0x97/0xc0 [ 148.591463][ T72] sysvec_apic_timer_interrupt+0x75/0x80 [ 148.591599][ T72] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 148.591755][ T72] default_idle+0xf/0x20 [ 148.591850][ T72] default_idle_call+0x6d/0xb0 [ 148.591977][ T72] cpuidle_idle_call+0x1f4/0x280 [ 148.592106][ T72] do_idle+0xf9/0x160 [ 148.592200][ T72] cpu_startup_entry+0x54/0x60 [ 148.592327][ T72] start_secondary+0x21c/0x2b0 [ 148.592454][ T72] common_startup_64+0x12c/0x138 [ 148.592582][ T72] [ 148.592582][ T72] to a SOFTIRQ-irq-unsafe lock: [ 148.592736][ T72] (&p->alloc_lock){+.+.}-{2:2} [ 148.592741][ T72] [ 148.592741][ T72] ... which became SOFTIRQ-irq-unsafe at: [ 148.593050][ T72] ... [ 148.593052][ T72] __lock_acquire+0x797/0x1570 [ 148.593242][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.593368][ T72] _raw_spin_lock+0x30/0x40 [ 148.593493][ T72] set_mems_allowed+0x1d/0x210 [ 148.593628][ T72] kernel_init_freeable+0x72/0x310 [ 148.593755][ T72] kernel_init+0x20/0x200 [ 148.593881][ T72] ret_from_fork+0x31/0x70 [ 148.594006][ T72] ret_from_fork_asm+0x1a/0x30 [ 148.594134][ T72] [ 148.594134][ T72] other info that might help us debug this: [ 148.594134][ T72] [ 148.594387][ T72] Possible interrupt unsafe locking scenario: [ 148.594387][ T72] [ 148.594574][ T72] CPU0 CPU1 [ 148.594698][ T72] ---- ---- [ 148.594824][ T72] lock(&p->alloc_lock); [ 148.594921][ T72] local_irq_disable(); [ 148.595084][ T72] lock(&ul->lock#2); [ 148.595243][ T72] lock(&p->alloc_lock); [ 148.595401][ T72] [ 148.595496][ T72] lock(&ul->lock#2); [ 148.595594][ T72] [ 148.595594][ T72] *** DEADLOCK *** [ 148.595594][ T72] [ 148.595778][ T72] 5 locks held by kworker/u16:1/72: [ 148.595905][ T72] #0: ffff88800190a948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 148.596176][ T72] #1: ffffc900004dfda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 148.596400][ T72] #2: ffffffffb479bc50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xb60 [ 148.596626][ T72] #3: ffffffffb47b7208 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0 [ 148.596879][ T72] #4: ffff88803613f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 148.597109][ T72] [ 148.597109][ T72] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 148.597351][ T72] -> (&ul->lock#2){+.-.}-{2:2} { [ 148.597482][ T72] HARDIRQ-ON-W at: [ 148.597581][ T72] __lock_acquire+0x797/0x1570 [ 148.597741][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.597901][ T72] _raw_spin_lock_bh+0x38/0x50 [ 148.598065][ T72] rt_add_uncached_list+0x55/0x1b0 [ 148.598223][ T72] rt_cache_route+0xfd/0x150 [ 148.598383][ T72] rt_set_nexthop.constprop.0+0x283/0xe90 [ 148.598575][ T72] __mkroute_output+0x4fa/0x1210 [ 148.598735][ T72] ip_route_output_key_hash+0x15d/0x2b0 [ 148.598924][ T72] ip_route_output_flow+0x23/0x140 [ 148.599087][ T72] ip_tunnel_xmit+0x7b7/0x2890 [ 148.599249][ T72] __gre_xmit+0x4af/0xa40 [ip_gre] [ 148.599440][ T72] gre_tap_xmit+0x431/0x580 [ip_gre] [ 148.599603][ T72] dev_hard_start_xmit+0x10e/0x360 [ 148.599764][ T72] sch_direct_xmit+0x203/0x11c0 [ 148.599927][ T72] __dev_xmit_skb+0x936/0xfa0 [ 148.600086][ T72] __dev_queue_xmit+0x8be/0x16e0 [ 148.600246][ T72] ip6_finish_output2+0x512/0xf60 [ 148.600405][ T72] ip6_finish_output+0x553/0xdf0 [ 148.600563][ T72] ip6_output+0x1f3/0x770 [ 148.600722][ T72] NF_HOOK.constprop.0+0xe2/0x680 [ 148.600886][ T72] mld_sendpack+0x62d/0xbc0 [ 148.601046][ T72] mld_ifc_work+0x32/0x1f0 [ 148.601205][ T72] process_one_work+0xe2c/0x1730 [ 148.601364][ T72] worker_thread+0x587/0xd30 [ 148.601524][ T72] kthread+0x28a/0x350 [ 148.601652][ T72] ret_from_fork+0x31/0x70 [ 148.601813][ T72] ret_from_fork_asm+0x1a/0x30 [ 148.601976][ T72] IN-SOFTIRQ-W at: [ 148.602072][ T72] __lock_acquire+0x797/0x1570 [ 148.602232][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.602391][ T72] _raw_spin_lock_bh+0x38/0x50 [ 148.602552][ T72] rt_del_uncached_list+0x77/0x200 [ 148.602711][ T72] dst_destroy+0xbc/0x370 [ 148.602871][ T72] rcu_do_batch+0x3c0/0xfb0 [ 148.603032][ T72] rcu_core+0x2be/0x500 [ 148.603192][ T72] __do_softirq+0x1f8/0x5df [ 148.603352][ T72] irq_exit_rcu+0x97/0xc0 [ 148.603518][ T72] sysvec_apic_timer_interrupt+0x75/0x80 [ 148.603710][ T72] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 148.603901][ T72] default_idle+0xf/0x20 [ 148.604062][ T72] default_idle_call+0x6d/0xb0 [ 148.604223][ T72] cpuidle_idle_call+0x1f4/0x280 [ 148.604388][ T72] do_idle+0xf9/0x160 [ 148.604516][ T72] cpu_startup_entry+0x54/0x60 [ 148.604749][ T72] start_secondary+0x21c/0x2b0 [ 148.604910][ T72] common_startup_64+0x12c/0x138 [ 148.605077][ T72] INITIAL USE at: [ 148.605174][ T72] __lock_acquire+0x797/0x1570 [ 148.605407][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.605566][ T72] _raw_spin_lock_bh+0x38/0x50 [ 148.605731][ T72] rt_add_uncached_list+0x55/0x1b0 [ 148.605893][ T72] rt_cache_route+0xfd/0x150 [ 148.606124][ T72] rt_set_nexthop.constprop.0+0x283/0xe90 [ 148.606317][ T72] __mkroute_output+0x4fa/0x1210 [ 148.606481][ T72] ip_route_output_key_hash+0x15d/0x2b0 [ 148.606789][ T72] ip_route_output_flow+0x23/0x140 [ 148.606948][ T72] ip_tunnel_xmit+0x7b7/0x2890 [ 148.607108][ T72] __gre_xmit+0x4af/0xa40 [ip_gre] [ 148.607351][ T72] gre_tap_xmit+0x431/0x580 [ip_gre] [ 148.607515][ T72] dev_hard_start_xmit+0x10e/0x360 [ 148.607674][ T72] sch_direct_xmit+0x203/0x11c0 [ 148.607833][ T72] __dev_xmit_skb+0x936/0xfa0 [ 148.608082][ T72] __dev_queue_xmit+0x8be/0x16e0 [ 148.608241][ T72] ip6_finish_output2+0x512/0xf60 [ 148.608399][ T72] ip6_finish_output+0x553/0xdf0 [ 148.608559][ T72] ip6_output+0x1f3/0x770 [ 148.608717][ T72] NF_HOOK.constprop.0+0xe2/0x680 [ 148.608884][ T72] mld_sendpack+0x62d/0xbc0 [ 148.609048][ T72] mld_ifc_work+0x32/0x1f0 [ 148.609206][ T72] process_one_work+0xe2c/0x1730 [ 148.609375][ T72] worker_thread+0x587/0xd30 [ 148.609534][ T72] kthread+0x28a/0x350 [ 148.609739][ T72] ret_from_fork+0x31/0x70 [ 148.609906][ T72] ret_from_fork_asm+0x1a/0x30 [ 148.610064][ T72] } [ 148.610130][ T72] ... key at: [] __key.7+0x0/0x40 [ 148.610361][ T72] [ 148.610361][ T72] the dependencies between the lock to be acquired [ 148.610363][ T72] and SOFTIRQ-irq-unsafe lock: [ 148.610706][ T72] -> (&p->alloc_lock){+.+.}-{2:2} { [ 148.610920][ T72] HARDIRQ-ON-W at: [ 148.611018][ T72] __lock_acquire+0x797/0x1570 [ 148.611182][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.611342][ T72] _raw_spin_lock+0x30/0x40 [ 148.611510][ T72] set_mems_allowed+0x1d/0x210 [ 148.611670][ T72] kernel_init_freeable+0x72/0x310 [ 148.611831][ T72] kernel_init+0x20/0x200 [ 148.612007][ T72] ret_from_fork+0x31/0x70 [ 148.612167][ T72] ret_from_fork_asm+0x1a/0x30 [ 148.612327][ T72] SOFTIRQ-ON-W at: [ 148.612423][ T72] __lock_acquire+0x797/0x1570 [ 148.612661][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.612828][ T72] _raw_spin_lock+0x30/0x40 [ 148.612996][ T72] set_mems_allowed+0x1d/0x210 [ 148.613156][ T72] kernel_init_freeable+0x72/0x310 [ 148.613390][ T72] kernel_init+0x20/0x200 [ 148.613549][ T72] ret_from_fork+0x31/0x70 [ 148.613708][ T72] ret_from_fork_asm+0x1a/0x30 [ 148.613954][ T72] INITIAL USE at: [ 148.614051][ T72] __lock_acquire+0x797/0x1570 [ 148.614214][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.614374][ T72] _raw_spin_lock+0x30/0x40 [ 148.614541][ T72] set_mems_allowed+0x1d/0x210 [ 148.614700][ T72] kernel_init_freeable+0x72/0x310 [ 148.614866][ T72] kernel_init+0x20/0x200 [ 148.615026][ T72] ret_from_fork+0x31/0x70 [ 148.615189][ T72] ret_from_fork_asm+0x1a/0x30 [ 148.615350][ T72] } [ 148.615415][ T72] ... key at: [] __key.388+0x0/0x40 [ 148.615683][ T72] ... acquired at: [ 148.615783][ T72] check_prev_add+0x194/0xce0 [ 148.615912][ T72] validate_chain+0x71d/0x9b0 [ 148.616038][ T72] __lock_acquire+0xaf0/0x1570 [ 148.616166][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.616436][ T72] _raw_spin_lock+0x30/0x40 [ 148.616566][ T72] __get_task_comm+0x27/0x70 [ 148.616694][ T72] ref_tracker_alloc+0x2ee/0x490 [ 148.616823][ T72] rt_flush_dev+0x371/0x670 [ 148.617028][ T72] fib_netdev_event+0x248/0x5a0 [ 148.617156][ T72] notifier_call_chain+0xcd/0x150 [ 148.617282][ T72] unregister_netdevice_many_notify+0x548/0x1190 [ 148.617439][ T72] default_device_exit_batch+0x234/0x2d0 [ 148.617596][ T72] cleanup_net+0x4f3/0xb60 [ 148.617723][ T72] process_one_work+0xe2c/0x1730 [ 148.617851][ T72] worker_thread+0x587/0xd30 [ 148.617978][ T72] kthread+0x28a/0x350 [ 148.618074][ T72] ret_from_fork+0x31/0x70 [ 148.618202][ T72] ret_from_fork_asm+0x1a/0x30 [ 148.618328][ T72] [ 148.618393][ T72] [ 148.618393][ T72] stack backtrace: [ 148.618546][ T72] CPU: 2 PID: 72 Comm: kworker/u16:1 Not tainted 6.9.0-rc1-virtme #1 [ 148.618809][ T72] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 148.619088][ T72] Workqueue: netns cleanup_net [ 148.619293][ T72] Call Trace: [ 148.619404][ T72] [ 148.619471][ T72] dump_stack_lvl+0x82/0xd0 [ 148.619602][ T72] print_bad_irq_dependency+0x957/0xd40 [ 148.619731][ T72] ? __pfx_print_bad_irq_dependency+0x10/0x10 [ 148.619969][ T72] ? __pfx_usage_skip+0x10/0x10 [ 148.620095][ T72] ? __bfs+0x246/0x650 [ 148.620191][ T72] ? __pfx_usage_match+0x10/0x10 [ 148.620319][ T72] check_irq_usage+0x596/0x850 [ 148.620448][ T72] ? arch_stack_walk+0x8c/0xf0 [ 148.620576][ T72] ? __pfx_check_irq_usage+0x10/0x10 [ 148.620704][ T72] ? __bfs+0x246/0x650 [ 148.620800][ T72] ? __pfx_hlock_conflict+0x10/0x10 [ 148.620927][ T72] ? check_path.constprop.0+0x24/0x50 [ 148.621064][ T72] ? check_noncircular+0x14c/0x3e0 [ 148.621192][ T72] ? __pfx_check_noncircular+0x10/0x10 [ 148.621319][ T72] ? find_held_lock+0x2c/0x110 [ 148.621448][ T72] ? check_prev_add+0x194/0xce0 [ 148.621648][ T72] check_prev_add+0x194/0xce0 [ 148.621775][ T72] ? validate_chain+0x130/0x9b0 [ 148.621902][ T72] validate_chain+0x71d/0x9b0 [ 148.622039][ T72] ? mark_lock+0x38/0x3e0 [ 148.622138][ T72] ? __pfx_validate_chain+0x10/0x10 [ 148.622337][ T72] ? hlock_class+0x4e/0x130 [ 148.622477][ T72] ? mark_lock+0x38/0x3e0 [ 148.622574][ T72] __lock_acquire+0xaf0/0x1570 [ 148.622705][ T72] lock_acquire.part.0+0xeb/0x330 [ 148.622907][ T72] ? __get_task_comm+0x27/0x70 [ 148.623036][ T72] ? mark_lock+0x38/0x3e0 [ 148.623132][ T72] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 148.623268][ T72] ? trace_lock_acquire+0x135/0x1c0 [ 148.623399][ T72] ? __get_task_comm+0x27/0x70 [ 148.623525][ T72] ? lock_acquire+0x32/0xc0 [ 148.623651][ T72] ? __get_task_comm+0x27/0x70 [ 148.623778][ T72] _raw_spin_lock+0x30/0x40 [ 148.623904][ T72] ? __get_task_comm+0x27/0x70 [ 148.624031][ T72] __get_task_comm+0x27/0x70 [ 148.624157][ T72] ref_tracker_alloc+0x2ee/0x490 [ 148.624283][ T72] ? lock_acquire.part.0+0xeb/0x330 [ 148.624409][ T72] ? __pfx_ref_tracker_alloc+0x10/0x10 [ 148.624608][ T72] ? rt_flush_dev+0x371/0x670 [ 148.624735][ T72] ? fib_netdev_event+0x248/0x5a0 [ 148.624860][ T72] ? notifier_call_chain+0xcd/0x150 [ 148.624993][ T72] ? unregister_netdevice_many_notify+0x548/0x1190 [ 148.625219][ T72] ? default_device_exit_batch+0x234/0x2d0 [ 148.625375][ T72] ? cleanup_net+0x4f3/0xb60 [ 148.625500][ T72] ? process_one_work+0xe2c/0x1730 [ 148.625642][ T72] ? worker_thread+0x587/0xd30 [ 148.625912][ T72] ? kthread+0x28a/0x350 [ 148.626008][ T72] ? ret_from_fork+0x31/0x70 [ 148.626137][ T72] ? ret_from_fork_asm+0x1a/0x30 [ 148.626264][ T72] ? do_raw_spin_lock+0x131/0x270 [ 148.626394][ T72] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 148.626519][ T72] ? lock_acquire+0x32/0xc0 [ 148.626649][ T72] ? rt_flush_dev+0x13b/0x670 [ 148.626776][ T72] rt_flush_dev+0x371/0x670 [ 148.626913][ T72] ? __timer_delete_sync+0xd3/0x110 [ 148.627041][ T72] ? __timer_delete_sync+0xa7/0x110 [ 148.627170][ T72] fib_netdev_event+0x248/0x5a0 [ 148.627295][ T72] notifier_call_chain+0xcd/0x150 [ 148.627423][ T72] unregister_netdevice_many_notify+0x548/0x1190 [ 148.627650][ T72] ? mutex_is_locked+0x17/0x50 [ 148.627781][ T72] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 148.627938][ T72] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 148.628100][ T72] default_device_exit_batch+0x234/0x2d0 [ 148.628304][ T72] ? __pfx_default_device_exit_batch+0x10/0x10 [ 148.628461][ T72] ? ops_exit_list+0xb4/0x170 [ 148.628588][ T72] cleanup_net+0x4f3/0xb60 [ 148.628714][ T72] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 148.628913][ T72] ? __pfx_cleanup_net+0x10/0x10 [ 148.629045][ T72] ? trace_lock_acquire+0x135/0x1c0 [ 148.629174][ T72] ? process_one_work+0xde2/0x1730 [ 148.629308][ T72] ? lock_acquire+0x32/0xc0 [ 148.629521][ T72] ? process_one_work+0xde2/0x1730 [ 148.629648][ T72] process_one_work+0xe2c/0x1730 [ 148.629776][ T72] ? __pfx___lock_release+0x10/0x10 [ 148.629904][ T72] ? __pfx_process_one_work+0x10/0x10 [ 148.630107][ T72] ? assign_work+0x16c/0x240 [ 148.630234][ T72] worker_thread+0x587/0xd30 [ 148.630371][ T72] ? __pfx_worker_thread+0x10/0x10 [ 148.630497][ T72] kthread+0x28a/0x350 [ 148.630592][ T72] ? __pfx_kthread+0x10/0x10 [ 148.630790][ T72] ret_from_fork+0x31/0x70 [ 148.630916][ T72] ? __pfx_kthread+0x10/0x10 [ 148.631041][ T72] ret_from_fork_asm+0x1a/0x30 [ 148.631170][ T72]