[ 24.277212][ T237] gre: GRE over IPv4 demultiplexor driver [ 26.097019][ T261] ip_gre: GRE over IPv4 tunneling driver [ 27.947274][ T275] ping (275) used greatest stack depth: 24368 bytes left [ 30.858276][ T320] ip (320) used greatest stack depth: 23808 bytes left [ 80.133304][ T835] ip6_gre: GRE over IPv6 tunneling driver [ 111.968249][ T1178] ping (1178) used greatest stack depth: 23632 bytes left [ 117.354863][ C1] ip6_tunnel: tep0 xmit: Local address not yet configured! [ 122.601804][ C0] ip6_tunnel: tep0 xmit: Local address not yet configured! [ 127.017796][ C0] ip6_tunnel: tep0 xmit: Local address not yet configured! [ 132.266794][ C1] ip6_tunnel: tep0 xmit: Local address not yet configured! [ 135.687726][ T70] [ 135.687896][ T70] ===================================================== [ 135.688199][ T70] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected [ 135.688394][ T70] 6.9.0-rc1-virtme #1 Not tainted [ 135.688523][ T70] ----------------------------------------------------- [ 135.688761][ T70] kworker/u16:1/70 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 135.688952][ T70] ffff888004e18968 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70 [ 135.689194][ T70] [ 135.689194][ T70] and this task is already holding: [ 135.689381][ T70] ffff88802f73f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 135.689618][ T70] which would create a new lock dependency: [ 135.689791][ T70] (&ul->lock#2){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2} [ 135.689988][ T70] [ 135.689988][ T70] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 135.690214][ T70] (&ul->lock#2){+.-.}-{2:2} [ 135.690219][ T70] [ 135.690219][ T70] ... which became SOFTIRQ-irq-safe at: [ 135.690601][ T70] __lock_acquire+0x797/0x1570 [ 135.690733][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.690859][ T70] _raw_spin_lock_bh+0x38/0x50 [ 135.690986][ T70] rt_del_uncached_list+0x77/0x200 [ 135.691189][ T70] dst_destroy+0xbc/0x370 [ 135.691317][ T70] rcu_do_batch+0x3c0/0xfb0 [ 135.691448][ T70] rcu_core+0x2be/0x500 [ 135.691545][ T70] __do_softirq+0x1f8/0x5df [ 135.691753][ T70] irq_exit_rcu+0x97/0xc0 [ 135.691881][ T70] sysvec_apic_timer_interrupt+0x75/0x80 [ 135.692007][ T70] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 135.692162][ T70] default_idle+0xf/0x20 [ 135.692259][ T70] default_idle_call+0x6d/0xb0 [ 135.692382][ T70] cpuidle_idle_call+0x1f4/0x280 [ 135.692509][ T70] do_idle+0xf9/0x160 [ 135.692607][ T70] cpu_startup_entry+0x54/0x60 [ 135.692733][ T70] start_secondary+0x21c/0x2b0 [ 135.692859][ T70] common_startup_64+0x12c/0x138 [ 135.692986][ T70] [ 135.692986][ T70] to a SOFTIRQ-irq-unsafe lock: [ 135.693141][ T70] (&p->alloc_lock){+.+.}-{2:2} [ 135.693145][ T70] [ 135.693145][ T70] ... which became SOFTIRQ-irq-unsafe at: [ 135.693522][ T70] ... [ 135.693524][ T70] __lock_acquire+0x797/0x1570 [ 135.693718][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.693845][ T70] _raw_spin_lock+0x30/0x40 [ 135.694046][ T70] set_mems_allowed+0x1d/0x210 [ 135.694176][ T70] kernel_init_freeable+0x72/0x310 [ 135.694306][ T70] kernel_init+0x20/0x200 [ 135.694434][ T70] ret_from_fork+0x31/0x70 [ 135.694563][ T70] ret_from_fork_asm+0x1a/0x30 [ 135.694762][ T70] [ 135.694762][ T70] other info that might help us debug this: [ 135.694762][ T70] [ 135.695010][ T70] Possible interrupt unsafe locking scenario: [ 135.695010][ T70] [ 135.695268][ T70] CPU0 CPU1 [ 135.695393][ T70] ---- ---- [ 135.695520][ T70] lock(&p->alloc_lock); [ 135.695617][ T70] local_irq_disable(); [ 135.695774][ T70] lock(&ul->lock#2); [ 135.696009][ T70] lock(&p->alloc_lock); [ 135.696167][ T70] [ 135.696264][ T70] lock(&ul->lock#2); [ 135.696362][ T70] [ 135.696362][ T70] *** DEADLOCK *** [ 135.696362][ T70] [ 135.696619][ T70] 5 locks held by kworker/u16:1/70: [ 135.696745][ T70] #0: ffff88800190a948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730 [ 135.697012][ T70] #1: ffffc900004bfda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730 [ 135.697308][ T70] #2: ffffffff96d9bc50 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xb60 [ 135.697533][ T70] #3: ffffffff96db7208 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0 [ 135.697946][ T70] #4: ffff88802f73f198 (&ul->lock#2){+.-.}-{2:2}, at: rt_flush_dev+0x13b/0x670 [ 135.698175][ T70] [ 135.698175][ T70] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 135.698420][ T70] -> (&ul->lock#2){+.-.}-{2:2} { [ 135.698553][ T70] HARDIRQ-ON-W at: [ 135.698649][ T70] __lock_acquire+0x797/0x1570 [ 135.698809][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.698971][ T70] _raw_spin_lock_bh+0x38/0x50 [ 135.699129][ T70] rt_add_uncached_list+0x55/0x1b0 [ 135.699293][ T70] rt_cache_route+0xfd/0x150 [ 135.699524][ T70] rt_set_nexthop.constprop.0+0x283/0xe90 [ 135.699713][ T70] __mkroute_output+0x4fa/0x1210 [ 135.699874][ T70] ip_route_output_key_hash+0x15d/0x2b0 [ 135.700136][ T70] ip_route_output_flow+0x23/0x140 [ 135.700304][ T70] ip_tunnel_xmit+0x7b7/0x2890 [ 135.700466][ T70] __gre_xmit+0x4af/0xa40 [ip_gre] [ 135.700638][ T70] gre_tap_xmit+0x431/0x580 [ip_gre] [ 135.700885][ T70] dev_hard_start_xmit+0x10e/0x360 [ 135.701045][ T70] sch_direct_xmit+0x203/0x11c0 [ 135.701206][ T70] __dev_xmit_skb+0x936/0xfa0 [ 135.701365][ T70] __dev_queue_xmit+0x8be/0x16e0 [ 135.701527][ T70] ip6_finish_output2+0x512/0xf60 [ 135.701686][ T70] ip6_finish_output+0x553/0xdf0 [ 135.701847][ T70] ip6_output+0x1f3/0x770 [ 135.702005][ T70] NF_HOOK.constprop.0+0xe2/0x680 [ 135.702165][ T70] mld_sendpack+0x62d/0xbc0 [ 135.702324][ T70] mld_ifc_work+0x32/0x1f0 [ 135.702558][ T70] process_one_work+0xe2c/0x1730 [ 135.702718][ T70] worker_thread+0x587/0xd30 [ 135.702879][ T70] kthread+0x28a/0x350 [ 135.703009][ T70] ret_from_fork+0x31/0x70 [ 135.703245][ T70] ret_from_fork_asm+0x1a/0x30 [ 135.703404][ T70] IN-SOFTIRQ-W at: [ 135.703499][ T70] __lock_acquire+0x797/0x1570 [ 135.703748][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.703911][ T70] _raw_spin_lock_bh+0x38/0x50 [ 135.704069][ T70] rt_del_uncached_list+0x77/0x200 [ 135.704229][ T70] dst_destroy+0xbc/0x370 [ 135.704388][ T70] rcu_do_batch+0x3c0/0xfb0 [ 135.704548][ T70] rcu_core+0x2be/0x500 [ 135.704711][ T70] __do_softirq+0x1f8/0x5df [ 135.704871][ T70] irq_exit_rcu+0x97/0xc0 [ 135.705031][ T70] sysvec_apic_timer_interrupt+0x75/0x80 [ 135.705222][ T70] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 135.705484][ T70] default_idle+0xf/0x20 [ 135.705644][ T70] default_idle_call+0x6d/0xb0 [ 135.705804][ T70] cpuidle_idle_call+0x1f4/0x280 [ 135.705970][ T70] do_idle+0xf9/0x160 [ 135.706170][ T70] cpu_startup_entry+0x54/0x60 [ 135.706331][ T70] start_secondary+0x21c/0x2b0 [ 135.706491][ T70] common_startup_64+0x12c/0x138 [ 135.706722][ T70] INITIAL USE at: [ 135.706817][ T70] __lock_acquire+0x797/0x1570 [ 135.706977][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.707136][ T70] _raw_spin_lock_bh+0x38/0x50 [ 135.707440][ T70] rt_add_uncached_list+0x55/0x1b0 [ 135.707598][ T70] rt_cache_route+0xfd/0x150 [ 135.707759][ T70] rt_set_nexthop.constprop.0+0x283/0xe90 [ 135.708022][ T70] __mkroute_output+0x4fa/0x1210 [ 135.708185][ T70] ip_route_output_key_hash+0x15d/0x2b0 [ 135.708344][ T70] ip_route_output_flow+0x23/0x140 [ 135.708504][ T70] ip_tunnel_xmit+0x7b7/0x2890 [ 135.708733][ T70] __gre_xmit+0x4af/0xa40 [ip_gre] [ 135.708895][ T70] gre_tap_xmit+0x431/0x580 [ip_gre] [ 135.709057][ T70] dev_hard_start_xmit+0x10e/0x360 [ 135.709294][ T70] sch_direct_xmit+0x203/0x11c0 [ 135.709453][ T70] __dev_xmit_skb+0x936/0xfa0 [ 135.709614][ T70] __dev_queue_xmit+0x8be/0x16e0 [ 135.709844][ T70] ip6_finish_output2+0x512/0xf60 [ 135.710004][ T70] ip6_finish_output+0x553/0xdf0 [ 135.710175][ T70] ip6_output+0x1f3/0x770 [ 135.710340][ T70] NF_HOOK.constprop.0+0xe2/0x680 [ 135.710570][ T70] mld_sendpack+0x62d/0xbc0 [ 135.710729][ T70] mld_ifc_work+0x32/0x1f0 [ 135.710888][ T70] process_one_work+0xe2c/0x1730 [ 135.711121][ T70] worker_thread+0x587/0xd30 [ 135.711280][ T70] kthread+0x28a/0x350 [ 135.711407][ T70] ret_from_fork+0x31/0x70 [ 135.711574][ T70] ret_from_fork_asm+0x1a/0x30 [ 135.711814][ T70] } [ 135.711880][ T70] ... key at: [] __key.7+0x0/0x40 [ 135.712044][ T70] [ 135.712044][ T70] the dependencies between the lock to be acquired [ 135.712046][ T70] and SOFTIRQ-irq-unsafe lock: [ 135.712387][ T70] -> (&p->alloc_lock){+.+.}-{2:2} { [ 135.712521][ T70] HARDIRQ-ON-W at: [ 135.712622][ T70] __lock_acquire+0x797/0x1570 [ 135.712789][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.712950][ T70] _raw_spin_lock+0x30/0x40 [ 135.713115][ T70] set_mems_allowed+0x1d/0x210 [ 135.713277][ T70] kernel_init_freeable+0x72/0x310 [ 135.713514][ T70] kernel_init+0x20/0x200 [ 135.713677][ T70] ret_from_fork+0x31/0x70 [ 135.713842][ T70] ret_from_fork_asm+0x1a/0x30 [ 135.714075][ T70] SOFTIRQ-ON-W at: [ 135.714171][ T70] __lock_acquire+0x797/0x1570 [ 135.714331][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.714491][ T70] _raw_spin_lock+0x30/0x40 [ 135.714727][ T70] set_mems_allowed+0x1d/0x210 [ 135.714895][ T70] kernel_init_freeable+0x72/0x310 [ 135.715056][ T70] kernel_init+0x20/0x200 [ 135.715215][ T70] ret_from_fork+0x31/0x70 [ 135.715374][ T70] ret_from_fork_asm+0x1a/0x30 [ 135.715532][ T70] INITIAL USE at: [ 135.715628][ T70] __lock_acquire+0x797/0x1570 [ 135.715790][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.715950][ T70] _raw_spin_lock+0x30/0x40 [ 135.716108][ T70] set_mems_allowed+0x1d/0x210 [ 135.716266][ T70] kernel_init_freeable+0x72/0x310 [ 135.716507][ T70] kernel_init+0x20/0x200 [ 135.716667][ T70] ret_from_fork+0x31/0x70 [ 135.716826][ T70] ret_from_fork_asm+0x1a/0x30 [ 135.717130][ T70] } [ 135.717196][ T70] ... key at: [] __key.388+0x0/0x40 [ 135.717390][ T70] ... acquired at: [ 135.717486][ T70] check_prev_add+0x194/0xce0 [ 135.717620][ T70] validate_chain+0x71d/0x9b0 [ 135.717827][ T70] __lock_acquire+0xaf0/0x1570 [ 135.717953][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.718079][ T70] _raw_spin_lock+0x30/0x40 [ 135.718205][ T70] __get_task_comm+0x27/0x70 [ 135.718332][ T70] ref_tracker_alloc+0x2ee/0x490 [ 135.718467][ T70] rt_flush_dev+0x371/0x670 [ 135.718593][ T70] fib_netdev_event+0x248/0x5a0 [ 135.718720][ T70] notifier_call_chain+0xcd/0x150 [ 135.718848][ T70] unregister_netdevice_many_notify+0x548/0x1190 [ 135.719005][ T70] default_device_exit_batch+0x234/0x2d0 [ 135.719161][ T70] cleanup_net+0x4f3/0xb60 [ 135.719291][ T70] process_one_work+0xe2c/0x1730 [ 135.719491][ T70] worker_thread+0x587/0xd30 [ 135.719618][ T70] kthread+0x28a/0x350 [ 135.719713][ T70] ret_from_fork+0x31/0x70 [ 135.719841][ T70] ret_from_fork_asm+0x1a/0x30 [ 135.720052][ T70] [ 135.720124][ T70] [ 135.720124][ T70] stack backtrace: [ 135.720303][ T70] CPU: 2 PID: 70 Comm: kworker/u16:1 Not tainted 6.9.0-rc1-virtme #1 [ 135.720512][ T70] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 135.720904][ T70] Workqueue: netns cleanup_net [ 135.721048][ T70] Call Trace: [ 135.721160][ T70] [ 135.721233][ T70] dump_stack_lvl+0x82/0xd0 [ 135.721378][ T70] print_bad_irq_dependency+0x957/0xd40 [ 135.721518][ T70] ? __pfx_print_bad_irq_dependency+0x10/0x10 [ 135.721693][ T70] ? __pfx_usage_skip+0x10/0x10 [ 135.721834][ T70] ? __bfs+0x246/0x650 [ 135.721943][ T70] ? __pfx_usage_match+0x10/0x10 [ 135.722088][ T70] check_irq_usage+0x596/0x850 [ 135.722227][ T70] ? arch_stack_walk+0x8c/0xf0 [ 135.722369][ T70] ? __pfx_check_irq_usage+0x10/0x10 [ 135.722510][ T70] ? __bfs+0x246/0x650 [ 135.722683][ T70] ? __pfx_hlock_conflict+0x10/0x10 [ 135.722810][ T70] ? check_path.constprop.0+0x24/0x50 [ 135.722939][ T70] ? check_noncircular+0x14c/0x3e0 [ 135.723069][ T70] ? __pfx_check_noncircular+0x10/0x10 [ 135.723265][ T70] ? find_held_lock+0x2c/0x110 [ 135.723393][ T70] ? check_prev_add+0x194/0xce0 [ 135.723518][ T70] check_prev_add+0x194/0xce0 [ 135.723644][ T70] ? validate_chain+0x130/0x9b0 [ 135.723841][ T70] validate_chain+0x71d/0x9b0 [ 135.723970][ T70] ? mark_lock+0x38/0x3e0 [ 135.724065][ T70] ? __pfx_validate_chain+0x10/0x10 [ 135.724192][ T70] ? hlock_class+0x4e/0x130 [ 135.724322][ T70] ? mark_lock+0x38/0x3e0 [ 135.724490][ T70] __lock_acquire+0xaf0/0x1570 [ 135.724619][ T70] lock_acquire.part.0+0xeb/0x330 [ 135.724748][ T70] ? __get_task_comm+0x27/0x70 [ 135.724873][ T70] ? mark_lock+0x38/0x3e0 [ 135.724970][ T70] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 135.725166][ T70] ? trace_lock_acquire+0x135/0x1c0 [ 135.725292][ T70] ? __get_task_comm+0x27/0x70 [ 135.725421][ T70] ? lock_acquire+0x32/0xc0 [ 135.725546][ T70] ? __get_task_comm+0x27/0x70 [ 135.725744][ T70] _raw_spin_lock+0x30/0x40 [ 135.725870][ T70] ? __get_task_comm+0x27/0x70 [ 135.725995][ T70] __get_task_comm+0x27/0x70 [ 135.726121][ T70] ref_tracker_alloc+0x2ee/0x490 [ 135.726320][ T70] ? lock_acquire.part.0+0xeb/0x330 [ 135.726446][ T70] ? __pfx_ref_tracker_alloc+0x10/0x10 [ 135.726573][ T70] ? rt_flush_dev+0x371/0x670 [ 135.726701][ T70] ? fib_netdev_event+0x248/0x5a0 [ 135.726965][ T70] ? notifier_call_chain+0xcd/0x150 [ 135.727090][ T70] ? unregister_netdevice_many_notify+0x548/0x1190 [ 135.727245][ T70] ? default_device_exit_batch+0x234/0x2d0 [ 135.727400][ T70] ? cleanup_net+0x4f3/0xb60 [ 135.727607][ T70] ? process_one_work+0xe2c/0x1730 [ 135.727733][ T70] ? worker_thread+0x587/0xd30 [ 135.727858][ T70] ? kthread+0x28a/0x350 [ 135.727956][ T70] ? ret_from_fork+0x31/0x70 [ 135.728081][ T70] ? ret_from_fork_asm+0x1a/0x30 [ 135.728277][ T70] ? do_raw_spin_lock+0x131/0x270 [ 135.728405][ T70] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 135.728531][ T70] ? lock_acquire+0x32/0xc0 [ 135.728656][ T70] ? rt_flush_dev+0x13b/0x670 [ 135.728862][ T70] rt_flush_dev+0x371/0x670 [ 135.728988][ T70] ? __timer_delete_sync+0xd3/0x110 [ 135.729117][ T70] ? __timer_delete_sync+0xa7/0x110 [ 135.729244][ T70] fib_netdev_event+0x248/0x5a0 [ 135.729373][ T70] notifier_call_chain+0xcd/0x150 [ 135.729500][ T70] unregister_netdevice_many_notify+0x548/0x1190 [ 135.729656][ T70] ? mutex_is_locked+0x17/0x50 [ 135.729783][ T70] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 135.729939][ T70] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 135.730099][ T70] default_device_exit_batch+0x234/0x2d0 [ 135.730236][ T70] ? __pfx_default_device_exit_batch+0x10/0x10 [ 135.730392][ T70] ? ops_exit_list+0xb4/0x170 [ 135.730593][ T70] cleanup_net+0x4f3/0xb60 [ 135.730718][ T70] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 135.730845][ T70] ? __pfx_cleanup_net+0x10/0x10 [ 135.730971][ T70] ? trace_lock_acquire+0x135/0x1c0 [ 135.731168][ T70] ? process_one_work+0xde2/0x1730 [ 135.731294][ T70] ? lock_acquire+0x32/0xc0 [ 135.731421][ T70] ? process_one_work+0xde2/0x1730 [ 135.731549][ T70] process_one_work+0xe2c/0x1730 [ 135.731753][ T70] ? __pfx___lock_release+0x10/0x10 [ 135.731881][ T70] ? __pfx_process_one_work+0x10/0x10 [ 135.732009][ T70] ? assign_work+0x16c/0x240 [ 135.732135][ T70] worker_thread+0x587/0xd30 [ 135.732261][ T70] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 135.732421][ T70] ? __pfx_worker_thread+0x10/0x10 [ 135.732547][ T70] ? __pfx_worker_thread+0x10/0x10 [ 135.732677][ T70] kthread+0x28a/0x350 [ 135.732774][ T70] ? __pfx_kthread+0x10/0x10 [ 135.732900][ T70] ret_from_fork+0x31/0x70 [ 135.733026][ T70] ? __pfx_kthread+0x10/0x10 [ 135.733152][ T70] ret_from_fork_asm+0x1a/0x30 [ 135.733280][ T70]