[ 71.006515][ T624] gre: GRE over IPv4 demultiplexor driver [ 71.025252][ T624] ip6_gre: GRE over IPv6 tunneling driver [ 71.168880][ T7] ip6_tunnel: er6 xmit: Local address not yet configured! [ 71.402848][ T50] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 71.650603][ T50] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 71.787842][ T10] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 72.837035][ T634] Mirror/redirect action on [ 76.339766][ C2] [ 76.339926][ C2] ======================================================== [ 76.340153][ C2] WARNING: possible irq lock inversion dependency detected [ 76.340372][ C2] 6.9.0-rc1-virtme #1 Not tainted [ 76.340515][ C2] -------------------------------------------------------- [ 76.340730][ C2] swapper/2/0 just changed the state of lock: [ 76.340906][ C2] ffff888009510ed8 (&qdisc_xmit_lock_key#4){+.-.}-{2:2}, at: sch_direct_xmit+0x2a1/0x11c0 [ 76.341175][ C2] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 76.341385][ C2] (&p->alloc_lock){+.+.}-{2:2} [ 76.341389][ C2] [ 76.341389][ C2] [ 76.341389][ C2] and interrupts could create inverse lock ordering between them. [ 76.341389][ C2] [ 76.341929][ C2] [ 76.341929][ C2] other info that might help us debug this: [ 76.342150][ C2] Possible interrupt unsafe locking scenario: [ 76.342150][ C2] [ 76.342372][ C2] CPU0 CPU1 [ 76.342514][ C2] ---- ---- [ 76.342657][ C2] lock(&p->alloc_lock); [ 76.342771][ C2] local_irq_disable(); [ 76.342957][ C2] lock(&qdisc_xmit_lock_key#4); [ 76.343186][ C2] lock(&p->alloc_lock); [ 76.343382][ C2] [ 76.343491][ C2] lock(&qdisc_xmit_lock_key#4); [ 76.343640][ C2] [ 76.343640][ C2] *** DEADLOCK *** [ 76.343640][ C2] [ 76.343853][ C2] 5 locks held by swapper/2/0: [ 76.343996][ C2] #0: ffffc90000230d68 ((&ndev->rs_timer)){+.-.}-{0:0}, at: call_timer_fn+0xe8/0x230 [ 76.344254][ C2] #1: ffffffff96b66360 (rcu_read_lock){....}-{1:2}, at: ndisc_send_skb+0x7c3/0x1230 [ 76.344514][ C2] #2: ffffffff96b66360 (rcu_read_lock){....}-{1:2}, at: ip6_finish_output2+0x32b/0xf60 [ 76.344766][ C2] #3: ffffffff96b66300 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1e5/0x16e0 [ 76.345023][ C2] #4: ffff888009a62258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb+0x792/0xfa0 [ 76.345342][ C2] [ 76.345342][ C2] the shortest dependencies between 2nd lock and 1st lock: [ 76.345590][ C2] -> (&p->alloc_lock){+.+.}-{2:2} { [ 76.345737][ C2] HARDIRQ-ON-W at: [ 76.345848][ C2] __lock_acquire+0x797/0x1570 [ 76.346037][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.346223][ C2] _raw_spin_lock+0x30/0x40 [ 76.346411][ C2] set_mems_allowed+0x1d/0x210 [ 76.346601][ C2] kernel_init_freeable+0x72/0x310 [ 76.346785][ C2] kernel_init+0x20/0x200 [ 76.346974][ C2] ret_from_fork+0x31/0x70 [ 76.347157][ C2] ret_from_fork_asm+0x1a/0x30 [ 76.347342][ C2] SOFTIRQ-ON-W at: [ 76.347452][ C2] __lock_acquire+0x797/0x1570 [ 76.347628][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.347810][ C2] _raw_spin_lock+0x30/0x40 [ 76.347996][ C2] set_mems_allowed+0x1d/0x210 [ 76.348176][ C2] kernel_init_freeable+0x72/0x310 [ 76.348355][ C2] kernel_init+0x20/0x200 [ 76.348538][ C2] ret_from_fork+0x31/0x70 [ 76.348719][ C2] ret_from_fork_asm+0x1a/0x30 [ 76.348899][ C2] INITIAL USE at: [ 76.349012][ C2] __lock_acquire+0x797/0x1570 [ 76.349205][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.349383][ C2] _raw_spin_lock+0x30/0x40 [ 76.349562][ C2] set_mems_allowed+0x1d/0x210 [ 76.349738][ C2] kernel_init_freeable+0x72/0x310 [ 76.349918][ C2] kernel_init+0x20/0x200 [ 76.350098][ C2] ret_from_fork+0x31/0x70 [ 76.350276][ C2] ret_from_fork_asm+0x1a/0x30 [ 76.350454][ C2] } [ 76.350526][ C2] ... key at: [] __key.388+0x0/0x40 [ 76.350743][ C2] ... acquired at: [ 76.350848][ C2] __lock_acquire+0xaf0/0x1570 [ 76.350995][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.351137][ C2] _raw_spin_lock+0x30/0x40 [ 76.351287][ C2] __get_task_comm+0x27/0x70 [ 76.351432][ C2] ref_tracker_alloc+0x2ee/0x490 [ 76.351576][ C2] dst_init+0x84/0x4a0 [ 76.351688][ C2] dst_alloc+0x97/0x150 [ 76.351796][ C2] ip6_dst_alloc+0x23/0x90 [ 76.351943][ C2] ip6_rt_pcpu_alloc+0x1e6/0x520 [ 76.352086][ C2] ip6_pol_route+0x56f/0x840 [ 76.352241][ C2] __fib6_rule_action+0x1f3/0x5b0 [ 76.352389][ C2] fib_rules_lookup+0x2b4/0xa20 [ 76.352537][ C2] fib6_rule_lookup+0x182/0x630 [ 76.352680][ C2] ip6_route_output_flags+0x259/0x480 [ 76.352823][ C2] ip6_tnl_xmit+0x1540/0x3490 [ 76.352968][ C2] ip6erspan_tunnel_xmit+0xf63/0x2ce0 [ip6_gre] [ 76.353153][ C2] dev_hard_start_xmit+0x10e/0x360 [ 76.353297][ C2] sch_direct_xmit+0x203/0x11c0 [ 76.353443][ C2] __dev_xmit_skb+0x936/0xfa0 [ 76.353587][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 76.353734][ C2] ip6_finish_output2+0x59b/0xf60 [ 76.353876][ C2] ip6_finish_output+0x553/0xdf0 [ 76.354019][ C2] ip6_output+0x1f3/0x770 [ 76.354165][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 76.354310][ C2] mld_sendpack+0x62d/0xbc0 [ 76.354452][ C2] mld_ifc_work+0x32/0x1f0 [ 76.354596][ C2] process_one_work+0xe2c/0x1730 [ 76.354749][ C2] worker_thread+0x587/0xd30 [ 76.354893][ C2] kthread+0x28a/0x350 [ 76.355010][ C2] ret_from_fork+0x31/0x70 [ 76.355154][ C2] ret_from_fork_asm+0x1a/0x30 [ 76.355297][ C2] [ 76.355374][ C2] -> (&qdisc_xmit_lock_key#4){+.-.}-{2:2} { [ 76.355557][ C2] HARDIRQ-ON-W at: [ 76.355667][ C2] __lock_acquire+0x797/0x1570 [ 76.355846][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.356033][ C2] _raw_spin_lock+0x30/0x40 [ 76.356214][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 76.356393][ C2] __dev_xmit_skb+0x936/0xfa0 [ 76.356573][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 76.356753][ C2] ip6_finish_output2+0x512/0xf60 [ 76.356931][ C2] ip6_finish_output+0x553/0xdf0 [ 76.357113][ C2] ip6_output+0x1f3/0x770 [ 76.357292][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 76.357469][ C2] mld_sendpack+0x62d/0xbc0 [ 76.357647][ C2] mld_ifc_work+0x32/0x1f0 [ 76.357827][ C2] process_one_work+0xe2c/0x1730 [ 76.358005][ C2] worker_thread+0x587/0xd30 [ 76.358192][ C2] kthread+0x28a/0x350 [ 76.358335][ C2] ret_from_fork+0x31/0x70 [ 76.358515][ C2] ret_from_fork_asm+0x1a/0x30 [ 76.358694][ C2] IN-SOFTIRQ-W at: [ 76.358807][ C2] __lock_acquire+0x797/0x1570 [ 76.358984][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.359190][ C2] _raw_spin_lock+0x30/0x40 [ 76.359370][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 76.359549][ C2] __dev_xmit_skb+0x936/0xfa0 [ 76.359728][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 76.359905][ C2] ip6_finish_output2+0x59b/0xf60 [ 76.360083][ C2] ip6_finish_output+0x553/0xdf0 [ 76.360261][ C2] ip6_output+0x1f3/0x770 [ 76.360444][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 76.360622][ C2] ndisc_send_skb+0x8e0/0x1230 [ 76.360803][ C2] addrconf_rs_timer+0x268/0x730 [ 76.360983][ C2] call_timer_fn+0x13b/0x230 [ 76.361170][ C2] __run_timers+0x545/0x810 [ 76.361348][ C2] run_timer_softirq+0xe8/0x1b0 [ 76.361526][ C2] __do_softirq+0x1f8/0x5df [ 76.361714][ C2] irq_exit_rcu+0x97/0xc0 [ 76.361896][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 76.362119][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 76.362335][ C2] default_idle+0xf/0x20 [ 76.362514][ C2] default_idle_call+0x6d/0xb0 [ 76.362692][ C2] cpuidle_idle_call+0x1f4/0x280 [ 76.362877][ C2] do_idle+0xf9/0x160 [ 76.363021][ C2] cpu_startup_entry+0x54/0x60 [ 76.363200][ C2] start_secondary+0x21c/0x2b0 [ 76.363383][ C2] common_startup_64+0x12c/0x138 [ 76.363563][ C2] INITIAL USE at: [ 76.363672][ C2] __lock_acquire+0x797/0x1570 [ 76.363868][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.364050][ C2] _raw_spin_lock+0x30/0x40 [ 76.364229][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 76.364406][ C2] __dev_xmit_skb+0x936/0xfa0 [ 76.364586][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 76.364765][ C2] ip6_finish_output2+0x512/0xf60 [ 76.364945][ C2] ip6_finish_output+0x553/0xdf0 [ 76.365127][ C2] ip6_output+0x1f3/0x770 [ 76.365309][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 76.365486][ C2] mld_sendpack+0x62d/0xbc0 [ 76.365667][ C2] mld_ifc_work+0x32/0x1f0 [ 76.365847][ C2] process_one_work+0xe2c/0x1730 [ 76.366031][ C2] worker_thread+0x587/0xd30 [ 76.366213][ C2] kthread+0x28a/0x350 [ 76.366364][ C2] ret_from_fork+0x31/0x70 [ 76.366545][ C2] ret_from_fork_asm+0x1a/0x30 [ 76.366725][ C2] } [ 76.366797][ C2] ... key at: [] qdisc_xmit_lock_key.143+0x0/0xffffffffffffb940 [ip6_gre] [ 76.367084][ C2] ... acquired at: [ 76.367197][ C2] mark_lock+0x28d/0x3e0 [ 76.367342][ C2] mark_usage+0xd9/0x2a0 [ 76.367488][ C2] __lock_acquire+0x797/0x1570 [ 76.367630][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.367773][ C2] _raw_spin_lock+0x30/0x40 [ 76.367916][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 76.368060][ C2] __dev_xmit_skb+0x936/0xfa0 [ 76.368204][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 76.368348][ C2] ip6_finish_output2+0x59b/0xf60 [ 76.368489][ C2] ip6_finish_output+0x553/0xdf0 [ 76.368634][ C2] ip6_output+0x1f3/0x770 [ 76.368775][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 76.368919][ C2] ndisc_send_skb+0x8e0/0x1230 [ 76.369068][ C2] addrconf_rs_timer+0x268/0x730 [ 76.369216][ C2] call_timer_fn+0x13b/0x230 [ 76.369359][ C2] __run_timers+0x545/0x810 [ 76.369504][ C2] run_timer_softirq+0xe8/0x1b0 [ 76.369648][ C2] __do_softirq+0x1f8/0x5df [ 76.369797][ C2] irq_exit_rcu+0x97/0xc0 [ 76.369940][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 76.370119][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 76.370300][ C2] default_idle+0xf/0x20 [ 76.370442][ C2] default_idle_call+0x6d/0xb0 [ 76.370593][ C2] cpuidle_idle_call+0x1f4/0x280 [ 76.370736][ C2] do_idle+0xf9/0x160 [ 76.370843][ C2] cpu_startup_entry+0x54/0x60 [ 76.370987][ C2] start_secondary+0x21c/0x2b0 [ 76.371132][ C2] common_startup_64+0x12c/0x138 [ 76.371274][ C2] [ 76.371348][ C2] [ 76.371348][ C2] stack backtrace: [ 76.371522][ C2] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.9.0-rc1-virtme #1 [ 76.371734][ C2] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 76.372054][ C2] Call Trace: [ 76.372170][ C2] [ 76.372248][ C2] dump_stack_lvl+0x82/0xd0 [ 76.372396][ C2] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 76.372575][ C2] ? common_startup_64+0x12c/0x138 [ 76.372719][ C2] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 76.372895][ C2] ? __pfx_usage_skip+0x10/0x10 [ 76.373039][ C2] ? __pfx_usage_match+0x10/0x10 [ 76.373185][ C2] ? arch_stack_walk+0x8c/0xf0 [ 76.373332][ C2] mark_lock_irq+0x4cd/0xa10 [ 76.373482][ C2] ? __pfx_mark_lock_irq+0x10/0x10 [ 76.373626][ C2] ? stack_trace_save+0x94/0xd0 [ 76.373771][ C2] ? __pfx_validate_chain+0x10/0x10 [ 76.373915][ C2] ? save_trace+0x8f/0x5b0 [ 76.374058][ C2] mark_lock+0x28d/0x3e0 [ 76.374166][ C2] mark_usage+0xd9/0x2a0 [ 76.374276][ C2] __lock_acquire+0x797/0x1570 [ 76.374421][ C2] lock_acquire.part.0+0xeb/0x330 [ 76.374568][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 76.374713][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 76.374856][ C2] ? trace_lock_acquire+0x135/0x1c0 [ 76.374999][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 76.375142][ C2] ? lock_acquire+0x32/0xc0 [ 76.375283][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 76.375430][ C2] _raw_spin_lock+0x30/0x40 [ 76.375574][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 76.375809][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 76.375956][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 76.376100][ C2] ? __pfx_sch_direct_xmit+0x10/0x10 [ 76.376242][ C2] ? __dev_xmit_skb+0x792/0xfa0 [ 76.376468][ C2] ? lock_acquire+0x32/0xc0 [ 76.376608][ C2] ? __dev_xmit_skb+0x792/0xfa0 [ 76.376751][ C2] __dev_xmit_skb+0x936/0xfa0 [ 76.376893][ C2] ? __pfx___dev_xmit_skb+0x10/0x10 [ 76.377033][ C2] ? __dev_queue_xmit+0x1e5/0x16e0 [ 76.377262][ C2] ? lock_acquire+0x32/0xc0 [ 76.377405][ C2] ? __dev_queue_xmit+0x1e5/0x16e0 [ 76.377548][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 76.377691][ C2] ? mark_held_locks+0x9e/0xe0 [ 76.377924][ C2] ? lockdep_hardirqs_on_prepare.part.0+0x14f/0x370 [ 76.378100][ C2] ? __pfx___dev_queue_xmit+0x10/0x10 [ 76.378242][ C2] ? neigh_hh_output+0x36d/0x5d0 [ 76.378389][ C2] ip6_finish_output2+0x59b/0xf60 [ 76.378627][ C2] ip6_finish_output+0x553/0xdf0 [ 76.378774][ C2] ? hlock_class+0x4e/0x130 [ 76.378916][ C2] ip6_output+0x1f3/0x770 [ 76.379040][ C2] ? __pfx_ip6_output+0x10/0x10 [ 76.379190][ C2] ? lock_acquire.part.0+0x22/0x330 [ 76.379333][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 76.379475][ C2] ? ndisc_send_skb+0x4a3/0x1230 [ 76.379616][ C2] ? __pfx_NF_HOOK.constprop.0+0x10/0x10 [ 76.379759][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 76.379988][ C2] ? trace_lock_acquire+0x135/0x1c0 [ 76.380131][ C2] ndisc_send_skb+0x8e0/0x1230 [ 76.380272][ C2] ? __pfx_ndisc_send_skb+0x10/0x10 [ 76.380415][ C2] ? __ndisc_fill_addr_option+0xb5/0x110 [ 76.380650][ C2] addrconf_rs_timer+0x268/0x730 [ 76.380793][ C2] ? __pfx_addrconf_rs_timer+0x10/0x10 [ 76.380942][ C2] ? call_timer_fn+0xe8/0x230 [ 76.381087][ C2] ? lock_acquire+0x32/0xc0 [ 76.381229][ C2] ? call_timer_fn+0xe8/0x230 [ 76.381377][ C2] ? __pfx_addrconf_rs_timer+0x10/0x10 [ 76.381517][ C2] call_timer_fn+0x13b/0x230 [ 76.381658][ C2] ? call_timer_fn+0xe8/0x230 [ 76.381882][ C2] ? call_timer_fn+0xe8/0x230 [ 76.382023][ C2] ? __pfx_call_timer_fn+0x10/0x10 [ 76.382163][ C2] ? mark_lock+0x38/0x3e0 [ 76.382274][ C2] __run_timers+0x545/0x810 [ 76.382420][ C2] ? __pfx_addrconf_rs_timer+0x10/0x10 [ 76.382652][ C2] ? __pfx___run_timers+0x10/0x10 [ 76.382795][ C2] ? do_raw_spin_lock+0x131/0x270 [ 76.382942][ C2] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 76.383087][ C2] ? lock_acquire+0x32/0xc0 [ 76.383234][ C2] ? run_timer_softirq+0xe0/0x1b0 [ 76.383380][ C2] run_timer_softirq+0xe8/0x1b0 [ 76.383521][ C2] __do_softirq+0x1f8/0x5df [ 76.383663][ C2] irq_exit_rcu+0x97/0xc0 [ 76.383772][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 76.383998][ C2] [ 76.384070][ C2] [ 76.384141][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 76.384317][ C2] RIP: 0010:default_idle+0xf/0x20 [ 76.384464][ C2] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d d3 eb 32 00 fb f4 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 [ 76.385062][ C2] RSP: 0018:ffffc9000015fdf8 EFLAGS: 00000242 [ 76.385246][ C2] RAX: 000000000023be25 RBX: 1ffff9200002bfc1 RCX: ffffffff95932ea5 [ 76.385462][ C2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff930ac1d4 [ 76.385676][ C2] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1005ee709c [ 76.386061][ C2] R10: ffff88802f7384e3 R11: ffff88802f72d9b0 R12: 0000000000000000 [ 76.386280][ C2] R13: ffff888001b8c5c0 R14: dffffc0000000000 R15: 0000000000000000 [ 76.386495][ C2] ? ct_kernel_exit.constprop.0+0xc5/0xf0 [ 76.386723][ C2] ? cpuidle_idle_call+0x1f4/0x280 [ 76.386865][ C2] default_idle_call+0x6d/0xb0 [ 76.387006][ C2] cpuidle_idle_call+0x1f4/0x280 [ 76.387147][ C2] ? __pfx_cpuidle_idle_call+0x10/0x10 [ 76.387374][ C2] ? tsc_verify_tsc_adjust+0x5e/0x2b0 [ 76.387520][ C2] do_idle+0xf9/0x160 [ 76.387628][ C2] cpu_startup_entry+0x54/0x60 [ 76.387770][ C2] start_secondary+0x21c/0x2b0 [ 76.387913][ C2] ? __pfx_start_secondary+0x10/0x10 [ 76.388142][ C2] common_startup_64+0x12c/0x138 [ 76.388285][ C2]