[ 84.557805][ T828] gre: GRE over IPv4 demultiplexor driver [ 84.581429][ T828] ip6_gre: GRE over IPv6 tunneling driver [ 84.727812][ T50] ip6_tunnel: er6 xmit: Local address not yet configured! [ 84.958663][ T51] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 84.969582][ T10] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 85.024586][ T51] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 86.388287][ T838] Mirror/redirect action on [ 89.384770][ C2] [ 89.384938][ C2] ======================================================== [ 89.385182][ C2] WARNING: possible irq lock inversion dependency detected [ 89.385412][ C2] 6.9.0-rc1-virtme #1 Not tainted [ 89.385569][ C2] -------------------------------------------------------- [ 89.385798][ C2] swapper/2/0 just changed the state of lock: [ 89.385990][ C2] ffff8880064ba0d8 (&qdisc_xmit_lock_key#6){+.-.}-{2:2}, at: sch_direct_xmit+0x2a1/0x11c0 [ 89.386293][ C2] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 89.386521][ C2] (&p->alloc_lock){+.+.}-{2:2} [ 89.386526][ C2] [ 89.386526][ C2] [ 89.386526][ C2] and interrupts could create inverse lock ordering between them. [ 89.386526][ C2] [ 89.387083][ C2] [ 89.387083][ C2] other info that might help us debug this: [ 89.387306][ C2] Possible interrupt unsafe locking scenario: [ 89.387306][ C2] [ 89.387536][ C2] CPU0 CPU1 [ 89.387690][ C2] ---- ---- [ 89.387842][ C2] lock(&p->alloc_lock); [ 89.387965][ C2] local_irq_disable(); [ 89.388153][ C2] lock(&qdisc_xmit_lock_key#6); [ 89.388385][ C2] lock(&p->alloc_lock); [ 89.388589][ C2] [ 89.388705][ C2] lock(&qdisc_xmit_lock_key#6); [ 89.388860][ C2] [ 89.388860][ C2] *** DEADLOCK *** [ 89.388860][ C2] [ 89.389084][ C2] 3 locks held by swapper/2/0: [ 89.389239][ C2] #0: ffffffffa1b66360 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1f8/0x670 [ 89.389515][ C2] #1: ffffffffa1b66300 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1e5/0x16e0 [ 89.389796][ C2] #2: ffff888002118a58 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb+0x792/0xfa0 [ 89.390140][ C2] [ 89.390140][ C2] the shortest dependencies between 2nd lock and 1st lock: [ 89.390410][ C2] -> (&p->alloc_lock){+.+.}-{2:2} { [ 89.390571][ C2] HARDIRQ-ON-W at: [ 89.390688][ C2] __lock_acquire+0x797/0x1570 [ 89.390895][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.391090][ C2] _raw_spin_lock+0x30/0x40 [ 89.391288][ C2] set_mems_allowed+0x1d/0x210 [ 89.391482][ C2] kernel_init_freeable+0x72/0x310 [ 89.391681][ C2] kernel_init+0x20/0x200 [ 89.391877][ C2] ret_from_fork+0x31/0x70 [ 89.392077][ C2] ret_from_fork_asm+0x1a/0x30 [ 89.392273][ C2] SOFTIRQ-ON-W at: [ 89.392390][ C2] __lock_acquire+0x797/0x1570 [ 89.392592][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.392786][ C2] _raw_spin_lock+0x30/0x40 [ 89.392982][ C2] set_mems_allowed+0x1d/0x210 [ 89.393176][ C2] kernel_init_freeable+0x72/0x310 [ 89.393368][ C2] kernel_init+0x20/0x200 [ 89.393563][ C2] ret_from_fork+0x31/0x70 [ 89.393757][ C2] ret_from_fork_asm+0x1a/0x30 [ 89.393949][ C2] INITIAL USE at: [ 89.394063][ C2] __lock_acquire+0x797/0x1570 [ 89.394260][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.394455][ C2] _raw_spin_lock+0x30/0x40 [ 89.394650][ C2] set_mems_allowed+0x1d/0x210 [ 89.394843][ C2] kernel_init_freeable+0x72/0x310 [ 89.395046][ C2] kernel_init+0x20/0x200 [ 89.395239][ C2] ret_from_fork+0x31/0x70 [ 89.395432][ C2] ret_from_fork_asm+0x1a/0x30 [ 89.395632][ C2] } [ 89.395714][ C2] ... key at: [] __key.388+0x0/0x40 [ 89.395946][ C2] ... acquired at: [ 89.396062][ C2] __lock_acquire+0xaf0/0x1570 [ 89.396223][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.396377][ C2] _raw_spin_lock+0x30/0x40 [ 89.396538][ C2] __get_task_comm+0x27/0x70 [ 89.396697][ C2] ref_tracker_alloc+0x2ee/0x490 [ 89.396857][ C2] dst_init+0x84/0x4a0 [ 89.396976][ C2] dst_alloc+0x97/0x150 [ 89.397091][ C2] ip6_dst_alloc+0x23/0x90 [ 89.397245][ C2] ip6_rt_pcpu_alloc+0x1e6/0x520 [ 89.397399][ C2] ip6_pol_route+0x56f/0x840 [ 89.397558][ C2] __fib6_rule_action+0x1f3/0x5b0 [ 89.397712][ C2] fib_rules_lookup+0x2b4/0xa20 [ 89.397869][ C2] fib6_rule_lookup+0x182/0x630 [ 89.398027][ C2] ip6_route_output_flags+0x259/0x480 [ 89.398181][ C2] ip6_tnl_xmit+0x1540/0x3490 [ 89.398340][ C2] ip6erspan_tunnel_xmit+0xf63/0x2ce0 [ip6_gre] [ 89.398540][ C2] dev_hard_start_xmit+0x10e/0x360 [ 89.398695][ C2] sch_direct_xmit+0x203/0x11c0 [ 89.398853][ C2] __dev_xmit_skb+0x936/0xfa0 [ 89.399008][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 89.399161][ C2] ip6_finish_output2+0x59b/0xf60 [ 89.399320][ C2] ip6_finish_output+0x553/0xdf0 [ 89.399473][ C2] ip6_output+0x1f3/0x770 [ 89.399630][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 89.399786][ C2] mld_sendpack+0x62d/0xbc0 [ 89.399943][ C2] mld_ifc_work+0x32/0x1f0 [ 89.400098][ C2] process_one_work+0xe2c/0x1730 [ 89.400256][ C2] worker_thread+0x587/0xd30 [ 89.400416][ C2] kthread+0x28a/0x350 [ 89.400540][ C2] ret_from_fork+0x31/0x70 [ 89.400695][ C2] ret_from_fork_asm+0x1a/0x30 [ 89.400851][ C2] [ 89.400935][ C2] -> (&qdisc_xmit_lock_key#6){+.-.}-{2:2} { [ 89.401134][ C2] HARDIRQ-ON-W at: [ 89.401251][ C2] __lock_acquire+0x797/0x1570 [ 89.401449][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.401646][ C2] _raw_spin_lock+0x30/0x40 [ 89.401846][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 89.402041][ C2] __dev_xmit_skb+0x936/0xfa0 [ 89.402237][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 89.402430][ C2] ip6_finish_output2+0x512/0xf60 [ 89.402622][ C2] ip6_finish_output+0x553/0xdf0 [ 89.402821][ C2] ip6_output+0x1f3/0x770 [ 89.403022][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 89.403217][ C2] mld_sendpack+0x62d/0xbc0 [ 89.403413][ C2] mld_ifc_work+0x32/0x1f0 [ 89.403608][ C2] process_one_work+0xe2c/0x1730 [ 89.403802][ C2] worker_thread+0x587/0xd30 [ 89.403996][ C2] kthread+0x28a/0x350 [ 89.404151][ C2] ret_from_fork+0x31/0x70 [ 89.404352][ C2] ret_from_fork_asm+0x1a/0x30 [ 89.404549][ C2] IN-SOFTIRQ-W at: [ 89.404665][ C2] __lock_acquire+0x797/0x1570 [ 89.404869][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.405194][ C2] _raw_spin_lock+0x30/0x40 [ 89.405382][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 89.405573][ C2] __dev_xmit_skb+0x936/0xfa0 [ 89.405760][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 89.405953][ C2] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 89.406181][ C2] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 89.406411][ C2] tcf_action_exec.part.0+0x112/0x3d0 [ 89.406597][ C2] __tcf_classify.constprop.0+0x17e/0x4e0 [ 89.406823][ C2] tcf_classify+0x72/0xa0 [ 89.407011][ C2] tc_run+0x2be/0x700 [ 89.407162][ C2] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 89.407397][ C2] __netif_receive_skb_one_core+0xaf/0x1b0 [ 89.407620][ C2] process_backlog+0x103/0x670 [ 89.407807][ C2] __napi_poll.constprop.0+0xa2/0x460 [ 89.407996][ C2] net_rx_action+0x49f/0xc00 [ 89.408182][ C2] __do_softirq+0x1f8/0x5df [ 89.408375][ C2] irq_exit_rcu+0x97/0xc0 [ 89.408562][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 89.408785][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 89.409012][ C2] default_idle+0xf/0x20 [ 89.409199][ C2] default_idle_call+0x6d/0xb0 [ 89.409389][ C2] cpuidle_idle_call+0x1f4/0x280 [ 89.409580][ C2] do_idle+0xf9/0x160 [ 89.409729][ C2] cpu_startup_entry+0x54/0x60 [ 89.409921][ C2] start_secondary+0x21c/0x2b0 [ 89.410110][ C2] common_startup_64+0x12c/0x138 [ 89.410298][ C2] INITIAL USE at: [ 89.410413][ C2] __lock_acquire+0x797/0x1570 [ 89.410599][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.410787][ C2] _raw_spin_lock+0x30/0x40 [ 89.410974][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 89.411166][ C2] __dev_xmit_skb+0x936/0xfa0 [ 89.411350][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 89.411539][ C2] ip6_finish_output2+0x512/0xf60 [ 89.411732][ C2] ip6_finish_output+0x553/0xdf0 [ 89.411916][ C2] ip6_output+0x1f3/0x770 [ 89.412101][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 89.412287][ C2] mld_sendpack+0x62d/0xbc0 [ 89.412474][ C2] mld_ifc_work+0x32/0x1f0 [ 89.412659][ C2] process_one_work+0xe2c/0x1730 [ 89.412851][ C2] worker_thread+0x587/0xd30 [ 89.413035][ C2] kthread+0x28a/0x350 [ 89.413183][ C2] ret_from_fork+0x31/0x70 [ 89.413372][ C2] ret_from_fork_asm+0x1a/0x30 [ 89.413559][ C2] } [ 89.413634][ C2] ... key at: [] qdisc_xmit_lock_key.143+0x0/0xffffffffffffb940 [ip6_gre] [ 89.413932][ C2] ... acquired at: [ 89.414047][ C2] mark_lock+0x28d/0x3e0 [ 89.414196][ C2] mark_usage+0xd9/0x2a0 [ 89.414345][ C2] __lock_acquire+0x797/0x1570 [ 89.414493][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.414650][ C2] _raw_spin_lock+0x30/0x40 [ 89.414805][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 89.414960][ C2] __dev_xmit_skb+0x936/0xfa0 [ 89.415109][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 89.415262][ C2] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 89.415452][ C2] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 89.415639][ C2] tcf_action_exec.part.0+0x112/0x3d0 [ 89.415787][ C2] __tcf_classify.constprop.0+0x17e/0x4e0 [ 89.415975][ C2] tcf_classify+0x72/0xa0 [ 89.416123][ C2] tc_run+0x2be/0x700 [ 89.416236][ C2] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 89.416421][ C2] __netif_receive_skb_one_core+0xaf/0x1b0 [ 89.416713][ C2] process_backlog+0x103/0x670 [ 89.416861][ C2] __napi_poll.constprop.0+0xa2/0x460 [ 89.417011][ C2] net_rx_action+0x49f/0xc00 [ 89.417161][ C2] __do_softirq+0x1f8/0x5df [ 89.417311][ C2] irq_exit_rcu+0x97/0xc0 [ 89.417550][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 89.417743][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 89.417930][ C2] default_idle+0xf/0x20 [ 89.418174][ C2] default_idle_call+0x6d/0xb0 [ 89.418324][ C2] cpuidle_idle_call+0x1f4/0x280 [ 89.418473][ C2] do_idle+0xf9/0x160 [ 89.418585][ C2] cpu_startup_entry+0x54/0x60 [ 89.418742][ C2] start_secondary+0x21c/0x2b0 [ 89.419073][ C2] common_startup_64+0x12c/0x138 [ 89.419221][ C2] [ 89.419297][ C2] [ 89.419297][ C2] stack backtrace: [ 89.419485][ C2] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.9.0-rc1-virtme #1 [ 89.419794][ C2] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 89.420120][ C2] Call Trace: [ 89.420242][ C2] [ 89.420325][ C2] dump_stack_lvl+0x82/0xd0 [ 89.420577][ C2] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 89.420761][ C2] ? kernel_text_address+0x17/0xe0 [ 89.420914][ C2] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 89.421098][ C2] ? __pfx_usage_skip+0x10/0x10 [ 89.421245][ C2] ? __pfx_usage_match+0x10/0x10 [ 89.421396][ C2] ? arch_stack_walk+0x8c/0xf0 [ 89.421544][ C2] mark_lock_irq+0x4cd/0xa10 [ 89.421693][ C2] ? common_startup_64+0x12c/0x138 [ 89.421932][ C2] ? __pfx_mark_lock_irq+0x10/0x10 [ 89.422080][ C2] ? __pfx_stack_trace_save+0x10/0x10 [ 89.422238][ C2] ? save_trace+0x8f/0x5b0 [ 89.422388][ C2] mark_lock+0x28d/0x3e0 [ 89.422593][ C2] mark_usage+0xd9/0x2a0 [ 89.422705][ C2] __lock_acquire+0x797/0x1570 [ 89.422854][ C2] ? tcf_mirred_act+0x302/0x780 [act_mirred] [ 89.423047][ C2] ? arch_stack_walk+0x68/0xf0 [ 89.423286][ C2] lock_acquire.part.0+0xeb/0x330 [ 89.423434][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 89.423583][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 89.423736][ C2] ? trace_lock_acquire+0x135/0x1c0 [ 89.423885][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 89.424125][ C2] ? lock_acquire+0x32/0xc0 [ 89.424272][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 89.424423][ C2] _raw_spin_lock+0x30/0x40 [ 89.424570][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 89.424809][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 89.424969][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 89.425122][ C2] ? __pfx_sch_direct_xmit+0x10/0x10 [ 89.425271][ C2] ? __dev_xmit_skb+0x792/0xfa0 [ 89.425507][ C2] ? lock_acquire+0x32/0xc0 [ 89.425656][ C2] ? __dev_xmit_skb+0x792/0xfa0 [ 89.425805][ C2] __dev_xmit_skb+0x936/0xfa0 [ 89.425954][ C2] ? __pfx___dev_xmit_skb+0x10/0x10 [ 89.426192][ C2] ? __dev_queue_xmit+0x1e5/0x16e0 [ 89.426342][ C2] ? lock_acquire+0x32/0xc0 [ 89.426493][ C2] ? __dev_queue_xmit+0x1e5/0x16e0 [ 89.426643][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 89.426793][ C2] ? lockdep_hardirqs_on_prepare.part.0+0x14f/0x370 [ 89.427070][ C2] ? __pfx___dev_queue_xmit+0x10/0x10 [ 89.427220][ C2] ? trace_kmem_cache_alloc+0x2a/0xd0 [ 89.427368][ C2] ? kmem_cache_alloc+0x132/0x270 [ 89.427519][ C2] ? __copy_skb_header+0xaf/0x490 [ 89.427764][ C2] ? __skb_clone+0x57a/0x760 [ 89.427919][ C2] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 89.428110][ C2] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 89.428387][ C2] ? hlock_class+0x4e/0x130 [ 89.428538][ C2] tcf_action_exec.part.0+0x112/0x3d0 [ 89.428687][ C2] ? mall_classify+0x173/0x240 [cls_matchall] [ 89.428875][ C2] __tcf_classify.constprop.0+0x17e/0x4e0 [ 89.429115][ C2] ? update_sd_lb_stats.constprop.0+0x9e4/0xd40 [ 89.429308][ C2] tcf_classify+0x72/0xa0 [ 89.429422][ C2] ? __pfx_tcf_classify+0x10/0x10 [ 89.429572][ C2] ? __pfx_update_sd_lb_stats.constprop.0+0x10/0x10 [ 89.429857][ C2] tc_run+0x2be/0x700 [ 89.429972][ C2] ? __pfx_tc_run+0x10/0x10 [ 89.430123][ C2] ? hlock_class+0x4e/0x130 [ 89.430273][ C2] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 89.430462][ C2] ? __lock_acquire+0xaf0/0x1570 [ 89.430798][ C2] ? __pfx___netif_receive_skb_core.constprop.0+0x10/0x10 [ 89.430987][ C2] ? lock_acquire.part.0+0xeb/0x330 [ 89.431136][ C2] ? process_backlog+0x1f8/0x670 [ 89.431287][ C2] __netif_receive_skb_one_core+0xaf/0x1b0 [ 89.431566][ C2] ? __pfx___netif_receive_skb_one_core+0x10/0x10 [ 89.431750][ C2] ? process_backlog+0x1f8/0x670 [ 89.431896][ C2] ? lock_acquire+0x32/0xc0 [ 89.432135][ C2] ? process_backlog+0x1f8/0x670 [ 89.432287][ C2] process_backlog+0x103/0x670 [ 89.432439][ C2] __napi_poll.constprop.0+0xa2/0x460 [ 89.432588][ C2] net_rx_action+0x49f/0xc00 [ 89.432740][ C2] ? __pfx_net_rx_action+0x10/0x10 [ 89.432982][ C2] __do_softirq+0x1f8/0x5df [ 89.433131][ C2] irq_exit_rcu+0x97/0xc0 [ 89.433244][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 89.433392][ C2] [ 89.433469][ C2] [ 89.433638][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 89.433824][ C2] RIP: 0010:default_idle+0xf/0x20 [ 89.433978][ C2] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d d3 eb 32 00 fb f4 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 [ 89.434597][ C2] RSP: 0018:ffffc9000015fdf8 EFLAGS: 00000242 [ 89.434796][ C2] RAX: 000000000032cca1 RBX: 1ffff9200002bfc1 RCX: ffffffffa0932ea5 [ 89.435125][ C2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9e0ac1d4 [ 89.435347][ C2] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1006c2709c [ 89.435569][ C2] R10: ffff8880361384e3 R11: ffff88803612d9b0 R12: 0000000000000000 [ 89.435886][ C2] R13: ffff888001b8c5c0 R14: dffffc0000000000 R15: 0000000000000000 [ 89.436113][ C2] ? ct_kernel_exit.constprop.0+0xc5/0xf0 [ 89.436261][ C2] ? cpuidle_idle_call+0x1f4/0x280 [ 89.436411][ C2] default_idle_call+0x6d/0xb0 [ 89.436654][ C2] cpuidle_idle_call+0x1f4/0x280 [ 89.436801][ C2] ? __pfx_cpuidle_idle_call+0x10/0x10 [ 89.436949][ C2] ? tsc_verify_tsc_adjust+0x5e/0x2b0 [ 89.437099][ C2] do_idle+0xf9/0x160 [ 89.437214][ C2] cpu_startup_entry+0x54/0x60 [ 89.437365][ C2] start_secondary+0x21c/0x2b0 [ 89.437513][ C2] ? __pfx_start_secondary+0x10/0x10 [ 89.437662][ C2] common_startup_64+0x12c/0x138 [ 89.437903][ C2]