[ 24.459206][ T263] gre: GRE over IPv4 demultiplexor driver [ 24.483909][ T263] ip6_gre: GRE over IPv6 tunneling driver [ 24.621170][ T49] ip6_tunnel: er6 xmit: Local address not yet configured! [ 24.851787][ T49] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 25.057856][ T49] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 25.194183][ T10] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 26.280249][ T273] Mirror/redirect action on [ 29.610029][ C1] [ 29.610324][ C1] ======================================================== [ 29.610551][ C1] WARNING: possible irq lock inversion dependency detected [ 29.610765][ C1] 6.9.0-rc1-virtme #1 Not tainted [ 29.610911][ C1] -------------------------------------------------------- [ 29.611123][ C1] swapper/1/0 just changed the state of lock: [ 29.611300][ C1] ffff8880084974d8 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}, at: sch_direct_xmit+0x2a1/0x11c0 [ 29.611564][ C1] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 29.611774][ C1] (&p->alloc_lock){+.+.}-{2:2} [ 29.611778][ C1] [ 29.611778][ C1] [ 29.611778][ C1] and interrupts could create inverse lock ordering between them. [ 29.611778][ C1] [ 29.612296][ C1] [ 29.612296][ C1] other info that might help us debug this: [ 29.612511][ C1] Possible interrupt unsafe locking scenario: [ 29.612511][ C1] [ 29.612724][ C1] CPU0 CPU1 [ 29.612876][ C1] ---- ---- [ 29.613021][ C1] lock(&p->alloc_lock); [ 29.613132][ C1] local_irq_disable(); [ 29.613310][ C1] lock(&qdisc_xmit_lock_key#2); [ 29.613527][ C1] lock(&p->alloc_lock); [ 29.613705][ C1] [ 29.613813][ C1] lock(&qdisc_xmit_lock_key#2); [ 29.613959][ C1] [ 29.613959][ C1] *** DEADLOCK *** [ 29.613959][ C1] [ 29.614167][ C1] 3 locks held by swapper/1/0: [ 29.614313][ C1] #0: ffffffffa2166360 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1f8/0x670 [ 29.614583][ C1] #1: ffffffffa2166300 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1e5/0x16e0 [ 29.614837][ C1] #2: ffff888005f6ba58 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb+0x792/0xfa0 [ 29.615164][ C1] [ 29.615164][ C1] the shortest dependencies between 2nd lock and 1st lock: [ 29.615412][ C1] -> (&p->alloc_lock){+.+.}-{2:2} { [ 29.615563][ C1] HARDIRQ-ON-W at: [ 29.615674][ C1] __lock_acquire+0x797/0x1570 [ 29.615863][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.616045][ C1] _raw_spin_lock+0x30/0x40 [ 29.616230][ C1] set_mems_allowed+0x1d/0x210 [ 29.616417][ C1] kernel_init_freeable+0x72/0x310 [ 29.616600][ C1] kernel_init+0x20/0x200 [ 29.616783][ C1] ret_from_fork+0x31/0x70 [ 29.616966][ C1] ret_from_fork_asm+0x1a/0x30 [ 29.617147][ C1] SOFTIRQ-ON-W at: [ 29.617254][ C1] __lock_acquire+0x797/0x1570 [ 29.617436][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.617615][ C1] _raw_spin_lock+0x30/0x40 [ 29.617796][ C1] set_mems_allowed+0x1d/0x210 [ 29.617977][ C1] kernel_init_freeable+0x72/0x310 [ 29.618157][ C1] kernel_init+0x20/0x200 [ 29.618339][ C1] ret_from_fork+0x31/0x70 [ 29.618522][ C1] ret_from_fork_asm+0x1a/0x30 [ 29.618705][ C1] INITIAL USE at: [ 29.618814][ C1] __lock_acquire+0x797/0x1570 [ 29.618996][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.619178][ C1] _raw_spin_lock+0x30/0x40 [ 29.619358][ C1] set_mems_allowed+0x1d/0x210 [ 29.619540][ C1] kernel_init_freeable+0x72/0x310 [ 29.619723][ C1] kernel_init+0x20/0x200 [ 29.619903][ C1] ret_from_fork+0x31/0x70 [ 29.620085][ C1] ret_from_fork_asm+0x1a/0x30 [ 29.620266][ C1] } [ 29.620341][ C1] ... key at: [] __key.388+0x0/0x40 [ 29.620558][ C1] ... acquired at: [ 29.620666][ C1] __lock_acquire+0xaf0/0x1570 [ 29.620817][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.620962][ C1] _raw_spin_lock+0x30/0x40 [ 29.621106][ C1] __get_task_comm+0x27/0x70 [ 29.621254][ C1] ref_tracker_alloc+0x2ee/0x490 [ 29.621401][ C1] dst_init+0x84/0x4a0 [ 29.621511][ C1] dst_alloc+0x97/0x150 [ 29.621620][ C1] ip6_dst_alloc+0x23/0x90 [ 29.621766][ C1] ip6_rt_pcpu_alloc+0x1e6/0x520 [ 29.621913][ C1] ip6_pol_route+0x56f/0x840 [ 29.622057][ C1] __fib6_rule_action+0x1f3/0x5b0 [ 29.622202][ C1] fib_rules_lookup+0x2b4/0xa20 [ 29.622347][ C1] fib6_rule_lookup+0x182/0x630 [ 29.622490][ C1] ip6_route_output_flags+0x259/0x480 [ 29.622644][ C1] ip6_tnl_xmit+0x1540/0x3490 [ 29.622791][ C1] ip6erspan_tunnel_xmit+0xf63/0x2ce0 [ip6_gre] [ 29.622995][ C1] dev_hard_start_xmit+0x10e/0x360 [ 29.623144][ C1] sch_direct_xmit+0x203/0x11c0 [ 29.623290][ C1] __dev_xmit_skb+0x936/0xfa0 [ 29.623435][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 29.623579][ C1] ip6_finish_output2+0x512/0xf60 [ 29.623723][ C1] ip6_finish_output+0x553/0xdf0 [ 29.623867][ C1] ip6_output+0x1f3/0x770 [ 29.624010][ C1] NF_HOOK.constprop.0+0xe2/0x680 [ 29.624157][ C1] ndisc_send_skb+0x8e0/0x1230 [ 29.624313][ C1] ndisc_send_ns+0x9e/0x100 [ 29.624457][ C1] addrconf_dad_work+0x7dd/0xab0 [ 29.624600][ C1] process_one_work+0xe2c/0x1730 [ 29.624748][ C1] worker_thread+0x587/0xd30 [ 29.624893][ C1] kthread+0x28a/0x350 [ 29.625002][ C1] ret_from_fork+0x31/0x70 [ 29.625147][ C1] ret_from_fork_asm+0x1a/0x30 [ 29.625292][ C1] [ 29.625365][ C1] -> (&qdisc_xmit_lock_key#2){+.-.}-{2:2} { [ 29.625553][ C1] HARDIRQ-ON-W at: [ 29.625662][ C1] __lock_acquire+0x797/0x1570 [ 29.625846][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.626028][ C1] _raw_spin_lock+0x30/0x40 [ 29.626209][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 29.626391][ C1] __dev_xmit_skb+0x936/0xfa0 [ 29.626571][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 29.626754][ C1] ip6_finish_output2+0x512/0xf60 [ 29.626941][ C1] ip6_finish_output+0x553/0xdf0 [ 29.627123][ C1] ip6_output+0x1f3/0x770 [ 29.627304][ C1] NF_HOOK.constprop.0+0xe2/0x680 [ 29.627485][ C1] mld_sendpack+0x62d/0xbc0 [ 29.627667][ C1] mld_ifc_work+0x32/0x1f0 [ 29.627854][ C1] process_one_work+0xe2c/0x1730 [ 29.628035][ C1] worker_thread+0x587/0xd30 [ 29.628218][ C1] kthread+0x28a/0x350 [ 29.628364][ C1] ret_from_fork+0x31/0x70 [ 29.628545][ C1] ret_from_fork_asm+0x1a/0x30 [ 29.628725][ C1] IN-SOFTIRQ-W at: [ 29.628834][ C1] __lock_acquire+0x797/0x1570 [ 29.629020][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.629200][ C1] _raw_spin_lock+0x30/0x40 [ 29.629383][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 29.629564][ C1] __dev_xmit_skb+0x936/0xfa0 [ 29.629744][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 29.629923][ C1] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 29.630148][ C1] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 29.630367][ C1] tcf_action_exec.part.0+0x112/0x3d0 [ 29.630549][ C1] __tcf_classify.constprop.0+0x17e/0x4e0 [ 29.630768][ C1] tcf_classify+0x72/0xa0 [ 29.630949][ C1] tc_run+0x2be/0x700 [ 29.631097][ C1] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 29.631318][ C1] __netif_receive_skb_one_core+0xaf/0x1b0 [ 29.631535][ C1] process_backlog+0x103/0x670 [ 29.631716][ C1] __napi_poll.constprop.0+0xa2/0x460 [ 29.631912][ C1] net_rx_action+0x49f/0xc00 [ 29.632096][ C1] __do_softirq+0x1f8/0x5df [ 29.632279][ C1] irq_exit_rcu+0x97/0xc0 [ 29.632463][ C1] sysvec_apic_timer_interrupt+0x75/0x80 [ 29.632682][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 29.632906][ C1] default_idle+0xf/0x20 [ 29.633086][ C1] default_idle_call+0x6d/0xb0 [ 29.633268][ C1] cpuidle_idle_call+0x1f4/0x280 [ 29.633451][ C1] do_idle+0xf9/0x160 [ 29.633599][ C1] cpu_startup_entry+0x54/0x60 [ 29.633780][ C1] start_secondary+0x21c/0x2b0 [ 29.633963][ C1] common_startup_64+0x12c/0x138 [ 29.634146][ C1] INITIAL USE at: [ 29.634255][ C1] __lock_acquire+0x797/0x1570 [ 29.634439][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.634620][ C1] _raw_spin_lock+0x30/0x40 [ 29.634809][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 29.634991][ C1] __dev_xmit_skb+0x936/0xfa0 [ 29.635173][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 29.635353][ C1] ip6_finish_output2+0x512/0xf60 [ 29.635533][ C1] ip6_finish_output+0x553/0xdf0 [ 29.635713][ C1] ip6_output+0x1f3/0x770 [ 29.635896][ C1] NF_HOOK.constprop.0+0xe2/0x680 [ 29.636077][ C1] mld_sendpack+0x62d/0xbc0 [ 29.636258][ C1] mld_ifc_work+0x32/0x1f0 [ 29.636441][ C1] process_one_work+0xe2c/0x1730 [ 29.636624][ C1] worker_thread+0x587/0xd30 [ 29.636805][ C1] kthread+0x28a/0x350 [ 29.636953][ C1] ret_from_fork+0x31/0x70 [ 29.637137][ C1] ret_from_fork_asm+0x1a/0x30 [ 29.637319][ C1] } [ 29.637392][ C1] ... key at: [] qdisc_xmit_lock_key.143+0x0/0xffffffffffffb940 [ip6_gre] [ 29.637685][ C1] ... acquired at: [ 29.637790][ C1] mark_lock+0x28d/0x3e0 [ 29.637934][ C1] mark_usage+0xd9/0x2a0 [ 29.638078][ C1] __lock_acquire+0x797/0x1570 [ 29.638316][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.638461][ C1] _raw_spin_lock+0x30/0x40 [ 29.638604][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 29.638748][ C1] __dev_xmit_skb+0x936/0xfa0 [ 29.638979][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 29.639123][ C1] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 29.639303][ C1] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 29.639484][ C1] tcf_action_exec.part.0+0x112/0x3d0 [ 29.639719][ C1] __tcf_classify.constprop.0+0x17e/0x4e0 [ 29.639900][ C1] tcf_classify+0x72/0xa0 [ 29.640045][ C1] tc_run+0x2be/0x700 [ 29.640154][ C1] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 29.640424][ C1] __netif_receive_skb_one_core+0xaf/0x1b0 [ 29.640603][ C1] process_backlog+0x103/0x670 [ 29.640749][ C1] __napi_poll.constprop.0+0xa2/0x460 [ 29.640893][ C1] net_rx_action+0x49f/0xc00 [ 29.641129][ C1] __do_softirq+0x1f8/0x5df [ 29.641274][ C1] irq_exit_rcu+0x97/0xc0 [ 29.641417][ C1] sysvec_apic_timer_interrupt+0x75/0x80 [ 29.641596][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 29.641865][ C1] default_idle+0xf/0x20 [ 29.642008][ C1] default_idle_call+0x6d/0xb0 [ 29.642152][ C1] cpuidle_idle_call+0x1f4/0x280 [ 29.642298][ C1] do_idle+0xf9/0x160 [ 29.642498][ C1] cpu_startup_entry+0x54/0x60 [ 29.642641][ C1] start_secondary+0x21c/0x2b0 [ 29.642786][ C1] common_startup_64+0x12c/0x138 [ 29.642955][ C1] [ 29.643029][ C1] [ 29.643029][ C1] stack backtrace: [ 29.643297][ C1] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.9.0-rc1-virtme #1 [ 29.643518][ C1] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 29.643937][ C1] Call Trace: [ 29.644065][ C1] [ 29.644140][ C1] dump_stack_lvl+0x82/0xd0 [ 29.644287][ C1] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 29.644465][ C1] ? kernel_text_address+0x17/0xe0 [ 29.644699][ C1] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 29.644875][ C1] ? __pfx_usage_skip+0x10/0x10 [ 29.645017][ C1] ? __pfx_usage_match+0x10/0x10 [ 29.645160][ C1] ? arch_stack_walk+0x8c/0xf0 [ 29.645393][ C1] mark_lock_irq+0x4cd/0xa10 [ 29.645536][ C1] ? common_startup_64+0x12c/0x138 [ 29.645680][ C1] ? __pfx_mark_lock_irq+0x10/0x10 [ 29.645821][ C1] ? __pfx_stack_trace_save+0x10/0x10 [ 29.646054][ C1] ? save_trace+0x8f/0x5b0 [ 29.646197][ C1] mark_lock+0x28d/0x3e0 [ 29.646306][ C1] mark_usage+0xd9/0x2a0 [ 29.646414][ C1] __lock_acquire+0x797/0x1570 [ 29.646558][ C1] ? tcf_mirred_act+0x302/0x780 [act_mirred] [ 29.646824][ C1] ? arch_stack_walk+0x68/0xf0 [ 29.646968][ C1] lock_acquire.part.0+0xeb/0x330 [ 29.647111][ C1] ? sch_direct_xmit+0x2a1/0x11c0 [ 29.647256][ C1] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 29.647488][ C1] ? trace_lock_acquire+0x135/0x1c0 [ 29.647630][ C1] ? sch_direct_xmit+0x2a1/0x11c0 [ 29.647773][ C1] ? lock_acquire+0x32/0xc0 [ 29.647916][ C1] ? sch_direct_xmit+0x2a1/0x11c0 [ 29.648149][ C1] _raw_spin_lock+0x30/0x40 [ 29.648292][ C1] ? sch_direct_xmit+0x2a1/0x11c0 [ 29.648433][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 29.648578][ C1] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 29.648722][ C1] ? __pfx_sch_direct_xmit+0x10/0x10 [ 29.649042][ C1] ? __dev_xmit_skb+0x792/0xfa0 [ 29.649183][ C1] ? lock_acquire+0x32/0xc0 [ 29.649325][ C1] ? __dev_xmit_skb+0x792/0xfa0 [ 29.649471][ C1] __dev_xmit_skb+0x936/0xfa0 [ 29.649703][ C1] ? __pfx___dev_xmit_skb+0x10/0x10 [ 29.649845][ C1] ? __dev_queue_xmit+0x1e5/0x16e0 [ 29.649987][ C1] ? lock_acquire+0x32/0xc0 [ 29.650130][ C1] ? __dev_queue_xmit+0x1e5/0x16e0 [ 29.650369][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 29.650512][ C1] ? lockdep_hardirqs_on_prepare.part.0+0x14f/0x370 [ 29.650693][ C1] ? __pfx___dev_queue_xmit+0x10/0x10 [ 29.650836][ C1] ? trace_kmem_cache_alloc+0x2a/0xd0 [ 29.651068][ C1] ? kmem_cache_alloc+0x132/0x270 [ 29.651214][ C1] ? __copy_skb_header+0xaf/0x490 [ 29.651358][ C1] ? __skb_clone+0x57a/0x760 [ 29.651503][ C1] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 29.651772][ C1] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 29.651953][ C1] tcf_action_exec.part.0+0x112/0x3d0 [ 29.652099][ C1] ? mall_classify+0x173/0x240 [cls_matchall] [ 29.652280][ C1] __tcf_classify.constprop.0+0x17e/0x4e0 [ 29.652512][ C1] ? update_sd_lb_stats.constprop.0+0x9e4/0xd40 [ 29.652691][ C1] tcf_classify+0x72/0xa0 [ 29.652799][ C1] ? __pfx_tcf_classify+0x10/0x10 [ 29.652948][ C1] ? __pfx_update_sd_lb_stats.constprop.0+0x10/0x10 [ 29.653214][ C1] ? validate_chain+0x130/0x9b0 [ 29.653357][ C1] tc_run+0x2be/0x700 [ 29.653471][ C1] ? __pfx_tc_run+0x10/0x10 [ 29.653614][ C1] ? hlock_class+0x4e/0x130 [ 29.653848][ C1] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 29.654027][ C1] ? __lock_acquire+0xaf0/0x1570 [ 29.654174][ C1] ? __pfx___netif_receive_skb_core.constprop.0+0x10/0x10 [ 29.654352][ C1] ? lock_acquire.part.0+0xeb/0x330 [ 29.654584][ C1] ? process_backlog+0x1f8/0x670 [ 29.654731][ C1] __netif_receive_skb_one_core+0xaf/0x1b0 [ 29.654910][ C1] ? __pfx___netif_receive_skb_one_core+0x10/0x10 [ 29.655090][ C1] ? process_backlog+0x1f8/0x670 [ 29.655323][ C1] ? lock_acquire+0x32/0xc0 [ 29.655466][ C1] ? process_backlog+0x1f8/0x670 [ 29.655611][ C1] process_backlog+0x103/0x670 [ 29.655757][ C1] ? __lock_release+0x103/0x460 [ 29.655988][ C1] ? rebalance_domains+0x278/0xe20 [ 29.656132][ C1] __napi_poll.constprop.0+0xa2/0x460 [ 29.656279][ C1] net_rx_action+0x49f/0xc00 [ 29.656424][ C1] ? __pfx_net_rx_action+0x10/0x10 [ 29.656654][ C1] ? tmigr_cpu_activate+0x280/0x290 [ 29.656798][ C1] ? spin_bug+0x190/0x1d0 [ 29.656907][ C1] ? __pfx_rebalance_domains+0x10/0x10 [ 29.657051][ C1] __do_softirq+0x1f8/0x5df [ 29.657194][ C1] irq_exit_rcu+0x97/0xc0 [ 29.657390][ C1] sysvec_apic_timer_interrupt+0x75/0x80 [ 29.657536][ C1] [ 29.657610][ C1] [ 29.657683][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 29.657860][ C1] RIP: 0010:default_idle+0xf/0x20 [ 29.658101][ C1] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d d3 eb 32 00 fb f4 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 [ 29.658609][ C1] RSP: 0018:ffffc9000014fdf8 EFLAGS: 00000242 [ 29.658887][ C1] RAX: 000000000005fbef RBX: 1ffff92000029fc1 RCX: ffffffffa0f32ea5 [ 29.659102][ C1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9e6ac1d4 [ 29.659317][ C1] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1006c1709c [ 29.659622][ C1] R10: ffff8880360b84e3 R11: ffff8880360ad9b0 R12: 0000000000000000 [ 29.659840][ C1] R13: ffff888001b8a300 R14: dffffc0000000000 R15: 0000000000000000 [ 29.660055][ C1] ? ct_kernel_exit.constprop.0+0xc5/0xf0 [ 29.660374][ C1] ? cpuidle_idle_call+0x1f4/0x280 [ 29.660519][ C1] default_idle_call+0x6d/0xb0 [ 29.660662][ C1] cpuidle_idle_call+0x1f4/0x280 [ 29.660804][ C1] ? __pfx_cpuidle_idle_call+0x10/0x10 [ 29.661034][ C1] ? tsc_verify_tsc_adjust+0x5e/0x2b0 [ 29.661181][ C1] do_idle+0xf9/0x160 [ 29.661289][ C1] cpu_startup_entry+0x54/0x60 [ 29.661433][ C1] start_secondary+0x21c/0x2b0 [ 29.661665][ C1] ? __pfx_start_secondary+0x10/0x10 [ 29.661808][ C1] common_startup_64+0x12c/0x138 [ 29.661953][ C1]