[ 24.484852][ T264] gre: GRE over IPv4 demultiplexor driver [ 24.512216][ T264] ip6_gre: GRE over IPv6 tunneling driver [ 24.650406][ T7] ip6_tunnel: er6 xmit: Local address not yet configured! [ 25.036343][ T7] ip6_tunnel: er6 xmit: Local address not yet configured! [ 26.407376][ T274] Mirror/redirect action on [ 29.672579][ C2] [ 29.672798][ C2] ======================================================== [ 29.673181][ C2] WARNING: possible irq lock inversion dependency detected [ 29.673467][ C2] 6.9.0-rc1-virtme #1 Not tainted [ 29.673623][ C2] -------------------------------------------------------- [ 29.673854][ C2] swapper/2/0 just changed the state of lock: [ 29.674054][ C2] ffff8880063532d8 (&qdisc_xmit_lock_key#2){+.-.}-{2:2}, at: sch_direct_xmit+0x2a1/0x11c0 [ 29.674350][ C2] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 29.674598][ C2] (&p->alloc_lock){+.+.}-{2:2} [ 29.674602][ C2] [ 29.674602][ C2] [ 29.674602][ C2] and interrupts could create inverse lock ordering between them. [ 29.674602][ C2] [ 29.675175][ C2] [ 29.675175][ C2] other info that might help us debug this: [ 29.675403][ C2] Possible interrupt unsafe locking scenario: [ 29.675403][ C2] [ 29.675640][ C2] CPU0 CPU1 [ 29.675796][ C2] ---- ---- [ 29.675953][ C2] lock(&p->alloc_lock); [ 29.676087][ C2] local_irq_disable(); [ 29.676284][ C2] lock(&qdisc_xmit_lock_key#2); [ 29.676519][ C2] lock(&p->alloc_lock); [ 29.676719][ C2] [ 29.676858][ C2] lock(&qdisc_xmit_lock_key#2); [ 29.677021][ C2] [ 29.677021][ C2] *** DEADLOCK *** [ 29.677021][ C2] [ 29.677281][ C2] 3 locks held by swapper/2/0: [ 29.677442][ C2] #0: ffffffffb9f66360 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1f8/0x670 [ 29.677752][ C2] #1: ffffffffb9f66300 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1e5/0x16e0 [ 29.678032][ C2] #2: ffff888006943a58 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb+0x792/0xfa0 [ 29.678390][ C2] [ 29.678390][ C2] the shortest dependencies between 2nd lock and 1st lock: [ 29.678660][ C2] -> (&p->alloc_lock){+.+.}-{2:2} { [ 29.678840][ C2] HARDIRQ-ON-W at: [ 29.678960][ C2] __lock_acquire+0x797/0x1570 [ 29.679174][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.679375][ C2] _raw_spin_lock+0x30/0x40 [ 29.679577][ C2] set_mems_allowed+0x1d/0x210 [ 29.679779][ C2] kernel_init_freeable+0x72/0x310 [ 29.679990][ C2] kernel_init+0x20/0x200 [ 29.680187][ C2] ret_from_fork+0x31/0x70 [ 29.680392][ C2] ret_from_fork_asm+0x1a/0x30 [ 29.680609][ C2] SOFTIRQ-ON-W at: [ 29.680730][ C2] __lock_acquire+0x797/0x1570 [ 29.680930][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.681133][ C2] _raw_spin_lock+0x30/0x40 [ 29.681333][ C2] set_mems_allowed+0x1d/0x210 [ 29.681534][ C2] kernel_init_freeable+0x72/0x310 [ 29.681738][ C2] kernel_init+0x20/0x200 [ 29.681938][ C2] ret_from_fork+0x31/0x70 [ 29.682138][ C2] ret_from_fork_asm+0x1a/0x30 [ 29.682336][ C2] INITIAL USE at: [ 29.682458][ C2] __lock_acquire+0x797/0x1570 [ 29.682654][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.682855][ C2] _raw_spin_lock+0x30/0x40 [ 29.683050][ C2] set_mems_allowed+0x1d/0x210 [ 29.683248][ C2] kernel_init_freeable+0x72/0x310 [ 29.683501][ C2] kernel_init+0x20/0x200 [ 29.683772][ C2] ret_from_fork+0x31/0x70 [ 29.684050][ C2] ret_from_fork_asm+0x1a/0x30 [ 29.684324][ C2] } [ 29.684436][ C2] ... key at: [] __key.388+0x0/0x40 [ 29.684721][ C2] ... acquired at: [ 29.684841][ C2] __lock_acquire+0xaf0/0x1570 [ 29.685002][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.685161][ C2] _raw_spin_lock+0x30/0x40 [ 29.685320][ C2] __get_task_comm+0x27/0x70 [ 29.685478][ C2] ref_tracker_alloc+0x2ee/0x490 [ 29.685637][ C2] ___neigh_create+0x11e/0xe10 [ 29.685798][ C2] ip6_finish_output2+0xabd/0xf60 [ 29.685958][ C2] ip6_finish_output+0x553/0xdf0 [ 29.686115][ C2] ip6_output+0x1f3/0x770 [ 29.686274][ C2] ip6tunnel_xmit.constprop.0+0x16b/0x360 [ 29.686471][ C2] ip6_tnl_xmit+0x1266/0x3490 [ 29.686628][ C2] ip6erspan_tunnel_xmit+0xf63/0x2ce0 [ip6_gre] [ 29.686835][ C2] dev_hard_start_xmit+0x10e/0x360 [ 29.686997][ C2] sch_direct_xmit+0x203/0x11c0 [ 29.687166][ C2] __dev_xmit_skb+0x936/0xfa0 [ 29.687326][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 29.687486][ C2] ip6_finish_output2+0x512/0xf60 [ 29.687647][ C2] ip6_finish_output+0x553/0xdf0 [ 29.687807][ C2] ip6_output+0x1f3/0x770 [ 29.687965][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 29.688124][ C2] ndisc_send_skb+0x8e0/0x1230 [ 29.688287][ C2] ndisc_send_ns+0x9e/0x100 [ 29.688444][ C2] addrconf_dad_work+0x7dd/0xab0 [ 29.688602][ C2] process_one_work+0xe2c/0x1730 [ 29.688763][ C2] worker_thread+0x587/0xd30 [ 29.688923][ C2] kthread+0x28a/0x350 [ 29.689054][ C2] ret_from_fork+0x31/0x70 [ 29.689212][ C2] ret_from_fork_asm+0x1a/0x30 [ 29.689406][ C2] [ 29.689518][ C2] -> (&qdisc_xmit_lock_key#2){+.-.}-{2:2} { [ 29.689798][ C2] HARDIRQ-ON-W at: [ 29.689962][ C2] __lock_acquire+0x797/0x1570 [ 29.690253][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.690524][ C2] _raw_spin_lock+0x30/0x40 [ 29.690750][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 29.690949][ C2] __dev_xmit_skb+0x936/0xfa0 [ 29.691145][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 29.691360][ C2] ip6_finish_output2+0x512/0xf60 [ 29.691559][ C2] ip6_finish_output+0x553/0xdf0 [ 29.691766][ C2] ip6_output+0x1f3/0x770 [ 29.691966][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 29.692182][ C2] mld_sendpack+0x62d/0xbc0 [ 29.692384][ C2] mld_ifc_work+0x32/0x1f0 [ 29.692593][ C2] process_one_work+0xe2c/0x1730 [ 29.692837][ C2] worker_thread+0x587/0xd30 [ 29.693119][ C2] kthread+0x28a/0x350 [ 29.693323][ C2] ret_from_fork+0x31/0x70 [ 29.693524][ C2] ret_from_fork_asm+0x1a/0x30 [ 29.693736][ C2] IN-SOFTIRQ-W at: [ 29.693854][ C2] __lock_acquire+0x797/0x1570 [ 29.694056][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.694254][ C2] _raw_spin_lock+0x30/0x40 [ 29.694462][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 29.694659][ C2] __dev_xmit_skb+0x936/0xfa0 [ 29.694859][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 29.695058][ C2] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 29.695300][ C2] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 29.695538][ C2] tcf_action_exec.part.0+0x112/0x3d0 [ 29.695737][ C2] __tcf_classify.constprop.0+0x17e/0x4e0 [ 29.695975][ C2] tcf_classify+0x72/0xa0 [ 29.696173][ C2] tc_run+0x2be/0x700 [ 29.696333][ C2] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 29.696572][ C2] __netif_receive_skb_one_core+0xaf/0x1b0 [ 29.696811][ C2] process_backlog+0x103/0x670 [ 29.697010][ C2] __napi_poll.constprop.0+0xa2/0x460 [ 29.697225][ C2] net_rx_action+0x49f/0xc00 [ 29.697426][ C2] __do_softirq+0x1f8/0x5df [ 29.697626][ C2] irq_exit_rcu+0x97/0xc0 [ 29.697834][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 29.698075][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 29.698326][ C2] default_idle+0xf/0x20 [ 29.698529][ C2] default_idle_call+0x6d/0xb0 [ 29.698734][ C2] cpuidle_idle_call+0x1f4/0x280 [ 29.698938][ C2] do_idle+0xf9/0x160 [ 29.699103][ C2] cpu_startup_entry+0x54/0x60 [ 29.699317][ C2] start_secondary+0x21c/0x2b0 [ 29.699517][ C2] common_startup_64+0x12c/0x138 [ 29.699716][ C2] INITIAL USE at: [ 29.699841][ C2] __lock_acquire+0x797/0x1570 [ 29.700044][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.700244][ C2] _raw_spin_lock+0x30/0x40 [ 29.700454][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 29.700707][ C2] __dev_xmit_skb+0x936/0xfa0 [ 29.700978][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 29.701250][ C2] ip6_finish_output2+0x512/0xf60 [ 29.701525][ C2] ip6_finish_output+0x553/0xdf0 [ 29.701731][ C2] ip6_output+0x1f3/0x770 [ 29.701931][ C2] NF_HOOK.constprop.0+0xe2/0x680 [ 29.702128][ C2] mld_sendpack+0x62d/0xbc0 [ 29.702325][ C2] mld_ifc_work+0x32/0x1f0 [ 29.702522][ C2] process_one_work+0xe2c/0x1730 [ 29.702720][ C2] worker_thread+0x587/0xd30 [ 29.702921][ C2] kthread+0x28a/0x350 [ 29.703124][ C2] ret_from_fork+0x31/0x70 [ 29.703399][ C2] ret_from_fork_asm+0x1a/0x30 [ 29.703648][ C2] } [ 29.703727][ C2] ... key at: [] qdisc_xmit_lock_key.143+0x0/0xffffffffffffb940 [ip6_gre] [ 29.704049][ C2] ... acquired at: [ 29.704166][ C2] mark_lock+0x28d/0x3e0 [ 29.704327][ C2] mark_usage+0xd9/0x2a0 [ 29.704552][ C2] __lock_acquire+0x797/0x1570 [ 29.704771][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.704989][ C2] _raw_spin_lock+0x30/0x40 [ 29.705207][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 29.705379][ C2] __dev_xmit_skb+0x936/0xfa0 [ 29.705551][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 29.705710][ C2] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 29.705908][ C2] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 29.706110][ C2] tcf_action_exec.part.0+0x112/0x3d0 [ 29.706270][ C2] __tcf_classify.constprop.0+0x17e/0x4e0 [ 29.706467][ C2] tcf_classify+0x72/0xa0 [ 29.706654][ C2] tc_run+0x2be/0x700 [ 29.706772][ C2] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 29.706967][ C2] __netif_receive_skb_one_core+0xaf/0x1b0 [ 29.707165][ C2] process_backlog+0x103/0x670 [ 29.707326][ C2] __napi_poll.constprop.0+0xa2/0x460 [ 29.707575][ C2] net_rx_action+0x49f/0xc00 [ 29.707738][ C2] __do_softirq+0x1f8/0x5df [ 29.707897][ C2] irq_exit_rcu+0x97/0xc0 [ 29.708057][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 29.708358][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 29.708556][ C2] default_idle+0xf/0x20 [ 29.708713][ C2] default_idle_call+0x6d/0xb0 [ 29.708871][ C2] cpuidle_idle_call+0x1f4/0x280 [ 29.709123][ C2] do_idle+0xf9/0x160 [ 29.709244][ C2] cpu_startup_entry+0x54/0x60 [ 29.709403][ C2] start_secondary+0x21c/0x2b0 [ 29.709563][ C2] common_startup_64+0x12c/0x138 [ 29.709857][ C2] [ 29.709965][ C2] [ 29.709965][ C2] stack backtrace: [ 29.710233][ C2] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.9.0-rc1-virtme #1 [ 29.710513][ C2] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 29.710955][ C2] Call Trace: [ 29.711079][ C2] [ 29.711163][ C2] dump_stack_lvl+0x82/0xd0 [ 29.711323][ C2] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 29.711700][ C2] ? kernel_text_address+0x17/0xe0 [ 29.711864][ C2] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 29.712077][ C2] ? __pfx_usage_skip+0x10/0x10 [ 29.712237][ C2] ? __pfx_usage_match+0x10/0x10 [ 29.712493][ C2] ? arch_stack_walk+0x8c/0xf0 [ 29.712704][ C2] mark_lock_irq+0x4cd/0xa10 [ 29.712924][ C2] ? common_startup_64+0x12c/0x138 [ 29.713140][ C2] ? __pfx_mark_lock_irq+0x10/0x10 [ 29.713358][ C2] ? __pfx_stack_trace_save+0x10/0x10 [ 29.713577][ C2] ? save_trace+0x8f/0x5b0 [ 29.713741][ C2] mark_lock+0x28d/0x3e0 [ 29.713861][ C2] mark_usage+0xd9/0x2a0 [ 29.713981][ C2] __lock_acquire+0x797/0x1570 [ 29.714235][ C2] ? tcf_mirred_act+0x302/0x780 [act_mirred] [ 29.714444][ C2] ? arch_stack_walk+0x68/0xf0 [ 29.714602][ C2] lock_acquire.part.0+0xeb/0x330 [ 29.714769][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 29.715064][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 29.715281][ C2] ? trace_lock_acquire+0x135/0x1c0 [ 29.715505][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 29.715665][ C2] ? lock_acquire+0x32/0xc0 [ 29.715916][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 29.716075][ C2] _raw_spin_lock+0x30/0x40 [ 29.716232][ C2] ? sch_direct_xmit+0x2a1/0x11c0 [ 29.716389][ C2] sch_direct_xmit+0x2a1/0x11c0 [ 29.716639][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 29.716799][ C2] ? __pfx_sch_direct_xmit+0x10/0x10 [ 29.716957][ C2] ? __dev_xmit_skb+0x792/0xfa0 [ 29.717119][ C2] ? lock_acquire+0x32/0xc0 [ 29.717372][ C2] ? __dev_xmit_skb+0x792/0xfa0 [ 29.717536][ C2] __dev_xmit_skb+0x936/0xfa0 [ 29.717697][ C2] ? __pfx___dev_xmit_skb+0x10/0x10 [ 29.717857][ C2] ? __dev_queue_xmit+0x1e5/0x16e0 [ 29.718025][ C2] ? lock_acquire+0x32/0xc0 [ 29.718280][ C2] ? __dev_queue_xmit+0x1e5/0x16e0 [ 29.718445][ C2] __dev_queue_xmit+0x8be/0x16e0 [ 29.718606][ C2] ? lockdep_hardirqs_on_prepare.part.0+0x14f/0x370 [ 29.718804][ C2] ? __pfx___dev_queue_xmit+0x10/0x10 [ 29.719054][ C2] ? trace_kmem_cache_alloc+0x2a/0xd0 [ 29.719213][ C2] ? kmem_cache_alloc+0x132/0x270 [ 29.719372][ C2] ? __copy_skb_header+0xaf/0x490 [ 29.719532][ C2] ? __skb_clone+0x57a/0x760 [ 29.719782][ C2] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 29.719982][ C2] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 29.720187][ C2] tcf_action_exec.part.0+0x112/0x3d0 [ 29.720349][ C2] ? mall_classify+0x173/0x240 [cls_matchall] [ 29.720640][ C2] __tcf_classify.constprop.0+0x17e/0x4e0 [ 29.720800][ C2] ? update_sd_lb_stats.constprop.0+0x9e4/0xd40 [ 29.721001][ C2] tcf_classify+0x72/0xa0 [ 29.721123][ C2] ? __pfx_tcf_classify+0x10/0x10 [ 29.721373][ C2] ? __pfx_update_sd_lb_stats.constprop.0+0x10/0x10 [ 29.721569][ C2] ? validate_chain+0x130/0x9b0 [ 29.721726][ C2] tc_run+0x2be/0x700 [ 29.721847][ C2] ? __pfx_tc_run+0x10/0x10 [ 29.722095][ C2] ? hlock_class+0x4e/0x130 [ 29.722252][ C2] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 29.722451][ C2] ? __lock_acquire+0xaf0/0x1570 [ 29.722609][ C2] ? __pfx___netif_receive_skb_core.constprop.0+0x10/0x10 [ 29.722897][ C2] ? lock_acquire.part.0+0xeb/0x330 [ 29.723069][ C2] ? process_backlog+0x1f8/0x670 [ 29.723233][ C2] __netif_receive_skb_one_core+0xaf/0x1b0 [ 29.723440][ C2] ? __pfx___netif_receive_skb_one_core+0x10/0x10 [ 29.723735][ C2] ? process_backlog+0x1f8/0x670 [ 29.723892][ C2] ? lock_acquire+0x32/0xc0 [ 29.724062][ C2] ? process_backlog+0x1f8/0x670 [ 29.724229][ C2] process_backlog+0x103/0x670 [ 29.724574][ C2] ? __lock_release+0x103/0x460 [ 29.724733][ C2] ? rebalance_domains+0x278/0xe20 [ 29.724894][ C2] __napi_poll.constprop.0+0xa2/0x460 [ 29.725054][ C2] net_rx_action+0x49f/0xc00 [ 29.725308][ C2] ? __pfx_net_rx_action+0x10/0x10 [ 29.725465][ C2] ? tmigr_cpu_activate+0x280/0x290 [ 29.725623][ C2] ? __pfx_rebalance_domains+0x10/0x10 [ 29.725781][ C2] __do_softirq+0x1f8/0x5df [ 29.726031][ C2] irq_exit_rcu+0x97/0xc0 [ 29.726149][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 29.726309][ C2] [ 29.726390][ C2] [ 29.726470][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 29.726759][ C2] RIP: 0010:default_idle+0xf/0x20 [ 29.726923][ C2] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d d3 fb 32 00 fb f4 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 [ 29.727598][ C2] RSP: 0018:ffffc9000015fdf8 EFLAGS: 00000242 [ 29.727805][ C2] RAX: 0000000000095957 RBX: 1ffff9200002bfc1 RCX: ffffffffb8d31ea5 [ 29.728045][ C2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffb64ac1d4 [ 29.728375][ C2] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1006c2709c [ 29.728612][ C2] R10: ffff8880361384e3 R11: ffff88803612d9b0 R12: 0000000000000000 [ 29.728847][ C2] R13: ffff888001b8c5c0 R14: dffffc0000000000 R15: 0000000000000000 [ 29.729191][ C2] ? ct_kernel_exit.constprop.0+0xc5/0xf0 [ 29.729362][ C2] ? cpuidle_idle_call+0x1f4/0x280 [ 29.729521][ C2] default_idle_call+0x6d/0xb0 [ 29.729678][ C2] cpuidle_idle_call+0x1f4/0x280 [ 29.729927][ C2] ? __pfx_cpuidle_idle_call+0x10/0x10 [ 29.730087][ C2] ? tsc_verify_tsc_adjust+0x5e/0x2b0 [ 29.730248][ C2] do_idle+0xf9/0x160 [ 29.730371][ C2] cpu_startup_entry+0x54/0x60 [ 29.730529][ C2] start_secondary+0x21c/0x2b0 [ 29.730779][ C2] ? __pfx_start_secondary+0x10/0x10 [ 29.730947][ C2] common_startup_64+0x12c/0x138 [ 29.731108][ C2]