[ 56.816734][ T506] gre: GRE over IPv4 demultiplexor driver [ 56.845763][ T506] ip6_gre: GRE over IPv6 tunneling driver [ 57.011402][ T7] ip6_tunnel: er6 xmit: Local address not yet configured! [ 57.166634][ T7] ip6_tunnel: er6 xmit: Local address not yet configured! [ 57.267345][ T64] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 57.849429][ T64] ip6_tunnel: gt6 xmit: Local address not yet configured! [ 58.707634][ T516] Mirror/redirect action on [ 62.073411][ C1] [ 62.073644][ C1] ======================================================== [ 62.073900][ C1] WARNING: possible irq lock inversion dependency detected [ 62.074163][ C1] 6.9.0-rc1-virtme #1 Not tainted [ 62.074343][ C1] -------------------------------------------------------- [ 62.074597][ C1] swapper/1/0 just changed the state of lock: [ 62.074805][ C1] ffff888008ddd0d8 (&qdisc_xmit_lock_key#4){+.-.}-{2:2}, at: sch_direct_xmit+0x2a1/0x11c0 [ 62.075114][ C1] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 62.075356][ C1] (&p->alloc_lock){+.+.}-{2:2} [ 62.075361][ C1] [ 62.075361][ C1] [ 62.075361][ C1] and interrupts could create inverse lock ordering between them. [ 62.075361][ C1] [ 62.075955][ C1] [ 62.075955][ C1] other info that might help us debug this: [ 62.076196][ C1] Possible interrupt unsafe locking scenario: [ 62.076196][ C1] [ 62.076448][ C1] CPU0 CPU1 [ 62.076609][ C1] ---- ---- [ 62.076778][ C1] lock(&p->alloc_lock); [ 62.076909][ C1] local_irq_disable(); [ 62.077112][ C1] lock(&qdisc_xmit_lock_key#4); [ 62.077359][ C1] lock(&p->alloc_lock); [ 62.077565][ C1] [ 62.077686][ C1] lock(&qdisc_xmit_lock_key#4); [ 62.077846][ C1] [ 62.077846][ C1] *** DEADLOCK *** [ 62.077846][ C1] [ 62.078091][ C1] 3 locks held by swapper/1/0: [ 62.078256][ C1] #0: ffffffff8a766360 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1f8/0x670 [ 62.078543][ C1] #1: ffffffff8a766300 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x1e5/0x16e0 [ 62.078835][ C1] #2: ffff88800901a258 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock){+...}-{2:2}, at: __dev_xmit_skb+0x792/0xfa0 [ 62.079206][ C1] [ 62.079206][ C1] the shortest dependencies between 2nd lock and 1st lock: [ 62.079489][ C1] -> (&p->alloc_lock){+.+.}-{2:2} { [ 62.079659][ C1] HARDIRQ-ON-W at: [ 62.079787][ C1] __lock_acquire+0x797/0x1570 [ 62.079997][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.080203][ C1] _raw_spin_lock+0x30/0x40 [ 62.080416][ C1] set_mems_allowed+0x1d/0x210 [ 62.080621][ C1] kernel_init_freeable+0x72/0x310 [ 62.080826][ C1] kernel_init+0x20/0x200 [ 62.081037][ C1] ret_from_fork+0x31/0x70 [ 62.081246][ C1] ret_from_fork_asm+0x1a/0x30 [ 62.081449][ C1] SOFTIRQ-ON-W at: [ 62.081573][ C1] __lock_acquire+0x797/0x1570 [ 62.081780][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.081987][ C1] _raw_spin_lock+0x30/0x40 [ 62.082189][ C1] set_mems_allowed+0x1d/0x210 [ 62.082397][ C1] kernel_init_freeable+0x72/0x310 [ 62.082601][ C1] kernel_init+0x20/0x200 [ 62.082809][ C1] ret_from_fork+0x31/0x70 [ 62.083016][ C1] ret_from_fork_asm+0x1a/0x30 [ 62.083221][ C1] INITIAL USE at: [ 62.083344][ C1] __lock_acquire+0x797/0x1570 [ 62.083551][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.083758][ C1] _raw_spin_lock+0x30/0x40 [ 62.083971][ C1] set_mems_allowed+0x1d/0x210 [ 62.084181][ C1] kernel_init_freeable+0x72/0x310 [ 62.084383][ C1] kernel_init+0x20/0x200 [ 62.084591][ C1] ret_from_fork+0x31/0x70 [ 62.084796][ C1] ret_from_fork_asm+0x1a/0x30 [ 62.085003][ C1] } [ 62.085086][ C1] ... key at: [] __key.388+0x0/0x40 [ 62.085350][ C1] ... acquired at: [ 62.085473][ C1] __lock_acquire+0xaf0/0x1570 [ 62.085636][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.085801][ C1] _raw_spin_lock+0x30/0x40 [ 62.085964][ C1] __get_task_comm+0x27/0x70 [ 62.086129][ C1] ref_tracker_alloc+0x2ee/0x490 [ 62.086293][ C1] ___neigh_create+0x11e/0xe10 [ 62.086466][ C1] ip6_finish_output2+0xabd/0xf60 [ 62.086639][ C1] ip6_finish_output+0x553/0xdf0 [ 62.086799][ C1] ip6_output+0x1f3/0x770 [ 62.086958][ C1] ip6tunnel_xmit.constprop.0+0x16b/0x360 [ 62.087161][ C1] ip6_tnl_xmit+0x1266/0x3490 [ 62.087327][ C1] ip6erspan_tunnel_xmit+0xf63/0x2ce0 [ip6_gre] [ 62.087536][ C1] dev_hard_start_xmit+0x10e/0x360 [ 62.087710][ C1] sch_direct_xmit+0x203/0x11c0 [ 62.087878][ C1] __dev_xmit_skb+0x936/0xfa0 [ 62.088049][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 62.088214][ C1] ip6_finish_output2+0x512/0xf60 [ 62.088383][ C1] ip6_finish_output+0x553/0xdf0 [ 62.088546][ C1] ip6_output+0x1f3/0x770 [ 62.088712][ C1] NF_HOOK.constprop.0+0xe2/0x680 [ 62.088874][ C1] ndisc_send_skb+0x8e0/0x1230 [ 62.089039][ C1] ndisc_send_ns+0x9e/0x100 [ 62.089211][ C1] addrconf_dad_work+0x7dd/0xab0 [ 62.089385][ C1] process_one_work+0xe2c/0x1730 [ 62.089548][ C1] worker_thread+0x587/0xd30 [ 62.089712][ C1] kthread+0x28a/0x350 [ 62.089836][ C1] ret_from_fork+0x31/0x70 [ 62.089998][ C1] ret_from_fork_asm+0x1a/0x30 [ 62.090171][ C1] [ 62.090255][ C1] -> (&qdisc_xmit_lock_key#4){+.-.}-{2:2} { [ 62.090462][ C1] HARDIRQ-ON-W at: [ 62.090585][ C1] __lock_acquire+0x797/0x1570 [ 62.090801][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.091006][ C1] _raw_spin_lock+0x30/0x40 [ 62.091210][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 62.091413][ C1] __dev_xmit_skb+0x936/0xfa0 [ 62.091625][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 62.091831][ C1] ip6_finish_output2+0x512/0xf60 [ 62.092038][ C1] ip6_finish_output+0x553/0xdf0 [ 62.092240][ C1] ip6_output+0x1f3/0x770 [ 62.092445][ C1] NF_HOOK.constprop.0+0xe2/0x680 [ 62.092651][ C1] mld_sendpack+0x62d/0xbc0 [ 62.092856][ C1] mld_ifc_work+0x32/0x1f0 [ 62.093063][ C1] process_one_work+0xe2c/0x1730 [ 62.093271][ C1] worker_thread+0x587/0xd30 [ 62.093474][ C1] kthread+0x28a/0x350 [ 62.093640][ C1] ret_from_fork+0x31/0x70 [ 62.093842][ C1] ret_from_fork_asm+0x1a/0x30 [ 62.094060][ C1] IN-SOFTIRQ-W at: [ 62.094185][ C1] __lock_acquire+0x797/0x1570 [ 62.094397][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.094608][ C1] _raw_spin_lock+0x30/0x40 [ 62.094812][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 62.095018][ C1] __dev_xmit_skb+0x936/0xfa0 [ 62.095233][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 62.095436][ C1] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 62.095681][ C1] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 62.095935][ C1] tcf_action_exec.part.0+0x112/0x3d0 [ 62.096142][ C1] __tcf_classify.constprop.0+0x17e/0x4e0 [ 62.096387][ C1] tcf_classify+0x72/0xa0 [ 62.096594][ C1] tc_run+0x2be/0x700 [ 62.096762][ C1] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 62.097004][ C1] __netif_receive_skb_one_core+0xaf/0x1b0 [ 62.097253][ C1] process_backlog+0x103/0x670 [ 62.097456][ C1] __napi_poll.constprop.0+0xa2/0x460 [ 62.097661][ C1] net_rx_action+0x49f/0xc00 [ 62.097873][ C1] __do_softirq+0x1f8/0x5df [ 62.098076][ C1] irq_exit_rcu+0x97/0xc0 [ 62.098282][ C1] sysvec_apic_timer_interrupt+0x75/0x80 [ 62.098535][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 62.098781][ C1] default_idle+0xf/0x20 [ 62.098986][ C1] default_idle_call+0x6d/0xb0 [ 62.099193][ C1] cpuidle_idle_call+0x1f4/0x280 [ 62.099405][ C1] do_idle+0xf9/0x160 [ 62.099568][ C1] cpu_startup_entry+0x54/0x60 [ 62.099768][ C1] start_secondary+0x21c/0x2b0 [ 62.099978][ C1] common_startup_64+0x12c/0x138 [ 62.100181][ C1] INITIAL USE at: [ 62.100303][ C1] __lock_acquire+0x797/0x1570 [ 62.100513][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.100719][ C1] _raw_spin_lock+0x30/0x40 [ 62.100920][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 62.101121][ C1] __dev_xmit_skb+0x936/0xfa0 [ 62.101334][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 62.101537][ C1] ip6_finish_output2+0x512/0xf60 [ 62.101741][ C1] ip6_finish_output+0x553/0xdf0 [ 62.101943][ C1] ip6_output+0x1f3/0x770 [ 62.102150][ C1] NF_HOOK.constprop.0+0xe2/0x680 [ 62.102355][ C1] mld_sendpack+0x62d/0xbc0 [ 62.102565][ C1] mld_ifc_work+0x32/0x1f0 [ 62.102771][ C1] process_one_work+0xe2c/0x1730 [ 62.102976][ C1] worker_thread+0x587/0xd30 [ 62.103182][ C1] kthread+0x28a/0x350 [ 62.103349][ C1] ret_from_fork+0x31/0x70 [ 62.103556][ C1] ret_from_fork_asm+0x1a/0x30 [ 62.103763][ C1] } [ 62.103845][ C1] ... key at: [] qdisc_xmit_lock_key.143+0x0/0xffffffffffffb940 [ip6_gre] [ 62.104209][ C1] ... acquired at: [ 62.104329][ C1] mark_lock+0x28d/0x3e0 [ 62.104493][ C1] mark_usage+0xd9/0x2a0 [ 62.104655][ C1] __lock_acquire+0x797/0x1570 [ 62.104817][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.104981][ C1] _raw_spin_lock+0x30/0x40 [ 62.105147][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 62.105310][ C1] __dev_xmit_skb+0x936/0xfa0 [ 62.105471][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 62.105636][ C1] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 62.105844][ C1] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 62.106049][ C1] tcf_action_exec.part.0+0x112/0x3d0 [ 62.106213][ C1] __tcf_classify.constprop.0+0x17e/0x4e0 [ 62.106416][ C1] tcf_classify+0x72/0xa0 [ 62.106582][ C1] tc_run+0x2be/0x700 [ 62.106704][ C1] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 62.106910][ C1] __netif_receive_skb_one_core+0xaf/0x1b0 [ 62.107111][ C1] process_backlog+0x103/0x670 [ 62.107272][ C1] __napi_poll.constprop.0+0xa2/0x460 [ 62.107434][ C1] net_rx_action+0x49f/0xc00 [ 62.107604][ C1] __do_softirq+0x1f8/0x5df [ 62.107862][ C1] irq_exit_rcu+0x97/0xc0 [ 62.108029][ C1] sysvec_apic_timer_interrupt+0x75/0x80 [ 62.108235][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 62.108436][ C1] default_idle+0xf/0x20 [ 62.108688][ C1] default_idle_call+0x6d/0xb0 [ 62.108850][ C1] cpuidle_idle_call+0x1f4/0x280 [ 62.109013][ C1] do_idle+0xf9/0x160 [ 62.109135][ C1] cpu_startup_entry+0x54/0x60 [ 62.109386][ C1] start_secondary+0x21c/0x2b0 [ 62.109551][ C1] common_startup_64+0x12c/0x138 [ 62.109717][ C1] [ 62.109798][ C1] [ 62.109798][ C1] stack backtrace: [ 62.110007][ C1] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.9.0-rc1-virtme #1 [ 62.110434][ C1] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 62.110792][ C1] Call Trace: [ 62.111016][ C1] [ 62.111108][ C1] dump_stack_lvl+0x82/0xd0 [ 62.111277][ C1] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 62.111480][ C1] ? kernel_text_address+0x17/0xe0 [ 62.111645][ C1] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 62.111941][ C1] ? __pfx_usage_skip+0x10/0x10 [ 62.112102][ C1] ? __pfx_usage_match+0x10/0x10 [ 62.112271][ C1] ? arch_stack_walk+0x8c/0xf0 [ 62.112434][ C1] mark_lock_irq+0x4cd/0xa10 [ 62.112693][ C1] ? common_startup_64+0x12c/0x138 [ 62.112854][ C1] ? __pfx_mark_lock_irq+0x10/0x10 [ 62.113016][ C1] ? __pfx_stack_trace_save+0x10/0x10 [ 62.113179][ C1] ? save_trace+0x8f/0x5b0 [ 62.113432][ C1] mark_lock+0x28d/0x3e0 [ 62.113556][ C1] mark_usage+0xd9/0x2a0 [ 62.113680][ C1] __lock_acquire+0x797/0x1570 [ 62.113843][ C1] ? tcf_mirred_act+0x302/0x780 [act_mirred] [ 62.114058][ C1] ? arch_stack_walk+0x68/0xf0 [ 62.114311][ C1] lock_acquire.part.0+0xeb/0x330 [ 62.114470][ C1] ? sch_direct_xmit+0x2a1/0x11c0 [ 62.114635][ C1] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 62.114799][ C1] ? trace_lock_acquire+0x135/0x1c0 [ 62.115060][ C1] ? sch_direct_xmit+0x2a1/0x11c0 [ 62.115226][ C1] ? lock_acquire+0x32/0xc0 [ 62.115389][ C1] ? sch_direct_xmit+0x2a1/0x11c0 [ 62.115553][ C1] _raw_spin_lock+0x30/0x40 [ 62.115816][ C1] ? sch_direct_xmit+0x2a1/0x11c0 [ 62.115978][ C1] sch_direct_xmit+0x2a1/0x11c0 [ 62.116142][ C1] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 62.116304][ C1] ? __pfx_sch_direct_xmit+0x10/0x10 [ 62.116557][ C1] ? __dev_xmit_skb+0x792/0xfa0 [ 62.116720][ C1] ? lock_acquire+0x32/0xc0 [ 62.116880][ C1] ? __dev_xmit_skb+0x792/0xfa0 [ 62.117045][ C1] __dev_xmit_skb+0x936/0xfa0 [ 62.117206][ C1] ? __pfx___dev_xmit_skb+0x10/0x10 [ 62.117458][ C1] ? __dev_queue_xmit+0x1e5/0x16e0 [ 62.117619][ C1] ? lock_acquire+0x32/0xc0 [ 62.117784][ C1] ? __dev_queue_xmit+0x1e5/0x16e0 [ 62.117944][ C1] __dev_queue_xmit+0x8be/0x16e0 [ 62.118205][ C1] ? lockdep_hardirqs_on_prepare.part.0+0x14f/0x370 [ 62.118406][ C1] ? __pfx___dev_queue_xmit+0x10/0x10 [ 62.118567][ C1] ? trace_kmem_cache_alloc+0x2a/0xd0 [ 62.118733][ C1] ? kmem_cache_alloc+0x132/0x270 [ 62.118896][ C1] ? __copy_skb_header+0xaf/0x490 [ 62.119060][ C1] ? __skb_clone+0x57a/0x760 [ 62.119227][ C1] tcf_mirred_to_dev+0x787/0xdf0 [act_mirred] [ 62.119434][ C1] tcf_mirred_act+0x33b/0x780 [act_mirred] [ 62.119732][ C1] tcf_action_exec.part.0+0x112/0x3d0 [ 62.119894][ C1] ? mall_classify+0x173/0x240 [cls_matchall] [ 62.120101][ C1] __tcf_classify.constprop.0+0x17e/0x4e0 [ 62.120266][ C1] ? update_sd_lb_stats.constprop.0+0x9e4/0xd40 [ 62.120562][ C1] tcf_classify+0x72/0xa0 [ 62.120685][ C1] ? __pfx_tcf_classify+0x10/0x10 [ 62.120848][ C1] ? __pfx_update_sd_lb_stats.constprop.0+0x10/0x10 [ 62.121056][ C1] ? validate_chain+0x130/0x9b0 [ 62.121308][ C1] tc_run+0x2be/0x700 [ 62.121432][ C1] ? __pfx_tc_run+0x10/0x10 [ 62.121594][ C1] ? hlock_class+0x4e/0x130 [ 62.121761][ C1] __netif_receive_skb_core.constprop.0+0x10ea/0x2d20 [ 62.122060][ C1] ? __lock_acquire+0xaf0/0x1570 [ 62.122225][ C1] ? __pfx___netif_receive_skb_core.constprop.0+0x10/0x10 [ 62.122427][ C1] ? lock_acquire.part.0+0xeb/0x330 [ 62.122589][ C1] ? process_backlog+0x1f8/0x670 [ 62.122932][ C1] __netif_receive_skb_one_core+0xaf/0x1b0 [ 62.123135][ C1] ? __pfx___netif_receive_skb_one_core+0x10/0x10 [ 62.123339][ C1] ? process_backlog+0x1f8/0x670 [ 62.123501][ C1] ? lock_acquire+0x32/0xc0 [ 62.123753][ C1] ? process_backlog+0x1f8/0x670 [ 62.123916][ C1] process_backlog+0x103/0x670 [ 62.124088][ C1] ? __lock_release+0x103/0x460 [ 62.124252][ C1] ? rebalance_domains+0x278/0xe20 [ 62.124507][ C1] __napi_poll.constprop.0+0xa2/0x460 [ 62.124671][ C1] net_rx_action+0x49f/0xc00 [ 62.124840][ C1] ? __pfx_net_rx_action+0x10/0x10 [ 62.125342][ C1] ? tmigr_cpu_activate+0x280/0x290 [ 62.125597][ C1] ? __pfx_rebalance_domains+0x10/0x10 [ 62.125762][ C1] __do_softirq+0x1f8/0x5df [ 62.125926][ C1] irq_exit_rcu+0x97/0xc0 [ 62.126051][ C1] sysvec_apic_timer_interrupt+0x75/0x80 [ 62.126214][ C1] [ 62.126394][ C1] [ 62.126476][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 62.126682][ C1] RIP: 0010:default_idle+0xf/0x20 [ 62.126855][ C1] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d d3 fb 32 00 fb f4 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 [ 62.127527][ C1] RSP: 0018:ffffc9000014fdf8 EFLAGS: 00000242 [ 62.127736][ C1] RAX: 00000000000d1f29 RBX: 1ffff92000029fc1 RCX: ffffffff89531ea5 [ 62.128088][ C1] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff86cac1d4 [ 62.128332][ C1] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1006c1709c [ 62.128574][ C1] R10: ffff8880360b84e3 R11: ffff8880360ad9b0 R12: 0000000000000000 [ 62.128912][ C1] R13: ffff888001b8a300 R14: dffffc0000000000 R15: 0000000000000000 [ 62.129161][ C1] ? ct_kernel_exit.constprop.0+0xc5/0xf0 [ 62.129330][ C1] ? cpuidle_idle_call+0x1f4/0x280 [ 62.129585][ C1] default_idle_call+0x6d/0xb0 [ 62.129748][ C1] cpuidle_idle_call+0x1f4/0x280 [ 62.129912][ C1] ? __pfx_cpuidle_idle_call+0x10/0x10 [ 62.130076][ C1] ? tsc_verify_tsc_adjust+0x5e/0x2b0 [ 62.130333][ C1] do_idle+0xf9/0x160 [ 62.130456][ C1] cpu_startup_entry+0x54/0x60 [ 62.130617][ C1] start_secondary+0x21c/0x2b0 [ 62.130781][ C1] ? __pfx_start_secondary+0x10/0x10 [ 62.130943][ C1] common_startup_64+0x12c/0x138 [ 62.131198][ C1]