[ 105.030669][ T1369] br1: port 1(veth1) entered blocking state [ 105.031518][ T1369] br1: port 1(veth1) entered disabled state [ 105.032715][ T1369] veth1: entered allmulticast mode [ 105.035302][ T1369] veth1: entered promiscuous mode [ 105.167134][ T8] br1: port 1(veth1) entered blocking state [ 105.167585][ T8] br1: port 1(veth1) entered forwarding state [ 105.293477][ T1371] br1: port 2(veth2) entered blocking state [ 105.294770][ T1371] br1: port 2(veth2) entered disabled state [ 105.295343][ T1371] veth2: entered allmulticast mode [ 105.297434][ T1371] veth2: entered promiscuous mode [ 105.433123][ T63] br1: port 2(veth2) entered blocking state [ 105.433486][ T63] br1: port 2(veth2) entered forwarding state [ 115.316012][ T1421] Mirror/redirect action on [ 120.650683][ C2] [ 120.650864][ C2] ======================================================== [ 120.651146][ C2] WARNING: possible irq lock inversion dependency detected [ 120.651431][ C2] 6.9.0-rc1-virtme #1 Not tainted [ 120.651604][ C2] -------------------------------------------------------- [ 120.651839][ C2] swapper/2/0 just changed the state of lock: [ 120.652036][ C2] ffff8880079c2c38 (&br->lock){+.-.}-{2:2}, at: br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 120.652432][ C2] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 120.652691][ C2] (&p->alloc_lock){+.+.}-{2:2} [ 120.652698][ C2] [ 120.652698][ C2] [ 120.652698][ C2] and interrupts could create inverse lock ordering between them. [ 120.652698][ C2] [ 120.653352][ C2] [ 120.653352][ C2] other info that might help us debug this: [ 120.653604][ C2] Chain exists of: [ 120.653604][ C2] &br->lock --> lweventlist_lock --> &p->alloc_lock [ 120.653604][ C2] [ 120.653988][ C2] Possible interrupt unsafe locking scenario: [ 120.653988][ C2] [ 120.654237][ C2] CPU0 CPU1 [ 120.654415][ C2] ---- ---- [ 120.654583][ C2] lock(&p->alloc_lock); [ 120.654722][ C2] local_irq_disable(); [ 120.654950][ C2] lock(&br->lock); [ 120.655181][ C2] lock(lweventlist_lock); [ 120.655390][ C2] [ 120.655527][ C2] lock(&br->lock); [ 120.655670][ C2] [ 120.655670][ C2] *** DEADLOCK *** [ 120.655670][ C2] [ 120.655906][ C2] 1 lock held by swapper/2/0: [ 120.656081][ C2] #0: ffffc90000230b68 ((&p->forward_delay_timer)){+.-.}-{0:0}, at: call_timer_fn+0xe8/0x230 [ 120.656412][ C2] [ 120.656412][ C2] the shortest dependencies between 2nd lock and 1st lock: [ 120.656700][ C2] -> (&p->alloc_lock){+.+.}-{2:2} { [ 120.656876][ C2] HARDIRQ-ON-W at: [ 120.657003][ C2] __lock_acquire+0x797/0x1570 [ 120.657236][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.657473][ C2] _raw_spin_lock+0x30/0x40 [ 120.657713][ C2] set_mems_allowed+0x1d/0x210 [ 120.657925][ C2] kernel_init_freeable+0x72/0x310 [ 120.658149][ C2] kernel_init+0x20/0x200 [ 120.658348][ C2] ret_from_fork+0x31/0x70 [ 120.658532][ C2] ret_from_fork_asm+0x1a/0x30 [ 120.658742][ C2] SOFTIRQ-ON-W at: [ 120.658855][ C2] __lock_acquire+0x797/0x1570 [ 120.659036][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.659211][ C2] _raw_spin_lock+0x30/0x40 [ 120.659395][ C2] set_mems_allowed+0x1d/0x210 [ 120.659591][ C2] kernel_init_freeable+0x72/0x310 [ 120.659788][ C2] kernel_init+0x20/0x200 [ 120.659975][ C2] ret_from_fork+0x31/0x70 [ 120.660154][ C2] ret_from_fork_asm+0x1a/0x30 [ 120.660327][ C2] INITIAL USE at: [ 120.660432][ C2] __lock_acquire+0x797/0x1570 [ 120.660620][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.660821][ C2] _raw_spin_lock+0x30/0x40 [ 120.661139][ C2] set_mems_allowed+0x1d/0x210 [ 120.661335][ C2] kernel_init_freeable+0x72/0x310 [ 120.661528][ C2] kernel_init+0x20/0x200 [ 120.661708][ C2] ret_from_fork+0x31/0x70 [ 120.661895][ C2] ret_from_fork_asm+0x1a/0x30 [ 120.662073][ C2] } [ 120.662158][ C2] ... key at: [] __key.387+0x0/0x40 [ 120.662395][ C2] ... acquired at: [ 120.662511][ C2] __lock_acquire+0xaf0/0x1570 [ 120.662673][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.662830][ C2] _raw_spin_lock+0x30/0x40 [ 120.662987][ C2] __get_task_comm+0x27/0x70 [ 120.663143][ C2] ref_tracker_alloc+0x2ee/0x490 [ 120.663300][ C2] linkwatch_fire_event+0x196/0x200 [ 120.663470][ C2] veth_newlink+0x367/0x830 [veth] [ 120.663637][ C2] rtnl_newlink_create+0x341/0x850 [ 120.663792][ C2] __rtnl_newlink+0xac9/0xd80 [ 120.663947][ C2] rtnl_newlink+0x63/0xa0 [ 120.664106][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 120.664260][ C2] netlink_rcv_skb+0x130/0x360 [ 120.664442][ C2] netlink_unicast+0x449/0x710 [ 120.664599][ C2] netlink_sendmsg+0x723/0xbe0 [ 120.664787][ C2] ____sys_sendmsg+0x7b2/0xa10 [ 120.664947][ C2] ___sys_sendmsg+0xee/0x170 [ 120.665135][ C2] __sys_sendmsg+0xcd/0x170 [ 120.665306][ C2] do_syscall_64+0xc6/0x1e0 [ 120.665465][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 120.665663][ C2] [ 120.665747][ C2] -> (lweventlist_lock){....}-{2:2} { [ 120.665929][ C2] INITIAL USE at: [ 120.666049][ C2] __lock_acquire+0x797/0x1570 [ 120.666276][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.666511][ C2] _raw_spin_lock_irqsave+0x3f/0x60 [ 120.666719][ C2] linkwatch_sync_dev+0x29/0x1f0 [ 120.666929][ C2] rtnl_getlink+0x4c9/0x970 [ 120.667130][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 120.667325][ C2] netlink_rcv_skb+0x130/0x360 [ 120.667541][ C2] netlink_unicast+0x449/0x710 [ 120.667746][ C2] netlink_sendmsg+0x723/0xbe0 [ 120.667966][ C2] __sys_sendto+0x37d/0x410 [ 120.668184][ C2] __x64_sys_sendto+0xe0/0x1c0 [ 120.668379][ C2] do_syscall_64+0xc6/0x1e0 [ 120.668577][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 120.668854][ C2] } [ 120.668924][ C2] ... key at: [] lweventlist_lock+0x18/0x60 [ 120.669162][ C2] ... acquired at: [ 120.669287][ C2] __lock_acquire+0xaf0/0x1570 [ 120.669466][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.669635][ C2] _raw_spin_lock_irqsave+0x3f/0x60 [ 120.669790][ C2] linkwatch_fire_event+0x74/0x200 [ 120.669981][ C2] br_stp_change_bridge_id+0x25f/0x360 [bridge] [ 120.670204][ C2] br_set_mac_address+0x1f3/0x260 [bridge] [ 120.670434][ C2] dev_set_mac_address+0x28a/0x410 [ 120.670598][ C2] dev_set_mac_address_user+0x31/0x50 [ 120.670752][ C2] do_setlink+0x17b4/0x21e0 [ 120.670932][ C2] rtnl_setlink+0x1f7/0x310 [ 120.671133][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 120.671313][ C2] netlink_rcv_skb+0x130/0x360 [ 120.671454][ C2] netlink_unicast+0x449/0x710 [ 120.671611][ C2] netlink_sendmsg+0x723/0xbe0 [ 120.671755][ C2] __sys_sendto+0x37d/0x410 [ 120.671912][ C2] __x64_sys_sendto+0xe0/0x1c0 [ 120.672067][ C2] do_syscall_64+0xc6/0x1e0 [ 120.672234][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 120.672407][ C2] [ 120.672501][ C2] -> (&br->lock){+.-.}-{2:2} { [ 120.672674][ C2] HARDIRQ-ON-W at: [ 120.672812][ C2] __lock_acquire+0x797/0x1570 [ 120.673015][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.673203][ C2] _raw_spin_lock_bh+0x38/0x50 [ 120.673396][ C2] recalculate_group_addr+0x55/0x130 [bridge] [ 120.673648][ C2] br_vlan_filter_toggle+0x77/0x130 [bridge] [ 120.673943][ C2] br_changelink+0x36c/0x1360 [bridge] [ 120.674177][ C2] br_dev_newlink+0xa5/0x100 [bridge] [ 120.674432][ C2] rtnl_newlink_create+0x341/0x850 [ 120.674622][ C2] __rtnl_newlink+0xac9/0xd80 [ 120.674809][ C2] rtnl_newlink+0x63/0xa0 [ 120.675006][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 120.675187][ C2] netlink_rcv_skb+0x130/0x360 [ 120.675391][ C2] netlink_unicast+0x449/0x710 [ 120.675574][ C2] netlink_sendmsg+0x723/0xbe0 [ 120.675774][ C2] ____sys_sendmsg+0x7b2/0xa10 [ 120.675981][ C2] ___sys_sendmsg+0xee/0x170 [ 120.676168][ C2] __sys_sendmsg+0xcd/0x170 [ 120.676370][ C2] do_syscall_64+0xc6/0x1e0 [ 120.676545][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 120.676790][ C2] IN-SOFTIRQ-W at: [ 120.676896][ C2] __lock_acquire+0x797/0x1570 [ 120.677098][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.677287][ C2] _raw_spin_lock+0x30/0x40 [ 120.677460][ C2] br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 120.677704][ C2] call_timer_fn+0x13b/0x230 [ 120.677885][ C2] __run_timers+0x545/0x810 [ 120.678068][ C2] timer_expire_remote+0x9b/0xe0 [ 120.678256][ C2] tmigr_handle_remote_cpu+0x270/0x430 [ 120.678439][ C2] tmigr_handle_remote_up+0x19f/0x2e0 [ 120.678638][ C2] tmigr_handle_remote+0x21d/0x3a0 [ 120.678840][ C2] __do_softirq+0x1f8/0x5df [ 120.679015][ C2] irq_exit_rcu+0x97/0xc0 [ 120.679229][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 120.679502][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 120.679714][ C2] default_idle+0xf/0x20 [ 120.679920][ C2] default_idle_call+0x6d/0xb0 [ 120.680126][ C2] cpuidle_idle_call+0x1f4/0x280 [ 120.680314][ C2] do_idle+0xf9/0x160 [ 120.680456][ C2] cpu_startup_entry+0x54/0x60 [ 120.680644][ C2] start_secondary+0x21c/0x2b0 [ 120.680830][ C2] common_startup_64+0x12c/0x138 [ 120.681039][ C2] INITIAL USE at: [ 120.681151][ C2] __lock_acquire+0x797/0x1570 [ 120.681355][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.681563][ C2] _raw_spin_lock_bh+0x38/0x50 [ 120.681740][ C2] recalculate_group_addr+0x55/0x130 [bridge] [ 120.681990][ C2] br_vlan_filter_toggle+0x77/0x130 [bridge] [ 120.682235][ C2] br_changelink+0x36c/0x1360 [bridge] [ 120.682444][ C2] br_dev_newlink+0xa5/0x100 [bridge] [ 120.682653][ C2] rtnl_newlink_create+0x341/0x850 [ 120.682833][ C2] __rtnl_newlink+0xac9/0xd80 [ 120.683005][ C2] rtnl_newlink+0x63/0xa0 [ 120.683176][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 120.683352][ C2] netlink_rcv_skb+0x130/0x360 [ 120.683548][ C2] netlink_unicast+0x449/0x710 [ 120.683722][ C2] netlink_sendmsg+0x723/0xbe0 [ 120.683892][ C2] ____sys_sendmsg+0x7b2/0xa10 [ 120.684071][ C2] ___sys_sendmsg+0xee/0x170 [ 120.684248][ C2] __sys_sendmsg+0xcd/0x170 [ 120.684424][ C2] do_syscall_64+0xc6/0x1e0 [ 120.684691][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 120.684897][ C2] } [ 120.684969][ C2] ... key at: [] __key.6+0x0/0xfffffffffffe5ac0 [bridge] [ 120.685240][ C2] ... acquired at: [ 120.685345][ C2] mark_lock+0x28d/0x3e0 [ 120.685484][ C2] mark_usage+0xd9/0x2a0 [ 120.685624][ C2] __lock_acquire+0x797/0x1570 [ 120.685761][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.685986][ C2] _raw_spin_lock+0x30/0x40 [ 120.686126][ C2] br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 120.686334][ C2] call_timer_fn+0x13b/0x230 [ 120.686488][ C2] __run_timers+0x545/0x810 [ 120.686632][ C2] timer_expire_remote+0x9b/0xe0 [ 120.686773][ C2] tmigr_handle_remote_cpu+0x270/0x430 [ 120.686910][ C2] tmigr_handle_remote_up+0x19f/0x2e0 [ 120.687088][ C2] tmigr_handle_remote+0x21d/0x3a0 [ 120.687240][ C2] __do_softirq+0x1f8/0x5df [ 120.687463][ C2] irq_exit_rcu+0x97/0xc0 [ 120.687603][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 120.687775][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 120.687979][ C2] default_idle+0xf/0x20 [ 120.688126][ C2] default_idle_call+0x6d/0xb0 [ 120.688298][ C2] cpuidle_idle_call+0x1f4/0x280 [ 120.688441][ C2] do_idle+0xf9/0x160 [ 120.688591][ C2] cpu_startup_entry+0x54/0x60 [ 120.688828][ C2] start_secondary+0x21c/0x2b0 [ 120.689000][ C2] common_startup_64+0x12c/0x138 [ 120.689152][ C2] [ 120.689261][ C2] [ 120.689261][ C2] stack backtrace: [ 120.689434][ C2] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.9.0-rc1-virtme #1 [ 120.689645][ C2] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 120.690000][ C2] Call Trace: [ 120.690200][ C2] [ 120.690305][ C2] dump_stack_lvl+0x82/0xd0 [ 120.690452][ C2] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 120.690668][ C2] ? common_startup_64+0x12c/0x138 [ 120.690812][ C2] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 120.691027][ C2] ? __pfx_usage_skip+0x10/0x10 [ 120.691166][ C2] ? __pfx_usage_match+0x10/0x10 [ 120.691304][ C2] ? arch_stack_walk+0x8c/0xf0 [ 120.691444][ C2] mark_lock_irq+0x4cd/0xa10 [ 120.691669][ C2] ? __pfx_mark_lock_irq+0x10/0x10 [ 120.691812][ C2] ? stack_trace_save+0x94/0xd0 [ 120.691961][ C2] ? add_chain_cache+0x248/0x8b0 [ 120.692122][ C2] ? save_trace+0x8f/0x5b0 [ 120.692261][ C2] mark_lock+0x28d/0x3e0 [ 120.692391][ C2] mark_usage+0xd9/0x2a0 [ 120.692497][ C2] __lock_acquire+0x797/0x1570 [ 120.692636][ C2] ? __lock_acquire+0xaf0/0x1570 [ 120.692783][ C2] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [bridge] [ 120.693057][ C2] lock_acquire.part.0+0xeb/0x330 [ 120.693197][ C2] ? br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 120.693438][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 120.693714][ C2] ? trace_lock_acquire+0x135/0x1c0 [ 120.693852][ C2] ? br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 120.694078][ C2] ? lock_acquire+0x32/0xc0 [ 120.694214][ C2] ? br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 120.694440][ C2] _raw_spin_lock+0x30/0x40 [ 120.694580][ C2] ? br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 120.694821][ C2] br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 120.695032][ C2] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [bridge] [ 120.695272][ C2] call_timer_fn+0x13b/0x230 [ 120.695415][ C2] ? call_timer_fn+0xe8/0x230 [ 120.695553][ C2] ? call_timer_fn+0xe8/0x230 [ 120.695798][ C2] ? __pfx_call_timer_fn+0x10/0x10 [ 120.695948][ C2] ? mark_lock+0x38/0x3e0 [ 120.696056][ C2] __run_timers+0x545/0x810 [ 120.696216][ C2] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [bridge] [ 120.696502][ C2] ? __pfx___run_timers+0x10/0x10 [ 120.696643][ C2] ? __lock_release+0x103/0x460 [ 120.696804][ C2] ? do_raw_spin_lock+0x131/0x270 [ 120.696942][ C2] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 120.697123][ C2] ? lock_acquire+0x32/0xc0 [ 120.697363][ C2] ? timer_expire_remote+0x93/0xe0 [ 120.697509][ C2] timer_expire_remote+0x9b/0xe0 [ 120.697651][ C2] tmigr_handle_remote_cpu+0x270/0x430 [ 120.697789][ C2] ? __pfx_tmigr_handle_remote_cpu+0x10/0x10 [ 120.697963][ C2] tmigr_handle_remote_up+0x19f/0x2e0 [ 120.698101][ C2] tmigr_handle_remote+0x21d/0x3a0 [ 120.698238][ C2] ? __pfx_tmigr_handle_remote+0x10/0x10 [ 120.698407][ C2] __do_softirq+0x1f8/0x5df [ 120.698640][ C2] irq_exit_rcu+0x97/0xc0 [ 120.698762][ C2] sysvec_apic_timer_interrupt+0x75/0x80 [ 120.698899][ C2] [ 120.698990][ C2] [ 120.699076][ C2] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 120.699276][ C2] RIP: 0010:default_idle+0xf/0x20 [ 120.699444][ C2] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d a3 30 35 00 fb f4 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 [ 120.700155][ C2] RSP: 0018:ffffc9000015fdf8 EFLAGS: 00000242 [ 120.700369][ C2] RAX: 00000000003cb351 RBX: 1ffff9200002bfc1 RCX: ffffffff9750e9d5 [ 120.700636][ C2] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff94eaa4a4 [ 120.700849][ C2] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1006c2709c [ 120.701069][ C2] R10: ffff8880361384e3 R11: ffff88803612d9b0 R12: 0000000000000000 [ 120.701311][ C2] R13: ffff888001b7c5c0 R14: dffffc0000000000 R15: 0000000000000000 [ 120.701663][ C2] ? ct_kernel_exit.constprop.0+0xc5/0xf0 [ 120.701803][ C2] ? cpuidle_idle_call+0x1f4/0x280 [ 120.701967][ C2] default_idle_call+0x6d/0xb0 [ 120.702107][ C2] cpuidle_idle_call+0x1f4/0x280 [ 120.702287][ C2] ? __pfx_cpuidle_idle_call+0x10/0x10 [ 120.702449][ C2] ? tsc_verify_tsc_adjust+0x5e/0x2b0 [ 120.702644][ C2] do_idle+0xf9/0x160 [ 120.702766][ C2] cpu_startup_entry+0x54/0x60 [ 120.703023][ C2] start_secondary+0x21c/0x2b0 [ 120.703191][ C2] ? __pfx_start_secondary+0x10/0x10 [ 120.703331][ C2] common_startup_64+0x12c/0x138 [ 120.703487][ C2] [ 151.090127][ T1594] br1: port 1(veth1) entered disabled state [ 151.192036][ T1595] br1: port 2(veth2) entered disabled state [ 151.287659][ T1596] veth2: left allmulticast mode [ 151.287854][ T1596] veth2: left promiscuous mode [ 151.288203][ T1596] br1: port 2(veth2) entered disabled state [ 151.289911][ T1596] veth1: left allmulticast mode [ 151.290092][ T1596] veth1: left promiscuous mode [ 151.290439][ T1596] br1: port 1(veth1) entered disabled state