[ 34.142991][ T387] fab-br0: port 1(fbond) entered blocking state [ 34.143275][ T387] fab-br0: port 1(fbond) entered disabled state [ 34.143492][ T387] fbond: entered allmulticast mode [ 34.145010][ T387] fbond: entered promiscuous mode [ 34.288226][ T390] veth1-bond: entered promiscuous mode [ 34.289072][ T390] veth1-bond: entered allmulticast mode [ 34.289863][ T390] fbond: (slave veth1-bond): Enslaving as a backup interface with a down link [ 34.413147][ T391] veth2-bond: entered promiscuous mode [ 34.413513][ T391] veth2-bond: entered allmulticast mode [ 34.413980][ T391] fbond: (slave veth2-bond): Enslaving as a backup interface with a down link [ 34.569698][ T73] fbond: Warning: No 802.3ad response from the link partner for any adapters in the bond [ 34.667985][ T73] fbond: (slave veth1-bond): link status definitely up, 10000 Mbps full duplex [ 34.668622][ T73] fbond: (slave veth2-bond): link status definitely up, 10000 Mbps full duplex [ 34.668886][ T73] fbond: active interface up! [ 34.757798][ T394] fab-br0: port 1(fbond) entered blocking state [ 34.758205][ T394] fab-br0: port 1(fbond) entered forwarding state [ 34.913816][ C3] [ 34.913913][ C3] ======================================================== [ 34.914117][ C3] WARNING: possible irq lock inversion dependency detected [ 34.914316][ C3] 6.9.0-rc1-virtme #1 Not tainted [ 34.914476][ C3] -------------------------------------------------------- [ 34.914671][ C3] swapper/3/0 just changed the state of lock: [ 34.914834][ C3] ffff88802c65ac18 (&br->lock){+.-.}-{2:2}, at: br_forward_delay_timer_expired+0x48/0x440 [ 34.915071][ C3] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 34.915265][ C3] (&p->alloc_lock){+.+.}-{2:2} [ 34.915270][ C3] [ 34.915270][ C3] [ 34.915270][ C3] and interrupts could create inverse lock ordering between them. [ 34.915270][ C3] [ 34.915745][ C3] [ 34.915745][ C3] other info that might help us debug this: [ 34.915954][ C3] Chain exists of: [ 34.915954][ C3] &br->lock --> lweventlist_lock --> &p->alloc_lock [ 34.915954][ C3] [ 34.916252][ C3] Possible interrupt unsafe locking scenario: [ 34.916252][ C3] [ 34.916442][ C3] CPU0 CPU1 [ 34.916572][ C3] ---- ---- [ 34.916702][ C3] lock(&p->alloc_lock); [ 34.916804][ C3] local_irq_disable(); [ 34.916963][ C3] lock(&br->lock); [ 34.917129][ C3] lock(lweventlist_lock); [ 34.917318][ C3] [ 34.917417][ C3] lock(&br->lock); [ 34.917517][ C3] [ 34.917517][ C3] *** DEADLOCK *** [ 34.917517][ C3] [ 34.917756][ C3] 1 lock held by swapper/3/0: [ 34.917897][ C3] #0: ffffc90000288d68 ((&p->forward_delay_timer)){+.-.}-{0:0}, at: call_timer_fn+0xe8/0x230 [ 34.918184][ C3] [ 34.918184][ C3] the shortest dependencies between 2nd lock and 1st lock: [ 34.918435][ C3] -> (&p->alloc_lock){+.+.}-{2:2} { [ 34.918572][ C3] HARDIRQ-ON-W at: [ 34.918695][ C3] __lock_acquire+0x797/0x1570 [ 34.918886][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.919071][ C3] _raw_spin_lock+0x30/0x40 [ 34.919238][ C3] set_mems_allowed+0x1d/0x210 [ 34.919425][ C3] kernel_init_freeable+0x72/0x310 [ 34.919618][ C3] kernel_init+0x20/0x200 [ 34.919788][ C3] ret_from_fork+0x31/0x70 [ 34.919955][ C3] ret_from_fork_asm+0x1a/0x30 [ 34.920123][ C3] SOFTIRQ-ON-W at: [ 34.920231][ C3] __lock_acquire+0x797/0x1570 [ 34.920415][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.920598][ C3] _raw_spin_lock+0x30/0x40 [ 34.920764][ C3] set_mems_allowed+0x1d/0x210 [ 34.920941][ C3] kernel_init_freeable+0x72/0x310 [ 34.921109][ C3] kernel_init+0x20/0x200 [ 34.921290][ C3] ret_from_fork+0x31/0x70 [ 34.921482][ C3] ret_from_fork_asm+0x1a/0x30 [ 34.921658][ C3] INITIAL USE at: [ 34.921786][ C3] __lock_acquire+0x797/0x1570 [ 34.921952][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.922134][ C3] _raw_spin_lock+0x30/0x40 [ 34.922316][ C3] set_mems_allowed+0x1d/0x210 [ 34.922486][ C3] kernel_init_freeable+0x72/0x310 [ 34.922676][ C3] kernel_init+0x20/0x200 [ 34.922841][ C3] ret_from_fork+0x31/0x70 [ 34.923008][ C3] ret_from_fork_asm+0x1a/0x30 [ 34.923174][ C3] } [ 34.923243][ C3] ... key at: [] __key.387+0x0/0x40 [ 34.923462][ C3] ... acquired at: [ 34.923571][ C3] __lock_acquire+0xaf0/0x1570 [ 34.923721][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.923862][ C3] _raw_spin_lock+0x30/0x40 [ 34.923997][ C3] __get_task_comm+0x27/0x70 [ 34.924149][ C3] ref_tracker_alloc+0x2ee/0x490 [ 34.924283][ C3] linkwatch_fire_event+0x196/0x200 [ 34.924431][ C3] bond_create+0xba/0x110 [ 34.924604][ C3] bonding_init+0x91/0xe0 [ 34.924758][ C3] do_one_initcall+0x8d/0x1e0 [ 34.924914][ C3] do_initcalls+0x1b2/0x3e0 [ 34.925046][ C3] kernel_init_freeable+0x232/0x310 [ 34.925198][ C3] kernel_init+0x20/0x200 [ 34.925337][ C3] ret_from_fork+0x31/0x70 [ 34.925486][ C3] ret_from_fork_asm+0x1a/0x30 [ 34.925626][ C3] [ 34.925701][ C3] -> (lweventlist_lock){....}-{2:2} { [ 34.925870][ C3] INITIAL USE at: [ 34.926001][ C3] __lock_acquire+0x797/0x1570 [ 34.926187][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.926370][ C3] _raw_spin_lock_irqsave+0x3f/0x60 [ 34.926548][ C3] linkwatch_fire_event+0x74/0x200 [ 34.926730][ C3] bond_create+0xba/0x110 [ 34.926914][ C3] bonding_init+0x91/0xe0 [ 34.927095][ C3] do_one_initcall+0x8d/0x1e0 [ 34.927288][ C3] do_initcalls+0x1b2/0x3e0 [ 34.927479][ C3] kernel_init_freeable+0x232/0x310 [ 34.927676][ C3] kernel_init+0x20/0x200 [ 34.927841][ C3] ret_from_fork+0x31/0x70 [ 34.928042][ C3] ret_from_fork_asm+0x1a/0x30 [ 34.928211][ C3] } [ 34.928289][ C3] ... key at: [] lweventlist_lock+0x18/0x60 [ 34.928504][ C3] ... acquired at: [ 34.928602][ C3] __lock_acquire+0xaf0/0x1570 [ 34.928761][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.928911][ C3] _raw_spin_lock_irqsave+0x3f/0x60 [ 34.929049][ C3] linkwatch_fire_event+0x74/0x200 [ 34.929198][ C3] br_stp_change_bridge_id+0x25f/0x360 [ 34.929334][ C3] br_stp_recalculate_bridge_id+0x275/0x330 [ 34.929525][ C3] br_add_if+0x9cd/0x1520 [ 34.929659][ C3] do_setlink+0x84b/0x21e0 [ 34.929800][ C3] __rtnl_newlink+0xaa7/0xd80 [ 34.929949][ C3] rtnl_newlink+0x63/0xa0 [ 34.930080][ C3] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 34.930221][ C3] netlink_rcv_skb+0x130/0x360 [ 34.930379][ C3] netlink_unicast+0x449/0x710 [ 34.930511][ C3] netlink_sendmsg+0x723/0xbe0 [ 34.930681][ C3] ____sys_sendmsg+0x7b2/0xa10 [ 34.930841][ C3] ___sys_sendmsg+0xee/0x170 [ 34.930983][ C3] __sys_sendmsg+0xcd/0x170 [ 34.931161][ C3] do_syscall_64+0xc6/0x1e0 [ 34.931315][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 34.931493][ C3] [ 34.931567][ C3] -> (&br->lock){+.-.}-{2:2} { [ 34.931719][ C3] HARDIRQ-ON-W at: [ 34.931819][ C3] __lock_acquire+0x797/0x1570 [ 34.932014][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.932180][ C3] _raw_spin_lock_bh+0x38/0x50 [ 34.932345][ C3] br_stp_enable_bridge+0x1f/0x2f0 [ 34.932510][ C3] br_dev_open+0xa2/0x120 [ 34.932678][ C3] __dev_open+0x221/0x430 [ 34.932844][ C3] __dev_change_flags+0x469/0x6c0 [ 34.933010][ C3] dev_change_flags+0x80/0x160 [ 34.933177][ C3] do_setlink+0x7d6/0x21e0 [ 34.933342][ C3] __rtnl_newlink+0xaa7/0xd80 [ 34.933508][ C3] rtnl_newlink+0x63/0xa0 [ 34.933673][ C3] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 34.933856][ C3] netlink_rcv_skb+0x130/0x360 [ 34.934022][ C3] netlink_unicast+0x449/0x710 [ 34.934192][ C3] netlink_sendmsg+0x723/0xbe0 [ 34.934357][ C3] ____sys_sendmsg+0x7b2/0xa10 [ 34.934523][ C3] ___sys_sendmsg+0xee/0x170 [ 34.934687][ C3] __sys_sendmsg+0xcd/0x170 [ 34.934851][ C3] do_syscall_64+0xc6/0x1e0 [ 34.935018][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 34.935235][ C3] IN-SOFTIRQ-W at: [ 34.935338][ C3] __lock_acquire+0x797/0x1570 [ 34.935506][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.935673][ C3] _raw_spin_lock+0x30/0x40 [ 34.935844][ C3] br_forward_delay_timer_expired+0x48/0x440 [ 34.936042][ C3] call_timer_fn+0x13b/0x230 [ 34.936314][ C3] __run_timers+0x545/0x810 [ 34.936520][ C3] run_timer_softirq+0xe8/0x1b0 [ 34.936691][ C3] __do_softirq+0x1f8/0x5df [ 34.936956][ C3] irq_exit_rcu+0x97/0xc0 [ 34.937138][ C3] sysvec_apic_timer_interrupt+0x75/0x80 [ 34.937353][ C3] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 34.937647][ C3] finish_task_switch.isra.0+0x21d/0x8c0 [ 34.937849][ C3] __schedule+0x6ea/0x1a20 [ 34.938033][ C3] schedule_idle+0x5e/0x90 [ 34.938282][ C3] do_idle+0x122/0x160 [ 34.938433][ C3] cpu_startup_entry+0x54/0x60 [ 34.938606][ C3] start_secondary+0x21c/0x2b0 [ 34.938783][ C3] common_startup_64+0x12c/0x138 [ 34.939062][ C3] INITIAL USE at: [ 34.939164][ C3] __lock_acquire+0x797/0x1570 [ 34.939358][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.939532][ C3] _raw_spin_lock_bh+0x38/0x50 [ 34.939813][ C3] br_stp_enable_bridge+0x1f/0x2f0 [ 34.939983][ C3] br_dev_open+0xa2/0x120 [ 34.940167][ C3] __dev_open+0x221/0x430 [ 34.940341][ C3] __dev_change_flags+0x469/0x6c0 [ 34.940523][ C3] dev_change_flags+0x80/0x160 [ 34.940710][ C3] do_setlink+0x7d6/0x21e0 [ 34.940903][ C3] __rtnl_newlink+0xaa7/0xd80 [ 34.941071][ C3] rtnl_newlink+0x63/0xa0 [ 34.941237][ C3] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 34.941400][ C3] netlink_rcv_skb+0x130/0x360 [ 34.941657][ C3] netlink_unicast+0x449/0x710 [ 34.941834][ C3] netlink_sendmsg+0x723/0xbe0 [ 34.942009][ C3] ____sys_sendmsg+0x7b2/0xa10 [ 34.942250][ C3] ___sys_sendmsg+0xee/0x170 [ 34.942413][ C3] __sys_sendmsg+0xcd/0x170 [ 34.942577][ C3] do_syscall_64+0xc6/0x1e0 [ 34.942742][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 34.943015][ C3] } [ 34.943082][ C3] ... key at: [] __key.6+0x0/0x40 [ 34.943248][ C3] ... acquired at: [ 34.943344][ C3] mark_lock+0x28d/0x3e0 [ 34.943551][ C3] mark_usage+0xd9/0x2a0 [ 34.943684][ C3] __lock_acquire+0x797/0x1570 [ 34.943816][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.943949][ C3] _raw_spin_lock+0x30/0x40 [ 34.944225][ C3] br_forward_delay_timer_expired+0x48/0x440 [ 34.944393][ C3] call_timer_fn+0x13b/0x230 [ 34.944525][ C3] __run_timers+0x545/0x810 [ 34.944656][ C3] run_timer_softirq+0xe8/0x1b0 [ 34.944861][ C3] __do_softirq+0x1f8/0x5df [ 34.944992][ C3] irq_exit_rcu+0x97/0xc0 [ 34.945123][ C3] sysvec_apic_timer_interrupt+0x75/0x80 [ 34.945286][ C3] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 34.945523][ C3] finish_task_switch.isra.0+0x21d/0x8c0 [ 34.945690][ C3] __schedule+0x6ea/0x1a20 [ 34.945828][ C3] schedule_idle+0x5e/0x90 [ 34.945960][ C3] do_idle+0x122/0x160 [ 34.946135][ C3] cpu_startup_entry+0x54/0x60 [ 34.946266][ C3] start_secondary+0x21c/0x2b0 [ 34.946410][ C3] common_startup_64+0x12c/0x138 [ 34.946542][ C3] [ 34.946628][ C3] [ 34.946628][ C3] stack backtrace: [ 34.946872][ C3] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.9.0-rc1-virtme #1 [ 34.947080][ C3] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 34.947484][ C3] Call Trace: [ 34.947589][ C3] [ 34.947658][ C3] dump_stack_lvl+0x82/0xd0 [ 34.947817][ C3] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 34.947990][ C3] ? common_startup_64+0x12c/0x138 [ 34.948216][ C3] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 34.948388][ C3] ? __pfx_usage_skip+0x10/0x10 [ 34.948537][ C3] ? __pfx_usage_match+0x10/0x10 [ 34.948670][ C3] ? arch_stack_walk+0x8c/0xf0 [ 34.948887][ C3] mark_lock_irq+0x4cd/0xa10 [ 34.949040][ C3] ? __pfx_mark_lock_irq+0x10/0x10 [ 34.949171][ C3] ? stack_trace_save+0x94/0xd0 [ 34.949321][ C3] ? add_chain_cache+0x248/0x8b0 [ 34.949537][ C3] ? save_trace+0x8f/0x5b0 [ 34.949684][ C3] mark_lock+0x28d/0x3e0 [ 34.949797][ C3] mark_usage+0xd9/0x2a0 [ 34.949908][ C3] __lock_acquire+0x797/0x1570 [ 34.950041][ C3] ? __lock_acquire+0xaf0/0x1570 [ 34.950264][ C3] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [ 34.950437][ C3] lock_acquire.part.0+0xeb/0x330 [ 34.950577][ C3] ? br_forward_delay_timer_expired+0x48/0x440 [ 34.950755][ C3] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 34.950997][ C3] ? trace_lock_acquire+0x135/0x1c0 [ 34.951129][ C3] ? br_forward_delay_timer_expired+0x48/0x440 [ 34.951291][ C3] ? lock_acquire+0x32/0xc0 [ 34.951422][ C3] ? br_forward_delay_timer_expired+0x48/0x440 [ 34.951660][ C3] _raw_spin_lock+0x30/0x40 [ 34.951804][ C3] ? br_forward_delay_timer_expired+0x48/0x440 [ 34.951971][ C3] br_forward_delay_timer_expired+0x48/0x440 [ 34.952232][ C3] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [ 34.952446][ C3] call_timer_fn+0x13b/0x230 [ 34.952584][ C3] ? call_timer_fn+0xe8/0x230 [ 34.952731][ C3] ? call_timer_fn+0xe8/0x230 [ 34.952961][ C3] ? __pfx_call_timer_fn+0x10/0x10 [ 34.953105][ C3] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 34.953273][ C3] __run_timers+0x545/0x810 [ 34.953404][ C3] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [ 34.953674][ C3] ? __pfx___run_timers+0x10/0x10 [ 34.953827][ C3] ? do_raw_spin_lock+0x131/0x270 [ 34.953982][ C3] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 34.954122][ C3] ? lock_acquire+0x32/0xc0 [ 34.954264][ C3] ? run_timer_softirq+0xe0/0x1b0 [ 34.954397][ C3] run_timer_softirq+0xe8/0x1b0 [ 34.954530][ C3] __do_softirq+0x1f8/0x5df [ 34.954685][ C3] irq_exit_rcu+0x97/0xc0 [ 34.954786][ C3] sysvec_apic_timer_interrupt+0x75/0x80 [ 34.955018][ C3] [ 34.955087][ C3] [ 34.955158][ C3] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 34.955344][ C3] RIP: 0010:finish_task_switch.isra.0+0x21d/0x8c0 [ 34.955516][ C3] Code: 4c 89 ff 48 c7 03 00 00 00 00 ff d2 0f 1f 00 4d 85 e4 75 ba 4c 89 ff e8 81 1c 76 02 e8 3c 98 2d 00 fb 65 48 8b 1d 93 a3 be 7a <66> 90 48 83 7d d0 00 74 55 65 48 8b 1d 82 a3 be 7a 48 8d bb 00 05 [ 34.956132][ C3] RSP: 0018:ffffc9000016fd08 EFLAGS: 00000206 [ 34.956317][ C3] RAX: 0000000000094a5f RBX: ffff888001b90040 RCX: 1ffffffff12e6f61 [ 34.956526][ C3] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff85452424 [ 34.956739][ C3] RBP: ffffc9000016fd48 R08: 0000000000000001 R09: fffffbfff12e77e2 [ 34.956969][ C3] R10: ffffffff8973bf17 R11: ffff8880361bdc40 R12: ffff8880361bda98 [ 34.957189][ C3] R13: ffff888005e82300 R14: ffff8880361bda80 R15: ffff8880361bda80 [ 34.957393][ C3] ? finish_task_switch.isra.0+0x214/0x8c0 [ 34.957659][ C3] ? finish_task_switch.isra.0+0x214/0x8c0 [ 34.957837][ C3] ? __switch_to+0x5ce/0xe50 [ 34.957991][ C3] __schedule+0x6ea/0x1a20 [ 34.958126][ C3] ? __pfx___schedule+0x10/0x10 [ 34.958360][ C3] ? trace_rcu_dyntick+0x112/0x180 [ 34.958510][ C3] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 34.958687][ C3] schedule_idle+0x5e/0x90 [ 34.958827][ C3] do_idle+0x122/0x160 [ 34.959003][ C3] cpu_startup_entry+0x54/0x60 [ 34.959135][ C3] start_secondary+0x21c/0x2b0 [ 34.959291][ C3] ? __pfx_start_secondary+0x10/0x10 [ 34.959425][ C3] common_startup_64+0x12c/0x138 [ 34.959660][ C3] [ 37.178984][ T431] fbond: left allmulticast mode [ 37.179168][ T431] veth1-bond: left allmulticast mode [ 37.179307][ T431] veth2-bond: left allmulticast mode [ 37.179445][ T431] fbond: left promiscuous mode [ 37.179582][ T431] veth1-bond: left promiscuous mode [ 37.179887][ T431] veth2-bond: left promiscuous mode [ 37.180268][ T431] fab-br0: port 1(fbond) entered disabled state [ 37.351022][ T432] fbond (unregistering): (slave veth1-bond): Removing an active aggregator [ 37.351848][ T432] fbond (unregistering): (slave veth1-bond): Releasing backup interface [ 37.378697][ T432] fbond (unregistering): (slave veth2-bond): Removing an active aggregator [ 37.379332][ T432] fbond (unregistering): (slave veth2-bond): Releasing backup interface [ 37.397354][ T432] fbond (unregistering): Released all slaves