[ 25.597443][ T237] ip (237) used greatest stack depth: 23408 bytes left [ 28.728672][ T281] 8021q: 802.1Q VLAN Support v1.8 [ 32.590150][ T323] GACT probability NOT on [ 34.001427][ T338] br10: port 1(veth1.10) entered blocking state [ 34.001809][ T338] br10: port 1(veth1.10) entered disabled state [ 34.002357][ T338] veth1.10: entered allmulticast mode [ 34.002540][ T338] veth1: entered allmulticast mode [ 34.004521][ T338] veth1.10: entered promiscuous mode [ 34.004707][ T338] veth1: entered promiscuous mode [ 34.517437][ T343] br11: port 1(veth1.11) entered blocking state [ 34.517715][ T343] br11: port 1(veth1.11) entered disabled state [ 34.517982][ T343] veth1.11: entered allmulticast mode [ 34.519796][ T343] veth1.11: entered promiscuous mode [ 35.412833][ T353] br10: port 2(veth2.10) entered blocking state [ 35.414033][ T353] br10: port 2(veth2.10) entered disabled state [ 35.414344][ T353] veth2.10: entered allmulticast mode [ 35.414517][ T353] veth2: entered allmulticast mode [ 35.416405][ T353] veth2.10: entered promiscuous mode [ 35.416584][ T353] veth2: entered promiscuous mode [ 35.955536][ T358] br11: port 2(veth2.11) entered blocking state [ 35.955810][ T358] br11: port 2(veth2.11) entered disabled state [ 35.957275][ T358] veth2.11: entered allmulticast mode [ 35.959141][ T358] veth2.11: entered promiscuous mode [ 36.496031][ T362] br10: port 2(veth2.10) entered blocking state [ 36.496519][ T362] br10: port 2(veth2.10) entered forwarding state [ 36.497216][ T362] br10: port 1(veth1.10) entered blocking state [ 36.497458][ T362] br10: port 1(veth1.10) entered forwarding state [ 36.613730][ T363] br11: port 2(veth2.11) entered blocking state [ 36.614021][ T363] br11: port 2(veth2.11) entered forwarding state [ 36.614465][ T363] br11: port 1(veth1.11) entered blocking state [ 36.614698][ T363] br11: port 1(veth1.11) entered forwarding state [ 51.559067][ C2] [ 51.559249][ C2] ======================================================== [ 51.559562][ C2] WARNING: possible irq lock inversion dependency detected [ 51.559916][ C2] 6.9.0-rc1-virtme #1 Not tainted [ 51.560138][ C2] -------------------------------------------------------- [ 51.560467][ C2] ksoftirqd/2/28 just changed the state of lock: [ 51.560741][ C2] ffff888005562c38 (&br->lock){+.-.}-{2:2}, at: br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 51.561326][ C2] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 51.561652][ C2] (&p->alloc_lock){+.+.}-{2:2} [ 51.561660][ C2] [ 51.561660][ C2] [ 51.561660][ C2] and interrupts could create inverse lock ordering between them. [ 51.561660][ C2] [ 51.562456][ C2] [ 51.562456][ C2] other info that might help us debug this: [ 51.562773][ C2] Chain exists of: [ 51.562773][ C2] &br->lock --> lweventlist_lock --> &p->alloc_lock [ 51.562773][ C2] [ 51.563263][ C2] Possible interrupt unsafe locking scenario: [ 51.563263][ C2] [ 51.563562][ C2] CPU0 CPU1 [ 51.563800][ C2] ---- ---- [ 51.564017][ C2] lock(&p->alloc_lock); [ 51.564186][ C2] local_irq_disable(); [ 51.564460][ C2] lock(&br->lock); [ 51.564646][ C2] lock(lweventlist_lock); [ 51.564828][ C2] [ 51.564956][ C2] lock(&br->lock); [ 51.565072][ C2] [ 51.565072][ C2] *** DEADLOCK *** [ 51.565072][ C2] [ 51.565295][ C2] 1 lock held by ksoftirqd/2/28: [ 51.565453][ C2] #0: ffffc90000207a20 ((&p->forward_delay_timer)){+.-.}-{0:0}, at: call_timer_fn+0xe8/0x230 [ 51.565768][ C2] [ 51.565768][ C2] the shortest dependencies between 2nd lock and 1st lock: [ 51.566020][ C2] -> (&p->alloc_lock){+.+.}-{2:2} { [ 51.566172][ C2] HARDIRQ-ON-W at: [ 51.566284][ C2] __lock_acquire+0x797/0x1570 [ 51.566468][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.566651][ C2] _raw_spin_lock+0x30/0x40 [ 51.566830][ C2] set_mems_allowed+0x1d/0x210 [ 51.567017][ C2] kernel_init_freeable+0x72/0x310 [ 51.567215][ C2] kernel_init+0x20/0x200 [ 51.567399][ C2] ret_from_fork+0x31/0x70 [ 51.567593][ C2] ret_from_fork_asm+0x1a/0x30 [ 51.567775][ C2] SOFTIRQ-ON-W at: [ 51.567884][ C2] __lock_acquire+0x797/0x1570 [ 51.568064][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.568250][ C2] _raw_spin_lock+0x30/0x40 [ 51.568432][ C2] set_mems_allowed+0x1d/0x210 [ 51.568615][ C2] kernel_init_freeable+0x72/0x310 [ 51.568801][ C2] kernel_init+0x20/0x200 [ 51.568981][ C2] ret_from_fork+0x31/0x70 [ 51.569157][ C2] ret_from_fork_asm+0x1a/0x30 [ 51.569344][ C2] INITIAL USE at: [ 51.569457][ C2] __lock_acquire+0x797/0x1570 [ 51.569635][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.569814][ C2] _raw_spin_lock+0x30/0x40 [ 51.569990][ C2] set_mems_allowed+0x1d/0x210 [ 51.570176][ C2] kernel_init_freeable+0x72/0x310 [ 51.570358][ C2] kernel_init+0x20/0x200 [ 51.570535][ C2] ret_from_fork+0x31/0x70 [ 51.570716][ C2] ret_from_fork_asm+0x1a/0x30 [ 51.570894][ C2] } [ 51.570970][ C2] ... key at: [] __key.387+0x0/0x40 [ 51.571191][ C2] ... acquired at: [ 51.571301][ C2] __lock_acquire+0xaf0/0x1570 [ 51.571459][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.571600][ C2] _raw_spin_lock+0x30/0x40 [ 51.571741][ C2] __get_task_comm+0x27/0x70 [ 51.571885][ C2] ref_tracker_alloc+0x2ee/0x490 [ 51.572031][ C2] linkwatch_fire_event+0x196/0x200 [ 51.572174][ C2] veth_newlink+0x367/0x830 [veth] [ 51.572325][ C2] rtnl_newlink_create+0x341/0x850 [ 51.572467][ C2] __rtnl_newlink+0xac9/0xd80 [ 51.572615][ C2] rtnl_newlink+0x63/0xa0 [ 51.572758][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 51.572904][ C2] netlink_rcv_skb+0x130/0x360 [ 51.573051][ C2] netlink_unicast+0x449/0x710 [ 51.573195][ C2] netlink_sendmsg+0x723/0xbe0 [ 51.573336][ C2] ____sys_sendmsg+0x7b2/0xa10 [ 51.573480][ C2] ___sys_sendmsg+0xee/0x170 [ 51.573624][ C2] __sys_sendmsg+0xcd/0x170 [ 51.573769][ C2] do_syscall_64+0xc6/0x1e0 [ 51.573922][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 51.574100][ C2] [ 51.574180][ C2] -> (lweventlist_lock){....}-{2:2} { [ 51.574329][ C2] INITIAL USE at: [ 51.574449][ C2] __lock_acquire+0x797/0x1570 [ 51.574633][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.574816][ C2] _raw_spin_lock_irqsave+0x3f/0x60 [ 51.575001][ C2] linkwatch_sync_dev+0x29/0x1f0 [ 51.575180][ C2] rtnl_getlink+0x4c9/0x970 [ 51.575359][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 51.575538][ C2] netlink_rcv_skb+0x130/0x360 [ 51.575722][ C2] netlink_unicast+0x449/0x710 [ 51.575901][ C2] netlink_sendmsg+0x723/0xbe0 [ 51.576082][ C2] __sys_sendto+0x37d/0x410 [ 51.576266][ C2] __x64_sys_sendto+0xe0/0x1c0 [ 51.576447][ C2] do_syscall_64+0xc6/0x1e0 [ 51.576629][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 51.576845][ C2] } [ 51.576916][ C2] ... key at: [] lweventlist_lock+0x18/0x60 [ 51.577134][ C2] ... acquired at: [ 51.577239][ C2] __lock_acquire+0xaf0/0x1570 [ 51.577384][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.577533][ C2] _raw_spin_lock_irqsave+0x3f/0x60 [ 51.577677][ C2] linkwatch_fire_event+0x74/0x200 [ 51.577820][ C2] br_stp_change_bridge_id+0x25f/0x360 [bridge] [ 51.578043][ C2] br_set_mac_address+0x1f3/0x260 [bridge] [ 51.578256][ C2] dev_set_mac_address+0x28a/0x410 [ 51.578405][ C2] dev_set_mac_address_user+0x31/0x50 [ 51.578550][ C2] do_setlink+0x17b4/0x21e0 [ 51.578692][ C2] rtnl_setlink+0x1f7/0x310 [ 51.578857][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 51.578987][ C2] netlink_rcv_skb+0x130/0x360 [ 51.579117][ C2] netlink_unicast+0x449/0x710 [ 51.579246][ C2] netlink_sendmsg+0x723/0xbe0 [ 51.579382][ C2] __sys_sendto+0x37d/0x410 [ 51.579518][ C2] __x64_sys_sendto+0xe0/0x1c0 [ 51.579650][ C2] do_syscall_64+0xc6/0x1e0 [ 51.579782][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 51.579950][ C2] [ 51.580015][ C2] -> (&br->lock){+.-.}-{2:2} { [ 51.580150][ C2] HARDIRQ-ON-W at: [ 51.580248][ C2] __lock_acquire+0x797/0x1570 [ 51.580416][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.580584][ C2] _raw_spin_lock_bh+0x38/0x50 [ 51.580747][ C2] br_set_mac_address+0xd2/0x260 [bridge] [ 51.580975][ C2] dev_set_mac_address+0x28a/0x410 [ 51.581157][ C2] dev_set_mac_address_user+0x31/0x50 [ 51.581318][ C2] do_setlink+0x17b4/0x21e0 [ 51.581484][ C2] rtnl_setlink+0x1f7/0x310 [ 51.581691][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 51.581875][ C2] netlink_rcv_skb+0x130/0x360 [ 51.582054][ C2] netlink_unicast+0x449/0x710 [ 51.582234][ C2] netlink_sendmsg+0x723/0xbe0 [ 51.582414][ C2] __sys_sendto+0x37d/0x410 [ 51.582593][ C2] __x64_sys_sendto+0xe0/0x1c0 [ 51.582771][ C2] do_syscall_64+0xc6/0x1e0 [ 51.582957][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 51.583169][ C2] IN-SOFTIRQ-W at: [ 51.583278][ C2] __lock_acquire+0x797/0x1570 [ 51.583456][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.583638][ C2] _raw_spin_lock+0x30/0x40 [ 51.583820][ C2] br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 51.584072][ C2] call_timer_fn+0x13b/0x230 [ 51.584259][ C2] __run_timers+0x545/0x810 [ 51.584451][ C2] timer_expire_remote+0x9b/0xe0 [ 51.584634][ C2] tmigr_handle_remote_cpu+0x270/0x430 [ 51.584816][ C2] tmigr_handle_remote_up+0x19f/0x2e0 [ 51.584996][ C2] tmigr_handle_remote+0x21d/0x3a0 [ 51.585174][ C2] __do_softirq+0x1f8/0x5df [ 51.585357][ C2] run_ksoftirqd+0x2e/0x60 [ 51.585537][ C2] smpboot_thread_fn+0x306/0x840 [ 51.585726][ C2] kthread+0x28a/0x350 [ 51.585871][ C2] ret_from_fork+0x31/0x70 [ 51.586053][ C2] ret_from_fork_asm+0x1a/0x30 [ 51.586233][ C2] INITIAL USE at: [ 51.586342][ C2] __lock_acquire+0x797/0x1570 [ 51.586528][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.586707][ C2] _raw_spin_lock_bh+0x38/0x50 [ 51.586887][ C2] br_set_mac_address+0xd2/0x260 [bridge] [ 51.587135][ C2] dev_set_mac_address+0x28a/0x410 [ 51.587316][ C2] dev_set_mac_address_user+0x31/0x50 [ 51.587575][ C2] do_setlink+0x17b4/0x21e0 [ 51.587756][ C2] rtnl_setlink+0x1f7/0x310 [ 51.587933][ C2] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 51.588189][ C2] netlink_rcv_skb+0x130/0x360 [ 51.588367][ C2] netlink_unicast+0x449/0x710 [ 51.588551][ C2] netlink_sendmsg+0x723/0xbe0 [ 51.588804][ C2] __sys_sendto+0x37d/0x410 [ 51.588988][ C2] __x64_sys_sendto+0xe0/0x1c0 [ 51.589168][ C2] do_syscall_64+0xc6/0x1e0 [ 51.589346][ C2] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 51.589636][ C2] } [ 51.589709][ C2] ... key at: [] __key.6+0x0/0xfffffffffffe5ac0 [bridge] [ 51.589991][ C2] ... acquired at: [ 51.590099][ C2] mark_lock+0x28d/0x3e0 [ 51.590312][ C2] mark_usage+0xd9/0x2a0 [ 51.590462][ C2] __lock_acquire+0x797/0x1570 [ 51.590609][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.590753][ C2] _raw_spin_lock+0x30/0x40 [ 51.590969][ C2] br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 51.591183][ C2] call_timer_fn+0x13b/0x230 [ 51.591326][ C2] __run_timers+0x545/0x810 [ 51.591470][ C2] timer_expire_remote+0x9b/0xe0 [ 51.591616][ C2] tmigr_handle_remote_cpu+0x270/0x430 [ 51.591764][ C2] tmigr_handle_remote_up+0x19f/0x2e0 [ 51.591907][ C2] tmigr_handle_remote+0x21d/0x3a0 [ 51.592048][ C2] __do_softirq+0x1f8/0x5df [ 51.592339][ C2] run_ksoftirqd+0x2e/0x60 [ 51.592482][ C2] smpboot_thread_fn+0x306/0x840 [ 51.592643][ C2] kthread+0x28a/0x350 [ 51.592745][ C2] ret_from_fork+0x31/0x70 [ 51.592875][ C2] ret_from_fork_asm+0x1a/0x30 [ 51.593090][ C2] [ 51.593155][ C2] [ 51.593155][ C2] stack backtrace: [ 51.593320][ C2] CPU: 2 PID: 28 Comm: ksoftirqd/2 Not tainted 6.9.0-rc1-virtme #1 [ 51.593520][ C2] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 51.593810][ C2] Call Trace: [ 51.593912][ C2] [ 51.593981][ C2] dump_stack_lvl+0x82/0xd0 [ 51.594199][ C2] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 51.594378][ C2] ? kernel_text_address+0x17/0xe0 [ 51.594533][ C2] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 51.594695][ C2] ? __pfx_usage_skip+0x10/0x10 [ 51.594920][ C2] ? __pfx_usage_match+0x10/0x10 [ 51.595048][ C2] ? arch_stack_walk+0x8c/0xf0 [ 51.595183][ C2] mark_lock_irq+0x4cd/0xa10 [ 51.595314][ C2] ? ret_from_fork_asm+0x1a/0x30 [ 51.595530][ C2] ? __pfx_mark_lock_irq+0x10/0x10 [ 51.595661][ C2] ? __pfx_stack_trace_save+0x10/0x10 [ 51.595796][ C2] ? add_chain_cache+0x248/0x8b0 [ 51.595926][ C2] ? save_trace+0x8f/0x5b0 [ 51.596059][ C2] mark_lock+0x28d/0x3e0 [ 51.596235][ C2] mark_usage+0xd9/0x2a0 [ 51.596337][ C2] __lock_acquire+0x797/0x1570 [ 51.596481][ C2] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [bridge] [ 51.596715][ C2] lock_acquire.part.0+0xeb/0x330 [ 51.596925][ C2] ? br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 51.597128][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 51.597279][ C2] ? trace_lock_acquire+0x135/0x1c0 [ 51.597413][ C2] ? br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 51.597687][ C2] ? lock_acquire+0x32/0xc0 [ 51.597819][ C2] ? br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 51.598022][ C2] _raw_spin_lock+0x30/0x40 [ 51.598153][ C2] ? br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 51.598441][ C2] br_forward_delay_timer_expired+0x48/0x440 [bridge] [ 51.598646][ C2] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [bridge] [ 51.598875][ C2] call_timer_fn+0x13b/0x230 [ 51.599078][ C2] ? call_timer_fn+0xe8/0x230 [ 51.599207][ C2] ? call_timer_fn+0xe8/0x230 [ 51.599342][ C2] ? __pfx_call_timer_fn+0x10/0x10 [ 51.599475][ C2] ? mark_lock+0x38/0x3e0 [ 51.599654][ C2] __run_timers+0x545/0x810 [ 51.599785][ C2] ? __pfx_br_forward_delay_timer_expired+0x10/0x10 [bridge] [ 51.600014][ C2] ? __pfx___run_timers+0x10/0x10 [ 51.600144][ C2] ? __lock_release+0x103/0x460 [ 51.600348][ C2] ? do_raw_spin_lock+0x131/0x270 [ 51.600481][ C2] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 51.600621][ C2] ? lock_acquire+0x32/0xc0 [ 51.600764][ C2] ? timer_expire_remote+0x93/0xe0 [ 51.600985][ C2] timer_expire_remote+0x9b/0xe0 [ 51.601118][ C2] tmigr_handle_remote_cpu+0x270/0x430 [ 51.601253][ C2] ? __pfx_tmigr_handle_remote_cpu+0x10/0x10 [ 51.601422][ C2] tmigr_handle_remote_up+0x19f/0x2e0 [ 51.601559][ C2] tmigr_handle_remote+0x21d/0x3a0 [ 51.601691][ C2] ? __pfx_tmigr_handle_remote+0x10/0x10 [ 51.601821][ C2] ? __pfx___lock_release+0x10/0x10 [ 51.601950][ C2] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 51.602080][ C2] __do_softirq+0x1f8/0x5df [ 51.602291][ C2] ? __pfx_run_ksoftirqd+0x10/0x10 [ 51.602420][ C2] run_ksoftirqd+0x2e/0x60 [ 51.602553][ C2] smpboot_thread_fn+0x306/0x840 [ 51.602683][ C2] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 51.602963][ C2] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 51.603093][ C2] kthread+0x28a/0x350 [ 51.603193][ C2] ? __pfx_kthread+0x10/0x10 [ 51.603331][ C2] ret_from_fork+0x31/0x70 [ 51.603535][ C2] ? __pfx_kthread+0x10/0x10 [ 51.603664][ C2] ret_from_fork_asm+0x1a/0x30 [ 51.603798][ C2] [ 63.614844][ T446] br11: port 2(veth2.11) entered disabled state [ 63.615255][ T446] br11: port 1(veth1.11) entered disabled state [ 63.714019][ T447] br10: port 2(veth2.10) entered disabled state [ 63.714484][ T447] br10: port 1(veth1.10) entered disabled state [ 63.929107][ T449] veth2.11: left allmulticast mode [ 63.929297][ T449] veth2.11: left promiscuous mode [ 63.929580][ T449] br11: port 2(veth2.11) entered disabled state [ 64.145652][ T451] veth2: left allmulticast mode [ 64.145843][ T451] veth2: left promiscuous mode [ 64.250134][ T454] veth2.10: left allmulticast mode [ 64.250316][ T454] veth2.10: left promiscuous mode [ 64.250585][ T454] br10: port 2(veth2.10) entered disabled state [ 64.792439][ T459] veth1.11: left allmulticast mode [ 64.792682][ T459] veth1.11: left promiscuous mode [ 64.792952][ T459] br11: port 1(veth1.11) entered disabled state [ 65.008157][ T461] veth1: left allmulticast mode [ 65.008344][ T461] veth1: left promiscuous mode [ 65.110157][ T462] veth1.10: left allmulticast mode [ 65.110330][ T462] veth1.10: left promiscuous mode [ 65.110579][ T462] br10: port 1(veth1.10) entered disabled state