[ 13.041516][ T301] lag1: Mode changed to "loadbalance" [ 13.441986][ T309] lag1: Port device veth0 added [ 13.529127][ T310] lag1: Port device veth6 added [ 14.725936][ T328] lag4: Mode changed to "loadbalance" [ 15.119364][ T335] lag4: Port device veth3 added [ 15.199087][ T336] lag4: Port device veth5 added [ 16.315331][ T353] lag2: Mode changed to "loadbalance" [ 16.709808][ T360] lag2: Port device veth1 added [ 16.710507][ T45] [ 16.710618][ T45] ====================================================== [ 16.710809][ T45] WARNING: possible circular locking dependency detected [ 16.711005][ T45] 6.13.0-rc5-virtme #1 Not tainted [ 16.711157][ T45] ------------------------------------------------------ [ 16.711345][ T45] kworker/u17:1/45 is trying to acquire lock: [ 16.711535][ T45] ffff88800aae2d80 (team->team_lock_key#3){+.+.}-{4:4}, at: team_port_change_check+0x43/0x130 [ 16.711850][ T45] [ 16.711850][ T45] but task is already holding lock: [ 16.712066][ T45] ffffffffbd9d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 16.712365][ T45] [ 16.712365][ T45] which lock already depends on the new lock. [ 16.712365][ T45] [ 16.712656][ T45] [ 16.712656][ T45] the existing dependency chain (in reverse order) is: [ 16.712911][ T45] [ 16.712911][ T45] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 16.713151][ T45] __lock_acquire+0xb9a/0x1680 [ 16.713307][ T45] lock_acquire.part.0+0xeb/0x330 [ 16.713461][ T45] down_read+0x9f/0x340 [ 16.713623][ T45] blocking_notifier_call_chain+0x50/0x90 [ 16.713808][ T45] __dev_open+0x15c/0x430 [ 16.713962][ T45] dev_open+0xc9/0x130 [ 16.714120][ T45] team_port_add+0x90a/0x1ce0 [ 16.714275][ T45] team_add_slave+0x41/0x80 [ 16.714430][ T45] do_setlink.constprop.0+0x80e/0x2300 [ 16.714623][ T45] rtnl_newlink+0x69c/0xa70 [ 16.714779][ T45] rtnetlink_rcv_msg+0x712/0xc10 [ 16.714946][ T45] netlink_rcv_skb+0x130/0x360 [ 16.715100][ T45] netlink_unicast+0x44b/0x710 [ 16.715250][ T45] netlink_sendmsg+0x723/0xbe0 [ 16.715399][ T45] ____sys_sendmsg+0x7ac/0xa10 [ 16.715549][ T45] ___sys_sendmsg+0xee/0x170 [ 16.715703][ T45] __sys_sendmsg+0x109/0x1a0 [ 16.715853][ T45] do_syscall_64+0xc1/0x1d0 [ 16.716006][ T45] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 16.716198][ T45] [ 16.716198][ T45] -> #0 (team->team_lock_key#3){+.+.}-{4:4}: [ 16.716439][ T45] check_prev_add+0x182/0xce0 [ 16.716592][ T45] validate_chain+0x7bd/0xae0 [ 16.716746][ T45] __lock_acquire+0xb9a/0x1680 [ 16.716894][ T45] lock_acquire.part.0+0xeb/0x330 [ 16.717048][ T45] __mutex_lock+0x169/0xbc0 [ 16.717189][ T45] team_port_change_check+0x43/0x130 [ 16.717369][ T45] team_device_event+0x3fe/0x520 [ 16.717514][ T45] notifier_call_chain+0xcd/0x150 [ 16.717661][ T45] blocking_notifier_call_chain+0x66/0x90 [ 16.717862][ T45] netdev_state_change+0xf5/0x120 [ 16.718011][ T45] linkwatch_do_dev+0xd2/0x100 [ 16.718158][ T45] __linkwatch_run_queue+0x1df/0x650 [ 16.718336][ T45] linkwatch_event+0x40/0x60 [ 16.718474][ T45] process_one_work+0xe55/0x16d0 [ 16.718625][ T45] worker_thread+0x58c/0xce0 [ 16.718770][ T45] kthread+0x28a/0x350 [ 16.718920][ T45] ret_from_fork+0x31/0x70 [ 16.719075][ T45] ret_from_fork_asm+0x1a/0x30 [ 16.719215][ T45] [ 16.719215][ T45] other info that might help us debug this: [ 16.719215][ T45] [ 16.719484][ T45] Possible unsafe locking scenario: [ 16.719484][ T45] [ 16.719689][ T45] CPU0 CPU1 [ 16.719827][ T45] ---- ---- [ 16.719966][ T45] rlock((netdev_chain).rwsem); [ 16.720110][ T45] lock(team->team_lock_key#3); [ 16.720331][ T45] lock((netdev_chain).rwsem); [ 16.720564][ T45] lock(team->team_lock_key#3); [ 16.720720][ T45] [ 16.720720][ T45] *** DEADLOCK *** [ 16.720720][ T45] [ 16.720937][ T45] 4 locks held by kworker/u17:1/45: [ 16.721078][ T45] #0: ffff8880010a9d48 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x7ec/0x16d0 [ 16.721368][ T45] #1: ffffc9000031fda0 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work+0xe0b/0x16d0 [ 16.721667][ T45] #2: ffffffffbd9e7a48 (rtnl_mutex){+.+.}-{4:4}, at: linkwatch_event+0xe/0x60 [ 16.721926][ T45] #3: ffffffffbd9d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 16.722213][ T45] [ 16.722213][ T45] stack backtrace: [ 16.722395][ T45] CPU: 3 UID: 0 PID: 45 Comm: kworker/u17:1 Not tainted 6.13.0-rc5-virtme #1 [ 16.722646][ T45] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 16.722816][ T45] Workqueue: events_unbound linkwatch_event [ 16.723014][ T45] Call Trace: [ 16.723126][ T45] [ 16.723211][ T45] dump_stack_lvl+0x82/0xd0 [ 16.723362][ T45] print_circular_bug+0x2e2/0x4b0 [ 16.723512][ T45] check_noncircular+0x2f9/0x3e0 [ 16.723650][ T45] ? rcu_read_lock_any_held+0x43/0xb0 [ 16.723803][ T45] ? __bfs+0x3f9/0x6c0 [ 16.723922][ T45] ? __pfx_check_noncircular+0x10/0x10 [ 16.724065][ T45] ? check_irq_usage+0x13a/0x850 [ 16.724216][ T45] ? alloc_chain_hlocks+0x4e0/0x720 [ 16.724352][ T45] ? __pfx_check_irq_usage+0x10/0x10 [ 16.724490][ T45] check_prev_add+0x182/0xce0 [ 16.724722][ T45] validate_chain+0x7bd/0xae0 [ 16.724875][ T45] ? __pfx_validate_chain+0x10/0x10 [ 16.725020][ T45] ? hlock_class+0x4e/0x130 [ 16.725161][ T45] ? mark_lock+0x38/0x3e0 [ 16.725363][ T45] __lock_acquire+0xb9a/0x1680 [ 16.725509][ T45] ? check_prev_add+0x6e0/0xce0 [ 16.725657][ T45] lock_acquire.part.0+0xeb/0x330 [ 16.725793][ T45] ? team_port_change_check+0x43/0x130 [ 16.726028][ T45] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 16.726179][ T45] ? trace_lock_acquire+0x14c/0x1f0 [ 16.726319][ T45] ? lock_acquire+0x32/0xc0 [ 16.726460][ T45] ? team_port_change_check+0x43/0x130 [ 16.726684][ T45] __mutex_lock+0x169/0xbc0 [ 16.726828][ T45] ? team_port_change_check+0x43/0x130 [ 16.726980][ T45] ? team_port_change_check+0x43/0x130 [ 16.727119][ T45] ? __pfx___mutex_lock+0x10/0x10 [ 16.727342][ T45] ? lock_acquire.part.0+0xeb/0x330 [ 16.727485][ T45] ? blocking_notifier_call_chain+0x50/0x90 [ 16.727656][ T45] ? find_held_lock+0x2c/0x110 [ 16.727804][ T45] ? __pfx_fib_sync_up+0x10/0x10 [ 16.728046][ T45] ? __pfx_rwsem_read_trylock+0x10/0x10 [ 16.728198][ T45] ? team_port_change_check+0x43/0x130 [ 16.728355][ T45] team_port_change_check+0x43/0x130 [ 16.728510][ T45] team_device_event+0x3fe/0x520 [ 16.728722][ T45] ? trace_notifier_run+0xe7/0x150 [ 16.728860][ T45] notifier_call_chain+0xcd/0x150 [ 16.729001][ T45] blocking_notifier_call_chain+0x66/0x90 [ 16.729136][ T45] netdev_state_change+0xf5/0x120 [ 16.729352][ T45] ? __pfx_netdev_state_change+0x10/0x10 [ 16.729492][ T45] linkwatch_do_dev+0xd2/0x100 [ 16.729631][ T45] __linkwatch_run_queue+0x1df/0x650 [ 16.729768][ T45] ? __pfx___linkwatch_run_queue+0x10/0x10 [ 16.730017][ T45] ? lock_acquire+0x32/0xc0 [ 16.730162][ T45] linkwatch_event+0x40/0x60 [ 16.730297][ T45] process_one_work+0xe55/0x16d0 [ 16.730513][ T45] ? __pfx___lock_release+0x10/0x10 [ 16.730693][ T45] ? __pfx_process_one_work+0x10/0x10 [ 16.730853][ T45] ? assign_work+0x16c/0x240 [ 16.731010][ T45] worker_thread+0x58c/0xce0 [ 16.731245][ T45] ? __pfx_worker_thread+0x10/0x10 [ 16.731401][ T45] kthread+0x28a/0x350 [ 16.731507][ T45] ? __pfx_kthread+0x10/0x10 [ 16.731644][ T45] ret_from_fork+0x31/0x70 [ 16.731786][ T45] ? __pfx_kthread+0x10/0x10 [ 16.732018][ T45] ret_from_fork_asm+0x1a/0x30 [ 16.732176][ T45] [ 16.796923][ T361] lag2: Port device veth7 added [ 17.134506][ T369] br1: port 1(lag2) entered blocking state [ 17.134742][ T369] br1: port 1(lag2) entered disabled state [ 17.134951][ T369] lag2: entered allmulticast mode [ 17.135090][ T369] veth1: entered allmulticast mode [ 17.135496][ T369] veth7: entered allmulticast mode [ 17.137278][ T369] veth1: entered promiscuous mode [ 17.137449][ T369] veth7: entered promiscuous mode [ 17.339721][ T373] br1: port 1(lag2) entered blocking state [ 17.339958][ T373] br1: port 1(lag2) entered forwarding state [ 17.578955][ T379] lag3: Mode changed to "loadbalance" [ 17.859924][ T386] lag3: Port device veth2 added [ 17.941419][ T387] lag3: Port device veth4 added [ 26.524225][ T450] veth7: left allmulticast mode [ 26.525570][ T450] lag2: Port device veth7 removed [ 26.576888][ T451] veth7: left promiscuous mode [ 39.326858][ T501] veth7: entered allmulticast mode [ 39.327097][ T501] veth7: entered promiscuous mode [ 39.327847][ T501] lag2: Port device veth7 added [ 39.577889][ T512] veth1: left allmulticast mode [ 39.579227][ T512] lag2: Port device veth1 removed [ 39.638565][ T513] veth1: left promiscuous mode [ 52.362450][ T40] br1: port 1(lag2) entered disabled state [ 52.410723][ T563] veth7: left allmulticast mode [ 52.412474][ T563] lag2: Port device veth7 removed [ 52.478003][ T564] veth7: left promiscuous mode [ 52.647098][ T574] veth1: entered allmulticast mode [ 52.647309][ T574] veth1: entered promiscuous mode [ 52.648181][ T574] lag2: Port device veth1 added [ 52.677853][ T70] br1: port 1(lag2) entered blocking state [ 52.678171][ T70] br1: port 1(lag2) entered forwarding state [ 52.862254][ T585] veth7: entered allmulticast mode [ 52.862460][ T585] veth7: entered promiscuous mode [ 52.863167][ T585] lag2: Port device veth7 added [ 65.733942][ T635] lag2: left allmulticast mode [ 65.734183][ T635] veth1: left allmulticast mode [ 65.735346][ T635] veth7: left allmulticast mode [ 65.735562][ T635] veth1: left promiscuous mode [ 65.735751][ T635] veth7: left promiscuous mode [ 65.735925][ T635] br1: port 1(lag2) entered disabled state [ 67.832132][ T637] br1: port 1(lag2) entered blocking state [ 67.832411][ T637] br1: port 1(lag2) entered disabled state [ 67.832629][ T637] lag2: entered allmulticast mode [ 67.832767][ T637] veth1: entered allmulticast mode [ 67.832905][ T637] veth7: entered allmulticast mode [ 67.834459][ T637] veth1: entered promiscuous mode [ 67.834635][ T637] veth7: entered promiscuous mode [ 67.835716][ T637] br1: port 1(lag2) entered blocking state [ 67.835910][ T637] br1: port 1(lag2) entered forwarding state [ 80.614508][ T687] lag3: Port device veth2 removed [ 93.450296][ T738] lag3: Port device veth2 added [ 93.694164][ T749] lag3: Port device veth4 removed [ 106.581617][ T800] lag3: Port device veth2 removed [ 106.811354][ T811] lag3: Port device veth4 added [ 107.034852][ T822] lag3: Port device veth2 added [ 119.842166][ T872] br1: port 2(lag3) entered blocking state [ 119.842408][ T872] br1: port 2(lag3) entered disabled state [ 119.842619][ T872] lag3: entered allmulticast mode [ 119.842760][ T872] veth4: entered allmulticast mode [ 119.842906][ T872] veth2: entered allmulticast mode [ 119.845410][ T872] lag3: entered promiscuous mode [ 119.845654][ T872] veth4: entered promiscuous mode [ 119.846001][ T872] veth2: entered promiscuous mode [ 119.846473][ T872] lag2: entered promiscuous mode [ 119.847768][ T872] br1: port 2(lag3) entered blocking state [ 119.847967][ T872] br1: port 2(lag3) entered forwarding state [ 121.946675][ T874] lag3: left allmulticast mode [ 121.946896][ T874] veth4: left allmulticast mode [ 121.947061][ T874] veth2: left allmulticast mode [ 121.947226][ T874] lag3: left promiscuous mode [ 121.947404][ T874] veth4: left promiscuous mode [ 121.947787][ T874] veth2: left promiscuous mode [ 121.948306][ T874] br1: port 2(lag3) entered disabled state [ 121.949111][ T874] lag2: left promiscuous mode [ 134.984327][ T927] lag3: Port device veth4 removed [ 135.037946][ T928] lag3: Port device veth2 removed [ 135.613161][ T39] br1: port 1(lag2) entered disabled state [ 135.757858][ T943] lag2: left allmulticast mode [ 135.758075][ T943] veth1: left allmulticast mode [ 135.758267][ T943] veth7: left allmulticast mode [ 135.758482][ T943] br1: port 1(lag2) entered disabled state [ 135.944544][ T945] lag2: Port device veth7 removed [ 136.020134][ T946] lag2: Port device veth1 removed [ 137.212207][ T960] lag4: Port device veth5 removed [ 137.273725][ T961] lag4: Port device veth3 removed [ 138.467277][ T975] lag1: Port device veth6 removed [ 138.516583][ T976] lag1: Port device veth0 removed