[ 12.627467][ T275] ip (275) used greatest stack depth: 23136 bytes left [ 14.359151][ T299] lag1: Mode changed to "loadbalance" [ 14.803539][ T308] lag1: Port device veth0 added [ 14.885443][ T309] lag1: Port device veth6 added [ 16.241053][ T327] lag4: Mode changed to "loadbalance" [ 16.671597][ T334] lag4: Port device veth3 added [ 16.766474][ T335] lag4: Port device veth5 added [ 18.034173][ T352] lag2: Mode changed to "loadbalance" [ 18.486317][ T359] lag2: Port device veth1 added [ 18.487083][ T40] [ 18.487218][ T40] ====================================================== [ 18.487435][ T40] WARNING: possible circular locking dependency detected [ 18.487671][ T40] 6.13.0-rc5-virtme #1 Not tainted [ 18.487834][ T40] ------------------------------------------------------ [ 18.488037][ T40] kworker/u20:0/40 is trying to acquire lock: [ 18.488239][ T40] ffff8880051f2d80 (team->team_lock_key#3){+.+.}-{4:4}, at: team_port_change_check+0x43/0x130 [ 18.488588][ T40] [ 18.488588][ T40] but task is already holding lock: [ 18.488824][ T40] ffffffff939d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 18.489155][ T40] [ 18.489155][ T40] which lock already depends on the new lock. [ 18.489155][ T40] [ 18.489465][ T40] [ 18.489465][ T40] the existing dependency chain (in reverse order) is: [ 18.489822][ T40] [ 18.489822][ T40] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 18.490066][ T40] __lock_acquire+0xb9a/0x1680 [ 18.490241][ T40] lock_acquire.part.0+0xeb/0x330 [ 18.490409][ T40] down_read+0x9f/0x340 [ 18.490571][ T40] blocking_notifier_call_chain+0x50/0x90 [ 18.490758][ T40] __dev_open+0x15c/0x430 [ 18.490909][ T40] dev_open+0xc9/0x130 [ 18.491052][ T40] team_port_add+0x90a/0x1ce0 [ 18.491196][ T40] team_add_slave+0x41/0x80 [ 18.491351][ T40] do_setlink.constprop.0+0x80e/0x2300 [ 18.491559][ T40] rtnl_newlink+0x69c/0xa70 [ 18.491700][ T40] rtnetlink_rcv_msg+0x712/0xc10 [ 18.491851][ T40] netlink_rcv_skb+0x130/0x360 [ 18.492001][ T40] netlink_unicast+0x44b/0x710 [ 18.492154][ T40] netlink_sendmsg+0x723/0xbe0 [ 18.492293][ T40] ____sys_sendmsg+0x7ac/0xa10 [ 18.492447][ T40] ___sys_sendmsg+0xee/0x170 [ 18.492607][ T40] __sys_sendmsg+0x109/0x1a0 [ 18.492750][ T40] do_syscall_64+0xc1/0x1d0 [ 18.492898][ T40] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 18.493080][ T40] [ 18.493080][ T40] -> #0 (team->team_lock_key#3){+.+.}-{4:4}: [ 18.493312][ T40] check_prev_add+0x182/0xce0 [ 18.493468][ T40] validate_chain+0x7bd/0xae0 [ 18.493623][ T40] __lock_acquire+0xb9a/0x1680 [ 18.493769][ T40] lock_acquire.part.0+0xeb/0x330 [ 18.493910][ T40] __mutex_lock+0x169/0xbc0 [ 18.494056][ T40] team_port_change_check+0x43/0x130 [ 18.494240][ T40] team_device_event+0x3fe/0x520 [ 18.494383][ T40] notifier_call_chain+0xcd/0x150 [ 18.494534][ T40] blocking_notifier_call_chain+0x66/0x90 [ 18.494710][ T40] netdev_state_change+0xf5/0x120 [ 18.494862][ T40] linkwatch_do_dev+0xd2/0x100 [ 18.495019][ T40] __linkwatch_run_queue+0x1df/0x650 [ 18.495198][ T40] linkwatch_event+0x40/0x60 [ 18.495342][ T40] process_one_work+0xe55/0x16d0 [ 18.495495][ T40] worker_thread+0x58c/0xce0 [ 18.495641][ T40] kthread+0x28a/0x350 [ 18.495789][ T40] ret_from_fork+0x31/0x70 [ 18.495934][ T40] ret_from_fork_asm+0x1a/0x30 [ 18.496082][ T40] [ 18.496082][ T40] other info that might help us debug this: [ 18.496082][ T40] [ 18.496359][ T40] Possible unsafe locking scenario: [ 18.496359][ T40] [ 18.496578][ T40] CPU0 CPU1 [ 18.496718][ T40] ---- ---- [ 18.496865][ T40] rlock((netdev_chain).rwsem); [ 18.497009][ T40] lock(team->team_lock_key#3); [ 18.497224][ T40] lock((netdev_chain).rwsem); [ 18.497437][ T40] lock(team->team_lock_key#3); [ 18.497597][ T40] [ 18.497597][ T40] *** DEADLOCK *** [ 18.497597][ T40] [ 18.497809][ T40] 4 locks held by kworker/u20:0/40: [ 18.497957][ T40] #0: ffff8880010a9d48 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x7ec/0x16d0 [ 18.498257][ T40] #1: ffffc900002cfda0 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work+0xe0b/0x16d0 [ 18.498561][ T40] #2: ffffffff939e7a48 (rtnl_mutex){+.+.}-{4:4}, at: linkwatch_event+0xe/0x60 [ 18.498820][ T40] #3: ffffffff939d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 18.499111][ T40] [ 18.499111][ T40] stack backtrace: [ 18.499297][ T40] CPU: 1 UID: 0 PID: 40 Comm: kworker/u20:0 Not tainted 6.13.0-rc5-virtme #1 [ 18.499545][ T40] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 18.499723][ T40] Workqueue: events_unbound linkwatch_event [ 18.499904][ T40] Call Trace: [ 18.500015][ T40] [ 18.500093][ T40] dump_stack_lvl+0x82/0xd0 [ 18.500251][ T40] print_circular_bug+0x2e2/0x4b0 [ 18.500443][ T40] check_noncircular+0x2f9/0x3e0 [ 18.500588][ T40] ? rcu_read_lock_any_held+0x43/0xb0 [ 18.500741][ T40] ? __bfs+0x3f9/0x6c0 [ 18.500851][ T40] ? __pfx_check_noncircular+0x10/0x10 [ 18.500994][ T40] ? check_irq_usage+0x13a/0x850 [ 18.501139][ T40] ? alloc_chain_hlocks+0x4e0/0x720 [ 18.501286][ T40] ? __pfx_check_irq_usage+0x10/0x10 [ 18.501435][ T40] check_prev_add+0x182/0xce0 [ 18.501583][ T40] validate_chain+0x7bd/0xae0 [ 18.501728][ T40] ? __pfx_validate_chain+0x10/0x10 [ 18.501880][ T40] ? hlock_class+0x4e/0x130 [ 18.502022][ T40] ? mark_lock+0x38/0x3e0 [ 18.502131][ T40] __lock_acquire+0xb9a/0x1680 [ 18.502275][ T40] ? check_prev_add+0x6e0/0xce0 [ 18.502446][ T40] lock_acquire.part.0+0xeb/0x330 [ 18.502591][ T40] ? team_port_change_check+0x43/0x130 [ 18.502735][ T40] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 18.502877][ T40] ? trace_lock_acquire+0x14c/0x1f0 [ 18.503020][ T40] ? lock_acquire+0x32/0xc0 [ 18.503162][ T40] ? team_port_change_check+0x43/0x130 [ 18.503304][ T40] __mutex_lock+0x169/0xbc0 [ 18.503453][ T40] ? team_port_change_check+0x43/0x130 [ 18.503599][ T40] ? team_port_change_check+0x43/0x130 [ 18.503741][ T40] ? __pfx___mutex_lock+0x10/0x10 [ 18.503884][ T40] ? lock_acquire.part.0+0xeb/0x330 [ 18.504027][ T40] ? blocking_notifier_call_chain+0x50/0x90 [ 18.504205][ T40] ? find_held_lock+0x2c/0x110 [ 18.504353][ T40] ? __pfx_fib_sync_up+0x10/0x10 [ 18.504504][ T40] ? __pfx_rwsem_read_trylock+0x10/0x10 [ 18.504650][ T40] ? team_port_change_check+0x43/0x130 [ 18.504794][ T40] team_port_change_check+0x43/0x130 [ 18.504938][ T40] team_device_event+0x3fe/0x520 [ 18.505080][ T40] ? trace_notifier_run+0xe7/0x150 [ 18.505222][ T40] notifier_call_chain+0xcd/0x150 [ 18.505367][ T40] blocking_notifier_call_chain+0x66/0x90 [ 18.505542][ T40] netdev_state_change+0xf5/0x120 [ 18.505686][ T40] ? __pfx_netdev_state_change+0x10/0x10 [ 18.505831][ T40] linkwatch_do_dev+0xd2/0x100 [ 18.505973][ T40] __linkwatch_run_queue+0x1df/0x650 [ 18.506116][ T40] ? __pfx___linkwatch_run_queue+0x10/0x10 [ 18.506292][ T40] ? lock_acquire+0x32/0xc0 [ 18.506438][ T40] linkwatch_event+0x40/0x60 [ 18.506607][ T40] process_one_work+0xe55/0x16d0 [ 18.506752][ T40] ? __pfx___lock_release+0x10/0x10 [ 18.506895][ T40] ? __pfx_process_one_work+0x10/0x10 [ 18.507038][ T40] ? assign_work+0x16c/0x240 [ 18.507184][ T40] worker_thread+0x58c/0xce0 [ 18.507331][ T40] ? __pfx_worker_thread+0x10/0x10 [ 18.507471][ T40] kthread+0x28a/0x350 [ 18.507581][ T40] ? __pfx_kthread+0x10/0x10 [ 18.507734][ T40] ret_from_fork+0x31/0x70 [ 18.507878][ T40] ? __pfx_kthread+0x10/0x10 [ 18.508019][ T40] ret_from_fork_asm+0x1a/0x30 [ 18.508166][ T40] [ 18.600638][ T360] lag2: Port device veth7 added [ 19.021785][ T367] br1: port 1(lag2) entered blocking state [ 19.022052][ T367] br1: port 1(lag2) entered disabled state [ 19.022296][ T367] lag2: entered allmulticast mode [ 19.022485][ T367] veth1: entered allmulticast mode [ 19.022674][ T367] veth7: entered allmulticast mode [ 19.024652][ T367] veth1: entered promiscuous mode [ 19.024844][ T367] veth7: entered promiscuous mode [ 19.220052][ T371] br1: port 1(lag2) entered blocking state [ 19.220310][ T371] br1: port 1(lag2) entered forwarding state [ 19.485508][ T377] lag3: Mode changed to "loadbalance" [ 19.812220][ T384] lag3: Port device veth2 added [ 19.886250][ T385] lag3: Port device veth4 added [ 28.720065][ T448] veth7: left allmulticast mode [ 28.731349][ T448] lag2: Port device veth7 removed [ 28.800433][ T449] veth7: left promiscuous mode [ 41.694218][ T499] veth7: entered allmulticast mode [ 41.694514][ T499] veth7: entered promiscuous mode [ 41.695299][ T499] lag2: Port device veth7 added [ 41.934257][ T510] veth1: left allmulticast mode [ 41.935734][ T510] lag2: Port device veth1 removed [ 42.000583][ T511] veth1: left promiscuous mode [ 54.853445][ T39] br1: port 1(lag2) entered disabled state [ 54.920583][ T561] veth7: left allmulticast mode [ 54.922667][ T561] lag2: Port device veth7 removed [ 54.982079][ T562] veth7: left promiscuous mode [ 55.164709][ T572] veth1: entered allmulticast mode [ 55.164945][ T572] veth1: entered promiscuous mode [ 55.165849][ T572] lag2: Port device veth1 added [ 55.188250][ T40] br1: port 1(lag2) entered blocking state [ 55.188498][ T40] br1: port 1(lag2) entered forwarding state [ 55.419032][ T583] veth7: entered allmulticast mode [ 55.419219][ T583] veth7: entered promiscuous mode [ 55.420035][ T583] lag2: Port device veth7 added [ 68.278367][ T633] lag2: left allmulticast mode [ 68.278592][ T633] veth1: left allmulticast mode [ 68.278743][ T633] veth7: left allmulticast mode [ 68.278919][ T633] veth1: left promiscuous mode [ 68.279094][ T633] veth7: left promiscuous mode [ 68.279244][ T633] br1: port 1(lag2) entered disabled state [ 70.386756][ T635] br1: port 1(lag2) entered blocking state [ 70.386986][ T635] br1: port 1(lag2) entered disabled state [ 70.387216][ T635] lag2: entered allmulticast mode [ 70.387357][ T635] veth1: entered allmulticast mode [ 70.387902][ T635] veth7: entered allmulticast mode [ 70.389532][ T635] veth1: entered promiscuous mode [ 70.389719][ T635] veth7: entered promiscuous mode [ 70.390392][ T635] br1: port 1(lag2) entered blocking state [ 70.390591][ T635] br1: port 1(lag2) entered forwarding state [ 83.281246][ T685] lag3: Port device veth2 removed [ 96.239067][ T736] lag3: Port device veth2 added [ 96.508909][ T747] lag3: Port device veth4 removed [ 109.463486][ T798] lag3: Port device veth2 removed [ 109.753986][ T809] lag3: Port device veth4 added [ 110.007449][ T820] lag3: Port device veth2 added [ 122.882291][ T870] br1: port 2(lag3) entered blocking state [ 122.882586][ T870] br1: port 2(lag3) entered disabled state [ 122.882832][ T870] lag3: entered allmulticast mode [ 122.882986][ T870] veth4: entered allmulticast mode [ 122.883143][ T870] veth2: entered allmulticast mode [ 122.884979][ T870] lag3: entered promiscuous mode [ 122.885151][ T870] veth4: entered promiscuous mode [ 122.885533][ T870] veth2: entered promiscuous mode [ 122.886024][ T870] lag2: entered promiscuous mode [ 122.887402][ T870] br1: port 2(lag3) entered blocking state [ 122.887625][ T870] br1: port 2(lag3) entered forwarding state [ 124.987992][ T872] lag3: left allmulticast mode [ 124.988192][ T872] veth4: left allmulticast mode [ 124.988349][ T872] veth2: left allmulticast mode [ 124.988677][ T872] lag3: left promiscuous mode [ 124.988831][ T872] veth4: left promiscuous mode [ 124.989215][ T872] veth2: left promiscuous mode [ 124.989745][ T872] br1: port 2(lag3) entered disabled state [ 124.990721][ T872] lag2: left promiscuous mode [ 138.191549][ T925] lag3: Port device veth4 removed [ 138.265196][ T926] lag3: Port device veth2 removed [ 138.913558][ T39] br1: port 1(lag2) entered disabled state [ 139.026074][ T941] lag2: left allmulticast mode [ 139.026270][ T941] veth1: left allmulticast mode [ 139.026437][ T941] veth7: left allmulticast mode [ 139.026620][ T941] br1: port 1(lag2) entered disabled state [ 139.224265][ T943] lag2: Port device veth7 removed [ 139.290797][ T944] lag2: Port device veth1 removed [ 140.406513][ T957] lag4: Port device veth5 removed [ 140.474424][ T958] lag4: Port device veth3 removed [ 141.779588][ T973] lag1: Port device veth6 removed [ 141.843575][ T974] lag1: Port device veth0 removed