[ 10.853086][ T253] ip (253) used greatest stack depth: 23248 bytes left [ 13.237416][ T301] lag1: Mode changed to "loadbalance" [ 13.647053][ T309] lag1: Port device veth0 added [ 13.755287][ T310] lag1: Port device veth6 added [ 15.127775][ T328] lag4: Mode changed to "loadbalance" [ 15.514515][ T335] lag4: Port device veth3 added [ 15.603776][ T336] lag4: Port device veth5 added [ 16.839748][ T353] lag2: Mode changed to "loadbalance" [ 17.319044][ T360] lag2: Port device veth1 added [ 17.319939][ T39] [ 17.320062][ T39] ====================================================== [ 17.320260][ T39] WARNING: possible circular locking dependency detected [ 17.320457][ T39] 6.13.0-rc5-virtme #1 Not tainted [ 17.320611][ T39] ------------------------------------------------------ [ 17.320795][ T39] kworker/u19:0/39 is trying to acquire lock: [ 17.321014][ T39] ffff88800d662d80 (team->team_lock_key#3){+.+.}-{4:4}, at: team_port_change_check+0x43/0x130 [ 17.321337][ T39] [ 17.321337][ T39] but task is already holding lock: [ 17.321577][ T39] ffffffffad3d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.321906][ T39] [ 17.321906][ T39] which lock already depends on the new lock. [ 17.321906][ T39] [ 17.322217][ T39] [ 17.322217][ T39] the existing dependency chain (in reverse order) is: [ 17.322477][ T39] [ 17.322477][ T39] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 17.322712][ T39] __lock_acquire+0xb9a/0x1680 [ 17.322871][ T39] lock_acquire.part.0+0xeb/0x330 [ 17.323046][ T39] down_read+0x9f/0x340 [ 17.323201][ T39] blocking_notifier_call_chain+0x50/0x90 [ 17.323397][ T39] __dev_open+0x15c/0x430 [ 17.323554][ T39] dev_open+0xc9/0x130 [ 17.323711][ T39] team_port_add+0x90a/0x1ce0 [ 17.323870][ T39] team_add_slave+0x41/0x80 [ 17.324039][ T39] do_setlink.constprop.0+0x80e/0x2300 [ 17.324230][ T39] rtnl_newlink+0x69c/0xa70 [ 17.324387][ T39] rtnetlink_rcv_msg+0x712/0xc10 [ 17.324540][ T39] netlink_rcv_skb+0x130/0x360 [ 17.324691][ T39] netlink_unicast+0x44b/0x710 [ 17.324851][ T39] netlink_sendmsg+0x723/0xbe0 [ 17.325002][ T39] ____sys_sendmsg+0x7ac/0xa10 [ 17.325157][ T39] ___sys_sendmsg+0xee/0x170 [ 17.325313][ T39] __sys_sendmsg+0x109/0x1a0 [ 17.325468][ T39] do_syscall_64+0xc1/0x1d0 [ 17.325617][ T39] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.325808][ T39] [ 17.325808][ T39] -> #0 (team->team_lock_key#3){+.+.}-{4:4}: [ 17.326041][ T39] check_prev_add+0x182/0xce0 [ 17.326196][ T39] validate_chain+0x7bd/0xae0 [ 17.326347][ T39] __lock_acquire+0xb9a/0x1680 [ 17.326501][ T39] lock_acquire.part.0+0xeb/0x330 [ 17.326654][ T39] __mutex_lock+0x169/0xbc0 [ 17.326814][ T39] team_port_change_check+0x43/0x130 [ 17.327005][ T39] team_device_event+0x3fe/0x520 [ 17.327158][ T39] notifier_call_chain+0xcd/0x150 [ 17.327310][ T39] blocking_notifier_call_chain+0x66/0x90 [ 17.327502][ T39] netdev_state_change+0xf5/0x120 [ 17.327652][ T39] linkwatch_do_dev+0xd2/0x100 [ 17.327809][ T39] __linkwatch_run_queue+0x1df/0x650 [ 17.328020][ T39] linkwatch_event+0x40/0x60 [ 17.328174][ T39] process_one_work+0xe55/0x16d0 [ 17.328330][ T39] worker_thread+0x58c/0xce0 [ 17.328493][ T39] kthread+0x28a/0x350 [ 17.328659][ T39] ret_from_fork+0x31/0x70 [ 17.328822][ T39] ret_from_fork_asm+0x1a/0x30 [ 17.328981][ T39] [ 17.328981][ T39] other info that might help us debug this: [ 17.328981][ T39] [ 17.329281][ T39] Possible unsafe locking scenario: [ 17.329281][ T39] [ 17.329512][ T39] CPU0 CPU1 [ 17.329667][ T39] ---- ---- [ 17.329822][ T39] rlock((netdev_chain).rwsem); [ 17.329977][ T39] lock(team->team_lock_key#3); [ 17.330220][ T39] lock((netdev_chain).rwsem); [ 17.330447][ T39] lock(team->team_lock_key#3); [ 17.330625][ T39] [ 17.330625][ T39] *** DEADLOCK *** [ 17.330625][ T39] [ 17.330858][ T39] 4 locks held by kworker/u19:0/39: [ 17.331022][ T39] #0: ffff8880010a9d48 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x7ec/0x16d0 [ 17.331333][ T39] #1: ffffc900002bfda0 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work+0xe0b/0x16d0 [ 17.331636][ T39] #2: ffffffffad3e7a48 (rtnl_mutex){+.+.}-{4:4}, at: linkwatch_event+0xe/0x60 [ 17.331935][ T39] #3: ffffffffad3d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.332249][ T39] [ 17.332249][ T39] stack backtrace: [ 17.332441][ T39] CPU: 2 UID: 0 PID: 39 Comm: kworker/u19:0 Not tainted 6.13.0-rc5-virtme #1 [ 17.332712][ T39] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 17.332902][ T39] Workqueue: events_unbound linkwatch_event [ 17.333123][ T39] Call Trace: [ 17.333256][ T39] [ 17.333338][ T39] dump_stack_lvl+0x82/0xd0 [ 17.333504][ T39] print_circular_bug+0x2e2/0x4b0 [ 17.333681][ T39] check_noncircular+0x2f9/0x3e0 [ 17.333858][ T39] ? rcu_read_lock_any_held+0x43/0xb0 [ 17.334012][ T39] ? __bfs+0x3f9/0x6c0 [ 17.334130][ T39] ? __pfx_check_noncircular+0x10/0x10 [ 17.334309][ T39] ? check_irq_usage+0x13a/0x850 [ 17.334463][ T39] ? alloc_chain_hlocks+0x4e0/0x720 [ 17.334613][ T39] ? __pfx_check_irq_usage+0x10/0x10 [ 17.334793][ T39] check_prev_add+0x182/0xce0 [ 17.334947][ T39] validate_chain+0x7bd/0xae0 [ 17.335099][ T39] ? __pfx_validate_chain+0x10/0x10 [ 17.335250][ T39] ? hlock_class+0x4e/0x130 [ 17.335421][ T39] ? mark_lock+0x38/0x3e0 [ 17.335548][ T39] __lock_acquire+0xb9a/0x1680 [ 17.335698][ T39] ? check_prev_add+0x6e0/0xce0 [ 17.335851][ T39] lock_acquire.part.0+0xeb/0x330 [ 17.336006][ T39] ? team_port_change_check+0x43/0x130 [ 17.336159][ T39] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.336310][ T39] ? trace_lock_acquire+0x14c/0x1f0 [ 17.336462][ T39] ? lock_acquire+0x32/0xc0 [ 17.336635][ T39] ? team_port_change_check+0x43/0x130 [ 17.336791][ T39] __mutex_lock+0x169/0xbc0 [ 17.336957][ T39] ? team_port_change_check+0x43/0x130 [ 17.337111][ T39] ? team_port_change_check+0x43/0x130 [ 17.337284][ T39] ? __pfx___mutex_lock+0x10/0x10 [ 17.337435][ T39] ? lock_acquire.part.0+0xeb/0x330 [ 17.337604][ T39] ? blocking_notifier_call_chain+0x50/0x90 [ 17.337800][ T39] ? find_held_lock+0x2c/0x110 [ 17.337979][ T39] ? __pfx_fib_sync_up+0x10/0x10 [ 17.338132][ T39] ? __pfx_rwsem_read_trylock+0x10/0x10 [ 17.338298][ T39] ? team_port_change_check+0x43/0x130 [ 17.338461][ T39] team_port_change_check+0x43/0x130 [ 17.338621][ T39] team_device_event+0x3fe/0x520 [ 17.338787][ T39] ? trace_notifier_run+0xe7/0x150 [ 17.338949][ T39] notifier_call_chain+0xcd/0x150 [ 17.339100][ T39] blocking_notifier_call_chain+0x66/0x90 [ 17.339254][ T39] netdev_state_change+0xf5/0x120 [ 17.339406][ T39] ? __pfx_netdev_state_change+0x10/0x10 [ 17.339559][ T39] linkwatch_do_dev+0xd2/0x100 [ 17.339729][ T39] __linkwatch_run_queue+0x1df/0x650 [ 17.339890][ T39] ? __pfx___linkwatch_run_queue+0x10/0x10 [ 17.340082][ T39] ? lock_acquire+0x32/0xc0 [ 17.340234][ T39] linkwatch_event+0x40/0x60 [ 17.340386][ T39] process_one_work+0xe55/0x16d0 [ 17.340566][ T39] ? __pfx___lock_release+0x10/0x10 [ 17.340716][ T39] ? __pfx_process_one_work+0x10/0x10 [ 17.340885][ T39] ? assign_work+0x16c/0x240 [ 17.341048][ T39] worker_thread+0x58c/0xce0 [ 17.341204][ T39] ? __pfx_worker_thread+0x10/0x10 [ 17.341365][ T39] kthread+0x28a/0x350 [ 17.341480][ T39] ? __pfx_kthread+0x10/0x10 [ 17.341636][ T39] ret_from_fork+0x31/0x70 [ 17.341798][ T39] ? __pfx_kthread+0x10/0x10 [ 17.341974][ T39] ret_from_fork_asm+0x1a/0x30 [ 17.342135][ T39] [ 17.416243][ T361] lag2: Port device veth7 added [ 17.825737][ T368] br1: port 1(lag2) entered blocking state [ 17.826012][ T368] br1: port 1(lag2) entered disabled state [ 17.826294][ T368] lag2: entered allmulticast mode [ 17.826483][ T368] veth1: entered allmulticast mode [ 17.826654][ T368] veth7: entered allmulticast mode [ 17.829094][ T368] veth1: entered promiscuous mode [ 17.829335][ T368] veth7: entered promiscuous mode [ 18.021149][ T372] br1: port 1(lag2) entered blocking state [ 18.021448][ T372] br1: port 1(lag2) entered forwarding state [ 18.297164][ T378] lag3: Mode changed to "loadbalance" [ 18.634679][ T385] lag3: Port device veth2 added [ 18.711854][ T386] lag3: Port device veth4 added [ 27.566168][ T449] veth7: left allmulticast mode [ 27.568514][ T449] lag2: Port device veth7 removed [ 27.640265][ T450] veth7: left promiscuous mode [ 40.494659][ T500] veth7: entered allmulticast mode [ 40.494867][ T500] veth7: entered promiscuous mode [ 40.495654][ T500] lag2: Port device veth7 added [ 40.757332][ T511] veth1: left allmulticast mode [ 40.759020][ T511] lag2: Port device veth1 removed [ 40.823113][ T512] veth1: left promiscuous mode [ 53.604792][ T123] br1: port 1(lag2) entered disabled state [ 53.667623][ T563] veth7: left allmulticast mode [ 53.669148][ T563] lag2: Port device veth7 removed [ 53.730027][ T564] veth7: left promiscuous mode [ 53.915628][ T574] veth1: entered allmulticast mode [ 53.915898][ T574] veth1: entered promiscuous mode [ 53.917140][ T574] lag2: Port device veth1 added [ 53.933783][ T524] br1: port 1(lag2) entered blocking state [ 53.934184][ T524] br1: port 1(lag2) entered forwarding state [ 54.167371][ T585] veth7: entered allmulticast mode [ 54.167595][ T585] veth7: entered promiscuous mode [ 54.168584][ T585] lag2: Port device veth7 added [ 66.947026][ T635] lag2: left allmulticast mode [ 66.947264][ T635] veth1: left allmulticast mode [ 66.947736][ T635] veth7: left allmulticast mode [ 66.947919][ T635] veth1: left promiscuous mode [ 66.948095][ T635] veth7: left promiscuous mode [ 66.948258][ T635] br1: port 1(lag2) entered disabled state [ 69.040774][ T637] br1: port 1(lag2) entered blocking state [ 69.041003][ T637] br1: port 1(lag2) entered disabled state [ 69.041221][ T637] lag2: entered allmulticast mode [ 69.041376][ T637] veth1: entered allmulticast mode [ 69.041518][ T637] veth7: entered allmulticast mode [ 69.043134][ T637] veth1: entered promiscuous mode [ 69.043337][ T637] veth7: entered promiscuous mode [ 69.044208][ T637] br1: port 1(lag2) entered blocking state [ 69.044432][ T637] br1: port 1(lag2) entered forwarding state [ 81.912024][ T687] lag3: Port device veth2 removed [ 94.780298][ T738] lag3: Port device veth2 added [ 95.020156][ T749] lag3: Port device veth4 removed [ 107.894931][ T800] lag3: Port device veth2 removed [ 108.127115][ T811] lag3: Port device veth4 added [ 108.393545][ T822] lag3: Port device veth2 added [ 121.278877][ T872] br1: port 2(lag3) entered blocking state [ 121.279196][ T872] br1: port 2(lag3) entered disabled state [ 121.279489][ T872] lag3: entered allmulticast mode [ 121.279674][ T872] veth4: entered allmulticast mode [ 121.279842][ T872] veth2: entered allmulticast mode [ 121.281572][ T872] lag3: entered promiscuous mode [ 121.281801][ T872] veth4: entered promiscuous mode [ 121.282215][ T872] veth2: entered promiscuous mode [ 121.282768][ T872] lag2: entered promiscuous mode [ 121.283965][ T872] br1: port 2(lag3) entered blocking state [ 121.284210][ T872] br1: port 2(lag3) entered forwarding state [ 123.408009][ T874] lag3: left allmulticast mode [ 123.408219][ T874] veth4: left allmulticast mode [ 123.408393][ T874] veth2: left allmulticast mode [ 123.408562][ T874] lag3: left promiscuous mode [ 123.408724][ T874] veth4: left promiscuous mode [ 123.409126][ T874] veth2: left promiscuous mode [ 123.409672][ T874] br1: port 2(lag3) entered disabled state [ 123.410778][ T874] lag2: left promiscuous mode [ 136.536893][ T927] lag3: Port device veth4 removed [ 136.609879][ T928] lag3: Port device veth2 removed [ 137.224007][ T123] br1: port 1(lag2) entered disabled state [ 137.347250][ T943] lag2: left allmulticast mode [ 137.347475][ T943] veth1: left allmulticast mode [ 137.347648][ T943] veth7: left allmulticast mode [ 137.347842][ T943] br1: port 1(lag2) entered disabled state [ 137.522507][ T945] lag2: Port device veth7 removed [ 137.590891][ T946] lag2: Port device veth1 removed [ 138.712620][ T961] lag4: Port device veth5 removed [ 138.775653][ T962] lag4: Port device veth3 removed [ 139.934199][ T975] lag1: Port device veth6 removed [ 140.006093][ T976] lag1: Port device veth0 removed