[ 13.425730][ T299] lag1: Mode changed to "loadbalance" [ 13.880312][ T307] lag1: Port device veth0 added [ 13.963636][ T308] lag1: Port device veth6 added [ 15.261643][ T326] lag4: Mode changed to "loadbalance" [ 15.663660][ T333] lag4: Port device veth3 added [ 15.749395][ T334] lag4: Port device veth5 added [ 16.998787][ T351] lag2: Mode changed to "loadbalance" [ 17.385230][ T358] lag2: Port device veth1 added [ 17.386018][ T351] [ 17.386115][ T351] ====================================================== [ 17.386301][ T351] WARNING: possible circular locking dependency detected [ 17.386490][ T351] 6.13.0-rc5-virtme #1 Not tainted [ 17.386644][ T351] ------------------------------------------------------ [ 17.386829][ T351] teamd/351 is trying to acquire lock: [ 17.386978][ T351] ffff8880137fad80 (team->team_lock_key#3){+.+.}-{4:4}, at: team_port_change_check+0x43/0x130 [ 17.387302][ T351] [ 17.387302][ T351] but task is already holding lock: [ 17.387518][ T351] ffffffffa1bd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.387826][ T351] [ 17.387826][ T351] which lock already depends on the new lock. [ 17.387826][ T351] [ 17.388147][ T351] [ 17.388147][ T351] the existing dependency chain (in reverse order) is: [ 17.388408][ T351] [ 17.388408][ T351] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 17.388636][ T351] __lock_acquire+0xb9a/0x1680 [ 17.388790][ T351] lock_acquire.part.0+0xeb/0x330 [ 17.388943][ T351] down_read+0x9f/0x340 [ 17.389096][ T351] blocking_notifier_call_chain+0x50/0x90 [ 17.389281][ T351] __dev_open+0x15c/0x430 [ 17.389431][ T351] dev_open+0xc9/0x130 [ 17.389602][ T351] team_port_add+0x90a/0x1ce0 [ 17.389758][ T351] team_add_slave+0x41/0x80 [ 17.389907][ T351] do_setlink.constprop.0+0x80e/0x2300 [ 17.390099][ T351] rtnl_newlink+0x69c/0xa70 [ 17.390249][ T351] rtnetlink_rcv_msg+0x712/0xc10 [ 17.390403][ T351] netlink_rcv_skb+0x130/0x360 [ 17.390553][ T351] netlink_unicast+0x44b/0x710 [ 17.390702][ T351] netlink_sendmsg+0x723/0xbe0 [ 17.390852][ T351] ____sys_sendmsg+0x7ac/0xa10 [ 17.391004][ T351] ___sys_sendmsg+0xee/0x170 [ 17.391159][ T351] __sys_sendmsg+0x109/0x1a0 [ 17.391308][ T351] do_syscall_64+0xc1/0x1d0 [ 17.391459][ T351] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.391647][ T351] [ 17.391647][ T351] -> #0 (team->team_lock_key#3){+.+.}-{4:4}: [ 17.391874][ T351] check_prev_add+0x182/0xce0 [ 17.392027][ T351] validate_chain+0x7bd/0xae0 [ 17.392180][ T351] __lock_acquire+0xb9a/0x1680 [ 17.392333][ T351] lock_acquire.part.0+0xeb/0x330 [ 17.392483][ T351] __mutex_lock+0x169/0xbc0 [ 17.392631][ T351] team_port_change_check+0x43/0x130 [ 17.392815][ T351] team_device_event+0x3fe/0x520 [ 17.392969][ T351] notifier_call_chain+0xcd/0x150 [ 17.393128][ T351] blocking_notifier_call_chain+0x66/0x90 [ 17.393312][ T351] netdev_state_change+0xf5/0x120 [ 17.393461][ T351] linkwatch_do_dev+0xd2/0x100 [ 17.393614][ T351] rtnl_getlink+0x4cf/0x970 [ 17.393759][ T351] rtnetlink_rcv_msg+0x2fb/0xc10 [ 17.393904][ T351] netlink_rcv_skb+0x130/0x360 [ 17.394053][ T351] netlink_unicast+0x44b/0x710 [ 17.394201][ T351] netlink_sendmsg+0x723/0xbe0 [ 17.394348][ T351] ____sys_sendmsg+0x7ac/0xa10 [ 17.394494][ T351] ___sys_sendmsg+0xee/0x170 [ 17.394644][ T351] __sys_sendmsg+0x109/0x1a0 [ 17.394790][ T351] do_syscall_64+0xc1/0x1d0 [ 17.394938][ T351] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.395122][ T351] [ 17.395122][ T351] other info that might help us debug this: [ 17.395122][ T351] [ 17.395411][ T351] Possible unsafe locking scenario: [ 17.395411][ T351] [ 17.395624][ T351] CPU0 CPU1 [ 17.395772][ T351] ---- ---- [ 17.395927][ T351] rlock((netdev_chain).rwsem); [ 17.396077][ T351] lock(team->team_lock_key#3); [ 17.396295][ T351] lock((netdev_chain).rwsem); [ 17.396516][ T351] lock(team->team_lock_key#3); [ 17.396668][ T351] [ 17.396668][ T351] *** DEADLOCK *** [ 17.396668][ T351] [ 17.396886][ T351] 2 locks held by teamd/351: [ 17.397039][ T351] #0: ffffffffa1be7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnetlink_rcv_msg+0x2af/0xc10 [ 17.397301][ T351] #1: ffffffffa1bd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.397598][ T351] [ 17.397598][ T351] stack backtrace: [ 17.397780][ T351] CPU: 2 UID: 0 PID: 351 Comm: teamd Not tainted 6.13.0-rc5-virtme #1 [ 17.398004][ T351] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 17.398190][ T351] Call Trace: [ 17.398301][ T351] [ 17.398380][ T351] dump_stack_lvl+0x82/0xd0 [ 17.398533][ T351] print_circular_bug+0x2e2/0x4b0 [ 17.398684][ T351] check_noncircular+0x2f9/0x3e0 [ 17.398831][ T351] ? __pfx_check_noncircular+0x10/0x10 [ 17.398985][ T351] ? alloc_chain_hlocks+0x4e0/0x720 [ 17.399136][ T351] ? __lock_release+0x103/0x460 [ 17.399286][ T351] check_prev_add+0x182/0xce0 [ 17.399439][ T351] validate_chain+0x7bd/0xae0 [ 17.399588][ T351] ? __pfx_validate_chain+0x10/0x10 [ 17.399733][ T351] ? hlock_class+0x4e/0x130 [ 17.399887][ T351] ? mark_lock+0x38/0x3e0 [ 17.400004][ T351] __lock_acquire+0xb9a/0x1680 [ 17.400152][ T351] lock_acquire.part.0+0xeb/0x330 [ 17.400306][ T351] ? team_port_change_check+0x43/0x130 [ 17.400456][ T351] ? mark_lock+0x38/0x3e0 [ 17.400587][ T351] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.400734][ T351] ? trace_lock_acquire+0x14c/0x1f0 [ 17.400881][ T351] ? lock_acquire+0x32/0xc0 [ 17.401029][ T351] ? team_port_change_check+0x43/0x130 [ 17.401175][ T351] __mutex_lock+0x169/0xbc0 [ 17.401325][ T351] ? team_port_change_check+0x43/0x130 [ 17.401474][ T351] ? team_port_change_check+0x43/0x130 [ 17.401622][ T351] ? __pfx___mutex_lock+0x10/0x10 [ 17.401768][ T351] ? lock_acquire.part.0+0xeb/0x330 [ 17.401913][ T351] ? blocking_notifier_call_chain+0x50/0x90 [ 17.402096][ T351] ? ref_tracker_free+0x356/0x910 [ 17.402248][ T351] ? __pfx_fib_sync_up+0x10/0x10 [ 17.402407][ T351] ? __pfx_rwsem_read_trylock+0x10/0x10 [ 17.402559][ T351] ? team_port_change_check+0x43/0x130 [ 17.402798][ T351] team_port_change_check+0x43/0x130 [ 17.402947][ T351] team_device_event+0x3fe/0x520 [ 17.403099][ T351] ? trace_notifier_run+0xe7/0x150 [ 17.403256][ T351] notifier_call_chain+0xcd/0x150 [ 17.403404][ T351] blocking_notifier_call_chain+0x66/0x90 [ 17.403560][ T351] netdev_state_change+0xf5/0x120 [ 17.403707][ T351] ? __pfx_netdev_state_change+0x10/0x10 [ 17.403853][ T351] ? __pfx_dev_activate+0x10/0x10 [ 17.404004][ T351] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 17.404192][ T351] linkwatch_do_dev+0xd2/0x100 [ 17.404342][ T351] rtnl_getlink+0x4cf/0x970 [ 17.404490][ T351] ? __pfx_rtnl_getlink+0x10/0x10 [ 17.404742][ T351] ? __mutex_lock+0x190/0xbc0 [ 17.404890][ T351] ? rtnetlink_rcv_msg+0x2af/0xc10 [ 17.405038][ T351] ? __pfx___mutex_lock+0x10/0x10 [ 17.405185][ T351] ? __lock_release+0x103/0x460 [ 17.405417][ T351] ? rtnetlink_rcv_msg+0x297/0xc10 [ 17.405566][ T351] ? __pfx___lock_release+0x10/0x10 [ 17.405712][ T351] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.405866][ T351] rtnetlink_rcv_msg+0x2fb/0xc10 [ 17.406104][ T351] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 17.406249][ T351] ? hlock_class+0x4e/0x130 [ 17.406396][ T351] ? mark_lock+0x38/0x3e0 [ 17.406508][ T351] ? __lock_acquire+0xb9a/0x1680 [ 17.406743][ T351] netlink_rcv_skb+0x130/0x360 [ 17.406890][ T351] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 17.407036][ T351] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 17.407193][ T351] ? netlink_deliver_tap+0x13e/0x340 [ 17.407430][ T351] ? netlink_deliver_tap+0xc3/0x340 [ 17.407578][ T351] netlink_unicast+0x44b/0x710 [ 17.407724][ T351] ? __pfx_netlink_unicast+0x10/0x10 [ 17.407872][ T351] ? find_held_lock+0x2c/0x110 [ 17.408109][ T351] netlink_sendmsg+0x723/0xbe0 [ 17.408256][ T351] ? __pfx_netlink_sendmsg+0x10/0x10 [ 17.408412][ T351] ____sys_sendmsg+0x7ac/0xa10 [ 17.408559][ T351] ? __pfx_____sys_sendmsg+0x10/0x10 [ 17.408789][ T351] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 17.408973][ T351] ___sys_sendmsg+0xee/0x170 [ 17.409122][ T351] ? hlock_class+0x4e/0x130 [ 17.409270][ T351] ? __pfx____sys_sendmsg+0x10/0x10 [ 17.409420][ T351] ? usage_skip+0x132/0x190 [ 17.409576][ T351] ? do_user_addr_fault+0x8fd/0xe30 [ 17.409723][ T351] ? do_user_addr_fault+0x8fd/0xe30 [ 17.409871][ T351] ? find_held_lock+0x2c/0x110 [ 17.410018][ T351] ? __lock_release+0x103/0x460 [ 17.410166][ T351] __sys_sendmsg+0x109/0x1a0 [ 17.410312][ T351] ? __pfx___sys_sendmsg+0x10/0x10 [ 17.410462][ T351] ? do_user_addr_fault+0x97c/0xe30 [ 17.410703][ T351] do_syscall_64+0xc1/0x1d0 [ 17.410851][ T351] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.411032][ T351] RIP: 0033:0x7fc7b20f79a7 [ 17.411277][ T351] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 17.411792][ T351] RSP: 002b:00007ffccb464838 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 17.412110][ T351] RAX: ffffffffffffffda RBX: 00000000147da320 RCX: 00007fc7b20f79a7 [ 17.412332][ T351] RDX: 0000000000000000 RSI: 00007ffccb464870 RDI: 0000000000000005 [ 17.412637][ T351] RBP: 00000000147da230 R08: 0000000000000008 R09: 0000000000000000 [ 17.412864][ T351] R10: 0000000000000200 R11: 0000000000000246 R12: 0000000014800ab0 [ 17.413085][ T351] R13: 00007ffccb464870 R14: 00007ffccb4649a8 R15: 0000000014800578 [ 17.413407][ T351] [ 17.470469][ T359] lag2: Port device veth7 added [ 17.854536][ T366] br1: port 1(lag2) entered blocking state [ 17.854812][ T366] br1: port 1(lag2) entered disabled state [ 17.855043][ T366] lag2: entered allmulticast mode [ 17.855197][ T366] veth1: entered allmulticast mode [ 17.855359][ T366] veth7: entered allmulticast mode [ 17.857396][ T366] veth1: entered promiscuous mode [ 17.857610][ T366] veth7: entered promiscuous mode [ 18.035377][ T370] br1: port 1(lag2) entered blocking state [ 18.035641][ T370] br1: port 1(lag2) entered forwarding state [ 18.265040][ T376] lag3: Mode changed to "loadbalance" [ 18.570383][ T383] lag3: Port device veth2 added [ 18.647118][ T384] lag3: Port device veth4 added [ 27.415694][ T447] veth7: left allmulticast mode [ 27.418020][ T447] lag2: Port device veth7 removed [ 27.481341][ T448] veth7: left promiscuous mode [ 40.344839][ T498] veth7: entered allmulticast mode [ 40.345057][ T498] veth7: entered promiscuous mode [ 40.345888][ T498] lag2: Port device veth7 added [ 40.586908][ T509] veth1: left allmulticast mode [ 40.588165][ T509] lag2: Port device veth1 removed [ 40.637668][ T510] veth1: left promiscuous mode [ 53.448536][ T37] br1: port 1(lag2) entered disabled state [ 53.513394][ T560] veth7: left allmulticast mode [ 53.514909][ T560] lag2: Port device veth7 removed [ 53.581093][ T561] veth7: left promiscuous mode [ 53.769237][ T571] veth1: entered allmulticast mode [ 53.769481][ T571] veth1: entered promiscuous mode [ 53.770264][ T571] lag2: Port device veth1 added [ 53.809910][ T37] br1: port 1(lag2) entered blocking state [ 53.810158][ T37] br1: port 1(lag2) entered forwarding state [ 54.005770][ T582] veth7: entered allmulticast mode [ 54.006077][ T582] veth7: entered promiscuous mode [ 54.006870][ T582] lag2: Port device veth7 added [ 66.860593][ T632] lag2: left allmulticast mode [ 66.860924][ T632] veth1: left allmulticast mode [ 66.861193][ T632] veth7: left allmulticast mode [ 66.861496][ T632] veth1: left promiscuous mode [ 66.861785][ T632] veth7: left promiscuous mode [ 66.862050][ T632] br1: port 1(lag2) entered disabled state [ 68.982172][ T634] br1: port 1(lag2) entered blocking state [ 68.982495][ T634] br1: port 1(lag2) entered disabled state [ 68.982754][ T634] lag2: entered allmulticast mode [ 68.982933][ T634] veth1: entered allmulticast mode [ 68.983088][ T634] veth7: entered allmulticast mode [ 68.984778][ T634] veth1: entered promiscuous mode [ 68.985009][ T634] veth7: entered promiscuous mode [ 68.985709][ T634] br1: port 1(lag2) entered blocking state [ 68.985936][ T634] br1: port 1(lag2) entered forwarding state [ 81.823915][ T684] lag3: Port device veth2 removed [ 94.709747][ T735] lag3: Port device veth2 added [ 94.960821][ T746] lag3: Port device veth4 removed [ 107.958998][ T798] lag3: Port device veth2 removed [ 108.176365][ T809] lag3: Port device veth4 added [ 108.429139][ T820] lag3: Port device veth2 added [ 121.378841][ T870] br1: port 2(lag3) entered blocking state [ 121.379197][ T870] br1: port 2(lag3) entered disabled state [ 121.379569][ T870] lag3: entered allmulticast mode [ 121.379786][ T870] veth4: entered allmulticast mode [ 121.380020][ T870] veth2: entered allmulticast mode [ 121.382587][ T870] lag3: entered promiscuous mode [ 121.382855][ T870] veth4: entered promiscuous mode [ 121.383427][ T870] veth2: entered promiscuous mode [ 121.384198][ T870] lag2: entered promiscuous mode [ 121.386055][ T870] br1: port 2(lag3) entered blocking state [ 121.386395][ T870] br1: port 2(lag3) entered forwarding state [ 123.491735][ T872] lag3: left allmulticast mode [ 123.491989][ T872] veth4: left allmulticast mode [ 123.492167][ T872] veth2: left allmulticast mode [ 123.492359][ T872] lag3: left promiscuous mode [ 123.492548][ T872] veth4: left promiscuous mode [ 123.492973][ T872] veth2: left promiscuous mode [ 123.493517][ T872] br1: port 2(lag3) entered disabled state [ 123.494383][ T872] lag2: left promiscuous mode [ 136.681004][ T925] lag3: Port device veth4 removed [ 136.751626][ T926] lag3: Port device veth2 removed [ 137.415816][ T56] br1: port 1(lag2) entered disabled state [ 137.554072][ T941] lag2: left allmulticast mode [ 137.554280][ T941] veth1: left allmulticast mode [ 137.554441][ T941] veth7: left allmulticast mode [ 137.554616][ T941] br1: port 1(lag2) entered disabled state [ 137.725093][ T943] lag2: Port device veth7 removed [ 137.782848][ T944] lag2: Port device veth1 removed [ 138.947951][ T959] lag4: Port device veth5 removed [ 139.006224][ T960] lag4: Port device veth3 removed [ 140.289909][ T973] lag1: Port device veth6 removed [ 140.363434][ T974] lag1: Port device veth0 removed