[ 13.795333][ T300] lag1: Mode changed to "loadbalance" [ 14.220488][ T308] lag1: Port device veth0 added [ 14.307647][ T309] lag1: Port device veth6 added [ 15.628829][ T327] lag4: Mode changed to "loadbalance" [ 16.041758][ T334] lag4: Port device veth3 added [ 16.138528][ T335] lag4: Port device veth5 added [ 17.392475][ T353] lag2: Mode changed to "loadbalance" [ 17.839838][ T360] lag2: Port device veth1 added [ 17.840714][ T300] [ 17.840822][ T300] ====================================================== [ 17.841014][ T300] WARNING: possible circular locking dependency detected [ 17.841200][ T300] 6.13.0-rc5-virtme #1 Not tainted [ 17.841355][ T300] ------------------------------------------------------ [ 17.841535][ T300] teamd/300 is trying to acquire lock: [ 17.841690][ T300] ffff888002452d80 (team->team_lock_key#3){+.+.}-{4:4}, at: team_port_change_check+0x43/0x130 [ 17.842090][ T300] [ 17.842090][ T300] but task is already holding lock: [ 17.842343][ T300] ffffffffaadd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.842693][ T300] [ 17.842693][ T300] which lock already depends on the new lock. [ 17.842693][ T300] [ 17.843018][ T300] [ 17.843018][ T300] the existing dependency chain (in reverse order) is: [ 17.843293][ T300] [ 17.843293][ T300] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 17.843521][ T300] __lock_acquire+0xb9a/0x1680 [ 17.843680][ T300] lock_acquire.part.0+0xeb/0x330 [ 17.843834][ T300] down_read+0x9f/0x340 [ 17.843991][ T300] blocking_notifier_call_chain+0x50/0x90 [ 17.844177][ T300] __dev_open+0x15c/0x430 [ 17.844328][ T300] dev_open+0xc9/0x130 [ 17.844481][ T300] team_port_add+0x90a/0x1ce0 [ 17.844658][ T300] team_add_slave+0x41/0x80 [ 17.844808][ T300] do_setlink.constprop.0+0x80e/0x2300 [ 17.844994][ T300] rtnl_newlink+0x69c/0xa70 [ 17.845142][ T300] rtnetlink_rcv_msg+0x712/0xc10 [ 17.845332][ T300] netlink_rcv_skb+0x130/0x360 [ 17.845499][ T300] netlink_unicast+0x44b/0x710 [ 17.845672][ T300] netlink_sendmsg+0x723/0xbe0 [ 17.845836][ T300] ____sys_sendmsg+0x7ac/0xa10 [ 17.846044][ T300] ___sys_sendmsg+0xee/0x170 [ 17.846219][ T300] __sys_sendmsg+0x109/0x1a0 [ 17.846402][ T300] do_syscall_64+0xc1/0x1d0 [ 17.846569][ T300] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.846781][ T300] [ 17.846781][ T300] -> #0 (team->team_lock_key#3){+.+.}-{4:4}: [ 17.847054][ T300] check_prev_add+0x182/0xce0 [ 17.847225][ T300] validate_chain+0x7bd/0xae0 [ 17.847390][ T300] __lock_acquire+0xb9a/0x1680 [ 17.847560][ T300] lock_acquire.part.0+0xeb/0x330 [ 17.847727][ T300] __mutex_lock+0x169/0xbc0 [ 17.847889][ T300] team_port_change_check+0x43/0x130 [ 17.848089][ T300] team_device_event+0x3fe/0x520 [ 17.848250][ T300] notifier_call_chain+0xcd/0x150 [ 17.848418][ T300] blocking_notifier_call_chain+0x66/0x90 [ 17.848620][ T300] netdev_state_change+0xf5/0x120 [ 17.848787][ T300] linkwatch_do_dev+0xd2/0x100 [ 17.848967][ T300] rtnl_getlink+0x4cf/0x970 [ 17.849148][ T300] rtnetlink_rcv_msg+0x2fb/0xc10 [ 17.849317][ T300] netlink_rcv_skb+0x130/0x360 [ 17.849488][ T300] netlink_unicast+0x44b/0x710 [ 17.849653][ T300] netlink_sendmsg+0x723/0xbe0 [ 17.849813][ T300] ____sys_sendmsg+0x7ac/0xa10 [ 17.849981][ T300] ___sys_sendmsg+0xee/0x170 [ 17.850147][ T300] __sys_sendmsg+0x109/0x1a0 [ 17.850309][ T300] do_syscall_64+0xc1/0x1d0 [ 17.850469][ T300] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.850674][ T300] [ 17.850674][ T300] other info that might help us debug this: [ 17.850674][ T300] [ 17.850988][ T300] Possible unsafe locking scenario: [ 17.850988][ T300] [ 17.851222][ T300] CPU0 CPU1 [ 17.851383][ T300] ---- ---- [ 17.851561][ T300] rlock((netdev_chain).rwsem); [ 17.851744][ T300] lock(team->team_lock_key#3); [ 17.851986][ T300] lock((netdev_chain).rwsem); [ 17.852225][ T300] lock(team->team_lock_key#3); [ 17.852393][ T300] [ 17.852393][ T300] *** DEADLOCK *** [ 17.852393][ T300] [ 17.852633][ T300] 2 locks held by teamd/300: [ 17.852803][ T300] #0: ffffffffaade7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnetlink_rcv_msg+0x2af/0xc10 [ 17.853094][ T300] #1: ffffffffaadd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.853434][ T300] [ 17.853434][ T300] stack backtrace: [ 17.853633][ T300] CPU: 2 UID: 0 PID: 300 Comm: teamd Not tainted 6.13.0-rc5-virtme #1 [ 17.853880][ T300] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 17.854082][ T300] Call Trace: [ 17.854203][ T300] [ 17.854287][ T300] dump_stack_lvl+0x82/0xd0 [ 17.854459][ T300] print_circular_bug+0x2e2/0x4b0 [ 17.854624][ T300] check_noncircular+0x2f9/0x3e0 [ 17.854785][ T300] ? __pfx_check_noncircular+0x10/0x10 [ 17.854974][ T300] ? alloc_chain_hlocks+0x4e0/0x720 [ 17.855149][ T300] ? __lock_release+0x103/0x460 [ 17.855317][ T300] check_prev_add+0x182/0xce0 [ 17.855482][ T300] validate_chain+0x7bd/0xae0 [ 17.855656][ T300] ? __pfx_validate_chain+0x10/0x10 [ 17.855818][ T300] ? hlock_class+0x4e/0x130 [ 17.855985][ T300] ? mark_lock+0x38/0x3e0 [ 17.856109][ T300] __lock_acquire+0xb9a/0x1680 [ 17.856280][ T300] lock_acquire.part.0+0xeb/0x330 [ 17.856442][ T300] ? team_port_change_check+0x43/0x130 [ 17.856607][ T300] ? mark_lock+0x38/0x3e0 [ 17.856728][ T300] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.856887][ T300] ? trace_lock_acquire+0x14c/0x1f0 [ 17.857049][ T300] ? lock_acquire+0x32/0xc0 [ 17.857237][ T300] ? team_port_change_check+0x43/0x130 [ 17.857404][ T300] __mutex_lock+0x169/0xbc0 [ 17.857562][ T300] ? team_port_change_check+0x43/0x130 [ 17.857723][ T300] ? team_port_change_check+0x43/0x130 [ 17.857884][ T300] ? __pfx___mutex_lock+0x10/0x10 [ 17.858050][ T300] ? lock_acquire.part.0+0xeb/0x330 [ 17.858214][ T300] ? blocking_notifier_call_chain+0x50/0x90 [ 17.858429][ T300] ? ref_tracker_free+0x356/0x910 [ 17.858606][ T300] ? __pfx_fib_sync_up+0x10/0x10 [ 17.858769][ T300] ? __pfx_rwsem_read_trylock+0x10/0x10 [ 17.858920][ T300] ? team_port_change_check+0x43/0x130 [ 17.859068][ T300] team_port_change_check+0x43/0x130 [ 17.859223][ T300] team_device_event+0x3fe/0x520 [ 17.859375][ T300] ? trace_notifier_run+0xe7/0x150 [ 17.859523][ T300] notifier_call_chain+0xcd/0x150 [ 17.859679][ T300] blocking_notifier_call_chain+0x66/0x90 [ 17.859826][ T300] netdev_state_change+0xf5/0x120 [ 17.859973][ T300] ? __pfx_netdev_state_change+0x10/0x10 [ 17.860135][ T300] ? __pfx_dev_activate+0x10/0x10 [ 17.860291][ T300] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 17.860492][ T300] linkwatch_do_dev+0xd2/0x100 [ 17.860652][ T300] rtnl_getlink+0x4cf/0x970 [ 17.860812][ T300] ? __pfx_rtnl_getlink+0x10/0x10 [ 17.860998][ T300] ? __mutex_lock+0x190/0xbc0 [ 17.861164][ T300] ? rtnetlink_rcv_msg+0x2af/0xc10 [ 17.861324][ T300] ? __pfx___mutex_lock+0x10/0x10 [ 17.861482][ T300] ? __lock_release+0x103/0x460 [ 17.861633][ T300] ? rtnetlink_rcv_msg+0x297/0xc10 [ 17.861787][ T300] ? __pfx___lock_release+0x10/0x10 [ 17.861945][ T300] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.862122][ T300] rtnetlink_rcv_msg+0x2fb/0xc10 [ 17.862283][ T300] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 17.862443][ T300] ? hlock_class+0x4e/0x130 [ 17.862602][ T300] ? mark_lock+0x38/0x3e0 [ 17.862725][ T300] ? __lock_acquire+0xb9a/0x1680 [ 17.862887][ T300] netlink_rcv_skb+0x130/0x360 [ 17.863048][ T300] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 17.863212][ T300] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 17.863387][ T300] ? netlink_deliver_tap+0x13e/0x340 [ 17.863549][ T300] ? netlink_deliver_tap+0xc3/0x340 [ 17.863713][ T300] netlink_unicast+0x44b/0x710 [ 17.863876][ T300] ? __pfx_netlink_unicast+0x10/0x10 [ 17.864039][ T300] ? find_held_lock+0x2c/0x110 [ 17.864193][ T300] netlink_sendmsg+0x723/0xbe0 [ 17.864349][ T300] ? __pfx_netlink_sendmsg+0x10/0x10 [ 17.864496][ T300] ____sys_sendmsg+0x7ac/0xa10 [ 17.864644][ T300] ? __pfx_____sys_sendmsg+0x10/0x10 [ 17.864800][ T300] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 17.864989][ T300] ___sys_sendmsg+0xee/0x170 [ 17.865135][ T300] ? hlock_class+0x4e/0x130 [ 17.865284][ T300] ? __pfx____sys_sendmsg+0x10/0x10 [ 17.865439][ T300] ? usage_skip+0x132/0x190 [ 17.865586][ T300] ? do_user_addr_fault+0x8fd/0xe30 [ 17.865735][ T300] ? do_user_addr_fault+0x8fd/0xe30 [ 17.865885][ T300] ? find_held_lock+0x2c/0x110 [ 17.866034][ T300] ? __lock_release+0x103/0x460 [ 17.866184][ T300] __sys_sendmsg+0x109/0x1a0 [ 17.866336][ T300] ? __pfx___sys_sendmsg+0x10/0x10 [ 17.866499][ T300] ? do_user_addr_fault+0x97c/0xe30 [ 17.866670][ T300] do_syscall_64+0xc1/0x1d0 [ 17.866828][ T300] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.867038][ T300] RIP: 0033:0x7f7e9e7b39a7 [ 17.867235][ T300] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 17.867774][ T300] RSP: 002b:00007ffd0fb7f248 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 17.867998][ T300] RAX: ffffffffffffffda RBX: 000000002b4dc320 RCX: 00007f7e9e7b39a7 [ 17.868219][ T300] RDX: 0000000000000000 RSI: 00007ffd0fb7f280 RDI: 0000000000000005 [ 17.868440][ T300] RBP: 000000002b4dc230 R08: 0000000000000008 R09: 0000000000000000 [ 17.868666][ T300] R10: 0000000000000400 R11: 0000000000000246 R12: 000000002b5035e0 [ 17.868884][ T300] R13: 00007ffd0fb7f280 R14: 00007ffd0fb7f3b8 R15: 000000002b501e78 [ 17.869107][ T300] [ 17.932900][ T361] lag2: Port device veth7 added [ 18.337872][ T368] br1: port 1(lag2) entered blocking state [ 18.338139][ T368] br1: port 1(lag2) entered disabled state [ 18.338390][ T368] lag2: entered allmulticast mode [ 18.338548][ T368] veth1: entered allmulticast mode [ 18.338695][ T368] veth7: entered allmulticast mode [ 18.340725][ T368] veth1: entered promiscuous mode [ 18.340917][ T368] veth7: entered promiscuous mode [ 18.525669][ T372] br1: port 1(lag2) entered blocking state [ 18.526098][ T372] br1: port 1(lag2) entered forwarding state [ 18.768448][ T378] lag3: Mode changed to "loadbalance" [ 19.110688][ T385] lag3: Port device veth2 added [ 19.181635][ T386] lag3: Port device veth4 added [ 28.070626][ T449] veth7: left allmulticast mode [ 28.073887][ T449] lag2: Port device veth7 removed [ 28.145732][ T450] veth7: left promiscuous mode [ 41.019281][ T500] veth7: entered allmulticast mode [ 41.019495][ T500] veth7: entered promiscuous mode [ 41.020262][ T500] lag2: Port device veth7 added [ 41.271689][ T511] veth1: left allmulticast mode [ 41.273172][ T511] lag2: Port device veth1 removed [ 41.350835][ T512] veth1: left promiscuous mode [ 54.164400][ T37] br1: port 1(lag2) entered disabled state [ 54.242098][ T562] veth7: left allmulticast mode [ 54.244408][ T562] lag2: Port device veth7 removed [ 54.305585][ T563] veth7: left promiscuous mode [ 54.508110][ T573] veth1: entered allmulticast mode [ 54.508327][ T573] veth1: entered promiscuous mode [ 54.516072][ T573] lag2: Port device veth1 added [ 54.527455][ T300] br1: port 1(lag2) entered blocking state [ 54.527742][ T300] br1: port 1(lag2) entered forwarding state [ 54.759605][ T584] veth7: entered allmulticast mode [ 54.759810][ T584] veth7: entered promiscuous mode [ 54.760550][ T584] lag2: Port device veth7 added [ 67.636072][ T634] lag2: left allmulticast mode [ 67.636280][ T634] veth1: left allmulticast mode [ 67.636442][ T634] veth7: left allmulticast mode [ 67.636601][ T634] veth1: left promiscuous mode [ 67.636761][ T634] veth7: left promiscuous mode [ 67.636923][ T634] br1: port 1(lag2) entered disabled state [ 69.770806][ T636] br1: port 1(lag2) entered blocking state [ 69.771225][ T636] br1: port 1(lag2) entered disabled state [ 69.771575][ T636] lag2: entered allmulticast mode [ 69.771812][ T636] veth1: entered allmulticast mode [ 69.772056][ T636] veth7: entered allmulticast mode [ 69.776058][ T636] veth1: entered promiscuous mode [ 69.776409][ T636] veth7: entered promiscuous mode [ 69.777246][ T636] br1: port 1(lag2) entered blocking state [ 69.777462][ T636] br1: port 1(lag2) entered forwarding state [ 82.720437][ T686] lag3: Port device veth2 removed [ 95.673834][ T737] lag3: Port device veth2 added [ 95.971317][ T748] lag3: Port device veth4 removed [ 108.920531][ T800] lag3: Port device veth2 removed [ 109.192764][ T811] lag3: Port device veth4 added [ 109.463367][ T822] lag3: Port device veth2 added [ 122.318398][ T872] br1: port 2(lag3) entered blocking state [ 122.318622][ T872] br1: port 2(lag3) entered disabled state [ 122.318840][ T872] lag3: entered allmulticast mode [ 122.318981][ T872] veth4: entered allmulticast mode [ 122.319751][ T872] veth2: entered allmulticast mode [ 122.321322][ T872] lag3: entered promiscuous mode [ 122.321488][ T872] veth4: entered promiscuous mode [ 122.321845][ T872] veth2: entered promiscuous mode [ 122.322345][ T872] lag2: entered promiscuous mode [ 122.323607][ T872] br1: port 2(lag3) entered blocking state [ 122.323878][ T872] br1: port 2(lag3) entered forwarding state [ 124.436081][ T874] lag3: left allmulticast mode [ 124.436318][ T874] veth4: left allmulticast mode [ 124.436503][ T874] veth2: left allmulticast mode [ 124.436683][ T874] lag3: left promiscuous mode [ 124.436860][ T874] veth4: left promiscuous mode [ 124.437270][ T874] veth2: left promiscuous mode [ 124.437787][ T874] br1: port 2(lag3) entered disabled state [ 124.438834][ T874] lag2: left promiscuous mode [ 137.653037][ T927] lag3: Port device veth4 removed [ 137.724999][ T928] lag3: Port device veth2 removed [ 138.388200][ T39] br1: port 1(lag2) entered disabled state [ 138.522033][ T943] lag2: left allmulticast mode [ 138.522251][ T943] veth1: left allmulticast mode [ 138.522403][ T943] veth7: left allmulticast mode [ 138.522595][ T943] br1: port 1(lag2) entered disabled state [ 138.727013][ T945] lag2: Port device veth7 removed [ 138.811261][ T946] lag2: Port device veth1 removed [ 140.078781][ T959] lag4: Port device veth5 removed [ 140.154803][ T960] lag4: Port device veth3 removed [ 141.495545][ T975] lag1: Port device veth6 removed [ 141.578504][ T976] lag1: Port device veth0 removed