[ 1372.942979][T18103] lag1: Mode changed to "loadbalance" [ 1373.043794][T18105] 8021q: adding VLAN 0 to HW filter on device lag1 [ 1373.399914][T18120] [ 1373.400059][T18120] ====================================================== [ 1373.400270][T18120] WARNING: possible circular locking dependency detected [ 1373.400497][T18120] 6.14.0-rc2-virtme #1 Not tainted [ 1373.400670][T18120] ------------------------------------------------------ [ 1373.400877][T18120] ip/18120 is trying to acquire lock: [ 1373.401043][T18120] ffff8880086f9cb0 (&dev->lock){+.+.}-{4:4}, at: __dev_open+0x57e/0x710 [ 1373.401310][T18120] [ 1373.401310][T18120] but task is already holding lock: [ 1373.401558][T18120] ffff888002152d80 (team->team_lock_key){+.+.}-{4:4}, at: team_add_slave+0x33/0x80 [ 1373.401861][T18120] [ 1373.401861][T18120] which lock already depends on the new lock. [ 1373.401861][T18120] [ 1373.402185][T18120] [ 1373.402185][T18120] the existing dependency chain (in reverse order) is: [ 1373.402462][T18120] [ 1373.402462][T18120] -> #1 (team->team_lock_key){+.+.}-{4:4}: [ 1373.402714][T18120] __lock_acquire+0xb9a/0x1680 [ 1373.402895][T18120] lock_acquire.part.0+0xeb/0x330 [ 1373.403070][T18120] __mutex_lock+0x169/0xc40 [ 1373.403244][T18120] team_set_mac_address+0x120/0x250 [ 1373.403468][T18120] netif_set_mac_address+0x28a/0x410 [ 1373.403707][T18120] do_setlink.constprop.0+0x1980/0x25f0 [ 1373.403914][T18120] rtnl_setlink+0x2b5/0x5a0 [ 1373.404078][T18120] rtnetlink_rcv_msg+0x712/0xc10 [ 1373.404243][T18120] netlink_rcv_skb+0x130/0x360 [ 1373.404412][T18120] netlink_unicast+0x44b/0x710 [ 1373.404575][T18120] netlink_sendmsg+0x723/0xbe0 [ 1373.404746][T18120] __sys_sendto+0x3c3/0x450 [ 1373.404911][T18120] __x64_sys_sendto+0xe0/0x1c0 [ 1373.405073][T18120] do_syscall_64+0xc1/0x1d0 [ 1373.405238][T18120] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1373.405457][T18120] [ 1373.405457][T18120] -> #0 (&dev->lock){+.+.}-{4:4}: [ 1373.405708][T18120] check_prev_add+0x182/0xce0 [ 1373.405874][T18120] validate_chain+0x7bd/0xae0 [ 1373.406053][T18120] __lock_acquire+0xb9a/0x1680 [ 1373.406218][T18120] lock_acquire.part.0+0xeb/0x330 [ 1373.406383][T18120] __mutex_lock+0x169/0xc40 [ 1373.406550][T18120] __dev_open+0x57e/0x710 [ 1373.406714][T18120] netif_open+0xc9/0x130 [ 1373.406877][T18120] dev_open+0x9f/0x1a0 [ 1373.407046][T18120] team_port_add+0x90a/0x1ce0 [ 1373.407220][T18120] team_add_slave+0x41/0x80 [ 1373.407410][T18120] do_set_master+0x302/0x610 [ 1373.407575][T18120] do_setlink.constprop.0+0x87b/0x25f0 [ 1373.407786][T18120] rtnl_newlink+0x69c/0xa70 [ 1373.407951][T18120] rtnetlink_rcv_msg+0x712/0xc10 [ 1373.408114][T18120] netlink_rcv_skb+0x130/0x360 [ 1373.408279][T18120] netlink_unicast+0x44b/0x710 [ 1373.408459][T18120] netlink_sendmsg+0x723/0xbe0 [ 1373.408625][T18120] ____sys_sendmsg+0x7ac/0xa10 [ 1373.408784][T18120] ___sys_sendmsg+0xee/0x170 [ 1373.408933][T18120] __sys_sendmsg+0x109/0x1a0 [ 1373.409082][T18120] do_syscall_64+0xc1/0x1d0 [ 1373.409236][T18120] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1373.409448][T18120] [ 1373.409448][T18120] other info that might help us debug this: [ 1373.409448][T18120] [ 1373.409755][T18120] Possible unsafe locking scenario: [ 1373.409755][T18120] [ 1373.409976][T18120] CPU0 CPU1 [ 1373.410124][T18120] ---- ---- [ 1373.410273][T18120] lock(team->team_lock_key); [ 1373.410431][T18120] lock(&dev->lock); [ 1373.410623][T18120] lock(team->team_lock_key); [ 1373.410847][T18120] lock(&dev->lock); [ 1373.410963][T18120] [ 1373.410963][T18120] *** DEADLOCK *** [ 1373.410963][T18120] [ 1373.411180][T18120] 2 locks held by ip/18120: [ 1373.411335][T18120] #0: ffffffff9f9eeb48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x653/0xa70 [ 1373.411610][T18120] #1: ffff888002152d80 (team->team_lock_key){+.+.}-{4:4}, at: team_add_slave+0x33/0x80 [ 1373.411894][T18120] [ 1373.411894][T18120] stack backtrace: [ 1373.412078][T18120] CPU: 1 UID: 0 PID: 18120 Comm: ip Not tainted 6.14.0-rc2-virtme #1 [ 1373.412084][T18120] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 1373.412086][T18120] Call Trace: [ 1373.412088][T18120] [ 1373.412090][T18120] dump_stack_lvl+0x82/0xd0 [ 1373.412099][T18120] print_circular_bug+0x1b7/0x240 [ 1373.412105][T18120] check_noncircular+0x2f9/0x3e0 [ 1373.412109][T18120] ? __pfx_check_noncircular+0x10/0x10 [ 1373.412113][T18120] ? find_held_lock+0x2c/0x110 [ 1373.412119][T18120] ? alloc_chain_hlocks+0x4e0/0x720 [ 1373.412123][T18120] check_prev_add+0x182/0xce0 [ 1373.412128][T18120] validate_chain+0x7bd/0xae0 [ 1373.412132][T18120] ? stack_depot_save_flags+0x27e/0x440 [ 1373.412140][T18120] ? __pfx_validate_chain+0x10/0x10 [ 1373.412143][T18120] ? hlock_class+0x4e/0x130 [ 1373.412145][T18120] ? mark_lock+0x38/0x3e0 [ 1373.412149][T18120] __lock_acquire+0xb9a/0x1680 [ 1373.412155][T18120] lock_acquire.part.0+0xeb/0x330 [ 1373.412158][T18120] ? __dev_open+0x57e/0x710 [ 1373.412162][T18120] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 1373.412165][T18120] ? trace_lock_acquire+0x14c/0x1f0 [ 1373.412168][T18120] ? lock_acquire+0x32/0xc0 [ 1373.412171][T18120] ? __dev_open+0x57e/0x710 [ 1373.412174][T18120] __mutex_lock+0x169/0xc40 [ 1373.412176][T18120] ? __dev_open+0x57e/0x710 [ 1373.412183][T18120] ? __dev_open+0x57e/0x710 [ 1373.412186][T18120] ? __pfx___mutex_lock+0x10/0x10 [ 1373.412190][T18120] ? packet_notifier+0x3b0/0x810 [ 1373.412198][T18120] ? __dev_open+0x57e/0x710 [ 1373.412200][T18120] __dev_open+0x57e/0x710 [ 1373.412203][T18120] ? mark_held_locks+0x9e/0xe0 [ 1373.412206][T18120] ? __pfx___dev_open+0x10/0x10 [ 1373.412209][T18120] ? trace_percpu_alloc_percpu+0x149/0x1d0 [ 1373.412216][T18120] netif_open+0xc9/0x130 [ 1373.412219][T18120] ? __pfx_netif_open+0x10/0x10 [ 1373.412223][T18120] dev_open+0x9f/0x1a0 [ 1373.412227][T18120] team_port_add+0x90a/0x1ce0 [ 1373.412233][T18120] ? __pfx_team_port_add+0x10/0x10 [ 1373.412236][T18120] ? rcu_read_lock_any_held+0x43/0xb0 [ 1373.412242][T18120] ? validate_chain+0x1fe/0xae0 [ 1373.412247][T18120] team_add_slave+0x41/0x80 [ 1373.412251][T18120] do_set_master+0x302/0x610 [ 1373.412255][T18120] do_setlink.constprop.0+0x87b/0x25f0 [ 1373.412260][T18120] ? __pfx_do_setlink.constprop.0+0x10/0x10 [ 1373.412263][T18120] ? lock_acquire.part.0+0xeb/0x330 [ 1373.412266][T18120] ? rtnl_newlink+0x653/0xa70 [ 1373.412269][T18120] ? rtnl_newlink+0xb9/0xa70 [ 1373.412272][T18120] ? rtnetlink_rcv_msg+0x712/0xc10 [ 1373.412275][T18120] ? __mutex_trylock_common+0xfa/0x260 [ 1373.412279][T18120] ? __pfx___mutex_trylock_common+0x10/0x10 [ 1373.412282][T18120] ? lock_acquire+0x32/0xc0 [ 1373.412286][T18120] ? trace_contention_end+0xef/0x150 [ 1373.412289][T18120] ? __mutex_lock+0x190/0xc40 [ 1373.412292][T18120] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 1373.412295][T18120] ? rtnl_newlink+0x653/0xa70 [ 1373.412298][T18120] ? __pfx___mutex_lock+0x10/0x10 [ 1373.412300][T18120] ? __pfx_validate_nla+0x10/0x10 [ 1373.412306][T18120] ? trace_cap_capable+0x11e/0x1a0 [ 1373.412312][T18120] ? __rtnl_newlink+0x3f2/0xa80 [ 1373.412316][T18120] rtnl_newlink+0x69c/0xa70 [ 1373.412321][T18120] ? __pfx_rtnl_newlink+0x10/0x10 [ 1373.412323][T18120] ? find_held_lock+0x2c/0x110 [ 1373.412328][T18120] ? __pfx___lock_release+0x10/0x10 [ 1373.412331][T18120] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 1373.412336][T18120] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 1373.412339][T18120] ? __pfx_rtnl_newlink+0x10/0x10 [ 1373.412342][T18120] rtnetlink_rcv_msg+0x712/0xc10 [ 1373.412346][T18120] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 1373.412348][T18120] ? hlock_class+0x4e/0x130 [ 1373.412351][T18120] ? mark_lock+0x38/0x3e0 [ 1373.412355][T18120] ? __lock_acquire+0xb9a/0x1680 [ 1373.412359][T18120] netlink_rcv_skb+0x130/0x360 [ 1373.412363][T18120] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 1373.412367][T18120] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 1373.412374][T18120] ? netlink_deliver_tap+0x13e/0x340 [ 1373.412378][T18120] ? netlink_deliver_tap+0xc3/0x340 [ 1373.412382][T18120] netlink_unicast+0x44b/0x710 [ 1373.412386][T18120] ? __pfx_netlink_unicast+0x10/0x10 [ 1373.412389][T18120] ? find_held_lock+0x2c/0x110 [ 1373.412394][T18120] netlink_sendmsg+0x723/0xbe0 [ 1373.412398][T18120] ? __pfx_netlink_sendmsg+0x10/0x10 [ 1373.412404][T18120] ____sys_sendmsg+0x7ac/0xa10 [ 1373.412408][T18120] ? __pfx_____sys_sendmsg+0x10/0x10 [ 1373.412410][T18120] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 1373.412417][T18120] ___sys_sendmsg+0xee/0x170 [ 1373.412421][T18120] ? __pfx____sys_sendmsg+0x10/0x10 [ 1373.412426][T18120] ? kasan_save_stack+0x34/0x50 [ 1373.412430][T18120] ? kasan_save_stack+0x24/0x50 [ 1373.412432][T18120] ? kasan_record_aux_stack+0x8c/0xa0 [ 1373.412436][T18120] ? __call_rcu_common.constprop.0+0xa1/0x5b0 [ 1373.412440][T18120] ? __x64_sys_close+0x7c/0xd0 [ 1373.412444][T18120] ? do_syscall_64+0xc1/0x1d0 [ 1373.412446][T18120] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1373.412450][T18120] ? __lock_acquire+0xb9a/0x1680 [ 1373.412453][T18120] ? lock_downgrade+0x52/0x110 [ 1373.412456][T18120] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 1373.412460][T18120] ? find_held_lock+0x2c/0x110 [ 1373.412464][T18120] ? __lock_release+0x103/0x460 [ 1373.412467][T18120] ? __virt_addr_valid+0x22b/0x430 [ 1373.412472][T18120] ? __pfx___lock_release+0x10/0x10 [ 1373.412475][T18120] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 1373.412479][T18120] __sys_sendmsg+0x109/0x1a0 [ 1373.412483][T18120] ? __pfx___sys_sendmsg+0x10/0x10 [ 1373.412491][T18120] do_syscall_64+0xc1/0x1d0 [ 1373.412493][T18120] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1373.412496][T18120] RIP: 0033:0x7f8c749a89a7 [ 1373.412501][T18120] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 1373.412504][T18120] RSP: 002b:00007ffe40f542a8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 1373.412508][T18120] RAX: ffffffffffffffda RBX: 00007ffe40f549d0 RCX: 00007f8c749a89a7 [ 1373.412510][T18120] RDX: 0000000000000000 RSI: 00007ffe40f54310 RDI: 0000000000000005 [ 1373.412512][T18120] RBP: 0000000000000004 R08: 0000000000000003 R09: 0000000000000000 [ 1373.412514][T18120] R10: 00000000149e6910 R11: 0000000000000246 R12: 0000000000000004 [ 1373.412515][T18120] R13: 0000000067b32d2f R14: 0000000000498600 R15: 0000000000000000 [ 1373.412521][T18120] [ 1373.432837][T18120] lag1: Port device veth0 added [ 1373.439023][T18120] ip (18120) used greatest stack depth: 23208 bytes left [ 1373.492351][T18122] lag1: Port device veth6 added [ 1373.825067][T18128] 8021q: adding VLAN 0 to HW filter on device lag1 [ 1375.002061][T18151] lag4: Mode changed to "loadbalance" [ 1375.092128][T18153] 8021q: adding VLAN 0 to HW filter on device lag4 [ 1375.373068][T18159] lag4: Port device veth3 added [ 1375.433708][T18160] lag4: Port device veth5 added [ 1375.778581][T18166] 8021q: adding VLAN 0 to HW filter on device lag4 [ 1377.027414][T18189] lag2: Mode changed to "loadbalance" [ 1377.114032][T18191] 8021q: adding VLAN 0 to HW filter on device lag2 [ 1377.415374][T18197] lag2: Port device veth1 added [ 1377.490705][T18198] lag2: Port device veth7 added [ 1378.093034][T18211] br1: port 1(lag2.100) entered blocking state [ 1378.093960][T18211] br1: port 1(lag2.100) entered disabled state [ 1378.094230][T18211] lag2.100: entered allmulticast mode [ 1378.094400][T18211] lag2: entered allmulticast mode [ 1378.094570][T18211] veth1: entered allmulticast mode [ 1378.094752][T18211] veth7: entered allmulticast mode [ 1378.096839][T18211] lag2.100: entered promiscuous mode [ 1378.097045][T18211] lag2: entered promiscuous mode [ 1378.097229][T18211] veth1: entered promiscuous mode [ 1378.097576][T18211] veth7: entered promiscuous mode [ 1378.388398][T18218] br2: port 1(lag2.200) entered blocking state [ 1378.388686][T18218] br2: port 1(lag2.200) entered disabled state [ 1378.388930][T18218] lag2.200: entered allmulticast mode [ 1378.390645][T18218] lag2.200: entered promiscuous mode [ 1378.585250][T18221] br1: port 1(lag2.100) entered blocking state [ 1378.585512][T18221] br1: port 1(lag2.100) entered forwarding state [ 1378.656005][T18222] br2: port 1(lag2.200) entered blocking state [ 1378.656294][T18222] br2: port 1(lag2.200) entered forwarding state [ 1379.035619][T18230] lag3: Mode changed to "loadbalance" [ 1379.113330][T18232] 8021q: adding VLAN 0 to HW filter on device lag3 [ 1379.419758][T18238] lag3: Port device veth2 added [ 1379.495686][T18240] lag3: Port device veth4 added [ 1390.897297][T18321] veth7: left promiscuous mode [ 1390.897714][T18321] veth7: left allmulticast mode [ 1390.899065][T18321] lag2: Port device veth7 removed [ 1406.025339][T18382] veth7: entered promiscuous mode [ 1406.025750][T18382] veth7: entered allmulticast mode [ 1406.026528][T18382] lag2: Port device veth7 added [ 1406.258728][T18393] veth1: left promiscuous mode [ 1406.259169][T18393] veth1: left allmulticast mode [ 1406.260700][T18393] lag2: Port device veth1 removed [ 1421.369174][ T137] br1: port 1(lag2.100) entered disabled state [ 1421.371072][ T137] br2: port 1(lag2.200) entered disabled state [ 1421.407522][T18454] veth7: left promiscuous mode [ 1421.407902][T18454] veth7: left allmulticast mode [ 1421.409035][T18454] lag2: Port device veth7 removed [ 1421.634085][T18465] veth1: entered promiscuous mode [ 1421.634457][T18465] veth1: entered allmulticast mode [ 1421.635251][T18465] lag2: Port device veth1 added [ 1421.662402][ T137] br1: port 1(lag2.100) entered blocking state [ 1421.662647][ T137] br1: port 1(lag2.100) entered forwarding state [ 1421.663359][ T137] br2: port 1(lag2.200) entered blocking state [ 1421.663550][ T137] br2: port 1(lag2.200) entered forwarding state [ 1421.880811][T18476] veth7: entered promiscuous mode [ 1421.881786][T18476] veth7: entered allmulticast mode [ 1421.882552][T18476] lag2: Port device veth7 added [ 1437.003003][T18536] lag2.200: left allmulticast mode [ 1437.003262][T18536] lag2.200: left promiscuous mode [ 1437.003818][T18536] br2: port 1(lag2.200) entered disabled state [ 1437.062289][T18537] lag2.100: left allmulticast mode [ 1437.062492][T18537] lag2: left allmulticast mode [ 1437.062651][T18537] veth1: left allmulticast mode [ 1437.062808][T18537] veth7: left allmulticast mode [ 1437.062968][T18537] lag2.100: left promiscuous mode [ 1437.063110][T18537] lag2: left promiscuous mode [ 1437.063344][T18537] veth1: left promiscuous mode [ 1437.063734][T18537] veth7: left promiscuous mode [ 1437.064457][T18537] br1: port 1(lag2.100) entered disabled state [ 1439.167131][T18539] br1: port 1(lag2.100) entered blocking state [ 1439.167414][T18539] br1: port 1(lag2.100) entered disabled state [ 1439.167658][T18539] lag2.100: entered allmulticast mode [ 1439.167813][T18539] lag2: entered allmulticast mode [ 1439.167971][T18539] veth1: entered allmulticast mode [ 1439.168149][T18539] veth7: entered allmulticast mode [ 1439.169789][T18539] lag2.100: entered promiscuous mode [ 1439.169969][T18539] lag2: entered promiscuous mode [ 1439.170135][T18539] veth1: entered promiscuous mode [ 1439.170958][T18539] veth7: entered promiscuous mode [ 1439.172819][T18539] br1: port 1(lag2.100) entered blocking state [ 1439.173047][T18539] br1: port 1(lag2.100) entered forwarding state [ 1439.232869][T18540] br2: port 1(lag2.200) entered blocking state [ 1439.233108][T18540] br2: port 1(lag2.200) entered disabled state [ 1439.234310][T18540] lag2.200: entered allmulticast mode [ 1439.235902][T18540] lag2.200: entered promiscuous mode [ 1439.236873][T18540] br2: port 1(lag2.200) entered blocking state [ 1439.237085][T18540] br2: port 1(lag2.200) entered forwarding state [ 1454.334848][T18600] lag3: Port device veth2 removed [ 1469.478137][T18661] lag3: Port device veth2 added [ 1469.712584][T18672] lag3: Port device veth4 removed [ 1484.738496][T18733] lag3: Port device veth2 removed [ 1484.954962][T18744] lag3: Port device veth4 added [ 1485.195019][T18755] lag3: Port device veth2 added [ 1500.950920][T18825] lag3: Port device veth4 removed [ 1501.004006][T18826] lag3: Port device veth2 removed [ 1501.666851][T18838] br2: port 1(lag2.200) entered disabled state [ 1501.725901][T18839] br1: port 1(lag2.100) entered disabled state [ 1501.903569][T18842] lag2.200: left allmulticast mode [ 1501.903787][T18842] lag2.200: left promiscuous mode [ 1501.904273][T18842] br2: port 1(lag2.200) entered disabled state [ 1502.085115][T18844] lag2.100: left allmulticast mode [ 1502.086061][T18844] lag2: left allmulticast mode [ 1502.086232][T18844] veth1: left allmulticast mode [ 1502.086392][T18844] veth7: left allmulticast mode [ 1502.086553][T18844] lag2.100: left promiscuous mode [ 1502.086694][T18844] lag2: left promiscuous mode [ 1502.086835][T18844] veth1: left promiscuous mode [ 1502.087199][T18844] veth7: left promiscuous mode [ 1502.087850][T18844] br1: port 1(lag2.100) entered disabled state [ 1502.454181][T18848] lag2: Port device veth7 removed [ 1502.513874][T18849] lag2: Port device veth1 removed [ 1503.733507][T18866] lag4: Port device veth5 removed [ 1503.797745][T18867] lag4: Port device veth3 removed [ 1505.133260][T18882] lag1: Port device veth6 removed [ 1505.191832][T18883] lag1: Port device veth0 removed