[ 13.666900][ T297] lag1: Mode changed to "loadbalance" [ 14.147576][ T307] lag1: Port device veth0 added [ 14.154635][ T307] ip (307) used greatest stack depth: 24136 bytes left [ 14.258283][ T308] lag1: Port device veth6 added [ 15.229890][ T320] 8021q: 802.1Q VLAN Support v1.8 [ 15.230284][ T320] 8021q: adding VLAN 0 to HW filter on device lag1 [ 16.481169][ T339] lag4: Mode changed to "loadbalance" [ 16.572082][ T341] 8021q: adding VLAN 0 to HW filter on device lag4 [ 16.974840][ T347] [ 16.974991][ T347] ====================================================== [ 16.975201][ T347] WARNING: possible circular locking dependency detected [ 16.975409][ T347] 6.13.0-rc5-virtme #1 Not tainted [ 16.975572][ T347] ------------------------------------------------------ [ 16.975785][ T347] ip/347 is trying to acquire lock: [ 16.975946][ T347] ffffffff863d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 16.976290][ T347] [ 16.976290][ T347] but task is already holding lock: [ 16.976529][ T347] ffff88800d25ad80 (team->team_lock_key#2){+.+.}-{4:4}, at: team_add_slave+0x33/0x80 [ 16.976829][ T347] [ 16.976829][ T347] which lock already depends on the new lock. [ 16.976829][ T347] [ 16.977142][ T347] [ 16.977142][ T347] the existing dependency chain (in reverse order) is: [ 16.977419][ T347] [ 16.977419][ T347] -> #1 (team->team_lock_key#2){+.+.}-{4:4}: [ 16.977668][ T347] __lock_acquire+0xb9a/0x1680 [ 16.977837][ T347] lock_acquire.part.0+0xeb/0x330 [ 16.977999][ T347] __mutex_lock+0x169/0xbc0 [ 16.978165][ T347] team_vlan_rx_add_vid+0x3b/0x1b0 [ 16.978356][ T347] vlan_vid_add+0x2bc/0x860 [ 16.978521][ T347] vlan_device_event+0x2f4/0x2130 [8021q] [ 16.978742][ T347] notifier_call_chain+0xcd/0x150 [ 16.978925][ T347] blocking_notifier_call_chain+0x66/0x90 [ 16.979130][ T347] __dev_notify_flags+0xe6/0x250 [ 16.979303][ T347] dev_change_flags+0xec/0x160 [ 16.979469][ T347] do_setlink.constprop.0+0x79d/0x2300 [ 16.979674][ T347] rtnl_newlink+0x69c/0xa70 [ 16.979836][ T347] rtnetlink_rcv_msg+0x712/0xc10 [ 16.980001][ T347] netlink_rcv_skb+0x130/0x360 [ 16.980168][ T347] netlink_unicast+0x44b/0x710 [ 16.980335][ T347] netlink_sendmsg+0x723/0xbe0 [ 16.980497][ T347] ____sys_sendmsg+0x7ac/0xa10 [ 16.980665][ T347] ___sys_sendmsg+0xee/0x170 [ 16.980858][ T347] __sys_sendmsg+0x109/0x1a0 [ 16.981025][ T347] do_syscall_64+0xc1/0x1d0 [ 16.981193][ T347] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 16.981399][ T347] [ 16.981399][ T347] -> #0 ((netdev_chain).rwsem){++++}-{4:4}: [ 16.981646][ T347] check_prev_add+0x182/0xce0 [ 16.981813][ T347] validate_chain+0x7bd/0xae0 [ 16.981977][ T347] __lock_acquire+0xb9a/0x1680 [ 16.982141][ T347] lock_acquire.part.0+0xeb/0x330 [ 16.982307][ T347] down_read+0x9f/0x340 [ 16.982473][ T347] blocking_notifier_call_chain+0x50/0x90 [ 16.982681][ T347] __dev_open+0x15c/0x430 [ 16.982845][ T347] dev_open+0xc9/0x130 [ 16.983010][ T347] team_port_add+0x90a/0x1ce0 [ 16.983181][ T347] team_add_slave+0x41/0x80 [ 16.983346][ T347] do_setlink.constprop.0+0x80e/0x2300 [ 16.983547][ T347] rtnl_newlink+0x69c/0xa70 [ 16.983721][ T347] rtnetlink_rcv_msg+0x712/0xc10 [ 16.983903][ T347] netlink_rcv_skb+0x130/0x360 [ 16.984076][ T347] netlink_unicast+0x44b/0x710 [ 16.984239][ T347] netlink_sendmsg+0x723/0xbe0 [ 16.984404][ T347] ____sys_sendmsg+0x7ac/0xa10 [ 16.984578][ T347] ___sys_sendmsg+0xee/0x170 [ 16.984741][ T347] __sys_sendmsg+0x109/0x1a0 [ 16.984902][ T347] do_syscall_64+0xc1/0x1d0 [ 16.985065][ T347] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 16.985266][ T347] [ 16.985266][ T347] other info that might help us debug this: [ 16.985266][ T347] [ 16.985578][ T347] Possible unsafe locking scenario: [ 16.985578][ T347] [ 16.985823][ T347] CPU0 CPU1 [ 16.985987][ T347] ---- ---- [ 16.986147][ T347] lock(team->team_lock_key#2); [ 16.986316][ T347] lock((netdev_chain).rwsem); [ 16.986553][ T347] lock(team->team_lock_key#2); [ 16.986806][ T347] rlock((netdev_chain).rwsem); [ 16.986980][ T347] [ 16.986980][ T347] *** DEADLOCK *** [ 16.986980][ T347] [ 16.987216][ T347] 2 locks held by ip/347: [ 16.987340][ T347] #0: ffffffff863e7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x653/0xa70 [ 16.987630][ T347] #1: ffff88800d25ad80 (team->team_lock_key#2){+.+.}-{4:4}, at: team_add_slave+0x33/0x80 [ 16.987920][ T347] [ 16.987920][ T347] stack backtrace: [ 16.988125][ T347] CPU: 3 UID: 0 PID: 347 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 16.988386][ T347] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 16.988587][ T347] Call Trace: [ 16.988720][ T347] [ 16.988824][ T347] dump_stack_lvl+0x82/0xd0 [ 16.988994][ T347] print_circular_bug+0x2e2/0x4b0 [ 16.989158][ T347] check_noncircular+0x2f9/0x3e0 [ 16.989325][ T347] ? __pfx_check_noncircular+0x10/0x10 [ 16.989498][ T347] ? is_bpf_text_address+0x67/0x120 [ 16.989664][ T347] ? alloc_chain_hlocks+0x4e0/0x720 [ 16.989828][ T347] check_prev_add+0x182/0xce0 [ 16.989991][ T347] validate_chain+0x7bd/0xae0 [ 16.990152][ T347] ? __pfx_validate_chain+0x10/0x10 [ 16.990318][ T347] ? hlock_class+0x4e/0x130 [ 16.990483][ T347] ? mark_lock+0x38/0x3e0 [ 16.990607][ T347] __lock_acquire+0xb9a/0x1680 [ 16.990771][ T347] lock_acquire.part.0+0xeb/0x330 [ 16.990933][ T347] ? blocking_notifier_call_chain+0x50/0x90 [ 16.991138][ T347] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 16.991300][ T347] ? trace_lock_acquire+0x14c/0x1f0 [ 16.991463][ T347] ? lock_acquire+0x32/0xc0 [ 16.991629][ T347] ? blocking_notifier_call_chain+0x50/0x90 [ 16.991831][ T347] down_read+0x9f/0x340 [ 16.991954][ T347] ? blocking_notifier_call_chain+0x50/0x90 [ 16.992156][ T347] ? __pfx_down_read+0x10/0x10 [ 16.992318][ T347] blocking_notifier_call_chain+0x50/0x90 [ 16.992479][ T347] __dev_open+0x15c/0x430 [ 16.992603][ T347] ? trace_percpu_alloc_percpu+0x149/0x1d0 [ 16.992840][ T347] ? __pfx___dev_open+0x10/0x10 [ 16.993004][ T347] ? pcpu_alloc_noprof+0x4fe/0xab0 [ 16.993171][ T347] dev_open+0xc9/0x130 [ 16.993295][ T347] ? __pfx_dev_open+0x10/0x10 [ 16.993459][ T347] team_port_add+0x90a/0x1ce0 [ 16.993623][ T347] ? __pfx_team_port_add+0x10/0x10 [ 16.993784][ T347] ? __pfx_validate_chain+0x10/0x10 [ 16.993945][ T347] ? __pfx_validate_linkmsg+0x10/0x10 [ 16.994112][ T347] team_add_slave+0x41/0x80 [ 16.994274][ T347] do_setlink.constprop.0+0x80e/0x2300 [ 16.994436][ T347] ? __pfx_do_setlink.constprop.0+0x10/0x10 [ 16.994635][ T347] ? lock_acquire.part.0+0xeb/0x330 [ 16.994798][ T347] ? rtnl_newlink+0x653/0xa70 [ 16.994960][ T347] ? rtnl_newlink+0xb9/0xa70 [ 16.995119][ T347] ? rtnetlink_rcv_msg+0x712/0xc10 [ 16.995285][ T347] ? __mutex_trylock_common+0xfa/0x260 [ 16.995448][ T347] ? __pfx___mutex_trylock_common+0x10/0x10 [ 16.995650][ T347] ? lock_acquire+0x32/0xc0 [ 16.995811][ T347] ? trace_contention_end+0xef/0x150 [ 16.995972][ T347] ? __mutex_lock+0x190/0xbc0 [ 16.996134][ T347] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 16.996301][ T347] ? rtnl_newlink+0x653/0xa70 [ 16.996462][ T347] ? __pfx___mutex_lock+0x10/0x10 [ 16.996625][ T347] ? __rtnl_newlink+0x3f2/0xa80 [ 16.996790][ T347] rtnl_newlink+0x69c/0xa70 [ 16.996954][ T347] ? __pfx_rtnl_newlink+0x10/0x10 [ 16.997119][ T347] ? find_held_lock+0x2c/0x110 [ 16.997285][ T347] ? __pfx___lock_release+0x10/0x10 [ 16.997451][ T347] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 16.997618][ T347] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 16.997811][ T347] ? __pfx_rtnl_newlink+0x10/0x10 [ 16.997972][ T347] rtnetlink_rcv_msg+0x712/0xc10 [ 16.998150][ T347] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 16.998328][ T347] ? hlock_class+0x4e/0x130 [ 16.998514][ T347] ? mark_lock+0x38/0x3e0 [ 16.998638][ T347] ? __lock_acquire+0xb9a/0x1680 [ 16.998799][ T347] netlink_rcv_skb+0x130/0x360 [ 16.998966][ T347] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 16.999128][ T347] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 16.999292][ T347] ? netlink_deliver_tap+0x13e/0x340 [ 16.999455][ T347] ? netlink_deliver_tap+0xc3/0x340 [ 16.999622][ T347] netlink_unicast+0x44b/0x710 [ 16.999786][ T347] ? __pfx_netlink_unicast+0x10/0x10 [ 16.999952][ T347] ? find_held_lock+0x2c/0x110 [ 17.000119][ T347] netlink_sendmsg+0x723/0xbe0 [ 17.000280][ T347] ? __pfx_netlink_sendmsg+0x10/0x10 [ 17.000443][ T347] ____sys_sendmsg+0x7ac/0xa10 [ 17.000607][ T347] ? __pfx_____sys_sendmsg+0x10/0x10 [ 17.000768][ T347] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 17.000975][ T347] ___sys_sendmsg+0xee/0x170 [ 17.001138][ T347] ? __pfx____sys_sendmsg+0x10/0x10 [ 17.001304][ T347] ? kasan_save_stack+0x34/0x50 [ 17.001469][ T347] ? kasan_save_stack+0x24/0x50 [ 17.001634][ T347] ? __kasan_record_aux_stack+0x8e/0xa0 [ 17.001799][ T347] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 17.002005][ T347] ? __x64_sys_close+0x7c/0xd0 [ 17.002171][ T347] ? do_syscall_64+0xc1/0x1d0 [ 17.002333][ T347] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.002546][ T347] ? __lock_acquire+0xb9a/0x1680 [ 17.002710][ T347] ? find_held_lock+0x2c/0x110 [ 17.002901][ T347] ? __lock_release+0x103/0x460 [ 17.003062][ T347] ? __virt_addr_valid+0x22b/0x430 [ 17.003228][ T347] ? __pfx___lock_release+0x10/0x10 [ 17.003390][ T347] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.003560][ T347] __sys_sendmsg+0x109/0x1a0 [ 17.003753][ T347] ? __pfx___sys_sendmsg+0x10/0x10 [ 17.003927][ T347] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 17.004139][ T347] do_syscall_64+0xc1/0x1d0 [ 17.004305][ T347] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.004505][ T347] RIP: 0033:0x7f17af9519a7 [ 17.004677][ T347] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 17.005247][ T347] RSP: 002b:00007ffc98021b68 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 17.005489][ T347] RAX: ffffffffffffffda RBX: 00007ffc98022290 RCX: 00007f17af9519a7 [ 17.005735][ T347] RDX: 0000000000000000 RSI: 00007ffc98021bd0 RDI: 0000000000000005 [ 17.005985][ T347] RBP: 0000000000000004 R08: 0000000000000003 R09: 0000000000000000 [ 17.006230][ T347] R10: 00000000006c0910 R11: 0000000000000246 R12: 0000000000000004 [ 17.006475][ T347] R13: 0000000067790255 R14: 0000000000498600 R15: 0000000000000000 [ 17.006720][ T347] [ 17.008594][ T347] lag4: Port device veth3 added [ 17.021468][ T347] ip (347) used greatest stack depth: 23800 bytes left [ 17.080502][ T348] lag4: Port device veth5 added [ 17.434619][ T354] 8021q: adding VLAN 0 to HW filter on device lag4 [ 18.621047][ T377] lag2: Mode changed to "loadbalance" [ 18.678043][ T379] 8021q: adding VLAN 0 to HW filter on device lag2 [ 18.974555][ T385] lag2: Port device veth1 added [ 19.056735][ T386] lag2: Port device veth7 added [ 19.810568][ T400] br1: port 1(lag2.100) entered blocking state [ 19.810806][ T400] br1: port 1(lag2.100) entered disabled state [ 19.811086][ T400] lag2.100: entered allmulticast mode [ 19.811234][ T400] lag2: entered allmulticast mode [ 19.811391][ T400] veth1: entered allmulticast mode [ 19.811542][ T400] veth7: entered allmulticast mode [ 19.813696][ T400] lag2.100: entered promiscuous mode [ 19.813857][ T400] lag2: entered promiscuous mode [ 19.814004][ T400] veth1: entered promiscuous mode [ 19.814339][ T400] veth7: entered promiscuous mode [ 20.096050][ T407] br2: port 1(lag2.200) entered blocking state [ 20.096328][ T407] br2: port 1(lag2.200) entered disabled state [ 20.096567][ T407] lag2.200: entered allmulticast mode [ 20.098118][ T407] lag2.200: entered promiscuous mode [ 20.289428][ T410] br1: port 1(lag2.100) entered blocking state [ 20.289782][ T410] br1: port 1(lag2.100) entered forwarding state [ 20.358708][ T411] br2: port 1(lag2.200) entered blocking state [ 20.358968][ T411] br2: port 1(lag2.200) entered forwarding state [ 20.705645][ T419] lag3: Mode changed to "loadbalance" [ 20.787479][ T421] 8021q: adding VLAN 0 to HW filter on device lag3 [ 21.084596][ T427] lag3: Port device veth2 added [ 21.152274][ T428] lag3: Port device veth4 added [ 32.687481][ T510] veth7: left promiscuous mode [ 32.687897][ T510] veth7: left allmulticast mode [ 32.689218][ T510] lag2: Port device veth7 removed [ 47.893829][ T571] veth7: entered promiscuous mode [ 47.894229][ T571] veth7: entered allmulticast mode [ 47.895167][ T571] lag2: Port device veth7 added [ 48.184813][ T582] veth1: left promiscuous mode [ 48.185213][ T582] veth1: left allmulticast mode [ 48.192929][ T582] lag2: Port device veth1 removed [ 63.320590][ T39] br1: port 1(lag2.100) entered disabled state [ 63.323466][ T39] br2: port 1(lag2.200) entered disabled state [ 63.405444][ T644] veth7: left promiscuous mode [ 63.406007][ T644] veth7: left allmulticast mode [ 63.407778][ T644] lag2: Port device veth7 removed [ 63.686701][ T655] veth1: entered promiscuous mode [ 63.687078][ T655] veth1: entered allmulticast mode [ 63.687935][ T655] lag2: Port device veth1 added [ 63.710829][ T377] br2: port 1(lag2.200) entered blocking state [ 63.711159][ T377] br2: port 1(lag2.200) entered forwarding state [ 63.713122][ T40] br1: port 1(lag2.100) entered blocking state [ 63.713387][ T40] br1: port 1(lag2.100) entered forwarding state [ 63.924082][ T666] veth7: entered promiscuous mode [ 63.924458][ T666] veth7: entered allmulticast mode [ 63.925184][ T666] lag2: Port device veth7 added [ 78.989796][ T726] lag2.200: left allmulticast mode [ 78.990047][ T726] lag2.200: left promiscuous mode [ 78.990610][ T726] br2: port 1(lag2.200) entered disabled state [ 79.056415][ T727] lag2.100: left allmulticast mode [ 79.056602][ T727] lag2: left allmulticast mode [ 79.056762][ T727] veth1: left allmulticast mode [ 79.056931][ T727] veth7: left allmulticast mode [ 79.057090][ T727] lag2.100: left promiscuous mode [ 79.057233][ T727] lag2: left promiscuous mode [ 79.058282][ T727] veth1: left promiscuous mode [ 79.058703][ T727] veth7: left promiscuous mode [ 79.059452][ T727] br1: port 1(lag2.100) entered disabled state [ 81.162776][ T729] br1: port 1(lag2.100) entered blocking state [ 81.163079][ T729] br1: port 1(lag2.100) entered disabled state [ 81.163379][ T729] lag2.100: entered allmulticast mode [ 81.163537][ T729] lag2: entered allmulticast mode [ 81.163702][ T729] veth1: entered allmulticast mode [ 81.163867][ T729] veth7: entered allmulticast mode [ 81.165866][ T729] lag2.100: entered promiscuous mode [ 81.166070][ T729] lag2: entered promiscuous mode [ 81.166254][ T729] veth1: entered promiscuous mode [ 81.166676][ T729] veth7: entered promiscuous mode [ 81.168282][ T729] br1: port 1(lag2.100) entered blocking state [ 81.168575][ T729] br1: port 1(lag2.100) entered forwarding state [ 81.236273][ T730] br2: port 1(lag2.200) entered blocking state [ 81.236597][ T730] br2: port 1(lag2.200) entered disabled state [ 81.236818][ T730] lag2.200: entered allmulticast mode [ 81.238456][ T730] lag2.200: entered promiscuous mode [ 81.239603][ T730] br2: port 1(lag2.200) entered blocking state [ 81.239843][ T730] br2: port 1(lag2.200) entered forwarding state [ 96.397567][ T790] lag3: Port device veth2 removed [ 111.535686][ T851] lag3: Port device veth2 added [ 111.786911][ T862] lag3: Port device veth4 removed [ 126.983600][ T923] lag3: Port device veth2 removed [ 127.258136][ T934] lag3: Port device veth4 added [ 127.497509][ T945] lag3: Port device veth2 added [ 143.246893][ T1015] lag3: Port device veth4 removed [ 143.306723][ T1016] lag3: Port device veth2 removed [ 143.997608][ T1028] br2: port 1(lag2.200) entered disabled state [ 144.058994][ T1029] br1: port 1(lag2.100) entered disabled state [ 144.251189][ T1032] lag2.200: left allmulticast mode [ 144.251412][ T1032] lag2.200: left promiscuous mode [ 144.251898][ T1032] br2: port 1(lag2.200) entered disabled state [ 144.434725][ T1034] lag2.100: left allmulticast mode [ 144.434963][ T1034] lag2: left allmulticast mode [ 144.435120][ T1034] veth1: left allmulticast mode [ 144.435277][ T1034] veth7: left allmulticast mode [ 144.437294][ T1034] lag2.100: left promiscuous mode [ 144.437764][ T1034] lag2: left promiscuous mode [ 144.437992][ T1034] veth1: left promiscuous mode [ 144.438528][ T1034] veth7: left promiscuous mode [ 144.439254][ T1034] br1: port 1(lag2.100) entered disabled state [ 144.810612][ T1038] lag2: Port device veth7 removed [ 144.873244][ T1039] lag2: Port device veth1 removed [ 146.194210][ T1056] lag4: Port device veth5 removed [ 146.254899][ T1057] lag4: Port device veth3 removed [ 147.590739][ T1072] lag1: Port device veth6 removed [ 147.662583][ T1073] lag1: Port device veth0 removed