[ 13.552248][ T300] lag1: Mode changed to "loadbalance" [ 14.061280][ T309] lag1: Port device veth0 added [ 14.154289][ T310] lag1: Port device veth6 added [ 15.174186][ T322] 8021q: 802.1Q VLAN Support v1.8 [ 15.174567][ T322] 8021q: adding VLAN 0 to HW filter on device lag1 [ 16.517758][ T341] lag4: Mode changed to "loadbalance" [ 16.610455][ T343] 8021q: adding VLAN 0 to HW filter on device lag4 [ 16.999587][ T349] [ 16.999718][ T349] ====================================================== [ 16.999926][ T349] WARNING: possible circular locking dependency detected [ 17.000132][ T349] 6.13.0-rc5-virtme #1 Not tainted [ 17.000300][ T349] ------------------------------------------------------ [ 17.000505][ T349] ip/349 is trying to acquire lock: [ 17.000671][ T349] ffffffff985d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.001015][ T349] [ 17.001015][ T349] but task is already holding lock: [ 17.001262][ T349] ffff88800da6ad80 (team->team_lock_key#2){+.+.}-{4:4}, at: team_add_slave+0x33/0x80 [ 17.001558][ T349] [ 17.001558][ T349] which lock already depends on the new lock. [ 17.001558][ T349] [ 17.001875][ T349] [ 17.001875][ T349] the existing dependency chain (in reverse order) is: [ 17.002160][ T349] [ 17.002160][ T349] -> #1 (team->team_lock_key#2){+.+.}-{4:4}: [ 17.002404][ T349] __lock_acquire+0xb9a/0x1680 [ 17.002571][ T349] lock_acquire.part.0+0xeb/0x330 [ 17.002741][ T349] __mutex_lock+0x169/0xbc0 [ 17.002908][ T349] team_vlan_rx_add_vid+0x3b/0x1b0 [ 17.003082][ T349] vlan_vid_add+0x2bc/0x860 [ 17.003253][ T349] vlan_device_event+0x2f4/0x2130 [8021q] [ 17.003465][ T349] notifier_call_chain+0xcd/0x150 [ 17.003629][ T349] blocking_notifier_call_chain+0x66/0x90 [ 17.003833][ T349] __dev_notify_flags+0xe6/0x250 [ 17.003999][ T349] dev_change_flags+0xec/0x160 [ 17.004162][ T349] do_setlink.constprop.0+0x79d/0x2300 [ 17.004363][ T349] rtnl_newlink+0x69c/0xa70 [ 17.004528][ T349] rtnetlink_rcv_msg+0x712/0xc10 [ 17.004744][ T349] netlink_rcv_skb+0x130/0x360 [ 17.004984][ T349] netlink_unicast+0x44b/0x710 [ 17.005228][ T349] netlink_sendmsg+0x723/0xbe0 [ 17.005485][ T349] ____sys_sendmsg+0x7ac/0xa10 [ 17.005685][ T349] ___sys_sendmsg+0xee/0x170 [ 17.005911][ T349] __sys_sendmsg+0x109/0x1a0 [ 17.006162][ T349] do_syscall_64+0xc1/0x1d0 [ 17.006388][ T349] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.006679][ T349] [ 17.006679][ T349] -> #0 ((netdev_chain).rwsem){++++}-{4:4}: [ 17.007051][ T349] check_prev_add+0x182/0xce0 [ 17.007307][ T349] validate_chain+0x7bd/0xae0 [ 17.007564][ T349] __lock_acquire+0xb9a/0x1680 [ 17.007818][ T349] lock_acquire.part.0+0xeb/0x330 [ 17.008064][ T349] down_read+0x9f/0x340 [ 17.008307][ T349] blocking_notifier_call_chain+0x50/0x90 [ 17.008612][ T349] __dev_open+0x15c/0x430 [ 17.008867][ T349] dev_open+0xc9/0x130 [ 17.009119][ T349] team_port_add+0x90a/0x1ce0 [ 17.009382][ T349] team_add_slave+0x41/0x80 [ 17.009636][ T349] do_setlink.constprop.0+0x80e/0x2300 [ 17.009954][ T349] rtnl_newlink+0x69c/0xa70 [ 17.010205][ T349] rtnetlink_rcv_msg+0x712/0xc10 [ 17.010458][ T349] netlink_rcv_skb+0x130/0x360 [ 17.010713][ T349] netlink_unicast+0x44b/0x710 [ 17.010967][ T349] netlink_sendmsg+0x723/0xbe0 [ 17.011222][ T349] ____sys_sendmsg+0x7ac/0xa10 [ 17.011475][ T349] ___sys_sendmsg+0xee/0x170 [ 17.011730][ T349] __sys_sendmsg+0x109/0x1a0 [ 17.011984][ T349] do_syscall_64+0xc1/0x1d0 [ 17.012236][ T349] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.012552][ T349] [ 17.012552][ T349] other info that might help us debug this: [ 17.012552][ T349] [ 17.013023][ T349] Possible unsafe locking scenario: [ 17.013023][ T349] [ 17.013346][ T349] CPU0 CPU1 [ 17.013508][ T349] ---- ---- [ 17.013668][ T349] lock(team->team_lock_key#2); [ 17.013832][ T349] lock((netdev_chain).rwsem); [ 17.014072][ T349] lock(team->team_lock_key#2); [ 17.014333][ T349] rlock((netdev_chain).rwsem); [ 17.014494][ T349] [ 17.014494][ T349] *** DEADLOCK *** [ 17.014494][ T349] [ 17.014731][ T349] 2 locks held by ip/349: [ 17.014855][ T349] #0: ffffffff985e7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x653/0xa70 [ 17.015160][ T349] #1: ffff88800da6ad80 (team->team_lock_key#2){+.+.}-{4:4}, at: team_add_slave+0x33/0x80 [ 17.015533][ T349] [ 17.015533][ T349] stack backtrace: [ 17.015734][ T349] CPU: 0 UID: 0 PID: 349 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 17.015980][ T349] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 17.016192][ T349] Call Trace: [ 17.016318][ T349] <TASK> [ 17.016403][ T349] dump_stack_lvl+0x82/0xd0 [ 17.016578][ T349] print_circular_bug+0x2e2/0x4b0 [ 17.016743][ T349] check_noncircular+0x2f9/0x3e0 [ 17.016912][ T349] ? __pfx_check_noncircular+0x10/0x10 [ 17.017079][ T349] ? is_bpf_text_address+0x67/0x120 [ 17.017258][ T349] ? alloc_chain_hlocks+0x4e0/0x720 [ 17.017423][ T349] check_prev_add+0x182/0xce0 [ 17.017590][ T349] validate_chain+0x7bd/0xae0 [ 17.017754][ T349] ? __pfx_validate_chain+0x10/0x10 [ 17.017923][ T349] ? hlock_class+0x4e/0x130 [ 17.018088][ T349] ? mark_lock+0x38/0x3e0 [ 17.018224][ T349] __lock_acquire+0xb9a/0x1680 [ 17.018388][ T349] lock_acquire.part.0+0xeb/0x330 [ 17.018561][ T349] ? blocking_notifier_call_chain+0x50/0x90 [ 17.018765][ T349] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.018929][ T349] ? trace_lock_acquire+0x14c/0x1f0 [ 17.019096][ T349] ? lock_acquire+0x32/0xc0 [ 17.019265][ T349] ? blocking_notifier_call_chain+0x50/0x90 [ 17.019475][ T349] down_read+0x9f/0x340 [ 17.019601][ T349] ? blocking_notifier_call_chain+0x50/0x90 [ 17.019803][ T349] ? __pfx_down_read+0x10/0x10 [ 17.019968][ T349] blocking_notifier_call_chain+0x50/0x90 [ 17.020144][ T349] __dev_open+0x15c/0x430 [ 17.020301][ T349] ? trace_percpu_alloc_percpu+0x149/0x1d0 [ 17.020506][ T349] ? __pfx___dev_open+0x10/0x10 [ 17.020668][ T349] ? pcpu_alloc_noprof+0x4fe/0xab0 [ 17.020834][ T349] dev_open+0xc9/0x130 [ 17.020967][ T349] ? __pfx_dev_open+0x10/0x10 [ 17.021142][ T349] team_port_add+0x90a/0x1ce0 [ 17.021308][ T349] ? __pfx_team_port_add+0x10/0x10 [ 17.021476][ T349] ? __pfx_validate_chain+0x10/0x10 [ 17.021641][ T349] ? __pfx_validate_linkmsg+0x10/0x10 [ 17.021806][ T349] team_add_slave+0x41/0x80 [ 17.021970][ T349] do_setlink.constprop.0+0x80e/0x2300 [ 17.022145][ T349] ? __pfx_do_setlink.constprop.0+0x10/0x10 [ 17.022351][ T349] ? lock_acquire.part.0+0xeb/0x330 [ 17.022515][ T349] ? rtnl_newlink+0x653/0xa70 [ 17.022683][ T349] ? rtnl_newlink+0xb9/0xa70 [ 17.022845][ T349] ? rtnetlink_rcv_msg+0x712/0xc10 [ 17.023007][ T349] ? __mutex_trylock_common+0xfa/0x260 [ 17.023183][ T349] ? __pfx___mutex_trylock_common+0x10/0x10 [ 17.023386][ T349] ? lock_acquire+0x32/0xc0 [ 17.023555][ T349] ? trace_contention_end+0xef/0x150 [ 17.023718][ T349] ? __mutex_lock+0x190/0xbc0 [ 17.023885][ T349] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 17.024052][ T349] ? rtnl_newlink+0x653/0xa70 [ 17.024222][ T349] ? __pfx___mutex_lock+0x10/0x10 [ 17.024386][ T349] ? __rtnl_newlink+0x3f2/0xa80 [ 17.024551][ T349] rtnl_newlink+0x69c/0xa70 [ 17.024716][ T349] ? __pfx_rtnl_newlink+0x10/0x10 [ 17.024879][ T349] ? find_held_lock+0x2c/0x110 [ 17.025043][ T349] ? __pfx___lock_release+0x10/0x10 [ 17.025219][ T349] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.025383][ T349] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 17.025559][ T349] ? __pfx_rtnl_newlink+0x10/0x10 [ 17.025721][ T349] rtnetlink_rcv_msg+0x712/0xc10 [ 17.025883][ T349] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 17.026049][ T349] ? hlock_class+0x4e/0x130 [ 17.026220][ T349] ? mark_lock+0x38/0x3e0 [ 17.026351][ T349] ? __lock_acquire+0xb9a/0x1680 [ 17.026514][ T349] netlink_rcv_skb+0x130/0x360 [ 17.026676][ T349] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 17.026842][ T349] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 17.027009][ T349] ? netlink_deliver_tap+0x13e/0x340 [ 17.027191][ T349] ? netlink_deliver_tap+0xc3/0x340 [ 17.027362][ T349] netlink_unicast+0x44b/0x710 [ 17.027534][ T349] ? __pfx_netlink_unicast+0x10/0x10 [ 17.027695][ T349] ? find_held_lock+0x2c/0x110 [ 17.027859][ T349] netlink_sendmsg+0x723/0xbe0 [ 17.028025][ T349] ? __pfx_netlink_sendmsg+0x10/0x10 [ 17.028202][ T349] ____sys_sendmsg+0x7ac/0xa10 [ 17.028368][ T349] ? __pfx_____sys_sendmsg+0x10/0x10 [ 17.028529][ T349] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 17.028741][ T349] ___sys_sendmsg+0xee/0x170 [ 17.028909][ T349] ? __pfx____sys_sendmsg+0x10/0x10 [ 17.029073][ T349] ? kasan_save_stack+0x34/0x50 [ 17.029249][ T349] ? kasan_save_stack+0x24/0x50 [ 17.029412][ T349] ? __kasan_record_aux_stack+0x8e/0xa0 [ 17.029577][ T349] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 17.029781][ T349] ? __x64_sys_close+0x7c/0xd0 [ 17.029951][ T349] ? do_syscall_64+0xc1/0x1d0 [ 17.030114][ T349] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.030332][ T349] ? __lock_acquire+0xb9a/0x1680 [ 17.030500][ T349] ? find_held_lock+0x2c/0x110 [ 17.030666][ T349] ? __lock_release+0x103/0x460 [ 17.030829][ T349] ? __virt_addr_valid+0x22b/0x430 [ 17.030994][ T349] ? __pfx___lock_release+0x10/0x10 [ 17.031177][ T349] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.031350][ T349] __sys_sendmsg+0x109/0x1a0 [ 17.031515][ T349] ? __pfx___sys_sendmsg+0x10/0x10 [ 17.031680][ T349] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 17.031889][ T349] do_syscall_64+0xc1/0x1d0 [ 17.032054][ T349] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.032268][ T349] RIP: 0033:0x7faf8a6e79a7 [ 17.032439][ T349] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 17.033020][ T349] RSP: 002b:00007ffdd84fa2f8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 17.033267][ T349] RAX: ffffffffffffffda RBX: 00007ffdd84faa20 RCX: 00007faf8a6e79a7 [ 17.033517][ T349] RDX: 0000000000000000 RSI: 00007ffdd84fa360 RDI: 0000000000000005 [ 17.033760][ T349] RBP: 0000000000000004 R08: 0000000000000003 R09: 0000000000000000 [ 17.034007][ T349] R10: 00000000062c7910 R11: 0000000000000246 R12: 0000000000000004 [ 17.034252][ T349] R13: 0000000067792ce7 R14: 0000000000498600 R15: 0000000000000000 [ 17.034501][ T349] </TASK> [ 17.036581][ T349] lag4: Port device veth3 added [ 17.042913][ T349] ip (349) used greatest stack depth: 23648 bytes left [ 17.102333][ T350] lag4: Port device veth5 added [ 17.425141][ T356] 8021q: adding VLAN 0 to HW filter on device lag4 [ 18.619624][ T379] lag2: Mode changed to "loadbalance" [ 18.687145][ T381] 8021q: adding VLAN 0 to HW filter on device lag2 [ 18.981341][ T387] lag2: Port device veth1 added [ 19.060176][ T388] lag2: Port device veth7 added [ 19.832703][ T403] br1: port 1(lag2.100) entered blocking state [ 19.833051][ T403] br1: port 1(lag2.100) entered disabled state [ 19.833421][ T403] lag2.100: entered allmulticast mode [ 19.833640][ T403] lag2: entered allmulticast mode [ 19.833830][ T403] veth1: entered allmulticast mode [ 19.834038][ T403] veth7: entered allmulticast mode [ 19.842021][ T403] lag2.100: entered promiscuous mode [ 19.842323][ T403] lag2: entered promiscuous mode [ 19.842498][ T403] veth1: entered promiscuous mode [ 19.842879][ T403] veth7: entered promiscuous mode [ 20.138418][ T410] br2: port 1(lag2.200) entered blocking state [ 20.138744][ T410] br2: port 1(lag2.200) entered disabled state [ 20.139066][ T410] lag2.200: entered allmulticast mode [ 20.141104][ T410] lag2.200: entered promiscuous mode [ 20.335991][ T413] br1: port 1(lag2.100) entered blocking state [ 20.336349][ T413] br1: port 1(lag2.100) entered forwarding state [ 20.395538][ T414] br2: port 1(lag2.200) entered blocking state [ 20.395779][ T414] br2: port 1(lag2.200) entered forwarding state [ 20.748208][ T422] lag3: Mode changed to "loadbalance" [ 20.822081][ T423] 8021q: adding VLAN 0 to HW filter on device lag3 [ 21.126201][ T430] lag3: Port device veth2 added [ 21.200269][ T431] lag3: Port device veth4 added [ 32.580055][ T512] veth7: left promiscuous mode [ 32.581057][ T512] veth7: left allmulticast mode [ 32.582202][ T512] lag2: Port device veth7 removed [ 47.759132][ T573] veth7: entered promiscuous mode [ 47.759743][ T573] veth7: entered allmulticast mode [ 47.760920][ T573] lag2: Port device veth7 added [ 48.023899][ T584] veth1: left promiscuous mode [ 48.024351][ T584] veth1: left allmulticast mode [ 48.026007][ T584] lag2: Port device veth1 removed [ 63.201564][ T45] br1: port 1(lag2.100) entered disabled state [ 63.202899][ T45] br2: port 1(lag2.200) entered disabled state [ 63.279253][ T645] veth7: left promiscuous mode [ 63.279758][ T645] veth7: left allmulticast mode [ 63.281805][ T645] lag2: Port device veth7 removed [ 63.637178][ T656] veth1: entered promiscuous mode [ 63.637662][ T656] veth1: entered allmulticast mode [ 63.638603][ T656] lag2: Port device veth1 added [ 63.669728][ T255] br1: port 1(lag2.100) entered blocking state [ 63.670095][ T255] br1: port 1(lag2.100) entered forwarding state [ 63.670886][ T255] br2: port 1(lag2.200) entered blocking state [ 63.671161][ T255] br2: port 1(lag2.200) entered forwarding state [ 63.913200][ T667] veth7: entered promiscuous mode [ 63.913720][ T667] veth7: entered allmulticast mode [ 63.916336][ T667] lag2: Port device veth7 added [ 78.971135][ T727] lag2.200: left allmulticast mode [ 78.971824][ T727] lag2.200: left promiscuous mode [ 78.972631][ T727] br2: port 1(lag2.200) entered disabled state [ 79.043380][ T728] lag2.100: left allmulticast mode [ 79.043588][ T728] lag2: left allmulticast mode [ 79.043754][ T728] veth1: left allmulticast mode [ 79.043930][ T728] veth7: left allmulticast mode [ 79.044100][ T728] lag2.100: left promiscuous mode [ 79.044389][ T728] lag2: left promiscuous mode [ 79.044546][ T728] veth1: left promiscuous mode [ 79.044930][ T728] veth7: left promiscuous mode [ 79.045638][ T728] br1: port 1(lag2.100) entered disabled state [ 81.156330][ T730] br1: port 1(lag2.100) entered blocking state [ 81.156556][ T730] br1: port 1(lag2.100) entered disabled state [ 81.156765][ T730] lag2.100: entered allmulticast mode [ 81.156904][ T730] lag2: entered allmulticast mode [ 81.157051][ T730] veth1: entered allmulticast mode [ 81.157215][ T730] veth7: entered allmulticast mode [ 81.158765][ T730] lag2.100: entered promiscuous mode [ 81.158931][ T730] lag2: entered promiscuous mode [ 81.159074][ T730] veth1: entered promiscuous mode [ 81.159436][ T730] veth7: entered promiscuous mode [ 81.160536][ T730] br1: port 1(lag2.100) entered blocking state [ 81.160750][ T730] br1: port 1(lag2.100) entered forwarding state [ 81.223607][ T731] br2: port 1(lag2.200) entered blocking state [ 81.223855][ T731] br2: port 1(lag2.200) entered disabled state [ 81.224078][ T731] lag2.200: entered allmulticast mode [ 81.225927][ T731] lag2.200: entered promiscuous mode [ 81.227060][ T731] br2: port 1(lag2.200) entered blocking state [ 81.227298][ T731] br2: port 1(lag2.200) entered forwarding state [ 96.373727][ T791] lag3: Port device veth2 removed [ 111.590464][ T852] lag3: Port device veth2 added [ 111.831966][ T863] lag3: Port device veth4 removed [ 127.005056][ T924] lag3: Port device veth2 removed [ 127.231011][ T935] lag3: Port device veth4 added [ 127.497437][ T946] lag3: Port device veth2 added [ 143.308872][ T1016] lag3: Port device veth4 removed [ 143.368162][ T1017] lag3: Port device veth2 removed [ 144.018176][ T1029] br2: port 1(lag2.200) entered disabled state [ 144.079668][ T1030] br1: port 1(lag2.100) entered disabled state [ 144.253672][ T1033] lag2.200: left allmulticast mode [ 144.253890][ T1033] lag2.200: left promiscuous mode [ 144.254380][ T1033] br2: port 1(lag2.200) entered disabled state [ 144.418467][ T1035] lag2.100: left allmulticast mode [ 144.418656][ T1035] lag2: left allmulticast mode [ 144.418807][ T1035] veth1: left allmulticast mode [ 144.418957][ T1035] veth7: left allmulticast mode [ 144.419110][ T1035] lag2.100: left promiscuous mode [ 144.419265][ T1035] lag2: left promiscuous mode [ 144.419413][ T1035] veth1: left promiscuous mode [ 144.419759][ T1035] veth7: left promiscuous mode [ 144.420381][ T1035] br1: port 1(lag2.100) entered disabled state [ 144.777008][ T1039] lag2: Port device veth7 removed [ 144.840666][ T1040] lag2: Port device veth1 removed [ 146.137747][ T1057] lag4: Port device veth5 removed [ 146.190654][ T1058] lag4: Port device veth3 removed [ 147.563250][ T1073] lag1: Port device veth6 removed [ 147.633814][ T1074] lag1: Port device veth0 removed