[ 10.540093][ T250] ip (250) used greatest stack depth: 23136 bytes left [ 12.926907][ T296] 8021q: 802.1Q VLAN Support v1.8 [ 16.486460][ T343] gre: GRE over IPv4 demultiplexor driver [ 16.500923][ T343] ip_gre: GRE over IPv4 tunneling driver [ 17.585503][ T360] lag: Mode changed to "loadbalance" [ 17.785026][ T363] lag: Port device veth2 added [ 17.786281][ T360] [ 17.786408][ T360] ====================================================== [ 17.786610][ T360] WARNING: possible circular locking dependency detected [ 17.786809][ T360] 6.13.0-rc5-virtme #1 Not tainted [ 17.786963][ T360] ------------------------------------------------------ [ 17.787158][ T360] teamd/360 is trying to acquire lock: [ 17.787311][ T360] ffff88800de4ad80 (team->team_lock_key){+.+.}-{4:4}, at: team_port_change_check+0x43/0x130 [ 17.787646][ T360] [ 17.787646][ T360] but task is already holding lock: [ 17.787876][ T360] ffffffffa17d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.788196][ T360] [ 17.788196][ T360] which lock already depends on the new lock. [ 17.788196][ T360] [ 17.788491][ T360] [ 17.788491][ T360] the existing dependency chain (in reverse order) is: [ 17.788782][ T360] [ 17.788782][ T360] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 17.789037][ T360] __lock_acquire+0xb9a/0x1680 [ 17.789193][ T360] lock_acquire.part.0+0xeb/0x330 [ 17.789371][ T360] down_read+0x9f/0x340 [ 17.789531][ T360] blocking_notifier_call_chain+0x50/0x90 [ 17.789723][ T360] __dev_open+0x15c/0x430 [ 17.789894][ T360] dev_open+0xc9/0x130 [ 17.790057][ T360] team_port_add+0x90a/0x1ce0 [ 17.790222][ T360] team_add_slave+0x41/0x80 [ 17.790374][ T360] do_setlink.constprop.0+0x80e/0x2300 [ 17.790569][ T360] rtnl_newlink+0x69c/0xa70 [ 17.790731][ T360] rtnetlink_rcv_msg+0x712/0xc10 [ 17.790888][ T360] netlink_rcv_skb+0x130/0x360 [ 17.791040][ T360] netlink_unicast+0x44b/0x710 [ 17.791204][ T360] netlink_sendmsg+0x723/0xbe0 [ 17.791379][ T360] ____sys_sendmsg+0x7ac/0xa10 [ 17.791531][ T360] ___sys_sendmsg+0xee/0x170 [ 17.791690][ T360] __sys_sendmsg+0x109/0x1a0 [ 17.791842][ T360] do_syscall_64+0xc1/0x1d0 [ 17.792028][ T360] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.792229][ T360] [ 17.792229][ T360] -> #0 (team->team_lock_key){+.+.}-{4:4}: [ 17.792476][ T360] check_prev_add+0x182/0xce0 [ 17.792643][ T360] validate_chain+0x7bd/0xae0 [ 17.792796][ T360] __lock_acquire+0xb9a/0x1680 [ 17.792950][ T360] lock_acquire.part.0+0xeb/0x330 [ 17.793120][ T360] __mutex_lock+0x169/0xbc0 [ 17.793286][ T360] team_port_change_check+0x43/0x130 [ 17.793506][ T360] team_device_event+0x3fe/0x520 [ 17.793660][ T360] notifier_call_chain+0xcd/0x150 [ 17.793827][ T360] blocking_notifier_call_chain+0x66/0x90 [ 17.794017][ T360] netdev_state_change+0xf5/0x120 [ 17.794180][ T360] linkwatch_do_dev+0xd2/0x100 [ 17.794345][ T360] rtnl_getlink+0x4cf/0x970 [ 17.794496][ T360] rtnetlink_rcv_msg+0x2fb/0xc10 [ 17.794660][ T360] netlink_rcv_skb+0x130/0x360 [ 17.794812][ T360] netlink_unicast+0x44b/0x710 [ 17.794969][ T360] netlink_sendmsg+0x723/0xbe0 [ 17.795125][ T360] ____sys_sendmsg+0x7ac/0xa10 [ 17.795288][ T360] ___sys_sendmsg+0xee/0x170 [ 17.795440][ T360] __sys_sendmsg+0x109/0x1a0 [ 17.795593][ T360] do_syscall_64+0xc1/0x1d0 [ 17.795756][ T360] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.795953][ T360] [ 17.795953][ T360] other info that might help us debug this: [ 17.795953][ T360] [ 17.796258][ T360] Possible unsafe locking scenario: [ 17.796258][ T360] [ 17.796483][ T360] CPU0 CPU1 [ 17.796639][ T360] ---- ---- [ 17.796790][ T360] rlock((netdev_chain).rwsem); [ 17.796952][ T360] lock(team->team_lock_key); [ 17.797176][ T360] lock((netdev_chain).rwsem); [ 17.797420][ T360] lock(team->team_lock_key); [ 17.797574][ T360] [ 17.797574][ T360] *** DEADLOCK *** [ 17.797574][ T360] [ 17.797798][ T360] 2 locks held by teamd/360: [ 17.797949][ T360] #0: ffffffffa17e7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnetlink_rcv_msg+0x2af/0xc10 [ 17.798236][ T360] #1: ffffffffa17d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 17.798555][ T360] [ 17.798555][ T360] stack backtrace: [ 17.798741][ T360] CPU: 1 UID: 0 PID: 360 Comm: teamd Not tainted 6.13.0-rc5-virtme #1 [ 17.798981][ T360] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 17.799182][ T360] Call Trace: [ 17.799298][ T360] [ 17.799376][ T360] dump_stack_lvl+0x82/0xd0 [ 17.799539][ T360] print_circular_bug+0x2e2/0x4b0 [ 17.799689][ T360] check_noncircular+0x2f9/0x3e0 [ 17.799847][ T360] ? __pfx_check_noncircular+0x10/0x10 [ 17.800002][ T360] ? alloc_chain_hlocks+0x4e0/0x720 [ 17.800151][ T360] ? __lock_release+0x103/0x460 [ 17.800301][ T360] check_prev_add+0x182/0xce0 [ 17.800462][ T360] validate_chain+0x7bd/0xae0 [ 17.800615][ T360] ? __pfx_validate_chain+0x10/0x10 [ 17.800785][ T360] ? hlock_class+0x4e/0x130 [ 17.800939][ T360] ? mark_lock+0x38/0x3e0 [ 17.801053][ T360] __lock_acquire+0xb9a/0x1680 [ 17.801204][ T360] lock_acquire.part.0+0xeb/0x330 [ 17.801355][ T360] ? team_port_change_check+0x43/0x130 [ 17.801518][ T360] ? __pfx___lock_release+0x10/0x10 [ 17.801669][ T360] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.801825][ T360] ? trace_lock_acquire+0x14c/0x1f0 [ 17.801979][ T360] ? lock_acquire+0x32/0xc0 [ 17.802130][ T360] ? team_port_change_check+0x43/0x130 [ 17.802281][ T360] __mutex_lock+0x169/0xbc0 [ 17.802432][ T360] ? team_port_change_check+0x43/0x130 [ 17.802585][ T360] ? team_port_change_check+0x43/0x130 [ 17.802737][ T360] ? __pfx___mutex_lock+0x10/0x10 [ 17.802889][ T360] ? lock_acquire.part.0+0xeb/0x330 [ 17.803048][ T360] ? blocking_notifier_call_chain+0x50/0x90 [ 17.803242][ T360] ? ref_tracker_free+0x356/0x910 [ 17.803412][ T360] ? __pfx_fib_sync_up+0x10/0x10 [ 17.803566][ T360] ? __pfx_rwsem_read_trylock+0x10/0x10 [ 17.803725][ T360] ? team_port_change_check+0x43/0x130 [ 17.803878][ T360] team_port_change_check+0x43/0x130 [ 17.804029][ T360] team_device_event+0x3fe/0x520 [ 17.804187][ T360] ? trace_notifier_run+0xe7/0x150 [ 17.804348][ T360] notifier_call_chain+0xcd/0x150 [ 17.804502][ T360] blocking_notifier_call_chain+0x66/0x90 [ 17.804658][ T360] netdev_state_change+0xf5/0x120 [ 17.804813][ T360] ? __pfx_netdev_state_change+0x10/0x10 [ 17.804966][ T360] ? __pfx_dev_activate+0x10/0x10 [ 17.805138][ T360] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 17.805349][ T360] linkwatch_do_dev+0xd2/0x100 [ 17.805517][ T360] rtnl_getlink+0x4cf/0x970 [ 17.805683][ T360] ? __pfx_rtnl_getlink+0x10/0x10 [ 17.805861][ T360] ? __mutex_lock+0x190/0xbc0 [ 17.806027][ T360] ? rtnetlink_rcv_msg+0x2af/0xc10 [ 17.806207][ T360] ? __pfx___mutex_lock+0x10/0x10 [ 17.806382][ T360] ? __lock_release+0x103/0x460 [ 17.806568][ T360] ? rtnetlink_rcv_msg+0x297/0xc10 [ 17.806736][ T360] ? __pfx___lock_release+0x10/0x10 [ 17.806908][ T360] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 17.807082][ T360] rtnetlink_rcv_msg+0x2fb/0xc10 [ 17.807263][ T360] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 17.807444][ T360] ? hlock_class+0x4e/0x130 [ 17.807610][ T360] ? mark_lock+0x38/0x3e0 [ 17.807750][ T360] ? __lock_acquire+0xb9a/0x1680 [ 17.807919][ T360] netlink_rcv_skb+0x130/0x360 [ 17.808099][ T360] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 17.808268][ T360] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 17.808448][ T360] ? netlink_deliver_tap+0x13e/0x340 [ 17.808619][ T360] ? netlink_deliver_tap+0xc3/0x340 [ 17.808797][ T360] netlink_unicast+0x44b/0x710 [ 17.808973][ T360] ? __pfx_netlink_unicast+0x10/0x10 [ 17.809148][ T360] ? find_held_lock+0x2c/0x110 [ 17.809324][ T360] netlink_sendmsg+0x723/0xbe0 [ 17.809501][ T360] ? __pfx_netlink_sendmsg+0x10/0x10 [ 17.809672][ T360] ____sys_sendmsg+0x7ac/0xa10 [ 17.809942][ T360] ? __pfx_____sys_sendmsg+0x10/0x10 [ 17.810115][ T360] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 17.810344][ T360] ___sys_sendmsg+0xee/0x170 [ 17.810523][ T360] ? hlock_class+0x4e/0x130 [ 17.810703][ T360] ? __pfx____sys_sendmsg+0x10/0x10 [ 17.810970][ T360] ? usage_skip+0x132/0x190 [ 17.811137][ T360] ? do_user_addr_fault+0x8fd/0xe30 [ 17.811301][ T360] ? do_user_addr_fault+0x8fd/0xe30 [ 17.811470][ T360] ? find_held_lock+0x2c/0x110 [ 17.811738][ T360] ? __lock_release+0x103/0x460 [ 17.811910][ T360] __sys_sendmsg+0x109/0x1a0 [ 17.812076][ T360] ? __pfx___sys_sendmsg+0x10/0x10 [ 17.812241][ T360] ? do_user_addr_fault+0x97c/0xe30 [ 17.812514][ T360] do_syscall_64+0xc1/0x1d0 [ 17.812693][ T360] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 17.812907][ T360] RIP: 0033:0x7f09aee9f9a7 [ 17.813090][ T360] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 17.813804][ T360] RSP: 002b:00007fff75447588 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 17.814054][ T360] RAX: ffffffffffffffda RBX: 000000002d928320 RCX: 00007f09aee9f9a7 [ 17.814377][ T360] RDX: 0000000000000000 RSI: 00007fff754475c0 RDI: 0000000000000005 [ 17.814623][ T360] RBP: 000000002d928230 R08: 0000000000000008 R09: 0000000000000000 [ 17.814891][ T360] R10: 0000000000000200 R11: 0000000000000246 R12: 000000002d950290 [ 17.815136][ T360] R13: 00007fff754475c0 R14: 00007fff754476f8 R15: 000000002d94fd58 [ 17.815468][ T360] [ 18.003031][ T367] lag: Port device veth4 added [ 18.121110][ T369] 8021q: adding VLAN 0 to HW filter on device lag [ 18.173231][ T370] br1: port 1(lag) entered blocking state [ 18.173480][ T370] br1: port 1(lag) entered disabled state [ 18.173665][ T370] lag: entered allmulticast mode [ 18.173821][ T370] veth2: entered allmulticast mode [ 18.173972][ T370] veth4: entered allmulticast mode [ 18.175722][ T370] veth2: entered promiscuous mode [ 18.175910][ T370] veth4: entered promiscuous mode [ 18.235738][ T371] br1: port 1(lag) entered blocking state [ 18.235994][ T371] br1: port 1(lag) entered forwarding state [ 18.707779][ T380] GACT probability NOT on [ 24.801947][ T407] Mirror/redirect action on [ 39.181619][ T149] br1: port 1(lag) entered disabled state [ 44.970526][ T149] br1: port 1(lag) entered blocking state [ 44.970781][ T149] br1: port 1(lag) entered forwarding state [ 59.740752][ T37] br1: port 1(lag) entered disabled state [ 65.559836][ T39] br1: port 1(lag) entered blocking state [ 65.560052][ T39] br1: port 1(lag) entered forwarding state [ 66.170947][ T511] lag: left allmulticast mode [ 66.171139][ T511] veth2: left allmulticast mode [ 66.171292][ T511] veth4: left allmulticast mode [ 66.171471][ T511] veth2: left promiscuous mode [ 66.171642][ T511] veth4: left promiscuous mode [ 66.171806][ T511] br1: port 1(lag) entered disabled state [ 66.244999][ T360] lag: Port device veth4 removed [ 66.248246][ T360] lag: Port device veth2 removed