[ 11.124141][ T249] ip (249) used greatest stack depth: 23632 bytes left [ 13.640437][ T294] 8021q: 802.1Q VLAN Support v1.8 [ 17.210828][ T341] gre: GRE over IPv4 demultiplexor driver [ 17.226955][ T341] ip_gre: GRE over IPv4 tunneling driver [ 18.276660][ T358] lag: Mode changed to "loadbalance" [ 18.515729][ T361] lag: Port device veth2 added [ 18.516497][ T45] [ 18.516608][ T45] ====================================================== [ 18.516849][ T45] WARNING: possible circular locking dependency detected [ 18.517052][ T45] 6.13.0-rc5-virtme #1 Not tainted [ 18.517228][ T45] ------------------------------------------------------ [ 18.517457][ T45] kworker/u20:1/45 is trying to acquire lock: [ 18.517690][ T45] ffff88800d652d80 (team->team_lock_key){+.+.}-{4:4}, at: team_port_change_check+0x43/0x130 [ 18.518040][ T45] [ 18.518040][ T45] but task is already holding lock: [ 18.518285][ T45] ffffffff8e5d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 18.518627][ T45] [ 18.518627][ T45] which lock already depends on the new lock. [ 18.518627][ T45] [ 18.518986][ T45] [ 18.518986][ T45] the existing dependency chain (in reverse order) is: [ 18.519293][ T45] [ 18.519293][ T45] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 18.519555][ T45] __lock_acquire+0xb9a/0x1680 [ 18.519739][ T45] lock_acquire.part.0+0xeb/0x330 [ 18.519907][ T45] down_read+0x9f/0x340 [ 18.520080][ T45] blocking_notifier_call_chain+0x50/0x90 [ 18.520300][ T45] __dev_open+0x15c/0x430 [ 18.520483][ T45] dev_open+0xc9/0x130 [ 18.520648][ T45] team_port_add+0x90a/0x1ce0 [ 18.520837][ T45] team_add_slave+0x41/0x80 [ 18.521013][ T45] do_setlink.constprop.0+0x80e/0x2300 [ 18.521247][ T45] rtnl_newlink+0x69c/0xa70 [ 18.521427][ T45] rtnetlink_rcv_msg+0x712/0xc10 [ 18.521595][ T45] netlink_rcv_skb+0x130/0x360 [ 18.521775][ T45] netlink_unicast+0x44b/0x710 [ 18.521943][ T45] netlink_sendmsg+0x723/0xbe0 [ 18.522118][ T45] ____sys_sendmsg+0x7ac/0xa10 [ 18.522290][ T45] ___sys_sendmsg+0xee/0x170 [ 18.522452][ T45] __sys_sendmsg+0x109/0x1a0 [ 18.522611][ T45] do_syscall_64+0xc1/0x1d0 [ 18.522772][ T45] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 18.522968][ T45] [ 18.522968][ T45] -> #0 (team->team_lock_key){+.+.}-{4:4}: [ 18.523219][ T45] check_prev_add+0x182/0xce0 [ 18.523387][ T45] validate_chain+0x7bd/0xae0 [ 18.523550][ T45] __lock_acquire+0xb9a/0x1680 [ 18.523700][ T45] lock_acquire.part.0+0xeb/0x330 [ 18.523854][ T45] __mutex_lock+0x169/0xbc0 [ 18.524016][ T45] team_port_change_check+0x43/0x130 [ 18.524209][ T45] team_device_event+0x3fe/0x520 [ 18.524372][ T45] notifier_call_chain+0xcd/0x150 [ 18.524525][ T45] blocking_notifier_call_chain+0x66/0x90 [ 18.524724][ T45] netdev_state_change+0xf5/0x120 [ 18.524887][ T45] linkwatch_do_dev+0xd2/0x100 [ 18.525072][ T45] __linkwatch_run_queue+0x1df/0x650 [ 18.525287][ T45] linkwatch_event+0x40/0x60 [ 18.525445][ T45] process_one_work+0xe55/0x16d0 [ 18.525626][ T45] worker_thread+0x58c/0xce0 [ 18.525786][ T45] kthread+0x28a/0x350 [ 18.525947][ T45] ret_from_fork+0x31/0x70 [ 18.526118][ T45] ret_from_fork_asm+0x1a/0x30 [ 18.526283][ T45] [ 18.526283][ T45] other info that might help us debug this: [ 18.526283][ T45] [ 18.526606][ T45] Possible unsafe locking scenario: [ 18.526606][ T45] [ 18.526861][ T45] CPU0 CPU1 [ 18.527036][ T45] ---- ---- [ 18.527234][ T45] rlock((netdev_chain).rwsem); [ 18.527406][ T45] lock(team->team_lock_key); [ 18.527677][ T45] lock((netdev_chain).rwsem); [ 18.527945][ T45] lock(team->team_lock_key); [ 18.528123][ T45] [ 18.528123][ T45] *** DEADLOCK *** [ 18.528123][ T45] [ 18.528379][ T45] 4 locks held by kworker/u20:1/45: [ 18.528552][ T45] #0: ffff8880010a9d48 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x7ec/0x16d0 [ 18.528920][ T45] #1: ffffc9000031fda0 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work+0xe0b/0x16d0 [ 18.529294][ T45] #2: ffffffff8e5e7a48 (rtnl_mutex){+.+.}-{4:4}, at: linkwatch_event+0xe/0x60 [ 18.529628][ T45] #3: ffffffff8e5d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 18.530008][ T45] [ 18.530008][ T45] stack backtrace: [ 18.530227][ T45] CPU: 3 UID: 0 PID: 45 Comm: kworker/u20:1 Not tainted 6.13.0-rc5-virtme #1 [ 18.530543][ T45] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 18.530788][ T45] Workqueue: events_unbound linkwatch_event [ 18.531023][ T45] Call Trace: [ 18.531169][ T45] [ 18.531286][ T45] dump_stack_lvl+0x82/0xd0 [ 18.531473][ T45] print_circular_bug+0x2e2/0x4b0 [ 18.531665][ T45] check_noncircular+0x2f9/0x3e0 [ 18.531843][ T45] ? unwind_next_frame+0x1478/0x1d00 [ 18.532042][ T45] ? __pfx_check_noncircular+0x10/0x10 [ 18.532310][ T45] ? unwind_next_frame+0x1d2/0x1d00 [ 18.532493][ T45] ? alloc_chain_hlocks+0x4e0/0x720 [ 18.532755][ T45] check_prev_add+0x182/0xce0 [ 18.532946][ T45] validate_chain+0x7bd/0xae0 [ 18.533133][ T45] ? __pfx_validate_chain+0x10/0x10 [ 18.533317][ T45] ? hlock_class+0x4e/0x130 [ 18.533491][ T45] ? mark_lock+0x38/0x3e0 [ 18.533629][ T45] __lock_acquire+0xb9a/0x1680 [ 18.533827][ T45] lock_acquire.part.0+0xeb/0x330 [ 18.534017][ T45] ? team_port_change_check+0x43/0x130 [ 18.534199][ T45] ? validate_chain+0x1fe/0xae0 [ 18.534379][ T45] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 18.534564][ T45] ? trace_lock_acquire+0x14c/0x1f0 [ 18.534758][ T45] ? lock_acquire+0x32/0xc0 [ 18.534979][ T45] ? team_port_change_check+0x43/0x130 [ 18.535163][ T45] __mutex_lock+0x169/0xbc0 [ 18.535337][ T45] ? team_port_change_check+0x43/0x130 [ 18.535515][ T45] ? team_port_change_check+0x43/0x130 [ 18.535691][ T45] ? __pfx___mutex_lock+0x10/0x10 [ 18.535866][ T45] ? lock_acquire.part.0+0xeb/0x330 [ 18.536050][ T45] ? blocking_notifier_call_chain+0x50/0x90 [ 18.536279][ T45] ? find_held_lock+0x2c/0x110 [ 18.536459][ T45] ? __pfx_fib_sync_up+0x10/0x10 [ 18.536647][ T45] ? __pfx_rwsem_read_trylock+0x10/0x10 [ 18.536833][ T45] ? team_port_change_check+0x43/0x130 [ 18.537012][ T45] team_port_change_check+0x43/0x130 [ 18.537198][ T45] team_device_event+0x3fe/0x520 [ 18.537382][ T45] ? trace_notifier_run+0xe7/0x150 [ 18.537562][ T45] notifier_call_chain+0xcd/0x150 [ 18.537761][ T45] blocking_notifier_call_chain+0x66/0x90 [ 18.537939][ T45] netdev_state_change+0xf5/0x120 [ 18.538130][ T45] ? __pfx_netdev_state_change+0x10/0x10 [ 18.538311][ T45] linkwatch_do_dev+0xd2/0x100 [ 18.538494][ T45] __linkwatch_run_queue+0x1df/0x650 [ 18.538681][ T45] ? __pfx___linkwatch_run_queue+0x10/0x10 [ 18.538998][ T45] ? lock_acquire+0x32/0xc0 [ 18.539199][ T45] linkwatch_event+0x40/0x60 [ 18.539376][ T45] process_one_work+0xe55/0x16d0 [ 18.539554][ T45] ? __pfx___lock_release+0x10/0x10 [ 18.539746][ T45] ? __pfx_process_one_work+0x10/0x10 [ 18.539928][ T45] ? assign_work+0x16c/0x240 [ 18.540114][ T45] worker_thread+0x58c/0xce0 [ 18.540296][ T45] ? __pfx_worker_thread+0x10/0x10 [ 18.540769][ T45] kthread+0x28a/0x350 [ 18.540958][ T45] ? __pfx_kthread+0x10/0x10 [ 18.541479][ T45] ret_from_fork+0x31/0x70 [ 18.541733][ T45] ? __pfx_kthread+0x10/0x10 [ 18.542146][ T45] ret_from_fork_asm+0x1a/0x30 [ 18.542419][ T45] [ 18.760891][ T364] lag: Port device veth4 added [ 18.905154][ T366] 8021q: adding VLAN 0 to HW filter on device lag [ 18.965301][ T367] br1: port 1(lag) entered blocking state [ 18.965525][ T367] br1: port 1(lag) entered disabled state [ 18.965738][ T367] lag: entered allmulticast mode [ 18.965912][ T367] veth2: entered allmulticast mode [ 18.966068][ T367] veth4: entered allmulticast mode [ 18.968392][ T367] veth2: entered promiscuous mode [ 18.968604][ T367] veth4: entered promiscuous mode [ 19.033389][ T368] br1: port 1(lag) entered blocking state [ 19.033598][ T368] br1: port 1(lag) entered forwarding state [ 19.565709][ T377] GACT probability NOT on [ 25.784511][ T406] Mirror/redirect action on [ 40.175315][ T37] br1: port 1(lag) entered disabled state [ 46.008249][ T40] br1: port 1(lag) entered blocking state [ 46.008638][ T40] br1: port 1(lag) entered forwarding state [ 60.885657][ T37] br1: port 1(lag) entered disabled state [ 66.717433][ T37] br1: port 1(lag) entered blocking state [ 66.717630][ T37] br1: port 1(lag) entered forwarding state [ 67.277034][ T510] lag: left allmulticast mode [ 67.277253][ T510] veth2: left allmulticast mode [ 67.277413][ T510] veth4: left allmulticast mode [ 67.277570][ T510] veth2: left promiscuous mode [ 67.277728][ T510] veth4: left promiscuous mode [ 67.277879][ T510] br1: port 1(lag) entered disabled state [ 67.371583][ T358] lag: Port device veth4 removed [ 67.375885][ T358] lag: Port device veth2 removed