[ 14.499883][ T299] lag1: Mode changed to "loadbalance" [ 14.932721][ T307] lag1: Port device veth0 added [ 15.024581][ T308] lag1: Port device veth6 added [ 16.396084][ T326] lag4: Mode changed to "loadbalance" [ 16.855685][ T333] lag4: Port device veth3 added [ 16.943128][ T334] lag4: Port device veth5 added [ 18.279832][ T351] lag2: Mode changed to "loadbalance" [ 18.764986][ T358] lag2: Port device veth1 added [ 18.766098][ T39] [ 18.766217][ T39] ====================================================== [ 18.766412][ T39] WARNING: possible circular locking dependency detected [ 18.766626][ T39] 6.13.0-rc5-virtme #1 Not tainted [ 18.766799][ T39] ------------------------------------------------------ [ 18.766999][ T39] kworker/u19:0/39 is trying to acquire lock: [ 18.767216][ T39] ffff8880050ead80 (team->team_lock_key#3){+.+.}-{4:4}, at: team_port_change_check+0x43/0x130 [ 18.767536][ T39] [ 18.767536][ T39] but task is already holding lock: [ 18.767787][ T39] ffffffff9ddd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 18.768136][ T39] [ 18.768136][ T39] which lock already depends on the new lock. [ 18.768136][ T39] [ 18.768467][ T39] [ 18.768467][ T39] the existing dependency chain (in reverse order) is: [ 18.768747][ T39] [ 18.768747][ T39] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 18.769002][ T39] __lock_acquire+0xb9a/0x1680 [ 18.769163][ T39] lock_acquire.part.0+0xeb/0x330 [ 18.769323][ T39] down_read+0x9f/0x340 [ 18.769484][ T39] blocking_notifier_call_chain+0x50/0x90 [ 18.769675][ T39] __dev_open+0x15c/0x430 [ 18.769826][ T39] dev_open+0xc9/0x130 [ 18.769981][ T39] team_port_add+0x90a/0x1ce0 [ 18.770134][ T39] team_add_slave+0x41/0x80 [ 18.770301][ T39] do_setlink.constprop.0+0x80e/0x2300 [ 18.770510][ T39] rtnl_newlink+0x69c/0xa70 [ 18.770655][ T39] rtnetlink_rcv_msg+0x712/0xc10 [ 18.770823][ T39] netlink_rcv_skb+0x130/0x360 [ 18.770993][ T39] netlink_unicast+0x44b/0x710 [ 18.771166][ T39] netlink_sendmsg+0x723/0xbe0 [ 18.771323][ T39] ____sys_sendmsg+0x7ac/0xa10 [ 18.771495][ T39] ___sys_sendmsg+0xee/0x170 [ 18.771646][ T39] __sys_sendmsg+0x109/0x1a0 [ 18.771798][ T39] do_syscall_64+0xc1/0x1d0 [ 18.771949][ T39] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 18.772136][ T39] [ 18.772136][ T39] -> #0 (team->team_lock_key#3){+.+.}-{4:4}: [ 18.772392][ T39] check_prev_add+0x182/0xce0 [ 18.772545][ T39] validate_chain+0x7bd/0xae0 [ 18.772688][ T39] __lock_acquire+0xb9a/0x1680 [ 18.772847][ T39] lock_acquire.part.0+0xeb/0x330 [ 18.772989][ T39] __mutex_lock+0x169/0xbc0 [ 18.773164][ T39] team_port_change_check+0x43/0x130 [ 18.773387][ T39] team_device_event+0x3fe/0x520 [ 18.773563][ T39] notifier_call_chain+0xcd/0x150 [ 18.773739][ T39] blocking_notifier_call_chain+0x66/0x90 [ 18.773952][ T39] netdev_state_change+0xf5/0x120 [ 18.774130][ T39] linkwatch_do_dev+0xd2/0x100 [ 18.774310][ T39] __linkwatch_run_queue+0x1df/0x650 [ 18.774505][ T39] linkwatch_event+0x40/0x60 [ 18.774668][ T39] process_one_work+0xe55/0x16d0 [ 18.774837][ T39] worker_thread+0x58c/0xce0 [ 18.774994][ T39] kthread+0x28a/0x350 [ 18.775149][ T39] ret_from_fork+0x31/0x70 [ 18.775298][ T39] ret_from_fork_asm+0x1a/0x30 [ 18.775455][ T39] [ 18.775455][ T39] other info that might help us debug this: [ 18.775455][ T39] [ 18.775766][ T39] Possible unsafe locking scenario: [ 18.775766][ T39] [ 18.776002][ T39] CPU0 CPU1 [ 18.776146][ T39] ---- ---- [ 18.776328][ T39] rlock((netdev_chain).rwsem); [ 18.776490][ T39] lock(team->team_lock_key#3); [ 18.776711][ T39] lock((netdev_chain).rwsem); [ 18.776937][ T39] lock(team->team_lock_key#3); [ 18.777090][ T39] [ 18.777090][ T39] *** DEADLOCK *** [ 18.777090][ T39] [ 18.777301][ T39] 4 locks held by kworker/u19:0/39: [ 18.777446][ T39] #0: ffff8880010a9d48 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x7ec/0x16d0 [ 18.777812][ T39] #1: ffffc900002bfda0 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work+0xe0b/0x16d0 [ 18.778135][ T39] #2: ffffffff9dde7a48 (rtnl_mutex){+.+.}-{4:4}, at: linkwatch_event+0xe/0x60 [ 18.778424][ T39] #3: ffffffff9ddd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 18.778738][ T39] [ 18.778738][ T39] stack backtrace: [ 18.778928][ T39] CPU: 3 UID: 0 PID: 39 Comm: kworker/u19:0 Not tainted 6.13.0-rc5-virtme #1 [ 18.779199][ T39] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 18.779396][ T39] Workqueue: events_unbound linkwatch_event [ 18.779589][ T39] Call Trace: [ 18.779723][ T39] [ 18.779806][ T39] dump_stack_lvl+0x82/0xd0 [ 18.779998][ T39] print_circular_bug+0x2e2/0x4b0 [ 18.780165][ T39] check_noncircular+0x2f9/0x3e0 [ 18.780316][ T39] ? rcu_read_lock_any_held+0x43/0xb0 [ 18.780489][ T39] ? __bfs+0x3f9/0x6c0 [ 18.780624][ T39] ? __pfx_check_noncircular+0x10/0x10 [ 18.780786][ T39] ? check_irq_usage+0x13a/0x850 [ 18.780939][ T39] ? alloc_chain_hlocks+0x4e0/0x720 [ 18.781093][ T39] ? __pfx_check_irq_usage+0x10/0x10 [ 18.781246][ T39] check_prev_add+0x182/0xce0 [ 18.781408][ T39] validate_chain+0x7bd/0xae0 [ 18.781571][ T39] ? __pfx_validate_chain+0x10/0x10 [ 18.781723][ T39] ? hlock_class+0x4e/0x130 [ 18.781891][ T39] ? mark_lock+0x38/0x3e0 [ 18.782015][ T39] __lock_acquire+0xb9a/0x1680 [ 18.782178][ T39] ? check_prev_add+0x6e0/0xce0 [ 18.782335][ T39] lock_acquire.part.0+0xeb/0x330 [ 18.782493][ T39] ? team_port_change_check+0x43/0x130 [ 18.782655][ T39] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 18.782805][ T39] ? trace_lock_acquire+0x14c/0x1f0 [ 18.782966][ T39] ? lock_acquire+0x32/0xc0 [ 18.783152][ T39] ? team_port_change_check+0x43/0x130 [ 18.783322][ T39] __mutex_lock+0x169/0xbc0 [ 18.783483][ T39] ? team_port_change_check+0x43/0x130 [ 18.783644][ T39] ? team_port_change_check+0x43/0x130 [ 18.783830][ T39] ? __pfx___mutex_lock+0x10/0x10 [ 18.784004][ T39] ? lock_acquire.part.0+0xeb/0x330 [ 18.784179][ T39] ? blocking_notifier_call_chain+0x50/0x90 [ 18.784374][ T39] ? find_held_lock+0x2c/0x110 [ 18.784529][ T39] ? __pfx_fib_sync_up+0x10/0x10 [ 18.784707][ T39] ? __pfx_rwsem_read_trylock+0x10/0x10 [ 18.784864][ T39] ? team_port_change_check+0x43/0x130 [ 18.785020][ T39] team_port_change_check+0x43/0x130 [ 18.785174][ T39] team_device_event+0x3fe/0x520 [ 18.785333][ T39] ? trace_notifier_run+0xe7/0x150 [ 18.785526][ T39] notifier_call_chain+0xcd/0x150 [ 18.785711][ T39] blocking_notifier_call_chain+0x66/0x90 [ 18.785875][ T39] netdev_state_change+0xf5/0x120 [ 18.786023][ T39] ? __pfx_netdev_state_change+0x10/0x10 [ 18.786182][ T39] linkwatch_do_dev+0xd2/0x100 [ 18.786366][ T39] __linkwatch_run_queue+0x1df/0x650 [ 18.786576][ T39] ? __pfx___linkwatch_run_queue+0x10/0x10 [ 18.786768][ T39] ? lock_acquire+0x32/0xc0 [ 18.786922][ T39] linkwatch_event+0x40/0x60 [ 18.787068][ T39] process_one_work+0xe55/0x16d0 [ 18.787226][ T39] ? __pfx___lock_release+0x10/0x10 [ 18.787405][ T39] ? __pfx_process_one_work+0x10/0x10 [ 18.787558][ T39] ? assign_work+0x16c/0x240 [ 18.787861][ T39] worker_thread+0x58c/0xce0 [ 18.788020][ T39] ? __pfx_worker_thread+0x10/0x10 [ 18.788193][ T39] kthread+0x28a/0x350 [ 18.788325][ T39] ? __pfx_kthread+0x10/0x10 [ 18.788566][ T39] ret_from_fork+0x31/0x70 [ 18.788775][ T39] ? __pfx_kthread+0x10/0x10 [ 18.788949][ T39] ret_from_fork_asm+0x1a/0x30 [ 18.789106][ T39] [ 18.852451][ T359] lag2: Port device veth7 added [ 19.259028][ T366] br1: port 1(lag2) entered blocking state [ 19.259477][ T366] br1: port 1(lag2) entered disabled state [ 19.259716][ T366] lag2: entered allmulticast mode [ 19.259882][ T366] veth1: entered allmulticast mode [ 19.260035][ T366] veth7: entered allmulticast mode [ 19.262120][ T366] veth1: entered promiscuous mode [ 19.262331][ T366] veth7: entered promiscuous mode [ 19.483406][ T370] br1: port 1(lag2) entered blocking state [ 19.483674][ T370] br1: port 1(lag2) entered forwarding state [ 19.717516][ T376] lag3: Mode changed to "loadbalance" [ 20.019937][ T383] lag3: Port device veth2 added [ 20.091816][ T384] lag3: Port device veth4 added [ 28.895976][ T447] veth7: left allmulticast mode [ 28.897650][ T447] lag2: Port device veth7 removed [ 28.972228][ T448] veth7: left promiscuous mode [ 41.858257][ T498] veth7: entered allmulticast mode [ 41.858513][ T498] veth7: entered promiscuous mode [ 41.859315][ T498] lag2: Port device veth7 added [ 42.115808][ T509] veth1: left allmulticast mode [ 42.117242][ T509] lag2: Port device veth1 removed [ 42.178273][ T510] veth1: left promiscuous mode [ 54.915894][ T58] br1: port 1(lag2) entered disabled state [ 54.971161][ T560] veth7: left allmulticast mode [ 54.972803][ T560] lag2: Port device veth7 removed [ 55.049115][ T561] veth7: left promiscuous mode [ 55.235880][ T571] veth1: entered allmulticast mode [ 55.236142][ T571] veth1: entered promiscuous mode [ 55.237442][ T571] lag2: Port device veth1 added [ 55.250341][ T40] br1: port 1(lag2) entered blocking state [ 55.250705][ T40] br1: port 1(lag2) entered forwarding state [ 55.481338][ T582] veth7: entered allmulticast mode [ 55.481564][ T582] veth7: entered promiscuous mode [ 55.482333][ T582] lag2: Port device veth7 added [ 68.378837][ T632] lag2: left allmulticast mode [ 68.379079][ T632] veth1: left allmulticast mode [ 68.379268][ T632] veth7: left allmulticast mode [ 68.379451][ T632] veth1: left promiscuous mode [ 68.379635][ T632] veth7: left promiscuous mode [ 68.379805][ T632] br1: port 1(lag2) entered disabled state [ 70.500269][ T634] br1: port 1(lag2) entered blocking state [ 70.500558][ T634] br1: port 1(lag2) entered disabled state [ 70.500802][ T634] lag2: entered allmulticast mode [ 70.500962][ T634] veth1: entered allmulticast mode [ 70.501124][ T634] veth7: entered allmulticast mode [ 70.502727][ T634] veth1: entered promiscuous mode [ 70.502940][ T634] veth7: entered promiscuous mode [ 70.503838][ T634] br1: port 1(lag2) entered blocking state [ 70.504070][ T634] br1: port 1(lag2) entered forwarding state [ 83.356512][ T684] lag3: Port device veth2 removed [ 96.294567][ T735] lag3: Port device veth2 added [ 96.559435][ T746] lag3: Port device veth4 removed [ 109.427116][ T797] lag3: Port device veth2 removed [ 109.692255][ T808] lag3: Port device veth4 added [ 109.946750][ T819] lag3: Port device veth2 added [ 122.810563][ T869] br1: port 2(lag3) entered blocking state [ 122.810940][ T869] br1: port 2(lag3) entered disabled state [ 122.811305][ T869] lag3: entered allmulticast mode [ 122.811526][ T869] veth4: entered allmulticast mode [ 122.811751][ T869] veth2: entered allmulticast mode [ 122.814723][ T869] lag3: entered promiscuous mode [ 122.815015][ T869] veth4: entered promiscuous mode [ 122.815570][ T869] veth2: entered promiscuous mode [ 122.816334][ T869] lag2: entered promiscuous mode [ 122.818080][ T869] br1: port 2(lag3) entered blocking state [ 122.818351][ T869] br1: port 2(lag3) entered forwarding state [ 124.936623][ T871] lag3: left allmulticast mode [ 124.936877][ T871] veth4: left allmulticast mode [ 124.937108][ T871] veth2: left allmulticast mode [ 124.937341][ T871] lag3: left promiscuous mode [ 124.937575][ T871] veth4: left promiscuous mode [ 124.938224][ T871] veth2: left promiscuous mode [ 124.938916][ T871] br1: port 2(lag3) entered disabled state [ 124.939979][ T871] lag2: left promiscuous mode [ 138.089132][ T924] lag3: Port device veth4 removed [ 138.154605][ T925] lag3: Port device veth2 removed [ 138.787622][ T45] br1: port 1(lag2) entered disabled state [ 138.913944][ T940] lag2: left allmulticast mode [ 138.914145][ T940] veth1: left allmulticast mode [ 138.914353][ T940] veth7: left allmulticast mode [ 138.914529][ T940] br1: port 1(lag2) entered disabled state [ 139.100710][ T942] lag2: Port device veth7 removed [ 139.178555][ T943] lag2: Port device veth1 removed [ 140.363295][ T956] lag4: Port device veth5 removed [ 140.429176][ T957] lag4: Port device veth3 removed [ 141.731834][ T972] lag1: Port device veth6 removed [ 141.797850][ T973] lag1: Port device veth0 removed