[ 1285.525346][T17350] 8021q: 802.1Q VLAN Support v1.8 [ 1287.346627][T17375] lag2: Mode changed to "loadbalance" [ 1287.522756][T17378] lag2: Port device veth3 added [ 1287.821716][T17389] lag2: Port device veth5 added [ 1287.995420][T17391] 8021q: adding VLAN 0 to HW filter on device lag2 [ 1287.995931][T17391] [ 1287.996014][T17391] ====================================================== [ 1287.996209][T17391] WARNING: possible circular locking dependency detected [ 1287.996408][T17391] 6.13.0-rc5-virtme #1 Not tainted [ 1287.996563][T17391] ------------------------------------------------------ [ 1287.996753][T17391] ip/17391 is trying to acquire lock: [ 1287.996902][T17391] ffff888005b0ad80 (team->team_lock_key){+.+.}-{4:4}, at: team_vlan_rx_add_vid+0x3b/0x1b0 [ 1287.997174][T17391] [ 1287.997174][T17391] but task is already holding lock: [ 1287.997409][T17391] ffffffff9ebd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 1287.997719][T17391] [ 1287.997719][T17391] which lock already depends on the new lock. [ 1287.997719][T17391] [ 1287.998017][T17391] [ 1287.998017][T17391] the existing dependency chain (in reverse order) is: [ 1287.998270][T17391] [ 1287.998270][T17391] -> #1 ((netdev_chain).rwsem){++++}-{4:4}: [ 1287.998495][T17391] __lock_acquire+0xb9a/0x1680 [ 1287.998654][T17391] lock_acquire.part.0+0xeb/0x330 [ 1287.998809][T17391] down_read+0x9f/0x340 [ 1287.998959][T17391] blocking_notifier_call_chain+0x50/0x90 [ 1287.999150][T17391] __dev_open+0x15c/0x430 [ 1287.999303][T17391] dev_open+0xc9/0x130 [ 1287.999453][T17391] team_port_add+0x90a/0x1ce0 [ 1287.999606][T17391] team_add_slave+0x41/0x80 [ 1287.999760][T17391] do_setlink.constprop.0+0x80e/0x2300 [ 1287.999941][T17391] rtnl_newlink+0x69c/0xa70 [ 1288.000087][T17391] rtnetlink_rcv_msg+0x712/0xc10 [ 1288.000260][T17391] netlink_rcv_skb+0x130/0x360 [ 1288.000415][T17391] netlink_unicast+0x44b/0x710 [ 1288.000567][T17391] netlink_sendmsg+0x723/0xbe0 [ 1288.000721][T17391] ____sys_sendmsg+0x7ac/0xa10 [ 1288.000873][T17391] ___sys_sendmsg+0xee/0x170 [ 1288.001025][T17391] __sys_sendmsg+0x109/0x1a0 [ 1288.001173][T17391] do_syscall_64+0xc1/0x1d0 [ 1288.001329][T17391] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1288.001519][T17391] [ 1288.001519][T17391] -> #0 (team->team_lock_key){+.+.}-{4:4}: [ 1288.001754][T17391] check_prev_add+0x182/0xce0 [ 1288.001906][T17391] validate_chain+0x7bd/0xae0 [ 1288.002060][T17391] __lock_acquire+0xb9a/0x1680 [ 1288.002220][T17391] lock_acquire.part.0+0xeb/0x330 [ 1288.002372][T17391] __mutex_lock+0x169/0xbc0 [ 1288.002527][T17391] team_vlan_rx_add_vid+0x3b/0x1b0 [ 1288.002685][T17391] vlan_vid_add+0x2bc/0x860 [ 1288.002868][T17391] vlan_device_event+0x2f4/0x2130 [8021q] [ 1288.003070][T17391] notifier_call_chain+0xcd/0x150 [ 1288.003238][T17391] blocking_notifier_call_chain+0x66/0x90 [ 1288.003440][T17391] __dev_notify_flags+0xe6/0x250 [ 1288.003604][T17391] dev_change_flags+0xec/0x160 [ 1288.003766][T17391] do_setlink.constprop.0+0x79d/0x2300 [ 1288.003967][T17391] rtnl_newlink+0x69c/0xa70 [ 1288.004130][T17391] rtnetlink_rcv_msg+0x712/0xc10 [ 1288.004293][T17391] netlink_rcv_skb+0x130/0x360 [ 1288.004462][T17391] netlink_unicast+0x44b/0x710 [ 1288.004622][T17391] netlink_sendmsg+0x723/0xbe0 [ 1288.004789][T17391] ____sys_sendmsg+0x7ac/0xa10 [ 1288.004960][T17391] ___sys_sendmsg+0xee/0x170 [ 1288.005124][T17391] __sys_sendmsg+0x109/0x1a0 [ 1288.005291][T17391] do_syscall_64+0xc1/0x1d0 [ 1288.005460][T17391] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1288.005670][T17391] [ 1288.005670][T17391] other info that might help us debug this: [ 1288.005670][T17391] [ 1288.005998][T17391] Possible unsafe locking scenario: [ 1288.005998][T17391] [ 1288.006216][T17391] CPU0 CPU1 [ 1288.006376][T17391] ---- ---- [ 1288.006529][T17391] rlock((netdev_chain).rwsem); [ 1288.006678][T17391] lock(team->team_lock_key); [ 1288.006901][T17391] lock((netdev_chain).rwsem); [ 1288.007118][T17391] lock(team->team_lock_key); [ 1288.007272][T17391] [ 1288.007272][T17391] *** DEADLOCK *** [ 1288.007272][T17391] [ 1288.007488][T17391] 2 locks held by ip/17391: [ 1288.007642][T17391] #0: ffffffff9ebe7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x653/0xa70 [ 1288.007912][T17391] #1: ffffffff9ebd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 1288.008225][T17391] [ 1288.008225][T17391] stack backtrace: [ 1288.008414][T17391] CPU: 2 UID: 0 PID: 17391 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 1288.008643][T17391] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 1288.008838][T17391] Call Trace: [ 1288.008957][T17391] [ 1288.009035][T17391] dump_stack_lvl+0x82/0xd0 [ 1288.009193][T17391] print_circular_bug+0x2e2/0x4b0 [ 1288.009349][T17391] check_noncircular+0x2f9/0x3e0 [ 1288.009503][T17391] ? __pfx_check_noncircular+0x10/0x10 [ 1288.009658][T17391] ? rcu_read_lock_any_held+0x43/0xb0 [ 1288.009816][T17391] ? validate_chain+0x1fe/0xae0 [ 1288.009971][T17391] ? alloc_chain_hlocks+0x4e0/0x720 [ 1288.010127][T17391] check_prev_add+0x182/0xce0 [ 1288.010286][T17391] validate_chain+0x7bd/0xae0 [ 1288.010440][T17391] ? find_held_lock+0x2c/0x110 [ 1288.010592][T17391] ? __pfx_validate_chain+0x10/0x10 [ 1288.010742][T17391] ? hlock_class+0x4e/0x130 [ 1288.010890][T17391] ? mark_lock+0x38/0x3e0 [ 1288.011009][T17391] __lock_acquire+0xb9a/0x1680 [ 1288.011166][T17391] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 1288.011351][T17391] lock_acquire.part.0+0xeb/0x330 [ 1288.011500][T17391] ? team_vlan_rx_add_vid+0x3b/0x1b0 [ 1288.011661][T17391] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 1288.011810][T17391] ? trace_lock_acquire+0x14c/0x1f0 [ 1288.011959][T17391] ? lock_acquire+0x32/0xc0 [ 1288.012111][T17391] ? team_vlan_rx_add_vid+0x3b/0x1b0 [ 1288.012265][T17391] __mutex_lock+0x169/0xbc0 [ 1288.012416][T17391] ? team_vlan_rx_add_vid+0x3b/0x1b0 [ 1288.012563][T17391] ? __create_object+0x5e/0xb0 [ 1288.012713][T17391] ? __pfx___lock_release+0x10/0x10 [ 1288.012862][T17391] ? team_vlan_rx_add_vid+0x3b/0x1b0 [ 1288.013012][T17391] ? __pfx___mutex_lock+0x10/0x10 [ 1288.013166][T17391] ? mark_held_locks+0x9e/0xe0 [ 1288.013314][T17391] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 1288.013500][T17391] ? __create_object+0x5e/0xb0 [ 1288.013649][T17391] ? team_vlan_rx_add_vid+0x3b/0x1b0 [ 1288.013798][T17391] team_vlan_rx_add_vid+0x3b/0x1b0 [ 1288.013948][T17391] vlan_vid_add+0x2bc/0x860 [ 1288.014100][T17391] vlan_device_event+0x2f4/0x2130 [8021q] [ 1288.014261][T17391] ? __pfx_vlan_device_event+0x10/0x10 [8021q] [ 1288.014448][T17391] ? packet_notifier+0x3b0/0x810 [ 1288.014597][T17391] notifier_call_chain+0xcd/0x150 [ 1288.014747][T17391] blocking_notifier_call_chain+0x66/0x90 [ 1288.014895][T17391] __dev_notify_flags+0xe6/0x250 [ 1288.015043][T17391] ? __pfx___dev_notify_flags+0x10/0x10 [ 1288.015206][T17391] ? __pfx___dev_change_flags+0x10/0x10 [ 1288.015362][T17391] ? unwind_get_return_address+0x5e/0xa0 [ 1288.015511][T17391] ? __pfx_stack_trace_consume_entry+0x10/0x10 [ 1288.015699][T17391] ? __pfx_validate_chain+0x10/0x10 [ 1288.015849][T17391] dev_change_flags+0xec/0x160 [ 1288.016007][T17391] do_setlink.constprop.0+0x79d/0x2300 [ 1288.016167][T17391] ? __pfx_do_setlink.constprop.0+0x10/0x10 [ 1288.016362][T17391] ? lock_acquire.part.0+0xeb/0x330 [ 1288.016518][T17391] ? rtnl_newlink+0x653/0xa70 [ 1288.016666][T17391] ? rtnl_newlink+0xb9/0xa70 [ 1288.016812][T17391] ? rtnetlink_rcv_msg+0x712/0xc10 [ 1288.016960][T17391] ? __mutex_trylock_common+0xfa/0x260 [ 1288.017110][T17391] ? __pfx___mutex_trylock_common+0x10/0x10 [ 1288.017301][T17391] ? lock_acquire+0x32/0xc0 [ 1288.017449][T17391] ? trace_contention_end+0xef/0x150 [ 1288.017605][T17391] ? __mutex_lock+0x190/0xbc0 [ 1288.017755][T17391] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 1288.017905][T17391] ? rtnl_newlink+0x653/0xa70 [ 1288.018059][T17391] ? __pfx___mutex_lock+0x10/0x10 [ 1288.018210][T17391] ? __create_object+0x5e/0xb0 [ 1288.018362][T17391] ? __rtnl_newlink+0x3f2/0xa80 [ 1288.018511][T17391] rtnl_newlink+0x69c/0xa70 [ 1288.018666][T17391] ? __pfx_rtnl_newlink+0x10/0x10 [ 1288.018813][T17391] ? find_held_lock+0x2c/0x110 [ 1288.018965][T17391] ? __pfx___lock_release+0x10/0x10 [ 1288.019114][T17391] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 1288.019269][T17391] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 1288.019419][T17391] ? __pfx_rtnl_newlink+0x10/0x10 [ 1288.019568][T17391] rtnetlink_rcv_msg+0x712/0xc10 [ 1288.019717][T17391] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 1288.019870][T17391] ? hlock_class+0x4e/0x130 [ 1288.020019][T17391] ? mark_lock+0x38/0x3e0 [ 1288.020135][T17391] ? __lock_acquire+0xb9a/0x1680 [ 1288.020284][T17391] netlink_rcv_skb+0x130/0x360 [ 1288.020431][T17391] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 1288.020579][T17391] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 1288.020734][T17391] ? netlink_deliver_tap+0x13e/0x340 [ 1288.020916][T17391] ? netlink_deliver_tap+0xc3/0x340 [ 1288.021076][T17391] netlink_unicast+0x44b/0x710 [ 1288.021229][T17391] ? __pfx_netlink_unicast+0x10/0x10 [ 1288.021390][T17391] ? find_held_lock+0x2c/0x110 [ 1288.021548][T17391] netlink_sendmsg+0x723/0xbe0 [ 1288.021703][T17391] ? __pfx_netlink_sendmsg+0x10/0x10 [ 1288.021868][T17391] ____sys_sendmsg+0x7ac/0xa10 [ 1288.022026][T17391] ? __pfx_____sys_sendmsg+0x10/0x10 [ 1288.022184][T17391] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 1288.022374][T17391] ___sys_sendmsg+0xee/0x170 [ 1288.022523][T17391] ? __pfx____sys_sendmsg+0x10/0x10 [ 1288.022674][T17391] ? kasan_save_stack+0x34/0x50 [ 1288.022827][T17391] ? kasan_save_stack+0x24/0x50 [ 1288.022975][T17391] ? __kasan_record_aux_stack+0x8e/0xa0 [ 1288.023136][T17391] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 1288.023324][T17391] ? __x64_sys_close+0x7c/0xd0 [ 1288.023477][T17391] ? do_syscall_64+0xc1/0x1d0 [ 1288.023628][T17391] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1288.023825][T17391] ? __lock_acquire+0xb9a/0x1680 [ 1288.023986][T17391] ? find_held_lock+0x2c/0x110 [ 1288.024140][T17391] ? __lock_release+0x103/0x460 [ 1288.024299][T17391] ? __virt_addr_valid+0x22b/0x430 [ 1288.024466][T17391] ? __pfx___lock_release+0x10/0x10 [ 1288.024624][T17391] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 1288.024778][T17391] __sys_sendmsg+0x109/0x1a0 [ 1288.024933][T17391] ? __pfx___sys_sendmsg+0x10/0x10 [ 1288.025089][T17391] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 1288.025287][T17391] do_syscall_64+0xc1/0x1d0 [ 1288.025452][T17391] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 1288.025646][T17391] RIP: 0033:0x7fac3f6419a7 [ 1288.025808][T17391] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 1288.026373][T17391] RSP: 002b:00007ffee560d688 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 1288.026614][T17391] RAX: ffffffffffffffda RBX: 00007ffee560ddb0 RCX: 00007fac3f6419a7 [ 1288.026849][T17391] RDX: 0000000000000000 RSI: 00007ffee560d6f0 RDI: 0000000000000005 [ 1288.027080][T17391] RBP: 0000000000000003 R08: 0000000000000003 R09: 0000000000000078 [ 1288.027333][T17391] R10: 00007fac3f4ffef8 R11: 0000000000000246 R12: 0000000000000003 [ 1288.027564][T17391] R13: 0000000067790036 R14: 0000000000498600 R15: 0000000000000000 [ 1288.027791][T17391] [ 1288.033570][T17391] ip (17391) used greatest stack depth: 23256 bytes left [ 1288.097122][T17392] 8021q: adding VLAN 0 to HW filter on device lag2 [ 1288.417041][T17397] gre: GRE over IPv4 demultiplexor driver [ 1288.430875][T17397] ip_gre: GRE over IPv4 tunneling driver [ 1289.779875][T17425] lag1: Mode changed to "loadbalance" [ 1289.978705][T17428] lag1: Port device veth2 added [ 1290.182719][T17431] lag1: Port device veth4 added [ 1290.334219][T17433] 8021q: adding VLAN 0 to HW filter on device lag1 [ 1296.365570][T17461] Mirror/redirect action on [ 1296.510647][T17464] GACT probability NOT on [ 1296.575786][T17465] lag2: Port device veth5 removed [ 1300.605866][T17475] lag2: Port device veth3 removed [ 1305.429539][T17504] lag2: Mode changed to "loadbalance" [ 1305.569791][T17507] lag2: Port device veth3 added [ 1305.803429][T17513] lag2: Port device veth5 added [ 1305.934616][T17515] 8021q: adding VLAN 0 to HW filter on device lag2 [ 1306.006070][T17516] 8021q: adding VLAN 0 to HW filter on device lag2 [ 1312.201276][T17550] lag2: Port device veth3 removed [ 1316.192575][T17560] lag2: Port device veth5 removed [ 1321.098512][T17590] lag2: Mode changed to "loadbalance" [ 1321.224690][T17593] lag2: Port device veth3 added [ 1321.428629][T17598] lag2: Port device veth5 added [ 1321.567805][T17600] 8021q: adding VLAN 0 to HW filter on device lag2 [ 1321.635933][T17601] 8021q: adding VLAN 0 to HW filter on device lag2 [ 1327.739053][T17425] lag1: Port device veth4 removed [ 1327.744548][T17425] lag1: Port device veth2 removed [ 1329.142591][T17590] lag2: Port device veth5 removed [ 1329.152666][T17590] lag2: Port device veth3 removed