[ 11.383764][ T249] ip (249) used greatest stack depth: 24264 bytes left [ 14.708032][ T300] 8021q: 802.1Q VLAN Support v1.8 [ 19.074225][ T350] br1: port 1(vx10) entered blocking state [ 19.075066][ T350] br1: port 1(vx10) entered disabled state [ 19.075551][ T350] vx10: entered allmulticast mode [ 19.085052][ T350] vx10: entered promiscuous mode [ 19.087116][ T350] br1: port 1(vx10) entered blocking state [ 19.087566][ T350] br1: port 1(vx10) entered forwarding state [ 19.096756][ T350] ip (350) used greatest stack depth: 24208 bytes left [ 19.511288][ T355] br1: port 2(vx20) entered blocking state [ 19.511814][ T355] br1: port 2(vx20) entered disabled state [ 19.512293][ T355] vx20: entered allmulticast mode [ 19.516500][ T355] vx20: entered promiscuous mode [ 19.517575][ T355] br1: port 2(vx20) entered blocking state [ 19.517889][ T355] br1: port 2(vx20) entered forwarding state [ 19.746514][ T357] br1: port 3(veth1) entered blocking state [ 19.746984][ T357] br1: port 3(veth1) entered disabled state [ 19.747450][ T357] veth1: entered allmulticast mode [ 19.750629][ T357] veth1: entered promiscuous mode [ 19.842952][ T56] br1: port 3(veth1) entered blocking state [ 19.843450][ T56] br1: port 3(veth1) entered forwarding state [ 20.298872][ T363] br1: port 4(veth2) entered blocking state [ 20.299312][ T363] br1: port 4(veth2) entered disabled state [ 20.299761][ T363] veth2: entered allmulticast mode [ 20.303422][ T363] veth2: entered promiscuous mode [ 20.417534][ T40] br1: port 4(veth2) entered blocking state [ 20.417852][ T40] br1: port 4(veth2) entered forwarding state [ 24.431440][ T413] br2: port 1(w1) entered blocking state [ 24.431780][ T413] br2: port 1(w1) entered disabled state [ 24.432123][ T413] w1: entered allmulticast mode [ 24.435492][ T413] w1: entered promiscuous mode [ 25.168855][ T421] br2: port 2(vx10) entered blocking state [ 25.169221][ T421] br2: port 2(vx10) entered disabled state [ 25.169560][ T421] vx10: entered allmulticast mode [ 25.171780][ T421] vx10: entered promiscuous mode [ 25.172438][ T421] br2: port 2(vx10) entered blocking state [ 25.172804][ T421] br2: port 2(vx10) entered forwarding state [ 25.847272][ T428] br2: port 3(vx20) entered blocking state [ 25.847634][ T428] br2: port 3(vx20) entered disabled state [ 25.847945][ T428] vx20: entered allmulticast mode [ 25.850120][ T428] vx20: entered promiscuous mode [ 25.859576][ T428] br2: port 3(vx20) entered blocking state [ 25.860006][ T428] br2: port 3(vx20) entered forwarding state [ 26.561819][ T45] br2: port 1(w1) entered blocking state [ 26.562365][ T45] br2: port 1(w1) entered forwarding state [ 29.239260][ T470] br2: port 1(w1) entered blocking state [ 29.239598][ T470] br2: port 1(w1) entered disabled state [ 29.240131][ T470] w1: entered allmulticast mode [ 29.242502][ T470] w1: entered promiscuous mode [ 30.016488][ T478] br2: port 2(vx10) entered blocking state [ 30.016822][ T478] br2: port 2(vx10) entered disabled state [ 30.017122][ T478] vx10: entered allmulticast mode [ 30.020159][ T478] vx10: entered promiscuous mode [ 30.020744][ T478] br2: port 2(vx10) entered blocking state [ 30.021008][ T478] br2: port 2(vx10) entered forwarding state [ 30.680488][ T485] br2: port 3(vx20) entered blocking state [ 30.680788][ T485] br2: port 3(vx20) entered disabled state [ 30.681061][ T485] vx20: entered allmulticast mode [ 30.683147][ T485] vx20: entered promiscuous mode [ 30.683683][ T485] br2: port 3(vx20) entered blocking state [ 30.683944][ T485] br2: port 3(vx20) entered forwarding state [ 31.452747][ T56] br2: port 1(w1) entered blocking state [ 31.453135][ T56] br2: port 1(w1) entered forwarding state [ 42.742694][ T572] GACT probability NOT on [ 174.149207][ T1373] veth3: entered promiscuous mode [ 230.853983][ T2300] veth3: left promiscuous mode [ 231.376060][ T2306] veth3: entered promiscuous mode [ 287.549778][ T3233] veth3: left promiscuous mode [ 288.494880][ T3251] vx20: left allmulticast mode [ 288.495135][ T3251] vx20: left promiscuous mode [ 288.495585][ T3251] br1: port 2(vx20) entered disabled state [ 288.590232][ T3252] vx10: left allmulticast mode [ 288.590508][ T3252] vx10: left promiscuous mode [ 288.590853][ T3252] br1: port 1(vx10) entered disabled state [ 294.018725][ T3257] br1: port 1(vx10) entered blocking state [ 294.019128][ T3257] br1: port 1(vx10) entered disabled state [ 294.019554][ T3257] vx10: entered allmulticast mode [ 294.021552][ T3257] vx10: entered promiscuous mode [ 294.022145][ T3257] br1: port 1(vx10) entered blocking state [ 294.022442][ T3257] br1: port 1(vx10) entered forwarding state [ 294.227229][ T3259] br1: port 2(vx20) entered blocking state [ 294.227626][ T3259] br1: port 2(vx20) entered disabled state [ 294.227920][ T3259] vx20: entered allmulticast mode [ 294.230000][ T3259] vx20: entered promiscuous mode [ 294.230623][ T3259] br1: port 2(vx20) entered blocking state [ 294.230899][ T3259] br1: port 2(vx20) entered forwarding state [ 441.607744][ T4089] veth3: entered promiscuous mode [ 498.668217][ T5016] veth3: left promiscuous mode [ 499.322602][ T5022] veth3: entered promiscuous mode [ 553.811602][ T5949] veth3: left promiscuous mode [ 895.912333][ T11] vx20: left allmulticast mode [ 895.912753][ T11] vx20: left promiscuous mode [ 895.913235][ T11] br2: port 3(vx20) entered disabled state [ 895.916659][ T11] vx10: left allmulticast mode [ 895.916886][ T11] vx10: left promiscuous mode [ 895.917238][ T11] br2: port 2(vx10) entered disabled state [ 895.919905][ T11] w1: left allmulticast mode [ 895.920144][ T11] w1: left promiscuous mode [ 895.920513][ T11] br2: port 1(w1) entered disabled state [ 896.400869][T10487] ip (10487) used greatest stack depth: 23904 bytes left [ 896.486659][ T11] vx20: left allmulticast mode [ 896.486905][ T11] vx20: left promiscuous mode [ 896.487228][ T11] br2: port 3(vx20) entered disabled state [ 896.489009][ T11] vx10: left allmulticast mode [ 896.489201][ T11] vx10: left promiscuous mode [ 896.489505][ T11] br2: port 2(vx10) entered disabled state [ 896.492361][ T11] w1: left allmulticast mode [ 896.492565][ T11] w1: left promiscuous mode [ 896.492865][ T11] br2: port 1(w1) entered disabled state [ 897.090795][T10493] ip (10493) used greatest stack depth: 23872 bytes left [ 899.059685][T10515] br1: port 4(veth2) entered disabled state [ 899.147999][T10516] veth2: left allmulticast mode [ 899.148295][T10516] veth2: left promiscuous mode [ 899.148676][T10516] br1: port 4(veth2) entered disabled state [ 899.556957][T10520] br1: port 3(veth1) entered disabled state [ 899.644121][T10522] veth1: left allmulticast mode [ 899.644389][T10522] veth1: left promiscuous mode [ 899.644768][T10522] br1: port 3(veth1) entered disabled state [ 899.828168][T10524] vx20: left allmulticast mode [ 899.828496][T10524] vx20: left promiscuous mode [ 899.828834][T10524] br1: port 2(vx20) entered disabled state [ 900.217222][T10528] vx10: left allmulticast mode [ 900.217574][T10528] vx10: left promiscuous mode [ 900.217889][T10528] br1: port 1(vx10) entered disabled state [ 901.325465][T10538] [ 901.325686][T10538] ============================================ [ 901.325960][T10538] WARNING: possible recursive locking detected [ 901.326162][T10538] 6.13.0-rc5-virtme #1 Not tainted [ 901.326361][T10538] -------------------------------------------- [ 901.326566][T10538] ip/10538 is trying to acquire lock: [ 901.326733][T10538] ffffffffb3bd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 901.327102][T10538] [ 901.327102][T10538] but task is already holding lock: [ 901.327349][T10538] ffffffffb3bd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 901.327692][T10538] [ 901.327692][T10538] other info that might help us debug this: [ 901.327936][T10538] Possible unsafe locking scenario: [ 901.327936][T10538] [ 901.328189][T10538] CPU0 [ 901.328319][T10538] ---- [ 901.328454][T10538] lock((netdev_chain).rwsem); [ 901.328635][T10538] lock((netdev_chain).rwsem); [ 901.328799][T10538] [ 901.328799][T10538] *** DEADLOCK *** [ 901.328799][T10538] [ 901.329039][T10538] May be due to missing lock nesting notation [ 901.329039][T10538] [ 901.329279][T10538] 2 locks held by ip/10538: [ 901.329442][T10538] #0: ffffffffb3be7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 901.329742][T10538] #1: ffffffffb3bd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 901.330095][T10538] [ 901.330095][T10538] stack backtrace: [ 901.330334][T10538] CPU: 2 UID: 0 PID: 10538 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 901.330568][T10538] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 901.330761][T10538] Call Trace: [ 901.330887][T10538] [ 901.330974][T10538] dump_stack_lvl+0x82/0xd0 [ 901.331159][T10538] print_deadlock_bug+0x40a/0x650 [ 901.331322][T10538] validate_chain+0x5bf/0xae0 [ 901.331487][T10538] ? __pfx_validate_chain+0x10/0x10 [ 901.331645][T10538] ? hlock_class+0x4e/0x130 [ 901.331800][T10538] ? mark_lock+0x38/0x3e0 [ 901.331939][T10538] __lock_acquire+0xb9a/0x1680 [ 901.332114][T10538] lock_acquire.part.0+0xeb/0x330 [ 901.332269][T10538] ? blocking_notifier_call_chain+0x50/0x90 [ 901.332457][T10538] ? netconsole_process_cleanups_core+0x2e/0x3d0 [ 901.332645][T10538] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 901.332796][T10538] ? trace_lock_acquire+0x14c/0x1f0 [ 901.332964][T10538] ? lock_acquire+0x32/0xc0 [ 901.333125][T10538] ? blocking_notifier_call_chain+0x50/0x90 [ 901.333323][T10538] down_read+0x9f/0x340 [ 901.333441][T10538] ? blocking_notifier_call_chain+0x50/0x90 [ 901.333629][T10538] ? __pfx_down_read+0x10/0x10 [ 901.333777][T10538] ? __lock_release+0x103/0x460 [ 901.333927][T10538] ? netconsole_netdev_event+0x2c1/0x620 [ 901.334086][T10538] ? __pfx___lock_release+0x10/0x10 [ 901.334245][T10538] blocking_notifier_call_chain+0x50/0x90 [ 901.334407][T10538] __netdev_upper_dev_unlink+0xd9/0x220 [ 901.334565][T10538] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 901.334763][T10538] ? __mutex_unlock_slowpath+0x14a/0x3d0 [ 901.334923][T10538] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 901.335128][T10538] ? rtnl_is_locked+0x15/0x20 [ 901.335284][T10538] netdev_upper_dev_unlink+0x71/0xa0 [ 901.335438][T10538] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 901.335628][T10538] ? netconsole_netdev_event+0x2c1/0x620 [ 901.335784][T10538] ? mutex_is_locked+0x1c/0x60 [ 901.335942][T10538] ? rtnl_is_locked+0x15/0x20 [ 901.336094][T10538] ? netdev_master_upper_dev_get+0x13/0x120 [ 901.336296][T10538] vrf_device_event+0x78/0xe0 [vrf] [ 901.336454][T10538] notifier_call_chain+0xcd/0x150 [ 901.336607][T10538] blocking_notifier_call_chain+0x66/0x90 [ 901.336762][T10538] unregister_netdevice_many_notify+0xa7b/0x1580 [ 901.336950][T10538] ? hlock_class+0x4e/0x130 [ 901.337116][T10538] ? mark_lock+0x38/0x3e0 [ 901.337237][T10538] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 901.337419][T10538] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 901.337630][T10538] ? kfree+0xf3/0x330 [ 901.337756][T10538] ? vlan_vid_del+0x310/0x5e0 [ 901.337913][T10538] ? vlan_vid_del+0x310/0x5e0 [ 901.338105][T10538] rtnl_dellink+0x348/0xa40 [ 901.338266][T10538] ? __pfx_rtnl_dellink+0x10/0x10 [ 901.338431][T10538] ? find_held_lock+0x2c/0x110 [ 901.338590][T10538] ? __lock_release+0x103/0x460 [ 901.338736][T10538] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 901.338882][T10538] ? __pfx___lock_release+0x10/0x10 [ 901.339027][T10538] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 901.339191][T10538] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 901.339340][T10538] ? __pfx_rtnl_dellink+0x10/0x10 [ 901.339491][T10538] rtnetlink_rcv_msg+0x712/0xc10 [ 901.339637][T10538] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 901.339787][T10538] ? hlock_class+0x4e/0x130 [ 901.339939][T10538] ? mark_lock+0x38/0x3e0 [ 901.340061][T10538] ? __lock_acquire+0xb9a/0x1680 [ 901.340219][T10538] netlink_rcv_skb+0x130/0x360 [ 901.340392][T10538] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 901.340538][T10538] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 901.340686][T10538] ? netlink_deliver_tap+0x13e/0x340 [ 901.340847][T10538] ? netlink_deliver_tap+0xc3/0x340 [ 901.340995][T10538] netlink_unicast+0x44b/0x710 [ 901.341201][T10538] ? __pfx_netlink_unicast+0x10/0x10 [ 901.341431][T10538] ? find_held_lock+0x2c/0x110 [ 901.341671][T10538] netlink_sendmsg+0x723/0xbe0 [ 901.341927][T10538] ? __pfx_netlink_sendmsg+0x10/0x10 [ 901.342103][T10538] ____sys_sendmsg+0x7ac/0xa10 [ 901.342253][T10538] ? __pfx_____sys_sendmsg+0x10/0x10 [ 901.342398][T10538] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 901.342628][T10538] ___sys_sendmsg+0xee/0x170 [ 901.342834][T10538] ? __pfx____sys_sendmsg+0x10/0x10 [ 901.342993][T10538] ? kasan_save_stack+0x34/0x50 [ 901.343211][T10538] ? kasan_save_stack+0x24/0x50 [ 901.343364][T10538] ? __kasan_record_aux_stack+0x8e/0xa0 [ 901.343557][T10538] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 901.343855][T10538] ? __x64_sys_close+0x7c/0xd0 [ 901.344028][T10538] ? do_syscall_64+0xc1/0x1d0 [ 901.344228][T10538] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 901.344435][T10538] ? __lock_acquire+0xb9a/0x1680 [ 901.344651][T10538] ? find_held_lock+0x2c/0x110 [ 901.344803][T10538] ? __lock_release+0x103/0x460 [ 901.344948][T10538] ? __virt_addr_valid+0x22b/0x430 [ 901.345099][T10538] ? __pfx___lock_release+0x10/0x10 [ 901.345247][T10538] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 901.345397][T10538] __sys_sendmsg+0x109/0x1a0 [ 901.345557][T10538] ? __pfx___sys_sendmsg+0x10/0x10 [ 901.345716][T10538] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 901.345907][T10538] do_syscall_64+0xc1/0x1d0 [ 901.346055][T10538] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 901.346237][T10538] RIP: 0033:0x7f59a59de9a7 [ 901.346396][T10538] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 901.346989][T10538] RSP: 002b:00007ffd9885ee08 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 901.347263][T10538] RAX: ffffffffffffffda RBX: 00007ffd9885f530 RCX: 00007f59a59de9a7 [ 901.347537][T10538] RDX: 0000000000000000 RSI: 00007ffd9885ee70 RDI: 0000000000000005 [ 901.347897][T10538] RBP: 0000000000000002 R08: 0000000000000003 R09: 0000000000000078 [ 901.348222][T10538] R10: 00007f59a589cef8 R11: 0000000000000246 R12: 0000000000000002 [ 901.348581][T10538] R13: 0000000067792955 R14: 0000000000498600 R15: 0000000000000000 [ 901.348861][T10538]