[ 31.486468][ T537] ip (537) used greatest stack depth: 23248 bytes left [ 33.890266][ T579] 8021q: 802.1Q VLAN Support v1.8 [ 37.418827][ T625] br1: port 1(vx10) entered blocking state [ 37.419566][ T625] br1: port 1(vx10) entered disabled state [ 37.419843][ T625] vx10: entered allmulticast mode [ 37.424117][ T625] vx10: entered promiscuous mode [ 37.425730][ T625] br1: port 1(vx10) entered blocking state [ 37.426312][ T625] br1: port 1(vx10) entered forwarding state [ 37.778200][ T630] br1: port 2(vx20) entered blocking state [ 37.778576][ T630] br1: port 2(vx20) entered disabled state [ 37.779001][ T630] vx20: entered allmulticast mode [ 37.782245][ T630] vx20: entered promiscuous mode [ 37.783173][ T630] br1: port 2(vx20) entered blocking state [ 37.783524][ T630] br1: port 2(vx20) entered forwarding state [ 37.967905][ T632] br1: port 3(veth1) entered blocking state [ 37.968316][ T632] br1: port 3(veth1) entered disabled state [ 37.968712][ T632] veth1: entered allmulticast mode [ 37.970725][ T632] veth1: entered promiscuous mode [ 38.047190][ T470] br1: port 3(veth1) entered blocking state [ 38.047600][ T470] br1: port 3(veth1) entered forwarding state [ 38.285136][ T636] br1: port 4(veth2) entered blocking state [ 38.285510][ T636] br1: port 4(veth2) entered disabled state [ 38.285852][ T636] veth2: entered allmulticast mode [ 38.288502][ T636] veth2: entered promiscuous mode [ 38.387100][ T40] br1: port 4(veth2) entered blocking state [ 38.387644][ T40] br1: port 4(veth2) entered forwarding state [ 41.909698][ T686] br2: port 1(w1) entered blocking state [ 41.909941][ T686] br2: port 1(w1) entered disabled state [ 41.910188][ T686] w1: entered allmulticast mode [ 41.912725][ T686] w1: entered promiscuous mode [ 42.546940][ T695] br2: port 2(vx10) entered blocking state [ 42.547249][ T695] br2: port 2(vx10) entered disabled state [ 42.547550][ T695] vx10: entered allmulticast mode [ 42.549514][ T695] vx10: entered promiscuous mode [ 42.550076][ T695] br2: port 2(vx10) entered blocking state [ 42.550630][ T695] br2: port 2(vx10) entered forwarding state [ 43.182574][ T702] br2: port 3(vx20) entered blocking state [ 43.183002][ T702] br2: port 3(vx20) entered disabled state [ 43.183471][ T702] vx20: entered allmulticast mode [ 43.186843][ T702] vx20: entered promiscuous mode [ 43.187654][ T702] br2: port 3(vx20) entered blocking state [ 43.188043][ T702] br2: port 3(vx20) entered forwarding state [ 43.922659][ T45] br2: port 1(w1) entered blocking state [ 43.923095][ T45] br2: port 1(w1) entered forwarding state [ 46.284278][ T742] br2: port 1(w1) entered blocking state [ 46.284540][ T742] br2: port 1(w1) entered disabled state [ 46.284784][ T742] w1: entered allmulticast mode [ 46.286830][ T742] w1: entered promiscuous mode [ 46.921922][ T750] br2: port 2(vx10) entered blocking state [ 46.922231][ T750] br2: port 2(vx10) entered disabled state [ 46.922489][ T750] vx10: entered allmulticast mode [ 46.924602][ T750] vx10: entered promiscuous mode [ 46.925120][ T750] br2: port 2(vx10) entered blocking state [ 46.925400][ T750] br2: port 2(vx10) entered forwarding state [ 47.500889][ T757] br2: port 3(vx20) entered blocking state [ 47.501186][ T757] br2: port 3(vx20) entered disabled state [ 47.501454][ T757] vx20: entered allmulticast mode [ 47.503422][ T757] vx20: entered promiscuous mode [ 47.504249][ T757] br2: port 3(vx20) entered blocking state [ 47.504513][ T757] br2: port 3(vx20) entered forwarding state [ 48.147694][ T40] br2: port 1(w1) entered blocking state [ 48.148001][ T40] br2: port 1(w1) entered forwarding state [ 95.284120][ T1490] veth3: entered promiscuous mode [ 142.857159][ T2417] veth3: left promiscuous mode [ 143.301495][ T2423] veth3: entered promiscuous mode [ 191.187073][ T3350] veth3: left promiscuous mode [ 192.093039][ T3368] vx20: left allmulticast mode [ 192.093340][ T3368] vx20: left promiscuous mode [ 192.093720][ T3368] br1: port 2(vx20) entered disabled state [ 192.192627][ T3369] vx10: left allmulticast mode [ 192.192868][ T3369] vx10: left promiscuous mode [ 192.193216][ T3369] br1: port 1(vx10) entered disabled state [ 197.623855][ T3374] br1: port 1(vx10) entered blocking state [ 197.624334][ T3374] br1: port 1(vx10) entered disabled state [ 197.624770][ T3374] vx10: entered allmulticast mode [ 197.626918][ T3374] vx10: entered promiscuous mode [ 197.627512][ T3374] br1: port 1(vx10) entered blocking state [ 197.627758][ T3374] br1: port 1(vx10) entered forwarding state [ 197.801314][ T3376] br1: port 2(vx20) entered blocking state [ 197.801609][ T3376] br1: port 2(vx20) entered disabled state [ 197.801891][ T3376] vx20: entered allmulticast mode [ 197.803845][ T3376] vx20: entered promiscuous mode [ 197.804428][ T3376] br1: port 2(vx20) entered blocking state [ 197.804698][ T3376] br1: port 2(vx20) entered forwarding state [ 244.144770][ T4054] veth3: entered promiscuous mode [ 290.549387][ T4981] veth3: left promiscuous mode [ 290.998290][ T4987] veth3: entered promiscuous mode [ 336.951993][ T5914] veth3: left promiscuous mode [ 338.218502][ T5937] vx20: left allmulticast mode [ 338.218802][ T5937] vx20: left promiscuous mode [ 338.219204][ T5937] br1: port 2(vx20) entered disabled state [ 338.298733][ T5938] vx10: left allmulticast mode [ 338.299044][ T5938] vx10: left promiscuous mode [ 338.299622][ T5938] br1: port 1(vx10) entered disabled state [ 343.668791][ T5943] br1: port 1(vx10) entered blocking state [ 343.669165][ T5943] br1: port 1(vx10) entered disabled state [ 343.669464][ T5943] vx10: entered allmulticast mode [ 343.671441][ T5943] vx10: entered promiscuous mode [ 343.671995][ T5943] br1: port 1(vx10) entered blocking state [ 343.672272][ T5943] br1: port 1(vx10) entered forwarding state [ 343.848097][ T5945] br1: port 2(vx20) entered blocking state [ 343.848601][ T5945] br1: port 2(vx20) entered disabled state [ 343.848904][ T5945] vx20: entered allmulticast mode [ 343.851027][ T5945] vx20: entered promiscuous mode [ 343.851631][ T5945] br1: port 2(vx20) entered blocking state [ 343.851902][ T5945] br1: port 2(vx20) entered forwarding state [ 650.255433][ T9220] vx20: left allmulticast mode [ 650.255738][ T9220] vx20: left promiscuous mode [ 650.256232][ T9220] br1: port 2(vx20) entered disabled state [ 650.337859][ T9221] vx10: left allmulticast mode [ 650.338098][ T9221] vx10: left promiscuous mode [ 650.338467][ T9221] br1: port 1(vx10) entered disabled state [ 655.700672][ T9226] br1: port 1(vx10) entered blocking state [ 655.701027][ T9226] br1: port 1(vx10) entered disabled state [ 655.701324][ T9226] vx10: entered allmulticast mode [ 655.703270][ T9226] vx10: entered promiscuous mode [ 655.703779][ T9226] br1: port 1(vx10) entered blocking state [ 655.704018][ T9226] br1: port 1(vx10) entered forwarding state [ 655.895696][ T9228] br1: port 2(vx20) entered blocking state [ 655.896006][ T9228] br1: port 2(vx20) entered disabled state [ 655.896305][ T9228] vx20: entered allmulticast mode [ 655.898503][ T9228] vx20: entered promiscuous mode [ 655.899026][ T9228] br1: port 2(vx20) entered blocking state [ 655.899278][ T9228] br1: port 2(vx20) entered forwarding state [ 760.428517][ T11] vx20: left allmulticast mode [ 760.428892][ T11] vx20: left promiscuous mode [ 760.429484][ T11] br2: port 3(vx20) entered disabled state [ 760.433969][ T11] vx10: left allmulticast mode [ 760.434221][ T11] vx10: left promiscuous mode [ 760.434534][ T11] br2: port 2(vx10) entered disabled state [ 760.437195][ T11] w1: left allmulticast mode [ 760.437426][ T11] w1: left promiscuous mode [ 760.437754][ T11] br2: port 1(w1) entered disabled state [ 760.964804][ T11] vx20: left allmulticast mode [ 760.965414][ T11] vx20: left promiscuous mode [ 760.966166][ T11] br2: port 3(vx20) entered disabled state [ 760.971241][ T11] vx10: left allmulticast mode [ 760.971464][ T11] vx10: left promiscuous mode [ 760.971827][ T11] br2: port 2(vx10) entered disabled state [ 760.974820][ T11] w1: left allmulticast mode [ 760.975132][ T11] w1: left promiscuous mode [ 760.975525][ T11] br2: port 1(w1) entered disabled state [ 763.356025][T11132] br1: port 4(veth2) entered disabled state [ 763.436444][T11133] veth2: left allmulticast mode [ 763.436725][T11133] veth2: left promiscuous mode [ 763.437072][T11133] br1: port 4(veth2) entered disabled state [ 763.713164][T11136] br1: port 3(veth1) entered disabled state [ 763.802761][T11137] veth1: left allmulticast mode [ 763.803007][T11137] veth1: left promiscuous mode [ 763.803347][T11137] br1: port 3(veth1) entered disabled state [ 763.963902][T11139] vx20: left allmulticast mode [ 763.964175][T11139] vx20: left promiscuous mode [ 763.964505][T11139] br1: port 2(vx20) entered disabled state [ 764.339312][T11143] vx10: left allmulticast mode [ 764.339638][T11143] vx10: left promiscuous mode [ 764.340013][T11143] br1: port 1(vx10) entered disabled state [ 765.322902][T11152] [ 765.323066][T11152] ============================================ [ 765.323274][T11152] WARNING: possible recursive locking detected [ 765.323472][T11152] 6.13.0-rc5-virtme #1 Not tainted [ 765.323632][T11152] -------------------------------------------- [ 765.323828][T11152] ip/11152 is trying to acquire lock: [ 765.323986][T11152] ffffffffacbd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 765.324333][T11152] [ 765.324333][T11152] but task is already holding lock: [ 765.324577][T11152] ffffffffacbd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 765.324898][T11152] [ 765.324898][T11152] other info that might help us debug this: [ 765.325131][T11152] Possible unsafe locking scenario: [ 765.325131][T11152] [ 765.325371][T11152] CPU0 [ 765.325492][T11152] ---- [ 765.325615][T11152] lock((netdev_chain).rwsem); [ 765.325778][T11152] lock((netdev_chain).rwsem); [ 765.325941][T11152] [ 765.325941][T11152] *** DEADLOCK *** [ 765.325941][T11152] [ 765.326184][T11152] May be due to missing lock nesting notation [ 765.326184][T11152] [ 765.326427][T11152] 2 locks held by ip/11152: [ 765.326593][T11152] #0: ffffffffacbe7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 765.326885][T11152] #1: ffffffffacbd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 765.327210][T11152] [ 765.327210][T11152] stack backtrace: [ 765.327411][T11152] CPU: 0 UID: 0 PID: 11152 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 765.327652][T11152] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 765.327849][T11152] Call Trace: [ 765.327976][T11152] <TASK> [ 765.328062][T11152] dump_stack_lvl+0x82/0xd0 [ 765.328228][T11152] print_deadlock_bug+0x40a/0x650 [ 765.328387][T11152] validate_chain+0x5bf/0xae0 [ 765.328552][T11152] ? __pfx_validate_chain+0x10/0x10 [ 765.328714][T11152] ? hlock_class+0x4e/0x130 [ 765.328872][T11152] ? mark_lock+0x38/0x3e0 [ 765.328999][T11152] __lock_acquire+0xb9a/0x1680 [ 765.329161][T11152] lock_acquire.part.0+0xeb/0x330 [ 765.329320][T11152] ? blocking_notifier_call_chain+0x50/0x90 [ 765.329519][T11152] ? netconsole_process_cleanups_core+0x2e/0x3d0 [ 765.329720][T11152] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 765.329879][T11152] ? trace_lock_acquire+0x14c/0x1f0 [ 765.330042][T11152] ? lock_acquire+0x32/0xc0 [ 765.330204][T11152] ? blocking_notifier_call_chain+0x50/0x90 [ 765.330412][T11152] down_read+0x9f/0x340 [ 765.330539][T11152] ? blocking_notifier_call_chain+0x50/0x90 [ 765.330745][T11152] ? __pfx_down_read+0x10/0x10 [ 765.330906][T11152] ? __lock_release+0x103/0x460 [ 765.331069][T11152] ? netconsole_netdev_event+0x2c1/0x620 [ 765.331233][T11152] ? __pfx___lock_release+0x10/0x10 [ 765.331398][T11152] blocking_notifier_call_chain+0x50/0x90 [ 765.331563][T11152] __netdev_upper_dev_unlink+0xd9/0x220 [ 765.331727][T11152] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 765.331934][T11152] ? __mutex_unlock_slowpath+0x14a/0x3d0 [ 765.332104][T11152] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 765.332312][T11152] ? rtnl_is_locked+0x15/0x20 [ 765.332479][T11152] netdev_upper_dev_unlink+0x71/0xa0 [ 765.332660][T11152] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 765.332858][T11152] ? netconsole_netdev_event+0x2c1/0x620 [ 765.333021][T11152] ? mutex_is_locked+0x1c/0x60 [ 765.333181][T11152] ? rtnl_is_locked+0x15/0x20 [ 765.333344][T11152] ? netdev_master_upper_dev_get+0x13/0x120 [ 765.333550][T11152] vrf_device_event+0x78/0xe0 [vrf] [ 765.333717][T11152] notifier_call_chain+0xcd/0x150 [ 765.333878][T11152] blocking_notifier_call_chain+0x66/0x90 [ 765.334041][T11152] unregister_netdevice_many_notify+0xa7b/0x1580 [ 765.334237][T11152] ? hlock_class+0x4e/0x130 [ 765.334396][T11152] ? mark_lock+0x38/0x3e0 [ 765.334519][T11152] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 765.334724][T11152] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 765.334925][T11152] ? kfree+0xf3/0x330 [ 765.335047][T11152] ? vlan_vid_del+0x310/0x5e0 [ 765.335210][T11152] ? vlan_vid_del+0x310/0x5e0 [ 765.335374][T11152] rtnl_dellink+0x348/0xa40 [ 765.335537][T11152] ? __pfx_rtnl_dellink+0x10/0x10 [ 765.335712][T11152] ? find_held_lock+0x2c/0x110 [ 765.335880][T11152] ? __lock_release+0x103/0x460 [ 765.336040][T11152] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 765.336198][T11152] ? __pfx___lock_release+0x10/0x10 [ 765.336357][T11152] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 765.336526][T11152] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 765.336690][T11152] ? __pfx_rtnl_dellink+0x10/0x10 [ 765.336851][T11152] rtnetlink_rcv_msg+0x712/0xc10 [ 765.337015][T11152] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 765.337177][T11152] ? hlock_class+0x4e/0x130 [ 765.337336][T11152] ? mark_lock+0x38/0x3e0 [ 765.337458][T11152] ? __lock_acquire+0xb9a/0x1680 [ 765.337619][T11152] netlink_rcv_skb+0x130/0x360 [ 765.337777][T11152] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 765.337938][T11152] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 765.338100][T11152] ? netlink_deliver_tap+0x13e/0x340 [ 765.338266][T11152] ? netlink_deliver_tap+0xc3/0x340 [ 765.338426][T11152] netlink_unicast+0x44b/0x710 [ 765.338589][T11152] ? __pfx_netlink_unicast+0x10/0x10 [ 765.338749][T11152] ? find_held_lock+0x2c/0x110 [ 765.338911][T11152] netlink_sendmsg+0x723/0xbe0 [ 765.339070][T11152] ? __pfx_netlink_sendmsg+0x10/0x10 [ 765.339236][T11152] ____sys_sendmsg+0x7ac/0xa10 [ 765.339401][T11152] ? __pfx_____sys_sendmsg+0x10/0x10 [ 765.339563][T11152] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 765.339766][T11152] ___sys_sendmsg+0xee/0x170 [ 765.339930][T11152] ? __pfx____sys_sendmsg+0x10/0x10 [ 765.340089][T11152] ? kasan_save_stack+0x34/0x50 [ 765.340252][T11152] ? kasan_save_stack+0x24/0x50 [ 765.340413][T11152] ? __kasan_record_aux_stack+0x8e/0xa0 [ 765.340583][T11152] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 765.340784][T11152] ? __x64_sys_close+0x7c/0xd0 [ 765.340945][T11152] ? do_syscall_64+0xc1/0x1d0 [ 765.341107][T11152] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 765.341311][T11152] ? __lock_acquire+0xb9a/0x1680 [ 765.341473][T11152] ? find_held_lock+0x2c/0x110 [ 765.341634][T11152] ? __lock_release+0x103/0x460 [ 765.341801][T11152] ? __virt_addr_valid+0x22b/0x430 [ 765.341966][T11152] ? __pfx___lock_release+0x10/0x10 [ 765.342127][T11152] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 765.342290][T11152] __sys_sendmsg+0x109/0x1a0 [ 765.342452][T11152] ? __pfx___sys_sendmsg+0x10/0x10 [ 765.342627][T11152] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 765.342831][T11152] do_syscall_64+0xc1/0x1d0 [ 765.342996][T11152] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 765.343193][T11152] RIP: 0033:0x7f4392ea39a7 [ 765.343366][T11152] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 765.343929][T11152] RSP: 002b:00007fff2bf5d6d8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 765.344176][T11152] RAX: ffffffffffffffda RBX: 00007fff2bf5de00 RCX: 00007f4392ea39a7 [ 765.344419][T11152] RDX: 0000000000000000 RSI: 00007fff2bf5d740 RDI: 0000000000000005 [ 765.344664][T11152] RBP: 0000000000000002 R08: 0000000000000003 R09: 0000000000000078 [ 765.344906][T11152] R10: 00007f4392d61ef8 R11: 0000000000000246 R12: 0000000000000002 [ 765.345153][T11152] R13: 000000006779579a R14: 0000000000498600 R15: 0000000000000000 [ 765.345398][T11152] </TASK>