[ 10.745895][ T249] ip (249) used greatest stack depth: 24400 bytes left [ 11.090525][ T259] ip (259) used greatest stack depth: 23424 bytes left [ 13.530697][ T301] 8021q: 802.1Q VLAN Support v1.8 [ 16.863375][ T348] br1: port 1(vx100) entered blocking state [ 16.863899][ T348] br1: port 1(vx100) entered disabled state [ 16.872747][ T348] vx100: entered allmulticast mode [ 16.875520][ T348] vx100: entered promiscuous mode [ 16.876659][ T348] br1: port 1(vx100) entered blocking state [ 16.877064][ T348] br1: port 1(vx100) entered forwarding state [ 17.035640][ T350] br1: port 2(veth1) entered blocking state [ 17.035976][ T350] br1: port 2(veth1) entered disabled state [ 17.043422][ T350] veth1: entered allmulticast mode [ 17.046167][ T350] veth1: entered promiscuous mode [ 17.126491][ T55] br1: port 2(veth1) entered blocking state [ 17.127176][ T55] br1: port 2(veth1) entered forwarding state [ 17.296694][ T354] br1: port 3(veth2) entered blocking state [ 17.297015][ T354] br1: port 3(veth2) entered disabled state [ 17.297329][ T354] veth2: entered allmulticast mode [ 17.299399][ T354] veth2: entered promiscuous mode [ 17.401405][ T39] br1: port 3(veth2) entered blocking state [ 17.401709][ T39] br1: port 3(veth2) entered forwarding state [ 20.779503][ T400] br2: port 1(w1) entered blocking state [ 20.779779][ T400] br2: port 1(w1) entered disabled state [ 20.780031][ T400] w1: entered allmulticast mode [ 20.782014][ T400] w1: entered promiscuous mode [ 21.329199][ T407] br2: port 2(vx100) entered blocking state [ 21.330416][ T407] br2: port 2(vx100) entered disabled state [ 21.330728][ T407] vx100: entered allmulticast mode [ 21.332761][ T407] vx100: entered promiscuous mode [ 21.333331][ T407] br2: port 2(vx100) entered blocking state [ 21.333628][ T407] br2: port 2(vx100) entered forwarding state [ 22.001774][ T39] br2: port 1(w1) entered blocking state [ 22.002243][ T39] br2: port 1(w1) entered forwarding state [ 24.304936][ T447] br2: port 1(w1) entered blocking state [ 24.305190][ T447] br2: port 1(w1) entered disabled state [ 24.305454][ T447] w1: entered allmulticast mode [ 24.307457][ T447] w1: entered promiscuous mode [ 24.861091][ T454] br2: port 2(vx100) entered blocking state [ 24.861415][ T454] br2: port 2(vx100) entered disabled state [ 24.861725][ T454] vx100: entered allmulticast mode [ 24.864011][ T454] vx100: entered promiscuous mode [ 24.865032][ T454] br2: port 2(vx100) entered blocking state [ 24.865321][ T454] br2: port 2(vx100) entered forwarding state [ 25.556070][ T45] br2: port 1(w1) entered blocking state [ 25.556407][ T45] br2: port 1(w1) entered forwarding state [ 37.750418][ T66] vx100: left allmulticast mode [ 37.750729][ T66] vx100: left promiscuous mode [ 37.751104][ T66] br2: port 2(vx100) entered disabled state [ 37.754652][ T66] w1: left allmulticast mode [ 37.754869][ T66] w1: left promiscuous mode [ 37.755185][ T66] br2: port 1(w1) entered disabled state [ 38.264638][ T66] vx100: left allmulticast mode [ 38.264946][ T66] vx100: left promiscuous mode [ 38.265274][ T66] br2: port 2(vx100) entered disabled state [ 38.268413][ T66] w1: left allmulticast mode [ 38.268608][ T66] w1: left promiscuous mode [ 38.268912][ T66] br2: port 1(w1) entered disabled state [ 40.319872][ T576] br1: port 3(veth2) entered disabled state [ 40.399081][ T577] veth2: left allmulticast mode [ 40.399386][ T577] veth2: left promiscuous mode [ 40.399732][ T577] br1: port 3(veth2) entered disabled state [ 40.565639][ T579] br1: port 2(veth1) entered disabled state [ 40.631984][ T580] veth1: left allmulticast mode [ 40.634022][ T580] veth1: left promiscuous mode [ 40.634607][ T580] br1: port 2(veth1) entered disabled state [ 40.721773][ T581] vx100: left allmulticast mode [ 40.722013][ T581] vx100: left promiscuous mode [ 40.723022][ T581] br1: port 1(vx100) entered disabled state [ 41.640181][ T590] [ 41.640387][ T590] ============================================ [ 41.640711][ T590] WARNING: possible recursive locking detected [ 41.641020][ T590] 6.13.0-rc5-virtme #1 Not tainted [ 41.641277][ T590] -------------------------------------------- [ 41.641594][ T590] ip/590 is trying to acquire lock: [ 41.641849][ T590] ffffffff9d5d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 41.642364][ T590] [ 41.642364][ T590] but task is already holding lock: [ 41.642749][ T590] ffffffff9d5d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 41.643254][ T590] [ 41.643254][ T590] other info that might help us debug this: [ 41.643614][ T590] Possible unsafe locking scenario: [ 41.643614][ T590] [ 41.643977][ T590] CPU0 [ 41.644148][ T590] ---- [ 41.644326][ T590] lock((netdev_chain).rwsem); [ 41.644584][ T590] lock((netdev_chain).rwsem); [ 41.644829][ T590] [ 41.644829][ T590] *** DEADLOCK *** [ 41.644829][ T590] [ 41.645180][ T590] May be due to missing lock nesting notation [ 41.645180][ T590] [ 41.645528][ T590] 2 locks held by ip/590: [ 41.645713][ T590] #0: ffffffff9d5e7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 41.646167][ T590] #1: ffffffff9d5d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 41.646680][ T590] [ 41.646680][ T590] stack backtrace: [ 41.646990][ T590] CPU: 3 UID: 0 PID: 590 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 41.647373][ T590] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 41.647699][ T590] Call Trace: [ 41.647891][ T590] [ 41.648027][ T590] dump_stack_lvl+0x82/0xd0 [ 41.648297][ T590] print_deadlock_bug+0x40a/0x650 [ 41.648549][ T590] validate_chain+0x5bf/0xae0 [ 41.648805][ T590] ? __pfx_validate_chain+0x10/0x10 [ 41.649070][ T590] ? hlock_class+0x4e/0x130 [ 41.649324][ T590] ? mark_lock+0x38/0x3e0 [ 41.649518][ T590] __lock_acquire+0xb9a/0x1680 [ 41.649776][ T590] lock_acquire.part.0+0xeb/0x330 [ 41.650028][ T590] ? blocking_notifier_call_chain+0x50/0x90 [ 41.650335][ T590] ? netconsole_process_cleanups_core+0x2e/0x3d0 [ 41.650641][ T590] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 41.650852][ T590] ? trace_lock_acquire+0x14c/0x1f0 [ 41.651017][ T590] ? lock_acquire+0x32/0xc0 [ 41.651177][ T590] ? blocking_notifier_call_chain+0x50/0x90 [ 41.651392][ T590] down_read+0x9f/0x340 [ 41.651519][ T590] ? blocking_notifier_call_chain+0x50/0x90 [ 41.651728][ T590] ? __pfx_down_read+0x10/0x10 [ 41.651891][ T590] ? __lock_release+0x103/0x460 [ 41.652054][ T590] ? netconsole_netdev_event+0x2c1/0x620 [ 41.652219][ T590] ? __pfx___lock_release+0x10/0x10 [ 41.652401][ T590] blocking_notifier_call_chain+0x50/0x90 [ 41.652569][ T590] __netdev_upper_dev_unlink+0xd9/0x220 [ 41.652738][ T590] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 41.652940][ T590] ? __mutex_unlock_slowpath+0x14a/0x3d0 [ 41.653104][ T590] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 41.653307][ T590] ? rtnl_is_locked+0x15/0x20 [ 41.653482][ T590] netdev_upper_dev_unlink+0x71/0xa0 [ 41.653648][ T590] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 41.653851][ T590] ? netconsole_netdev_event+0x2c1/0x620 [ 41.654019][ T590] ? mutex_is_locked+0x1c/0x60 [ 41.654181][ T590] ? rtnl_is_locked+0x15/0x20 [ 41.654351][ T590] ? netdev_master_upper_dev_get+0x13/0x120 [ 41.654554][ T590] vrf_device_event+0x78/0xe0 [vrf] [ 41.654722][ T590] notifier_call_chain+0xcd/0x150 [ 41.654887][ T590] blocking_notifier_call_chain+0x66/0x90 [ 41.655050][ T590] unregister_netdevice_many_notify+0xa7b/0x1580 [ 41.655260][ T590] ? hlock_class+0x4e/0x130 [ 41.655432][ T590] ? mark_lock+0x38/0x3e0 [ 41.655557][ T590] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 41.655760][ T590] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 41.655966][ T590] ? kfree+0xf3/0x330 [ 41.656096][ T590] ? vlan_vid_del+0x310/0x5e0 [ 41.656264][ T590] ? vlan_vid_del+0x310/0x5e0 [ 41.656440][ T590] rtnl_dellink+0x348/0xa40 [ 41.656605][ T590] ? __pfx_rtnl_dellink+0x10/0x10 [ 41.656781][ T590] ? find_held_lock+0x2c/0x110 [ 41.656945][ T590] ? __lock_release+0x103/0x460 [ 41.657106][ T590] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 41.657267][ T590] ? __pfx___lock_release+0x10/0x10 [ 41.657438][ T590] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 41.657608][ T590] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 41.657768][ T590] ? __pfx_rtnl_dellink+0x10/0x10 [ 41.657932][ T590] rtnetlink_rcv_msg+0x712/0xc10 [ 41.658096][ T590] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 41.658260][ T590] ? hlock_class+0x4e/0x130 [ 41.658449][ T590] ? mark_lock+0x38/0x3e0 [ 41.658573][ T590] ? __lock_acquire+0xb9a/0x1680 [ 41.658737][ T590] netlink_rcv_skb+0x130/0x360 [ 41.658903][ T590] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 41.659065][ T590] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 41.659234][ T590] ? netlink_deliver_tap+0x13e/0x340 [ 41.659424][ T590] ? netlink_deliver_tap+0xc3/0x340 [ 41.659588][ T590] netlink_unicast+0x44b/0x710 [ 41.659749][ T590] ? __pfx_netlink_unicast+0x10/0x10 [ 41.659915][ T590] ? find_held_lock+0x2c/0x110 [ 41.660083][ T590] netlink_sendmsg+0x723/0xbe0 [ 41.660246][ T590] ? __pfx_netlink_sendmsg+0x10/0x10 [ 41.660426][ T590] ____sys_sendmsg+0x7ac/0xa10 [ 41.660591][ T590] ? __pfx_____sys_sendmsg+0x10/0x10 [ 41.660753][ T590] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 41.660961][ T590] ___sys_sendmsg+0xee/0x170 [ 41.661126][ T590] ? __pfx____sys_sendmsg+0x10/0x10 [ 41.661296][ T590] ? kasan_save_stack+0x34/0x50 [ 41.661495][ T590] ? kasan_save_stack+0x24/0x50 [ 41.661660][ T590] ? __kasan_record_aux_stack+0x8e/0xa0 [ 41.661823][ T590] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 41.662031][ T590] ? __x64_sys_close+0x7c/0xd0 [ 41.662196][ T590] ? do_syscall_64+0xc1/0x1d0 [ 41.662379][ T590] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 41.662639][ T590] ? __lock_acquire+0xb9a/0x1680 [ 41.662807][ T590] ? find_held_lock+0x2c/0x110 [ 41.662977][ T590] ? __lock_release+0x103/0x460 [ 41.663140][ T590] ? __virt_addr_valid+0x22b/0x430 [ 41.663302][ T590] ? __pfx___lock_release+0x10/0x10 [ 41.663496][ T590] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 41.663660][ T590] __sys_sendmsg+0x109/0x1a0 [ 41.663830][ T590] ? __pfx___sys_sendmsg+0x10/0x10 [ 41.663990][ T590] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 41.664199][ T590] do_syscall_64+0xc1/0x1d0 [ 41.664388][ T590] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 41.664592][ T590] RIP: 0033:0x7f63834879a7 [ 41.664770][ T590] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 41.665358][ T590] RSP: 002b:00007ffe27ac6968 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 41.665610][ T590] RAX: ffffffffffffffda RBX: 00007ffe27ac7090 RCX: 00007f63834879a7 [ 41.665851][ T590] RDX: 0000000000000000 RSI: 00007ffe27ac69d0 RDI: 0000000000000005 [ 41.666099][ T590] RBP: 0000000000000002 R08: 0000000000000003 R09: 0000000000000078 [ 41.666355][ T590] R10: 00007f6383345ef8 R11: 0000000000000246 R12: 0000000000000002 [ 41.666597][ T590] R13: 0000000067790499 R14: 0000000000498600 R15: 0000000000000000 [ 41.666845][ T590]