[ 74.200864][ T997] 8021q: 802.1Q VLAN Support v1.8 [ 79.521526][ T1127] br10: port 1(veth1.10) entered blocking state [ 79.521819][ T1127] br10: port 1(veth1.10) entered disabled state [ 79.522480][ T1127] veth1.10: entered allmulticast mode [ 79.522668][ T1127] veth1: entered allmulticast mode [ 79.524674][ T1127] veth1.10: entered promiscuous mode [ 79.524876][ T1127] veth1: entered promiscuous mode [ 79.680664][ T1132] br10: port 2(veth2.10) entered blocking state [ 79.680974][ T1132] br10: port 2(veth2.10) entered disabled state [ 79.681333][ T1132] veth2.10: entered allmulticast mode [ 79.681532][ T1132] veth2: entered allmulticast mode [ 79.683775][ T1132] veth2.10: entered promiscuous mode [ 79.684054][ T1132] veth2: entered promiscuous mode [ 79.810025][ T1137] br10: port 2(veth2.10) entered blocking state [ 79.810705][ T1137] br10: port 2(veth2.10) entered forwarding state [ 79.811534][ T1137] br10: port 1(veth1.10) entered blocking state [ 79.811792][ T1137] br10: port 1(veth1.10) entered forwarding state [ 80.962747][ T1173] br11: port 1(veth1.11) entered blocking state [ 80.963062][ T1173] br11: port 1(veth1.11) entered disabled state [ 80.963647][ T1173] veth1.11: entered allmulticast mode [ 80.965660][ T1173] veth1.11: entered promiscuous mode [ 81.098901][ T1178] br11: port 2(veth2.11) entered blocking state [ 81.099278][ T1178] br11: port 2(veth2.11) entered disabled state [ 81.100023][ T1178] veth2.11: entered allmulticast mode [ 81.102928][ T1178] veth2.11: entered promiscuous mode [ 81.210010][ T1183] br11: port 2(veth2.11) entered blocking state [ 81.210410][ T1183] br11: port 2(veth2.11) entered forwarding state [ 81.210927][ T1183] br11: port 1(veth1.11) entered blocking state [ 81.211231][ T1183] br11: port 1(veth1.11) entered forwarding state [ 82.230800][ T1219] br12: port 1(veth1.12) entered blocking state [ 82.231140][ T1219] br12: port 1(veth1.12) entered disabled state [ 82.231451][ T1219] veth1.12: entered allmulticast mode [ 82.233463][ T1219] veth1.12: entered promiscuous mode [ 82.378368][ T1225] br12: port 2(veth2.12) entered blocking state [ 82.378662][ T1225] br12: port 2(veth2.12) entered disabled state [ 82.378938][ T1225] veth2.12: entered allmulticast mode [ 82.381198][ T1225] veth2.12: entered promiscuous mode [ 82.517892][ T1230] br12: port 2(veth2.12) entered blocking state [ 82.518232][ T1230] br12: port 2(veth2.12) entered forwarding state [ 82.519078][ T1230] br12: port 1(veth1.12) entered blocking state [ 82.519367][ T1230] br12: port 1(veth1.12) entered forwarding state [ 492.154216][ T2660] br12: port 2(veth2.12) entered disabled state [ 492.246510][ T2662] br12: port 1(veth1.12) entered disabled state [ 492.431871][ T2666] veth2.12: left allmulticast mode [ 492.433078][ T2666] veth2.12: left promiscuous mode [ 492.433415][ T2666] br12: port 2(veth2.12) entered disabled state [ 492.514796][ T2668] veth1.12: left allmulticast mode [ 492.515022][ T2668] veth1.12: left promiscuous mode [ 492.515352][ T2668] br12: port 1(veth1.12) entered disabled state [ 493.050552][ T2678] br11: port 2(veth2.11) entered disabled state [ 493.139173][ T2680] br11: port 1(veth1.11) entered disabled state [ 493.337262][ T2684] veth2.11: left allmulticast mode [ 493.337524][ T2684] veth2.11: left promiscuous mode [ 493.337857][ T2684] br11: port 2(veth2.11) entered disabled state [ 493.433571][ T2686] veth1.11: left allmulticast mode [ 493.433794][ T2686] veth1.11: left promiscuous mode [ 493.434102][ T2686] br11: port 1(veth1.11) entered disabled state [ 493.952344][ T2694] veth2: left allmulticast mode [ 493.952676][ T2694] veth2: left promiscuous mode [ 493.954577][ T2694] br10: port 2(veth2.10) entered disabled state [ 494.038423][ T2696] veth1: left allmulticast mode [ 494.038681][ T2696] veth1: left promiscuous mode [ 494.041235][ T2696] br10: port 1(veth1.10) entered disabled state [ 494.242305][ T2700] veth2.10: left allmulticast mode [ 494.242566][ T2700] veth2.10: left promiscuous mode [ 494.242894][ T2700] br10: port 2(veth2.10) entered disabled state [ 494.388169][ T2702] veth1.10: left allmulticast mode [ 494.388420][ T2702] veth1.10: left promiscuous mode [ 494.388746][ T2702] br10: port 1(veth1.10) entered disabled state [ 495.281199][ T2719] [ 495.281330][ T2719] ============================================ [ 495.281533][ T2719] WARNING: possible recursive locking detected [ 495.281717][ T2719] 6.13.0-rc5-virtme #1 Not tainted [ 495.281861][ T2719] -------------------------------------------- [ 495.282057][ T2719] ip/2719 is trying to acquire lock: [ 495.282210][ T2719] ffffffffa71d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 495.282526][ T2719] [ 495.282526][ T2719] but task is already holding lock: [ 495.282739][ T2719] ffffffffa71d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 495.283057][ T2719] [ 495.283057][ T2719] other info that might help us debug this: [ 495.283282][ T2719] Possible unsafe locking scenario: [ 495.283282][ T2719] [ 495.283498][ T2719] CPU0 [ 495.283616][ T2719] ---- [ 495.283726][ T2719] lock((netdev_chain).rwsem); [ 495.283872][ T2719] lock((netdev_chain).rwsem); [ 495.284016][ T2719] [ 495.284016][ T2719] *** DEADLOCK *** [ 495.284016][ T2719] [ 495.284241][ T2719] May be due to missing lock nesting notation [ 495.284241][ T2719] [ 495.284463][ T2719] 2 locks held by ip/2719: [ 495.284618][ T2719] #0: ffffffffa71e7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 495.284902][ T2719] #1: ffffffffa71d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 495.285219][ T2719] [ 495.285219][ T2719] stack backtrace: [ 495.285409][ T2719] CPU: 0 UID: 0 PID: 2719 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 495.285634][ T2719] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 495.285824][ T2719] Call Trace: [ 495.285938][ T2719] [ 495.286018][ T2719] dump_stack_lvl+0x82/0xd0 [ 495.286187][ T2719] print_deadlock_bug+0x40a/0x650 [ 495.286341][ T2719] validate_chain+0x5bf/0xae0 [ 495.286498][ T2719] ? __pfx_validate_chain+0x10/0x10 [ 495.286646][ T2719] ? hlock_class+0x4e/0x130 [ 495.286792][ T2719] ? mark_lock+0x38/0x3e0 [ 495.286910][ T2719] __lock_acquire+0xb9a/0x1680 [ 495.287061][ T2719] lock_acquire.part.0+0xeb/0x330 [ 495.287211][ T2719] ? blocking_notifier_call_chain+0x50/0x90 [ 495.287395][ T2719] ? netconsole_process_cleanups_core+0x2e/0x3d0 [ 495.287591][ T2719] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 495.287748][ T2719] ? trace_lock_acquire+0x14c/0x1f0 [ 495.287900][ T2719] ? lock_acquire+0x32/0xc0 [ 495.288052][ T2719] ? blocking_notifier_call_chain+0x50/0x90 [ 495.288237][ T2719] down_read+0x9f/0x340 [ 495.288347][ T2719] ? blocking_notifier_call_chain+0x50/0x90 [ 495.288530][ T2719] ? __pfx_down_read+0x10/0x10 [ 495.288677][ T2719] ? __lock_release+0x103/0x460 [ 495.288823][ T2719] ? netconsole_netdev_event+0x2c1/0x620 [ 495.288976][ T2719] ? __pfx___lock_release+0x10/0x10 [ 495.289131][ T2719] blocking_notifier_call_chain+0x50/0x90 [ 495.289278][ T2719] __netdev_upper_dev_unlink+0xd9/0x220 [ 495.289423][ T2719] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 495.289604][ T2719] ? __mutex_unlock_slowpath+0x14a/0x3d0 [ 495.289749][ T2719] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 495.289941][ T2719] ? rtnl_is_locked+0x15/0x20 [ 495.290095][ T2719] netdev_upper_dev_unlink+0x71/0xa0 [ 495.290240][ T2719] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 495.290420][ T2719] ? netconsole_netdev_event+0x2c1/0x620 [ 495.290568][ T2719] ? mutex_is_locked+0x1c/0x60 [ 495.290714][ T2719] ? rtnl_is_locked+0x15/0x20 [ 495.290859][ T2719] ? netdev_master_upper_dev_get+0x13/0x120 [ 495.291049][ T2719] vrf_device_event+0x78/0xe0 [vrf] [ 495.291212][ T2719] notifier_call_chain+0xcd/0x150 [ 495.291358][ T2719] blocking_notifier_call_chain+0x66/0x90 [ 495.291513][ T2719] unregister_netdevice_many_notify+0xa7b/0x1580 [ 495.291697][ T2719] ? hlock_class+0x4e/0x130 [ 495.291841][ T2719] ? mark_lock+0x38/0x3e0 [ 495.291962][ T2719] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 495.292147][ T2719] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 495.292334][ T2719] ? kfree+0xf3/0x330 [ 495.292446][ T2719] ? vlan_vid_del+0x310/0x5e0 [ 495.292596][ T2719] ? vlan_vid_del+0x310/0x5e0 [ 495.292741][ T2719] rtnl_dellink+0x348/0xa40 [ 495.292888][ T2719] ? __pfx_rtnl_dellink+0x10/0x10 [ 495.293056][ T2719] ? find_held_lock+0x2c/0x110 [ 495.293203][ T2719] ? __lock_release+0x103/0x460 [ 495.293348][ T2719] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 495.293497][ T2719] ? __pfx___lock_release+0x10/0x10 [ 495.293642][ T2719] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 495.293788][ T2719] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 495.293936][ T2719] ? __pfx_rtnl_dellink+0x10/0x10 [ 495.294084][ T2719] rtnetlink_rcv_msg+0x712/0xc10 [ 495.294230][ T2719] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 495.294385][ T2719] ? hlock_class+0x4e/0x130 [ 495.294542][ T2719] ? mark_lock+0x38/0x3e0 [ 495.294655][ T2719] ? __lock_acquire+0xb9a/0x1680 [ 495.294811][ T2719] netlink_rcv_skb+0x130/0x360 [ 495.294967][ T2719] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 495.295136][ T2719] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 495.295305][ T2719] ? netlink_deliver_tap+0x13e/0x340 [ 495.295469][ T2719] ? netlink_deliver_tap+0xc3/0x340 [ 495.295637][ T2719] netlink_unicast+0x44b/0x710 [ 495.295800][ T2719] ? __pfx_netlink_unicast+0x10/0x10 [ 495.295962][ T2719] ? find_held_lock+0x2c/0x110 [ 495.296123][ T2719] netlink_sendmsg+0x723/0xbe0 [ 495.296281][ T2719] ? __pfx_netlink_sendmsg+0x10/0x10 [ 495.296444][ T2719] ____sys_sendmsg+0x7ac/0xa10 [ 495.296607][ T2719] ? __pfx_____sys_sendmsg+0x10/0x10 [ 495.296769][ T2719] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 495.296971][ T2719] ___sys_sendmsg+0xee/0x170 [ 495.297135][ T2719] ? __pfx____sys_sendmsg+0x10/0x10 [ 495.297297][ T2719] ? kasan_save_stack+0x34/0x50 [ 495.297458][ T2719] ? kasan_save_stack+0x24/0x50 [ 495.297617][ T2719] ? __kasan_record_aux_stack+0x8e/0xa0 [ 495.297779][ T2719] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 495.297985][ T2719] ? __x64_sys_close+0x7c/0xd0 [ 495.298145][ T2719] ? do_syscall_64+0xc1/0x1d0 [ 495.298304][ T2719] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 495.298504][ T2719] ? __lock_acquire+0xb9a/0x1680 [ 495.298669][ T2719] ? find_held_lock+0x2c/0x110 [ 495.298828][ T2719] ? __lock_release+0x103/0x460 [ 495.298984][ T2719] ? __virt_addr_valid+0x22b/0x430 [ 495.299146][ T2719] ? __pfx___lock_release+0x10/0x10 [ 495.299310][ T2719] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 495.299472][ T2719] __sys_sendmsg+0x109/0x1a0 [ 495.299639][ T2719] ? __pfx___sys_sendmsg+0x10/0x10 [ 495.299803][ T2719] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 495.300008][ T2719] do_syscall_64+0xc1/0x1d0 [ 495.300169][ T2719] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 495.300378][ T2719] RIP: 0033:0x7f2eb69909a7 [ 495.300538][ T2719] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 495.301068][ T2719] RSP: 002b:00007fffd20e9038 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 495.301299][ T2719] RAX: ffffffffffffffda RBX: 00007fffd20e9760 RCX: 00007f2eb69909a7 [ 495.301525][ T2719] RDX: 0000000000000000 RSI: 00007fffd20e90a0 RDI: 0000000000000005 [ 495.301755][ T2719] RBP: 0000000000000002 R08: 0000000000000003 R09: 0000000000000078 [ 495.301972][ T2719] R10: 00007f2eb684eef8 R11: 0000000000000246 R12: 0000000000000002 [ 495.302191][ T2719] R13: 000000006779565c R14: 0000000000498600 R15: 0000000000000000 [ 495.302417][ T2719]