[ 10.382367][ T249] ip (249) used greatest stack depth: 24352 bytes left [ 11.131332][ T271] ip (271) used greatest stack depth: 23312 bytes left [ 13.138343][ T300] 8021q: 802.1Q VLAN Support v1.8 [ 16.944608][ T349] br1: port 1(vx10) entered blocking state [ 16.945076][ T349] br1: port 1(vx10) entered disabled state [ 16.945475][ T349] vx10: entered allmulticast mode [ 16.947889][ T349] vx10: entered promiscuous mode [ 16.948830][ T349] br1: port 1(vx10) entered blocking state [ 16.949176][ T349] br1: port 1(vx10) entered forwarding state [ 17.297361][ T354] br1: port 2(vx20) entered blocking state [ 17.297645][ T354] br1: port 2(vx20) entered disabled state [ 17.297908][ T354] vx20: entered allmulticast mode [ 17.299828][ T354] vx20: entered promiscuous mode [ 17.300664][ T354] br1: port 2(vx20) entered blocking state [ 17.300907][ T354] br1: port 2(vx20) entered forwarding state [ 17.446771][ T356] br1: port 3(veth1) entered blocking state [ 17.447163][ T356] br1: port 3(veth1) entered disabled state [ 17.447467][ T356] veth1: entered allmulticast mode [ 17.449374][ T356] veth1: entered promiscuous mode [ 17.520329][ T150] br1: port 3(veth1) entered blocking state [ 17.520795][ T150] br1: port 3(veth1) entered forwarding state [ 17.882600][ T361] br1: port 4(veth2) entered blocking state [ 17.882948][ T361] br1: port 4(veth2) entered disabled state [ 17.883210][ T361] veth2: entered allmulticast mode [ 17.886145][ T361] veth2: entered promiscuous mode [ 17.968223][ T37] br1: port 4(veth2) entered blocking state [ 17.968517][ T37] br1: port 4(veth2) entered forwarding state [ 21.079936][ T411] br2: port 1(w1) entered blocking state [ 21.080239][ T411] br2: port 1(w1) entered disabled state [ 21.080491][ T411] w1: entered allmulticast mode [ 21.082356][ T411] w1: entered promiscuous mode [ 21.685735][ T419] br2: port 2(vx10) entered blocking state [ 21.686070][ T419] br2: port 2(vx10) entered disabled state [ 21.686342][ T419] vx10: entered allmulticast mode [ 21.688236][ T419] vx10: entered promiscuous mode [ 21.688794][ T419] br2: port 2(vx10) entered blocking state [ 21.689068][ T419] br2: port 2(vx10) entered forwarding state [ 22.242645][ T426] br2: port 3(vx20) entered blocking state [ 22.242919][ T426] br2: port 3(vx20) entered disabled state [ 22.243184][ T426] vx20: entered allmulticast mode [ 22.245054][ T426] vx20: entered promiscuous mode [ 22.245919][ T426] br2: port 3(vx20) entered blocking state [ 22.246160][ T426] br2: port 3(vx20) entered forwarding state [ 22.876907][ T44] br2: port 1(w1) entered blocking state [ 22.877327][ T44] br2: port 1(w1) entered forwarding state [ 25.307478][ T468] br2: port 1(w1) entered blocking state [ 25.307764][ T468] br2: port 1(w1) entered disabled state [ 25.308294][ T468] w1: entered allmulticast mode [ 25.310226][ T468] w1: entered promiscuous mode [ 25.950596][ T476] br2: port 2(vx10) entered blocking state [ 25.951156][ T476] br2: port 2(vx10) entered disabled state [ 25.951469][ T476] vx10: entered allmulticast mode [ 25.953726][ T476] vx10: entered promiscuous mode [ 25.954360][ T476] br2: port 2(vx10) entered blocking state [ 25.954865][ T476] br2: port 2(vx10) entered forwarding state [ 26.536211][ T483] br2: port 3(vx20) entered blocking state [ 26.536527][ T483] br2: port 3(vx20) entered disabled state [ 26.536823][ T483] vx20: entered allmulticast mode [ 26.538712][ T483] vx20: entered promiscuous mode [ 26.547724][ T483] br2: port 3(vx20) entered blocking state [ 26.548027][ T483] br2: port 3(vx20) entered forwarding state [ 27.167943][ T150] br2: port 1(w1) entered blocking state [ 27.168265][ T150] br2: port 1(w1) entered forwarding state [ 38.079853][ T570] GACT probability NOT on [ 165.748679][ T1371] veth3: entered promiscuous mode [ 215.565636][ T2298] veth3: left promiscuous mode [ 216.025184][ T2304] veth3: entered promiscuous mode [ 264.610553][ T3231] veth3: left promiscuous mode [ 265.554690][ T3249] vx20: left allmulticast mode [ 265.554938][ T3249] vx20: left promiscuous mode [ 265.555280][ T3249] br1: port 2(vx20) entered disabled state [ 265.632895][ T3250] vx10: left allmulticast mode [ 265.633203][ T3250] vx10: left promiscuous mode [ 265.633604][ T3250] br1: port 1(vx10) entered disabled state [ 271.030320][ T3255] br1: port 1(vx10) entered blocking state [ 271.030636][ T3255] br1: port 1(vx10) entered disabled state [ 271.030918][ T3255] vx10: entered allmulticast mode [ 271.032850][ T3255] vx10: entered promiscuous mode [ 271.033348][ T3255] br1: port 1(vx10) entered blocking state [ 271.033599][ T3255] br1: port 1(vx10) entered forwarding state [ 271.175717][ T3257] br1: port 2(vx20) entered blocking state [ 271.175999][ T3257] br1: port 2(vx20) entered disabled state [ 271.176279][ T3257] vx20: entered allmulticast mode [ 271.178224][ T3257] vx20: entered promiscuous mode [ 271.178737][ T3257] br1: port 2(vx20) entered blocking state [ 271.178980][ T3257] br1: port 2(vx20) entered forwarding state [ 404.758217][ T4086] veth3: entered promiscuous mode [ 452.869158][ T5013] veth3: left promiscuous mode [ 453.329496][ T5019] veth3: entered promiscuous mode [ 502.721376][ T5946] veth3: left promiscuous mode [ 830.809472][ T11] vx20: left allmulticast mode [ 830.809823][ T11] vx20: left promiscuous mode [ 830.810251][ T11] br2: port 3(vx20) entered disabled state [ 830.813973][ T11] vx10: left allmulticast mode [ 830.814198][ T11] vx10: left promiscuous mode [ 830.814556][ T11] br2: port 2(vx10) entered disabled state [ 830.816437][ T11] w1: left allmulticast mode [ 830.817174][ T11] w1: left promiscuous mode [ 830.817507][ T11] br2: port 1(w1) entered disabled state [ 831.385044][ T11] vx20: left allmulticast mode [ 831.385290][ T11] vx20: left promiscuous mode [ 831.385638][ T11] br2: port 3(vx20) entered disabled state [ 831.387293][ T11] vx10: left allmulticast mode [ 831.387543][ T11] vx10: left promiscuous mode [ 831.387865][ T11] br2: port 2(vx10) entered disabled state [ 831.390172][ T11] w1: left allmulticast mode [ 831.390370][ T11] w1: left promiscuous mode [ 831.390930][ T11] br2: port 1(w1) entered disabled state [ 833.933672][T10513] br1: port 4(veth2) entered disabled state [ 834.020998][T10515] veth2: left allmulticast mode [ 834.021307][T10515] veth2: left promiscuous mode [ 834.021684][T10515] br1: port 4(veth2) entered disabled state [ 834.460739][T10519] br1: port 3(veth1) entered disabled state [ 834.539735][T10520] veth1: left allmulticast mode [ 834.539984][T10520] veth1: left promiscuous mode [ 834.540307][T10520] br1: port 3(veth1) entered disabled state [ 834.730626][T10522] vx20: left allmulticast mode [ 834.730906][T10522] vx20: left promiscuous mode [ 834.731265][T10522] br1: port 2(vx20) entered disabled state [ 835.149646][T10526] vx10: left allmulticast mode [ 835.149908][T10526] vx10: left promiscuous mode [ 835.150237][T10526] br1: port 1(vx10) entered disabled state [ 836.177797][T10536] [ 836.177997][T10536] ============================================ [ 836.178221][T10536] WARNING: possible recursive locking detected [ 836.178422][T10536] 6.13.0-rc5-virtme #1 Not tainted [ 836.178590][T10536] -------------------------------------------- [ 836.178790][T10536] ip/10536 is trying to acquire lock: [ 836.178947][T10536] ffffffff9c7d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 836.179285][T10536] [ 836.179285][T10536] but task is already holding lock: [ 836.179536][T10536] ffffffff9c7d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 836.179882][T10536] [ 836.179882][T10536] other info that might help us debug this: [ 836.180133][T10536] Possible unsafe locking scenario: [ 836.180133][T10536] [ 836.180372][T10536] CPU0 [ 836.180493][T10536] ---- [ 836.180611][T10536] lock((netdev_chain).rwsem); [ 836.180770][T10536] lock((netdev_chain).rwsem); [ 836.180941][T10536] [ 836.180941][T10536] *** DEADLOCK *** [ 836.180941][T10536] [ 836.181189][T10536] May be due to missing lock nesting notation [ 836.181189][T10536] [ 836.181431][T10536] 2 locks held by ip/10536: [ 836.181592][T10536] #0: ffffffff9c7e7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 836.181904][T10536] #1: ffffffff9c7d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 836.182241][T10536] [ 836.182241][T10536] stack backtrace: [ 836.182446][T10536] CPU: 2 UID: 0 PID: 10536 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 836.182688][T10536] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 836.182900][T10536] Call Trace: [ 836.183038][T10536] [ 836.183125][T10536] dump_stack_lvl+0x82/0xd0 [ 836.183295][T10536] print_deadlock_bug+0x40a/0x650 [ 836.183474][T10536] validate_chain+0x5bf/0xae0 [ 836.183633][T10536] ? __pfx_validate_chain+0x10/0x10 [ 836.183804][T10536] ? hlock_class+0x4e/0x130 [ 836.183976][T10536] ? mark_lock+0x38/0x3e0 [ 836.184109][T10536] __lock_acquire+0xb9a/0x1680 [ 836.184281][T10536] lock_acquire.part.0+0xeb/0x330 [ 836.184435][T10536] ? blocking_notifier_call_chain+0x50/0x90 [ 836.184651][T10536] ? netconsole_process_cleanups_core+0x2e/0x3d0 [ 836.184861][T10536] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 836.185028][T10536] ? trace_lock_acquire+0x14c/0x1f0 [ 836.185197][T10536] ? lock_acquire+0x32/0xc0 [ 836.185359][T10536] ? blocking_notifier_call_chain+0x50/0x90 [ 836.185563][T10536] down_read+0x9f/0x340 [ 836.185682][T10536] ? blocking_notifier_call_chain+0x50/0x90 [ 836.185886][T10536] ? __pfx_down_read+0x10/0x10 [ 836.186050][T10536] ? __lock_release+0x103/0x460 [ 836.186214][T10536] ? netconsole_netdev_event+0x2c1/0x620 [ 836.186376][T10536] ? __pfx___lock_release+0x10/0x10 [ 836.186536][T10536] blocking_notifier_call_chain+0x50/0x90 [ 836.186702][T10536] __netdev_upper_dev_unlink+0xd9/0x220 [ 836.186867][T10536] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 836.187071][T10536] ? __mutex_unlock_slowpath+0x14a/0x3d0 [ 836.187240][T10536] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 836.187440][T10536] ? rtnl_is_locked+0x15/0x20 [ 836.187601][T10536] netdev_upper_dev_unlink+0x71/0xa0 [ 836.187764][T10536] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 836.187979][T10536] ? netconsole_netdev_event+0x2c1/0x620 [ 836.188140][T10536] ? mutex_is_locked+0x1c/0x60 [ 836.188300][T10536] ? rtnl_is_locked+0x15/0x20 [ 836.188456][T10536] ? netdev_master_upper_dev_get+0x13/0x120 [ 836.188652][T10536] vrf_device_event+0x78/0xe0 [vrf] [ 836.188818][T10536] notifier_call_chain+0xcd/0x150 [ 836.188981][T10536] blocking_notifier_call_chain+0x66/0x90 [ 836.189167][T10536] unregister_netdevice_many_notify+0xa7b/0x1580 [ 836.189389][T10536] ? hlock_class+0x4e/0x130 [ 836.189555][T10536] ? mark_lock+0x38/0x3e0 [ 836.189684][T10536] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 836.189882][T10536] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 836.190087][T10536] ? kfree+0xf3/0x330 [ 836.190225][T10536] ? vlan_vid_del+0x310/0x5e0 [ 836.190397][T10536] ? vlan_vid_del+0x310/0x5e0 [ 836.190566][T10536] rtnl_dellink+0x348/0xa40 [ 836.190733][T10536] ? __pfx_rtnl_dellink+0x10/0x10 [ 836.190915][T10536] ? find_held_lock+0x2c/0x110 [ 836.191079][T10536] ? __lock_release+0x103/0x460 [ 836.191236][T10536] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 836.191406][T10536] ? __pfx___lock_release+0x10/0x10 [ 836.191571][T10536] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 836.191750][T10536] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 836.191909][T10536] ? __pfx_rtnl_dellink+0x10/0x10 [ 836.192081][T10536] rtnetlink_rcv_msg+0x712/0xc10 [ 836.192238][T10536] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 836.192414][T10536] ? hlock_class+0x4e/0x130 [ 836.192574][T10536] ? mark_lock+0x38/0x3e0 [ 836.192695][T10536] ? __lock_acquire+0xb9a/0x1680 [ 836.192866][T10536] netlink_rcv_skb+0x130/0x360 [ 836.193025][T10536] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 836.193205][T10536] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 836.193380][T10536] ? netlink_deliver_tap+0x13e/0x340 [ 836.193560][T10536] ? netlink_deliver_tap+0xc3/0x340 [ 836.193737][T10536] netlink_unicast+0x44b/0x710 [ 836.193911][T10536] ? __pfx_netlink_unicast+0x10/0x10 [ 836.194082][T10536] ? find_held_lock+0x2c/0x110 [ 836.194256][T10536] netlink_sendmsg+0x723/0xbe0 [ 836.194416][T10536] ? __pfx_netlink_sendmsg+0x10/0x10 [ 836.194576][T10536] ____sys_sendmsg+0x7ac/0xa10 [ 836.194737][T10536] ? __pfx_____sys_sendmsg+0x10/0x10 [ 836.194899][T10536] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 836.195098][T10536] ___sys_sendmsg+0xee/0x170 [ 836.195258][T10536] ? __pfx____sys_sendmsg+0x10/0x10 [ 836.195417][T10536] ? kasan_save_stack+0x34/0x50 [ 836.195573][T10536] ? kasan_save_stack+0x24/0x50 [ 836.195727][T10536] ? __kasan_record_aux_stack+0x8e/0xa0 [ 836.195887][T10536] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 836.196091][T10536] ? __x64_sys_close+0x7c/0xd0 [ 836.196254][T10536] ? do_syscall_64+0xc1/0x1d0 [ 836.196412][T10536] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 836.196615][T10536] ? __lock_acquire+0xb9a/0x1680 [ 836.196784][T10536] ? find_held_lock+0x2c/0x110 [ 836.196944][T10536] ? __lock_release+0x103/0x460 [ 836.197101][T10536] ? __virt_addr_valid+0x22b/0x430 [ 836.197259][T10536] ? __pfx___lock_release+0x10/0x10 [ 836.197418][T10536] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 836.197584][T10536] __sys_sendmsg+0x109/0x1a0 [ 836.197747][T10536] ? __pfx___sys_sendmsg+0x10/0x10 [ 836.197906][T10536] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 836.198108][T10536] do_syscall_64+0xc1/0x1d0 [ 836.198265][T10536] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 836.198458][T10536] RIP: 0033:0x7fcb9521e9a7 [ 836.198626][T10536] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 836.199193][T10536] RSP: 002b:00007ffdd19b2fe8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 836.199437][T10536] RAX: ffffffffffffffda RBX: 00007ffdd19b3710 RCX: 00007fcb9521e9a7 [ 836.199677][T10536] RDX: 0000000000000000 RSI: 00007ffdd19b3050 RDI: 0000000000000005 [ 836.199923][T10536] RBP: 0000000000000002 R08: 0000000000000003 R09: 0000000000000078 [ 836.200170][T10536] R10: 00007fcb950dcef8 R11: 0000000000000246 R12: 0000000000000002 [ 836.200405][T10536] R13: 0000000067792f81 R14: 0000000000498600 R15: 0000000000000000 [ 836.200642][T10536]