[ 11.881818][ T250] ip (250) used greatest stack depth: 23440 bytes left [ 15.105429][ T300] 8021q: 802.1Q VLAN Support v1.8 [ 18.986525][ T346] br1: port 1(vx10) entered blocking state [ 18.987021][ T346] br1: port 1(vx10) entered disabled state [ 18.987741][ T346] vx10: entered allmulticast mode [ 18.989912][ T346] vx10: entered promiscuous mode [ 18.990943][ T346] br1: port 1(vx10) entered blocking state [ 18.991592][ T346] br1: port 1(vx10) entered forwarding state [ 19.390701][ T351] br1: port 2(vx20) entered blocking state [ 19.391015][ T351] br1: port 2(vx20) entered disabled state [ 19.391323][ T351] vx20: entered allmulticast mode [ 19.394027][ T351] vx20: entered promiscuous mode [ 19.394947][ T351] br1: port 2(vx20) entered blocking state [ 19.395224][ T351] br1: port 2(vx20) entered forwarding state [ 19.581336][ T353] br1: port 3(veth1) entered blocking state [ 19.581731][ T353] br1: port 3(veth1) entered disabled state [ 19.582050][ T353] veth1: entered allmulticast mode [ 19.584041][ T353] veth1: entered promiscuous mode [ 19.689745][ T150] br1: port 3(veth1) entered blocking state [ 19.690242][ T150] br1: port 3(veth1) entered forwarding state [ 19.969301][ T357] br1: port 4(veth2) entered blocking state [ 19.970837][ T357] br1: port 4(veth2) entered disabled state [ 19.971202][ T357] veth2: entered allmulticast mode [ 19.973277][ T357] veth2: entered promiscuous mode [ 20.085179][ T37] br1: port 4(veth2) entered blocking state [ 20.085704][ T37] br1: port 4(veth2) entered forwarding state [ 24.038300][ T407] br2: port 1(w1) entered blocking state [ 24.039261][ T407] br2: port 1(w1) entered disabled state [ 24.039636][ T407] w1: entered allmulticast mode [ 24.042899][ T407] w1: entered promiscuous mode [ 24.741183][ T415] br2: port 2(vx10) entered blocking state [ 24.741565][ T415] br2: port 2(vx10) entered disabled state [ 24.741873][ T415] vx10: entered allmulticast mode [ 24.744204][ T415] vx10: entered promiscuous mode [ 24.745010][ T415] br2: port 2(vx10) entered blocking state [ 24.745393][ T415] br2: port 2(vx10) entered forwarding state [ 25.401457][ T422] br2: port 3(vx20) entered blocking state [ 25.401837][ T422] br2: port 3(vx20) entered disabled state [ 25.402162][ T422] vx20: entered allmulticast mode [ 25.404177][ T422] vx20: entered promiscuous mode [ 25.404810][ T422] br2: port 3(vx20) entered blocking state [ 25.405103][ T422] br2: port 3(vx20) entered forwarding state [ 26.245744][ T69] br2: port 1(w1) entered blocking state [ 26.246691][ T69] br2: port 1(w1) entered forwarding state [ 28.869610][ T462] br2: port 1(w1) entered blocking state [ 28.869984][ T462] br2: port 1(w1) entered disabled state [ 28.870365][ T462] w1: entered allmulticast mode [ 28.874120][ T462] w1: entered promiscuous mode [ 29.631528][ T470] br2: port 2(vx10) entered blocking state [ 29.631952][ T470] br2: port 2(vx10) entered disabled state [ 29.632399][ T470] vx10: entered allmulticast mode [ 29.636119][ T470] vx10: entered promiscuous mode [ 29.637101][ T470] br2: port 2(vx10) entered blocking state [ 29.637523][ T470] br2: port 2(vx10) entered forwarding state [ 30.308538][ T477] br2: port 3(vx20) entered blocking state [ 30.308902][ T477] br2: port 3(vx20) entered disabled state [ 30.309223][ T477] vx20: entered allmulticast mode [ 30.311236][ T477] vx20: entered promiscuous mode [ 30.312492][ T477] br2: port 3(vx20) entered blocking state [ 30.312799][ T477] br2: port 3(vx20) entered forwarding state [ 31.081636][ T37] br2: port 1(w1) entered blocking state [ 31.082041][ T37] br2: port 1(w1) entered forwarding state [ 47.466160][ T593] GACT probability NOT on [ 83.130007][ T1214] veth3: entered promiscuous mode [ 137.984209][ T2141] veth3: left promiscuous mode [ 138.562784][ T2147] veth3: entered promiscuous mode [ 194.504184][ T3074] veth3: left promiscuous mode [ 195.564518][ T3092] vx20: left allmulticast mode [ 195.564779][ T3092] vx20: left promiscuous mode [ 195.565125][ T3092] br1: port 2(vx20) entered disabled state [ 195.682978][ T3093] vx10: left allmulticast mode [ 195.683302][ T3093] vx10: left promiscuous mode [ 195.683876][ T3093] br1: port 1(vx10) entered disabled state [ 201.137126][ T3098] br1: port 1(vx10) entered blocking state [ 201.137645][ T3098] br1: port 1(vx10) entered disabled state [ 201.137979][ T3098] vx10: entered allmulticast mode [ 201.140080][ T3098] vx10: entered promiscuous mode [ 201.140711][ T3098] br1: port 1(vx10) entered blocking state [ 201.141030][ T3098] br1: port 1(vx10) entered forwarding state [ 201.336496][ T3100] br1: port 2(vx20) entered blocking state [ 201.336883][ T3100] br1: port 2(vx20) entered disabled state [ 201.337234][ T3100] vx20: entered allmulticast mode [ 201.339363][ T3100] vx20: entered promiscuous mode [ 201.340299][ T3100] br1: port 2(vx20) entered blocking state [ 201.340621][ T3100] br1: port 2(vx20) entered forwarding state [ 252.098264][ T3778] veth3: entered promiscuous mode [ 307.070194][ T4705] veth3: left promiscuous mode [ 307.595408][ T4711] veth3: entered promiscuous mode [ 362.221792][ T5638] veth3: left promiscuous mode [ 363.724205][ T5661] vx20: left allmulticast mode [ 363.724537][ T5661] vx20: left promiscuous mode [ 363.724910][ T5661] br1: port 2(vx20) entered disabled state [ 363.833900][ T5662] vx10: left allmulticast mode [ 363.834311][ T5662] vx10: left promiscuous mode [ 363.834805][ T5662] br1: port 1(vx10) entered disabled state [ 369.350703][ T5667] br1: port 1(vx10) entered blocking state [ 369.351145][ T5667] br1: port 1(vx10) entered disabled state [ 369.351579][ T5667] vx10: entered allmulticast mode [ 369.354963][ T5667] vx10: entered promiscuous mode [ 369.355697][ T5667] br1: port 1(vx10) entered blocking state [ 369.355963][ T5667] br1: port 1(vx10) entered forwarding state [ 369.581591][ T5669] br1: port 2(vx20) entered blocking state [ 369.582070][ T5669] br1: port 2(vx20) entered disabled state [ 369.582529][ T5669] vx20: entered allmulticast mode [ 369.586316][ T5669] vx20: entered promiscuous mode [ 369.587257][ T5669] br1: port 2(vx20) entered blocking state [ 369.587730][ T5669] br1: port 2(vx20) entered forwarding state [ 702.260327][ T8945] vx20: left allmulticast mode [ 702.260726][ T8945] vx20: left promiscuous mode [ 702.261080][ T8945] br1: port 2(vx20) entered disabled state [ 702.347070][ T8946] vx10: left allmulticast mode [ 702.347334][ T8946] vx10: left promiscuous mode [ 702.347687][ T8946] br1: port 1(vx10) entered disabled state [ 707.767259][ T8951] br1: port 1(vx10) entered blocking state [ 707.767588][ T8951] br1: port 1(vx10) entered disabled state [ 707.767868][ T8951] vx10: entered allmulticast mode [ 707.769854][ T8951] vx10: entered promiscuous mode [ 707.770357][ T8951] br1: port 1(vx10) entered blocking state [ 707.770614][ T8951] br1: port 1(vx10) entered forwarding state [ 707.982051][ T8953] br1: port 2(vx20) entered blocking state [ 707.982520][ T8953] br1: port 2(vx20) entered disabled state [ 707.982934][ T8953] vx20: entered allmulticast mode [ 707.986529][ T8953] vx20: entered promiscuous mode [ 707.987397][ T8953] br1: port 2(vx20) entered blocking state [ 707.987825][ T8953] br1: port 2(vx20) entered forwarding state [ 817.359617][ T11] vx20: left allmulticast mode [ 817.359983][ T11] vx20: left promiscuous mode [ 817.360414][ T11] br2: port 3(vx20) entered disabled state [ 817.380321][ T11] vx10: left allmulticast mode [ 817.380738][ T11] vx10: left promiscuous mode [ 817.381211][ T11] br2: port 2(vx10) entered disabled state [ 817.385148][ T11] w1: left allmulticast mode [ 817.385628][ T11] w1: left promiscuous mode [ 817.386104][ T11] br2: port 1(w1) entered disabled state [ 817.873899][ T11] vx20: left allmulticast mode [ 817.874221][ T11] vx20: left promiscuous mode [ 817.875608][ T11] br2: port 3(vx20) entered disabled state [ 817.879132][ T11] vx10: left allmulticast mode [ 817.879603][ T11] vx10: left promiscuous mode [ 817.880034][ T11] br2: port 2(vx10) entered disabled state [ 817.884667][ T11] w1: left allmulticast mode [ 817.884986][ T11] w1: left promiscuous mode [ 817.885507][ T11] br2: port 1(w1) entered disabled state [ 820.527338][T10857] br1: port 4(veth2) entered disabled state [ 820.622721][T10858] veth2: left allmulticast mode [ 820.622981][T10858] veth2: left promiscuous mode [ 820.623311][T10858] br1: port 4(veth2) entered disabled state [ 820.892579][T10861] br1: port 3(veth1) entered disabled state [ 820.991211][T10862] veth1: left allmulticast mode [ 820.991494][T10862] veth1: left promiscuous mode [ 820.991831][T10862] br1: port 3(veth1) entered disabled state [ 821.159719][T10864] vx20: left allmulticast mode [ 821.159987][T10864] vx20: left promiscuous mode [ 821.160327][T10864] br1: port 2(vx20) entered disabled state [ 821.556717][T10868] vx10: left allmulticast mode [ 821.556991][T10868] vx10: left promiscuous mode [ 821.557330][T10868] br1: port 1(vx10) entered disabled state [ 822.522074][T10877] [ 822.522241][T10877] ============================================ [ 822.522434][T10877] WARNING: possible recursive locking detected [ 822.522617][T10877] 6.13.0-rc5-virtme #1 Not tainted [ 822.522775][T10877] -------------------------------------------- [ 822.522974][T10877] ip/10877 is trying to acquire lock: [ 822.523123][T10877] ffffffff997d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 822.523433][T10877] [ 822.523433][T10877] but task is already holding lock: [ 822.523649][T10877] ffffffff997d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 822.523964][T10877] [ 822.523964][T10877] other info that might help us debug this: [ 822.524190][T10877] Possible unsafe locking scenario: [ 822.524190][T10877] [ 822.524406][T10877] CPU0 [ 822.524521][T10877] ---- [ 822.524631][T10877] lock((netdev_chain).rwsem); [ 822.524790][T10877] lock((netdev_chain).rwsem); [ 822.524950][T10877] [ 822.524950][T10877] *** DEADLOCK *** [ 822.524950][T10877] [ 822.525169][T10877] May be due to missing lock nesting notation [ 822.525169][T10877] [ 822.525386][T10877] 2 locks held by ip/10877: [ 822.525536][T10877] #0: ffffffff997e7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 822.525815][T10877] #1: ffffffff997d6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 822.526132][T10877] [ 822.526132][T10877] stack backtrace: [ 822.526321][T10877] CPU: 3 UID: 0 PID: 10877 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 822.526549][T10877] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 822.526741][T10877] Call Trace: [ 822.526861][T10877] [ 822.526956][T10877] dump_stack_lvl+0x82/0xd0 [ 822.527114][T10877] print_deadlock_bug+0x40a/0x650 [ 822.527266][T10877] validate_chain+0x5bf/0xae0 [ 822.527416][T10877] ? __pfx_validate_chain+0x10/0x10 [ 822.527570][T10877] ? hlock_class+0x4e/0x130 [ 822.527722][T10877] ? mark_lock+0x38/0x3e0 [ 822.527837][T10877] __lock_acquire+0xb9a/0x1680 [ 822.527998][T10877] lock_acquire.part.0+0xeb/0x330 [ 822.528147][T10877] ? blocking_notifier_call_chain+0x50/0x90 [ 822.528332][T10877] ? netconsole_process_cleanups_core+0x2e/0x3d0 [ 822.528530][T10877] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 822.528686][T10877] ? trace_lock_acquire+0x14c/0x1f0 [ 822.528839][T10877] ? lock_acquire+0x32/0xc0 [ 822.528999][T10877] ? blocking_notifier_call_chain+0x50/0x90 [ 822.529185][T10877] down_read+0x9f/0x340 [ 822.529304][T10877] ? blocking_notifier_call_chain+0x50/0x90 [ 822.529492][T10877] ? __pfx_down_read+0x10/0x10 [ 822.529641][T10877] ? __lock_release+0x103/0x460 [ 822.529844][T10877] ? netconsole_netdev_event+0x2c1/0x620 [ 822.530090][T10877] ? __pfx___lock_release+0x10/0x10 [ 822.530348][T10877] blocking_notifier_call_chain+0x50/0x90 [ 822.530602][T10877] __netdev_upper_dev_unlink+0xd9/0x220 [ 822.530853][T10877] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 822.531164][T10877] ? __mutex_unlock_slowpath+0x14a/0x3d0 [ 822.531414][T10877] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 822.531714][T10877] ? rtnl_is_locked+0x15/0x20 [ 822.531966][T10877] netdev_upper_dev_unlink+0x71/0xa0 [ 822.532219][T10877] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 822.532516][T10877] ? netconsole_netdev_event+0x2c1/0x620 [ 822.532758][T10877] ? mutex_is_locked+0x1c/0x60 [ 822.532999][T10877] ? rtnl_is_locked+0x15/0x20 [ 822.533236][T10877] ? netdev_master_upper_dev_get+0x13/0x120 [ 822.533533][T10877] vrf_device_event+0x78/0xe0 [vrf] [ 822.533783][T10877] notifier_call_chain+0xcd/0x150 [ 822.534024][T10877] blocking_notifier_call_chain+0x66/0x90 [ 822.534263][T10877] unregister_netdevice_many_notify+0xa7b/0x1580 [ 822.534563][T10877] ? hlock_class+0x4e/0x130 [ 822.534804][T10877] ? mark_lock+0x38/0x3e0 [ 822.534989][T10877] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 822.535294][T10877] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 822.535596][T10877] ? kfree+0xf3/0x330 [ 822.535781][T10877] ? vlan_vid_del+0x310/0x5e0 [ 822.536033][T10877] ? vlan_vid_del+0x310/0x5e0 [ 822.536282][T10877] rtnl_dellink+0x348/0xa40 [ 822.536528][T10877] ? __pfx_rtnl_dellink+0x10/0x10 [ 822.536796][T10877] ? find_held_lock+0x2c/0x110 [ 822.537041][T10877] ? __lock_release+0x103/0x460 [ 822.537291][T10877] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 822.537536][T10877] ? __pfx___lock_release+0x10/0x10 [ 822.537775][T10877] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 822.538027][T10877] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 822.538273][T10877] ? __pfx_rtnl_dellink+0x10/0x10 [ 822.538516][T10877] rtnetlink_rcv_msg+0x712/0xc10 [ 822.538760][T10877] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 822.538999][T10877] ? hlock_class+0x4e/0x130 [ 822.539239][T10877] ? mark_lock+0x38/0x3e0 [ 822.539425][T10877] ? __lock_acquire+0xb9a/0x1680 [ 822.539668][T10877] netlink_rcv_skb+0x130/0x360 [ 822.539907][T10877] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 822.540162][T10877] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 822.540428][T10877] ? netlink_deliver_tap+0x13e/0x340 [ 822.540670][T10877] ? netlink_deliver_tap+0xc3/0x340 [ 822.540909][T10877] netlink_unicast+0x44b/0x710 [ 822.541152][T10877] ? __pfx_netlink_unicast+0x10/0x10 [ 822.541392][T10877] ? find_held_lock+0x2c/0x110 [ 822.541634][T10877] netlink_sendmsg+0x723/0xbe0 [ 822.541880][T10877] ? __pfx_netlink_sendmsg+0x10/0x10 [ 822.542123][T10877] ____sys_sendmsg+0x7ac/0xa10 [ 822.542363][T10877] ? __pfx_____sys_sendmsg+0x10/0x10 [ 822.542601][T10877] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 822.542907][T10877] ___sys_sendmsg+0xee/0x170 [ 822.543147][T10877] ? __pfx____sys_sendmsg+0x10/0x10 [ 822.543386][T10877] ? kasan_save_stack+0x34/0x50 [ 822.543632][T10877] ? kasan_save_stack+0x24/0x50 [ 822.543869][T10877] ? __kasan_record_aux_stack+0x8e/0xa0 [ 822.544110][T10877] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 822.544405][T10877] ? __x64_sys_close+0x7c/0xd0 [ 822.544643][T10877] ? do_syscall_64+0xc1/0x1d0 [ 822.544883][T10877] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 822.545184][T10877] ? __lock_acquire+0xb9a/0x1680 [ 822.545434][T10877] ? find_held_lock+0x2c/0x110 [ 822.545675][T10877] ? __lock_release+0x103/0x460 [ 822.545913][T10877] ? __virt_addr_valid+0x22b/0x430 [ 822.546156][T10877] ? __pfx___lock_release+0x10/0x10 [ 822.546392][T10877] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 822.546632][T10877] __sys_sendmsg+0x109/0x1a0 [ 822.546870][T10877] ? __pfx___sys_sendmsg+0x10/0x10 [ 822.547115][T10877] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 822.547420][T10877] do_syscall_64+0xc1/0x1d0 [ 822.547656][T10877] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 822.547951][T10877] RIP: 0033:0x7f8467e719a7 [ 822.548204][T10877] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 822.549060][T10877] RSP: 002b:00007ffc74d0fa48 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 822.549420][T10877] RAX: ffffffffffffffda RBX: 00007ffc74d10170 RCX: 00007f8467e719a7 [ 822.549777][T10877] RDX: 0000000000000000 RSI: 00007ffc74d0fab0 RDI: 0000000000000005 [ 822.550134][T10877] RBP: 0000000000000002 R08: 0000000000000003 R09: 0000000000000078 [ 822.550511][T10877] R10: 00007f8467d2fef8 R11: 0000000000000246 R12: 0000000000000002 [ 822.550871][T10877] R13: 0000000067792908 R14: 0000000000498600 R15: 0000000000000000 [ 822.551238][T10877]