[ 10.948004][ T248] ip (248) used greatest stack depth: 23680 bytes left [ 13.788725][ T301] 8021q: 802.1Q VLAN Support v1.8 [ 17.119362][ T347] br1: port 1(vx10) entered blocking state [ 17.119792][ T347] br1: port 1(vx10) entered disabled state [ 17.120156][ T347] vx10: entered allmulticast mode [ 17.122165][ T347] vx10: entered promiscuous mode [ 17.123099][ T347] br1: port 1(vx10) entered blocking state [ 17.123452][ T347] br1: port 1(vx10) entered forwarding state [ 17.476793][ T352] br1: port 2(vx20) entered blocking state [ 17.477278][ T352] br1: port 2(vx20) entered disabled state [ 17.477762][ T352] vx20: entered allmulticast mode [ 17.481075][ T352] vx20: entered promiscuous mode [ 17.482317][ T352] br1: port 2(vx20) entered blocking state [ 17.482958][ T352] br1: port 2(vx20) entered forwarding state [ 17.661946][ T354] br1: port 3(veth1) entered blocking state [ 17.662300][ T354] br1: port 3(veth1) entered disabled state [ 17.662735][ T354] veth1: entered allmulticast mode [ 17.665673][ T354] veth1: entered promiscuous mode [ 17.751269][ T37] br1: port 3(veth1) entered blocking state [ 17.751733][ T37] br1: port 3(veth1) entered forwarding state [ 17.971668][ T358] br1: port 4(veth2) entered blocking state [ 17.972184][ T358] br1: port 4(veth2) entered disabled state [ 17.972483][ T358] veth2: entered allmulticast mode [ 17.976492][ T358] veth2: entered promiscuous mode [ 18.052962][ T37] br1: port 4(veth2) entered blocking state [ 18.053419][ T37] br1: port 4(veth2) entered forwarding state [ 21.382890][ T407] br2: port 1(w1) entered blocking state [ 21.383168][ T407] br2: port 1(w1) entered disabled state [ 21.383443][ T407] w1: entered allmulticast mode [ 21.391486][ T407] w1: entered promiscuous mode [ 22.053658][ T415] br2: port 2(vx10) entered blocking state [ 22.053959][ T415] br2: port 2(vx10) entered disabled state [ 22.054234][ T415] vx10: entered allmulticast mode [ 22.056245][ T415] vx10: entered promiscuous mode [ 22.056861][ T415] br2: port 2(vx10) entered blocking state [ 22.057128][ T415] br2: port 2(vx10) entered forwarding state [ 22.668161][ T422] br2: port 3(vx20) entered blocking state [ 22.668720][ T422] br2: port 3(vx20) entered disabled state [ 22.669031][ T422] vx20: entered allmulticast mode [ 22.670969][ T422] vx20: entered promiscuous mode [ 22.671522][ T422] br2: port 3(vx20) entered blocking state [ 22.671779][ T422] br2: port 3(vx20) entered forwarding state [ 23.356004][ T39] br2: port 1(w1) entered blocking state [ 23.356546][ T39] br2: port 1(w1) entered forwarding state [ 25.852876][ T462] br2: port 1(w1) entered blocking state [ 25.853201][ T462] br2: port 1(w1) entered disabled state [ 25.853512][ T462] w1: entered allmulticast mode [ 25.855474][ T462] w1: entered promiscuous mode [ 26.579837][ T470] br2: port 2(vx10) entered blocking state [ 26.580299][ T470] br2: port 2(vx10) entered disabled state [ 26.580773][ T470] vx10: entered allmulticast mode [ 26.583643][ T470] vx10: entered promiscuous mode [ 26.584285][ T470] br2: port 2(vx10) entered blocking state [ 26.584586][ T470] br2: port 2(vx10) entered forwarding state [ 27.236094][ T477] br2: port 3(vx20) entered blocking state [ 27.236544][ T477] br2: port 3(vx20) entered disabled state [ 27.236983][ T477] vx20: entered allmulticast mode [ 27.240702][ T477] vx20: entered promiscuous mode [ 27.241265][ T477] br2: port 3(vx20) entered blocking state [ 27.241540][ T477] br2: port 3(vx20) entered forwarding state [ 27.989286][ T39] br2: port 1(w1) entered blocking state [ 27.989610][ T39] br2: port 1(w1) entered forwarding state [ 43.619218][ T592] GACT probability NOT on [ 76.640550][ T1213] veth3: entered promiscuous mode [ 124.864626][ T2140] veth3: left promiscuous mode [ 125.303115][ T2146] veth3: entered promiscuous mode [ 175.922716][ T3073] veth3: left promiscuous mode [ 176.843034][ T3091] vx20: left allmulticast mode [ 176.843329][ T3091] vx20: left promiscuous mode [ 176.844802][ T3091] br1: port 2(vx20) entered disabled state [ 176.937367][ T3092] vx10: left allmulticast mode [ 176.937631][ T3092] vx10: left promiscuous mode [ 176.937967][ T3092] br1: port 1(vx10) entered disabled state [ 182.375837][ T3097] br1: port 1(vx10) entered blocking state [ 182.376178][ T3097] br1: port 1(vx10) entered disabled state [ 182.376505][ T3097] vx10: entered allmulticast mode [ 182.379064][ T3097] vx10: entered promiscuous mode [ 182.379624][ T3097] br1: port 1(vx10) entered blocking state [ 182.379898][ T3097] br1: port 1(vx10) entered forwarding state [ 182.562768][ T3099] br1: port 2(vx20) entered blocking state [ 182.563270][ T3099] br1: port 2(vx20) entered disabled state [ 182.563758][ T3099] vx20: entered allmulticast mode [ 182.567581][ T3099] vx20: entered promiscuous mode [ 182.568499][ T3099] br1: port 2(vx20) entered blocking state [ 182.568936][ T3099] br1: port 2(vx20) entered forwarding state [ 228.841290][ T3777] veth3: entered promiscuous mode [ 280.689578][ T4704] veth3: left promiscuous mode [ 281.158338][ T4710] veth3: entered promiscuous mode [ 332.247104][ T5637] veth3: left promiscuous mode [ 333.574403][ T5660] vx20: left allmulticast mode [ 333.574667][ T5660] vx20: left promiscuous mode [ 333.574998][ T5660] br1: port 2(vx20) entered disabled state [ 333.654389][ T5661] vx10: left allmulticast mode [ 333.654640][ T5661] vx10: left promiscuous mode [ 333.654950][ T5661] br1: port 1(vx10) entered disabled state [ 339.026062][ T5666] br1: port 1(vx10) entered blocking state [ 339.026346][ T5666] br1: port 1(vx10) entered disabled state [ 339.027963][ T5666] vx10: entered allmulticast mode [ 339.029958][ T5666] vx10: entered promiscuous mode [ 339.037242][ T5666] br1: port 1(vx10) entered blocking state [ 339.037546][ T5666] br1: port 1(vx10) entered forwarding state [ 339.214189][ T5668] br1: port 2(vx20) entered blocking state [ 339.214563][ T5668] br1: port 2(vx20) entered disabled state [ 339.214851][ T5668] vx20: entered allmulticast mode [ 339.217216][ T5668] vx20: entered promiscuous mode [ 339.217837][ T5668] br1: port 2(vx20) entered blocking state [ 339.218117][ T5668] br1: port 2(vx20) entered forwarding state [ 663.813834][ T8943] vx20: left allmulticast mode [ 663.814226][ T8943] vx20: left promiscuous mode [ 663.814754][ T8943] br1: port 2(vx20) entered disabled state [ 663.939321][ T8944] vx10: left allmulticast mode [ 663.939608][ T8944] vx10: left promiscuous mode [ 663.939958][ T8944] br1: port 1(vx10) entered disabled state [ 669.387644][ T8949] br1: port 1(vx10) entered blocking state [ 669.387963][ T8949] br1: port 1(vx10) entered disabled state [ 669.388246][ T8949] vx10: entered allmulticast mode [ 669.390332][ T8949] vx10: entered promiscuous mode [ 669.390890][ T8949] br1: port 1(vx10) entered blocking state [ 669.391146][ T8949] br1: port 1(vx10) entered forwarding state [ 669.652410][ T8951] br1: port 2(vx20) entered blocking state [ 669.652854][ T8951] br1: port 2(vx20) entered disabled state [ 669.653292][ T8951] vx20: entered allmulticast mode [ 669.657536][ T8951] vx20: entered promiscuous mode [ 669.658210][ T8951] br1: port 2(vx20) entered blocking state [ 669.658554][ T8951] br1: port 2(vx20) entered forwarding state [ 793.181997][ T11] vx20: left allmulticast mode [ 793.182392][ T11] vx20: left promiscuous mode [ 793.182770][ T11] br2: port 3(vx20) entered disabled state [ 793.186435][ T11] vx10: left allmulticast mode [ 793.186739][ T11] vx10: left promiscuous mode [ 793.187062][ T11] br2: port 2(vx10) entered disabled state [ 793.190518][ T11] w1: left allmulticast mode [ 793.190748][ T11] w1: left promiscuous mode [ 793.191086][ T11] br2: port 1(w1) entered disabled state [ 793.743421][ T11] vx20: left allmulticast mode [ 793.743663][ T11] vx20: left promiscuous mode [ 793.744005][ T11] br2: port 3(vx20) entered disabled state [ 793.750729][ T11] vx10: left allmulticast mode [ 793.750969][ T11] vx10: left promiscuous mode [ 793.751288][ T11] br2: port 2(vx10) entered disabled state [ 793.757255][ T11] w1: left allmulticast mode [ 793.757569][ T11] w1: left promiscuous mode [ 793.758029][ T11] br2: port 1(w1) entered disabled state [ 794.268459][T10834] ip (10834) used greatest stack depth: 23352 bytes left [ 796.271534][T10855] br1: port 4(veth2) entered disabled state [ 796.352736][T10856] veth2: left allmulticast mode [ 796.353015][T10856] veth2: left promiscuous mode [ 796.353345][T10856] br1: port 4(veth2) entered disabled state [ 796.640295][T10859] br1: port 3(veth1) entered disabled state [ 796.726469][T10860] veth1: left allmulticast mode [ 796.726769][T10860] veth1: left promiscuous mode [ 796.727118][T10860] br1: port 3(veth1) entered disabled state [ 796.898786][T10862] vx20: left allmulticast mode [ 796.899065][T10862] vx20: left promiscuous mode [ 796.900628][T10862] br1: port 2(vx20) entered disabled state [ 797.311076][T10866] vx10: left allmulticast mode [ 797.311314][T10866] vx10: left promiscuous mode [ 797.311640][T10866] br1: port 1(vx10) entered disabled state [ 798.230529][T10875] [ 798.230678][T10875] ============================================ [ 798.230948][T10875] WARNING: possible recursive locking detected [ 798.231225][T10875] 6.13.0-rc5-virtme #1 Not tainted [ 798.231467][T10875] -------------------------------------------- [ 798.231747][T10875] ip/10875 is trying to acquire lock: [ 798.231974][T10875] ffffffff88bd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 798.232447][T10875] [ 798.232447][T10875] but task is already holding lock: [ 798.232785][T10875] ffffffff88bd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 798.233270][T10875] [ 798.233270][T10875] other info that might help us debug this: [ 798.233602][T10875] Possible unsafe locking scenario: [ 798.233602][T10875] [ 798.233938][T10875] CPU0 [ 798.234110][T10875] ---- [ 798.234279][T10875] lock((netdev_chain).rwsem); [ 798.234506][T10875] lock((netdev_chain).rwsem); [ 798.234732][T10875] [ 798.234732][T10875] *** DEADLOCK *** [ 798.234732][T10875] [ 798.235087][T10875] May be due to missing lock nesting notation [ 798.235087][T10875] [ 798.235422][T10875] 2 locks held by ip/10875: [ 798.235649][T10875] #0: ffffffff88be7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 798.236063][T10875] #1: ffffffff88bd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 798.236523][T10875] [ 798.236523][T10875] stack backtrace: [ 798.236811][T10875] CPU: 3 UID: 0 PID: 10875 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 798.237159][T10875] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 798.237452][T10875] Call Trace: [ 798.237629][T10875] [ 798.237751][T10875] dump_stack_lvl+0x82/0xd0 [ 798.237987][T10875] print_deadlock_bug+0x40a/0x650 [ 798.238222][T10875] validate_chain+0x5bf/0xae0 [ 798.238469][T10875] ? __pfx_validate_chain+0x10/0x10 [ 798.238701][T10875] ? hlock_class+0x4e/0x130 [ 798.238933][T10875] ? mark_lock+0x38/0x3e0 [ 798.239111][T10875] __lock_acquire+0xb9a/0x1680 [ 798.239347][T10875] lock_acquire.part.0+0xeb/0x330 [ 798.239573][T10875] ? blocking_notifier_call_chain+0x50/0x90 [ 798.239859][T10875] ? netconsole_process_cleanups_core+0x2e/0x3d0 [ 798.240148][T10875] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 798.240381][T10875] ? trace_lock_acquire+0x14c/0x1f0 [ 798.240613][T10875] ? lock_acquire+0x32/0xc0 [ 798.240845][T10875] ? blocking_notifier_call_chain+0x50/0x90 [ 798.241136][T10875] down_read+0x9f/0x340 [ 798.241317][T10875] ? blocking_notifier_call_chain+0x50/0x90 [ 798.241607][T10875] ? __pfx_down_read+0x10/0x10 [ 798.241836][T10875] ? __lock_release+0x103/0x460 [ 798.242062][T10875] ? netconsole_netdev_event+0x2c1/0x620 [ 798.242297][T10875] ? __pfx___lock_release+0x10/0x10 [ 798.242527][T10875] blocking_notifier_call_chain+0x50/0x90 [ 798.242756][T10875] __netdev_upper_dev_unlink+0xd9/0x220 [ 798.242988][T10875] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 798.243270][T10875] ? __mutex_unlock_slowpath+0x14a/0x3d0 [ 798.243499][T10875] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 798.243787][T10875] ? rtnl_is_locked+0x15/0x20 [ 798.244023][T10875] netdev_upper_dev_unlink+0x71/0xa0 [ 798.244257][T10875] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 798.244580][T10875] ? netconsole_netdev_event+0x2c1/0x620 [ 798.244827][T10875] ? mutex_is_locked+0x1c/0x60 [ 798.245069][T10875] ? rtnl_is_locked+0x15/0x20 [ 798.245299][T10875] ? netdev_master_upper_dev_get+0x13/0x120 [ 798.245578][T10875] vrf_device_event+0x78/0xe0 [vrf] [ 798.245811][T10875] notifier_call_chain+0xcd/0x150 [ 798.246044][T10875] blocking_notifier_call_chain+0x66/0x90 [ 798.246284][T10875] unregister_netdevice_many_notify+0xa7b/0x1580 [ 798.246568][T10875] ? hlock_class+0x4e/0x130 [ 798.246793][T10875] ? mark_lock+0x38/0x3e0 [ 798.246972][T10875] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 798.247254][T10875] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 798.247543][T10875] ? kfree+0xf3/0x330 [ 798.247718][T10875] ? vlan_vid_del+0x310/0x5e0 [ 798.247959][T10875] ? vlan_vid_del+0x310/0x5e0 [ 798.248190][T10875] rtnl_dellink+0x348/0xa40 [ 798.248424][T10875] ? __pfx_rtnl_dellink+0x10/0x10 [ 798.248680][T10875] ? find_held_lock+0x2c/0x110 [ 798.248919][T10875] ? __lock_release+0x103/0x460 [ 798.249152][T10875] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 798.249419][T10875] ? __pfx___lock_release+0x10/0x10 [ 798.249663][T10875] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 798.249916][T10875] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 798.250160][T10875] ? __pfx_rtnl_dellink+0x10/0x10 [ 798.250401][T10875] rtnetlink_rcv_msg+0x712/0xc10 [ 798.250643][T10875] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 798.250884][T10875] ? hlock_class+0x4e/0x130 [ 798.251125][T10875] ? mark_lock+0x38/0x3e0 [ 798.251312][T10875] ? __lock_acquire+0xb9a/0x1680 [ 798.251563][T10875] netlink_rcv_skb+0x130/0x360 [ 798.251809][T10875] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 798.252051][T10875] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 798.252300][T10875] ? netlink_deliver_tap+0x13e/0x340 [ 798.252542][T10875] ? netlink_deliver_tap+0xc3/0x340 [ 798.252785][T10875] netlink_unicast+0x44b/0x710 [ 798.253025][T10875] ? __pfx_netlink_unicast+0x10/0x10 [ 798.253271][T10875] ? find_held_lock+0x2c/0x110 [ 798.253516][T10875] netlink_sendmsg+0x723/0xbe0 [ 798.253758][T10875] ? __pfx_netlink_sendmsg+0x10/0x10 [ 798.254003][T10875] ____sys_sendmsg+0x7ac/0xa10 [ 798.254246][T10875] ? __pfx_____sys_sendmsg+0x10/0x10 [ 798.254486][T10875] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 798.254801][T10875] ___sys_sendmsg+0xee/0x170 [ 798.255054][T10875] ? __pfx____sys_sendmsg+0x10/0x10 [ 798.255297][T10875] ? kasan_save_stack+0x34/0x50 [ 798.255541][T10875] ? kasan_save_stack+0x24/0x50 [ 798.255781][T10875] ? __kasan_record_aux_stack+0x8e/0xa0 [ 798.256025][T10875] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 798.256328][T10875] ? __x64_sys_close+0x7c/0xd0 [ 798.256568][T10875] ? do_syscall_64+0xc1/0x1d0 [ 798.256809][T10875] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 798.257115][T10875] ? __lock_acquire+0xb9a/0x1680 [ 798.257363][T10875] ? find_held_lock+0x2c/0x110 [ 798.257608][T10875] ? __lock_release+0x103/0x460 [ 798.257851][T10875] ? __virt_addr_valid+0x22b/0x430 [ 798.258094][T10875] ? __pfx___lock_release+0x10/0x10 [ 798.258334][T10875] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 798.258581][T10875] __sys_sendmsg+0x109/0x1a0 [ 798.258832][T10875] ? __pfx___sys_sendmsg+0x10/0x10 [ 798.259072][T10875] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 798.259383][T10875] do_syscall_64+0xc1/0x1d0 [ 798.259623][T10875] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 798.259922][T10875] RIP: 0033:0x7f2bb3d629a7 [ 798.260174][T10875] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 798.261036][T10875] RSP: 002b:00007ffd2e6db778 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 798.261411][T10875] RAX: ffffffffffffffda RBX: 00007ffd2e6dbea0 RCX: 00007f2bb3d629a7 [ 798.261788][T10875] RDX: 0000000000000000 RSI: 00007ffd2e6db7e0 RDI: 0000000000000005 [ 798.262134][T10875] RBP: 0000000000000002 R08: 0000000000000003 R09: 0000000000000078 [ 798.262506][T10875] R10: 00007f2bb3c20ef8 R11: 0000000000000246 R12: 0000000000000002 [ 798.262850][T10875] R13: 0000000067795193 R14: 0000000000498600 R15: 0000000000000000 [ 798.263199][T10875]