[ 14.653448][ T301] 8021q: 802.1Q VLAN Support v1.8 [ 18.397184][ T345] br1: port 1(vx10) entered blocking state [ 18.398719][ T345] br1: port 1(vx10) entered disabled state [ 18.399185][ T345] vx10: entered allmulticast mode [ 18.401698][ T345] vx10: entered promiscuous mode [ 18.402865][ T345] br1: port 1(vx10) entered blocking state [ 18.403254][ T345] br1: port 1(vx10) entered forwarding state [ 18.413769][ T345] ip (345) used greatest stack depth: 23736 bytes left [ 18.797614][ T350] br1: port 2(vx20) entered blocking state [ 18.797964][ T350] br1: port 2(vx20) entered disabled state [ 18.798283][ T350] vx20: entered allmulticast mode [ 18.800360][ T350] vx20: entered promiscuous mode [ 18.801429][ T350] br1: port 2(vx20) entered blocking state [ 18.801708][ T350] br1: port 2(vx20) entered forwarding state [ 18.975944][ T352] br1: port 3(veth1) entered blocking state [ 18.976303][ T352] br1: port 3(veth1) entered disabled state [ 18.977092][ T352] veth1: entered allmulticast mode [ 18.979060][ T352] veth1: entered promiscuous mode [ 19.053579][ T37] br1: port 3(veth1) entered blocking state [ 19.054173][ T37] br1: port 3(veth1) entered forwarding state [ 19.347686][ T356] br1: port 4(veth2) entered blocking state [ 19.348072][ T356] br1: port 4(veth2) entered disabled state [ 19.348394][ T356] veth2: entered allmulticast mode [ 19.350714][ T356] veth2: entered promiscuous mode [ 19.457833][ T37] br1: port 4(veth2) entered blocking state [ 19.458299][ T37] br1: port 4(veth2) entered forwarding state [ 23.144278][ T406] br2: port 1(w1) entered blocking state [ 23.145005][ T406] br2: port 1(w1) entered disabled state [ 23.145278][ T406] w1: entered allmulticast mode [ 23.147455][ T406] w1: entered promiscuous mode [ 23.860215][ T414] br2: port 2(vx10) entered blocking state [ 23.861188][ T414] br2: port 2(vx10) entered disabled state [ 23.861714][ T414] vx10: entered allmulticast mode [ 23.865448][ T414] vx10: entered promiscuous mode [ 23.866457][ T414] br2: port 2(vx10) entered blocking state [ 23.866896][ T414] br2: port 2(vx10) entered forwarding state [ 24.570637][ T421] br2: port 3(vx20) entered blocking state [ 24.570925][ T421] br2: port 3(vx20) entered disabled state [ 24.571212][ T421] vx20: entered allmulticast mode [ 24.573176][ T421] vx20: entered promiscuous mode [ 24.573715][ T421] br2: port 3(vx20) entered blocking state [ 24.573981][ T421] br2: port 3(vx20) entered forwarding state [ 25.299866][ T68] br2: port 1(w1) entered blocking state [ 25.300405][ T68] br2: port 1(w1) entered forwarding state [ 27.827702][ T461] br2: port 1(w1) entered blocking state [ 27.828008][ T461] br2: port 1(w1) entered disabled state [ 27.828281][ T461] w1: entered allmulticast mode [ 27.830898][ T461] w1: entered promiscuous mode [ 28.574985][ T469] br2: port 2(vx10) entered blocking state [ 28.575343][ T469] br2: port 2(vx10) entered disabled state [ 28.575644][ T469] vx10: entered allmulticast mode [ 28.577702][ T469] vx10: entered promiscuous mode [ 28.578509][ T469] br2: port 2(vx10) entered blocking state [ 28.578845][ T469] br2: port 2(vx10) entered forwarding state [ 29.264655][ T476] br2: port 3(vx20) entered blocking state [ 29.264944][ T476] br2: port 3(vx20) entered disabled state [ 29.265222][ T476] vx20: entered allmulticast mode [ 29.267148][ T476] vx20: entered promiscuous mode [ 29.267666][ T476] br2: port 3(vx20) entered blocking state [ 29.267909][ T476] br2: port 3(vx20) entered forwarding state [ 29.978228][ T37] br2: port 1(w1) entered blocking state [ 29.978515][ T37] br2: port 1(w1) entered forwarding state [ 45.959764][ T591] GACT probability NOT on [ 80.296331][ T1213] veth3: entered promiscuous mode [ 132.746773][ T2140] veth3: left promiscuous mode [ 133.228514][ T2146] veth3: entered promiscuous mode [ 188.398372][ T3073] veth3: left promiscuous mode [ 189.401559][ T3091] vx20: left allmulticast mode [ 189.401970][ T3091] vx20: left promiscuous mode [ 189.402495][ T3091] br1: port 2(vx20) entered disabled state [ 189.521780][ T3092] vx10: left allmulticast mode [ 189.522070][ T3092] vx10: left promiscuous mode [ 189.522438][ T3092] br1: port 1(vx10) entered disabled state [ 194.930801][ T3097] br1: port 1(vx10) entered blocking state [ 194.931129][ T3097] br1: port 1(vx10) entered disabled state [ 194.931454][ T3097] vx10: entered allmulticast mode [ 194.934417][ T3097] vx10: entered promiscuous mode [ 194.934989][ T3097] br1: port 1(vx10) entered blocking state [ 194.935268][ T3097] br1: port 1(vx10) entered forwarding state [ 195.140355][ T3099] br1: port 2(vx20) entered blocking state [ 195.140688][ T3099] br1: port 2(vx20) entered disabled state [ 195.141013][ T3099] vx20: entered allmulticast mode [ 195.143289][ T3099] vx20: entered promiscuous mode [ 195.143857][ T3099] br1: port 2(vx20) entered blocking state [ 195.144122][ T3099] br1: port 2(vx20) entered forwarding state [ 246.198214][ T3777] veth3: entered promiscuous mode [ 307.464789][ T4705] veth3: left promiscuous mode [ 308.028033][ T4711] veth3: entered promiscuous mode [ 362.755118][ T5638] veth3: left promiscuous mode [ 364.194178][ T5661] vx20: left allmulticast mode [ 364.194453][ T5661] vx20: left promiscuous mode [ 364.194797][ T5661] br1: port 2(vx20) entered disabled state [ 364.288712][ T5662] vx10: left allmulticast mode [ 364.288953][ T5662] vx10: left promiscuous mode [ 364.289394][ T5662] br1: port 1(vx10) entered disabled state [ 369.733635][ T5667] br1: port 1(vx10) entered blocking state [ 369.733953][ T5667] br1: port 1(vx10) entered disabled state [ 369.734244][ T5667] vx10: entered allmulticast mode [ 369.737198][ T5667] vx10: entered promiscuous mode [ 369.737910][ T5667] br1: port 1(vx10) entered blocking state [ 369.738263][ T5667] br1: port 1(vx10) entered forwarding state [ 369.916781][ T5669] br1: port 2(vx20) entered blocking state [ 369.917079][ T5669] br1: port 2(vx20) entered disabled state [ 369.917420][ T5669] vx20: entered allmulticast mode [ 369.919291][ T5669] vx20: entered promiscuous mode [ 369.928745][ T5669] br1: port 2(vx20) entered blocking state [ 369.929062][ T5669] br1: port 2(vx20) entered forwarding state [ 698.704200][ T8944] vx20: left allmulticast mode [ 698.704453][ T8944] vx20: left promiscuous mode [ 698.704775][ T8944] br1: port 2(vx20) entered disabled state [ 698.788502][ T8945] vx10: left allmulticast mode [ 698.788725][ T8945] vx10: left promiscuous mode [ 698.789058][ T8945] br1: port 1(vx10) entered disabled state [ 704.175585][ T8950] br1: port 1(vx10) entered blocking state [ 704.176096][ T8950] br1: port 1(vx10) entered disabled state [ 704.176561][ T8950] vx10: entered allmulticast mode [ 704.179940][ T8950] vx10: entered promiscuous mode [ 704.181476][ T8950] br1: port 1(vx10) entered blocking state [ 704.181883][ T8950] br1: port 1(vx10) entered forwarding state [ 704.357574][ T8952] br1: port 2(vx20) entered blocking state [ 704.357928][ T8952] br1: port 2(vx20) entered disabled state [ 704.358276][ T8952] vx20: entered allmulticast mode [ 704.361157][ T8952] vx20: entered promiscuous mode [ 704.361814][ T8952] br1: port 2(vx20) entered blocking state [ 704.362132][ T8952] br1: port 2(vx20) entered forwarding state [ 815.837584][ T67] vx20: left allmulticast mode [ 815.838220][ T67] vx20: left promiscuous mode [ 815.838804][ T67] br2: port 3(vx20) entered disabled state [ 815.847596][ T67] vx10: left allmulticast mode [ 815.847900][ T67] vx10: left promiscuous mode [ 815.848394][ T67] br2: port 2(vx10) entered disabled state [ 815.853628][ T67] w1: left allmulticast mode [ 815.853974][ T67] w1: left promiscuous mode [ 815.856014][ T67] br2: port 1(w1) entered disabled state [ 816.432764][ T67] vx20: left allmulticast mode [ 816.433030][ T67] vx20: left promiscuous mode [ 816.433385][ T67] br2: port 3(vx20) entered disabled state [ 816.435544][ T67] vx10: left allmulticast mode [ 816.435867][ T67] vx10: left promiscuous mode [ 816.436360][ T67] br2: port 2(vx10) entered disabled state [ 816.439648][ T67] w1: left allmulticast mode [ 816.439851][ T67] w1: left promiscuous mode [ 816.440150][ T67] br2: port 1(w1) entered disabled state [ 819.110845][T10856] br1: port 4(veth2) entered disabled state [ 819.195458][T10857] veth2: left allmulticast mode [ 819.195747][T10857] veth2: left promiscuous mode [ 819.196077][T10857] br1: port 4(veth2) entered disabled state [ 819.467091][T10860] br1: port 3(veth1) entered disabled state [ 819.545443][T10861] veth1: left allmulticast mode [ 819.545772][T10861] veth1: left promiscuous mode [ 819.546136][T10861] br1: port 3(veth1) entered disabled state [ 819.709110][T10863] vx20: left allmulticast mode [ 819.709430][T10863] vx20: left promiscuous mode [ 819.709787][T10863] br1: port 2(vx20) entered disabled state [ 820.129522][T10867] vx10: left allmulticast mode [ 820.129923][T10867] vx10: left promiscuous mode [ 820.130450][T10867] br1: port 1(vx10) entered disabled state [ 821.061635][T10876] [ 821.061860][T10876] ============================================ [ 821.062134][T10876] WARNING: possible recursive locking detected [ 821.062338][T10876] 6.13.0-rc5-virtme #1 Not tainted [ 821.062498][T10876] -------------------------------------------- [ 821.062717][T10876] ip/10876 is trying to acquire lock: [ 821.062882][T10876] ffffffff8abd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 821.063210][T10876] [ 821.063210][T10876] but task is already holding lock: [ 821.063430][T10876] ffffffff8abd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 821.063746][T10876] [ 821.063746][T10876] other info that might help us debug this: [ 821.063970][T10876] Possible unsafe locking scenario: [ 821.063970][T10876] [ 821.064197][T10876] CPU0 [ 821.064321][T10876] ---- [ 821.064444][T10876] lock((netdev_chain).rwsem); [ 821.064600][T10876] lock((netdev_chain).rwsem); [ 821.064772][T10876] [ 821.064772][T10876] *** DEADLOCK *** [ 821.064772][T10876] [ 821.065025][T10876] May be due to missing lock nesting notation [ 821.065025][T10876] [ 821.065260][T10876] 2 locks held by ip/10876: [ 821.065425][T10876] #0: ffffffff8abe7a48 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 821.065733][T10876] #1: ffffffff8abd6630 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 821.066067][T10876] [ 821.066067][T10876] stack backtrace: [ 821.066269][T10876] CPU: 2 UID: 0 PID: 10876 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 821.066529][T10876] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 821.066766][T10876] Call Trace: [ 821.066962][T10876] [ 821.067090][T10876] dump_stack_lvl+0x82/0xd0 [ 821.067349][T10876] print_deadlock_bug+0x40a/0x650 [ 821.067604][T10876] validate_chain+0x5bf/0xae0 [ 821.067852][T10876] ? __pfx_validate_chain+0x10/0x10 [ 821.068113][T10876] ? hlock_class+0x4e/0x130 [ 821.068328][T10876] ? mark_lock+0x38/0x3e0 [ 821.068453][T10876] __lock_acquire+0xb9a/0x1680 [ 821.068619][T10876] lock_acquire.part.0+0xeb/0x330 [ 821.068793][T10876] ? blocking_notifier_call_chain+0x50/0x90 [ 821.068999][T10876] ? netconsole_process_cleanups_core+0x2e/0x3d0 [ 821.069209][T10876] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 821.069377][T10876] ? trace_lock_acquire+0x14c/0x1f0 [ 821.069544][T10876] ? lock_acquire+0x32/0xc0 [ 821.069722][T10876] ? blocking_notifier_call_chain+0x50/0x90 [ 821.069934][T10876] down_read+0x9f/0x340 [ 821.070067][T10876] ? blocking_notifier_call_chain+0x50/0x90 [ 821.070277][T10876] ? __pfx_down_read+0x10/0x10 [ 821.070453][T10876] ? __lock_release+0x103/0x460 [ 821.070607][T10876] ? netconsole_netdev_event+0x2c1/0x620 [ 821.070768][T10876] ? __pfx___lock_release+0x10/0x10 [ 821.070915][T10876] blocking_notifier_call_chain+0x50/0x90 [ 821.071067][T10876] __netdev_upper_dev_unlink+0xd9/0x220 [ 821.071239][T10876] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 821.071442][T10876] ? __mutex_unlock_slowpath+0x14a/0x3d0 [ 821.071615][T10876] ? __pfx___mutex_unlock_slowpath+0x10/0x10 [ 821.071822][T10876] ? rtnl_is_locked+0x15/0x20 [ 821.071988][T10876] netdev_upper_dev_unlink+0x71/0xa0 [ 821.072149][T10876] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 821.072348][T10876] ? netconsole_netdev_event+0x2c1/0x620 [ 821.072508][T10876] ? mutex_is_locked+0x1c/0x60 [ 821.072656][T10876] ? rtnl_is_locked+0x15/0x20 [ 821.072820][T10876] ? netdev_master_upper_dev_get+0x13/0x120 [ 821.073005][T10876] vrf_device_event+0x78/0xe0 [vrf] [ 821.073157][T10876] notifier_call_chain+0xcd/0x150 [ 821.073309][T10876] blocking_notifier_call_chain+0x66/0x90 [ 821.073457][T10876] unregister_netdevice_many_notify+0xa7b/0x1580 [ 821.073641][T10876] ? hlock_class+0x4e/0x130 [ 821.073801][T10876] ? mark_lock+0x38/0x3e0 [ 821.073918][T10876] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 821.074098][T10876] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 821.074283][T10876] ? kfree+0xf3/0x330 [ 821.074398][T10876] ? vlan_vid_del+0x310/0x5e0 [ 821.074546][T10876] ? vlan_vid_del+0x310/0x5e0 [ 821.074702][T10876] rtnl_dellink+0x348/0xa40 [ 821.074851][T10876] ? __pfx_rtnl_dellink+0x10/0x10 [ 821.075016][T10876] ? find_held_lock+0x2c/0x110 [ 821.075164][T10876] ? __lock_release+0x103/0x460 [ 821.075311][T10876] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 821.075457][T10876] ? __pfx___lock_release+0x10/0x10 [ 821.075604][T10876] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 821.075779][T10876] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 821.075926][T10876] ? __pfx_rtnl_dellink+0x10/0x10 [ 821.076082][T10876] rtnetlink_rcv_msg+0x712/0xc10 [ 821.076229][T10876] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 821.076381][T10876] ? hlock_class+0x4e/0x130 [ 821.076530][T10876] ? mark_lock+0x38/0x3e0 [ 821.076641][T10876] ? __lock_acquire+0xb9a/0x1680 [ 821.076800][T10876] netlink_rcv_skb+0x130/0x360 [ 821.076946][T10876] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 821.077091][T10876] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 821.077248][T10876] ? netlink_deliver_tap+0x13e/0x340 [ 821.077406][T10876] ? netlink_deliver_tap+0xc3/0x340 [ 821.077552][T10876] netlink_unicast+0x44b/0x710 [ 821.077713][T10876] ? __pfx_netlink_unicast+0x10/0x10 [ 821.077859][T10876] ? find_held_lock+0x2c/0x110 [ 821.078008][T10876] netlink_sendmsg+0x723/0xbe0 [ 821.078153][T10876] ? __pfx_netlink_sendmsg+0x10/0x10 [ 821.078329][T10876] ____sys_sendmsg+0x7ac/0xa10 [ 821.078490][T10876] ? __pfx_____sys_sendmsg+0x10/0x10 [ 821.078652][T10876] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 821.078851][T10876] ___sys_sendmsg+0xee/0x170 [ 821.079000][T10876] ? __pfx____sys_sendmsg+0x10/0x10 [ 821.079147][T10876] ? kasan_save_stack+0x34/0x50 [ 821.079297][T10876] ? kasan_save_stack+0x24/0x50 [ 821.079465][T10876] ? __kasan_record_aux_stack+0x8e/0xa0 [ 821.079628][T10876] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 821.079845][T10876] ? __x64_sys_close+0x7c/0xd0 [ 821.080008][T10876] ? do_syscall_64+0xc1/0x1d0 [ 821.080168][T10876] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 821.080371][T10876] ? __lock_acquire+0xb9a/0x1680 [ 821.080532][T10876] ? find_held_lock+0x2c/0x110 [ 821.080702][T10876] ? __lock_release+0x103/0x460 [ 821.080860][T10876] ? __virt_addr_valid+0x22b/0x430 [ 821.081024][T10876] ? __pfx___lock_release+0x10/0x10 [ 821.081193][T10876] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 821.081429][T10876] __sys_sendmsg+0x109/0x1a0 [ 821.081681][T10876] ? __pfx___sys_sendmsg+0x10/0x10 [ 821.081926][T10876] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 821.082246][T10876] do_syscall_64+0xc1/0x1d0 [ 821.082485][T10876] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 821.082789][T10876] RIP: 0033:0x7f696771f9a7 [ 821.083054][T10876] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 821.083917][T10876] RSP: 002b:00007ffc652a7618 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 821.084315][T10876] RAX: ffffffffffffffda RBX: 00007ffc652a7d40 RCX: 00007f696771f9a7 [ 821.084666][T10876] RDX: 0000000000000000 RSI: 00007ffc652a7680 RDI: 0000000000000005 [ 821.085068][T10876] RBP: 0000000000000002 R08: 0000000000000003 R09: 0000000000000078 [ 821.085451][T10876] R10: 00007f69675ddef8 R11: 0000000000000246 R12: 0000000000000002 [ 821.085826][T10876] R13: 000000006778fe69 R14: 0000000000498600 R15: 0000000000000000 [ 821.086210][T10876]