[ 12.112554][ T248] IPVS: Registered protocols (TCP) [ 12.113876][ T248] IPVS: Connection hash table configured (size=4096, memory=32Kbytes) [ 12.115205][ T248] IPVS: ipvs loaded. [ 14.338258][ T273] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. [ 14.528001][ T275] br0: port 1(veth01) entered blocking state [ 14.528536][ T275] br0: port 1(veth01) entered disabled state [ 14.528925][ T275] veth01: entered allmulticast mode [ 14.530988][ T275] veth01: entered promiscuous mode [ 14.546940][ T275] ip (275) used greatest stack depth: 24040 bytes left [ 14.716117][ T276] br0: port 2(veth02) entered blocking state [ 14.716432][ T276] br0: port 2(veth02) entered disabled state [ 14.717049][ T276] veth02: entered allmulticast mode [ 14.718975][ T276] veth02: entered promiscuous mode [ 15.240676][ T56] br0: port 1(veth01) entered blocking state [ 15.241085][ T56] br0: port 1(veth01) entered forwarding state [ 16.299505][ T40] br0: port 2(veth02) entered blocking state [ 16.299832][ T40] br0: port 2(veth02) entered forwarding state [ 18.146791][ T293] IPVS: [rr] scheduler registered. [ 18.161190][ T76] IPVS: starting estimator thread 0... [ 18.226189][ T294] IPVS: using max 43 ests per chain, 103200 per kthread [ 19.986185][ T11] veth02: left allmulticast mode [ 19.986531][ T11] veth02: left promiscuous mode [ 19.986894][ T11] br0: port 2(veth02) entered disabled state [ 19.989820][ T11] veth01: left allmulticast mode [ 19.990022][ T11] veth01: left promiscuous mode [ 19.990354][ T11] br0: port 1(veth01) entered disabled state [ 20.466227][ T11] IPVS: stop unused estimator thread 0... [ 23.664936][ T355] br0: port 1(veth01) entered blocking state [ 23.665306][ T355] br0: port 1(veth01) entered disabled state [ 23.665573][ T355] veth01: entered allmulticast mode [ 23.667726][ T355] veth01: entered promiscuous mode [ 23.856066][ T356] br0: port 2(veth02) entered blocking state [ 23.856654][ T356] br0: port 2(veth02) entered disabled state [ 23.856936][ T356] veth02: entered allmulticast mode [ 23.858921][ T356] veth02: entered promiscuous mode [ 24.368888][ T37] br0: port 1(veth01) entered blocking state [ 24.369199][ T37] br0: port 1(veth01) entered forwarding state [ 25.432907][ T40] br0: port 2(veth02) entered blocking state [ 25.433203][ T40] br0: port 2(veth02) entered forwarding state [ 27.197196][ T9] IPVS: starting estimator thread 0... [ 27.261249][ T372] IPVS: using max 43 ests per chain, 103200 per kthread [ 27.694536][ T375] br0: port 2(veth02) entered disabled state [ 27.850872][ T375] veth02 (unregistering): left allmulticast mode [ 27.851324][ T375] veth02 (unregistering): left promiscuous mode [ 27.851585][ T375] br0: port 2(veth02) entered disabled state [ 27.852886][ T375] [ 27.852989][ T375] ============================================ [ 27.853178][ T375] WARNING: possible recursive locking detected [ 27.853370][ T375] 6.13.0-rc5-virtme #1 Not tainted [ 27.853522][ T375] -------------------------------------------- [ 27.853708][ T375] ip/375 is trying to acquire lock: [ 27.853861][ T375] ffffffff96bdfc30 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 27.854185][ T375] [ 27.854185][ T375] but task is already holding lock: [ 27.854408][ T375] ffffffff96bdfc30 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 27.854715][ T375] [ 27.854715][ T375] other info that might help us debug this: [ 27.854943][ T375] Possible unsafe locking scenario: [ 27.854943][ T375] [ 27.855169][ T375] CPU0 [ 27.855283][ T375] ---- [ 27.855397][ T375] lock((netdev_chain).rwsem); [ 27.855551][ T375] lock((netdev_chain).rwsem); [ 27.855704][ T375] [ 27.855704][ T375] *** DEADLOCK *** [ 27.855704][ T375] [ 27.855938][ T375] May be due to missing lock nesting notation [ 27.855938][ T375] [ 27.856162][ T375] 2 locks held by ip/375: [ 27.856288][ T375] #0: ffffffff96bf1088 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_dellink+0x228/0xa40 [ 27.856573][ T375] #1: ffffffff96bdfc30 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 27.856880][ T375] [ 27.856880][ T375] stack backtrace: [ 27.857071][ T375] CPU: 2 UID: 0 PID: 375 Comm: ip Not tainted 6.13.0-rc5-virtme #1 [ 27.857307][ T375] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 27.857498][ T375] Call Trace: [ 27.857619][ T375] [ 27.857699][ T375] dump_stack_lvl+0x82/0xd0 [ 27.857860][ T375] print_deadlock_bug+0x40a/0x650 [ 27.858022][ T375] validate_chain+0x5bf/0xae0 [ 27.858179][ T375] ? __pfx_validate_chain+0x10/0x10 [ 27.858336][ T375] ? hlock_class+0x4e/0x130 [ 27.858497][ T375] ? mark_lock+0x38/0x3e0 [ 27.858614][ T375] __lock_acquire+0xb9a/0x1680 [ 27.858770][ T375] lock_acquire.part.0+0xeb/0x330 [ 27.858924][ T375] ? blocking_notifier_call_chain+0x50/0x90 [ 27.859112][ T375] ? find_held_lock+0x2c/0x110 [ 27.859267][ T375] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 27.859419][ T375] ? trace_lock_acquire+0x14c/0x1f0 [ 27.859582][ T375] ? lock_acquire+0x32/0xc0 [ 27.859750][ T375] ? blocking_notifier_call_chain+0x50/0x90 [ 27.859941][ T375] down_read+0x9f/0x340 [ 27.860058][ T375] ? blocking_notifier_call_chain+0x50/0x90 [ 27.860249][ T375] ? __pfx_down_read+0x10/0x10 [ 27.860401][ T375] ? lockdep_hardirqs_on_prepare+0x12b/0x410 [ 27.860592][ T375] blocking_notifier_call_chain+0x50/0x90 [ 27.860751][ T375] __netdev_upper_dev_unlink+0xd9/0x220 [ 27.860915][ T375] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 27.861104][ T375] ? find_held_lock+0x2c/0x110 [ 27.861259][ T375] ? del_nbp+0x377/0xbe0 [bridge] [ 27.861450][ T375] ? hlock_class+0x4e/0x130 [ 27.861603][ T375] netdev_upper_dev_unlink+0x71/0xa0 [ 27.861760][ T375] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 27.861953][ T375] ? mutex_spin_on_owner+0x282/0x2b0 [ 27.862109][ T375] ? rtnl_is_locked+0x15/0x20 [ 27.862264][ T375] del_nbp+0x602/0xbe0 [bridge] [ 27.862447][ T375] br_del_if+0xa3/0x1f0 [bridge] [ 27.862629][ T375] br_device_event+0x53d/0x800 [bridge] [ 27.862811][ T375] ? __pfx_br_device_event+0x10/0x10 [bridge] [ 27.863026][ T375] ? ip_vs_dst_event+0x3d/0x770 [ip_vs] [ 27.863200][ T375] ? packet_notifier+0x3b0/0x810 [ 27.863355][ T375] notifier_call_chain+0xcd/0x150 [ 27.863509][ T375] blocking_notifier_call_chain+0x66/0x90 [ 27.863675][ T375] unregister_netdevice_many_notify+0xa7b/0x1580 [ 27.863874][ T375] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 27.864064][ T375] ? rtnl_dellink+0x228/0xa40 [ 27.864217][ T375] ? sock_def_readable+0x11e/0x300 [ 27.864378][ T375] ? rtnl_is_locked+0x15/0x20 [ 27.864531][ T375] ? unregister_netdevice_queue+0x70/0x410 [ 27.864720][ T375] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 27.864919][ T375] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 27.865110][ T375] rtnl_dellink+0x348/0xa40 [ 27.865267][ T375] ? __pfx_rtnl_dellink+0x10/0x10 [ 27.865434][ T375] ? find_held_lock+0x2c/0x110 [ 27.865591][ T375] ? __lock_release+0x103/0x460 [ 27.865761][ T375] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 27.865917][ T375] ? __pfx___lock_release+0x10/0x10 [ 27.866072][ T375] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 27.866227][ T375] ? rtnetlink_rcv_msg+0x6ef/0xc10 [ 27.866379][ T375] ? __pfx_rtnl_dellink+0x10/0x10 [ 27.866532][ T375] rtnetlink_rcv_msg+0x712/0xc10 [ 27.866690][ T375] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 27.866846][ T375] ? hlock_class+0x4e/0x130 [ 27.866999][ T375] ? mark_lock+0x38/0x3e0 [ 27.867116][ T375] ? __lock_acquire+0xb9a/0x1680 [ 27.867270][ T375] netlink_rcv_skb+0x130/0x360 [ 27.867428][ T375] ? __pfx_rtnetlink_rcv_msg+0x10/0x10 [ 27.867581][ T375] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 27.867748][ T375] ? netlink_deliver_tap+0x13e/0x340 [ 27.867912][ T375] ? netlink_deliver_tap+0xc3/0x340 [ 27.868067][ T375] netlink_unicast+0x44b/0x710 [ 27.868220][ T375] ? __pfx_netlink_unicast+0x10/0x10 [ 27.868371][ T375] ? find_held_lock+0x2c/0x110 [ 27.868530][ T375] ? __lock_release+0x103/0x460 [ 27.868686][ T375] netlink_sendmsg+0x723/0xbe0 [ 27.868846][ T375] ? __pfx_netlink_sendmsg+0x10/0x10 [ 27.869013][ T375] ? _copy_from_user+0x55/0xa0 [ 27.869174][ T375] ____sys_sendmsg+0x7ac/0xa10 [ 27.869330][ T375] ? __pfx_____sys_sendmsg+0x10/0x10 [ 27.869483][ T375] ? __pfx_copy_msghdr_from_user+0x10/0x10 [ 27.869682][ T375] ___sys_sendmsg+0xee/0x170 [ 27.869843][ T375] ? __pfx____sys_sendmsg+0x10/0x10 [ 27.869998][ T375] ? kasan_save_stack+0x34/0x50 [ 27.870161][ T375] ? kasan_save_stack+0x24/0x50 [ 27.870315][ T375] ? __kasan_record_aux_stack+0x8e/0xa0 [ 27.870470][ T375] ? __call_rcu_common.constprop.0+0xa1/0x4b0 [ 27.870662][ T375] ? __x64_sys_close+0x7c/0xd0 [ 27.870819][ T375] ? do_syscall_64+0xc1/0x1d0 [ 27.870977][ T375] ? entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 27.871167][ T375] ? __lock_acquire+0xb9a/0x1680 [ 27.871335][ T375] ? find_held_lock+0x2c/0x110 [ 27.871488][ T375] ? __lock_release+0x103/0x460 [ 27.871642][ T375] ? __virt_addr_valid+0x22b/0x430 [ 27.871806][ T375] ? __pfx___lock_release+0x10/0x10 [ 27.871959][ T375] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 27.872113][ T375] __sys_sendmsg+0x109/0x1a0 [ 27.872267][ T375] ? __pfx___sys_sendmsg+0x10/0x10 [ 27.872423][ T375] ? __pfx_slab_free_after_rcu_debug+0x10/0x10 [ 27.872620][ T375] do_syscall_64+0xc1/0x1d0 [ 27.872775][ T375] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 27.872965][ T375] RIP: 0033:0x7f188555d9a7 [ 27.873129][ T375] Code: 0a 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b9 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10 [ 27.873681][ T375] RSP: 002b:00007ffc28a35808 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 27.873920][ T375] RAX: ffffffffffffffda RBX: 00007ffc28a35f30 RCX: 00007f188555d9a7 [ 27.874153][ T375] RDX: 0000000000000000 RSI: 00007ffc28a35870 RDI: 0000000000000005 [ 27.874382][ T375] RBP: 0000000000000001 R08: 0000000000000003 R09: 0000000000000078 [ 27.874618][ T375] R10: 00007f188541bef8 R11: 0000000000000246 R12: 0000000000000001 [ 27.874848][ T375] R13: 00000000677958ef R14: 0000000000498600 R15: 0000000000000000 [ 27.875085][ T375] [ 27.908002][ T375] ip (375) used greatest stack depth: 23576 bytes left [ 28.607471][ T11] veth01: left allmulticast mode [ 28.607699][ T11] veth01: left promiscuous mode [ 28.607990][ T11] br0: port 1(veth01) entered disabled state [ 29.031009][ T11] IPVS: stop unused estimator thread 0... [ 31.241850][ T427] br0: port 1(veth01) entered blocking state [ 31.242116][ T427] br0: port 1(veth01) entered disabled state [ 31.242335][ T427] veth01: entered allmulticast mode [ 31.243796][ T427] veth01: entered promiscuous mode [ 31.362881][ T428] br0: port 2(veth02) entered blocking state [ 31.363258][ T428] br0: port 2(veth02) entered disabled state [ 31.363486][ T428] veth02: entered allmulticast mode [ 31.364946][ T428] veth02: entered promiscuous mode [ 31.744758][ T56] br0: port 1(veth01) entered blocking state [ 31.745128][ T56] br0: port 1(veth01) entered forwarding state [ 32.520171][ T40] br0: port 2(veth02) entered blocking state [ 32.520401][ T40] br0: port 2(veth02) entered forwarding state [ 34.032757][ T442] ipip: IPv4 and MPLS over IPv4 tunneling driver [ 34.607299][ T76] IPVS: starting estimator thread 0... [ 34.670148][ T451] IPVS: using max 92 ests per chain, 220800 per kthread [ 36.416351][ T11] veth02: left allmulticast mode [ 36.416538][ T11] veth02: left promiscuous mode [ 36.416832][ T11] br0: port 2(veth02) entered disabled state [ 36.418083][ T11] veth01: left allmulticast mode [ 36.418267][ T11] veth01: left promiscuous mode [ 36.418515][ T11] br0: port 1(veth01) entered disabled state [ 36.818734][ T11] IPVS: stop unused estimator thread 0...