[ 11.204326][ T280] ip (280) used greatest stack depth: 24320 bytes left [ 11.304610][ T281] eth0: renamed from r1h1 [ 11.495517][ T283] eth1: renamed from r1h2 [ 12.608181][ T298] ip (298) used greatest stack depth: 24080 bytes left [ 16.871043][ T66] [ 16.871151][ T66] ============================================ [ 16.871330][ T66] WARNING: possible recursive locking detected [ 16.871497][ T66] 6.13.0-rc5-virtme #1 Not tainted [ 16.871632][ T66] -------------------------------------------- [ 16.871805][ T66] kworker/u16:1/66 is trying to acquire lock: [ 16.871976][ T66] ffffffffa93f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 16.872257][ T66] [ 16.872257][ T66] but task is already holding lock: [ 16.872464][ T66] ffffffffa93f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 16.872738][ T66] [ 16.872738][ T66] other info that might help us debug this: [ 16.872938][ T66] Possible unsafe locking scenario: [ 16.872938][ T66] [ 16.873139][ T66] CPU0 [ 16.873242][ T66] ---- [ 16.873346][ T66] lock((netdev_chain).rwsem); [ 16.873483][ T66] lock((netdev_chain).rwsem); [ 16.873621][ T66] [ 16.873621][ T66] *** DEADLOCK *** [ 16.873621][ T66] [ 16.873819][ T66] May be due to missing lock nesting notation [ 16.873819][ T66] [ 16.874061][ T66] 5 locks held by kworker/u16:1/66: [ 16.874203][ T66] #0: ffff8880010b5948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0x7ec/0x16d0 [ 16.874484][ T66] #1: ffffc90000477da0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xe0b/0x16d0 [ 16.874728][ T66] #2: ffffffffa93ec4d0 (pernet_ops_rwsem){++++}-{4:4}, at: cleanup_net+0xbc/0xba0 [ 16.874977][ T66] #3: ffffffffa9407e88 (rtnl_mutex){+.+.}-{4:4}, at: default_device_exit_batch+0x81/0x2e0 [ 16.875255][ T66] #4: ffffffffa93f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 16.875532][ T66] [ 16.875532][ T66] stack backtrace: [ 16.875715][ T66] CPU: 1 UID: 0 PID: 66 Comm: kworker/u16:1 Not tainted 6.13.0-rc5-virtme #1 [ 16.875948][ T66] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 16.876120][ T66] Workqueue: netns cleanup_net [ 16.876259][ T66] Call Trace: [ 16.876360][ T66] [ 16.876435][ T66] dump_stack_lvl+0x82/0xd0 [ 16.876575][ T66] print_deadlock_bug+0x40a/0x650 [ 16.876731][ T66] validate_chain+0x5bf/0xae0 [ 16.876869][ T66] ? __pfx_validate_chain+0x10/0x10 [ 16.877007][ T66] ? hlock_class+0x4e/0x130 [ 16.877146][ T66] ? mark_lock+0x38/0x3e0 [ 16.877251][ T66] __lock_acquire+0xb9a/0x1680 [ 16.877392][ T66] lock_acquire.part.0+0xeb/0x330 [ 16.877529][ T66] ? blocking_notifier_call_chain+0x50/0x90 [ 16.877702][ T66] ? __lock_acquire+0xb9a/0x1680 [ 16.877840][ T66] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 16.877981][ T66] ? trace_lock_acquire+0x14c/0x1f0 [ 16.878119][ T66] ? lock_acquire+0x32/0xc0 [ 16.878256][ T66] ? blocking_notifier_call_chain+0x50/0x90 [ 16.878434][ T66] down_read+0x9f/0x340 [ 16.878538][ T66] ? blocking_notifier_call_chain+0x50/0x90 [ 16.878706][ T66] ? __pfx_down_read+0x10/0x10 [ 16.878840][ T66] ? __lock_release+0x103/0x460 [ 16.878976][ T66] blocking_notifier_call_chain+0x50/0x90 [ 16.879110][ T66] __netdev_upper_dev_unlink+0xd9/0x220 [ 16.879249][ T66] ? mark_held_locks+0x9e/0xe0 [ 16.879385][ T66] ? __pfx___netdev_upper_dev_unlink+0x10/0x10 [ 16.879549][ T66] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 16.879716][ T66] ? __try_to_del_timer_sync+0xe7/0x140 [ 16.879853][ T66] ? __pfx___try_to_del_timer_sync+0x10/0x10 [ 16.880020][ T66] ? mark_held_locks+0x9e/0xe0 [ 16.880154][ T66] netdev_upper_dev_unlink+0x71/0xa0 [ 16.880290][ T66] ? __pfx_netdev_upper_dev_unlink+0x10/0x10 [ 16.880457][ T66] ? mutex_is_locked+0x1c/0x60 [ 16.880607][ T66] ? rtnl_is_locked+0x15/0x20 [ 16.880744][ T66] ? netdev_master_upper_dev_get+0x13/0x120 [ 16.880915][ T66] vrf_device_event+0x78/0xe0 [ 16.881054][ T66] notifier_call_chain+0xcd/0x150 [ 16.881196][ T66] blocking_notifier_call_chain+0x66/0x90 [ 16.881335][ T66] unregister_netdevice_many_notify+0xa83/0x1580 [ 16.881508][ T66] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 16.881678][ T66] ? default_device_exit_batch+0x81/0x2e0 [ 16.881816][ T66] ? rtnl_is_locked+0x15/0x20 [ 16.881959][ T66] ? unregister_netdevice_queue+0x70/0x410 [ 16.882124][ T66] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 16.882292][ T66] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 16.882481][ T66] default_device_exit_batch+0x241/0x2e0 [ 16.882620][ T66] ? __pfx_default_device_exit_batch+0x10/0x10 [ 16.882791][ T66] ? ops_exit_list+0xb4/0x170 [ 16.882930][ T66] cleanup_net+0x4ef/0xba0 [ 16.883067][ T66] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 16.883205][ T66] ? __pfx_cleanup_net+0x10/0x10 [ 16.883343][ T66] ? trace_lock_acquire+0x14c/0x1f0 [ 16.883480][ T66] ? lock_acquire+0x32/0xc0 [ 16.883616][ T66] ? process_one_work+0xe0b/0x16d0 [ 16.883765][ T66] process_one_work+0xe55/0x16d0 [ 16.883901][ T66] ? __pfx___lock_release+0x10/0x10 [ 16.884035][ T66] ? __pfx_process_one_work+0x10/0x10 [ 16.884194][ T66] ? assign_work+0x16c/0x240 [ 16.884336][ T66] worker_thread+0x58c/0xce0 [ 16.884475][ T66] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 16.884646][ T66] ? __pfx_worker_thread+0x10/0x10 [ 16.884782][ T66] ? __pfx_worker_thread+0x10/0x10 [ 16.884919][ T66] kthread+0x28a/0x350 [ 16.885023][ T66] ? __pfx_kthread+0x10/0x10 [ 16.885160][ T66] ret_from_fork+0x31/0x70 [ 16.885299][ T66] ? __pfx_kthread+0x10/0x10 [ 16.885436][ T66] ret_from_fork_asm+0x1a/0x30 [ 16.885577][ T66] [ 18.938497][ T378] eth0: renamed from r1h1 [ 19.073823][ T380] eth1: renamed from r1h2 [ 25.545396][ T473] eth0: renamed from r1h1 [ 25.683069][ T475] eth1: renamed from r1h2 [ 32.093583][ T570] eth0: renamed from r1h1 [ 32.222619][ T572] eth1: renamed from r1h2 [ 38.696877][ T663] eth0: renamed from r1h1 [ 38.840805][ T665] eth1: renamed from r1h2 [ 46.255453][ T760] eth0: renamed from r1h1 [ 46.386674][ T762] eth1: renamed from r1h2 [ 54.147505][ T866] eth0: renamed from r1h1 [ 54.279712][ T868] eth0: renamed from r2h1 [ 54.426637][ T870] eth1: renamed from r1h2 [ 54.569139][ T872] eth1: renamed from r2h2 [ 54.836307][ T877] br0: port 1(eth0) entered blocking state [ 54.836578][ T877] br0: port 1(eth0) entered disabled state [ 54.836769][ T877] eth0: entered allmulticast mode [ 54.837676][ T877] eth0: entered promiscuous mode [ 54.839228][ T40] br0: port 1(eth0) entered blocking state [ 54.839426][ T40] br0: port 1(eth0) entered forwarding state [ 54.894203][ T878] br0: port 2(eth1) entered blocking state [ 54.894481][ T878] br0: port 2(eth1) entered disabled state [ 54.894885][ T878] eth1: entered allmulticast mode [ 54.895818][ T878] eth1: entered promiscuous mode [ 54.896930][ T39] br0: port 2(eth1) entered blocking state [ 54.897108][ T39] br0: port 2(eth1) entered forwarding state [ 55.270718][ T885] br0: port 1(eth0) entered blocking state [ 55.271015][ T885] br0: port 1(eth0) entered disabled state [ 55.271210][ T885] eth0: entered allmulticast mode [ 55.272531][ T885] eth0: entered promiscuous mode [ 55.333930][ T886] br0: port 2(eth1) entered blocking state [ 55.334193][ T886] br0: port 2(eth1) entered disabled state [ 55.334370][ T886] eth1: entered allmulticast mode [ 55.335271][ T886] eth1: entered promiscuous mode [ 55.336355][ T40] br0: port 2(eth1) entered blocking state [ 55.336551][ T40] br0: port 2(eth1) entered forwarding state [ 56.118140][ T56] br0: port 1(eth0) entered blocking state [ 56.118393][ T56] br0: port 1(eth0) entered forwarding state [ 59.381417][ T66] eth1: left allmulticast mode [ 59.381587][ T66] eth1: left promiscuous mode [ 59.381873][ T66] br0: port 2(eth1) entered disabled state [ 59.383104][ T66] eth0: left allmulticast mode [ 59.383251][ T66] eth0: left promiscuous mode [ 59.383477][ T66] br0: port 1(eth0) entered disabled state [ 59.638274][ T66] eth1: left allmulticast mode [ 59.638458][ T66] eth1: left promiscuous mode [ 59.638726][ T66] br0: port 2(eth1) entered disabled state [ 59.645747][ T66] eth0: left allmulticast mode [ 59.645910][ T66] eth0: left promiscuous mode [ 59.646158][ T66] br0: port 1(eth0) entered disabled state [ 62.243673][ T990] eth0: renamed from r1h1 [ 62.388810][ T992] eth0: renamed from r2h1 [ 62.523465][ T994] eth1: renamed from r1h2 [ 62.660492][ T996] eth1: renamed from r2h2 [ 62.929840][ T1001] br0: port 1(eth0) entered blocking state [ 62.930070][ T1001] br0: port 1(eth0) entered disabled state [ 62.930260][ T1001] eth0: entered allmulticast mode [ 62.931160][ T1001] eth0: entered promiscuous mode [ 62.938481][ T48] br0: port 1(eth0) entered blocking state [ 62.938732][ T48] br0: port 1(eth0) entered forwarding state [ 62.988920][ T1002] br0: port 2(eth1) entered blocking state [ 62.989118][ T1002] br0: port 2(eth1) entered disabled state [ 62.989292][ T1002] eth1: entered allmulticast mode [ 62.990141][ T1002] eth1: entered promiscuous mode [ 62.991576][ T48] br0: port 2(eth1) entered blocking state [ 62.991830][ T48] br0: port 2(eth1) entered forwarding state [ 63.465424][ T1009] br0: port 1(eth0) entered blocking state [ 63.465750][ T1009] br0: port 1(eth0) entered disabled state [ 63.465961][ T1009] eth0: entered allmulticast mode [ 63.467082][ T1009] eth0: entered promiscuous mode [ 63.534043][ T1010] br0: port 2(eth1) entered blocking state [ 63.534352][ T1010] br0: port 2(eth1) entered disabled state [ 63.534570][ T1010] eth1: entered allmulticast mode [ 63.535693][ T1010] eth1: entered promiscuous mode [ 63.536869][ T39] br0: port 2(eth1) entered blocking state [ 63.537103][ T39] br0: port 2(eth1) entered forwarding state [ 64.182127][ T56] br0: port 1(eth0) entered blocking state [ 64.182370][ T56] br0: port 1(eth0) entered forwarding state [ 67.765512][ T66] eth1: left allmulticast mode [ 67.765690][ T66] eth1: left promiscuous mode [ 67.765963][ T66] br0: port 2(eth1) entered disabled state [ 67.767160][ T66] eth0: left allmulticast mode [ 67.767298][ T66] eth0: left promiscuous mode [ 67.767527][ T66] br0: port 1(eth0) entered disabled state [ 68.034761][ T66] eth1: left allmulticast mode [ 68.034956][ T66] eth1: left promiscuous mode [ 68.035192][ T66] br0: port 2(eth1) entered disabled state [ 68.036341][ T66] eth0: left allmulticast mode [ 68.036480][ T66] eth0: left promiscuous mode [ 68.036848][ T66] br0: port 1(eth0) entered disabled state [ 70.112783][ T1102] eth0: renamed from r1h1 [ 70.248001][ T1104] eth1: renamed from r1h2 [ 76.991632][ T1203] eth0: renamed from r1h1 [ 77.140598][ T1205] eth1: renamed from r1h2 [ 83.839079][ T1304] eth0: renamed from r1h1 [ 83.967513][ T1306] eth1: renamed from r1h2 [ 90.768761][ T1402] eth0: renamed from r1h1 [ 90.916738][ T1404] eth1: renamed from r1h2 [ 98.591585][ T1504] eth0: renamed from r1h1 [ 98.718431][ T1506] eth1: renamed from r1h2 [ 106.862791][ T1617] eth0: renamed from r1h1 [ 107.016538][ T1619] eth0: renamed from r2h1 [ 107.141614][ T1621] eth1: renamed from r1h2 [ 107.282150][ T1623] eth1: renamed from r2h2 [ 107.568008][ T1628] br0: port 1(eth0) entered blocking state [ 107.568283][ T1628] br0: port 1(eth0) entered disabled state [ 107.568514][ T1628] eth0: entered allmulticast mode [ 107.569451][ T1628] eth0: entered promiscuous mode [ 107.571192][ T48] br0: port 1(eth0) entered blocking state [ 107.571426][ T48] br0: port 1(eth0) entered forwarding state [ 107.626990][ T1629] br0: port 2(eth1) entered blocking state [ 107.627195][ T1629] br0: port 2(eth1) entered disabled state [ 107.627372][ T1629] eth1: entered allmulticast mode [ 107.628235][ T1629] eth1: entered promiscuous mode [ 107.629661][ T865] br0: port 2(eth1) entered blocking state [ 107.629866][ T865] br0: port 2(eth1) entered forwarding state [ 108.019856][ T1636] br0: port 1(eth0) entered blocking state [ 108.020128][ T1636] br0: port 1(eth0) entered disabled state [ 108.020358][ T1636] eth0: entered allmulticast mode [ 108.021363][ T1636] eth0: entered promiscuous mode [ 108.077106][ T1637] br0: port 2(eth1) entered blocking state [ 108.077343][ T1637] br0: port 2(eth1) entered disabled state [ 108.077535][ T1637] eth1: entered allmulticast mode [ 108.078424][ T1637] eth1: entered promiscuous mode [ 108.079536][ T524] br0: port 2(eth1) entered blocking state [ 108.079728][ T524] br0: port 2(eth1) entered forwarding state [ 108.856328][ T865] br0: port 1(eth0) entered blocking state [ 108.856719][ T865] br0: port 1(eth0) entered forwarding state [ 112.205765][ T66] eth1: left allmulticast mode [ 112.206010][ T66] eth1: left promiscuous mode [ 112.206424][ T66] br0: port 2(eth1) entered disabled state [ 112.208161][ T66] eth0: left allmulticast mode [ 112.208392][ T66] eth0: left promiscuous mode [ 112.208757][ T66] br0: port 1(eth0) entered disabled state [ 112.544392][ T66] eth1: left allmulticast mode [ 112.544700][ T66] eth1: left promiscuous mode [ 112.545059][ T66] br0: port 2(eth1) entered disabled state [ 112.546732][ T66] eth0: left allmulticast mode [ 112.547019][ T66] eth0: left promiscuous mode [ 112.547489][ T66] br0: port 1(eth0) entered disabled state [ 115.248633][ T1741] eth0: renamed from r1h1 [ 115.383474][ T1743] eth0: renamed from r2h1 [ 115.518530][ T1745] eth1: renamed from r1h2 [ 115.656473][ T1747] eth1: renamed from r2h2 [ 115.937617][ T1752] br0: port 1(eth0) entered blocking state [ 115.937894][ T1752] br0: port 1(eth0) entered disabled state [ 115.938076][ T1752] eth0: entered allmulticast mode [ 115.938939][ T1752] eth0: entered promiscuous mode [ 115.940523][ T865] br0: port 1(eth0) entered blocking state [ 115.940712][ T865] br0: port 1(eth0) entered forwarding state [ 115.993070][ T1753] br0: port 2(eth1) entered blocking state [ 115.993283][ T1753] br0: port 2(eth1) entered disabled state [ 115.993475][ T1753] eth1: entered allmulticast mode [ 115.994392][ T1753] eth1: entered promiscuous mode [ 115.995979][ T865] br0: port 2(eth1) entered blocking state [ 115.996219][ T865] br0: port 2(eth1) entered forwarding state [ 116.373500][ T1760] br0: port 1(eth0) entered blocking state [ 116.374419][ T1760] br0: port 1(eth0) entered disabled state [ 116.374629][ T1760] eth0: entered allmulticast mode [ 116.375553][ T1760] eth0: entered promiscuous mode [ 116.437538][ T1761] br0: port 2(eth1) entered blocking state [ 116.437796][ T1761] br0: port 2(eth1) entered disabled state [ 116.437989][ T1761] eth1: entered allmulticast mode [ 116.438902][ T1761] eth1: entered promiscuous mode [ 116.440052][ T524] br0: port 2(eth1) entered blocking state [ 116.440238][ T524] br0: port 2(eth1) entered forwarding state [ 117.238698][ T56] br0: port 1(eth0) entered blocking state [ 117.238997][ T56] br0: port 1(eth0) entered forwarding state [ 120.491798][ T66] eth1: left allmulticast mode [ 120.492026][ T66] eth1: left promiscuous mode [ 120.492269][ T66] br0: port 2(eth1) entered disabled state [ 120.493384][ T66] eth0: left allmulticast mode [ 120.493522][ T66] eth0: left promiscuous mode [ 120.493783][ T66] br0: port 1(eth0) entered disabled state [ 120.777265][ T66] eth1: left allmulticast mode [ 120.777429][ T66] eth1: left promiscuous mode [ 120.777675][ T66] br0: port 2(eth1) entered disabled state [ 120.784678][ T66] eth0: left allmulticast mode [ 120.784849][ T66] eth0: left promiscuous mode [ 120.785084][ T66] br0: port 1(eth0) entered disabled state