[ 836.854853][ T8966] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 836.857739][ T8956] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 836.886304][ T8971] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 836.999207][ T8959] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 842.586099][ T8977] devlink (8977) used greatest stack depth: 22256 bytes left [ 866.913246][ T9026] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 866.927935][ T9028] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 867.046606][ T9025] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 867.049311][ T9036] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 869.996626][ T9036] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 870.006729][ T9027] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 870.019876][ T9124] [ 870.020021][ T9124] ====================================================== [ 870.020287][ T9124] WARNING: possible circular locking dependency detected [ 870.020541][ T9124] 6.13.0-rc2-virtme #1 Not tainted [ 870.020741][ T9124] ------------------------------------------------------ [ 870.020988][ T9124] devlink/9124 is trying to acquire lock: [ 870.021199][ T9124] ffff88800f770fd0 (kn->active#25){++++}-{0:0}, at: __kernfs_remove+0x1f9/0x590 [ 870.021555][ T9124] [ 870.021555][ T9124] but task is already holding lock: [ 870.021830][ T9124] ffffffffb0be1548 (rtnl_mutex){+.+.}-{4:4}, at: nsim_destroy+0x6b/0x5f0 [netdevsim] [ 870.022174][ T9124] [ 870.022174][ T9124] which lock already depends on the new lock. [ 870.022174][ T9124] [ 870.022536][ T9124] [ 870.022536][ T9124] the existing dependency chain (in reverse order) is: [ 870.022851][ T9124] [ 870.022851][ T9124] -> #1 (rtnl_mutex){+.+.}-{4:4}: [ 870.023132][ T9124] __lock_acquire+0xb9a/0x1680 [ 870.023330][ T9124] lock_acquire.part.0+0xeb/0x330 [ 870.023522][ T9124] __mutex_lock+0x169/0xbc0 [ 870.023703][ T9124] phys_port_name_show+0xbe/0x200 [ 870.023887][ T9124] dev_attr_show+0x43/0xc0 [ 870.024068][ T9124] sysfs_kf_seq_show+0x1fb/0x3d0 [ 870.024247][ T9124] seq_read_iter+0x40e/0x1090 [ 870.024426][ T9124] vfs_read+0x755/0xce0 [ 870.024605][ T9124] ksys_read+0xf8/0x1d0 [ 870.024792][ T9124] do_syscall_64+0xc1/0x1d0 [ 870.024976][ T9124] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 870.025211][ T9124] [ 870.025211][ T9124] -> #0 (kn->active#25){++++}-{0:0}: [ 870.025482][ T9124] check_prev_add+0x182/0xce0 [ 870.025670][ T9124] validate_chain+0x7bd/0xae0 [ 870.025856][ T9124] __lock_acquire+0xb9a/0x1680 [ 870.026042][ T9124] lock_acquire.part.0+0xeb/0x330 [ 870.026225][ T9124] kernfs_drain+0x384/0x460 [ 870.026403][ T9124] __kernfs_remove+0x1f9/0x590 [ 870.026582][ T9124] kernfs_remove_by_name_ns+0xa5/0x100 [ 870.026807][ T9124] remove_files+0x8c/0x1a0 [ 870.026991][ T9124] sysfs_remove_group+0x7b/0x160 [ 870.027176][ T9124] sysfs_remove_groups+0x53/0xa0 [ 870.027348][ T9124] device_remove_attrs+0x1db/0x260 [ 870.027524][ T9124] device_del+0x337/0x8c0 [ 870.027704][ T9124] unregister_netdevice_many_notify+0x43d/0x1580 [ 870.027934][ T9124] unregister_netdevice_queue+0x2a4/0x410 [ 870.028166][ T9124] nsim_destroy+0xe8/0x5f0 [netdevsim] [ 870.028425][ T9124] __nsim_dev_port_del+0x17e/0x250 [netdevsim] [ 870.028669][ T9124] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim] [ 870.028898][ T9124] nsim_dev_reload_down+0x62/0xc0 [netdevsim] [ 870.029133][ T9124] devlink_reload+0x172/0x450 [ 870.029315][ T9124] devlink_nl_reload_doit+0xb75/0x10b0 [ 870.029555][ T9124] genl_family_rcv_msg_doit+0x1d4/0x2b0 [ 870.029774][ T9124] genl_family_rcv_msg+0x347/0x5b0 [ 870.029962][ T9124] genl_rcv_msg+0xa3/0x140 [ 870.030131][ T9124] netlink_rcv_skb+0x130/0x360 [ 870.030305][ T9124] genl_rcv+0x28/0x40 [ 870.030477][ T9124] netlink_unicast+0x44b/0x710 [ 870.030650][ T9124] netlink_sendmsg+0x723/0xbe0 [ 870.030833][ T9124] __sys_sendto+0x3c3/0x450 [ 870.031004][ T9124] __x64_sys_sendto+0xe0/0x1c0 [ 870.031192][ T9124] do_syscall_64+0xc1/0x1d0 [ 870.031362][ T9124] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 870.031582][ T9124] [ 870.031582][ T9124] other info that might help us debug this: [ 870.031582][ T9124] [ 870.031926][ T9124] Possible unsafe locking scenario: [ 870.031926][ T9124] [ 870.032190][ T9124] CPU0 CPU1 [ 870.032367][ T9124] ---- ---- [ 870.032553][ T9124] lock(rtnl_mutex); [ 870.032685][ T9124] lock(kn->active#25); [ 870.032899][ T9124] lock(rtnl_mutex); [ 870.033112][ T9124] lock(kn->active#25); [ 870.033246][ T9124] [ 870.033246][ T9124] *** DEADLOCK *** [ 870.033246][ T9124] [ 870.033514][ T9124] 4 locks held by devlink/9124: [ 870.033689][ T9124] #0: ffffffffb0c36d30 (cb_lock){++++}-{4:4}, at: genl_rcv+0x19/0x40 [ 870.033970][ T9124] #1: ffff88800ed5b8e8 (&dev->mutex){....}-{4:4}, at: devlink_get_from_attrs_lock+0x28a/0x3a0 [ 870.034330][ T9124] #2: ffff88800ed5d250 (&devlink->lock_key){+.+.}-{4:4}, at: devlink_get_from_attrs_lock+0x292/0x3a0 [ 870.034680][ T9124] #3: ffffffffb0be1548 (rtnl_mutex){+.+.}-{4:4}, at: nsim_destroy+0x6b/0x5f0 [netdevsim] [ 870.034991][ T9124] [ 870.034991][ T9124] stack backtrace: [ 870.035202][ T9124] CPU: 2 UID: 0 PID: 9124 Comm: devlink Not tainted 6.13.0-rc2-virtme #1 [ 870.035461][ T9124] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 870.035674][ T9124] Call Trace: [ 870.035807][ T9124] [ 870.035901][ T9124] dump_stack_lvl+0x82/0xd0 [ 870.036082][ T9124] print_circular_bug+0x2e2/0x4b0 [ 870.036257][ T9124] check_noncircular+0x2f9/0x3e0 [ 870.036430][ T9124] ? __pfx_check_noncircular+0x10/0x10 [ 870.036617][ T9124] ? __kernel_text_address+0x12/0x40 [ 870.036799][ T9124] ? unwind_get_return_address+0x5e/0xa0 [ 870.036973][ T9124] ? __pfx_stack_trace_consume_entry+0x10/0x10 [ 870.037193][ T9124] ? arch_stack_walk+0xa2/0xf0 [ 870.037369][ T9124] ? alloc_chain_hlocks+0x4e0/0x720 [ 870.037540][ T9124] check_prev_add+0x182/0xce0 [ 870.037713][ T9124] ? __pfx_stack_trace_save+0x10/0x10 [ 870.037897][ T9124] validate_chain+0x7bd/0xae0 [ 870.038069][ T9124] ? __pfx_validate_chain+0x10/0x10 [ 870.038249][ T9124] ? mark_lock+0x335/0x3e0 [ 870.038426][ T9124] __lock_acquire+0xb9a/0x1680 [ 870.038596][ T9124] ? kasan_addr_to_slab+0x40/0x80 [ 870.038783][ T9124] lock_acquire.part.0+0xeb/0x330 [ 870.038965][ T9124] ? __kernfs_remove+0x1f9/0x590 [ 870.039150][ T9124] ? __pfx___lock_release+0x10/0x10 [ 870.039320][ T9124] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 870.039510][ T9124] ? trace_lock_acquire+0x14c/0x1f0 [ 870.039697][ T9124] ? __virt_addr_valid+0x22b/0x430 [ 870.039879][ T9124] ? lock_acquire+0x32/0xc0 [ 870.040058][ T9124] ? __kernfs_remove+0x1f9/0x590 [ 870.040236][ T9124] kernfs_drain+0x384/0x460 [ 870.040425][ T9124] ? __kernfs_remove+0x1f9/0x590 [ 870.040599][ T9124] ? __pfx_kernfs_drain+0x10/0x10 [ 870.040776][ T9124] ? lock_acquire+0x32/0xc0 [ 870.040952][ T9124] __kernfs_remove+0x1f9/0x590 [ 870.041131][ T9124] kernfs_remove_by_name_ns+0xa5/0x100 [ 870.041323][ T9124] remove_files+0x8c/0x1a0 [ 870.041503][ T9124] sysfs_remove_group+0x7b/0x160 [ 870.041689][ T9124] sysfs_remove_groups+0x53/0xa0 [ 870.041864][ T9124] device_remove_attrs+0x1db/0x260 [ 870.042039][ T9124] ? __pfx_device_remove_attrs+0x10/0x10 [ 870.042228][ T9124] device_del+0x337/0x8c0 [ 870.042361][ T9124] ? __pfx_device_del+0x10/0x10 [ 870.042559][ T9124] unregister_netdevice_many_notify+0x43d/0x1580 [ 870.042784][ T9124] ? __mutex_trylock_common+0xfa/0x260 [ 870.042962][ T9124] ? __pfx___mutex_trylock_common+0x10/0x10 [ 870.043187][ T9124] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 870.043401][ T9124] ? __mutex_lock+0x190/0xbc0 [ 870.043578][ T9124] ? nsim_destroy+0x6b/0x5f0 [netdevsim] [ 870.043771][ T9124] ? __pfx___mutex_lock+0x10/0x10 [ 870.043953][ T9124] unregister_netdevice_queue+0x2a4/0x410 [ 870.044128][ T9124] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 870.044301][ T9124] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 870.044526][ T9124] nsim_destroy+0xe8/0x5f0 [netdevsim] [ 870.044712][ T9124] __nsim_dev_port_del+0x17e/0x250 [netdevsim] [ 870.044945][ T9124] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim] [ 870.045174][ T9124] nsim_dev_reload_down+0x62/0xc0 [netdevsim] [ 870.045399][ T9124] devlink_reload+0x172/0x450 [ 870.045580][ T9124] ? __pfx_devlink_reload+0x10/0x10 [ 870.045761][ T9124] ? devlink_resources_validate+0x9c/0x100 [ 870.045982][ T9124] devlink_nl_reload_doit+0xb75/0x10b0 [ 870.046158][ T9124] ? __pfx_devlink_nl_reload_doit+0x10/0x10 [ 870.046386][ T9124] ? __nla_parse+0x26/0x30 [ 870.046568][ T9124] genl_family_rcv_msg_doit+0x1d4/0x2b0 [ 870.046742][ T9124] ? __pfx_genl_family_rcv_msg_doit+0x10/0x10 [ 870.046977][ T9124] ? validate_chain+0x1fe/0xae0 [ 870.047154][ T9124] ? rcu_read_lock_any_held+0x43/0xb0 [ 870.047325][ T9124] ? validate_chain+0x1fe/0xae0 [ 870.047505][ T9124] genl_family_rcv_msg+0x347/0x5b0 [ 870.047683][ T9124] ? __pfx_genl_family_rcv_msg+0x10/0x10 [ 870.047857][ T9124] ? __pfx_devlink_nl_pre_doit_dev_lock+0x10/0x10 [ 870.048085][ T9124] ? __pfx_devlink_nl_reload_doit+0x10/0x10 [ 870.048302][ T9124] ? __pfx_devlink_nl_post_doit_dev_lock+0x10/0x10 [ 870.048530][ T9124] genl_rcv_msg+0xa3/0x140 [ 870.048704][ T9124] netlink_rcv_skb+0x130/0x360 [ 870.048888][ T9124] ? __pfx_genl_rcv_msg+0x10/0x10 [ 870.049072][ T9124] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 870.049245][ T9124] ? genl_rcv+0x19/0x40 [ 870.049380][ T9124] ? __pfx_down_read+0x10/0x10 [ 870.049578][ T9124] ? netlink_deliver_tap+0x13e/0x340 [ 870.049767][ T9124] genl_rcv+0x28/0x40 [ 870.049895][ T9124] netlink_unicast+0x44b/0x710 [ 870.050069][ T9124] ? __pfx_netlink_unicast+0x10/0x10 [ 870.050261][ T9124] ? find_held_lock+0x2c/0x110 [ 870.050438][ T9124] netlink_sendmsg+0x723/0xbe0 [ 870.050619][ T9124] ? __pfx_netlink_sendmsg+0x10/0x10 [ 870.050801][ T9124] ? lock_acquire+0x32/0xc0 [ 870.050972][ T9124] ? __might_fault+0x11b/0x170 [ 870.051153][ T9124] __sys_sendto+0x3c3/0x450 [ 870.051333][ T9124] ? __pfx___sys_sendto+0x10/0x10 [ 870.051503][ T9124] ? __lock_release+0x103/0x460 [ 870.051680][ T9124] ? do_user_addr_fault+0x972/0xe30 [ 870.051863][ T9124] ? __pfx___lock_release+0x10/0x10 [ 870.052034][ T9124] ? __pfx___up_read+0x10/0x10 [ 870.052226][ T9124] ? do_user_addr_fault+0x972/0xe30 [ 870.052395][ T9124] ? do_user_addr_fault+0x97c/0xe30 [ 870.052569][ T9124] __x64_sys_sendto+0xe0/0x1c0 [ 870.052740][ T9124] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 870.052956][ T9124] do_syscall_64+0xc1/0x1d0 [ 870.053136][ T9124] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 870.053361][ T9124] RIP: 0033:0x7fe993fe2a4a [ 870.053547][ T9124] Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89 [ 870.054172][ T9124] RSP: 002b:00007ffd53cbca38 EFLAGS: 00000246 ORIG_RAX: 000000000000002c [ 870.054431][ T9124] RAX: ffffffffffffffda RBX: 0000000031b41b00 RCX: 00007fe993fe2a4a [ 870.054697][ T9124] RDX: 0000000000000034 RSI: 0000000031b41b00 RDI: 0000000000000005 [ 870.054955][ T9124] RBP: 0000000031b41910 R08: 00007fe994186200 R09: 000000000000000c [ 870.055224][ T9124] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000040f200 [ 870.055500][ T9124] R13: 0000000031b41910 R14: 0000000000000000 R15: 0000000000000001 [ 870.055779][ T9124] [ 1050.577867][ T9027] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 1050.606666][ T9137] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 1050.924552][ T9026] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 1051.022785][ T9027] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 1051.138993][ T9116] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 1051.153529][ T9025] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 1051.924770][ T9027] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 1052.017841][ T9149] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 1052.027937][ T9137] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 1052.127716][ T9025] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 1054.062301][ T9246] Failed to register fib notifier [ 1054.355306][ T9027] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 1054.386384][ T9026] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 1054.489529][ T9025] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 1054.565355][ T9137] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 1060.139766][ T9494] netdevsim netdevsim10 eni10npf0vf0: renamed from eth0 [ 1060.258938][ T9495] netdevsim netdevsim10 eni10npf0vf1: renamed from eth1 [ 1060.361689][ T9496] netdevsim netdevsim10 eni10npf0vf2: renamed from eth2 [ 1060.439039][ T9498] netdevsim netdevsim10 eni10npf0vf3: renamed from eth3