[ 882.069995][T10680] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 882.112558][T10751] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 882.164124][T10754] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 882.203858][T10746] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 887.812162][T10759] devlink (10759) used greatest stack depth: 21440 bytes left [ 912.054319][T10810] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 912.061353][T10806] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 912.182125][T10816] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 912.276408][T10808] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 914.987515][T10807] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 915.046984][T10906] [ 915.047178][T10906] ====================================================== [ 915.047526][T10906] WARNING: possible circular locking dependency detected [ 915.047872][T10906] 6.13.0-rc2-virtme #1 Not tainted [ 915.048162][T10906] ------------------------------------------------------ [ 915.048505][T10906] devlink/10906 is trying to acquire lock: [ 915.048848][T10906] ffff8880094d5690 (kn->active#26){++++}-{0:0}, at: __kernfs_remove+0x1f9/0x590 [ 915.049232][T10906] [ 915.049232][T10906] but task is already holding lock: [ 915.049499][T10906] ffffffffa1be1208 (rtnl_mutex){+.+.}-{4:4}, at: nsim_destroy+0x6b/0x5f0 [netdevsim] [ 915.049847][T10906] [ 915.049847][T10906] which lock already depends on the new lock. [ 915.049847][T10906] [ 915.050203][T10906] [ 915.050203][T10906] the existing dependency chain (in reverse order) is: [ 915.050520][T10906] [ 915.050520][T10906] -> #1 (rtnl_mutex){+.+.}-{4:4}: [ 915.050793][T10906] __lock_acquire+0xb9a/0x1680 [ 915.050982][T10906] lock_acquire.part.0+0xeb/0x330 [ 915.051158][T10906] __mutex_lock+0x169/0xbc0 [ 915.051337][T10906] phys_port_name_show+0xbe/0x200 [ 915.051520][T10906] dev_attr_show+0x43/0xc0 [ 915.051698][T10906] sysfs_kf_seq_show+0x1fb/0x3d0 [ 915.051878][T10906] seq_read_iter+0x40e/0x1090 [ 915.052057][T10906] vfs_read+0x755/0xce0 [ 915.052237][T10906] ksys_read+0xf8/0x1d0 [ 915.052412][T10906] do_syscall_64+0xc1/0x1d0 [ 915.052591][T10906] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 915.052815][T10906] [ 915.052815][T10906] -> #0 (kn->active#26){++++}-{0:0}: [ 915.053086][T10906] check_prev_add+0x182/0xce0 [ 915.053272][T10906] validate_chain+0x7bd/0xae0 [ 915.053449][T10906] __lock_acquire+0xb9a/0x1680 [ 915.053623][T10906] lock_acquire.part.0+0xeb/0x330 [ 915.053796][T10906] kernfs_drain+0x384/0x460 [ 915.053969][T10906] __kernfs_remove+0x1f9/0x590 [ 915.054147][T10906] kernfs_remove_by_name_ns+0xa5/0x100 [ 915.054377][T10906] remove_files+0x8c/0x1a0 [ 915.054555][T10906] sysfs_remove_group+0x7b/0x160 [ 915.054734][T10906] sysfs_remove_groups+0x53/0xa0 [ 915.054913][T10906] device_remove_attrs+0x1db/0x260 [ 915.055089][T10906] device_del+0x337/0x8c0 [ 915.055267][T10906] unregister_netdevice_many_notify+0x43d/0x1580 [ 915.055489][T10906] unregister_netdevice_queue+0x2a4/0x410 [ 915.055710][T10906] nsim_destroy+0xe8/0x5f0 [netdevsim] [ 915.055940][T10906] __nsim_dev_port_del+0x17e/0x250 [netdevsim] [ 915.056175][T10906] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim] [ 915.056402][T10906] nsim_dev_reload_down+0x62/0xc0 [netdevsim] [ 915.056632][T10906] devlink_reload+0x172/0x450 [ 915.056813][T10906] devlink_nl_reload_doit+0xb75/0x10b0 [ 915.057029][T10906] genl_family_rcv_msg_doit+0x1d4/0x2b0 [ 915.057244][T10906] genl_family_rcv_msg+0x347/0x5b0 [ 915.057419][T10906] genl_rcv_msg+0xa3/0x140 [ 915.057596][T10906] netlink_rcv_skb+0x130/0x360 [ 915.057774][T10906] genl_rcv+0x28/0x40 [ 915.057952][T10906] netlink_unicast+0x44b/0x710 [ 915.058132][T10906] netlink_sendmsg+0x723/0xbe0 [ 915.058317][T10906] __sys_sendto+0x3c3/0x450 [ 915.058495][T10906] __x64_sys_sendto+0xe0/0x1c0 [ 915.058673][T10906] do_syscall_64+0xc1/0x1d0 [ 915.058857][T10906] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 915.059089][T10906] [ 915.059089][T10906] other info that might help us debug this: [ 915.059089][T10906] [ 915.059436][T10906] Possible unsafe locking scenario: [ 915.059436][T10906] [ 915.059693][T10906] CPU0 CPU1 [ 915.059864][T10906] ---- ---- [ 915.060039][T10906] lock(rtnl_mutex); [ 915.060172][T10906] lock(kn->active#26); [ 915.060401][T10906] lock(rtnl_mutex); [ 915.060622][T10906] lock(kn->active#26); [ 915.060757][T10906] [ 915.060757][T10906] *** DEADLOCK *** [ 915.060757][T10906] [ 915.061012][T10906] 4 locks held by devlink/10906: [ 915.061188][T10906] #0: ffffffffa1c367b0 (cb_lock){++++}-{4:4}, at: genl_rcv+0x19/0x40 [ 915.061459][T10906] #1: ffff88800da168e8 (&dev->mutex){....}-{4:4}, at: devlink_get_from_attrs_lock+0x28a/0x3a0 [ 915.061818][T10906] #2: ffff88800eab5250 (&devlink->lock_key){+.+.}-{4:4}, at: devlink_get_from_attrs_lock+0x292/0x3a0 [ 915.062167][T10906] #3: ffffffffa1be1208 (rtnl_mutex){+.+.}-{4:4}, at: nsim_destroy+0x6b/0x5f0 [netdevsim] [ 915.062489][T10906] [ 915.062489][T10906] stack backtrace: [ 915.062709][T10906] CPU: 0 UID: 0 PID: 10906 Comm: devlink Not tainted 6.13.0-rc2-virtme #1 [ 915.062973][T10906] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 915.063196][T10906] Call Trace: [ 915.063331][T10906] [ 915.063423][T10906] dump_stack_lvl+0x82/0xd0 [ 915.063603][T10906] print_circular_bug+0x2e2/0x4b0 [ 915.063778][T10906] check_noncircular+0x2f9/0x3e0 [ 915.063954][T10906] ? __pfx_check_noncircular+0x10/0x10 [ 915.064138][T10906] ? __kernel_text_address+0x12/0x40 [ 915.064316][T10906] ? unwind_get_return_address+0x5e/0xa0 [ 915.064493][T10906] ? __pfx_stack_trace_consume_entry+0x10/0x10 [ 915.064712][T10906] ? arch_stack_walk+0xa2/0xf0 [ 915.064890][T10906] ? alloc_chain_hlocks+0x4e0/0x720 [ 915.065064][T10906] check_prev_add+0x182/0xce0 [ 915.065244][T10906] ? __pfx_stack_trace_save+0x10/0x10 [ 915.065421][T10906] validate_chain+0x7bd/0xae0 [ 915.065597][T10906] ? __pfx_validate_chain+0x10/0x10 [ 915.065772][T10906] ? mark_lock+0x335/0x3e0 [ 915.065948][T10906] __lock_acquire+0xb9a/0x1680 [ 915.066123][T10906] ? kasan_addr_to_slab+0x40/0x80 [ 915.066308][T10906] lock_acquire.part.0+0xeb/0x330 [ 915.066481][T10906] ? __kernfs_remove+0x1f9/0x590 [ 915.066665][T10906] ? __pfx___lock_release+0x10/0x10 [ 915.066847][T10906] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 915.067021][T10906] ? trace_lock_acquire+0x14c/0x1f0 [ 915.067196][T10906] ? __virt_addr_valid+0x22b/0x430 [ 915.067373][T10906] ? lock_acquire+0x32/0xc0 [ 915.067546][T10906] ? __kernfs_remove+0x1f9/0x590 [ 915.067726][T10906] kernfs_drain+0x384/0x460 [ 915.067906][T10906] ? __kernfs_remove+0x1f9/0x590 [ 915.068084][T10906] ? __pfx_kernfs_drain+0x10/0x10 [ 915.068265][T10906] ? lock_acquire+0x32/0xc0 [ 915.068443][T10906] __kernfs_remove+0x1f9/0x590 [ 915.068621][T10906] kernfs_remove_by_name_ns+0xa5/0x100 [ 915.068796][T10906] remove_files+0x8c/0x1a0 [ 915.068982][T10906] sysfs_remove_group+0x7b/0x160 [ 915.069159][T10906] sysfs_remove_groups+0x53/0xa0 [ 915.069340][T10906] device_remove_attrs+0x1db/0x260 [ 915.069517][T10906] ? __pfx_device_remove_attrs+0x10/0x10 [ 915.069694][T10906] device_del+0x337/0x8c0 [ 915.069827][T10906] ? __pfx_device_del+0x10/0x10 [ 915.070003][T10906] unregister_netdevice_many_notify+0x43d/0x1580 [ 915.070229][T10906] ? __mutex_trylock_common+0xfa/0x260 [ 915.070410][T10906] ? __pfx___mutex_trylock_common+0x10/0x10 [ 915.070632][T10906] ? __pfx_unregister_netdevice_many_notify+0x10/0x10 [ 915.070853][T10906] ? __mutex_lock+0x190/0xbc0 [ 915.071028][T10906] ? nsim_destroy+0x6b/0x5f0 [netdevsim] [ 915.071214][T10906] ? __pfx___mutex_lock+0x10/0x10 [ 915.071387][T10906] unregister_netdevice_queue+0x2a4/0x410 [ 915.071564][T10906] ? __pfx_do_raw_spin_lock+0x10/0x10 [ 915.071745][T10906] ? __pfx_unregister_netdevice_queue+0x10/0x10 [ 915.071962][T10906] nsim_destroy+0xe8/0x5f0 [netdevsim] [ 915.072156][T10906] __nsim_dev_port_del+0x17e/0x250 [netdevsim] [ 915.072383][T10906] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim] [ 915.072611][T10906] nsim_dev_reload_down+0x62/0xc0 [netdevsim] [ 915.072836][T10906] devlink_reload+0x172/0x450 [ 915.073014][T10906] ? __pfx_devlink_reload+0x10/0x10 [ 915.073196][T10906] ? devlink_resources_validate+0x9c/0x100 [ 915.073418][T10906] devlink_nl_reload_doit+0xb75/0x10b0 [ 915.073593][T10906] ? __pfx_devlink_nl_reload_doit+0x10/0x10 [ 915.073811][T10906] ? __nla_parse+0x26/0x30 [ 915.073993][T10906] genl_family_rcv_msg_doit+0x1d4/0x2b0 [ 915.074170][T10906] ? __pfx_genl_family_rcv_msg_doit+0x10/0x10 [ 915.074399][T10906] ? validate_chain+0x1fe/0xae0 [ 915.074576][T10906] ? rcu_read_lock_any_held+0x43/0xb0 [ 915.074751][T10906] ? validate_chain+0x1fe/0xae0 [ 915.074930][T10906] genl_family_rcv_msg+0x347/0x5b0 [ 915.075104][T10906] ? __pfx_genl_family_rcv_msg+0x10/0x10 [ 915.075280][T10906] ? __pfx_devlink_nl_pre_doit_dev_lock+0x10/0x10 [ 915.075497][T10906] ? __pfx_devlink_nl_reload_doit+0x10/0x10 [ 915.075717][T10906] ? __pfx_devlink_nl_post_doit_dev_lock+0x10/0x10 [ 915.075937][T10906] genl_rcv_msg+0xa3/0x140 [ 915.076113][T10906] netlink_rcv_skb+0x130/0x360 [ 915.076289][T10906] ? __pfx_genl_rcv_msg+0x10/0x10 [ 915.076463][T10906] ? __pfx_netlink_rcv_skb+0x10/0x10 [ 915.076641][T10906] ? genl_rcv+0x19/0x40 [ 915.076775][T10906] ? __pfx_down_read+0x10/0x10 [ 915.076954][T10906] ? netlink_deliver_tap+0x13e/0x340 [ 915.077129][T10906] genl_rcv+0x28/0x40 [ 915.077259][T10906] netlink_unicast+0x44b/0x710 [ 915.077437][T10906] ? __pfx_netlink_unicast+0x10/0x10 [ 915.077614][T10906] ? find_held_lock+0x2c/0x110 [ 915.077790][T10906] netlink_sendmsg+0x723/0xbe0 [ 915.077966][T10906] ? __pfx_netlink_sendmsg+0x10/0x10 [ 915.078142][T10906] ? lock_acquire+0x32/0xc0 [ 915.078331][T10906] ? __might_fault+0x11b/0x170 [ 915.078510][T10906] __sys_sendto+0x3c3/0x450 [ 915.078684][T10906] ? __pfx___sys_sendto+0x10/0x10 [ 915.078863][T10906] ? __lock_release+0x103/0x460 [ 915.079046][T10906] ? do_user_addr_fault+0x972/0xe30 [ 915.079222][T10906] ? __pfx___lock_release+0x10/0x10 [ 915.079398][T10906] ? __pfx___up_read+0x10/0x10 [ 915.079581][T10906] ? do_user_addr_fault+0x972/0xe30 [ 915.079770][T10906] ? do_user_addr_fault+0x97c/0xe30 [ 915.079963][T10906] __x64_sys_sendto+0xe0/0x1c0 [ 915.080158][T10906] ? lockdep_hardirqs_on_prepare+0x275/0x410 [ 915.080402][T10906] do_syscall_64+0xc1/0x1d0 [ 915.080582][T10906] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 915.080821][T10906] RIP: 0033:0x7fe520ecda4a [ 915.081030][T10906] Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89 [ 915.081696][T10906] RSP: 002b:00007ffe63c25c98 EFLAGS: 00000246 ORIG_RAX: 000000000000002c [ 915.081981][T10906] RAX: ffffffffffffffda RBX: 000000003cbc8b00 RCX: 00007fe520ecda4a [ 915.082274][T10906] RDX: 0000000000000034 RSI: 000000003cbc8b00 RDI: 0000000000000005 [ 915.082562][T10906] RBP: 000000003cbc8910 R08: 00007fe521071200 R09: 000000000000000c [ 915.082846][T10906] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000040f200 [ 915.083250][T10906] R13: 000000003cbc8910 R14: 0000000000000000 R15: 0000000000000001 [ 915.083547][T10906] [ 1096.023319][T10806] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 1096.107955][T10917] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 1096.120432][T10810] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 1096.460350][T10917] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 1096.500445][T10816] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 1096.605975][T10918] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 1096.671886][T10810] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 1097.341787][T10809] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 1097.445535][T10918] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 1097.552107][T10917] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 1097.638007][T10806] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 1099.559657][T11026] Failed to register fib notifier [ 1099.822579][T10918] netdevsim netdevsim10 eni10np2: renamed from eth1 [ 1099.907475][T10809] netdevsim netdevsim10 eni10np4: renamed from eth3 [ 1099.931307][T10806] netdevsim netdevsim10 eni10np3: renamed from eth2 [ 1100.045328][T10917] netdevsim netdevsim10 eni10np1: renamed from eth0 [ 1105.631370][T11272] netdevsim netdevsim10 eni10npf0vf0: renamed from eth0 [ 1105.642727][T11274] netdevsim netdevsim10 eni10npf0vf1: renamed from eth1 [ 1105.652796][T11277] netdevsim netdevsim10 eni10npf0vf2: renamed from eth2 [ 1105.763268][T11279] netdevsim netdevsim10 eni10npf0vf3: renamed from eth3