[ 955.297271][T11635] netdevsim netdevsim1337 eni1337np1: renamed from eth0
[ 955.301141][T11634] netdevsim netdevsim1057 eni1057np1: renamed from eth1
[ 955.439256][T11647] netdevsim netdevsim5417 eni5417np1: renamed from eth2
[ 955.626242][T11657]
[ 955.626370][T11657] ======================================================
[ 955.626598][T11657] WARNING: possible circular locking dependency detected
[ 955.626820][T11657] 6.13.0-rc2-virtme #1 Not tainted
[ 955.626994][T11657] ------------------------------------------------------
[ 955.627208][T11657] devlink/11657 is trying to acquire lock:
[ 955.627427][T11657] ffff888009782250 (kn->active#25){++++}-{0:0}, at: __kernfs_remove+0x1f9/0x590
[ 955.627753][T11657]
[ 955.627753][T11657] but task is already holding lock:
[ 955.628007][T11657] ffffffffb0fe1208 (rtnl_mutex){+.+.}-{4:4}, at: nsim_destroy+0x6b/0x5f0 [netdevsim]
[ 955.628332][T11657]
[ 955.628332][T11657] which lock already depends on the new lock.
[ 955.628332][T11657]
[ 955.628680][T11657]
[ 955.628680][T11657] the existing dependency chain (in reverse order) is:
[ 955.628974][T11657]
[ 955.628974][T11657] -> #1 (rtnl_mutex){+.+.}-{4:4}:
[ 955.629236][T11657] __lock_acquire+0xb9a/0x1680
[ 955.629418][T11657] lock_acquire.part.0+0xeb/0x330
[ 955.629593][T11657] __mutex_lock+0x169/0xbc0
[ 955.629831][T11657] phys_port_name_show+0xbe/0x200
[ 955.630009][T11657] dev_attr_show+0x43/0xc0
[ 955.630183][T11657] sysfs_kf_seq_show+0x1fb/0x3d0
[ 955.630361][T11657] seq_read_iter+0x40e/0x1090
[ 955.630536][T11657] vfs_read+0x755/0xce0
[ 955.630708][T11657] ksys_read+0xf8/0x1d0
[ 955.630878][T11657] do_syscall_64+0xc1/0x1d0
[ 955.631064][T11657] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 955.631288][T11657]
[ 955.631288][T11657] -> #0 (kn->active#25){++++}-{0:0}:
[ 955.631551][T11657] check_prev_add+0x182/0xce0
[ 955.631729][T11657] validate_chain+0x7bd/0xae0
[ 955.631914][T11657] __lock_acquire+0xb9a/0x1680
[ 955.632092][T11657] lock_acquire.part.0+0xeb/0x330
[ 955.632273][T11657] kernfs_drain+0x384/0x460
[ 955.632448][T11657] __kernfs_remove+0x1f9/0x590
[ 955.632621][T11657] kernfs_remove_by_name_ns+0xa5/0x100
[ 955.632833][T11657] remove_files+0x8c/0x1a0
[ 955.633004][T11657] sysfs_remove_group+0x7b/0x160
[ 955.633179][T11657] sysfs_remove_groups+0x53/0xa0
[ 955.633352][T11657] device_remove_attrs+0x1db/0x260
[ 955.633528][T11657] device_del+0x337/0x8c0
[ 955.633730][T11657] unregister_netdevice_many_notify+0x43d/0x1580
[ 955.633948][T11657] unregister_netdevice_queue+0x2a4/0x410
[ 955.634164][T11657] nsim_destroy+0xe8/0x5f0 [netdevsim]
[ 955.634391][T11657] __nsim_dev_port_del+0x17e/0x250 [netdevsim]
[ 955.634613][T11657] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim]
[ 955.634843][T11657] nsim_dev_reload_down+0x62/0xc0 [netdevsim]
[ 955.635072][T11657] devlink_reload+0x172/0x450
[ 955.635246][T11657] devlink_nl_reload_doit+0x395/0x10b0
[ 955.635458][T11657] genl_family_rcv_msg_doit+0x1d4/0x2b0
[ 955.635674][T11657] genl_family_rcv_msg+0x347/0x5b0
[ 955.635846][T11657] genl_rcv_msg+0xa3/0x140
[ 955.636027][T11657] netlink_rcv_skb+0x130/0x360
[ 955.636204][T11657] genl_rcv+0x28/0x40
[ 955.636375][T11657] netlink_unicast+0x44b/0x710
[ 955.636548][T11657] netlink_sendmsg+0x723/0xbe0
[ 955.636722][T11657] __sys_sendto+0x3c3/0x450
[ 955.636895][T11657] __x64_sys_sendto+0xe0/0x1c0
[ 955.637067][T11657] do_syscall_64+0xc1/0x1d0
[ 955.637241][T11657] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 955.637455][T11657]
[ 955.637455][T11657] other info that might help us debug this:
[ 955.637455][T11657]
[ 955.637834][T11657] Possible unsafe locking scenario:
[ 955.637834][T11657]
[ 955.638111][T11657] CPU0 CPU1
[ 955.638298][T11657] ---- ----
[ 955.638490][T11657] lock(rtnl_mutex);
[ 955.638637][T11657] lock(kn->active#25);
[ 955.638875][T11657] lock(rtnl_mutex);
[ 955.639110][T11657] lock(kn->active#25);
[ 955.639254][T11657]
[ 955.639254][T11657] *** DEADLOCK ***
[ 955.639254][T11657]
[ 955.639528][T11657] 4 locks held by devlink/11657:
[ 955.639718][T11657] #0: ffffffffb10367b0 (cb_lock){++++}-{4:4}, at: genl_rcv+0x19/0x40
[ 955.640006][T11657] #1: ffff88800b75e8e8 (&dev->mutex){....}-{4:4}, at: devlink_get_from_attrs_lock+0x28a/0x3a0
[ 955.640385][T11657] #2: ffff88800abc3a50 (&devlink->lock_key){+.+.}-{4:4}, at: devlink_get_from_attrs_lock+0x292/0x3a0
[ 955.640762][T11657] #3: ffffffffb0fe1208 (rtnl_mutex){+.+.}-{4:4}, at: nsim_destroy+0x6b/0x5f0 [netdevsim]
[ 955.641127][T11657]
[ 955.641127][T11657] stack backtrace:
[ 955.641361][T11657] CPU: 1 UID: 0 PID: 11657 Comm: devlink Not tainted 6.13.0-rc2-virtme #1
[ 955.641645][T11657] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 955.641912][T11657] Call Trace:
[ 955.642057][T11657]
[ 955.642157][T11657] dump_stack_lvl+0x82/0xd0
[ 955.642358][T11657] print_circular_bug+0x2e2/0x4b0
[ 955.642550][T11657] check_noncircular+0x2f9/0x3e0
[ 955.642738][T11657] ? __pfx_check_noncircular+0x10/0x10
[ 955.642930][T11657] ? __kernel_text_address+0x12/0x40
[ 955.643120][T11657] ? unwind_get_return_address+0x5e/0xa0
[ 955.643309][T11657] ? __pfx_stack_trace_consume_entry+0x10/0x10
[ 955.643544][T11657] ? arch_stack_walk+0xa2/0xf0
[ 955.643817][T11657] check_prev_add+0x182/0xce0
[ 955.644115][T11657] ? __pfx_stack_trace_save+0x10/0x10
[ 955.644366][T11657] validate_chain+0x7bd/0xae0
[ 955.644667][T11657] ? __pfx_validate_chain+0x10/0x10
[ 955.644966][T11657] ? mark_lock+0x335/0x3e0
[ 955.645254][T11657] __lock_acquire+0xb9a/0x1680
[ 955.645524][T11657] ? kasan_addr_to_slab+0x40/0x80
[ 955.645822][T11657] lock_acquire.part.0+0xeb/0x330
[ 955.646106][T11657] ? __kernfs_remove+0x1f9/0x590
[ 955.646411][T11657] ? __pfx___lock_release+0x10/0x10
[ 955.646722][T11657] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 955.647004][T11657] ? trace_lock_acquire+0x14c/0x1f0
[ 955.647280][T11657] ? __virt_addr_valid+0x22b/0x430
[ 955.647566][T11657] ? lock_acquire+0x32/0xc0
[ 955.647841][T11657] ? __kernfs_remove+0x1f9/0x590
[ 955.648137][T11657] kernfs_drain+0x384/0x460
[ 955.648422][T11657] ? __kernfs_remove+0x1f9/0x590
[ 955.648707][T11657] ? __pfx_kernfs_drain+0x10/0x10
[ 955.648980][T11657] ? lock_acquire+0x32/0xc0
[ 955.649234][T11657] __kernfs_remove+0x1f9/0x590
[ 955.649512][T11657] kernfs_remove_by_name_ns+0xa5/0x100
[ 955.649796][T11657] remove_files+0x8c/0x1a0
[ 955.650079][T11657] sysfs_remove_group+0x7b/0x160
[ 955.650372][T11657] sysfs_remove_groups+0x53/0xa0
[ 955.650649][T11657] device_remove_attrs+0x1db/0x260
[ 955.650888][T11657] ? __pfx_device_remove_attrs+0x10/0x10
[ 955.651179][T11657] device_del+0x337/0x8c0
[ 955.651394][T11657] ? __pfx_device_del+0x10/0x10
[ 955.651684][T11657] unregister_netdevice_many_notify+0x43d/0x1580
[ 955.652038][T11657] ? __mutex_trylock_common+0xfa/0x260
[ 955.652326][T11657] ? __pfx___mutex_trylock_common+0x10/0x10
[ 955.652671][T11657] ? __pfx_unregister_netdevice_many_notify+0x10/0x10
[ 955.653021][T11657] ? __mutex_lock+0x190/0xbc0
[ 955.653301][T11657] ? nsim_destroy+0x6b/0x5f0 [netdevsim]
[ 955.653599][T11657] ? __pfx___mutex_lock+0x10/0x10
[ 955.653884][T11657] unregister_netdevice_queue+0x2a4/0x410
[ 955.654165][T11657] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 955.654454][T11657] ? __pfx_unregister_netdevice_queue+0x10/0x10
[ 955.654798][T11657] nsim_destroy+0xe8/0x5f0 [netdevsim]
[ 955.655098][T11657] __nsim_dev_port_del+0x17e/0x250 [netdevsim]
[ 955.655464][T11657] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim]
[ 955.655829][T11657] nsim_dev_reload_down+0x62/0xc0 [netdevsim]
[ 955.656193][T11657] devlink_reload+0x172/0x450
[ 955.656488][T11657] ? __pfx_devlink_reload+0x10/0x10
[ 955.656765][T11657] ? fdget+0x52/0x1e0
[ 955.656976][T11657] devlink_nl_reload_doit+0x395/0x10b0
[ 955.657251][T11657] ? __pfx_devlink_nl_reload_doit+0x10/0x10
[ 955.657598][T11657] ? __nla_parse+0x26/0x30
[ 955.657876][T11657] genl_family_rcv_msg_doit+0x1d4/0x2b0
[ 955.658152][T11657] ? __pfx_genl_family_rcv_msg_doit+0x10/0x10
[ 955.658507][T11657] ? validate_chain+0x1fe/0xae0
[ 955.658783][T11657] ? rcu_read_lock_any_held+0x43/0xb0
[ 955.659066][T11657] ? validate_chain+0x1fe/0xae0
[ 955.659353][T11657] genl_family_rcv_msg+0x347/0x5b0
[ 955.659632][T11657] ? __pfx_genl_family_rcv_msg+0x10/0x10
[ 955.659904][T11657] ? __pfx_devlink_nl_pre_doit_dev_lock+0x10/0x10
[ 955.660244][T11657] ? __pfx_devlink_nl_reload_doit+0x10/0x10
[ 955.660595][T11657] ? __pfx_devlink_nl_post_doit_dev_lock+0x10/0x10
[ 955.660934][T11657] genl_rcv_msg+0xa3/0x140
[ 955.661220][T11657] netlink_rcv_skb+0x130/0x360
[ 955.661498][T11657] ? __pfx_genl_rcv_msg+0x10/0x10
[ 955.661772][T11657] ? __pfx_netlink_rcv_skb+0x10/0x10
[ 955.662054][T11657] ? genl_rcv+0x19/0x40
[ 955.662258][T11657] ? __pfx_down_read+0x10/0x10
[ 955.662552][T11657] ? netlink_deliver_tap+0x13e/0x340
[ 955.662827][T11657] genl_rcv+0x28/0x40
[ 955.663039][T11657] netlink_unicast+0x44b/0x710
[ 955.663312][T11657] ? __pfx_netlink_unicast+0x10/0x10
[ 955.663587][T11657] ? find_held_lock+0x2c/0x110
[ 955.663872][T11657] netlink_sendmsg+0x723/0xbe0
[ 955.664157][T11657] ? __pfx_netlink_sendmsg+0x10/0x10
[ 955.664444][T11657] ? lock_acquire+0x32/0xc0
[ 955.664718][T11657] ? __might_fault+0x11b/0x170
[ 955.664998][T11657] __sys_sendto+0x3c3/0x450
[ 955.665276][T11657] ? __pfx___sys_sendto+0x10/0x10
[ 955.665555][T11657] ? __pfx___lock_release+0x10/0x10
[ 955.665825][T11657] ? trace_lock_acquire+0x14c/0x1f0
[ 955.666101][T11657] ? rseq_update_cpu_node_id+0xe6/0x140
[ 955.666387][T11657] ? __rseq_handle_notify_resume+0x288/0x3e0
[ 955.666723][T11657] ? __pfx___rseq_handle_notify_resume+0x10/0x10
[ 955.667063][T11657] ? __pfx___x64_sys_openat+0x10/0x10
[ 955.667335][T11657] __x64_sys_sendto+0xe0/0x1c0
[ 955.667606][T11657] ? lockdep_hardirqs_on_prepare+0x275/0x410
[ 955.667938][T11657] do_syscall_64+0xc1/0x1d0
[ 955.668202][T11657] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 955.668541][T11657] RIP: 0033:0x7fd2f13b1a4a
[ 955.668818][T11657] Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89
[ 955.669764][T11657] RSP: 002b:00007ffefa4979b8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[ 955.670164][T11657] RAX: ffffffffffffffda RBX: 00000000331cfb00 RCX: 00007fd2f13b1a4a
[ 955.670577][T11657] RDX: 0000000000000040 RSI: 00000000331cfb00 RDI: 0000000000000005
[ 955.670985][T11657] RBP: 00000000331cf910 R08: 00007fd2f1555200 R09: 000000000000000c
[ 955.671351][T11657] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000040f200
[ 955.671622][T11657] R13: 00000000331cf910 R14: 0000000000000000 R15: 0000000000000001
[ 955.671898][T11657]
[ 1107.427499][ T43] INFO: task devlink:11657 blocked for more than 122 seconds.
[ 1107.427834][ T43] Not tainted 6.13.0-rc2-virtme #1
[ 1107.428004][ T43] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1107.428259][ T43] task:devlink state:D stack:23480 pid:11657 tgid:11657 ppid:11614 flags:0x00004002
[ 1107.428590][ T43] Call Trace:
[ 1107.428698][ T43]
[ 1107.428772][ T43] __schedule+0x6fc/0x1800
[ 1107.428930][ T43] ? __pfx___schedule+0x10/0x10
[ 1107.429112][ T43] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 1107.429251][ T43] ? trace_lock_release+0x10e/0x180
[ 1107.429388][ T43] schedule+0xdf/0x210
[ 1107.429501][ T43] kernfs_drain+0x2d3/0x460
[ 1107.429645][ T43] ? __pfx_kernfs_drain+0x10/0x10
[ 1107.429778][ T43] ? __pfx_autoremove_wake_function+0x10/0x10
[ 1107.429949][ T43] __kernfs_remove+0x1f9/0x590
[ 1107.430109][ T43] kernfs_remove_by_name_ns+0xa5/0x100
[ 1107.430248][ T43] remove_files+0x8c/0x1a0
[ 1107.430385][ T43] sysfs_remove_group+0x7b/0x160
[ 1107.430528][ T43] sysfs_remove_groups+0x53/0xa0
[ 1107.430668][ T43] device_remove_attrs+0x1db/0x260
[ 1107.430805][ T43] ? __pfx_device_remove_attrs+0x10/0x10
[ 1107.430941][ T43] device_del+0x337/0x8c0
[ 1107.431052][ T43] ? __pfx_device_del+0x10/0x10
[ 1107.431184][ T43] unregister_netdevice_many_notify+0x43d/0x1580
[ 1107.431353][ T43] ? __mutex_trylock_common+0xfa/0x260
[ 1107.431512][ T43] ? __pfx___mutex_trylock_common+0x10/0x10
[ 1107.431680][ T43] ? __pfx_unregister_netdevice_many_notify+0x10/0x10
[ 1107.431846][ T43] ? __mutex_lock+0x190/0xbc0
[ 1107.431985][ T43] ? nsim_destroy+0x6b/0x5f0 [netdevsim]
[ 1107.432144][ T43] ? __pfx___mutex_lock+0x10/0x10
[ 1107.432280][ T43] unregister_netdevice_queue+0x2a4/0x410
[ 1107.432414][ T43] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 1107.432557][ T43] ? __pfx_unregister_netdevice_queue+0x10/0x10
[ 1107.432728][ T43] nsim_destroy+0xe8/0x5f0 [netdevsim]
[ 1107.432875][ T43] __nsim_dev_port_del+0x17e/0x250 [netdevsim]
[ 1107.433059][ T43] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim]
[ 1107.433238][ T43] nsim_dev_reload_down+0x62/0xc0 [netdevsim]
[ 1107.433413][ T43] devlink_reload+0x172/0x450
[ 1107.433567][ T43] ? __pfx_devlink_reload+0x10/0x10
[ 1107.433702][ T43] ? fdget+0x52/0x1e0
[ 1107.433807][ T43] devlink_nl_reload_doit+0x395/0x10b0
[ 1107.433940][ T43] ? __pfx_devlink_nl_reload_doit+0x10/0x10
[ 1107.434111][ T43] ? __nla_parse+0x26/0x30
[ 1107.434249][ T43] genl_family_rcv_msg_doit+0x1d4/0x2b0
[ 1107.434383][ T43] ? __pfx_genl_family_rcv_msg_doit+0x10/0x10
[ 1107.434559][ T43] ? validate_chain+0x1fe/0xae0
[ 1107.434700][ T43] ? rcu_read_lock_any_held+0x43/0xb0
[ 1107.434834][ T43] ? validate_chain+0x1fe/0xae0
[ 1107.434967][ T43] genl_family_rcv_msg+0x347/0x5b0
[ 1107.435108][ T43] ? __pfx_genl_family_rcv_msg+0x10/0x10
[ 1107.435244][ T43] ? __pfx_devlink_nl_pre_doit_dev_lock+0x10/0x10
[ 1107.435411][ T43] ? __pfx_devlink_nl_reload_doit+0x10/0x10
[ 1107.435597][ T43] ? __pfx_devlink_nl_post_doit_dev_lock+0x10/0x10
[ 1107.435766][ T43] genl_rcv_msg+0xa3/0x140
[ 1107.435899][ T43] netlink_rcv_skb+0x130/0x360
[ 1107.436034][ T43] ? __pfx_genl_rcv_msg+0x10/0x10
[ 1107.436175][ T43] ? __pfx_netlink_rcv_skb+0x10/0x10
[ 1107.436312][ T43] ? genl_rcv+0x19/0x40
[ 1107.436412][ T43] ? __pfx_down_read+0x10/0x10
[ 1107.436557][ T43] ? netlink_deliver_tap+0x13e/0x340
[ 1107.436693][ T43] genl_rcv+0x28/0x40
[ 1107.436793][ T43] netlink_unicast+0x44b/0x710
[ 1107.436929][ T43] ? __pfx_netlink_unicast+0x10/0x10
[ 1107.437068][ T43] ? find_held_lock+0x2c/0x110
[ 1107.437203][ T43] netlink_sendmsg+0x723/0xbe0
[ 1107.437338][ T43] ? __pfx_netlink_sendmsg+0x10/0x10
[ 1107.437478][ T43] ? lock_acquire+0x32/0xc0
[ 1107.437609][ T43] ? __might_fault+0x11b/0x170
[ 1107.437747][ T43] __sys_sendto+0x3c3/0x450
[ 1107.437884][ T43] ? __pfx___sys_sendto+0x10/0x10
[ 1107.438017][ T43] ? __pfx___lock_release+0x10/0x10
[ 1107.438155][ T43] ? trace_lock_acquire+0x14c/0x1f0
[ 1107.438292][ T43] ? rseq_update_cpu_node_id+0xe6/0x140
[ 1107.438428][ T43] ? __rseq_handle_notify_resume+0x288/0x3e0
[ 1107.438603][ T43] ? __pfx___rseq_handle_notify_resume+0x10/0x10
[ 1107.438770][ T43] ? __pfx___x64_sys_openat+0x10/0x10
[ 1107.438906][ T43] __x64_sys_sendto+0xe0/0x1c0
[ 1107.439044][ T43] ? lockdep_hardirqs_on_prepare+0x275/0x410
[ 1107.439209][ T43] do_syscall_64+0xc1/0x1d0
[ 1107.439342][ T43] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 1107.439517][ T43] RIP: 0033:0x7fd2f13b1a4a
[ 1107.439661][ T43] RSP: 002b:00007ffefa4979b8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[ 1107.439863][ T43] RAX: ffffffffffffffda RBX: 00000000331cfb00 RCX: 00007fd2f13b1a4a
[ 1107.440068][ T43] RDX: 0000000000000040 RSI: 00000000331cfb00 RDI: 0000000000000005
[ 1107.440318][ T43] RBP: 00000000331cf910 R08: 00007fd2f1555200 R09: 000000000000000c
[ 1107.440530][ T43] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000040f200
[ 1107.440730][ T43] R13: 00000000331cf910 R14: 0000000000000000 R15: 0000000000000001
[ 1107.440939][ T43]
[ 1107.441046][ T43] INFO: lockdep is turned off.