[ 12.261784][ T252] netdevsim netdevsim1337 eni1337np1: renamed from eth0
[ 12.440059][ T257] ip (257) used greatest stack depth: 24304 bytes left
[ 14.233825][ T293] devlink (293) used greatest stack depth: 24064 bytes left
[ 306.864751][ T1144] netdevsim: Exceeded number of supported nexthops
[ 309.255751][ T1190] netdevsim: Failed to replace nexthop bucket
[ 309.256100][ T1190] netdevsim: Failed to replace nexthop bucket
[ 311.803117][ T1245] Failed to register nexthop notifier
[ 312.465506][ T228]
[ 312.465732][ T228] ======================================================
[ 312.466066][ T228] WARNING: possible circular locking dependency detected
[ 312.466418][ T228] 6.13.0-rc2-virtme #1 Not tainted
[ 312.466725][ T228] ------------------------------------------------------
[ 312.467085][ T228] nexthop.sh/228 is trying to acquire lock:
[ 312.467473][ T228] ffff88800924fc30 (kn->active#25){++++}-{0:0}, at: __kernfs_remove+0x1f9/0x590
[ 312.468038][ T228]
[ 312.468038][ T228] but task is already holding lock:
[ 312.468498][ T228] ffffffffb9de1248 (rtnl_mutex){+.+.}-{4:4}, at: nsim_destroy+0x6b/0x5f0 [netdevsim]
[ 312.469023][ T228]
[ 312.469023][ T228] which lock already depends on the new lock.
[ 312.469023][ T228]
[ 312.469606][ T228]
[ 312.469606][ T228] the existing dependency chain (in reverse order) is:
[ 312.470145][ T228]
[ 312.470145][ T228] -> #1 (rtnl_mutex){+.+.}-{4:4}:
[ 312.470611][ T228] __lock_acquire+0xb9a/0x1680
[ 312.470923][ T228] lock_acquire.part.0+0xeb/0x330
[ 312.471221][ T228] __mutex_lock+0x169/0xbc0
[ 312.471532][ T228] phys_port_name_show+0xbe/0x200
[ 312.471838][ T228] dev_attr_show+0x43/0xc0
[ 312.472146][ T228] sysfs_kf_seq_show+0x1fb/0x3d0
[ 312.472446][ T228] seq_read_iter+0x40e/0x1090
[ 312.472745][ T228] vfs_read+0x755/0xce0
[ 312.473036][ T228] ksys_read+0xf8/0x1d0
[ 312.473304][ T228] do_syscall_64+0xc1/0x1d0
[ 312.473581][ T228] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 312.473919][ T228]
[ 312.473919][ T228] -> #0 (kn->active#25){++++}-{0:0}:
[ 312.474372][ T228] check_prev_add+0x182/0xce0
[ 312.474685][ T228] validate_chain+0x7bd/0xae0
[ 312.474984][ T228] __lock_acquire+0xb9a/0x1680
[ 312.475289][ T228] lock_acquire.part.0+0xeb/0x330
[ 312.475618][ T228] kernfs_drain+0x384/0x460
[ 312.475937][ T228] __kernfs_remove+0x1f9/0x590
[ 312.476233][ T228] kernfs_remove_by_name_ns+0xa5/0x100
[ 312.476615][ T228] remove_files+0x8c/0x1a0
[ 312.476909][ T228] sysfs_remove_group+0x7b/0x160
[ 312.477211][ T228] sysfs_remove_groups+0x53/0xa0
[ 312.477490][ T228] device_remove_attrs+0x1db/0x260
[ 312.477788][ T228] device_del+0x337/0x8c0
[ 312.478072][ T228] unregister_netdevice_many_notify+0x43d/0x1580
[ 312.478434][ T228] unregister_netdevice_queue+0x2a4/0x410
[ 312.478782][ T228] nsim_destroy+0xe8/0x5f0 [netdevsim]
[ 312.479204][ T228] __nsim_dev_port_del+0x17e/0x250 [netdevsim]
[ 312.479601][ T228] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim]
[ 312.479963][ T228] nsim_drv_remove+0x51/0x1d0 [netdevsim]
[ 312.480391][ T228] device_release_driver_internal+0x3bf/0x590
[ 312.480770][ T228] bus_remove_device+0x1f1/0x3f0
[ 312.481073][ T228] device_del+0x33f/0x8c0
[ 312.481364][ T228] device_unregister+0x17/0xb0
[ 312.481661][ T228] del_device_store+0x2f3/0x4f0 [netdevsim]
[ 312.482044][ T228] kernfs_fop_write_iter+0x2ba/0x460
[ 312.482414][ T228] vfs_write+0xa81/0x11e0
[ 312.482721][ T228] ksys_write+0xf8/0x1d0
[ 312.483013][ T228] do_syscall_64+0xc1/0x1d0
[ 312.483306][ T228] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 312.483669][ T228]
[ 312.483669][ T228] other info that might help us debug this:
[ 312.483669][ T228]
[ 312.484039][ T228] Possible unsafe locking scenario:
[ 312.484039][ T228]
[ 312.484333][ T228] CPU0 CPU1
[ 312.484528][ T228] ---- ----
[ 312.484727][ T228] lock(rtnl_mutex);
[ 312.484873][ T228] lock(kn->active#25);
[ 312.485114][ T228] lock(rtnl_mutex);
[ 312.485365][ T228] lock(kn->active#25);
[ 312.485511][ T228]
[ 312.485511][ T228] *** DEADLOCK ***
[ 312.485511][ T228]
[ 312.485792][ T228] 7 locks held by nexthop.sh/228:
[ 312.485988][ T228] #0: ffff88800a3f13f0 (sb_writers#8){.+.+}-{0:0}, at: ksys_write+0xf8/0x1d0
[ 312.486339][ T228] #1: ffff888009200e88 (&of->mutex){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x20b/0x460
[ 312.486672][ T228] #2: ffff888009092370 (kn->active#34){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x22e/0x460
[ 312.487052][ T228] #3: ffffffffc03695e8 (nsim_bus_dev_list_lock){+.+.}-{4:4}, at: del_device_store+0x9f/0x4f0 [netdevsim]
[ 312.487472][ T228] #4: ffff888005e120e8 (&dev->mutex){....}-{4:4}, at: device_release_driver_internal+0xa7/0x590
[ 312.487838][ T228] #5: ffff888005e13a50 (&devlink->lock_key){+.+.}-{4:4}, at: nsim_drv_remove+0x49/0x1d0 [netdevsim]
[ 312.488210][ T228] #6: ffffffffb9de1248 (rtnl_mutex){+.+.}-{4:4}, at: nsim_destroy+0x6b/0x5f0 [netdevsim]
[ 312.488694][ T228]
[ 312.488694][ T228] stack backtrace:
[ 312.488916][ T228] CPU: 3 UID: 0 PID: 228 Comm: nexthop.sh Not tainted 6.13.0-rc2-virtme #1
[ 312.489259][ T228] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 312.489489][ T228] Call Trace:
[ 312.489623][ T228]
[ 312.489716][ T228] dump_stack_lvl+0x82/0xd0
[ 312.489940][ T228] print_circular_bug+0x2e2/0x4b0
[ 312.490150][ T228] check_noncircular+0x2f9/0x3e0
[ 312.490344][ T228] ? __pfx_check_noncircular+0x10/0x10
[ 312.490526][ T228] ? __kernel_text_address+0x12/0x40
[ 312.490709][ T228] ? unwind_get_return_address+0x5e/0xa0
[ 312.490910][ T228] ? __pfx_stack_trace_consume_entry+0x10/0x10
[ 312.491148][ T228] ? arch_stack_walk+0xa2/0xf0
[ 312.491351][ T228] ? alloc_chain_hlocks+0x4e0/0x720
[ 312.491583][ T228] check_prev_add+0x182/0xce0
[ 312.491764][ T228] ? __pfx_stack_trace_save+0x10/0x10
[ 312.491944][ T228] validate_chain+0x7bd/0xae0
[ 312.492120][ T228] ? __pfx_validate_chain+0x10/0x10
[ 312.492322][ T228] ? mark_lock+0x335/0x3e0
[ 312.492498][ T228] __lock_acquire+0xb9a/0x1680
[ 312.492670][ T228] ? kasan_addr_to_slab+0x40/0x80
[ 312.492858][ T228] lock_acquire.part.0+0xeb/0x330
[ 312.493041][ T228] ? __kernfs_remove+0x1f9/0x590
[ 312.493232][ T228] ? __pfx___lock_release+0x10/0x10
[ 312.493417][ T228] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 312.493597][ T228] ? trace_lock_acquire+0x14c/0x1f0
[ 312.493776][ T228] ? __virt_addr_valid+0x22b/0x430
[ 312.493955][ T228] ? lock_acquire+0x32/0xc0
[ 312.494165][ T228] ? __kernfs_remove+0x1f9/0x590
[ 312.494458][ T228] kernfs_drain+0x384/0x460
[ 312.494664][ T228] ? __kernfs_remove+0x1f9/0x590
[ 312.494849][ T228] ? __pfx_kernfs_drain+0x10/0x10
[ 312.495036][ T228] ? lock_acquire+0x32/0xc0
[ 312.495238][ T228] __kernfs_remove+0x1f9/0x590
[ 312.495430][ T228] kernfs_remove_by_name_ns+0xa5/0x100
[ 312.495621][ T228] remove_files+0x8c/0x1a0
[ 312.495805][ T228] sysfs_remove_group+0x7b/0x160
[ 312.496006][ T228] sysfs_remove_groups+0x53/0xa0
[ 312.496198][ T228] device_remove_attrs+0x1db/0x260
[ 312.496375][ T228] ? __pfx_device_remove_attrs+0x10/0x10
[ 312.496554][ T228] device_del+0x337/0x8c0
[ 312.496687][ T228] ? __pfx_device_del+0x10/0x10
[ 312.496883][ T228] unregister_netdevice_many_notify+0x43d/0x1580
[ 312.497104][ T228] ? __pfx_unregister_netdevice_many_notify+0x10/0x10
[ 312.497333][ T228] ? lockdep_hardirqs_on_prepare+0x275/0x410
[ 312.497548][ T228] ? __mutex_lock+0x34d/0xbc0
[ 312.497738][ T228] ? nsim_destroy+0x6b/0x5f0 [netdevsim]
[ 312.497938][ T228] ? __pfx___mutex_lock+0x10/0x10
[ 312.498119][ T228] unregister_netdevice_queue+0x2a4/0x410
[ 312.498392][ T228] ? __pfx_unregister_netdevice_queue+0x10/0x10
[ 312.498657][ T228] nsim_destroy+0xe8/0x5f0 [netdevsim]
[ 312.498843][ T228] __nsim_dev_port_del+0x17e/0x250 [netdevsim]
[ 312.499099][ T228] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim]
[ 312.499357][ T228] nsim_drv_remove+0x51/0x1d0 [netdevsim]
[ 312.499551][ T228] device_release_driver_internal+0x3bf/0x590
[ 312.499791][ T228] ? klist_put+0xb1/0x170
[ 312.499927][ T228] bus_remove_device+0x1f1/0x3f0
[ 312.500113][ T228] device_del+0x33f/0x8c0
[ 312.500255][ T228] ? __pfx_device_del+0x10/0x10
[ 312.500427][ T228] ? lock_acquire.part.0+0xeb/0x330
[ 312.500600][ T228] ? kernfs_fop_write_iter+0x22e/0x460
[ 312.500826][ T228] device_unregister+0x17/0xb0
[ 312.501001][ T228] del_device_store+0x2f3/0x4f0 [netdevsim]
[ 312.501270][ T228] ? __pfx_del_device_store+0x10/0x10 [netdevsim]
[ 312.501515][ T228] ? __pfx_sysfs_kf_write+0x10/0x10
[ 312.501689][ T228] ? sysfs_file_ops+0x11e/0x170
[ 312.501895][ T228] ? __pfx_sysfs_kf_write+0x10/0x10
[ 312.502092][ T228] kernfs_fop_write_iter+0x2ba/0x460
[ 312.502301][ T228] vfs_write+0xa81/0x11e0
[ 312.502451][ T228] ? __pfx_vfs_write+0x10/0x10
[ 312.502644][ T228] ? __lock_release+0x103/0x460
[ 312.502824][ T228] ? do_user_addr_fault+0x972/0xe30
[ 312.503015][ T228] ? __pfx___up_read+0x10/0x10
[ 312.503201][ T228] ksys_write+0xf8/0x1d0
[ 312.503357][ T228] ? __pfx_ksys_write+0x10/0x10
[ 312.503536][ T228] ? do_user_addr_fault+0x97c/0xe30
[ 312.503709][ T228] do_syscall_64+0xc1/0x1d0
[ 312.503899][ T228] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 312.504130][ T228] RIP: 0033:0x7f954fc2ab77
[ 312.504337][ T228] Code: 0b 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[ 312.504971][ T228] RSP: 002b:00007fffe8f67518 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 312.505256][ T228] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f954fc2ab77
[ 312.505527][ T228] RDX: 0000000000000005 RSI: 000055d669ed2c90 RDI: 0000000000000001
[ 312.505785][ T228] RBP: 000055d669ed2c90 R08: 0000000000000000 R09: 00007f954fc9d4e0
[ 312.506074][ T228] R10: 00007f954fc9d3e0 R11: 0000000000000246 R12: 0000000000000005
[ 312.506476][ T228] R13: 00007f954fce6760 R14: 0000000000000005 R15: 00007f954fce19c0
[ 312.506760][ T228]
[ 493.041424][ T43] INFO: task kworker/u16:0:11 blocked for more than 122 seconds.
[ 493.041787][ T43] Not tainted 6.13.0-rc2-virtme #1
[ 493.041963][ T43] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 493.042210][ T43] task:kworker/u16:0 state:D stack:26224 pid:11 tgid:11 ppid:2 flags:0x00004000
[ 493.042547][ T43] Workqueue: netns cleanup_net
[ 493.042728][ T43] Call Trace:
[ 493.042852][ T43]
[ 493.042949][ T43] __schedule+0x6fc/0x1800
[ 493.043119][ T43] ? __pfx___schedule+0x10/0x10
[ 493.043282][ T43] ? trace_lock_release+0x10e/0x180
[ 493.043449][ T43] ? trace_irq_enable.constprop.0+0xe9/0x150
[ 493.043644][ T43] ? trace_lock_release+0x10e/0x180
[ 493.043806][ T43] schedule+0xdf/0x210
[ 493.043927][ T43] schedule_preempt_disabled+0x15/0x30
[ 493.044085][ T43] __mutex_lock+0x6d3/0xbc0
[ 493.044239][ T43] ? do_raw_spin_lock+0x131/0x270
[ 493.044423][ T43] ? default_device_exit_batch+0x81/0x2e0
[ 493.044580][ T43] ? __pfx___mutex_lock+0x10/0x10
[ 493.044743][ T43] ? __wake_up+0x44/0x60
[ 493.044872][ T43] ? default_device_exit_batch+0x81/0x2e0
[ 493.045021][ T43] default_device_exit_batch+0x81/0x2e0
[ 493.045181][ T43] ? __pfx_default_device_exit_batch+0x10/0x10
[ 493.045382][ T43] ? mutex_is_locked+0x1c/0x60
[ 493.045542][ T43] ? nexthop_net_exit_batch_rtnl+0x83/0x210
[ 493.045735][ T43] cleanup_net+0x4ef/0xa50
[ 493.045892][ T43] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 493.046041][ T43] ? __pfx_cleanup_net+0x10/0x10
[ 493.046196][ T43] ? trace_lock_acquire+0x14c/0x1f0
[ 493.046349][ T43] ? lock_acquire+0x32/0xc0
[ 493.046520][ T43] ? process_one_work+0xe0b/0x16d0
[ 493.046689][ T43] process_one_work+0xe55/0x16d0
[ 493.046852][ T43] ? __pfx___lock_release+0x10/0x10
[ 493.047000][ T43] ? __pfx_process_one_work+0x10/0x10
[ 493.047153][ T43] ? assign_work+0x16c/0x240
[ 493.047308][ T43] worker_thread+0x58c/0xce0
[ 493.047488][ T43] ? __pfx_worker_thread+0x10/0x10
[ 493.047640][ T43] kthread+0x28a/0x350
[ 493.047767][ T43] ? __pfx_kthread+0x10/0x10
[ 493.047921][ T43] ret_from_fork+0x31/0x70
[ 493.048076][ T43] ? __pfx_kthread+0x10/0x10
[ 493.048230][ T43] ret_from_fork_asm+0x1a/0x30
[ 493.048419][ T43]
[ 493.048569][ T43] INFO: task nexthop.sh:228 blocked for more than 122 seconds.
[ 493.048784][ T43] Not tainted 6.13.0-rc2-virtme #1
[ 493.048941][ T43] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 493.049217][ T43] task:nexthop.sh state:D stack:24832 pid:228 tgid:228 ppid:227 flags:0x00004002
[ 493.049543][ T43] Call Trace:
[ 493.049667][ T43]
[ 493.049753][ T43] __schedule+0x6fc/0x1800
[ 493.049923][ T43] ? __pfx___schedule+0x10/0x10
[ 493.050089][ T43] ? __pfx_do_raw_spin_lock+0x10/0x10
[ 493.050274][ T43] ? trace_lock_release+0x10e/0x180
[ 493.050470][ T43] schedule+0xdf/0x210
[ 493.050597][ T43] kernfs_drain+0x2d3/0x460
[ 493.050759][ T43] ? __pfx_kernfs_drain+0x10/0x10
[ 493.050918][ T43] ? __pfx_autoremove_wake_function+0x10/0x10
[ 493.051119][ T43] __kernfs_remove+0x1f9/0x590
[ 493.051282][ T43] kernfs_remove_by_name_ns+0xa5/0x100
[ 493.051456][ T43] remove_files+0x8c/0x1a0
[ 493.051610][ T43] sysfs_remove_group+0x7b/0x160
[ 493.051762][ T43] sysfs_remove_groups+0x53/0xa0
[ 493.051909][ T43] device_remove_attrs+0x1db/0x260
[ 493.052058][ T43] ? __pfx_device_remove_attrs+0x10/0x10
[ 493.052211][ T43] device_del+0x337/0x8c0
[ 493.052331][ T43] ? __pfx_device_del+0x10/0x10
[ 493.052506][ T43] unregister_netdevice_many_notify+0x43d/0x1580
[ 493.052705][ T43] ? __pfx_unregister_netdevice_many_notify+0x10/0x10
[ 493.052889][ T43] ? lockdep_hardirqs_on_prepare+0x275/0x410
[ 493.053070][ T43] ? __mutex_lock+0x34d/0xbc0
[ 493.053221][ T43] ? nsim_destroy+0x6b/0x5f0 [netdevsim]
[ 493.053394][ T43] ? __pfx___mutex_lock+0x10/0x10
[ 493.053546][ T43] unregister_netdevice_queue+0x2a4/0x410
[ 493.053703][ T43] ? __pfx_unregister_netdevice_queue+0x10/0x10
[ 493.053889][ T43] nsim_destroy+0xe8/0x5f0 [netdevsim]
[ 493.054084][ T43] __nsim_dev_port_del+0x17e/0x250 [netdevsim]
[ 493.054318][ T43] nsim_dev_reload_destroy+0xe0/0x470 [netdevsim]
[ 493.054531][ T43] nsim_drv_remove+0x51/0x1d0 [netdevsim]
[ 493.054696][ T43] device_release_driver_internal+0x3bf/0x590
[ 493.054887][ T43] ? klist_put+0xb1/0x170
[ 493.055011][ T43] bus_remove_device+0x1f1/0x3f0
[ 493.055160][ T43] device_del+0x33f/0x8c0
[ 493.055284][ T43] ? __pfx_device_del+0x10/0x10
[ 493.055456][ T43] ? lock_acquire.part.0+0xeb/0x330
[ 493.055604][ T43] ? kernfs_fop_write_iter+0x22e/0x460
[ 493.055755][ T43] device_unregister+0x17/0xb0
[ 493.055900][ T43] del_device_store+0x2f3/0x4f0 [netdevsim]
[ 493.056102][ T43] ? __pfx_del_device_store+0x10/0x10 [netdevsim]
[ 493.056291][ T43] ? __pfx_sysfs_kf_write+0x10/0x10
[ 493.056465][ T43] ? sysfs_file_ops+0x11e/0x170
[ 493.056617][ T43] ? __pfx_sysfs_kf_write+0x10/0x10
[ 493.056766][ T43] kernfs_fop_write_iter+0x2ba/0x460
[ 493.056917][ T43] vfs_write+0xa81/0x11e0
[ 493.057038][ T43] ? __pfx_vfs_write+0x10/0x10
[ 493.057195][ T43] ? __lock_release+0x103/0x460
[ 493.057343][ T43] ? do_user_addr_fault+0x972/0xe30
[ 493.057506][ T43] ? __pfx___up_read+0x10/0x10
[ 493.057657][ T43] ksys_write+0xf8/0x1d0
[ 493.057774][ T43] ? __pfx_ksys_write+0x10/0x10
[ 493.057920][ T43] ? do_user_addr_fault+0x97c/0xe30
[ 493.058072][ T43] do_syscall_64+0xc1/0x1d0
[ 493.058233][ T43] entry_SYSCALL_64_after_hwframe+0x77/0x7f
[ 493.058440][ T43] RIP: 0033:0x7f954fc2ab77
[ 493.058641][ T43] RSP: 002b:00007fffe8f67518 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 493.058884][ T43] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f954fc2ab77
[ 493.059122][ T43] RDX: 0000000000000005 RSI: 000055d669ed2c90 RDI: 0000000000000001
[ 493.059360][ T43] RBP: 000055d669ed2c90 R08: 0000000000000000 R09: 00007f954fc9d4e0
[ 493.059578][ T43] R10: 00007f954fc9d3e0 R11: 0000000000000246 R12: 0000000000000005
[ 493.059799][ T43] R13: 00007f954fce6760 R14: 0000000000000005 R15: 00007f954fce19c0
[ 493.060022][ T43]
[ 493.060144][ T43] INFO: lockdep is turned off.