[ 4392.065384][ T2441] pktgen: Packet Generator for packet performance testing. Version: 2.75 [ 4548.161824][ T44] INFO: task modprobe:2441 blocked for more than 122 seconds. [ 4548.162311][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 4548.162518][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4548.162931][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 4548.163449][ T44] Call Trace: [ 4548.163614][ T44] [ 4548.163745][ T44] __schedule+0x880/0x1b10 [ 4548.163999][ T44] ? io_schedule_timeout+0x150/0x150 [ 4548.164230][ T44] ? schedule+0x1ee/0x270 [ 4548.164390][ T44] ? __lock_release+0x5d/0x170 [ 4548.164613][ T44] schedule+0xe2/0x270 [ 4548.164832][ T44] schedule_preempt_disabled+0x18/0x30 [ 4548.165055][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 4548.165271][ T44] ? __lock_acquire+0x449/0x7e0 [ 4548.165487][ T44] ? down_read+0x4a0/0x4a0 [ 4548.165709][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.165954][ T44] ? lock_acquire+0x104/0x160 [ 4548.166170][ T44] ? register_pernet_subsys+0x1a/0x40 [ 4548.166392][ T44] down_write+0x1d6/0x200 [ 4548.166556][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 4548.166855][ T44] ? find_held_lock+0x2b/0x80 [ 4548.167073][ T44] ? __create_object+0x5e/0xb0 [ 4548.167289][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 4548.167510][ T44] register_pernet_subsys+0x1a/0x40 [ 4548.167724][ T44] pg_init+0x2c/0x1000 [pktgen] [ 4548.167961][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 4548.168189][ T44] do_one_initcall+0x8c/0x1d0 [ 4548.168425][ T44] ? trace_initcall_level+0x130/0x130 [ 4548.169038][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.169281][ T44] ? trace_kmalloc+0x26/0xd0 [ 4548.169508][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 4548.169719][ T44] ? debug_objects_oom+0x160/0x160 [ 4548.169959][ T44] ? kasan_unpoison+0x27/0x50 [ 4548.170178][ T44] ? __asan_register_globals+0x5d/0x70 [ 4548.170398][ T44] do_init_module+0x27a/0x860 [ 4548.170620][ T44] ? free_module+0x2f0/0x2f0 [ 4548.170856][ T44] ? kfree+0x144/0x320 [ 4548.171018][ T44] ? find_vm_area+0x12/0x50 [ 4548.171238][ T44] load_module+0x1f40/0x2dd0 [ 4548.171463][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4548.171727][ T44] ? kernel_read_file+0x3ed/0x550 [ 4548.171970][ T44] ? kernel_read_file+0x3c8/0x550 [ 4548.172193][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4548.172413][ T44] ? init_module_from_file+0xe9/0x150 [ 4548.172628][ T44] init_module_from_file+0xe9/0x150 [ 4548.172877][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4548.173129][ T44] ? idempotent_init_module+0x316/0x610 [ 4548.173342][ T44] ? __lock_release+0x5d/0x170 [ 4548.173561][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4548.173805][ T44] idempotent_init_module+0x331/0x610 [ 4548.174028][ T44] ? init_module_from_file+0x150/0x150 [ 4548.174255][ T44] ? cap_capable+0x94/0x230 [ 4548.174474][ T44] __x64_sys_finit_module+0xca/0x150 [ 4548.174687][ T44] ? do_syscall_64+0x85/0x300 [ 4548.174926][ T44] do_syscall_64+0xc1/0x300 [ 4548.175143][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4548.175421][ T44] RIP: 0033:0x7fc9db13ba0d [ 4548.175649][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4548.176002][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 4548.176364][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 4548.176696][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 4548.177080][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 4548.177408][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 4548.177757][ T44] [ 4548.177942][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 4548.178408][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 4548.178959][ T44] Workqueue: netns cleanup_net [ 4548.179181][ T44] Call Trace: [ 4548.179343][ T44] [ 4548.179463][ T44] ? update_curr+0x39f/0x500 [ 4548.179681][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 4548.179934][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.180155][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.180369][ T44] ? trace_contention_end+0xd8/0x140 [ 4548.180585][ T44] ? __mutex_lock+0x19f/0x11a0 [ 4548.180838][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 4548.181066][ T44] ? irqentry_exit+0x3b/0x80 [ 4548.181280][ T44] ? lockdep_hardirqs_on+0x7c/0x110 [ 4548.181507][ T44] ? nf_ct_iterate_cleanup+0x6d/0x340 [nf_conntrack] [ 4548.181815][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 4548.182150][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 4548.182481][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 4548.182843][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 4548.183137][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 4548.183432][ T44] ? ops_undo_list+0x2bf/0x890 [ 4548.183653][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 4548.183893][ T44] ? cleanup_net+0x2c5/0x890 [ 4548.184116][ T44] ? cleanup_net+0x3a1/0x890 [ 4548.184332][ T44] ? net_passive_dec+0x190/0x190 [ 4548.184549][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.184800][ T44] ? process_one_work+0xe35/0x1650 [ 4548.185036][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 4548.185258][ T44] ? assign_work+0x168/0x240 [ 4548.185474][ T44] ? worker_thread+0x591/0xcf0 [ 4548.185697][ T44] ? rescuer_thread+0xd10/0xd10 [ 4548.185937][ T44] ? kthread+0x37b/0x5f0 [ 4548.186102][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4548.186314][ T44] ? ret_from_fork+0x1b/0x270 [ 4548.186529][ T44] ? __lock_release+0x5d/0x170 [ 4548.186760][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.186988][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4548.187205][ T44] ? ret_from_fork+0x1db/0x270 [ 4548.187419][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4548.187633][ T44] ? ret_from_fork_asm+0x11/0x20 [ 4548.187884][ T44] [ 4548.188074][ T44] [ 4548.188074][ T44] Showing all locks held in the system: [ 4548.188381][ T44] 4 locks held by kworker/u16:0/12: [ 4548.188606][ T44] 1 lock held by khungtaskd/44: [ 4548.189216][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4548.189609][ T44] 1 lock held by modprobe/2441: [ 4548.189840][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4548.190254][ T44] [ 4548.190367][ T44] ============================================= [ 4548.190367][ T44] [ 4671.042811][ T44] INFO: task modprobe:2441 blocked for more than 245 seconds. [ 4671.043262][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 4671.043464][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4671.043865][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 4671.044358][ T44] Call Trace: [ 4671.044521][ T44] [ 4671.044636][ T44] __schedule+0x880/0x1b10 [ 4671.044874][ T44] ? io_schedule_timeout+0x150/0x150 [ 4671.045102][ T44] ? schedule+0x1ee/0x270 [ 4671.045266][ T44] ? __lock_release+0x5d/0x170 [ 4671.045488][ T44] schedule+0xe2/0x270 [ 4671.045650][ T44] schedule_preempt_disabled+0x18/0x30 [ 4671.045881][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 4671.046100][ T44] ? __lock_acquire+0x449/0x7e0 [ 4671.046313][ T44] ? down_read+0x4a0/0x4a0 [ 4671.046536][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.046759][ T44] ? lock_acquire+0x104/0x160 [ 4671.046971][ T44] ? register_pernet_subsys+0x1a/0x40 [ 4671.047194][ T44] down_write+0x1d6/0x200 [ 4671.047364][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 4671.047630][ T44] ? find_held_lock+0x2b/0x80 [ 4671.047871][ T44] ? __create_object+0x5e/0xb0 [ 4671.048091][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 4671.048310][ T44] register_pernet_subsys+0x1a/0x40 [ 4671.048525][ T44] pg_init+0x2c/0x1000 [pktgen] [ 4671.048749][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 4671.048971][ T44] do_one_initcall+0x8c/0x1d0 [ 4671.049187][ T44] ? trace_initcall_level+0x130/0x130 [ 4671.049403][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.049616][ T44] ? trace_kmalloc+0x26/0xd0 [ 4671.049838][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 4671.050053][ T44] ? debug_objects_oom+0x160/0x160 [ 4671.050267][ T44] ? kasan_unpoison+0x27/0x50 [ 4671.050483][ T44] ? __asan_register_globals+0x5d/0x70 [ 4671.050702][ T44] do_init_module+0x27a/0x860 [ 4671.050936][ T44] ? free_module+0x2f0/0x2f0 [ 4671.051152][ T44] ? kfree+0x144/0x320 [ 4671.051313][ T44] ? find_vm_area+0x12/0x50 [ 4671.051534][ T44] load_module+0x1f40/0x2dd0 [ 4671.051768][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4671.052042][ T44] ? kernel_read_file+0x3ed/0x550 [ 4671.052268][ T44] ? kernel_read_file+0x3c8/0x550 [ 4671.052489][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4671.052710][ T44] ? init_module_from_file+0xe9/0x150 [ 4671.052947][ T44] init_module_from_file+0xe9/0x150 [ 4671.053163][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4671.053393][ T44] ? idempotent_init_module+0x316/0x610 [ 4671.053605][ T44] ? __lock_release+0x5d/0x170 [ 4671.053835][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4671.054053][ T44] idempotent_init_module+0x331/0x610 [ 4671.054268][ T44] ? init_module_from_file+0x150/0x150 [ 4671.054495][ T44] ? cap_capable+0x94/0x230 [ 4671.054713][ T44] __x64_sys_finit_module+0xca/0x150 [ 4671.054935][ T44] ? do_syscall_64+0x85/0x300 [ 4671.055152][ T44] do_syscall_64+0xc1/0x300 [ 4671.055374][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4671.055638][ T44] RIP: 0033:0x7fc9db13ba0d [ 4671.055866][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4671.056191][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 4671.056510][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 4671.056858][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 4671.057180][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 4671.057501][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 4671.057844][ T44] [ 4671.058010][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 4671.058467][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 4671.058997][ T44] Workqueue: netns cleanup_net [ 4671.059215][ T44] Call Trace: [ 4671.059375][ T44] [ 4671.059490][ T44] ? __mutex_trylock_common+0xf9/0x260 [ 4671.059715][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 4671.059939][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.060157][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.060370][ T44] ? trace_contention_end+0xd8/0x140 [ 4671.060584][ T44] ? __mutex_lock+0x19f/0x11a0 [ 4671.060820][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 4671.061044][ T44] ? ww_mutex_lock+0x160/0x160 [ 4671.061258][ T44] ? irqentry_exit+0x3b/0x80 [ 4671.061479][ T44] ? lockdep_hardirqs_on+0x7c/0x110 [ 4671.061706][ T44] ? nf_ct_iterate_cleanup+0x92/0x340 [nf_conntrack] [ 4671.062002][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 4671.062333][ T44] ? schedule+0xe2/0x270 [ 4671.062496][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 4671.062837][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 4671.063176][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 4671.063454][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 4671.063744][ T44] ? ops_undo_list+0x2bf/0x890 [ 4671.063963][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 4671.064175][ T44] ? cleanup_net+0x2c5/0x890 [ 4671.064397][ T44] ? cleanup_net+0x3a1/0x890 [ 4671.064616][ T44] ? net_passive_dec+0x190/0x190 [ 4671.064857][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.065077][ T44] ? process_one_work+0xe35/0x1650 [ 4671.065301][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 4671.065522][ T44] ? assign_work+0x168/0x240 [ 4671.065746][ T44] ? worker_thread+0x591/0xcf0 [ 4671.065970][ T44] ? rescuer_thread+0xd10/0xd10 [ 4671.066186][ T44] ? kthread+0x37b/0x5f0 [ 4671.066356][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4671.066568][ T44] ? ret_from_fork+0x1b/0x270 [ 4671.066789][ T44] ? __lock_release+0x5d/0x170 [ 4671.067006][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.067220][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4671.067435][ T44] ? ret_from_fork+0x1db/0x270 [ 4671.067646][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4671.067866][ T44] ? ret_from_fork_asm+0x11/0x20 [ 4671.068099][ T44] [ 4671.068283][ T44] [ 4671.068283][ T44] Showing all locks held in the system: [ 4671.068586][ T44] 4 locks held by kworker/u16:0/12: [ 4671.068947][ T44] 1 lock held by khungtaskd/44: [ 4671.069153][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4671.069541][ T44] 1 lock held by modprobe/2441: [ 4671.069756][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4671.070169][ T44] [ 4671.070273][ T44] ============================================= [ 4671.070273][ T44] [ 4793.921880][ T44] INFO: task modprobe:2441 blocked for more than 368 seconds. [ 4793.922530][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 4793.922962][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4793.923568][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 4793.924424][ T44] Call Trace: [ 4793.924681][ T44] [ 4793.924874][ T44] __schedule+0x880/0x1b10 [ 4793.925233][ T44] ? io_schedule_timeout+0x150/0x150 [ 4793.925561][ T44] ? schedule+0x1ee/0x270 [ 4793.925831][ T44] ? __lock_release+0x5d/0x170 [ 4793.926192][ T44] schedule+0xe2/0x270 [ 4793.926480][ T44] schedule_preempt_disabled+0x18/0x30 [ 4793.926893][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 4793.927288][ T44] ? __lock_acquire+0x449/0x7e0 [ 4793.927654][ T44] ? down_read+0x4a0/0x4a0 [ 4793.928071][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.928448][ T44] ? lock_acquire+0x104/0x160 [ 4793.928829][ T44] ? register_pernet_subsys+0x1a/0x40 [ 4793.929222][ T44] down_write+0x1d6/0x200 [ 4793.929501][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 4793.929965][ T44] ? find_held_lock+0x2b/0x80 [ 4793.930297][ T44] ? __create_object+0x5e/0xb0 [ 4793.930629][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 4793.931042][ T44] register_pernet_subsys+0x1a/0x40 [ 4793.931442][ T44] pg_init+0x2c/0x1000 [pktgen] [ 4793.931838][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 4793.932222][ T44] do_one_initcall+0x8c/0x1d0 [ 4793.932635][ T44] ? trace_initcall_level+0x130/0x130 [ 4793.933031][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.933400][ T44] ? trace_kmalloc+0x26/0xd0 [ 4793.933794][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 4793.934141][ T44] ? debug_objects_oom+0x160/0x160 [ 4793.934461][ T44] ? kasan_unpoison+0x27/0x50 [ 4793.934868][ T44] ? __asan_register_globals+0x5d/0x70 [ 4793.935306][ T44] do_init_module+0x27a/0x860 [ 4793.935679][ T44] ? free_module+0x2f0/0x2f0 [ 4793.936030][ T44] ? kfree+0x144/0x320 [ 4793.936300][ T44] ? find_vm_area+0x12/0x50 [ 4793.936699][ T44] load_module+0x1f40/0x2dd0 [ 4793.937151][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4793.937582][ T44] ? kernel_read_file+0x3ed/0x550 [ 4793.937971][ T44] ? kernel_read_file+0x3c8/0x550 [ 4793.938341][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4793.938838][ T44] ? init_module_from_file+0xe9/0x150 [ 4793.939216][ T44] init_module_from_file+0xe9/0x150 [ 4793.939589][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4793.939928][ T44] ? idempotent_init_module+0x316/0x610 [ 4793.940291][ T44] ? __lock_release+0x5d/0x170 [ 4793.940679][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4793.941148][ T44] idempotent_init_module+0x331/0x610 [ 4793.941515][ T44] ? init_module_from_file+0x150/0x150 [ 4793.941921][ T44] ? cap_capable+0x94/0x230 [ 4793.942243][ T44] __x64_sys_finit_module+0xca/0x150 [ 4793.942559][ T44] ? do_syscall_64+0x85/0x300 [ 4793.942869][ T44] do_syscall_64+0xc1/0x300 [ 4793.943142][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4793.943506][ T44] RIP: 0033:0x7fc9db13ba0d [ 4793.943820][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4793.944264][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 4793.944695][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 4793.945144][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 4793.945578][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 4793.946022][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 4793.946472][ T44] [ 4793.946702][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 4793.947367][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 4793.948053][ T44] Workqueue: netns cleanup_net [ 4793.948337][ T44] Call Trace: [ 4793.948548][ T44] [ 4793.948693][ T44] ? sched_clock_cpu+0x6c/0x540 [ 4793.948986][ T44] ? __mutex_trylock_common+0xf9/0x260 [ 4793.949278][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 4793.949557][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.949856][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.950135][ T44] ? trace_contention_end+0xd8/0x140 [ 4793.950419][ T44] ? __mutex_lock+0x19f/0x11a0 [ 4793.950708][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 4793.951028][ T44] ? nf_ct_iterate_cleanup+0x4a/0x340 [nf_conntrack] [ 4793.951425][ T44] ? irqentry_exit+0x3b/0x80 [ 4793.951724][ T44] ? nf_ct_iterate_cleanup+0x4a/0x340 [nf_conntrack] [ 4793.952084][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 4793.952500][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 4793.952921][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 4793.953341][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 4793.953701][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 4793.954084][ T44] ? ops_undo_list+0x2bf/0x890 [ 4793.954406][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 4793.954725][ T44] ? cleanup_net+0x2c5/0x890 [ 4793.955084][ T44] ? cleanup_net+0x3a1/0x890 [ 4793.955401][ T44] ? net_passive_dec+0x190/0x190 [ 4793.955726][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.956062][ T44] ? process_one_work+0xe35/0x1650 [ 4793.956363][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 4793.956698][ T44] ? assign_work+0x168/0x240 [ 4793.957042][ T44] ? worker_thread+0x591/0xcf0 [ 4793.957392][ T44] ? rescuer_thread+0xd10/0xd10 [ 4793.957663][ T44] ? kthread+0x37b/0x5f0 [ 4793.957893][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4793.958184][ T44] ? ret_from_fork+0x1b/0x270 [ 4793.958481][ T44] ? __lock_release+0x5d/0x170 [ 4793.958822][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.959123][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4793.959427][ T44] ? ret_from_fork+0x1db/0x270 [ 4793.959718][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4793.960040][ T44] ? ret_from_fork_asm+0x11/0x20 [ 4793.960363][ T44] [ 4793.960629][ T44] [ 4793.960629][ T44] Showing all locks held in the system: [ 4793.961167][ T44] 3 locks held by kworker/u16:0/12: [ 4793.961455][ T44] 1 lock held by khungtaskd/44: [ 4793.961714][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4793.962262][ T44] 1 lock held by modprobe/2441: [ 4793.962542][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4793.963116][ T44] [ 4793.963248][ T44] ============================================= [ 4793.963248][ T44] [ 4916.801808][ T44] INFO: task modprobe:2441 blocked for more than 491 seconds. [ 4916.802257][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 4916.802474][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4916.802881][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 4916.803368][ T44] Call Trace: [ 4916.803537][ T44] [ 4916.803661][ T44] __schedule+0x880/0x1b10 [ 4916.803903][ T44] ? io_schedule_timeout+0x150/0x150 [ 4916.804142][ T44] ? schedule+0x1ee/0x270 [ 4916.804302][ T44] ? __lock_release+0x5d/0x170 [ 4916.804526][ T44] schedule+0xe2/0x270 [ 4916.804689][ T44] schedule_preempt_disabled+0x18/0x30 [ 4916.804913][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 4916.805131][ T44] ? __lock_acquire+0x449/0x7e0 [ 4916.805345][ T44] ? down_read+0x4a0/0x4a0 [ 4916.805577][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.805804][ T44] ? lock_acquire+0x104/0x160 [ 4916.806017][ T44] ? register_pernet_subsys+0x1a/0x40 [ 4916.806239][ T44] down_write+0x1d6/0x200 [ 4916.806402][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 4916.806666][ T44] ? find_held_lock+0x2b/0x80 [ 4916.806908][ T44] ? __create_object+0x5e/0xb0 [ 4916.807126][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 4916.807353][ T44] register_pernet_subsys+0x1a/0x40 [ 4916.807568][ T44] pg_init+0x2c/0x1000 [pktgen] [ 4916.807794][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 4916.808010][ T44] do_one_initcall+0x8c/0x1d0 [ 4916.808223][ T44] ? trace_initcall_level+0x130/0x130 [ 4916.808440][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.808652][ T44] ? trace_kmalloc+0x26/0xd0 [ 4916.808882][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 4916.809094][ T44] ? debug_objects_oom+0x160/0x160 [ 4916.809308][ T44] ? kasan_unpoison+0x27/0x50 [ 4916.809524][ T44] ? __asan_register_globals+0x5d/0x70 [ 4916.809749][ T44] do_init_module+0x27a/0x860 [ 4916.809971][ T44] ? free_module+0x2f0/0x2f0 [ 4916.810184][ T44] ? kfree+0x144/0x320 [ 4916.810358][ T44] ? find_vm_area+0x12/0x50 [ 4916.810578][ T44] load_module+0x1f40/0x2dd0 [ 4916.810826][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4916.811093][ T44] ? kernel_read_file+0x3ed/0x550 [ 4916.811311][ T44] ? kernel_read_file+0x3c8/0x550 [ 4916.811527][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4916.811757][ T44] ? init_module_from_file+0xe9/0x150 [ 4916.811978][ T44] init_module_from_file+0xe9/0x150 [ 4916.812192][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4916.812422][ T44] ? idempotent_init_module+0x316/0x610 [ 4916.812634][ T44] ? __lock_release+0x5d/0x170 [ 4916.812860][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4916.813077][ T44] idempotent_init_module+0x331/0x610 [ 4916.813293][ T44] ? init_module_from_file+0x150/0x150 [ 4916.813527][ T44] ? cap_capable+0x94/0x230 [ 4916.813755][ T44] __x64_sys_finit_module+0xca/0x150 [ 4916.813969][ T44] ? do_syscall_64+0x85/0x300 [ 4916.814185][ T44] do_syscall_64+0xc1/0x300 [ 4916.814399][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4916.814664][ T44] RIP: 0033:0x7fc9db13ba0d [ 4916.814908][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4916.815244][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 4916.815565][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 4916.815896][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 4916.816217][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 4916.816543][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 4916.816887][ T44] [ 4916.817051][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 4916.817505][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 4916.818043][ T44] Workqueue: netns cleanup_net [ 4916.818264][ T44] Call Trace: [ 4916.818424][ T44] [ 4916.818539][ T44] ? __mutex_trylock_common+0xf9/0x260 [ 4916.818776][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 4916.818992][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.819210][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.819425][ T44] ? trace_contention_end+0xd8/0x140 [ 4916.819645][ T44] ? __mutex_lock+0x19f/0x11a0 [ 4916.819866][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 4916.820090][ T44] ? irqentry_exit+0x3b/0x80 [ 4916.820325][ T44] ? nf_ct_iterate_cleanup+0x4a/0x340 [nf_conntrack] [ 4916.820609][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 4916.820948][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 4916.821281][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 4916.821610][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 4916.821899][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 4916.822184][ T44] ? ops_undo_list+0x2bf/0x890 [ 4916.822407][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 4916.822621][ T44] ? cleanup_net+0x2c5/0x890 [ 4916.822867][ T44] ? cleanup_net+0x3a1/0x890 [ 4916.823083][ T44] ? net_passive_dec+0x190/0x190 [ 4916.823301][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.823519][ T44] ? process_one_work+0xe35/0x1650 [ 4916.823752][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 4916.823974][ T44] ? assign_work+0x168/0x240 [ 4916.824190][ T44] ? worker_thread+0x591/0xcf0 [ 4916.824416][ T44] ? rescuer_thread+0xd10/0xd10 [ 4916.824632][ T44] ? kthread+0x37b/0x5f0 [ 4916.824803][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4916.825016][ T44] ? ret_from_fork+0x1b/0x270 [ 4916.825230][ T44] ? __lock_release+0x5d/0x170 [ 4916.825444][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.825656][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4916.825879][ T44] ? ret_from_fork+0x1db/0x270 [ 4916.826092][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 4916.826309][ T44] ? ret_from_fork_asm+0x11/0x20 [ 4916.826533][ T44] [ 4916.826721][ T44] [ 4916.826721][ T44] Showing all locks held in the system: [ 4916.827168][ T44] 4 locks held by kworker/u16:0/12: [ 4916.827381][ T44] 1 lock held by khungtaskd/44: [ 4916.827584][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4916.827978][ T44] 1 lock held by modprobe/2441: [ 4916.828180][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4916.828595][ T44] [ 4916.828698][ T44] ============================================= [ 4916.828698][ T44] [ 5039.682812][ T44] INFO: task modprobe:2441 blocked for more than 614 seconds. [ 5039.683266][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 5039.683490][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5039.683882][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 5039.684361][ T44] Call Trace: [ 5039.684523][ T44] [ 5039.684639][ T44] __schedule+0x880/0x1b10 [ 5039.684874][ T44] ? io_schedule_timeout+0x150/0x150 [ 5039.685111][ T44] ? schedule+0x1ee/0x270 [ 5039.685271][ T44] ? __lock_release+0x5d/0x170 [ 5039.685494][ T44] schedule+0xe2/0x270 [ 5039.685657][ T44] schedule_preempt_disabled+0x18/0x30 [ 5039.685880][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 5039.686097][ T44] ? __lock_acquire+0x449/0x7e0 [ 5039.686310][ T44] ? down_read+0x4a0/0x4a0 [ 5039.686543][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.686775][ T44] ? lock_acquire+0x104/0x160 [ 5039.686991][ T44] ? register_pernet_subsys+0x1a/0x40 [ 5039.687214][ T44] down_write+0x1d6/0x200 [ 5039.687377][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 5039.687640][ T44] ? find_held_lock+0x2b/0x80 [ 5039.687864][ T44] ? __create_object+0x5e/0xb0 [ 5039.688081][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5039.688308][ T44] register_pernet_subsys+0x1a/0x40 [ 5039.688522][ T44] pg_init+0x2c/0x1000 [pktgen] [ 5039.688746][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5039.688963][ T44] do_one_initcall+0x8c/0x1d0 [ 5039.689178][ T44] ? trace_initcall_level+0x130/0x130 [ 5039.689408][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.689621][ T44] ? trace_kmalloc+0x26/0xd0 [ 5039.689847][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 5039.690068][ T44] ? debug_objects_oom+0x160/0x160 [ 5039.690282][ T44] ? kasan_unpoison+0x27/0x50 [ 5039.690498][ T44] ? __asan_register_globals+0x5d/0x70 [ 5039.690713][ T44] do_init_module+0x27a/0x860 [ 5039.690945][ T44] ? free_module+0x2f0/0x2f0 [ 5039.691159][ T44] ? kfree+0x144/0x320 [ 5039.691320][ T44] ? find_vm_area+0x12/0x50 [ 5039.691547][ T44] load_module+0x1f40/0x2dd0 [ 5039.691779][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5039.692045][ T44] ? kernel_read_file+0x3ed/0x550 [ 5039.692262][ T44] ? kernel_read_file+0x3c8/0x550 [ 5039.692482][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5039.692702][ T44] ? init_module_from_file+0xe9/0x150 [ 5039.692924][ T44] init_module_from_file+0xe9/0x150 [ 5039.693143][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5039.693376][ T44] ? idempotent_init_module+0x316/0x610 [ 5039.693589][ T44] ? __lock_release+0x5d/0x170 [ 5039.693813][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5039.694030][ T44] idempotent_init_module+0x331/0x610 [ 5039.694248][ T44] ? init_module_from_file+0x150/0x150 [ 5039.694476][ T44] ? cap_capable+0x94/0x230 [ 5039.694703][ T44] __x64_sys_finit_module+0xca/0x150 [ 5039.694924][ T44] ? do_syscall_64+0x85/0x300 [ 5039.695144][ T44] do_syscall_64+0xc1/0x300 [ 5039.695363][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5039.695626][ T44] RIP: 0033:0x7fc9db13ba0d [ 5039.695855][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5039.696187][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 5039.696509][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 5039.696837][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 5039.697159][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 5039.697477][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 5039.697827][ T44] [ 5039.697994][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 5039.698455][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 5039.698996][ T44] Workqueue: netns cleanup_net [ 5039.699217][ T44] Call Trace: [ 5039.699386][ T44] [ 5039.699503][ T44] ? __mutex_trylock_common+0xf9/0x260 [ 5039.699725][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 5039.699950][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.700167][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.700385][ T44] ? __mutex_lock+0x19f/0x11a0 [ 5039.700599][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 5039.700836][ T44] ? irqentry_exit+0x3b/0x80 [ 5039.701054][ T44] ? lockdep_hardirqs_on+0x7c/0x110 [ 5039.701281][ T44] ? nf_ct_iterate_cleanup+0x73/0x340 [nf_conntrack] [ 5039.701563][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 5039.701903][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 5039.702230][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 5039.702565][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 5039.702850][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 5039.703134][ T44] ? ops_undo_list+0x2bf/0x890 [ 5039.703353][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 5039.703565][ T44] ? cleanup_net+0x2c5/0x890 [ 5039.703797][ T44] ? cleanup_net+0x3a1/0x890 [ 5039.704019][ T44] ? net_passive_dec+0x190/0x190 [ 5039.704233][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.704455][ T44] ? process_one_work+0xe35/0x1650 [ 5039.704678][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 5039.704908][ T44] ? assign_work+0x168/0x240 [ 5039.705129][ T44] ? worker_thread+0x591/0xcf0 [ 5039.705351][ T44] ? rescuer_thread+0xd10/0xd10 [ 5039.705575][ T44] ? kthread+0x37b/0x5f0 [ 5039.705746][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5039.705960][ T44] ? ret_from_fork+0x1b/0x270 [ 5039.706172][ T44] ? __lock_release+0x5d/0x170 [ 5039.706387][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.706600][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5039.706823][ T44] ? ret_from_fork+0x1db/0x270 [ 5039.707037][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5039.707251][ T44] ? ret_from_fork_asm+0x11/0x20 [ 5039.707484][ T44] [ 5039.707657][ T44] [ 5039.707657][ T44] Showing all locks held in the system: [ 5039.707965][ T44] 4 locks held by kworker/u16:0/12: [ 5039.708177][ T44] 1 lock held by khungtaskd/44: [ 5039.708380][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5039.708769][ T44] 1 lock held by modprobe/2441: [ 5039.708979][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5039.709420][ T44] [ 5039.709522][ T44] ============================================= [ 5039.709522][ T44] [ 5162.562813][ T44] INFO: task modprobe:2441 blocked for more than 737 seconds. [ 5162.563203][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 5162.563414][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5162.563822][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 5162.564330][ T44] Call Trace: [ 5162.564492][ T44] [ 5162.564607][ T44] __schedule+0x880/0x1b10 [ 5162.564853][ T44] ? io_schedule_timeout+0x150/0x150 [ 5162.565079][ T44] ? schedule+0x1ee/0x270 [ 5162.565240][ T44] ? __lock_release+0x5d/0x170 [ 5162.565461][ T44] schedule+0xe2/0x270 [ 5162.565624][ T44] schedule_preempt_disabled+0x18/0x30 [ 5162.565856][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 5162.566075][ T44] ? __lock_acquire+0x449/0x7e0 [ 5162.566288][ T44] ? down_read+0x4a0/0x4a0 [ 5162.566510][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.566725][ T44] ? lock_acquire+0x104/0x160 [ 5162.566948][ T44] ? register_pernet_subsys+0x1a/0x40 [ 5162.567172][ T44] down_write+0x1d6/0x200 [ 5162.567345][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 5162.567610][ T44] ? find_held_lock+0x2b/0x80 [ 5162.567853][ T44] ? __create_object+0x5e/0xb0 [ 5162.568074][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5162.568293][ T44] register_pernet_subsys+0x1a/0x40 [ 5162.568516][ T44] pg_init+0x2c/0x1000 [pktgen] [ 5162.568733][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5162.568965][ T44] do_one_initcall+0x8c/0x1d0 [ 5162.569181][ T44] ? trace_initcall_level+0x130/0x130 [ 5162.569396][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.569611][ T44] ? trace_kmalloc+0x26/0xd0 [ 5162.569835][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 5162.570049][ T44] ? debug_objects_oom+0x160/0x160 [ 5162.570264][ T44] ? kasan_unpoison+0x27/0x50 [ 5162.570480][ T44] ? __asan_register_globals+0x5d/0x70 [ 5162.570703][ T44] do_init_module+0x27a/0x860 [ 5162.570938][ T44] ? free_module+0x2f0/0x2f0 [ 5162.571155][ T44] ? kfree+0x144/0x320 [ 5162.571319][ T44] ? find_vm_area+0x12/0x50 [ 5162.571539][ T44] load_module+0x1f40/0x2dd0 [ 5162.571785][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5162.572054][ T44] ? kernel_read_file+0x3ed/0x550 [ 5162.572277][ T44] ? kernel_read_file+0x3c8/0x550 [ 5162.572496][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5162.572717][ T44] ? init_module_from_file+0xe9/0x150 [ 5162.572942][ T44] init_module_from_file+0xe9/0x150 [ 5162.573158][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5162.573387][ T44] ? idempotent_init_module+0x316/0x610 [ 5162.573600][ T44] ? __lock_release+0x5d/0x170 [ 5162.573834][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5162.574054][ T44] idempotent_init_module+0x331/0x610 [ 5162.574272][ T44] ? init_module_from_file+0x150/0x150 [ 5162.574498][ T44] ? cap_capable+0x94/0x230 [ 5162.574717][ T44] __x64_sys_finit_module+0xca/0x150 [ 5162.574945][ T44] ? do_syscall_64+0x85/0x300 [ 5162.575162][ T44] do_syscall_64+0xc1/0x300 [ 5162.575383][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5162.575647][ T44] RIP: 0033:0x7fc9db13ba0d [ 5162.575893][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5162.576219][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 5162.576539][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 5162.576873][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 5162.577192][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 5162.577511][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 5162.577856][ T44] [ 5162.578021][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 5162.578492][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 5162.579024][ T44] Workqueue: netns cleanup_net [ 5162.579242][ T44] Call Trace: [ 5162.579404][ T44] [ 5162.579522][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 5162.579761][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.579995][ T44] ? trace_contention_end+0xd8/0x140 [ 5162.580212][ T44] ? __mutex_lock+0x19f/0x11a0 [ 5162.580426][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 5162.580649][ T44] ? irqentry_exit+0x3b/0x80 [ 5162.580869][ T44] ? lockdep_hardirqs_on+0x7c/0x110 [ 5162.581097][ T44] ? nf_ct_iterate_cleanup+0x79/0x340 [nf_conntrack] [ 5162.581383][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 5162.581716][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 5162.582053][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 5162.582382][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 5162.582658][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 5162.582959][ T44] ? ops_undo_list+0x2bf/0x890 [ 5162.583179][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 5162.583390][ T44] ? cleanup_net+0x2c5/0x890 [ 5162.583612][ T44] ? cleanup_net+0x3a1/0x890 [ 5162.583853][ T44] ? net_passive_dec+0x190/0x190 [ 5162.584073][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.584291][ T44] ? process_one_work+0xe35/0x1650 [ 5162.584514][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 5162.584750][ T44] ? assign_work+0x168/0x240 [ 5162.584969][ T44] ? worker_thread+0x591/0xcf0 [ 5162.585192][ T44] ? rescuer_thread+0xd10/0xd10 [ 5162.585407][ T44] ? kthread+0x37b/0x5f0 [ 5162.585571][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5162.585790][ T44] ? ret_from_fork+0x1b/0x270 [ 5162.586006][ T44] ? __lock_release+0x5d/0x170 [ 5162.586221][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.586442][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5162.586658][ T44] ? ret_from_fork+0x1db/0x270 [ 5162.586881][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5162.587095][ T44] ? ret_from_fork_asm+0x11/0x20 [ 5162.587322][ T44] [ 5162.587515][ T44] [ 5162.587515][ T44] Showing all locks held in the system: [ 5162.587963][ T44] 4 locks held by kworker/u16:0/12: [ 5162.588176][ T44] 1 lock held by khungtaskd/44: [ 5162.588388][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5162.588790][ T44] 1 lock held by modprobe/2441: [ 5162.588994][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5162.589415][ T44] [ 5162.589520][ T44] ============================================= [ 5162.589520][ T44] [ 5285.441805][ T44] INFO: task modprobe:2441 blocked for more than 860 seconds. [ 5285.442233][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 5285.442433][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5285.442820][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 5285.443308][ T44] Call Trace: [ 5285.443473][ T44] [ 5285.443587][ T44] __schedule+0x880/0x1b10 [ 5285.443832][ T44] ? io_schedule_timeout+0x150/0x150 [ 5285.444060][ T44] ? schedule+0x1ee/0x270 [ 5285.444230][ T44] ? __lock_release+0x5d/0x170 [ 5285.444452][ T44] schedule+0xe2/0x270 [ 5285.444615][ T44] schedule_preempt_disabled+0x18/0x30 [ 5285.444841][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 5285.445062][ T44] ? __lock_acquire+0x449/0x7e0 [ 5285.445291][ T44] ? down_read+0x4a0/0x4a0 [ 5285.445512][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.445726][ T44] ? lock_acquire+0x104/0x160 [ 5285.445950][ T44] ? register_pernet_subsys+0x1a/0x40 [ 5285.446171][ T44] down_write+0x1d6/0x200 [ 5285.446345][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 5285.446610][ T44] ? find_held_lock+0x2b/0x80 [ 5285.446838][ T44] ? __create_object+0x5e/0xb0 [ 5285.447054][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5285.447292][ T44] register_pernet_subsys+0x1a/0x40 [ 5285.447522][ T44] pg_init+0x2c/0x1000 [pktgen] [ 5285.447751][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5285.447965][ T44] do_one_initcall+0x8c/0x1d0 [ 5285.448180][ T44] ? trace_initcall_level+0x130/0x130 [ 5285.448415][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.448631][ T44] ? trace_kmalloc+0x26/0xd0 [ 5285.448853][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 5285.449064][ T44] ? debug_objects_oom+0x160/0x160 [ 5285.449278][ T44] ? kasan_unpoison+0x27/0x50 [ 5285.449504][ T44] ? __asan_register_globals+0x5d/0x70 [ 5285.449719][ T44] do_init_module+0x27a/0x860 [ 5285.449948][ T44] ? free_module+0x2f0/0x2f0 [ 5285.450166][ T44] ? kfree+0x144/0x320 [ 5285.450333][ T44] ? find_vm_area+0x12/0x50 [ 5285.450554][ T44] load_module+0x1f40/0x2dd0 [ 5285.450787][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5285.451051][ T44] ? kernel_read_file+0x3ed/0x550 [ 5285.451267][ T44] ? kernel_read_file+0x3c8/0x550 [ 5285.451495][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5285.451720][ T44] ? init_module_from_file+0xe9/0x150 [ 5285.451942][ T44] init_module_from_file+0xe9/0x150 [ 5285.452155][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5285.452390][ T44] ? idempotent_init_module+0x316/0x610 [ 5285.452603][ T44] ? __lock_release+0x5d/0x170 [ 5285.452827][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5285.453042][ T44] idempotent_init_module+0x331/0x610 [ 5285.453264][ T44] ? init_module_from_file+0x150/0x150 [ 5285.453520][ T44] ? cap_capable+0x94/0x230 [ 5285.453751][ T44] __x64_sys_finit_module+0xca/0x150 [ 5285.453964][ T44] ? do_syscall_64+0x85/0x300 [ 5285.454178][ T44] do_syscall_64+0xc1/0x300 [ 5285.454406][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5285.454670][ T44] RIP: 0033:0x7fc9db13ba0d [ 5285.454906][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5285.455230][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 5285.455562][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 5285.455890][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 5285.456212][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 5285.456543][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 5285.456883][ T44] [ 5285.457047][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 5285.457514][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 5285.458057][ T44] Workqueue: netns cleanup_net [ 5285.458277][ T44] Call Trace: [ 5285.458438][ T44] [ 5285.458567][ T44] ? __mutex_trylock_common+0xf9/0x260 [ 5285.458795][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 5285.459009][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.459229][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.459456][ T44] ? trace_contention_end+0xd8/0x140 [ 5285.459702][ T44] ? __mutex_lock+0x19f/0x11a0 [ 5285.459931][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 5285.460152][ T44] ? nf_ct_iterate_cleanup+0x4a/0x340 [nf_conntrack] [ 5285.460438][ T44] ? irqentry_exit+0x3b/0x80 [ 5285.460670][ T44] ? lockdep_hardirqs_on+0x7c/0x110 [ 5285.460914][ T44] ? nf_ct_iterate_cleanup+0x73/0x340 [nf_conntrack] [ 5285.461193][ T44] ? net_generic+0x31/0x1f0 [nf_conntrack] [ 5285.461482][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 5285.461834][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 5285.462164][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 5285.462493][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 5285.462789][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 5285.463074][ T44] ? ops_undo_list+0x2bf/0x890 [ 5285.463294][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 5285.463504][ T44] ? cleanup_net+0x2c5/0x890 [ 5285.463747][ T44] ? cleanup_net+0x3a1/0x890 [ 5285.463963][ T44] ? net_passive_dec+0x190/0x190 [ 5285.464184][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.464403][ T44] ? process_one_work+0xe35/0x1650 [ 5285.464641][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 5285.464870][ T44] ? assign_work+0x168/0x240 [ 5285.465088][ T44] ? worker_thread+0x591/0xcf0 [ 5285.465311][ T44] ? rescuer_thread+0xd10/0xd10 [ 5285.465526][ T44] ? kthread+0x37b/0x5f0 [ 5285.465718][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5285.465945][ T44] ? ret_from_fork+0x1b/0x270 [ 5285.466159][ T44] ? __lock_release+0x5d/0x170 [ 5285.466374][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.466589][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5285.466826][ T44] ? ret_from_fork+0x1db/0x270 [ 5285.467039][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5285.467252][ T44] ? ret_from_fork_asm+0x11/0x20 [ 5285.467477][ T44] [ 5285.467675][ T44] [ 5285.467675][ T44] Showing all locks held in the system: [ 5285.467992][ T44] 4 locks held by kworker/u16:0/12: [ 5285.468205][ T44] 1 lock held by khungtaskd/44: [ 5285.468408][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5285.468809][ T44] 1 lock held by modprobe/2441: [ 5285.469012][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5285.469433][ T44] [ 5285.469536][ T44] ============================================= [ 5285.469536][ T44] [ 5408.322800][ T44] INFO: task modprobe:2441 blocked for more than 983 seconds. [ 5408.323219][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 5408.323460][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5408.323922][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 5408.324497][ T44] Call Trace: [ 5408.324688][ T44] [ 5408.324836][ T44] __schedule+0x880/0x1b10 [ 5408.325103][ T44] ? io_schedule_timeout+0x150/0x150 [ 5408.325368][ T44] ? schedule+0x1ee/0x270 [ 5408.325558][ T44] ? __lock_release+0x5d/0x170 [ 5408.325829][ T44] schedule+0xe2/0x270 [ 5408.326022][ T44] schedule_preempt_disabled+0x18/0x30 [ 5408.326280][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 5408.326537][ T44] ? __lock_acquire+0x449/0x7e0 [ 5408.326820][ T44] ? down_read+0x4a0/0x4a0 [ 5408.327086][ T44] ? rcu_is_watching+0x12/0xb0 [ 5408.327342][ T44] ? lock_acquire+0x104/0x160 [ 5408.327593][ T44] ? register_pernet_subsys+0x1a/0x40 [ 5408.327885][ T44] down_write+0x1d6/0x200 [ 5408.328083][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 5408.328404][ T44] ? find_held_lock+0x2b/0x80 [ 5408.328659][ T44] ? __create_object+0x5e/0xb0 [ 5408.328925][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5408.329187][ T44] register_pernet_subsys+0x1a/0x40 [ 5408.329441][ T44] pg_init+0x2c/0x1000 [pktgen] [ 5408.329697][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5408.329959][ T44] do_one_initcall+0x8c/0x1d0 [ 5408.330220][ T44] ? trace_initcall_level+0x130/0x130 [ 5408.330475][ T44] ? rcu_is_watching+0x12/0xb0 [ 5408.330728][ T44] ? trace_kmalloc+0x26/0xd0 [ 5408.330991][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 5408.331243][ T44] ? debug_objects_oom+0x160/0x160 [ 5408.331499][ T44] ? kasan_unpoison+0x27/0x50 [ 5408.331780][ T44] ? __asan_register_globals+0x5d/0x70 [ 5408.332044][ T44] do_init_module+0x27a/0x860 [ 5408.332305][ T44] ? free_module+0x2f0/0x2f0 [ 5408.332558][ T44] ? kfree+0x144/0x320 [ 5408.332758][ T44] ? find_vm_area+0x12/0x50 [ 5408.333019][ T44] load_module+0x1f40/0x2dd0 [ 5408.333286][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5408.333605][ T44] ? kernel_read_file+0x3ed/0x550 [ 5408.333874][ T44] ? kernel_read_file+0x3c8/0x550 [ 5408.334137][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5408.334396][ T44] ? init_module_from_file+0xe9/0x150 [ 5408.334648][ T44] init_module_from_file+0xe9/0x150 [ 5408.334910][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5408.335180][ T44] ? idempotent_init_module+0x316/0x610 [ 5408.335430][ T44] ? __lock_release+0x5d/0x170 [ 5408.335690][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5408.335968][ T44] idempotent_init_module+0x331/0x610 [ 5408.336229][ T44] ? init_module_from_file+0x150/0x150 [ 5408.336494][ T44] ? cap_capable+0x94/0x230 [ 5408.336769][ T44] __x64_sys_finit_module+0xca/0x150 [ 5408.337021][ T44] ? do_syscall_64+0x85/0x300 [ 5408.337277][ T44] do_syscall_64+0xc1/0x300 [ 5408.337536][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5408.337859][ T44] RIP: 0033:0x7fc9db13ba0d [ 5408.338122][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5408.338505][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 5408.338890][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 5408.339284][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 5408.339661][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 5408.340065][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 5408.340461][ T44] [ 5408.340656][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 5408.341209][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 5408.341833][ T44] Workqueue: netns cleanup_net [ 5408.342090][ T44] Call Trace: [ 5408.342278][ T44] [ 5408.342412][ T44] ? __mutex_trylock_common+0xf9/0x260 [ 5408.342668][ T44] ? update_curr+0x39f/0x500 [ 5408.342938][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 5408.343192][ T44] ? rcu_is_watching+0x12/0xb0 [ 5408.343448][ T44] ? rcu_is_watching+0x12/0xb0 [ 5408.343700][ T44] ? trace_contention_end+0xd8/0x140 [ 5408.343976][ T44] ? __mutex_lock+0x19f/0x11a0 [ 5408.344234][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 5408.344498][ T44] ? irqentry_exit+0x3b/0x80 [ 5408.344782][ T44] ? nf_ct_iterate_cleanup+0x4a/0x340 [nf_conntrack] [ 5408.345114][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 5408.345503][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 5408.345895][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 5408.346283][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 5408.346613][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 5408.346962][ T44] ? ops_undo_list+0x2bf/0x890 [ 5408.347222][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 5408.347472][ T44] ? cleanup_net+0x2c5/0x890 [ 5408.347730][ T44] ? cleanup_net+0x3a1/0x890 [ 5408.348016][ T44] ? net_passive_dec+0x190/0x190 [ 5408.348274][ T44] ? rcu_is_watching+0x12/0xb0 [ 5408.348532][ T44] ? process_one_work+0xe35/0x1650 [ 5408.348809][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 5408.349070][ T44] ? assign_work+0x168/0x240 [ 5408.349328][ T44] ? worker_thread+0x591/0xcf0 [ 5408.349590][ T44] ? rescuer_thread+0xd10/0xd10 [ 5408.349855][ T44] ? kthread+0x37b/0x5f0 [ 5408.350050][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5408.350302][ T44] ? ret_from_fork+0x1b/0x270 [ 5408.350558][ T44] ? __lock_release+0x5d/0x170 [ 5408.350822][ T44] ? rcu_is_watching+0x12/0xb0 [ 5408.351081][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5408.351335][ T44] ? ret_from_fork+0x1db/0x270 [ 5408.351587][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5408.351863][ T44] ? ret_from_fork_asm+0x11/0x20 [ 5408.352134][ T44] [ 5408.352360][ T44] [ 5408.352360][ T44] Showing all locks held in the system: [ 5408.352727][ T44] 4 locks held by kworker/u16:0/12: [ 5408.353123][ T44] 1 lock held by khungtaskd/44: [ 5408.353367][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5408.353830][ T44] 1 lock held by modprobe/2441: [ 5408.354073][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5408.354571][ T44] [ 5408.354699][ T44] ============================================= [ 5408.354699][ T44] [ 5531.202799][ T44] INFO: task modprobe:2441 blocked for more than 1105 seconds. [ 5531.203219][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 5531.203461][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5531.203916][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 5531.204483][ T44] Call Trace: [ 5531.204674][ T44] [ 5531.204821][ T44] __schedule+0x880/0x1b10 [ 5531.205089][ T44] ? io_schedule_timeout+0x150/0x150 [ 5531.205357][ T44] ? schedule+0x1ee/0x270 [ 5531.205547][ T44] ? __lock_release+0x5d/0x170 [ 5531.205831][ T44] schedule+0xe2/0x270 [ 5531.206035][ T44] schedule_preempt_disabled+0x18/0x30 [ 5531.206285][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 5531.206540][ T44] ? __lock_acquire+0x449/0x7e0 [ 5531.206806][ T44] ? down_read+0x4a0/0x4a0 [ 5531.207070][ T44] ? rcu_is_watching+0x12/0xb0 [ 5531.207324][ T44] ? lock_acquire+0x104/0x160 [ 5531.207574][ T44] ? register_pernet_subsys+0x1a/0x40 [ 5531.207869][ T44] down_write+0x1d6/0x200 [ 5531.208067][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 5531.208384][ T44] ? find_held_lock+0x2b/0x80 [ 5531.208636][ T44] ? __create_object+0x5e/0xb0 [ 5531.208903][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5531.209164][ T44] register_pernet_subsys+0x1a/0x40 [ 5531.209416][ T44] pg_init+0x2c/0x1000 [pktgen] [ 5531.209679][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5531.209943][ T44] do_one_initcall+0x8c/0x1d0 [ 5531.210200][ T44] ? trace_initcall_level+0x130/0x130 [ 5531.210453][ T44] ? rcu_is_watching+0x12/0xb0 [ 5531.210706][ T44] ? trace_kmalloc+0x26/0xd0 [ 5531.210969][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 5531.211221][ T44] ? debug_objects_oom+0x160/0x160 [ 5531.211479][ T44] ? kasan_unpoison+0x27/0x50 [ 5531.211733][ T44] ? __asan_register_globals+0x5d/0x70 [ 5531.212021][ T44] do_init_module+0x27a/0x860 [ 5531.212281][ T44] ? free_module+0x2f0/0x2f0 [ 5531.212534][ T44] ? kfree+0x144/0x320 [ 5531.212723][ T44] ? find_vm_area+0x12/0x50 [ 5531.212993][ T44] load_module+0x1f40/0x2dd0 [ 5531.213261][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5531.213579][ T44] ? kernel_read_file+0x3ed/0x550 [ 5531.213847][ T44] ? kernel_read_file+0x3c8/0x550 [ 5531.214102][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5531.214364][ T44] ? init_module_from_file+0xe9/0x150 [ 5531.214615][ T44] init_module_from_file+0xe9/0x150 [ 5531.214876][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5531.215147][ T44] ? idempotent_init_module+0x316/0x610 [ 5531.215418][ T44] ? __lock_release+0x5d/0x170 [ 5531.215676][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5531.215966][ T44] idempotent_init_module+0x331/0x610 [ 5531.216228][ T44] ? init_module_from_file+0x150/0x150 [ 5531.216493][ T44] ? cap_capable+0x94/0x230 [ 5531.216763][ T44] __x64_sys_finit_module+0xca/0x150 [ 5531.217016][ T44] ? do_syscall_64+0x85/0x300 [ 5531.217277][ T44] do_syscall_64+0xc1/0x300 [ 5531.217532][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5531.217855][ T44] RIP: 0033:0x7fc9db13ba0d [ 5531.218117][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5531.218500][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 5531.218886][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 5531.219269][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 5531.219644][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 5531.220048][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 5531.220443][ T44] [ 5531.220643][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 5531.221191][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 5531.221816][ T44] Workqueue: netns cleanup_net [ 5531.222073][ T44] Call Trace: [ 5531.222259][ T44] [ 5531.222401][ T44] ? __mutex_trylock_common+0xf9/0x260 [ 5531.222660][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 5531.222925][ T44] ? rcu_is_watching+0x12/0xb0 [ 5531.223187][ T44] ? rcu_is_watching+0x12/0xb0 [ 5531.223441][ T44] ? trace_contention_end+0xd8/0x140 [ 5531.223697][ T44] ? __mutex_lock+0x19f/0x11a0 [ 5531.223974][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 5531.224240][ T44] ? irqentry_exit+0x3b/0x80 [ 5531.224516][ T44] ? nf_ct_iterate_cleanup+0x4a/0x340 [nf_conntrack] [ 5531.224853][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 5531.225244][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 5531.225628][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 5531.226039][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 5531.226377][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 5531.226711][ T44] ? ops_undo_list+0x2bf/0x890 [ 5531.226980][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 5531.227233][ T44] ? cleanup_net+0x2c5/0x890 [ 5531.227493][ T44] ? cleanup_net+0x3a1/0x890 [ 5531.227772][ T44] ? net_passive_dec+0x190/0x190 [ 5531.228039][ T44] ? rcu_is_watching+0x12/0xb0 [ 5531.228302][ T44] ? process_one_work+0xe35/0x1650 [ 5531.228567][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 5531.228837][ T44] ? assign_work+0x168/0x240 [ 5531.229094][ T44] ? worker_thread+0x591/0xcf0 [ 5531.229355][ T44] ? rescuer_thread+0xd10/0xd10 [ 5531.229609][ T44] ? kthread+0x37b/0x5f0 [ 5531.229813][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5531.230065][ T44] ? ret_from_fork+0x1b/0x270 [ 5531.230320][ T44] ? __lock_release+0x5d/0x170 [ 5531.230573][ T44] ? rcu_is_watching+0x12/0xb0 [ 5531.230832][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5531.231086][ T44] ? ret_from_fork+0x1db/0x270 [ 5531.231337][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5531.231588][ T44] ? ret_from_fork_asm+0x11/0x20 [ 5531.231877][ T44] [ 5531.232107][ T44] [ 5531.232107][ T44] Showing all locks held in the system: [ 5531.232474][ T44] 4 locks held by kworker/u16:0/12: [ 5531.232725][ T44] 1 lock held by khungtaskd/44: [ 5531.233110][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5531.233565][ T44] 1 lock held by modprobe/2441: [ 5531.233814][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5531.234317][ T44] [ 5531.234440][ T44] ============================================= [ 5531.234440][ T44] [ 5654.082797][ T44] INFO: task modprobe:2441 blocked for more than 1228 seconds. [ 5654.083206][ T44] Not tainted 6.17.0-rc5-virtme #1 [ 5654.083407][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5654.083782][ T44] task:modprobe state:D stack:25712 pid:2441 tgid:2441 ppid:1 task_flags:0x400100 flags:0x00004006 [ 5654.084263][ T44] Call Trace: [ 5654.084423][ T44] [ 5654.084537][ T44] __schedule+0x880/0x1b10 [ 5654.084771][ T44] ? io_schedule_timeout+0x150/0x150 [ 5654.085013][ T44] ? schedule+0x1ee/0x270 [ 5654.085172][ T44] ? __lock_release+0x5d/0x170 [ 5654.085394][ T44] schedule+0xe2/0x270 [ 5654.085560][ T44] schedule_preempt_disabled+0x18/0x30 [ 5654.085794][ T44] rwsem_down_write_slowpath+0x3c5/0xcb0 [ 5654.086010][ T44] ? __lock_acquire+0x449/0x7e0 [ 5654.086223][ T44] ? down_read+0x4a0/0x4a0 [ 5654.086445][ T44] ? rcu_is_watching+0x12/0xb0 [ 5654.086661][ T44] ? lock_acquire+0x104/0x160 [ 5654.086883][ T44] ? register_pernet_subsys+0x1a/0x40 [ 5654.087111][ T44] down_write+0x1d6/0x200 [ 5654.087272][ T44] ? rwsem_down_write_slowpath+0xcb0/0xcb0 [ 5654.087536][ T44] ? find_held_lock+0x2b/0x80 [ 5654.087757][ T44] ? __create_object+0x5e/0xb0 [ 5654.087974][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5654.088193][ T44] register_pernet_subsys+0x1a/0x40 [ 5654.088406][ T44] pg_init+0x2c/0x1000 [pktgen] [ 5654.088626][ T44] ? _sub_I_65535_1+0x200/0x200 [pktgen] [ 5654.088850][ T44] do_one_initcall+0x8c/0x1d0 [ 5654.089065][ T44] ? trace_initcall_level+0x130/0x130 [ 5654.089278][ T44] ? rcu_is_watching+0x12/0xb0 [ 5654.089492][ T44] ? trace_kmalloc+0x26/0xd0 [ 5654.089702][ T44] ? __kmalloc_cache_noprof+0x15f/0x360 [ 5654.089924][ T44] ? debug_objects_oom+0x160/0x160 [ 5654.090143][ T44] ? kasan_unpoison+0x27/0x50 [ 5654.090357][ T44] ? __asan_register_globals+0x5d/0x70 [ 5654.090572][ T44] do_init_module+0x27a/0x860 [ 5654.090800][ T44] ? free_module+0x2f0/0x2f0 [ 5654.091015][ T44] ? kfree+0x144/0x320 [ 5654.091173][ T44] ? find_vm_area+0x12/0x50 [ 5654.091393][ T44] load_module+0x1f40/0x2dd0 [ 5654.091617][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5654.091893][ T44] ? kernel_read_file+0x3ed/0x550 [ 5654.092112][ T44] ? kernel_read_file+0x3c8/0x550 [ 5654.092327][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5654.092547][ T44] ? init_module_from_file+0xe9/0x150 [ 5654.092764][ T44] init_module_from_file+0xe9/0x150 [ 5654.092979][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5654.093206][ T44] ? idempotent_init_module+0x316/0x610 [ 5654.093422][ T44] ? __lock_release+0x5d/0x170 [ 5654.093638][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5654.093862][ T44] idempotent_init_module+0x331/0x610 [ 5654.094078][ T44] ? init_module_from_file+0x150/0x150 [ 5654.094301][ T44] ? cap_capable+0x94/0x230 [ 5654.094519][ T44] __x64_sys_finit_module+0xca/0x150 [ 5654.094732][ T44] ? do_syscall_64+0x85/0x300 [ 5654.094967][ T44] do_syscall_64+0xc1/0x300 [ 5654.095182][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5654.095446][ T44] RIP: 0033:0x7fc9db13ba0d [ 5654.095667][ T44] RSP: 002b:00007ffd8a06c468 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5654.095997][ T44] RAX: ffffffffffffffda RBX: 000056436e7ffac0 RCX: 00007fc9db13ba0d [ 5654.096317][ T44] RDX: 0000000000000000 RSI: 000056434a324a2a RDI: 0000000000000005 [ 5654.096640][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffd8a06c5a0 [ 5654.096965][ T44] R10: 0000000000000005 R11: 0000000000000246 R12: 000056434a324a2a [ 5654.097283][ T44] R13: 000056436e7ffb60 R14: 000056436e7ffda0 R15: 0000000000000000 [ 5654.097617][ T44] [ 5654.097791][ T44] INFO: task modprobe:2441 blocked on an rw-semaphore likely owned by task kworker/u16:0:12 [ 5654.098243][ T44] task:kworker/u16:0 state:R running task stack:26136 pid:12 tgid:12 ppid:2 task_flags:0x4208060 flags:0x00004000 [ 5654.098771][ T44] Workqueue: netns cleanup_net [ 5654.098987][ T44] Call Trace: [ 5654.099148][ T44] [ 5654.099265][ T44] ? __mutex_trylock_common+0xf9/0x260 [ 5654.099480][ T44] ? update_curr+0x76/0x500 [ 5654.099694][ T44] ? __mutex_handoff+0x2b0/0x2b0 [ 5654.099913][ T44] ? rcu_is_watching+0x12/0xb0 [ 5654.100126][ T44] ? __schedule+0x13c2/0x1b10 [ 5654.100339][ T44] ? rcu_is_watching+0x12/0xb0 [ 5654.100551][ T44] ? trace_contention_end+0xd8/0x140 [ 5654.100771][ T44] ? __mutex_lock+0x19f/0x11a0 [ 5654.100988][ T44] ? trace_sched_exit_tp+0xcd/0x130 [ 5654.101209][ T44] ? irqentry_exit+0x3b/0x80 [ 5654.101422][ T44] ? lockdep_hardirqs_on+0x7c/0x110 [ 5654.101648][ T44] ? nf_ct_iterate_cleanup+0x4a/0x340 [nf_conntrack] [ 5654.101933][ T44] ? __nf_conntrack_hash_insert+0x200/0x200 [nf_conntrack] [ 5654.102265][ T44] ? nf_conntrack_cleanup_net_list+0x19a/0x4b0 [nf_conntrack] [ 5654.102595][ T44] ? nf_conntrack_cleanup_net_list+0x10e/0x4b0 [nf_conntrack] [ 5654.102931][ T44] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 5654.103207][ T44] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 5654.103489][ T44] ? ops_undo_list+0x2bf/0x890 [ 5654.103707][ T44] ? rtnl_net_dumpid_one+0x270/0x270 [ 5654.103925][ T44] ? cleanup_net+0x2c5/0x890 [ 5654.104149][ T44] ? cleanup_net+0x3a1/0x890 [ 5654.104363][ T44] ? net_passive_dec+0x190/0x190 [ 5654.104579][ T44] ? rcu_is_watching+0x12/0xb0 [ 5654.104802][ T44] ? process_one_work+0xe35/0x1650 [ 5654.105035][ T44] ? pwq_dec_nr_in_flight+0x550/0x550 [ 5654.105256][ T44] ? assign_work+0x168/0x240 [ 5654.105473][ T44] ? worker_thread+0x591/0xcf0 [ 5654.105696][ T44] ? rescuer_thread+0xd10/0xd10 [ 5654.105918][ T44] ? kthread+0x37b/0x5f0 [ 5654.106080][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5654.106292][ T44] ? ret_from_fork+0x1b/0x270 [ 5654.106503][ T44] ? __lock_release+0x5d/0x170 [ 5654.106718][ T44] ? rcu_is_watching+0x12/0xb0 [ 5654.106938][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5654.107152][ T44] ? ret_from_fork+0x1db/0x270 [ 5654.107363][ T44] ? kthread_is_per_cpu+0xc0/0xc0 [ 5654.107590][ T44] ? ret_from_fork_asm+0x11/0x20 [ 5654.107821][ T44] [ 5654.107982][ T44] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings [ 5654.108355][ T44] [ 5654.108355][ T44] Showing all locks held in the system: [ 5654.108660][ T44] 4 locks held by kworker/u16:0/12: [ 5654.109015][ T44] 1 lock held by khungtaskd/44: [ 5654.109223][ T44] #0: ffffffffa5b83f80 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5654.109606][ T44] 1 lock held by modprobe/2441: [ 5654.109815][ T44] #0: ffffffffa6270330 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5654.110235][ T44] [ 5654.110337][ T44] ============================================= [ 5654.110337][ T44] WAIT TIMEOUT stderr Ctrl-C stderr Ctrl-C stderr