[ 3985.860997][ T5909] GACT probability NOT on [ 4033.621475][ T12] ------------[ cut here ]------------ [ 4033.622109][ T12] conntrack cleanup blocked for 60s [ 4033.622113][ T12] WARNING: net/netfilter/nf_conntrack_core.c:2511 at 0x0, CPU#3: kworker/u16:0/12 [ 4033.622829][ T12] Modules linked in: act_gact(+) sch_ingress xt_conntrack nft_chain_nat xt_nat nft_compat nf_tables ip6_gre ip_gre gre xfrm_user openvswitch psample nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nsh geneve vxlan act_csum act_pedit cls_flower sch_prio [ 4033.624207][ T12] CPU: 3 UID: 0 PID: 12 Comm: kworker/u16:0 Not tainted 6.18.0-virtme #1 PREEMPT(full) [ 4033.624661][ T12] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 4033.624979][ T12] Workqueue: netns cleanup_net [ 4033.625252][ T12] RIP: 0010:nf_conntrack_cleanup_net_list+0x4a0/0x540 [nf_conntrack] [ 4033.625682][ T12] Code: 00 00 00 65 48 2b 05 ff 11 46 e5 0f 85 a0 00 00 00 48 81 c4 c0 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8d 3d 00 c2 e5 ff <67> 48 0f b9 3a e9 0f ff ff ff 48 c7 c7 bc ff 50 c0 e8 0a 4f 04 e1 [ 4033.626591][ T12] RSP: 0018:ffffc900000c7988 EFLAGS: 00010297 [ 4033.626916][ T12] RAX: 000000010038f5e2 RBX: ffffc900000c7ae8 RCX: ffffffffc06a6b0a [ 4033.627297][ T12] RDX: 000000010038f5e1 RSI: 0000000000000004 RDI: ffffffffc0503030 [ 4033.627688][ T12] RBP: ffff88800a3904a0 R08: 0000000000000000 R09: ffffed1001472094 [ 4033.628063][ T12] R10: ffff88800a3904a3 R11: ffff888001abd090 R12: 0000000000000001 [ 4033.628447][ T12] R13: ffff88801b9bca00 R14: dffffc0000000000 R15: ffffc900000c79e8 [ 4033.628822][ T12] FS: 0000000000000000(0000) GS:ffff8880c7698000(0000) knlGS:0000000000000000 [ 4033.629272][ T12] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4033.629599][ T12] CR2: 0000000000430c80 CR3: 000000000f561001 CR4: 0000000000772ef0 [ 4033.630516][ T12] PKRU: 55555554 [ 4033.630719][ T12] Call Trace: [ 4033.630913][ T12] [ 4033.631048][ T12] ? nf_conntrack_cleanup_net_list+0x137/0x540 [nf_conntrack] [ 4033.631465][ T12] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 4033.631785][ T12] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 4033.632107][ T12] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 4033.632432][ T12] ? _raw_spin_unlock+0x2d/0x50 [ 4033.632690][ T12] ops_undo_list+0x2bf/0x890 [ 4033.632952][ T12] ? netns_install+0x280/0x280 [ 4033.633208][ T12] ? cleanup_net+0x2d6/0x8e0 [ 4033.633470][ T12] cleanup_net+0x3b2/0x8e0 [ 4033.633721][ T12] ? net_passive_dec+0x190/0x190 [ 4033.633970][ T12] ? rcu_is_watching+0x12/0xb0 [ 4033.634222][ T12] ? lock_acquire+0x104/0x140 [ 4033.634491][ T12] process_one_work+0x880/0x1810 [ 4033.634755][ T12] ? pwq_dec_nr_in_flight+0x550/0x550 [ 4033.635011][ T12] ? assign_work+0x168/0x240 [ 4033.635266][ T12] worker_thread+0x591/0xcf0 [ 4033.635539][ T12] ? bh_worker+0x6f0/0x6f0 [ 4033.635789][ T12] kthread+0x37b/0x5f0 [ 4033.635982][ T12] ? kthread_is_per_cpu+0xc0/0xc0 [ 4033.636227][ T12] ? ret_from_fork+0x71/0x540 [ 4033.636483][ T12] ? __lock_release+0x5d/0x160 [ 4033.636760][ T12] ? lock_acquire+0x104/0x140 [ 4033.637015][ T12] ? rcu_is_watching+0x12/0xb0 [ 4033.637263][ T12] ? kthread_is_per_cpu+0xc0/0xc0 [ 4033.637538][ T12] ret_from_fork+0x42f/0x540 [ 4033.637788][ T12] ? arch_exit_to_user_mode_prepare.constprop.0+0x140/0x140 [ 4033.638158][ T12] ? kthread_is_per_cpu+0xc0/0xc0 [ 4033.638415][ T12] ? __switch_to+0x5c8/0xd50 [ 4033.638666][ T12] ? kthread_is_per_cpu+0xc0/0xc0 [ 4033.638915][ T12] ret_from_fork_asm+0x11/0x20 [ 4033.639183][ T12] [ 4033.639375][ T12] irq event stamp: 36035809 [ 4033.639644][ T12] hardirqs last enabled at (36035819): [] __up_console_sem+0x67/0x70 [ 4033.640082][ T12] hardirqs last disabled at (36035826): [] __up_console_sem+0x4c/0x70 [ 4033.640521][ T12] softirqs last enabled at (36035680): [] handle_softirqs+0x524/0x820 [ 4033.640955][ T12] softirqs last disabled at (36035669): [] __irq_exit_rcu+0x6c/0xe0 [ 4033.641400][ T12] ---[ end trace 0000000000000000 ]--- [ 4179.572505][ T44] INFO: task modprobe:5909 blocked for more than 122 seconds. [ 4179.573226][ T44] Tainted: G W 6.18.0-virtme #1 [ 4179.573561][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4179.573948][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 4179.574575][ T44] Call Trace: [ 4179.574768][ T44] [ 4179.574914][ T44] __schedule+0x90a/0x1930 [ 4179.575176][ T44] ? io_schedule_timeout+0x150/0x150 [ 4179.575439][ T44] ? __lock_release+0x5d/0x160 [ 4179.575697][ T44] schedule+0xe2/0x270 [ 4179.575886][ T44] schedule_preempt_disabled+0x18/0x30 [ 4179.576124][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4179.576374][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4179.576637][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.576887][ T44] ? lock_acquire+0x104/0x140 [ 4179.577147][ T44] down_write+0x1d6/0x200 [ 4179.577339][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4179.577671][ T44] ? console_prepend_message+0x220/0x220 [ 4179.577915][ T44] ? find_held_lock+0x2b/0x80 [ 4179.578179][ T44] register_pernet_subsys+0x1a/0x40 [ 4179.578440][ T44] tcf_register_action+0xa3/0x600 [ 4179.578688][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4179.578985][ T44] do_one_initcall+0xc5/0x3d0 [ 4179.579231][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4179.579535][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.579778][ T44] ? trace_kmalloc+0x24/0xb0 [ 4179.580034][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4179.580276][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4179.580533][ T44] ? kasan_unpoison+0x27/0x50 [ 4179.580781][ T44] do_init_module+0x27a/0x860 [ 4179.581646][ T44] ? free_module+0x3f0/0x3f0 [ 4179.581892][ T44] ? kfree+0x2bb/0x5f0 [ 4179.582081][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4179.582384][ T44] ? load_module+0x20c5/0x2dd0 [ 4179.582643][ T44] load_module+0x1f40/0x2dd0 [ 4179.582893][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4179.583184][ T44] ? kernel_read_file+0x3ed/0x550 [ 4179.583443][ T44] ? kernel_read_file+0x3c8/0x550 [ 4179.583685][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4179.583929][ T44] ? init_module_from_file+0x159/0x180 [ 4179.584167][ T44] init_module_from_file+0x159/0x180 [ 4179.584426][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4179.584677][ T44] ? __lock_acquire+0x664/0xca0 [ 4179.584928][ T44] ? idempotent_init_module+0x316/0x610 [ 4179.585166][ T44] ? __lock_release+0x5d/0x160 [ 4179.585417][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4179.585660][ T44] idempotent_init_module+0x331/0x610 [ 4179.585909][ T44] ? init_module_from_file+0x180/0x180 [ 4179.586160][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.586400][ T44] ? cap_capable+0x17b/0x340 [ 4179.586681][ T44] __x64_sys_finit_module+0xca/0x150 [ 4179.586921][ T44] ? do_syscall_64+0x85/0xfc0 [ 4179.587177][ T44] do_syscall_64+0xc1/0xfc0 [ 4179.587424][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4179.587716][ T44] RIP: 0033:0x7f1384ab692d [ 4179.587975][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4179.588334][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 4179.588700][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 4179.589051][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 4179.589418][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 4179.589765][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 4179.590127][ T44] [ 4179.590318][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 4179.590717][ T44] [ 4179.590717][ T44] Showing all locks held in the system: [ 4179.591053][ T44] 3 locks held by kworker/u16:0/12: [ 4179.591292][ T44] 2 locks held by kworker/u20:0/38: [ 4179.591545][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4179.591987][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4179.592587][ T44] 1 lock held by khungtaskd/44: [ 4179.592818][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4179.593228][ T44] 1 lock held by modprobe/5909: [ 4179.593464][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4179.593902][ T44] [ 4179.594036][ T44] ============================================= [ 4179.594036][ T44] [ 4302.452497][ T44] INFO: task modprobe:5909 blocked for more than 245 seconds. [ 4302.453030][ T44] Tainted: G W 6.18.0-virtme #1 [ 4302.453306][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4302.453731][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 4302.454250][ T44] Call Trace: [ 4302.454452][ T44] [ 4302.454597][ T44] __schedule+0x90a/0x1930 [ 4302.454855][ T44] ? io_schedule_timeout+0x150/0x150 [ 4302.455107][ T44] ? __lock_release+0x5d/0x160 [ 4302.455359][ T44] schedule+0xe2/0x270 [ 4302.455562][ T44] schedule_preempt_disabled+0x18/0x30 [ 4302.455805][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4302.456067][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4302.456321][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.456580][ T44] ? lock_acquire+0x104/0x140 [ 4302.456836][ T44] down_write+0x1d6/0x200 [ 4302.457025][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4302.457328][ T44] ? console_prepend_message+0x220/0x220 [ 4302.457608][ T44] ? find_held_lock+0x2b/0x80 [ 4302.457877][ T44] register_pernet_subsys+0x1a/0x40 [ 4302.458126][ T44] tcf_register_action+0xa3/0x600 [ 4302.458372][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4302.458689][ T44] do_one_initcall+0xc5/0x3d0 [ 4302.458937][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4302.459232][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.459488][ T44] ? trace_kmalloc+0x24/0xb0 [ 4302.459742][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4302.459981][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4302.460223][ T44] ? kasan_unpoison+0x27/0x50 [ 4302.460482][ T44] do_init_module+0x27a/0x860 [ 4302.460738][ T44] ? free_module+0x3f0/0x3f0 [ 4302.460975][ T44] ? kfree+0x2bb/0x5f0 [ 4302.461163][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4302.461500][ T44] ? load_module+0x20c5/0x2dd0 [ 4302.461750][ T44] load_module+0x1f40/0x2dd0 [ 4302.462004][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4302.462295][ T44] ? kernel_read_file+0x3ed/0x550 [ 4302.462570][ T44] ? kernel_read_file+0x3c8/0x550 [ 4302.462815][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4302.463060][ T44] ? init_module_from_file+0x159/0x180 [ 4302.463310][ T44] init_module_from_file+0x159/0x180 [ 4302.463564][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4302.463808][ T44] ? __lock_acquire+0x664/0xca0 [ 4302.464066][ T44] ? idempotent_init_module+0x316/0x610 [ 4302.464307][ T44] ? __lock_release+0x5d/0x160 [ 4302.464562][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4302.464806][ T44] idempotent_init_module+0x331/0x610 [ 4302.465057][ T44] ? init_module_from_file+0x180/0x180 [ 4302.465307][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.465561][ T44] ? cap_capable+0x17b/0x340 [ 4302.465812][ T44] __x64_sys_finit_module+0xca/0x150 [ 4302.466050][ T44] ? do_syscall_64+0x85/0xfc0 [ 4302.466293][ T44] do_syscall_64+0xc1/0xfc0 [ 4302.466559][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4302.466860][ T44] RIP: 0033:0x7f1384ab692d [ 4302.467112][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4302.467489][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 4302.467839][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 4302.468184][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 4302.468554][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 4302.468899][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 4302.469263][ T44] [ 4302.469459][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 4302.469845][ T44] [ 4302.469845][ T44] Showing all locks held in the system: [ 4302.470186][ T44] 4 locks held by kworker/u16:0/12: [ 4302.470585][ T44] 2 locks held by kworker/u20:0/38: [ 4302.470811][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4302.471251][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4302.471748][ T44] 1 lock held by khungtaskd/44: [ 4302.471974][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4302.472381][ T44] 1 lock held by modprobe/5909: [ 4302.472621][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4302.473057][ T44] [ 4302.473184][ T44] ============================================= [ 4302.473184][ T44] [ 4425.332483][ T44] INFO: task modprobe:5909 blocked for more than 368 seconds. [ 4425.333053][ T44] Tainted: G W 6.18.0-virtme #1 [ 4425.333330][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4425.333746][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 4425.334259][ T44] Call Trace: [ 4425.334464][ T44] [ 4425.334608][ T44] __schedule+0x90a/0x1930 [ 4425.334868][ T44] ? io_schedule_timeout+0x150/0x150 [ 4425.335118][ T44] ? __lock_release+0x5d/0x160 [ 4425.335369][ T44] schedule+0xe2/0x270 [ 4425.335580][ T44] schedule_preempt_disabled+0x18/0x30 [ 4425.335833][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4425.336078][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4425.336324][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.336581][ T44] ? lock_acquire+0x104/0x140 [ 4425.336829][ T44] down_write+0x1d6/0x200 [ 4425.337016][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4425.337307][ T44] ? console_prepend_message+0x220/0x220 [ 4425.337603][ T44] ? find_held_lock+0x2b/0x80 [ 4425.337852][ T44] register_pernet_subsys+0x1a/0x40 [ 4425.338096][ T44] tcf_register_action+0xa3/0x600 [ 4425.338342][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4425.338654][ T44] do_one_initcall+0xc5/0x3d0 [ 4425.338899][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4425.339193][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.339449][ T44] ? trace_kmalloc+0x24/0xb0 [ 4425.339689][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4425.339931][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4425.340172][ T44] ? kasan_unpoison+0x27/0x50 [ 4425.340424][ T44] do_init_module+0x27a/0x860 [ 4425.340674][ T44] ? free_module+0x3f0/0x3f0 [ 4425.340914][ T44] ? kfree+0x2bb/0x5f0 [ 4425.341100][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4425.341424][ T44] ? load_module+0x20c5/0x2dd0 [ 4425.341671][ T44] load_module+0x1f40/0x2dd0 [ 4425.341922][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4425.342211][ T44] ? kernel_read_file+0x3ed/0x550 [ 4425.342484][ T44] ? kernel_read_file+0x3c8/0x550 [ 4425.342731][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4425.342977][ T44] ? init_module_from_file+0x159/0x180 [ 4425.343223][ T44] init_module_from_file+0x159/0x180 [ 4425.343475][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4425.343717][ T44] ? __lock_acquire+0x664/0xca0 [ 4425.343969][ T44] ? idempotent_init_module+0x316/0x610 [ 4425.344207][ T44] ? __lock_release+0x5d/0x160 [ 4425.344460][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4425.344704][ T44] idempotent_init_module+0x331/0x610 [ 4425.344956][ T44] ? init_module_from_file+0x180/0x180 [ 4425.345207][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.345455][ T44] ? cap_capable+0x17b/0x340 [ 4425.345702][ T44] __x64_sys_finit_module+0xca/0x150 [ 4425.345941][ T44] ? do_syscall_64+0x85/0xfc0 [ 4425.346182][ T44] do_syscall_64+0xc1/0xfc0 [ 4425.346444][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4425.346747][ T44] RIP: 0033:0x7f1384ab692d [ 4425.346996][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4425.347352][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 4425.347719][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 4425.348068][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 4425.348439][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 4425.348789][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 4425.349150][ T44] [ 4425.349337][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 4425.349738][ T44] [ 4425.349738][ T44] Showing all locks held in the system: [ 4425.350068][ T44] 4 locks held by kworker/u16:0/12: [ 4425.350305][ T44] 2 locks held by kworker/u20:0/38: [ 4425.350556][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4425.350997][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4425.351647][ T44] 1 lock held by khungtaskd/44: [ 4425.351872][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4425.352277][ T44] 1 lock held by modprobe/5909: [ 4425.352509][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4425.352948][ T44] [ 4425.353081][ T44] ============================================= [ 4425.353081][ T44] [ 4548.212482][ T44] INFO: task modprobe:5909 blocked for more than 491 seconds. [ 4548.212966][ T44] Tainted: G W 6.18.0-virtme #1 [ 4548.213293][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4548.213772][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 4548.214379][ T44] Call Trace: [ 4548.214621][ T44] [ 4548.214762][ T44] __schedule+0x90a/0x1930 [ 4548.215053][ T44] ? io_schedule_timeout+0x150/0x150 [ 4548.215337][ T44] ? __lock_release+0x5d/0x160 [ 4548.215639][ T44] schedule+0xe2/0x270 [ 4548.215864][ T44] schedule_preempt_disabled+0x18/0x30 [ 4548.216152][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4548.216450][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4548.216740][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.217028][ T44] ? lock_acquire+0x104/0x140 [ 4548.217319][ T44] down_write+0x1d6/0x200 [ 4548.217586][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4548.217942][ T44] ? console_prepend_message+0x220/0x220 [ 4548.218239][ T44] ? find_held_lock+0x2b/0x80 [ 4548.218539][ T44] register_pernet_subsys+0x1a/0x40 [ 4548.218825][ T44] tcf_register_action+0xa3/0x600 [ 4548.219111][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4548.219479][ T44] do_one_initcall+0xc5/0x3d0 [ 4548.219770][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4548.220113][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.220422][ T44] ? trace_kmalloc+0x24/0xb0 [ 4548.220706][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4548.220995][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4548.221255][ T44] ? kasan_unpoison+0x27/0x50 [ 4548.221583][ T44] do_init_module+0x27a/0x860 [ 4548.221886][ T44] ? free_module+0x3f0/0x3f0 [ 4548.222170][ T44] ? kfree+0x2bb/0x5f0 [ 4548.222430][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4548.222778][ T44] ? load_module+0x20c5/0x2dd0 [ 4548.223067][ T44] load_module+0x1f40/0x2dd0 [ 4548.223357][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4548.223711][ T44] ? kernel_read_file+0x3ed/0x550 [ 4548.223997][ T44] ? kernel_read_file+0x3c8/0x550 [ 4548.224282][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4548.224579][ T44] ? init_module_from_file+0x159/0x180 [ 4548.224866][ T44] init_module_from_file+0x159/0x180 [ 4548.225146][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4548.225455][ T44] ? __lock_acquire+0x664/0xca0 [ 4548.225762][ T44] ? idempotent_init_module+0x316/0x610 [ 4548.226040][ T44] ? __lock_release+0x5d/0x160 [ 4548.226322][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4548.226628][ T44] idempotent_init_module+0x331/0x610 [ 4548.226913][ T44] ? init_module_from_file+0x180/0x180 [ 4548.227201][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.227499][ T44] ? cap_capable+0x17b/0x340 [ 4548.227789][ T44] __x64_sys_finit_module+0xca/0x150 [ 4548.228067][ T44] ? do_syscall_64+0x85/0xfc0 [ 4548.228348][ T44] do_syscall_64+0xc1/0xfc0 [ 4548.228637][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4548.228988][ T44] RIP: 0033:0x7f1384ab692d [ 4548.229274][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4548.229709][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 4548.230119][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 4548.230532][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 4548.230944][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 4548.231345][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 4548.231772][ T44] [ 4548.231992][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 4548.232465][ T44] [ 4548.232465][ T44] Showing all locks held in the system: [ 4548.232853][ T44] 4 locks held by kworker/u16:0/12: [ 4548.233128][ T44] 2 locks held by kworker/u20:0/38: [ 4548.233392][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4548.234073][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4548.234668][ T44] 1 lock held by khungtaskd/44: [ 4548.234935][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4548.235414][ T44] 1 lock held by modprobe/5909: [ 4548.235679][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4548.236197][ T44] [ 4548.236344][ T44] ============================================= [ 4548.236344][ T44] [ 4671.092474][ T44] INFO: task modprobe:5909 blocked for more than 614 seconds. [ 4671.092945][ T44] Tainted: G W 6.18.0-virtme #1 [ 4671.093265][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4671.093743][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 4671.094339][ T44] Call Trace: [ 4671.094576][ T44] [ 4671.094739][ T44] __schedule+0x90a/0x1930 [ 4671.095029][ T44] ? io_schedule_timeout+0x150/0x150 [ 4671.095312][ T44] ? __lock_release+0x5d/0x160 [ 4671.095610][ T44] schedule+0xe2/0x270 [ 4671.095832][ T44] schedule_preempt_disabled+0x18/0x30 [ 4671.096093][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4671.096371][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4671.096670][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.096951][ T44] ? lock_acquire+0x104/0x140 [ 4671.097234][ T44] down_write+0x1d6/0x200 [ 4671.097483][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4671.097836][ T44] ? console_prepend_message+0x220/0x220 [ 4671.098130][ T44] ? find_held_lock+0x2b/0x80 [ 4671.098426][ T44] register_pernet_subsys+0x1a/0x40 [ 4671.098711][ T44] tcf_register_action+0xa3/0x600 [ 4671.098995][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4671.099336][ T44] do_one_initcall+0xc5/0x3d0 [ 4671.099632][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4671.099984][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.100265][ T44] ? trace_kmalloc+0x24/0xb0 [ 4671.100585][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4671.100862][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4671.101141][ T44] ? kasan_unpoison+0x27/0x50 [ 4671.101453][ T44] do_init_module+0x27a/0x860 [ 4671.101747][ T44] ? free_module+0x3f0/0x3f0 [ 4671.102025][ T44] ? kfree+0x2bb/0x5f0 [ 4671.102253][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4671.102608][ T44] ? load_module+0x20c5/0x2dd0 [ 4671.102894][ T44] load_module+0x1f40/0x2dd0 [ 4671.103183][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4671.103537][ T44] ? kernel_read_file+0x3ed/0x550 [ 4671.103822][ T44] ? kernel_read_file+0x3c8/0x550 [ 4671.104106][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4671.104400][ T44] ? init_module_from_file+0x159/0x180 [ 4671.104691][ T44] init_module_from_file+0x159/0x180 [ 4671.104970][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4671.105250][ T44] ? __lock_acquire+0x664/0xca0 [ 4671.105569][ T44] ? idempotent_init_module+0x316/0x610 [ 4671.105852][ T44] ? __lock_release+0x5d/0x160 [ 4671.106133][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4671.106431][ T44] idempotent_init_module+0x331/0x610 [ 4671.106715][ T44] ? init_module_from_file+0x180/0x180 [ 4671.107002][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.107279][ T44] ? cap_capable+0x17b/0x340 [ 4671.107577][ T44] __x64_sys_finit_module+0xca/0x150 [ 4671.107865][ T44] ? do_syscall_64+0x85/0xfc0 [ 4671.108148][ T44] do_syscall_64+0xc1/0xfc0 [ 4671.108452][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4671.108794][ T44] RIP: 0033:0x7f1384ab692d [ 4671.109081][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4671.109516][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 4671.109927][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 4671.110339][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 4671.110758][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 4671.111160][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 4671.111586][ T44] [ 4671.111802][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 4671.112262][ T44] [ 4671.112262][ T44] Showing all locks held in the system: [ 4671.112799][ T44] 4 locks held by kworker/u16:0/12: [ 4671.113073][ T44] 2 locks held by kworker/u20:0/38: [ 4671.113335][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4671.113865][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4671.114452][ T44] 1 lock held by khungtaskd/44: [ 4671.114718][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4671.115180][ T44] 1 lock held by modprobe/5909: [ 4671.115477][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4671.116001][ T44] [ 4671.116149][ T44] ============================================= [ 4671.116149][ T44] [ 4793.972473][ T44] INFO: task modprobe:5909 blocked for more than 737 seconds. [ 4793.972953][ T44] Tainted: G W 6.18.0-virtme #1 [ 4793.973228][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4793.973647][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 4793.974139][ T44] Call Trace: [ 4793.974326][ T44] [ 4793.974481][ T44] __schedule+0x90a/0x1930 [ 4793.974742][ T44] ? io_schedule_timeout+0x150/0x150 [ 4793.974989][ T44] ? __lock_release+0x5d/0x160 [ 4793.975242][ T44] schedule+0xe2/0x270 [ 4793.975452][ T44] schedule_preempt_disabled+0x18/0x30 [ 4793.975691][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4793.975936][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4793.976183][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.976436][ T44] ? lock_acquire+0x104/0x140 [ 4793.976680][ T44] down_write+0x1d6/0x200 [ 4793.976868][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4793.977162][ T44] ? console_prepend_message+0x220/0x220 [ 4793.977435][ T44] ? find_held_lock+0x2b/0x80 [ 4793.977684][ T44] register_pernet_subsys+0x1a/0x40 [ 4793.977939][ T44] tcf_register_action+0xa3/0x600 [ 4793.978187][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4793.978505][ T44] do_one_initcall+0xc5/0x3d0 [ 4793.978751][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4793.979070][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.979312][ T44] ? trace_kmalloc+0x24/0xb0 [ 4793.979566][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4793.979816][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4793.980059][ T44] ? kasan_unpoison+0x27/0x50 [ 4793.980305][ T44] do_init_module+0x27a/0x860 [ 4793.980558][ T44] ? free_module+0x3f0/0x3f0 [ 4793.980803][ T44] ? kfree+0x2bb/0x5f0 [ 4793.980990][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4793.981279][ T44] ? load_module+0x20c5/0x2dd0 [ 4793.981550][ T44] load_module+0x1f40/0x2dd0 [ 4793.981803][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4793.982091][ T44] ? kernel_read_file+0x3ed/0x550 [ 4793.982335][ T44] ? kernel_read_file+0x3c8/0x550 [ 4793.982587][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4793.982841][ T44] ? init_module_from_file+0x159/0x180 [ 4793.983079][ T44] init_module_from_file+0x159/0x180 [ 4793.983319][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4793.983569][ T44] ? __lock_acquire+0x664/0xca0 [ 4793.983823][ T44] ? idempotent_init_module+0x316/0x610 [ 4793.984059][ T44] ? __lock_release+0x5d/0x160 [ 4793.984302][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4793.984564][ T44] idempotent_init_module+0x331/0x610 [ 4793.984806][ T44] ? init_module_from_file+0x180/0x180 [ 4793.985054][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.985294][ T44] ? cap_capable+0x17b/0x340 [ 4793.985568][ T44] __x64_sys_finit_module+0xca/0x150 [ 4793.985807][ T44] ? do_syscall_64+0x85/0xfc0 [ 4793.986047][ T44] do_syscall_64+0xc1/0xfc0 [ 4793.986298][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4793.986597][ T44] RIP: 0033:0x7f1384ab692d [ 4793.986846][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4793.987196][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 4793.987554][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 4793.987913][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 4793.988260][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 4793.988616][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 4793.988980][ T44] [ 4793.989178][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 4793.989576][ T44] [ 4793.989576][ T44] Showing all locks held in the system: [ 4793.989898][ T44] 4 locks held by kworker/u16:0/12: [ 4793.990129][ T44] 2 locks held by kworker/u20:0/38: [ 4793.990353][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4793.990822][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4793.991304][ T44] 1 lock held by khungtaskd/44: [ 4793.991683][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4793.992083][ T44] 1 lock held by modprobe/5909: [ 4793.992303][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4793.992752][ T44] [ 4793.992879][ T44] ============================================= [ 4793.992879][ T44] [ 4916.852471][ T44] INFO: task modprobe:5909 blocked for more than 860 seconds. [ 4916.852953][ T44] Tainted: G W 6.18.0-virtme #1 [ 4916.853226][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4916.853656][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 4916.854149][ T44] Call Trace: [ 4916.854342][ T44] [ 4916.854496][ T44] __schedule+0x90a/0x1930 [ 4916.854746][ T44] ? io_schedule_timeout+0x150/0x150 [ 4916.854991][ T44] ? __lock_release+0x5d/0x160 [ 4916.855237][ T44] schedule+0xe2/0x270 [ 4916.855448][ T44] schedule_preempt_disabled+0x18/0x30 [ 4916.855688][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4916.855934][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4916.856183][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.856434][ T44] ? lock_acquire+0x104/0x140 [ 4916.856681][ T44] down_write+0x1d6/0x200 [ 4916.856875][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4916.857173][ T44] ? console_prepend_message+0x220/0x220 [ 4916.857443][ T44] ? find_held_lock+0x2b/0x80 [ 4916.857691][ T44] register_pernet_subsys+0x1a/0x40 [ 4916.857934][ T44] tcf_register_action+0xa3/0x600 [ 4916.858190][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4916.858501][ T44] do_one_initcall+0xc5/0x3d0 [ 4916.858746][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4916.859048][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.859288][ T44] ? trace_kmalloc+0x24/0xb0 [ 4916.859540][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4916.859782][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4916.860024][ T44] ? kasan_unpoison+0x27/0x50 [ 4916.860271][ T44] do_init_module+0x27a/0x860 [ 4916.860530][ T44] ? free_module+0x3f0/0x3f0 [ 4916.860796][ T44] ? kfree+0x2bb/0x5f0 [ 4916.860984][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 4916.861277][ T44] ? load_module+0x20c5/0x2dd0 [ 4916.861552][ T44] load_module+0x1f40/0x2dd0 [ 4916.861803][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4916.862095][ T44] ? kernel_read_file+0x3ed/0x550 [ 4916.862351][ T44] ? kernel_read_file+0x3c8/0x550 [ 4916.862605][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4916.862858][ T44] ? init_module_from_file+0x159/0x180 [ 4916.863095][ T44] init_module_from_file+0x159/0x180 [ 4916.863335][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4916.863587][ T44] ? __lock_acquire+0x664/0xca0 [ 4916.863837][ T44] ? idempotent_init_module+0x316/0x610 [ 4916.864075][ T44] ? __lock_release+0x5d/0x160 [ 4916.864325][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4916.864579][ T44] idempotent_init_module+0x331/0x610 [ 4916.864822][ T44] ? init_module_from_file+0x180/0x180 [ 4916.865073][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.865312][ T44] ? cap_capable+0x17b/0x340 [ 4916.865586][ T44] __x64_sys_finit_module+0xca/0x150 [ 4916.865825][ T44] ? do_syscall_64+0x85/0xfc0 [ 4916.866076][ T44] do_syscall_64+0xc1/0xfc0 [ 4916.866317][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4916.866616][ T44] RIP: 0033:0x7f1384ab692d [ 4916.866861][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4916.867211][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 4916.867572][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 4916.867927][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 4916.868281][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 4916.868636][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 4916.869003][ T44] [ 4916.869188][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 4916.869595][ T44] [ 4916.869595][ T44] Showing all locks held in the system: [ 4916.869916][ T44] 4 locks held by kworker/u16:0/12: [ 4916.870147][ T44] 2 locks held by kworker/u20:0/38: [ 4916.870371][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4916.870846][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4916.871326][ T44] 1 lock held by khungtaskd/44: [ 4916.871706][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4916.872104][ T44] 1 lock held by modprobe/5909: [ 4916.872327][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4916.872778][ T44] [ 4916.872904][ T44] ============================================= [ 4916.872904][ T44] [ 5039.732470][ T44] INFO: task modprobe:5909 blocked for more than 983 seconds. [ 5039.732952][ T44] Tainted: G W 6.18.0-virtme #1 [ 5039.733225][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5039.733647][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 5039.734138][ T44] Call Trace: [ 5039.734325][ T44] [ 5039.734479][ T44] __schedule+0x90a/0x1930 [ 5039.734729][ T44] ? io_schedule_timeout+0x150/0x150 [ 5039.734971][ T44] ? __lock_release+0x5d/0x160 [ 5039.735226][ T44] schedule+0xe2/0x270 [ 5039.735426][ T44] schedule_preempt_disabled+0x18/0x30 [ 5039.735665][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5039.735907][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5039.736153][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.736393][ T44] ? lock_acquire+0x104/0x140 [ 5039.736647][ T44] down_write+0x1d6/0x200 [ 5039.736836][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5039.737133][ T44] ? console_prepend_message+0x220/0x220 [ 5039.737376][ T44] ? find_held_lock+0x2b/0x80 [ 5039.737648][ T44] register_pernet_subsys+0x1a/0x40 [ 5039.737894][ T44] tcf_register_action+0xa3/0x600 [ 5039.738137][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 5039.738460][ T44] do_one_initcall+0xc5/0x3d0 [ 5039.738713][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5039.739030][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.739267][ T44] ? trace_kmalloc+0x24/0xb0 [ 5039.739516][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5039.739756][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5039.739996][ T44] ? kasan_unpoison+0x27/0x50 [ 5039.740241][ T44] do_init_module+0x27a/0x860 [ 5039.740501][ T44] ? free_module+0x3f0/0x3f0 [ 5039.740744][ T44] ? kfree+0x2bb/0x5f0 [ 5039.740930][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 5039.741217][ T44] ? load_module+0x20c5/0x2dd0 [ 5039.741489][ T44] load_module+0x1f40/0x2dd0 [ 5039.741740][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5039.742032][ T44] ? kernel_read_file+0x3ed/0x550 [ 5039.742274][ T44] ? kernel_read_file+0x3c8/0x550 [ 5039.742532][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5039.742775][ T44] ? init_module_from_file+0x159/0x180 [ 5039.743013][ T44] init_module_from_file+0x159/0x180 [ 5039.743253][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5039.743506][ T44] ? __lock_acquire+0x664/0xca0 [ 5039.743754][ T44] ? idempotent_init_module+0x316/0x610 [ 5039.743990][ T44] ? __lock_release+0x5d/0x160 [ 5039.744236][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5039.744488][ T44] idempotent_init_module+0x331/0x610 [ 5039.744731][ T44] ? init_module_from_file+0x180/0x180 [ 5039.744981][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.745221][ T44] ? cap_capable+0x17b/0x340 [ 5039.745490][ T44] __x64_sys_finit_module+0xca/0x150 [ 5039.745727][ T44] ? do_syscall_64+0x85/0xfc0 [ 5039.745976][ T44] do_syscall_64+0xc1/0xfc0 [ 5039.746214][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5039.746518][ T44] RIP: 0033:0x7f1384ab692d [ 5039.746764][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5039.747114][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 5039.747468][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 5039.747825][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 5039.748170][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 5039.748534][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 5039.748897][ T44] [ 5039.749083][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 5039.749483][ T44] [ 5039.749483][ T44] Showing all locks held in the system: [ 5039.749806][ T44] 4 locks held by kworker/u16:0/12: [ 5039.750038][ T44] 2 locks held by kworker/u20:0/38: [ 5039.750261][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5039.750718][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5039.751199][ T44] 1 lock held by khungtaskd/44: [ 5039.751429][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5039.751829][ T44] 1 lock held by modprobe/5909: [ 5039.752050][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5039.752517][ T44] [ 5039.752643][ T44] ============================================= [ 5039.752643][ T44] [ 5162.612466][ T44] INFO: task modprobe:5909 blocked for more than 1105 seconds. [ 5162.612941][ T44] Tainted: G W 6.18.0-virtme #1 [ 5162.613213][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5162.613634][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 5162.614139][ T44] Call Trace: [ 5162.614328][ T44] [ 5162.614481][ T44] __schedule+0x90a/0x1930 [ 5162.614737][ T44] ? io_schedule_timeout+0x150/0x150 [ 5162.614982][ T44] ? __lock_release+0x5d/0x160 [ 5162.615238][ T44] schedule+0xe2/0x270 [ 5162.615442][ T44] schedule_preempt_disabled+0x18/0x30 [ 5162.615682][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5162.615926][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5162.616177][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.616426][ T44] ? lock_acquire+0x104/0x140 [ 5162.616670][ T44] down_write+0x1d6/0x200 [ 5162.616865][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5162.617156][ T44] ? console_prepend_message+0x220/0x220 [ 5162.617398][ T44] ? find_held_lock+0x2b/0x80 [ 5162.617675][ T44] register_pernet_subsys+0x1a/0x40 [ 5162.617916][ T44] tcf_register_action+0xa3/0x600 [ 5162.618161][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 5162.618476][ T44] do_one_initcall+0xc5/0x3d0 [ 5162.618745][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5162.619042][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.619283][ T44] ? trace_kmalloc+0x24/0xb0 [ 5162.619533][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5162.619771][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5162.620010][ T44] ? kasan_unpoison+0x27/0x50 [ 5162.620257][ T44] do_init_module+0x27a/0x860 [ 5162.620526][ T44] ? free_module+0x3f0/0x3f0 [ 5162.620766][ T44] ? kfree+0x2bb/0x5f0 [ 5162.620954][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 5162.621243][ T44] ? load_module+0x20c5/0x2dd0 [ 5162.621516][ T44] load_module+0x1f40/0x2dd0 [ 5162.621768][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5162.622060][ T44] ? kernel_read_file+0x3ed/0x550 [ 5162.622302][ T44] ? kernel_read_file+0x3c8/0x550 [ 5162.622559][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5162.622805][ T44] ? init_module_from_file+0x159/0x180 [ 5162.623043][ T44] init_module_from_file+0x159/0x180 [ 5162.623282][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5162.623533][ T44] ? __lock_acquire+0x664/0xca0 [ 5162.623783][ T44] ? idempotent_init_module+0x316/0x610 [ 5162.624034][ T44] ? __lock_release+0x5d/0x160 [ 5162.624274][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5162.624527][ T44] idempotent_init_module+0x331/0x610 [ 5162.624767][ T44] ? init_module_from_file+0x180/0x180 [ 5162.625018][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.625258][ T44] ? cap_capable+0x17b/0x340 [ 5162.625531][ T44] __x64_sys_finit_module+0xca/0x150 [ 5162.625779][ T44] ? do_syscall_64+0x85/0xfc0 [ 5162.626020][ T44] do_syscall_64+0xc1/0xfc0 [ 5162.626261][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5162.626573][ T44] RIP: 0033:0x7f1384ab692d [ 5162.626820][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5162.627171][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 5162.627537][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 5162.627882][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 5162.628232][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 5162.628592][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 5162.628955][ T44] [ 5162.629153][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 5162.629547][ T44] [ 5162.629547][ T44] Showing all locks held in the system: [ 5162.629869][ T44] 3 locks held by kworker/u16:0/12: [ 5162.630100][ T44] 2 locks held by kworker/u20:0/38: [ 5162.630322][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5162.630793][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5162.631274][ T44] 1 lock held by khungtaskd/44: [ 5162.631649][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5162.632051][ T44] 1 lock held by modprobe/5909: [ 5162.632281][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5162.632721][ T44] [ 5162.632847][ T44] ============================================= [ 5162.632847][ T44] [ 5285.492462][ T44] INFO: task modprobe:5909 blocked for more than 1228 seconds. [ 5285.492917][ T44] Tainted: G W 6.18.0-virtme #1 [ 5285.493249][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5285.493720][ T44] task:modprobe state:D stack:25208 pid:5909 tgid:5909 ppid:38 task_flags:0x400100 flags:0x00080000 [ 5285.494302][ T44] Call Trace: [ 5285.494535][ T44] [ 5285.494695][ T44] __schedule+0x90a/0x1930 [ 5285.494983][ T44] ? io_schedule_timeout+0x150/0x150 [ 5285.495275][ T44] ? __lock_release+0x5d/0x160 [ 5285.495577][ T44] schedule+0xe2/0x270 [ 5285.495778][ T44] schedule_preempt_disabled+0x18/0x30 [ 5285.496050][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5285.496333][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5285.496632][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.496912][ T44] ? lock_acquire+0x104/0x140 [ 5285.497196][ T44] down_write+0x1d6/0x200 [ 5285.497460][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5285.497804][ T44] ? console_prepend_message+0x220/0x220 [ 5285.498085][ T44] ? find_held_lock+0x2b/0x80 [ 5285.498372][ T44] register_pernet_subsys+0x1a/0x40 [ 5285.498684][ T44] tcf_register_action+0xa3/0x600 [ 5285.498974][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 5285.499321][ T44] do_one_initcall+0xc5/0x3d0 [ 5285.499626][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5285.499969][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.500254][ T44] ? trace_kmalloc+0x24/0xb0 [ 5285.500545][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5285.500824][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5285.501105][ T44] ? kasan_unpoison+0x27/0x50 [ 5285.501389][ T44] do_init_module+0x27a/0x860 [ 5285.501732][ T44] ? free_module+0x3f0/0x3f0 [ 5285.502017][ T44] ? kfree+0x2bb/0x5f0 [ 5285.502235][ T44] ? gact_cleanup_module+0x360/0x360 [act_gact] [ 5285.502584][ T44] ? load_module+0x20c5/0x2dd0 [ 5285.502872][ T44] load_module+0x1f40/0x2dd0 [ 5285.503170][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5285.503521][ T44] ? kernel_read_file+0x3ed/0x550 [ 5285.503814][ T44] ? kernel_read_file+0x3c8/0x550 [ 5285.504094][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5285.504377][ T44] ? init_module_from_file+0x159/0x180 [ 5285.504670][ T44] init_module_from_file+0x159/0x180 [ 5285.504970][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5285.505253][ T44] ? __lock_acquire+0x664/0xca0 [ 5285.505576][ T44] ? idempotent_init_module+0x316/0x610 [ 5285.505868][ T44] ? __lock_release+0x5d/0x160 [ 5285.506152][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5285.506447][ T44] idempotent_init_module+0x331/0x610 [ 5285.506732][ T44] ? init_module_from_file+0x180/0x180 [ 5285.507022][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.507300][ T44] ? cap_capable+0x17b/0x340 [ 5285.507603][ T44] __x64_sys_finit_module+0xca/0x150 [ 5285.507892][ T44] ? do_syscall_64+0x85/0xfc0 [ 5285.508172][ T44] do_syscall_64+0xc1/0xfc0 [ 5285.508462][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5285.508810][ T44] RIP: 0033:0x7f1384ab692d [ 5285.509097][ T44] RSP: 002b:00007fffa8712858 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5285.509534][ T44] RAX: ffffffffffffffda RBX: 0000555e29f63cc0 RCX: 00007f1384ab692d [ 5285.509953][ T44] RDX: 0000000000000000 RSI: 0000555e2999ca2a RDI: 0000000000000000 [ 5285.510354][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007fffa8712990 [ 5285.510774][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 0000555e2999ca2a [ 5285.511177][ T44] R13: 0000555e29f63b00 R14: 0000555e29f640f0 R15: 0000555e29f63d31 [ 5285.511618][ T44] [ 5285.511837][ T44] INFO: task modprobe:5909 is blocked on an rw-semaphore, but the owner is not found. [ 5285.512265][ T44] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings [ 5285.512737][ T44] [ 5285.512737][ T44] Showing all locks held in the system: [ 5285.513123][ T44] 4 locks held by kworker/u16:0/12: [ 5285.513393][ T44] 2 locks held by kworker/u20:0/38: [ 5285.513823][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5285.514341][ T44] #1: ffffc900002afcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5285.514916][ T44] 1 lock held by khungtaskd/44: [ 5285.515180][ T44] #0: ffffffffa477fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5285.515661][ T44] 1 lock held by modprobe/5909: [ 5285.515923][ T44] #0: ffffffffa4e68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5285.516466][ T44] [ 5285.516618][ T44] ============================================= [ 5285.516618][ T44] WAIT TIMEOUT stderr Ctrl-C stderr Ctrl-C stderr