[ 3999.316685][ T5710] GACT probability NOT on [ 4046.800070][ T66] ------------[ cut here ]------------ [ 4046.800657][ T66] conntrack cleanup blocked for 60s [ 4046.800661][ T66] WARNING: net/netfilter/nf_conntrack_core.c:2511 at 0x0, CPU#0: kworker/u16:1/66 [ 4046.801436][ T66] Modules linked in: act_gact(+) xt_conntrack nft_compat nf_tables cls_bpf sch_ingress sctp_diag sctp ip6_gre ip_gre gre xfrm_user openvswitch psample nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nsh geneve vxlan act_csum act_pedit cls_flower sch_prio [ 4046.802850][ T66] CPU: 0 UID: 0 PID: 66 Comm: kworker/u16:1 Not tainted 6.18.0-virtme #1 PREEMPT(full) [ 4046.803333][ T66] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 4046.803667][ T66] Workqueue: netns cleanup_net [ 4046.803952][ T66] RIP: 0010:nf_conntrack_cleanup_net_list+0x4a0/0x540 [nf_conntrack] [ 4046.804405][ T66] Code: 00 00 00 65 48 2b 05 ff 11 e6 d3 0f 85 a0 00 00 00 48 81 c4 c0 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8d 3d 00 b2 e3 ff <67> 48 0f b9 3a e9 0f ff ff ff 48 c7 c7 bc ef 4e c0 e8 0a 4f a4 cf [ 4046.805367][ T66] RSP: 0000:ffffc90000477988 EFLAGS: 00010287 [ 4046.805707][ T66] RAX: 0000000100392960 RBX: ffffc90000477ae8 RCX: ffffffffc06a6b0a [ 4046.806123][ T66] RDX: 000000010039295f RSI: 0000000000000004 RDI: ffffffffc04e2030 [ 4046.806522][ T66] RBP: ffff88800a9beda0 R08: 0000000000000000 R09: ffffed1001537db4 [ 4046.806921][ T66] R10: ffff88800a9beda3 R11: ffff888005f4d090 R12: 0000000000000001 [ 4046.807323][ T66] R13: ffff88800f046800 R14: dffffc0000000000 R15: ffffc900004779e8 [ 4046.807731][ T66] FS: 0000000000000000(0000) GS:ffff8880d8b18000(0000) knlGS:0000000000000000 [ 4046.808200][ T66] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4046.808534][ T66] CR2: 0000000000430c80 CR3: 000000001a03c004 CR4: 0000000000772ef0 [ 4046.809495][ T66] PKRU: 55555554 [ 4046.809707][ T66] Call Trace: [ 4046.809907][ T66] [ 4046.810056][ T66] ? nf_conntrack_cleanup_net_list+0x137/0x540 [nf_conntrack] [ 4046.810479][ T66] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 4046.810815][ T66] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 4046.811163][ T66] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 4046.811495][ T66] ? _raw_spin_unlock+0x2d/0x50 [ 4046.811769][ T66] ops_undo_list+0x2bf/0x890 [ 4046.812061][ T66] ? netns_install+0x280/0x280 [ 4046.812322][ T66] ? cleanup_net+0x2d6/0x8e0 [ 4046.812592][ T66] cleanup_net+0x3b2/0x8e0 [ 4046.812864][ T66] ? net_passive_dec+0x190/0x190 [ 4046.813134][ T66] ? rcu_is_watching+0x12/0xb0 [ 4046.813400][ T66] ? lock_acquire+0x104/0x140 [ 4046.813667][ T66] process_one_work+0x880/0x1810 [ 4046.813944][ T66] ? pwq_dec_nr_in_flight+0x550/0x550 [ 4046.814229][ T66] ? assign_work+0x168/0x240 [ 4046.814498][ T66] worker_thread+0x591/0xcf0 [ 4046.814771][ T66] ? bh_worker+0x6f0/0x6f0 [ 4046.815041][ T66] kthread+0x37b/0x5f0 [ 4046.815252][ T66] ? kthread_is_per_cpu+0xc0/0xc0 [ 4046.815510][ T66] ? ret_from_fork+0x71/0x540 [ 4046.815773][ T66] ? __lock_release+0x5d/0x160 [ 4046.816040][ T66] ? lock_acquire+0x104/0x140 [ 4046.816301][ T66] ? rcu_is_watching+0x12/0xb0 [ 4046.816573][ T66] ? kthread_is_per_cpu+0xc0/0xc0 [ 4046.816835][ T66] ret_from_fork+0x42f/0x540 [ 4046.817107][ T66] ? arch_exit_to_user_mode_prepare.constprop.0+0x140/0x140 [ 4046.817496][ T66] ? kthread_is_per_cpu+0xc0/0xc0 [ 4046.817760][ T66] ? __switch_to+0x5c8/0xd50 [ 4046.818034][ T66] ? kthread_is_per_cpu+0xc0/0xc0 [ 4046.818298][ T66] ret_from_fork_asm+0x11/0x20 [ 4046.818593][ T66] [ 4046.818795][ T66] irq event stamp: 33600611 [ 4046.819070][ T66] hardirqs last enabled at (33600621): [] __up_console_sem+0x67/0x70 [ 4046.819529][ T66] hardirqs last disabled at (33600628): [] __up_console_sem+0x4c/0x70 [ 4046.819989][ T66] softirqs last enabled at (33600256): [] handle_softirqs+0x524/0x820 [ 4046.820466][ T66] softirqs last disabled at (33600251): [] __irq_exit_rcu+0x6c/0xe0 [ 4046.820928][ T66] ---[ end trace 0000000000000000 ]--- [ 4179.568081][ T44] INFO: task modprobe:5710 blocked for more than 122 seconds. [ 4179.568643][ T44] Tainted: G W 6.18.0-virtme #1 [ 4179.568895][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4179.569360][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 4179.569838][ T44] Call Trace: [ 4179.570024][ T44] [ 4179.570145][ T44] __schedule+0x90a/0x1930 [ 4179.570396][ T44] ? io_schedule_timeout+0x150/0x150 [ 4179.570631][ T44] ? __lock_release+0x5d/0x160 [ 4179.570875][ T44] schedule+0xe2/0x270 [ 4179.571051][ T44] schedule_preempt_disabled+0x18/0x30 [ 4179.571280][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4179.571502][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4179.571727][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.571949][ T44] ? lock_acquire+0x104/0x140 [ 4179.572181][ T44] down_write+0x1d6/0x200 [ 4179.572352][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4179.572621][ T44] ? console_prepend_message+0x220/0x220 [ 4179.572855][ T44] ? find_held_lock+0x2b/0x80 [ 4179.573087][ T44] register_pernet_subsys+0x1a/0x40 [ 4179.573315][ T44] tcf_register_action+0xa3/0x600 [ 4179.573533][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 4179.573810][ T44] do_one_initcall+0xc5/0x3d0 [ 4179.574062][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4179.574347][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.574569][ T44] ? trace_kmalloc+0x24/0xb0 [ 4179.575292][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4179.575544][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4179.575770][ T44] ? kasan_unpoison+0x27/0x50 [ 4179.576015][ T44] do_init_module+0x27a/0x860 [ 4179.576250][ T44] ? free_module+0x3f0/0x3f0 [ 4179.576469][ T44] ? kfree+0x2bb/0x5f0 [ 4179.576644][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 4179.576916][ T44] ? load_module+0x20c5/0x2dd0 [ 4179.577149][ T44] load_module+0x1f40/0x2dd0 [ 4179.577378][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4179.577651][ T44] ? kernel_read_file+0x3ed/0x550 [ 4179.577871][ T44] ? kernel_read_file+0x3c8/0x550 [ 4179.578118][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4179.578348][ T44] ? init_module_from_file+0x159/0x180 [ 4179.578575][ T44] init_module_from_file+0x159/0x180 [ 4179.578794][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4179.579026][ T44] ? __lock_acquire+0x664/0xca0 [ 4179.579263][ T44] ? idempotent_init_module+0x316/0x610 [ 4179.579488][ T44] ? __lock_release+0x5d/0x160 [ 4179.579713][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4179.579944][ T44] idempotent_init_module+0x331/0x610 [ 4179.580173][ T44] ? init_module_from_file+0x180/0x180 [ 4179.580411][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.580633][ T44] ? cap_capable+0x17b/0x340 [ 4179.580856][ T44] __x64_sys_finit_module+0xca/0x150 [ 4179.581081][ T44] ? do_syscall_64+0x85/0xfc0 [ 4179.581311][ T44] do_syscall_64+0xc1/0xfc0 [ 4179.581532][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4179.581801][ T44] RIP: 0033:0x7f71a481c92d [ 4179.582054][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4179.582403][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 4179.582732][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 4179.583084][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 4179.583409][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 4179.583754][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 4179.584118][ T44] [ 4179.584296][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 4179.584651][ T44] [ 4179.584651][ T44] Showing all locks held in the system: [ 4179.584983][ T44] 1 lock held by khungtaskd/44: [ 4179.585208][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4179.585598][ T44] 2 locks held by kworker/u17:1/45: [ 4179.585810][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4179.586238][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4179.586708][ T44] 4 locks held by kworker/u16:1/66: [ 4179.586935][ T44] 1 lock held by modprobe/5710: [ 4179.587145][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4179.587564][ T44] [ 4179.587678][ T44] ============================================= [ 4179.587678][ T44] [ 4302.448094][ T44] INFO: task modprobe:5710 blocked for more than 245 seconds. [ 4302.448534][ T44] Tainted: G W 6.18.0-virtme #1 [ 4302.448783][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4302.449164][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 4302.449636][ T44] Call Trace: [ 4302.449802][ T44] [ 4302.449921][ T44] __schedule+0x90a/0x1930 [ 4302.450166][ T44] ? io_schedule_timeout+0x150/0x150 [ 4302.450390][ T44] ? __lock_release+0x5d/0x160 [ 4302.450622][ T44] schedule+0xe2/0x270 [ 4302.450786][ T44] schedule_preempt_disabled+0x18/0x30 [ 4302.451014][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4302.451237][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4302.451461][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.451678][ T44] ? lock_acquire+0x104/0x140 [ 4302.451901][ T44] down_write+0x1d6/0x200 [ 4302.452086][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4302.452357][ T44] ? console_prepend_message+0x220/0x220 [ 4302.452576][ T44] ? find_held_lock+0x2b/0x80 [ 4302.452796][ T44] register_pernet_subsys+0x1a/0x40 [ 4302.453023][ T44] tcf_register_action+0xa3/0x600 [ 4302.453245][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 4302.453516][ T44] do_one_initcall+0xc5/0x3d0 [ 4302.453742][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4302.454019][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.454239][ T44] ? trace_kmalloc+0x24/0xb0 [ 4302.454456][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4302.454671][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4302.454888][ T44] ? kasan_unpoison+0x27/0x50 [ 4302.455116][ T44] do_init_module+0x27a/0x860 [ 4302.455366][ T44] ? free_module+0x3f0/0x3f0 [ 4302.455600][ T44] ? kfree+0x2bb/0x5f0 [ 4302.455766][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 4302.456042][ T44] ? load_module+0x20c5/0x2dd0 [ 4302.456266][ T44] load_module+0x1f40/0x2dd0 [ 4302.456497][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4302.456764][ T44] ? kernel_read_file+0x3ed/0x550 [ 4302.456984][ T44] ? kernel_read_file+0x3c8/0x550 [ 4302.457218][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4302.457440][ T44] ? init_module_from_file+0x159/0x180 [ 4302.457654][ T44] init_module_from_file+0x159/0x180 [ 4302.457868][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4302.458096][ T44] ? __lock_acquire+0x664/0xca0 [ 4302.458321][ T44] ? idempotent_init_module+0x316/0x610 [ 4302.458539][ T44] ? __lock_release+0x5d/0x160 [ 4302.458754][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4302.458971][ T44] idempotent_init_module+0x331/0x610 [ 4302.459197][ T44] ? init_module_from_file+0x180/0x180 [ 4302.459423][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.459639][ T44] ? cap_capable+0x17b/0x340 [ 4302.459859][ T44] __x64_sys_finit_module+0xca/0x150 [ 4302.460086][ T44] ? do_syscall_64+0x85/0xfc0 [ 4302.460303][ T44] do_syscall_64+0xc1/0xfc0 [ 4302.460521][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4302.460787][ T44] RIP: 0033:0x7f71a481c92d [ 4302.461018][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4302.461343][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 4302.461673][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 4302.461994][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 4302.462322][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 4302.462645][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 4302.462982][ T44] [ 4302.463156][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 4302.463496][ T44] [ 4302.463496][ T44] Showing all locks held in the system: [ 4302.463806][ T44] 1 lock held by khungtaskd/44: [ 4302.464015][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4302.464378][ T44] 2 locks held by kworker/u17:1/45: [ 4302.464582][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4302.464991][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4302.465460][ T44] 4 locks held by kworker/u16:1/66: [ 4302.465685][ T44] 1 lock held by modprobe/5710: [ 4302.465892][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4302.466305][ T44] [ 4302.466407][ T44] ============================================= [ 4302.466407][ T44] [ 4425.328088][ T44] INFO: task modprobe:5710 blocked for more than 368 seconds. [ 4425.328471][ T44] Tainted: G W 6.18.0-virtme #1 [ 4425.328735][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4425.329100][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 4425.329573][ T44] Call Trace: [ 4425.329736][ T44] [ 4425.329852][ T44] __schedule+0x90a/0x1930 [ 4425.330088][ T44] ? io_schedule_timeout+0x150/0x150 [ 4425.330316][ T44] ? __lock_release+0x5d/0x160 [ 4425.330541][ T44] schedule+0xe2/0x270 [ 4425.330707][ T44] schedule_preempt_disabled+0x18/0x30 [ 4425.330922][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4425.331155][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4425.331380][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.331597][ T44] ? lock_acquire+0x104/0x140 [ 4425.331819][ T44] down_write+0x1d6/0x200 [ 4425.331991][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4425.332268][ T44] ? console_prepend_message+0x220/0x220 [ 4425.332488][ T44] ? find_held_lock+0x2b/0x80 [ 4425.332707][ T44] register_pernet_subsys+0x1a/0x40 [ 4425.332925][ T44] tcf_register_action+0xa3/0x600 [ 4425.333156][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 4425.333430][ T44] do_one_initcall+0xc5/0x3d0 [ 4425.333658][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4425.333926][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.334151][ T44] ? trace_kmalloc+0x24/0xb0 [ 4425.334366][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4425.334582][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4425.334799][ T44] ? kasan_unpoison+0x27/0x50 [ 4425.335028][ T44] do_init_module+0x27a/0x860 [ 4425.335280][ T44] ? free_module+0x3f0/0x3f0 [ 4425.335493][ T44] ? kfree+0x2bb/0x5f0 [ 4425.335658][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 4425.335931][ T44] ? load_module+0x20c5/0x2dd0 [ 4425.336161][ T44] load_module+0x1f40/0x2dd0 [ 4425.336390][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4425.336656][ T44] ? kernel_read_file+0x3ed/0x550 [ 4425.336880][ T44] ? kernel_read_file+0x3c8/0x550 [ 4425.337105][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4425.337328][ T44] ? init_module_from_file+0x159/0x180 [ 4425.337543][ T44] init_module_from_file+0x159/0x180 [ 4425.337760][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4425.337979][ T44] ? __lock_acquire+0x664/0xca0 [ 4425.338218][ T44] ? idempotent_init_module+0x316/0x610 [ 4425.338441][ T44] ? __lock_release+0x5d/0x160 [ 4425.338657][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4425.338877][ T44] idempotent_init_module+0x331/0x610 [ 4425.339103][ T44] ? init_module_from_file+0x180/0x180 [ 4425.339330][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.339545][ T44] ? cap_capable+0x17b/0x340 [ 4425.339765][ T44] __x64_sys_finit_module+0xca/0x150 [ 4425.339986][ T44] ? do_syscall_64+0x85/0xfc0 [ 4425.340210][ T44] do_syscall_64+0xc1/0xfc0 [ 4425.340427][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4425.340693][ T44] RIP: 0033:0x7f71a481c92d [ 4425.340918][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4425.341253][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 4425.341584][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 4425.341903][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 4425.342233][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 4425.342557][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 4425.342901][ T44] [ 4425.343070][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 4425.343411][ T44] [ 4425.343411][ T44] Showing all locks held in the system: [ 4425.343723][ T44] 1 lock held by khungtaskd/44: [ 4425.343924][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4425.344295][ T44] 2 locks held by kworker/u17:1/45: [ 4425.344501][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4425.344913][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4425.345387][ T44] 4 locks held by kworker/u16:1/66: [ 4425.345600][ T44] 1 lock held by modprobe/5710: [ 4425.345818][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4425.346237][ T44] [ 4425.346340][ T44] ============================================= [ 4425.346340][ T44] [ 4548.208082][ T44] INFO: task modprobe:5710 blocked for more than 491 seconds. [ 4548.208542][ T44] Tainted: G W 6.18.0-virtme #1 [ 4548.208802][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4548.209168][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 4548.209659][ T44] Call Trace: [ 4548.209822][ T44] [ 4548.209940][ T44] __schedule+0x90a/0x1930 [ 4548.210180][ T44] ? io_schedule_timeout+0x150/0x150 [ 4548.210404][ T44] ? __lock_release+0x5d/0x160 [ 4548.210626][ T44] schedule+0xe2/0x270 [ 4548.210792][ T44] schedule_preempt_disabled+0x18/0x30 [ 4548.211024][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4548.211246][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4548.211471][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.211691][ T44] ? lock_acquire+0x104/0x140 [ 4548.211913][ T44] down_write+0x1d6/0x200 [ 4548.212086][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4548.212354][ T44] ? console_prepend_message+0x220/0x220 [ 4548.212581][ T44] ? find_held_lock+0x2b/0x80 [ 4548.212802][ T44] register_pernet_subsys+0x1a/0x40 [ 4548.213028][ T44] tcf_register_action+0xa3/0x600 [ 4548.213248][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 4548.213520][ T44] do_one_initcall+0xc5/0x3d0 [ 4548.213737][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4548.214015][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.214237][ T44] ? trace_kmalloc+0x24/0xb0 [ 4548.214453][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4548.214666][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4548.214882][ T44] ? kasan_unpoison+0x27/0x50 [ 4548.215112][ T44] do_init_module+0x27a/0x860 [ 4548.215339][ T44] ? free_module+0x3f0/0x3f0 [ 4548.215555][ T44] ? kfree+0x2bb/0x5f0 [ 4548.215717][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 4548.216029][ T44] ? load_module+0x20c5/0x2dd0 [ 4548.216252][ T44] load_module+0x1f40/0x2dd0 [ 4548.216476][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4548.216745][ T44] ? kernel_read_file+0x3ed/0x550 [ 4548.216964][ T44] ? kernel_read_file+0x3c8/0x550 [ 4548.217189][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4548.217411][ T44] ? init_module_from_file+0x159/0x180 [ 4548.217630][ T44] init_module_from_file+0x159/0x180 [ 4548.217847][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4548.218073][ T44] ? __lock_acquire+0x664/0xca0 [ 4548.218301][ T44] ? idempotent_init_module+0x316/0x610 [ 4548.218515][ T44] ? __lock_release+0x5d/0x160 [ 4548.218734][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4548.218951][ T44] idempotent_init_module+0x331/0x610 [ 4548.219184][ T44] ? init_module_from_file+0x180/0x180 [ 4548.219411][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.219626][ T44] ? cap_capable+0x17b/0x340 [ 4548.219846][ T44] __x64_sys_finit_module+0xca/0x150 [ 4548.220068][ T44] ? do_syscall_64+0x85/0xfc0 [ 4548.220286][ T44] do_syscall_64+0xc1/0xfc0 [ 4548.220503][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4548.220776][ T44] RIP: 0033:0x7f71a481c92d [ 4548.221000][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4548.221334][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 4548.221659][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 4548.221981][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 4548.222316][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 4548.222641][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 4548.222979][ T44] [ 4548.223150][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 4548.223491][ T44] [ 4548.223491][ T44] Showing all locks held in the system: [ 4548.223812][ T44] 1 lock held by khungtaskd/44: [ 4548.224022][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4548.224383][ T44] 2 locks held by kworker/u17:1/45: [ 4548.224585][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4548.224999][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4548.225465][ T44] 4 locks held by kworker/u16:1/66: [ 4548.225674][ T44] 1 lock held by modprobe/5710: [ 4548.225883][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4548.226301][ T44] [ 4548.226411][ T44] ============================================= [ 4548.226411][ T44] [ 4671.088156][ T44] INFO: task modprobe:5710 blocked for more than 614 seconds. [ 4671.088793][ T44] Tainted: G W 6.18.0-virtme #1 [ 4671.089067][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4671.089413][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 4671.089902][ T44] Call Trace: [ 4671.090084][ T44] [ 4671.090203][ T44] __schedule+0x90a/0x1930 [ 4671.090432][ T44] ? io_schedule_timeout+0x150/0x150 [ 4671.090656][ T44] ? __lock_release+0x5d/0x160 [ 4671.090880][ T44] schedule+0xe2/0x270 [ 4671.091054][ T44] schedule_preempt_disabled+0x18/0x30 [ 4671.091279][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4671.091499][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4671.091737][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.091952][ T44] ? lock_acquire+0x104/0x140 [ 4671.092186][ T44] down_write+0x1d6/0x200 [ 4671.092350][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4671.092615][ T44] ? console_prepend_message+0x220/0x220 [ 4671.092844][ T44] ? find_held_lock+0x2b/0x80 [ 4671.093087][ T44] register_pernet_subsys+0x1a/0x40 [ 4671.093307][ T44] tcf_register_action+0xa3/0x600 [ 4671.093523][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 4671.093797][ T44] do_one_initcall+0xc5/0x3d0 [ 4671.094026][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4671.094301][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.094529][ T44] ? trace_kmalloc+0x24/0xb0 [ 4671.094751][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4671.094970][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4671.095204][ T44] ? kasan_unpoison+0x27/0x50 [ 4671.095424][ T44] do_init_module+0x27a/0x860 [ 4671.095649][ T44] ? free_module+0x3f0/0x3f0 [ 4671.095863][ T44] ? kfree+0x2bb/0x5f0 [ 4671.096047][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 4671.096339][ T44] ? load_module+0x20c5/0x2dd0 [ 4671.096560][ T44] load_module+0x1f40/0x2dd0 [ 4671.096788][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4671.097063][ T44] ? kernel_read_file+0x3ed/0x550 [ 4671.097282][ T44] ? kernel_read_file+0x3c8/0x550 [ 4671.097498][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4671.097720][ T44] ? init_module_from_file+0x159/0x180 [ 4671.097941][ T44] init_module_from_file+0x159/0x180 [ 4671.098168][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4671.098388][ T44] ? __lock_acquire+0x664/0xca0 [ 4671.098617][ T44] ? idempotent_init_module+0x316/0x610 [ 4671.098836][ T44] ? __lock_release+0x5d/0x160 [ 4671.099061][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4671.099279][ T44] idempotent_init_module+0x331/0x610 [ 4671.099506][ T44] ? init_module_from_file+0x180/0x180 [ 4671.099731][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.099945][ T44] ? cap_capable+0x17b/0x340 [ 4671.100174][ T44] __x64_sys_finit_module+0xca/0x150 [ 4671.100388][ T44] ? do_syscall_64+0x85/0xfc0 [ 4671.100607][ T44] do_syscall_64+0xc1/0xfc0 [ 4671.100825][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4671.101104][ T44] RIP: 0033:0x7f71a481c92d [ 4671.101329][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4671.101654][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 4671.101976][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 4671.102306][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 4671.102635][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 4671.102957][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 4671.103301][ T44] [ 4671.103463][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 4671.103808][ T44] [ 4671.103808][ T44] Showing all locks held in the system: [ 4671.104118][ T44] 1 lock held by khungtaskd/44: [ 4671.104325][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4671.104685][ T44] 2 locks held by kworker/u17:1/45: [ 4671.104885][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4671.105308][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4671.105769][ T44] 3 locks held by kworker/u16:1/66: [ 4671.105985][ T44] 1 lock held by modprobe/5710: [ 4671.106193][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4671.106605][ T44] [ 4671.106714][ T44] ============================================= [ 4671.106714][ T44] [ 4793.968083][ T44] INFO: task modprobe:5710 blocked for more than 737 seconds. [ 4793.968538][ T44] Tainted: G W 6.18.0-virtme #1 [ 4793.968784][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4793.969165][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 4793.969666][ T44] Call Trace: [ 4793.969832][ T44] [ 4793.969948][ T44] __schedule+0x90a/0x1930 [ 4793.970193][ T44] ? io_schedule_timeout+0x150/0x150 [ 4793.970414][ T44] ? __lock_release+0x5d/0x160 [ 4793.970650][ T44] schedule+0xe2/0x270 [ 4793.970815][ T44] schedule_preempt_disabled+0x18/0x30 [ 4793.971055][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4793.971281][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4793.971509][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.971740][ T44] ? lock_acquire+0x104/0x140 [ 4793.971965][ T44] down_write+0x1d6/0x200 [ 4793.972142][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4793.972410][ T44] ? console_prepend_message+0x220/0x220 [ 4793.972648][ T44] ? find_held_lock+0x2b/0x80 [ 4793.972869][ T44] register_pernet_subsys+0x1a/0x40 [ 4793.973103][ T44] tcf_register_action+0xa3/0x600 [ 4793.973322][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 4793.973608][ T44] do_one_initcall+0xc5/0x3d0 [ 4793.973844][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4793.974126][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.974379][ T44] ? trace_kmalloc+0x24/0xb0 [ 4793.974596][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4793.974825][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4793.975057][ T44] ? kasan_unpoison+0x27/0x50 [ 4793.975279][ T44] do_init_module+0x27a/0x860 [ 4793.975502][ T44] ? free_module+0x3f0/0x3f0 [ 4793.975732][ T44] ? kfree+0x2bb/0x5f0 [ 4793.975903][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 4793.976191][ T44] ? load_module+0x20c5/0x2dd0 [ 4793.976414][ T44] load_module+0x1f40/0x2dd0 [ 4793.976646][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4793.976925][ T44] ? kernel_read_file+0x3ed/0x550 [ 4793.977158][ T44] ? kernel_read_file+0x3c8/0x550 [ 4793.977376][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4793.977600][ T44] ? init_module_from_file+0x159/0x180 [ 4793.977838][ T44] init_module_from_file+0x159/0x180 [ 4793.978068][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4793.978287][ T44] ? __lock_acquire+0x664/0xca0 [ 4793.978513][ T44] ? idempotent_init_module+0x316/0x610 [ 4793.978740][ T44] ? __lock_release+0x5d/0x160 [ 4793.978958][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4793.979193][ T44] idempotent_init_module+0x331/0x610 [ 4793.979415][ T44] ? init_module_from_file+0x180/0x180 [ 4793.979642][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.979885][ T44] ? cap_capable+0x17b/0x340 [ 4793.980120][ T44] __x64_sys_finit_module+0xca/0x150 [ 4793.980336][ T44] ? do_syscall_64+0x85/0xfc0 [ 4793.980554][ T44] do_syscall_64+0xc1/0xfc0 [ 4793.980781][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4793.981069][ T44] RIP: 0033:0x7f71a481c92d [ 4793.981295][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4793.981625][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 4793.981961][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 4793.982292][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 4793.982622][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 4793.982961][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 4793.983306][ T44] [ 4793.983468][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 4793.983826][ T44] [ 4793.983826][ T44] Showing all locks held in the system: [ 4793.984148][ T44] 1 lock held by khungtaskd/44: [ 4793.984350][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4793.984712][ T44] 2 locks held by kworker/u17:1/45: [ 4793.984927][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4793.985356][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4793.985818][ T44] 4 locks held by kworker/u16:1/66: [ 4793.986073][ T44] 1 lock held by modprobe/5710: [ 4793.986284][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4793.986700][ T44] [ 4793.986803][ T44] ============================================= [ 4793.986803][ T44] [ 4916.848080][ T44] INFO: task modprobe:5710 blocked for more than 860 seconds. [ 4916.848490][ T44] Tainted: G W 6.18.0-virtme #1 [ 4916.848756][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4916.849160][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 4916.849663][ T44] Call Trace: [ 4916.849846][ T44] [ 4916.849970][ T44] __schedule+0x90a/0x1930 [ 4916.850230][ T44] ? io_schedule_timeout+0x150/0x150 [ 4916.850469][ T44] ? __lock_release+0x5d/0x160 [ 4916.850706][ T44] schedule+0xe2/0x270 [ 4916.850892][ T44] schedule_preempt_disabled+0x18/0x30 [ 4916.851133][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4916.851373][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4916.851610][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.851841][ T44] ? lock_acquire+0x104/0x140 [ 4916.852086][ T44] down_write+0x1d6/0x200 [ 4916.852262][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4916.852550][ T44] ? console_prepend_message+0x220/0x220 [ 4916.852786][ T44] ? find_held_lock+0x2b/0x80 [ 4916.853028][ T44] register_pernet_subsys+0x1a/0x40 [ 4916.853264][ T44] tcf_register_action+0xa3/0x600 [ 4916.853498][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 4916.853786][ T44] do_one_initcall+0xc5/0x3d0 [ 4916.854025][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4916.854341][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.854569][ T44] ? trace_kmalloc+0x24/0xb0 [ 4916.854797][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4916.855035][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4916.855268][ T44] ? kasan_unpoison+0x27/0x50 [ 4916.855502][ T44] do_init_module+0x27a/0x860 [ 4916.855740][ T44] ? free_module+0x3f0/0x3f0 [ 4916.855966][ T44] ? kfree+0x2bb/0x5f0 [ 4916.856155][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 4916.856454][ T44] ? load_module+0x20c5/0x2dd0 [ 4916.856689][ T44] load_module+0x1f40/0x2dd0 [ 4916.856932][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4916.857225][ T44] ? kernel_read_file+0x3ed/0x550 [ 4916.857458][ T44] ? kernel_read_file+0x3c8/0x550 [ 4916.857690][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4916.857931][ T44] ? init_module_from_file+0x159/0x180 [ 4916.858168][ T44] init_module_from_file+0x159/0x180 [ 4916.858397][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4916.858630][ T44] ? __lock_acquire+0x664/0xca0 [ 4916.858867][ T44] ? idempotent_init_module+0x316/0x610 [ 4916.859104][ T44] ? __lock_release+0x5d/0x160 [ 4916.859338][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4916.859575][ T44] idempotent_init_module+0x331/0x610 [ 4916.859811][ T44] ? init_module_from_file+0x180/0x180 [ 4916.860057][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.860290][ T44] ? cap_capable+0x17b/0x340 [ 4916.860525][ T44] __x64_sys_finit_module+0xca/0x150 [ 4916.860753][ T44] ? do_syscall_64+0x85/0xfc0 [ 4916.860982][ T44] do_syscall_64+0xc1/0xfc0 [ 4916.861229][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4916.861517][ T44] RIP: 0033:0x7f71a481c92d [ 4916.861762][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4916.862117][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 4916.862463][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 4916.862812][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 4916.863165][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 4916.863511][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 4916.863867][ T44] [ 4916.864048][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 4916.864420][ T44] [ 4916.864420][ T44] Showing all locks held in the system: [ 4916.864746][ T44] 1 lock held by khungtaskd/44: [ 4916.864961][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4916.865359][ T44] 2 locks held by kworker/u17:1/45: [ 4916.865577][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4916.866029][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4916.866529][ T44] 4 locks held by kworker/u16:1/66: [ 4916.866756][ T44] 1 lock held by modprobe/5710: [ 4916.866968][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4916.867418][ T44] [ 4916.867529][ T44] ============================================= [ 4916.867529][ T44] [ 5039.728073][ T44] INFO: task modprobe:5710 blocked for more than 983 seconds. [ 5039.728529][ T44] Tainted: G W 6.18.0-virtme #1 [ 5039.728776][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5039.729165][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 5039.729653][ T44] Call Trace: [ 5039.729819][ T44] [ 5039.729936][ T44] __schedule+0x90a/0x1930 [ 5039.730178][ T44] ? io_schedule_timeout+0x150/0x150 [ 5039.730401][ T44] ? __lock_release+0x5d/0x160 [ 5039.730639][ T44] schedule+0xe2/0x270 [ 5039.730813][ T44] schedule_preempt_disabled+0x18/0x30 [ 5039.731039][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5039.731259][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5039.731486][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.731715][ T44] ? lock_acquire+0x104/0x140 [ 5039.731936][ T44] down_write+0x1d6/0x200 [ 5039.732109][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5039.732386][ T44] ? console_prepend_message+0x220/0x220 [ 5039.732622][ T44] ? find_held_lock+0x2b/0x80 [ 5039.732843][ T44] register_pernet_subsys+0x1a/0x40 [ 5039.733074][ T44] tcf_register_action+0xa3/0x600 [ 5039.733340][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 5039.733630][ T44] do_one_initcall+0xc5/0x3d0 [ 5039.733850][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5039.734134][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.734350][ T44] ? trace_kmalloc+0x24/0xb0 [ 5039.734564][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5039.734791][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5039.735018][ T44] ? kasan_unpoison+0x27/0x50 [ 5039.735239][ T44] do_init_module+0x27a/0x860 [ 5039.735464][ T44] ? free_module+0x3f0/0x3f0 [ 5039.735711][ T44] ? kfree+0x2bb/0x5f0 [ 5039.735874][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 5039.736151][ T44] ? load_module+0x20c5/0x2dd0 [ 5039.736372][ T44] load_module+0x1f40/0x2dd0 [ 5039.736616][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5039.736905][ T44] ? kernel_read_file+0x3ed/0x550 [ 5039.737134][ T44] ? kernel_read_file+0x3c8/0x550 [ 5039.737351][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5039.737582][ T44] ? init_module_from_file+0x159/0x180 [ 5039.737811][ T44] init_module_from_file+0x159/0x180 [ 5039.738037][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5039.738258][ T44] ? __lock_acquire+0x664/0xca0 [ 5039.738486][ T44] ? idempotent_init_module+0x316/0x610 [ 5039.738710][ T44] ? __lock_release+0x5d/0x160 [ 5039.738933][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5039.739169][ T44] idempotent_init_module+0x331/0x610 [ 5039.739388][ T44] ? init_module_from_file+0x180/0x180 [ 5039.739614][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.739840][ T44] ? cap_capable+0x17b/0x340 [ 5039.740070][ T44] __x64_sys_finit_module+0xca/0x150 [ 5039.740285][ T44] ? do_syscall_64+0x85/0xfc0 [ 5039.740502][ T44] do_syscall_64+0xc1/0xfc0 [ 5039.740736][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5039.741019][ T44] RIP: 0033:0x7f71a481c92d [ 5039.741246][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5039.741570][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 5039.741903][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 5039.742240][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 5039.742564][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 5039.742894][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 5039.743241][ T44] [ 5039.743403][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 5039.743752][ T44] [ 5039.743752][ T44] Showing all locks held in the system: [ 5039.744076][ T44] 1 lock held by khungtaskd/44: [ 5039.744280][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5039.744640][ T44] 2 locks held by kworker/u17:1/45: [ 5039.744856][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5039.745279][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5039.745740][ T44] 4 locks held by kworker/u16:1/66: [ 5039.745964][ T44] 1 lock held by modprobe/5710: [ 5039.746173][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5039.746599][ T44] [ 5039.746703][ T44] ============================================= [ 5039.746703][ T44] [ 5162.608086][ T44] INFO: task modprobe:5710 blocked for more than 1105 seconds. [ 5162.608486][ T44] Tainted: G W 6.18.0-virtme #1 [ 5162.608738][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5162.609125][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 5162.609606][ T44] Call Trace: [ 5162.609769][ T44] [ 5162.609888][ T44] __schedule+0x90a/0x1930 [ 5162.610131][ T44] ? io_schedule_timeout+0x150/0x150 [ 5162.610357][ T44] ? __lock_release+0x5d/0x160 [ 5162.610590][ T44] schedule+0xe2/0x270 [ 5162.610768][ T44] schedule_preempt_disabled+0x18/0x30 [ 5162.610983][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5162.611220][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5162.611456][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.611674][ T44] ? lock_acquire+0x104/0x140 [ 5162.611902][ T44] down_write+0x1d6/0x200 [ 5162.612078][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5162.612354][ T44] ? console_prepend_message+0x220/0x220 [ 5162.612586][ T44] ? find_held_lock+0x2b/0x80 [ 5162.612811][ T44] register_pernet_subsys+0x1a/0x40 [ 5162.613043][ T44] tcf_register_action+0xa3/0x600 [ 5162.613265][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 5162.613560][ T44] do_one_initcall+0xc5/0x3d0 [ 5162.613784][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5162.614084][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.614302][ T44] ? trace_kmalloc+0x24/0xb0 [ 5162.614529][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5162.614745][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5162.614964][ T44] ? kasan_unpoison+0x27/0x50 [ 5162.615191][ T44] do_init_module+0x27a/0x860 [ 5162.615414][ T44] ? free_module+0x3f0/0x3f0 [ 5162.615646][ T44] ? kfree+0x2bb/0x5f0 [ 5162.615813][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 5162.616093][ T44] ? load_module+0x20c5/0x2dd0 [ 5162.616314][ T44] load_module+0x1f40/0x2dd0 [ 5162.616553][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5162.616836][ T44] ? kernel_read_file+0x3ed/0x550 [ 5162.617065][ T44] ? kernel_read_file+0x3c8/0x550 [ 5162.617283][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5162.617512][ T44] ? init_module_from_file+0x159/0x180 [ 5162.617739][ T44] init_module_from_file+0x159/0x180 [ 5162.617955][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5162.618184][ T44] ? __lock_acquire+0x664/0xca0 [ 5162.618412][ T44] ? idempotent_init_module+0x316/0x610 [ 5162.618637][ T44] ? __lock_release+0x5d/0x160 [ 5162.618863][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5162.619087][ T44] idempotent_init_module+0x331/0x610 [ 5162.619307][ T44] ? init_module_from_file+0x180/0x180 [ 5162.619533][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.619762][ T44] ? cap_capable+0x17b/0x340 [ 5162.619988][ T44] __x64_sys_finit_module+0xca/0x150 [ 5162.620214][ T44] ? do_syscall_64+0x85/0xfc0 [ 5162.620440][ T44] do_syscall_64+0xc1/0xfc0 [ 5162.620670][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5162.620939][ T44] RIP: 0033:0x7f71a481c92d [ 5162.621174][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5162.621504][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 5162.621839][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 5162.622180][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 5162.622508][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 5162.622850][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 5162.623197][ T44] [ 5162.623359][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 5162.623719][ T44] [ 5162.623719][ T44] Showing all locks held in the system: [ 5162.624038][ T44] 1 lock held by khungtaskd/44: [ 5162.624241][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5162.624605][ T44] 2 locks held by kworker/u17:1/45: [ 5162.624816][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5162.625247][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5162.625706][ T44] 4 locks held by kworker/u16:1/66: [ 5162.625938][ T44] 1 lock held by modprobe/5710: [ 5162.626149][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5162.626566][ T44] [ 5162.626677][ T44] ============================================= [ 5162.626677][ T44] [ 5285.488068][ T44] INFO: task modprobe:5710 blocked for more than 1228 seconds. [ 5285.488449][ T44] Tainted: G W 6.18.0-virtme #1 [ 5285.488695][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5285.489097][ T44] task:modprobe state:D stack:25208 pid:5710 tgid:5710 ppid:45 task_flags:0x400100 flags:0x00080000 [ 5285.489592][ T44] Call Trace: [ 5285.489754][ T44] [ 5285.489869][ T44] __schedule+0x90a/0x1930 [ 5285.490112][ T44] ? io_schedule_timeout+0x150/0x150 [ 5285.490334][ T44] ? __lock_release+0x5d/0x160 [ 5285.490558][ T44] schedule+0xe2/0x270 [ 5285.490724][ T44] schedule_preempt_disabled+0x18/0x30 [ 5285.490945][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5285.491177][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5285.491403][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.491619][ T44] ? lock_acquire+0x104/0x140 [ 5285.491843][ T44] down_write+0x1d6/0x200 [ 5285.492018][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5285.492285][ T44] ? console_prepend_message+0x220/0x220 [ 5285.492517][ T44] ? find_held_lock+0x2b/0x80 [ 5285.492738][ T44] register_pernet_subsys+0x1a/0x40 [ 5285.492958][ T44] tcf_register_action+0xa3/0x600 [ 5285.493183][ T44] ? nf_nat_cleanup+0x10c0/0x10c0 [nf_nat] [ 5285.493462][ T44] do_one_initcall+0xc5/0x3d0 [ 5285.493686][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5285.493956][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.494202][ T44] ? trace_kmalloc+0x24/0xb0 [ 5285.494425][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5285.494641][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5285.494858][ T44] ? kasan_unpoison+0x27/0x50 [ 5285.495085][ T44] do_init_module+0x27a/0x860 [ 5285.495309][ T44] ? free_module+0x3f0/0x3f0 [ 5285.495525][ T44] ? kfree+0x2bb/0x5f0 [ 5285.495687][ T44] ? conntrack_mt_exit+0xcf0/0xcf0 [xt_conntrack] [ 5285.495959][ T44] ? load_module+0x20c5/0x2dd0 [ 5285.496185][ T44] load_module+0x1f40/0x2dd0 [ 5285.496411][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5285.496681][ T44] ? kernel_read_file+0x3ed/0x550 [ 5285.496910][ T44] ? kernel_read_file+0x3c8/0x550 [ 5285.497133][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5285.497356][ T44] ? init_module_from_file+0x159/0x180 [ 5285.497582][ T44] init_module_from_file+0x159/0x180 [ 5285.497796][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5285.498021][ T44] ? __lock_acquire+0x664/0xca0 [ 5285.498246][ T44] ? idempotent_init_module+0x316/0x610 [ 5285.498459][ T44] ? __lock_release+0x5d/0x160 [ 5285.498678][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5285.498893][ T44] idempotent_init_module+0x331/0x610 [ 5285.499125][ T44] ? init_module_from_file+0x180/0x180 [ 5285.499349][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.499571][ T44] ? cap_capable+0x17b/0x340 [ 5285.499791][ T44] __x64_sys_finit_module+0xca/0x150 [ 5285.500013][ T44] ? do_syscall_64+0x85/0xfc0 [ 5285.500232][ T44] do_syscall_64+0xc1/0xfc0 [ 5285.500453][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5285.500724][ T44] RIP: 0033:0x7f71a481c92d [ 5285.500948][ T44] RSP: 002b:00007ffc1282d1c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5285.501279][ T44] RAX: ffffffffffffffda RBX: 0000560253498cc0 RCX: 00007f71a481c92d [ 5285.501608][ T44] RDX: 0000000000000000 RSI: 000056023956aa2a RDI: 0000000000000000 [ 5285.501931][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 00007ffc1282d300 [ 5285.502269][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 000056023956aa2a [ 5285.502595][ T44] R13: 0000560253498b00 R14: 00005602534990f0 R15: 0000560253498d31 [ 5285.502933][ T44] [ 5285.503103][ T44] INFO: task modprobe:5710 is blocked on an rw-semaphore, but the owner is not found. [ 5285.503440][ T44] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings [ 5285.503794][ T44] [ 5285.503794][ T44] Showing all locks held in the system: [ 5285.504107][ T44] 1 lock held by khungtaskd/44: [ 5285.504309][ T44] #0: ffffffff9317fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5285.504667][ T44] 2 locks held by kworker/u17:1/45: [ 5285.504866][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5285.505291][ T44] #1: ffffc9000031fcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5285.505755][ T44] 4 locks held by kworker/u16:1/66: [ 5285.505963][ T44] 1 lock held by modprobe/5710: [ 5285.506168][ T44] #0: ffffffff93868cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5285.506583][ T44] [ 5285.506688][ T44] ============================================= [ 5285.506688][ T44] WAIT TIMEOUT stderr Ctrl-C stderr Ctrl-C stderr