[ 3974.868918][ T12] ------------[ cut here ]------------ [ 3974.869554][ T12] conntrack cleanup blocked for 60s [ 3974.869557][ T12] WARNING: net/netfilter/nf_conntrack_core.c:2511 at 0x0, CPU#0: kworker/u16:0/12 [ 3974.870286][ T12] Modules linked in: mpls_router(+) sch_ingress xt_conntrack nft_compat nf_tables ip6_gre ip_gre gre xfrm_user openvswitch psample nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nsh geneve vxlan act_csum act_pedit cls_flower sch_prio [last unloaded: ila] [ 3974.871582][ T12] CPU: 0 UID: 0 PID: 12 Comm: kworker/u16:0 Not tainted 6.18.0-virtme #1 PREEMPT(full) [ 3974.872041][ T12] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 3974.872363][ T12] Workqueue: netns cleanup_net [ 3974.872637][ T12] RIP: 0010:nf_conntrack_cleanup_net_list+0x4a0/0x540 [nf_conntrack] [ 3974.873055][ T12] Code: 00 00 00 65 48 2b 05 ff 11 26 d9 0f 85 a0 00 00 00 48 81 c4 c0 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8d 3d 00 32 e1 ff <67> 48 0f b9 3a e9 0f ff ff ff 48 c7 c7 bc 6f 4c c0 e8 0a 4f e4 d4 [ 3974.873995][ T12] RSP: 0018:ffffc900000c7988 EFLAGS: 00010297 [ 3974.874320][ T12] RAX: 000000010038106b RBX: ffffc900000c7ae8 RCX: ffffffffc06a6b0a [ 3974.874703][ T12] RDX: 000000010038106a RSI: 0000000000000004 RDI: ffffffffc04ba030 [ 3974.875105][ T12] RBP: ffff888013526560 R08: 0000000000000000 R09: ffffed10026a4cac [ 3974.875498][ T12] R10: ffff888013526563 R11: ffff888001abd090 R12: 0000000000000001 [ 3974.875892][ T12] R13: ffff888004f9ac00 R14: dffffc0000000000 R15: ffffc900000c79e8 [ 3974.876277][ T12] FS: 0000000000000000(0000) GS:ffff8880d3718000(0000) knlGS:0000000000000000 [ 3974.876731][ T12] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3974.877064][ T12] CR2: 00007fae53bb1760 CR3: 0000000025d45005 CR4: 0000000000772ef0 [ 3974.877452][ T12] PKRU: 55555554 [ 3974.877649][ T12] Call Trace: [ 3974.878351][ T12] [ 3974.878491][ T12] ? nf_conntrack_cleanup_net_list+0x137/0x540 [nf_conntrack] [ 3974.878910][ T12] ? nf_conntrack_cleanup_end+0x70/0x70 [nf_conntrack] [ 3974.879234][ T12] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 3974.879564][ T12] ? nf_conntrack_pernet_exit+0xa5/0x140 [nf_conntrack] [ 3974.879895][ T12] ? _raw_spin_unlock+0x2d/0x50 [ 3974.880157][ T12] ops_undo_list+0x2bf/0x890 [ 3974.880421][ T12] ? netns_install+0x280/0x280 [ 3974.880669][ T12] ? cleanup_net+0x2d6/0x8e0 [ 3974.880954][ T12] cleanup_net+0x3b2/0x8e0 [ 3974.881217][ T12] ? net_passive_dec+0x190/0x190 [ 3974.881476][ T12] ? rcu_is_watching+0x12/0xb0 [ 3974.881737][ T12] ? lock_acquire+0x104/0x140 [ 3974.882006][ T12] process_one_work+0x880/0x1810 [ 3974.882278][ T12] ? pwq_dec_nr_in_flight+0x550/0x550 [ 3974.882538][ T12] ? assign_work+0x168/0x240 [ 3974.882798][ T12] worker_thread+0x591/0xcf0 [ 3974.883082][ T12] ? bh_worker+0x6f0/0x6f0 [ 3974.883341][ T12] kthread+0x37b/0x5f0 [ 3974.883539][ T12] ? kthread_is_per_cpu+0xc0/0xc0 [ 3974.883792][ T12] ? ret_from_fork+0x71/0x540 [ 3974.884061][ T12] ? __lock_release+0x5d/0x160 [ 3974.884313][ T12] ? lock_acquire+0x104/0x140 [ 3974.884570][ T12] ? rcu_is_watching+0x12/0xb0 [ 3974.884823][ T12] ? kthread_is_per_cpu+0xc0/0xc0 [ 3974.885089][ T12] ret_from_fork+0x42f/0x540 [ 3974.885342][ T12] ? arch_exit_to_user_mode_prepare.constprop.0+0x140/0x140 [ 3974.885716][ T12] ? kthread_is_per_cpu+0xc0/0xc0 [ 3974.885982][ T12] ? __switch_to+0x5c8/0xd50 [ 3974.886241][ T12] ? kthread_is_per_cpu+0xc0/0xc0 [ 3974.886500][ T12] ret_from_fork_asm+0x11/0x20 [ 3974.886780][ T12] [ 3974.886986][ T12] irq event stamp: 34991561 [ 3974.887262][ T12] hardirqs last enabled at (34991569): [] __up_console_sem+0x67/0x70 [ 3974.887709][ T12] hardirqs last disabled at (34991576): [] __up_console_sem+0x4c/0x70 [ 3974.888166][ T12] softirqs last enabled at (34991130): [] handle_softirqs+0x524/0x820 [ 3974.888613][ T12] softirqs last disabled at (34991125): [] __irq_exit_rcu+0x6c/0xe0 [ 3974.889072][ T12] ---[ end trace 0000000000000000 ]--- [ 4179.563028][ T44] INFO: task modprobe:1901 blocked for more than 122 seconds. [ 4179.563738][ T44] Tainted: G W 6.18.0-virtme #1 [ 4179.564074][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4179.564476][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 4179.565128][ T44] Call Trace: [ 4179.565331][ T44] [ 4179.565482][ T44] __schedule+0x90a/0x1930 [ 4179.565754][ T44] ? io_schedule_timeout+0x150/0x150 [ 4179.566033][ T44] ? __lock_release+0x5d/0x160 [ 4179.566300][ T44] schedule+0xe2/0x270 [ 4179.566500][ T44] schedule_preempt_disabled+0x18/0x30 [ 4179.566781][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4179.567056][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4179.567324][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.567584][ T44] ? lock_acquire+0x104/0x140 [ 4179.567848][ T44] down_write+0x1d6/0x200 [ 4179.568061][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4179.568377][ T44] ? __create_object+0x5e/0xb0 [ 4179.568644][ T44] ? __lock_release+0x5d/0x160 [ 4179.568912][ T44] register_pernet_subsys+0x1a/0x40 [ 4179.569169][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4179.569424][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 4179.569686][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4179.569954][ T44] do_one_initcall+0xc5/0x3d0 [ 4179.570214][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4179.570534][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.570787][ T44] ? trace_kmalloc+0x24/0xb0 [ 4179.571052][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4179.571882][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4179.572152][ T44] ? kasan_unpoison+0x27/0x50 [ 4179.572431][ T44] do_init_module+0x27a/0x860 [ 4179.572694][ T44] ? free_module+0x3f0/0x3f0 [ 4179.572967][ T44] ? kfree+0x2bb/0x5f0 [ 4179.573165][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 4179.573474][ T44] ? load_module+0x20c5/0x2dd0 [ 4179.573733][ T44] load_module+0x1f40/0x2dd0 [ 4179.574007][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4179.574314][ T44] ? kernel_read_file+0x3ed/0x550 [ 4179.574571][ T44] ? kernel_read_file+0x3c8/0x550 [ 4179.574834][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4179.575104][ T44] ? init_module_from_file+0x159/0x180 [ 4179.575356][ T44] init_module_from_file+0x159/0x180 [ 4179.575610][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4179.575863][ T44] ? __lock_acquire+0x664/0xca0 [ 4179.576136][ T44] ? idempotent_init_module+0x316/0x610 [ 4179.576388][ T44] ? __lock_release+0x5d/0x160 [ 4179.576652][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4179.576916][ T44] idempotent_init_module+0x331/0x610 [ 4179.577172][ T44] ? init_module_from_file+0x180/0x180 [ 4179.577433][ T44] ? rcu_is_watching+0x12/0xb0 [ 4179.577686][ T44] ? cap_capable+0x17b/0x340 [ 4179.577956][ T44] __x64_sys_finit_module+0xca/0x150 [ 4179.578211][ T44] ? do_syscall_64+0x85/0xfc0 [ 4179.578482][ T44] do_syscall_64+0xc1/0xfc0 [ 4179.578737][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4179.579055][ T44] RIP: 0033:0x7f1501e4292d [ 4179.579321][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4179.579701][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 4179.580083][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 4179.580463][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 4179.580832][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 4179.581211][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 4179.581606][ T44] [ 4179.581806][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 4179.582220][ T44] [ 4179.582220][ T44] Showing all locks held in the system: [ 4179.582570][ T44] 4 locks held by kworker/u16:0/12: [ 4179.582826][ T44] 1 lock held by khungtaskd/44: [ 4179.583172][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4179.583617][ T44] 2 locks held by kworker/u19:2/1585: [ 4179.583854][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4179.584328][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4179.584847][ T44] 1 lock held by modprobe/1901: [ 4179.585092][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4179.585558][ T44] [ 4179.585694][ T44] ============================================= [ 4179.585694][ T44] [ 4302.441965][ T44] INFO: task modprobe:1901 blocked for more than 245 seconds. [ 4302.442617][ T44] Tainted: G W 6.18.0-virtme #1 [ 4302.443310][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4302.443773][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 4302.444493][ T44] Call Trace: [ 4302.444699][ T44] [ 4302.444896][ T44] __schedule+0x90a/0x1930 [ 4302.445198][ T44] ? io_schedule_timeout+0x150/0x150 [ 4302.445467][ T44] ? __lock_release+0x5d/0x160 [ 4302.445736][ T44] schedule+0xe2/0x270 [ 4302.445947][ T44] schedule_preempt_disabled+0x18/0x30 [ 4302.446200][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4302.446471][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4302.446796][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.447129][ T44] ? lock_acquire+0x104/0x140 [ 4302.447450][ T44] down_write+0x1d6/0x200 [ 4302.447676][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4302.448005][ T44] ? __create_object+0x5e/0xb0 [ 4302.448259][ T44] ? __lock_release+0x5d/0x160 [ 4302.448549][ T44] register_pernet_subsys+0x1a/0x40 [ 4302.448881][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4302.449203][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 4302.449523][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4302.449845][ T44] do_one_initcall+0xc5/0x3d0 [ 4302.450175][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4302.450498][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.450816][ T44] ? trace_kmalloc+0x24/0xb0 [ 4302.451141][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4302.451454][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4302.451783][ T44] ? kasan_unpoison+0x27/0x50 [ 4302.452127][ T44] do_init_module+0x27a/0x860 [ 4302.452409][ T44] ? free_module+0x3f0/0x3f0 [ 4302.452722][ T44] ? kfree+0x2bb/0x5f0 [ 4302.452960][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 4302.453271][ T44] ? load_module+0x20c5/0x2dd0 [ 4302.453536][ T44] load_module+0x1f40/0x2dd0 [ 4302.453856][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4302.454254][ T44] ? kernel_read_file+0x3ed/0x550 [ 4302.454550][ T44] ? kernel_read_file+0x3c8/0x550 [ 4302.454866][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4302.455157][ T44] ? init_module_from_file+0x159/0x180 [ 4302.455410][ T44] init_module_from_file+0x159/0x180 [ 4302.455735][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4302.456059][ T44] ? __lock_acquire+0x664/0xca0 [ 4302.456393][ T44] ? idempotent_init_module+0x316/0x610 [ 4302.456707][ T44] ? __lock_release+0x5d/0x160 [ 4302.457042][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4302.457295][ T44] idempotent_init_module+0x331/0x610 [ 4302.457617][ T44] ? init_module_from_file+0x180/0x180 [ 4302.457957][ T44] ? rcu_is_watching+0x12/0xb0 [ 4302.458209][ T44] ? cap_capable+0x17b/0x340 [ 4302.458505][ T44] __x64_sys_finit_module+0xca/0x150 [ 4302.458819][ T44] ? do_syscall_64+0x85/0xfc0 [ 4302.459102][ T44] do_syscall_64+0xc1/0xfc0 [ 4302.459423][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4302.459808][ T44] RIP: 0033:0x7f1501e4292d [ 4302.460138][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4302.460530][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 4302.461011][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 4302.461471][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 4302.461951][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 4302.462364][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 4302.462844][ T44] [ 4302.463095][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 4302.463562][ T44] [ 4302.463562][ T44] Showing all locks held in the system: [ 4302.464103][ T44] 4 locks held by kworker/u16:0/12: [ 4302.464404][ T44] 1 lock held by khungtaskd/44: [ 4302.464641][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4302.465194][ T44] 2 locks held by kworker/u19:2/1585: [ 4302.465455][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4302.465929][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4302.466595][ T44] 1 lock held by modprobe/1901: [ 4302.466902][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4302.467408][ T44] [ 4302.467572][ T44] ============================================= [ 4302.467572][ T44] [ 4425.321961][ T44] INFO: task modprobe:1901 blocked for more than 368 seconds. [ 4425.322629][ T44] Tainted: G W 6.18.0-virtme #1 [ 4425.323069][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4425.323535][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 4425.324191][ T44] Call Trace: [ 4425.324392][ T44] [ 4425.324556][ T44] __schedule+0x90a/0x1930 [ 4425.324888][ T44] ? io_schedule_timeout+0x150/0x150 [ 4425.325218][ T44] ? __lock_release+0x5d/0x160 [ 4425.325538][ T44] schedule+0xe2/0x270 [ 4425.325779][ T44] schedule_preempt_disabled+0x18/0x30 [ 4425.326099][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4425.326413][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4425.326736][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.327052][ T44] ? lock_acquire+0x104/0x140 [ 4425.327306][ T44] down_write+0x1d6/0x200 [ 4425.327500][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4425.327839][ T44] ? __create_object+0x5e/0xb0 [ 4425.328169][ T44] ? __lock_release+0x5d/0x160 [ 4425.328481][ T44] register_pernet_subsys+0x1a/0x40 [ 4425.328799][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4425.329121][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 4425.329379][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4425.329664][ T44] do_one_initcall+0xc5/0x3d0 [ 4425.329985][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4425.330303][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.330625][ T44] ? trace_kmalloc+0x24/0xb0 [ 4425.330879][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4425.331135][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4425.331422][ T44] ? kasan_unpoison+0x27/0x50 [ 4425.331733][ T44] do_init_module+0x27a/0x860 [ 4425.332045][ T44] ? free_module+0x3f0/0x3f0 [ 4425.332362][ T44] ? kfree+0x2bb/0x5f0 [ 4425.332605][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 4425.333005][ T44] ? load_module+0x20c5/0x2dd0 [ 4425.333287][ T44] load_module+0x1f40/0x2dd0 [ 4425.333586][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4425.333923][ T44] ? kernel_read_file+0x3ed/0x550 [ 4425.334238][ T44] ? kernel_read_file+0x3c8/0x550 [ 4425.334551][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4425.334864][ T44] ? init_module_from_file+0x159/0x180 [ 4425.335153][ T44] init_module_from_file+0x159/0x180 [ 4425.335433][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4425.335752][ T44] ? __lock_acquire+0x664/0xca0 [ 4425.336018][ T44] ? idempotent_init_module+0x316/0x610 [ 4425.336330][ T44] ? __lock_release+0x5d/0x160 [ 4425.336639][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4425.336957][ T44] idempotent_init_module+0x331/0x610 [ 4425.337268][ T44] ? init_module_from_file+0x180/0x180 [ 4425.337585][ T44] ? rcu_is_watching+0x12/0xb0 [ 4425.337908][ T44] ? cap_capable+0x17b/0x340 [ 4425.338223][ T44] __x64_sys_finit_module+0xca/0x150 [ 4425.338534][ T44] ? do_syscall_64+0x85/0xfc0 [ 4425.338856][ T44] do_syscall_64+0xc1/0xfc0 [ 4425.339112][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4425.339409][ T44] RIP: 0033:0x7f1501e4292d [ 4425.339662][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4425.340044][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 4425.340502][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 4425.340939][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 4425.341296][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 4425.341660][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 4425.342136][ T44] [ 4425.342378][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 4425.342885][ T44] [ 4425.342885][ T44] Showing all locks held in the system: [ 4425.343317][ T44] 4 locks held by kworker/u16:0/12: [ 4425.343634][ T44] 1 lock held by khungtaskd/44: [ 4425.343986][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4425.344509][ T44] 2 locks held by kworker/u19:2/1585: [ 4425.344801][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4425.345327][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4425.345839][ T44] 1 lock held by modprobe/1901: [ 4425.346080][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4425.346528][ T44] [ 4425.346658][ T44] ============================================= [ 4425.346658][ T44] [ 4548.201974][ T44] INFO: task modprobe:1901 blocked for more than 491 seconds. [ 4548.202652][ T44] Tainted: G W 6.18.0-virtme #1 [ 4548.202973][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4548.203380][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 4548.203977][ T44] Call Trace: [ 4548.204232][ T44] [ 4548.204412][ T44] __schedule+0x90a/0x1930 [ 4548.204730][ T44] ? io_schedule_timeout+0x150/0x150 [ 4548.205097][ T44] ? __lock_release+0x5d/0x160 [ 4548.205442][ T44] schedule+0xe2/0x270 [ 4548.205686][ T44] schedule_preempt_disabled+0x18/0x30 [ 4548.206010][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4548.206264][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4548.206538][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.206796][ T44] ? lock_acquire+0x104/0x140 [ 4548.207124][ T44] down_write+0x1d6/0x200 [ 4548.207373][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4548.207755][ T44] ? __create_object+0x5e/0xb0 [ 4548.208025][ T44] ? __lock_release+0x5d/0x160 [ 4548.208337][ T44] register_pernet_subsys+0x1a/0x40 [ 4548.208646][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4548.208914][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 4548.209195][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4548.209509][ T44] do_one_initcall+0xc5/0x3d0 [ 4548.209822][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4548.210211][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.210521][ T44] ? trace_kmalloc+0x24/0xb0 [ 4548.210836][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4548.211155][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4548.211471][ T44] ? kasan_unpoison+0x27/0x50 [ 4548.211741][ T44] do_init_module+0x27a/0x860 [ 4548.212081][ T44] ? free_module+0x3f0/0x3f0 [ 4548.212388][ T44] ? kfree+0x2bb/0x5f0 [ 4548.212628][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 4548.213021][ T44] ? load_module+0x20c5/0x2dd0 [ 4548.213339][ T44] load_module+0x1f40/0x2dd0 [ 4548.213658][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4548.213968][ T44] ? kernel_read_file+0x3ed/0x550 [ 4548.214288][ T44] ? kernel_read_file+0x3c8/0x550 [ 4548.214599][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4548.214853][ T44] ? init_module_from_file+0x159/0x180 [ 4548.215173][ T44] init_module_from_file+0x159/0x180 [ 4548.215491][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4548.215802][ T44] ? __lock_acquire+0x664/0xca0 [ 4548.216090][ T44] ? idempotent_init_module+0x316/0x610 [ 4548.216335][ T44] ? __lock_release+0x5d/0x160 [ 4548.216584][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4548.216832][ T44] idempotent_init_module+0x331/0x610 [ 4548.217102][ T44] ? init_module_from_file+0x180/0x180 [ 4548.217360][ T44] ? rcu_is_watching+0x12/0xb0 [ 4548.217607][ T44] ? cap_capable+0x17b/0x340 [ 4548.217862][ T44] __x64_sys_finit_module+0xca/0x150 [ 4548.218118][ T44] ? do_syscall_64+0x85/0xfc0 [ 4548.218367][ T44] do_syscall_64+0xc1/0xfc0 [ 4548.218623][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4548.218937][ T44] RIP: 0033:0x7f1501e4292d [ 4548.219201][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4548.219565][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 4548.219991][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 4548.220444][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 4548.220824][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 4548.221258][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 4548.221724][ T44] [ 4548.221924][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 4548.222367][ T44] [ 4548.222367][ T44] Showing all locks held in the system: [ 4548.222818][ T44] 4 locks held by kworker/u16:0/12: [ 4548.223167][ T44] 1 lock held by khungtaskd/44: [ 4548.223463][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4548.223959][ T44] 2 locks held by kworker/u19:2/1585: [ 4548.224196][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4548.224767][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4548.225421][ T44] 1 lock held by modprobe/1901: [ 4548.225684][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4548.226227][ T44] [ 4548.226388][ T44] ============================================= [ 4548.226388][ T44] [ 4671.081972][ T44] INFO: task modprobe:1901 blocked for more than 614 seconds. [ 4671.082609][ T44] Tainted: G W 6.18.0-virtme #1 [ 4671.082940][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4671.083360][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 4671.084134][ T44] Call Trace: [ 4671.084404][ T44] [ 4671.084593][ T44] __schedule+0x90a/0x1930 [ 4671.084929][ T44] ? io_schedule_timeout+0x150/0x150 [ 4671.085303][ T44] ? __lock_release+0x5d/0x160 [ 4671.085571][ T44] schedule+0xe2/0x270 [ 4671.085821][ T44] schedule_preempt_disabled+0x18/0x30 [ 4671.086152][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4671.086457][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4671.086777][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.087113][ T44] ? lock_acquire+0x104/0x140 [ 4671.087452][ T44] down_write+0x1d6/0x200 [ 4671.087666][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4671.087991][ T44] ? __create_object+0x5e/0xb0 [ 4671.088314][ T44] ? __lock_release+0x5d/0x160 [ 4671.088635][ T44] register_pernet_subsys+0x1a/0x40 [ 4671.088918][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4671.089249][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 4671.089576][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4671.089860][ T44] do_one_initcall+0xc5/0x3d0 [ 4671.090196][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4671.090525][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.090843][ T44] ? trace_kmalloc+0x24/0xb0 [ 4671.091175][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4671.091493][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4671.091830][ T44] ? kasan_unpoison+0x27/0x50 [ 4671.092099][ T44] do_init_module+0x27a/0x860 [ 4671.092387][ T44] ? free_module+0x3f0/0x3f0 [ 4671.092643][ T44] ? kfree+0x2bb/0x5f0 [ 4671.092862][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 4671.093261][ T44] ? load_module+0x20c5/0x2dd0 [ 4671.093554][ T44] load_module+0x1f40/0x2dd0 [ 4671.093830][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4671.094157][ T44] ? kernel_read_file+0x3ed/0x550 [ 4671.094482][ T44] ? kernel_read_file+0x3c8/0x550 [ 4671.094785][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4671.095061][ T44] ? init_module_from_file+0x159/0x180 [ 4671.095319][ T44] init_module_from_file+0x159/0x180 [ 4671.095577][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4671.095862][ T44] ? __lock_acquire+0x664/0xca0 [ 4671.096201][ T44] ? idempotent_init_module+0x316/0x610 [ 4671.096519][ T44] ? __lock_release+0x5d/0x160 [ 4671.096830][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4671.097128][ T44] idempotent_init_module+0x331/0x610 [ 4671.097447][ T44] ? init_module_from_file+0x180/0x180 [ 4671.097772][ T44] ? rcu_is_watching+0x12/0xb0 [ 4671.098107][ T44] ? cap_capable+0x17b/0x340 [ 4671.098430][ T44] __x64_sys_finit_module+0xca/0x150 [ 4671.098710][ T44] ? do_syscall_64+0x85/0xfc0 [ 4671.098975][ T44] do_syscall_64+0xc1/0xfc0 [ 4671.099290][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4671.099687][ T44] RIP: 0033:0x7f1501e4292d [ 4671.100024][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4671.100514][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 4671.100948][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 4671.101321][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 4671.101779][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 4671.102292][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 4671.102769][ T44] [ 4671.103021][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 4671.103497][ T44] [ 4671.103497][ T44] Showing all locks held in the system: [ 4671.103845][ T44] 4 locks held by kworker/u16:0/12: [ 4671.104213][ T44] 1 lock held by khungtaskd/44: [ 4671.104514][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4671.105045][ T44] 2 locks held by kworker/u19:2/1585: [ 4671.105288][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4671.105844][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4671.106528][ T44] 1 lock held by modprobe/1901: [ 4671.106770][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4671.107368][ T44] [ 4671.107505][ T44] ============================================= [ 4671.107505][ T44] [ 4793.961964][ T44] INFO: task modprobe:1901 blocked for more than 737 seconds. [ 4793.962592][ T44] Tainted: G W 6.18.0-virtme #1 [ 4793.962928][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4793.963411][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 4793.964111][ T44] Call Trace: [ 4793.964372][ T44] [ 4793.964566][ T44] __schedule+0x90a/0x1930 [ 4793.964917][ T44] ? io_schedule_timeout+0x150/0x150 [ 4793.965241][ T44] ? __lock_release+0x5d/0x160 [ 4793.965577][ T44] schedule+0xe2/0x270 [ 4793.965784][ T44] schedule_preempt_disabled+0x18/0x30 [ 4793.966114][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4793.966437][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4793.966770][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.967099][ T44] ? lock_acquire+0x104/0x140 [ 4793.967423][ T44] down_write+0x1d6/0x200 [ 4793.967671][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4793.968010][ T44] ? __create_object+0x5e/0xb0 [ 4793.968343][ T44] ? __lock_release+0x5d/0x160 [ 4793.968648][ T44] register_pernet_subsys+0x1a/0x40 [ 4793.968936][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4793.969268][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 4793.969561][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4793.969815][ T44] do_one_initcall+0xc5/0x3d0 [ 4793.970148][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4793.970524][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.970782][ T44] ? trace_kmalloc+0x24/0xb0 [ 4793.971096][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4793.971415][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4793.971703][ T44] ? kasan_unpoison+0x27/0x50 [ 4793.971978][ T44] do_init_module+0x27a/0x860 [ 4793.972241][ T44] ? free_module+0x3f0/0x3f0 [ 4793.972514][ T44] ? kfree+0x2bb/0x5f0 [ 4793.972773][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 4793.973155][ T44] ? load_module+0x20c5/0x2dd0 [ 4793.973427][ T44] load_module+0x1f40/0x2dd0 [ 4793.973745][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4793.974143][ T44] ? kernel_read_file+0x3ed/0x550 [ 4793.974452][ T44] ? kernel_read_file+0x3c8/0x550 [ 4793.974706][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4793.974991][ T44] ? init_module_from_file+0x159/0x180 [ 4793.975246][ T44] init_module_from_file+0x159/0x180 [ 4793.975575][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4793.975905][ T44] ? __lock_acquire+0x664/0xca0 [ 4793.976233][ T44] ? idempotent_init_module+0x316/0x610 [ 4793.976547][ T44] ? __lock_release+0x5d/0x160 [ 4793.976864][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4793.977192][ T44] idempotent_init_module+0x331/0x610 [ 4793.977512][ T44] ? init_module_from_file+0x180/0x180 [ 4793.977846][ T44] ? rcu_is_watching+0x12/0xb0 [ 4793.978174][ T44] ? cap_capable+0x17b/0x340 [ 4793.978499][ T44] __x64_sys_finit_module+0xca/0x150 [ 4793.978768][ T44] ? do_syscall_64+0x85/0xfc0 [ 4793.979088][ T44] do_syscall_64+0xc1/0xfc0 [ 4793.979369][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4793.979758][ T44] RIP: 0033:0x7f1501e4292d [ 4793.980099][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4793.980566][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 4793.981040][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 4793.981476][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 4793.981988][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 4793.982401][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 4793.982788][ T44] [ 4793.983049][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 4793.983560][ T44] [ 4793.983560][ T44] Showing all locks held in the system: [ 4793.984118][ T44] 4 locks held by kworker/u16:0/12: [ 4793.984432][ T44] 1 lock held by khungtaskd/44: [ 4793.984738][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4793.985285][ T44] 2 locks held by kworker/u19:2/1585: [ 4793.985526][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4793.986036][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4793.986702][ T44] 1 lock held by modprobe/1901: [ 4793.987016][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4793.987611][ T44] [ 4793.987779][ T44] ============================================= [ 4793.987779][ T44] [ 4916.841978][ T44] INFO: task modprobe:1901 blocked for more than 860 seconds. [ 4916.842595][ T44] Tainted: G W 6.18.0-virtme #1 [ 4916.843005][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 4916.843446][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 4916.844105][ T44] Call Trace: [ 4916.844355][ T44] [ 4916.844538][ T44] __schedule+0x90a/0x1930 [ 4916.844826][ T44] ? io_schedule_timeout+0x150/0x150 [ 4916.845095][ T44] ? __lock_release+0x5d/0x160 [ 4916.845392][ T44] schedule+0xe2/0x270 [ 4916.845639][ T44] schedule_preempt_disabled+0x18/0x30 [ 4916.845968][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 4916.846297][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 4916.846595][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.846846][ T44] ? lock_acquire+0x104/0x140 [ 4916.847115][ T44] down_write+0x1d6/0x200 [ 4916.847366][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 4916.847751][ T44] ? __create_object+0x5e/0xb0 [ 4916.848074][ T44] ? __lock_release+0x5d/0x160 [ 4916.848399][ T44] register_pernet_subsys+0x1a/0x40 [ 4916.848720][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4916.849045][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 4916.849302][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 4916.849583][ T44] do_one_initcall+0xc5/0x3d0 [ 4916.849911][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 4916.850294][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.850554][ T44] ? trace_kmalloc+0x24/0xb0 [ 4916.850803][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 4916.851129][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 4916.851442][ T44] ? kasan_unpoison+0x27/0x50 [ 4916.851693][ T44] do_init_module+0x27a/0x860 [ 4916.851963][ T44] ? free_module+0x3f0/0x3f0 [ 4916.852275][ T44] ? kfree+0x2bb/0x5f0 [ 4916.852550][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 4916.852949][ T44] ? load_module+0x20c5/0x2dd0 [ 4916.853252][ T44] load_module+0x1f40/0x2dd0 [ 4916.853517][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 4916.853837][ T44] ? kernel_read_file+0x3ed/0x550 [ 4916.854149][ T44] ? kernel_read_file+0x3c8/0x550 [ 4916.854434][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 4916.854752][ T44] ? init_module_from_file+0x159/0x180 [ 4916.855079][ T44] init_module_from_file+0x159/0x180 [ 4916.855327][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 4916.855649][ T44] ? __lock_acquire+0x664/0xca0 [ 4916.855979][ T44] ? idempotent_init_module+0x316/0x610 [ 4916.856289][ T44] ? __lock_release+0x5d/0x160 [ 4916.856568][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 4916.856861][ T44] idempotent_init_module+0x331/0x610 [ 4916.857186][ T44] ? init_module_from_file+0x180/0x180 [ 4916.857523][ T44] ? rcu_is_watching+0x12/0xb0 [ 4916.857833][ T44] ? cap_capable+0x17b/0x340 [ 4916.858161][ T44] __x64_sys_finit_module+0xca/0x150 [ 4916.858423][ T44] ? do_syscall_64+0x85/0xfc0 [ 4916.858698][ T44] do_syscall_64+0xc1/0xfc0 [ 4916.859024][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 4916.859411][ T44] RIP: 0033:0x7f1501e4292d [ 4916.859728][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 4916.860194][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 4916.860594][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 4916.861055][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 4916.861432][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 4916.861843][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 4916.862292][ T44] [ 4916.862490][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 4916.863020][ T44] [ 4916.863020][ T44] Showing all locks held in the system: [ 4916.863395][ T44] 4 locks held by kworker/u16:0/12: [ 4916.863646][ T44] 1 lock held by khungtaskd/44: [ 4916.863989][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 4916.864526][ T44] 2 locks held by kworker/u19:2/1585: [ 4916.864826][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 4916.865430][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 4916.866095][ T44] 1 lock held by modprobe/1901: [ 4916.866401][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 4916.866880][ T44] [ 4916.867051][ T44] ============================================= [ 4916.867051][ T44] [ 5039.721962][ T44] INFO: task modprobe:1901 blocked for more than 983 seconds. [ 5039.722535][ T44] Tainted: G W 6.18.0-virtme #1 [ 5039.722843][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5039.723415][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 5039.724162][ T44] Call Trace: [ 5039.724364][ T44] [ 5039.724512][ T44] __schedule+0x90a/0x1930 [ 5039.724784][ T44] ? io_schedule_timeout+0x150/0x150 [ 5039.725125][ T44] ? __lock_release+0x5d/0x160 [ 5039.725422][ T44] schedule+0xe2/0x270 [ 5039.725634][ T44] schedule_preempt_disabled+0x18/0x30 [ 5039.725908][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5039.726199][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5039.726523][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.726840][ T44] ? lock_acquire+0x104/0x140 [ 5039.727140][ T44] down_write+0x1d6/0x200 [ 5039.727338][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5039.727657][ T44] ? __create_object+0x5e/0xb0 [ 5039.727959][ T44] ? __lock_release+0x5d/0x160 [ 5039.728255][ T44] register_pernet_subsys+0x1a/0x40 [ 5039.728513][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 5039.728767][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 5039.729080][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 5039.729332][ T44] do_one_initcall+0xc5/0x3d0 [ 5039.729594][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5039.729915][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.730169][ T44] ? trace_kmalloc+0x24/0xb0 [ 5039.730420][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5039.730672][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5039.730936][ T44] ? kasan_unpoison+0x27/0x50 [ 5039.731253][ T44] do_init_module+0x27a/0x860 [ 5039.731587][ T44] ? free_module+0x3f0/0x3f0 [ 5039.731910][ T44] ? kfree+0x2bb/0x5f0 [ 5039.732139][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 5039.732478][ T44] ? load_module+0x20c5/0x2dd0 [ 5039.732807][ T44] load_module+0x1f40/0x2dd0 [ 5039.733102][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5039.733498][ T44] ? kernel_read_file+0x3ed/0x550 [ 5039.733815][ T44] ? kernel_read_file+0x3c8/0x550 [ 5039.734148][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5039.734422][ T44] ? init_module_from_file+0x159/0x180 [ 5039.734736][ T44] init_module_from_file+0x159/0x180 [ 5039.735063][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5039.735334][ T44] ? __lock_acquire+0x664/0xca0 [ 5039.735662][ T44] ? idempotent_init_module+0x316/0x610 [ 5039.735983][ T44] ? __lock_release+0x5d/0x160 [ 5039.736259][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5039.736587][ T44] idempotent_init_module+0x331/0x610 [ 5039.736918][ T44] ? init_module_from_file+0x180/0x180 [ 5039.737202][ T44] ? rcu_is_watching+0x12/0xb0 [ 5039.737519][ T44] ? cap_capable+0x17b/0x340 [ 5039.737839][ T44] __x64_sys_finit_module+0xca/0x150 [ 5039.738127][ T44] ? do_syscall_64+0x85/0xfc0 [ 5039.738389][ T44] do_syscall_64+0xc1/0xfc0 [ 5039.738712][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5039.739086][ T44] RIP: 0033:0x7f1501e4292d [ 5039.739405][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5039.739860][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 5039.740282][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 5039.740746][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 5039.741155][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 5039.741616][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 5039.742099][ T44] [ 5039.742301][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 5039.742749][ T44] [ 5039.742749][ T44] Showing all locks held in the system: [ 5039.743293][ T44] 4 locks held by kworker/u16:0/12: [ 5039.743608][ T44] 1 lock held by khungtaskd/44: [ 5039.743914][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5039.744341][ T44] 2 locks held by kworker/u19:2/1585: [ 5039.744574][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5039.745162][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5039.745698][ T44] 1 lock held by modprobe/1901: [ 5039.746004][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5039.746540][ T44] [ 5039.746704][ T44] ============================================= [ 5039.746704][ T44] [ 5162.601958][ T44] INFO: task modprobe:1901 blocked for more than 1105 seconds. [ 5162.602614][ T44] Tainted: G W 6.18.0-virtme #1 [ 5162.603022][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5162.603488][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 5162.604176][ T44] Call Trace: [ 5162.604424][ T44] [ 5162.604605][ T44] __schedule+0x90a/0x1930 [ 5162.604927][ T44] ? io_schedule_timeout+0x150/0x150 [ 5162.605239][ T44] ? __lock_release+0x5d/0x160 [ 5162.605588][ T44] schedule+0xe2/0x270 [ 5162.605837][ T44] schedule_preempt_disabled+0x18/0x30 [ 5162.606108][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5162.606431][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5162.606757][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.607084][ T44] ? lock_acquire+0x104/0x140 [ 5162.607405][ T44] down_write+0x1d6/0x200 [ 5162.607662][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5162.607980][ T44] ? __create_object+0x5e/0xb0 [ 5162.608235][ T44] ? __lock_release+0x5d/0x160 [ 5162.608497][ T44] register_pernet_subsys+0x1a/0x40 [ 5162.608754][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 5162.609033][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 5162.609288][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 5162.609549][ T44] do_one_initcall+0xc5/0x3d0 [ 5162.609806][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5162.610141][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.610454][ T44] ? trace_kmalloc+0x24/0xb0 [ 5162.610767][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5162.611098][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5162.611356][ T44] ? kasan_unpoison+0x27/0x50 [ 5162.611618][ T44] do_init_module+0x27a/0x860 [ 5162.611886][ T44] ? free_module+0x3f0/0x3f0 [ 5162.612200][ T44] ? kfree+0x2bb/0x5f0 [ 5162.612443][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 5162.612832][ T44] ? load_module+0x20c5/0x2dd0 [ 5162.613176][ T44] load_module+0x1f40/0x2dd0 [ 5162.613502][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5162.613905][ T44] ? kernel_read_file+0x3ed/0x550 [ 5162.614226][ T44] ? kernel_read_file+0x3c8/0x550 [ 5162.614543][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5162.614817][ T44] ? init_module_from_file+0x159/0x180 [ 5162.615143][ T44] init_module_from_file+0x159/0x180 [ 5162.615462][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5162.615779][ T44] ? __lock_acquire+0x664/0xca0 [ 5162.616104][ T44] ? idempotent_init_module+0x316/0x610 [ 5162.616355][ T44] ? __lock_release+0x5d/0x160 [ 5162.616607][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5162.616862][ T44] idempotent_init_module+0x331/0x610 [ 5162.617130][ T44] ? init_module_from_file+0x180/0x180 [ 5162.617391][ T44] ? rcu_is_watching+0x12/0xb0 [ 5162.617644][ T44] ? cap_capable+0x17b/0x340 [ 5162.617951][ T44] __x64_sys_finit_module+0xca/0x150 [ 5162.618266][ T44] ? do_syscall_64+0x85/0xfc0 [ 5162.618583][ T44] do_syscall_64+0xc1/0xfc0 [ 5162.618915][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5162.619224][ T44] RIP: 0033:0x7f1501e4292d [ 5162.619553][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5162.619930][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 5162.620383][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 5162.620820][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 5162.621241][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 5162.621701][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 5162.622178][ T44] [ 5162.622372][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 5162.622767][ T44] [ 5162.622767][ T44] Showing all locks held in the system: [ 5162.623214][ T44] 4 locks held by kworker/u16:0/12: [ 5162.623534][ T44] 1 lock held by khungtaskd/44: [ 5162.623830][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5162.624325][ T44] 2 locks held by kworker/u19:2/1585: [ 5162.624628][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5162.625149][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5162.625808][ T44] 1 lock held by modprobe/1901: [ 5162.626110][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5162.626661][ T44] [ 5162.626825][ T44] ============================================= [ 5162.626825][ T44] [ 5285.481961][ T44] INFO: task modprobe:1901 blocked for more than 1228 seconds. [ 5285.482599][ T44] Tainted: G W 6.18.0-virtme #1 [ 5285.483060][ T44] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5285.483579][ T44] task:modprobe state:D stack:25208 pid:1901 tgid:1901 ppid:1585 task_flags:0x400100 flags:0x00080000 [ 5285.484277][ T44] Call Trace: [ 5285.484536][ T44] [ 5285.484687][ T44] __schedule+0x90a/0x1930 [ 5285.485010][ T44] ? io_schedule_timeout+0x150/0x150 [ 5285.485354][ T44] ? __lock_release+0x5d/0x160 [ 5285.485682][ T44] schedule+0xe2/0x270 [ 5285.485898][ T44] schedule_preempt_disabled+0x18/0x30 [ 5285.486216][ T44] rwsem_down_write_slowpath+0x3c5/0xc60 [ 5285.486536][ T44] ? rwsem_down_read_slowpath+0xd80/0xd80 [ 5285.486797][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.487088][ T44] ? lock_acquire+0x104/0x140 [ 5285.487395][ T44] down_write+0x1d6/0x200 [ 5285.487645][ T44] ? rwsem_down_write_slowpath+0xc60/0xc60 [ 5285.488000][ T44] ? __create_object+0x5e/0xb0 [ 5285.488323][ T44] ? __lock_release+0x5d/0x160 [ 5285.488641][ T44] register_pernet_subsys+0x1a/0x40 [ 5285.488916][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 5285.489173][ T44] mpls_init+0x17/0x1000 [mpls_router] [ 5285.489455][ T44] ? ip6gre_fini+0xf00/0xf00 [ip6_gre] [ 5285.489740][ T44] do_one_initcall+0xc5/0x3d0 [ 5285.490068][ T44] ? trace_event_raw_event_initcall_level+0x1f0/0x1f0 [ 5285.490455][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.490749][ T44] ? trace_kmalloc+0x24/0xb0 [ 5285.491042][ T44] ? __kmalloc_cache_noprof+0x2e4/0x710 [ 5285.491368][ T44] ? debug_objects_fill_pool+0x3e0/0x3e0 [ 5285.491688][ T44] ? kasan_unpoison+0x27/0x50 [ 5285.491955][ T44] do_init_module+0x27a/0x860 [ 5285.492242][ T44] ? free_module+0x3f0/0x3f0 [ 5285.492550][ T44] ? kfree+0x2bb/0x5f0 [ 5285.492747][ T44] ? ingress_module_exit+0xb0/0xb0 [sch_ingress] [ 5285.493150][ T44] ? load_module+0x20c5/0x2dd0 [ 5285.493466][ T44] load_module+0x1f40/0x2dd0 [ 5285.493749][ T44] ? layout_and_allocate.constprop.0+0x630/0x630 [ 5285.494127][ T44] ? kernel_read_file+0x3ed/0x550 [ 5285.494444][ T44] ? kernel_read_file+0x3c8/0x550 [ 5285.494742][ T44] ? __ia32_sys_fsconfig+0x150/0x150 [ 5285.495013][ T44] ? init_module_from_file+0x159/0x180 [ 5285.495265][ T44] init_module_from_file+0x159/0x180 [ 5285.495568][ T44] ? __do_sys_init_module+0x1e0/0x1e0 [ 5285.495899][ T44] ? __lock_acquire+0x664/0xca0 [ 5285.496222][ T44] ? idempotent_init_module+0x316/0x610 [ 5285.496532][ T44] ? __lock_release+0x5d/0x160 [ 5285.496848][ T44] ? do_raw_spin_unlock+0x58/0x220 [ 5285.497177][ T44] idempotent_init_module+0x331/0x610 [ 5285.497492][ T44] ? init_module_from_file+0x180/0x180 [ 5285.497785][ T44] ? rcu_is_watching+0x12/0xb0 [ 5285.498054][ T44] ? cap_capable+0x17b/0x340 [ 5285.498311][ T44] __x64_sys_finit_module+0xca/0x150 [ 5285.498561][ T44] ? do_syscall_64+0x85/0xfc0 [ 5285.498895][ T44] do_syscall_64+0xc1/0xfc0 [ 5285.499208][ T44] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 5285.499596][ T44] RIP: 0033:0x7f1501e4292d [ 5285.499864][ T44] RSP: 002b:00007ffc28d67588 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 [ 5285.500249][ T44] RAX: ffffffffffffffda RBX: 0000558915902df0 RCX: 00007f1501e4292d [ 5285.500615][ T44] RDX: 0000000000000000 RSI: 00005588f0699a2a RDI: 0000000000000000 [ 5285.501009][ T44] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000 [ 5285.501465][ T44] R10: 0000000000000000 R11: 0000000000000246 R12: 00005588f0699a2a [ 5285.501936][ T44] R13: 0000558915903000 R14: 00005589159032a0 R15: 0000000000000000 [ 5285.502409][ T44] [ 5285.502661][ T44] INFO: task modprobe:1901 is blocked on an rw-semaphore, but the owner is not found. [ 5285.503122][ T44] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings [ 5285.503627][ T44] [ 5285.503627][ T44] Showing all locks held in the system: [ 5285.504110][ T44] 4 locks held by kworker/u16:0/12: [ 5285.504423][ T44] 1 lock held by khungtaskd/44: [ 5285.504724][ T44] #0: ffffffff9857fe20 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x260 [ 5285.505174][ T44] 2 locks held by kworker/u19:2/1585: [ 5285.505411][ T44] #0: ffff888001099d48 ((wq_completion)events_unbound#2){+.+.}-{0:0}, at: process_one_work+0x1034/0x1810 [ 5285.506023][ T44] #1: ffffc90000aefcb8 ((work_completion)(&sub_info->work)){+.+.}-{0:0}, at: process_one_work+0x802/0x1810 [ 5285.506538][ T44] 1 lock held by modprobe/1901: [ 5285.506827][ T44] #0: ffffffff98c68cd0 (pernet_ops_rwsem){++++}-{4:4}, at: register_pernet_subsys+0x1a/0x40 [ 5285.507414][ T44] [ 5285.507578][ T44] ============================================= [ 5285.507578][ T44] WAIT TIMEOUT stderr Ctrl-C stderr Ctrl-C stderr