[ 632.195259][ T15] sched: DL replenish lagged too much [ 632.196157][ C1] [ 632.196161][ C1] ======================================================== [ 632.196164][ C1] WARNING: possible irq lock inversion dependency detected [ 632.196168][ C1] 6.17.0-rc1-virtme #1 Not tainted [ 632.196174][ C1] -------------------------------------------------------- [ 632.196176][ C1] swapper/1/0 just changed the state of lock: [ 632.196181][ C1] ffffffff9f877e60 (console_owner){-...}-{0:0}, at: console_lock_spinning_enable+0x7d/0xb0 [ 632.196214][ C1] but this lock took another, HARDIRQ-unsafe lock in the past: [ 632.196217][ C1] (_xmit_ETHER#2){+.-.}-{3:3} [ 632.196226][ C1] [ 632.196226][ C1] [ 632.196226][ C1] and interrupts could create inverse lock ordering between them. [ 632.196226][ C1] [ 632.196229][ C1] [ 632.196229][ C1] other info that might help us debug this: [ 632.196232][ C1] Chain exists of: [ 632.196232][ C1] console_owner --> target_list_lock --> _xmit_ETHER#2 [ 632.196232][ C1] [ 632.196246][ C1] Possible interrupt unsafe locking scenario: [ 632.196246][ C1] [ 632.196249][ C1] CPU0 CPU1 [ 632.196251][ C1] ---- ---- [ 632.196253][ C1] lock(_xmit_ETHER#2); [ 632.196259][ C1] local_irq_disable(); [ 632.196262][ C1] lock(console_owner); [ 632.196266][ C1] lock(target_list_lock); [ 632.196272][ C1] [ 632.196273][ C1] lock(console_owner); [ 632.196278][ C1] [ 632.196278][ C1] *** DEADLOCK *** [ 632.196278][ C1] [ 632.196280][ C1] 2 locks held by swapper/1/0: [ 632.196285][ C1] #0: ffffffff9f9584c0 (console_lock){+.+.}-{0:0}, at: irq_work_single+0x108/0x210 [ 632.196308][ C1] #1: ffffffff9f958530 (console_srcu){....}-{0:0}, at: console_flush_all+0x12f/0x620 [ 632.196326][ C1] [ 632.196326][ C1] the shortest dependencies between 2nd lock and 1st lock: [ 632.196344][ C1] -> (_xmit_ETHER#2){+.-.}-{3:3} { [ 632.196355][ C1] HARDIRQ-ON-W at: [ 632.196360][ C1] __lock_acquire+0x20b/0x7e0 [ 632.196370][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.196378][ C1] _raw_spin_trylock+0x69/0x80 [ 632.196391][ C1] virtnet_poll_cleantx.isra.0+0x1fe/0x500 [ 632.196402][ C1] virtnet_poll+0xf9/0xad0 [ 632.196409][ C1] __napi_poll.constprop.0+0x9f/0x460 [ 632.196421][ C1] net_rx_action+0x54f/0xda0 [ 632.196430][ C1] handle_softirqs+0x215/0x610 [ 632.196441][ C1] do_softirq+0xad/0xe0 [ 632.196448][ C1] __local_bh_enable_ip+0x101/0x120 [ 632.196455][ C1] virtnet_open+0x228/0xb40 [ 632.196462][ C1] __dev_open+0x224/0x670 [ 632.196468][ C1] __dev_change_flags+0x460/0x6c0 [ 632.196474][ C1] netif_change_flags+0x80/0x160 [ 632.196480][ C1] do_setlink.constprop.0+0x984/0x2640 [ 632.196489][ C1] rtnl_newlink+0x693/0xa60 [ 632.196495][ C1] rtnetlink_rcv_msg+0x709/0xc00 [ 632.196502][ C1] netlink_rcv_skb+0x121/0x340 [ 632.196511][ C1] netlink_unicast+0x4aa/0x780 [ 632.196517][ C1] netlink_sendmsg+0x71d/0xbe0 [ 632.196524][ C1] ____sys_sendmsg+0x3dd/0x890 [ 632.196533][ C1] ___sys_sendmsg+0xed/0x170 [ 632.196542][ C1] __sys_sendmsg+0x10b/0x1a0 [ 632.196551][ C1] do_syscall_64+0xc1/0x370 [ 632.196562][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 632.196570][ C1] IN-SOFTIRQ-W at: [ 632.196574][ C1] __lock_acquire+0x20b/0x7e0 [ 632.196581][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.196587][ C1] _raw_spin_lock+0x33/0x40 [ 632.196596][ C1] virtnet_poll_tx+0x1da/0x6f0 [ 632.196602][ C1] __napi_poll.constprop.0+0x9f/0x460 [ 632.196611][ C1] net_rx_action+0x54f/0xda0 [ 632.196620][ C1] handle_softirqs+0x215/0x610 [ 632.196627][ C1] do_softirq+0xad/0xe0 [ 632.196634][ C1] __local_bh_enable_ip+0x101/0x120 [ 632.196641][ C1] virtnet_napi_tx_enable+0x1b1/0x270 [ 632.196653][ C1] virtnet_open+0x27c/0xb40 [ 632.196671][ C1] __dev_open+0x224/0x670 [ 632.196672][ C1] __dev_change_flags+0x460/0x6c0 [ 632.196674][ C1] netif_change_flags+0x80/0x160 [ 632.196676][ C1] do_setlink.constprop.0+0x984/0x2640 [ 632.196679][ C1] rtnl_newlink+0x693/0xa60 [ 632.196681][ C1] rtnetlink_rcv_msg+0x709/0xc00 [ 632.196683][ C1] netlink_rcv_skb+0x121/0x340 [ 632.196685][ C1] netlink_unicast+0x4aa/0x780 [ 632.196688][ C1] netlink_sendmsg+0x71d/0xbe0 [ 632.196690][ C1] ____sys_sendmsg+0x3dd/0x890 [ 632.196692][ C1] ___sys_sendmsg+0xed/0x170 [ 632.196695][ C1] __sys_sendmsg+0x10b/0x1a0 [ 632.196698][ C1] do_syscall_64+0xc1/0x370 [ 632.196700][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 632.196702][ C1] INITIAL USE at: [ 632.196704][ C1] __lock_acquire+0x20b/0x7e0 [ 632.196706][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.196708][ C1] _raw_spin_trylock+0x69/0x80 [ 632.196710][ C1] virtnet_poll_cleantx.isra.0+0x1fe/0x500 [ 632.196713][ C1] virtnet_poll+0xf9/0xad0 [ 632.196715][ C1] __napi_poll.constprop.0+0x9f/0x460 [ 632.196718][ C1] net_rx_action+0x54f/0xda0 [ 632.196720][ C1] handle_softirqs+0x215/0x610 [ 632.196723][ C1] do_softirq+0xad/0xe0 [ 632.196725][ C1] __local_bh_enable_ip+0x101/0x120 [ 632.196727][ C1] virtnet_open+0x228/0xb40 [ 632.196729][ C1] __dev_open+0x224/0x670 [ 632.196731][ C1] __dev_change_flags+0x460/0x6c0 [ 632.196733][ C1] netif_change_flags+0x80/0x160 [ 632.196746][ C1] do_setlink.constprop.0+0x984/0x2640 [ 632.196749][ C1] rtnl_newlink+0x693/0xa60 [ 632.196751][ C1] rtnetlink_rcv_msg+0x709/0xc00 [ 632.196753][ C1] netlink_rcv_skb+0x121/0x340 [ 632.196754][ C1] netlink_unicast+0x4aa/0x780 [ 632.196756][ C1] netlink_sendmsg+0x71d/0xbe0 [ 632.196758][ C1] ____sys_sendmsg+0x3dd/0x890 [ 632.196761][ C1] ___sys_sendmsg+0xed/0x170 [ 632.196763][ C1] __sys_sendmsg+0x10b/0x1a0 [ 632.196766][ C1] do_syscall_64+0xc1/0x370 [ 632.196768][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 632.196770][ C1] } [ 632.196771][ C1] ... key at: [] netdev_xmit_lock_key+0x10/0x3c0 [ 632.196776][ C1] ... acquired at: [ 632.196777][ C1] __lock_acquire+0x449/0x7e0 [ 632.196779][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.196781][ C1] _raw_spin_lock+0x33/0x40 [ 632.196783][ C1] virtnet_poll_tx+0x1da/0x6f0 [ 632.196785][ C1] poll_napi+0xf8/0x250 [ 632.196789][ C1] netpoll_poll_dev+0x264/0x2f0 [ 632.196791][ C1] __netpoll_send_skb+0x6ba/0x8a0 [ 632.196793][ C1] netpoll_send_skb+0x2a/0x90 [ 632.196795][ C1] send_ext_msg_udp+0x422/0x570 [netconsole] [ 632.196802][ C1] write_ext_msg+0x1c6/0x220 [netconsole] [ 632.196806][ C1] console_emit_next_record+0x20a/0x430 [ 632.196808][ C1] console_flush_all+0x3b4/0x620 [ 632.196810][ C1] console_unlock+0xba/0x190 [ 632.196813][ C1] vprintk_emit+0x290/0x2d0 [ 632.196815][ C1] _printk+0xb6/0xe0 [ 632.196817][ C1] enabled_store+0x23b/0x850 [netconsole] [ 632.196821][ C1] configfs_write_iter+0x2c7/0x490 [ 632.196826][ C1] vfs_write+0xaae/0x12c0 [ 632.196832][ C1] ksys_write+0xf7/0x1d0 [ 632.196833][ C1] do_syscall_64+0xc1/0x370 [ 632.196835][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 632.196837][ C1] [ 632.196838][ C1] -> (target_list_lock){....}-{3:3} { [ 632.196841][ C1] INITIAL USE at: [ 632.196842][ C1] __lock_acquire+0x20b/0x7e0 [ 632.196845][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.196847][ C1] _raw_spin_lock_irqsave+0x42/0x60 [ 632.196850][ C1] make_netconsole_target+0x10b/0x220 [netconsole] [ 632.196853][ C1] configfs_mkdir+0x46e/0xef0 [ 632.196856][ C1] vfs_mkdir+0x36e/0x600 [ 632.196860][ C1] do_mkdirat+0x143/0x3b0 [ 632.196863][ C1] __x64_sys_mkdir+0x63/0x80 [ 632.196866][ C1] do_syscall_64+0xc1/0x370 [ 632.196869][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 632.196871][ C1] } [ 632.196871][ C1] ... key at: [] target_list_lock+0x18/0x60 [netconsole] [ 632.196875][ C1] ... acquired at: [ 632.196876][ C1] __lock_acquire+0x449/0x7e0 [ 632.196878][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.196880][ C1] _raw_spin_lock_irqsave+0x42/0x60 [ 632.196883][ C1] write_ext_msg+0x8e/0x220 [netconsole] [ 632.196886][ C1] console_emit_next_record+0x20a/0x430 [ 632.196889][ C1] console_flush_all+0x3b4/0x620 [ 632.196891][ C1] console_unlock+0xba/0x190 [ 632.196893][ C1] vprintk_emit+0x290/0x2d0 [ 632.196896][ C1] _printk+0xb6/0xe0 [ 632.196897][ C1] register_console+0x8dd/0xf30 [ 632.196900][ C1] enabled_store+0x623/0x850 [netconsole] [ 632.196904][ C1] configfs_write_iter+0x2c7/0x490 [ 632.196906][ C1] vfs_write+0xaae/0x12c0 [ 632.196908][ C1] ksys_write+0xf7/0x1d0 [ 632.196910][ C1] do_syscall_64+0xc1/0x370 [ 632.196912][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 632.196914][ C1] [ 632.196914][ C1] -> (console_owner){-...}-{0:0} { [ 632.196917][ C1] IN-HARDIRQ-W at: [ 632.196919][ C1] __lock_acquire+0x20b/0x7e0 [ 632.196921][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.196923][ C1] console_lock_spinning_enable+0x99/0xb0 [ 632.196925][ C1] console_emit_next_record+0x1b2/0x430 [ 632.196928][ C1] console_flush_all+0x3b4/0x620 [ 632.196930][ C1] console_unlock+0xba/0x190 [ 632.196932][ C1] wake_up_klogd_work_func+0x5b/0x80 [ 632.196935][ C1] irq_work_single+0x108/0x210 [ 632.196937][ C1] irq_work_run_list+0x6a/0x90 [ 632.196940][ C1] update_process_times+0x136/0x1a0 [ 632.196945][ C1] tick_nohz_handler+0x319/0x4e0 [ 632.196951][ C1] __run_hrtimer+0x158/0x370 [ 632.196953][ C1] __hrtimer_run_queues+0x194/0x260 [ 632.196956][ C1] hrtimer_interrupt+0x339/0x850 [ 632.196959][ C1] __sysvec_apic_timer_interrupt+0xc6/0x150 [ 632.196966][ C1] sysvec_apic_timer_interrupt+0xa3/0xc0 [ 632.196969][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 632.196971][ C1] pv_native_safe_halt+0xf/0x10 [ 632.196974][ C1] default_idle+0x9/0x10 [ 632.196977][ C1] default_idle_call+0x6d/0xa0 [ 632.196980][ C1] cpuidle_idle_call+0x23c/0x370 [ 632.196982][ C1] do_idle+0xe1/0x140 [ 632.196984][ C1] cpu_startup_entry+0x54/0x60 [ 632.196986][ C1] start_secondary+0x1fd/0x290 [ 632.196988][ C1] common_startup_64+0x13e/0x148 [ 632.196992][ C1] INITIAL USE at: [ 632.196994][ C1] } [ 632.196994][ C1] ... key at: [] console_owner_dep_map+0x0/0x60 [ 632.197000][ C1] ... acquired at: [ 632.197001][ C1] mark_lock+0x16c/0x2f0 [ 632.197003][ C1] mark_usage+0x11d/0x140 [ 632.197005][ C1] __lock_acquire+0x20b/0x7e0 [ 632.197007][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.197008][ C1] console_lock_spinning_enable+0x99/0xb0 [ 632.197011][ C1] console_emit_next_record+0x1b2/0x430 [ 632.197013][ C1] console_flush_all+0x3b4/0x620 [ 632.197016][ C1] console_unlock+0xba/0x190 [ 632.197018][ C1] wake_up_klogd_work_func+0x5b/0x80 [ 632.197020][ C1] irq_work_single+0x108/0x210 [ 632.197023][ C1] irq_work_run_list+0x6a/0x90 [ 632.197025][ C1] update_process_times+0x136/0x1a0 [ 632.197028][ C1] tick_nohz_handler+0x319/0x4e0 [ 632.197030][ C1] __run_hrtimer+0x158/0x370 [ 632.197032][ C1] __hrtimer_run_queues+0x194/0x260 [ 632.197035][ C1] hrtimer_interrupt+0x339/0x850 [ 632.197038][ C1] __sysvec_apic_timer_interrupt+0xc6/0x150 [ 632.197040][ C1] sysvec_apic_timer_interrupt+0xa3/0xc0 [ 632.197043][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 632.197045][ C1] pv_native_safe_halt+0xf/0x10 [ 632.197048][ C1] default_idle+0x9/0x10 [ 632.197050][ C1] default_idle_call+0x6d/0xa0 [ 632.197053][ C1] cpuidle_idle_call+0x23c/0x370 [ 632.197055][ C1] do_idle+0xe1/0x140 [ 632.197057][ C1] cpu_startup_entry+0x54/0x60 [ 632.197058][ C1] start_secondary+0x1fd/0x290 [ 632.197061][ C1] common_startup_64+0x13e/0x148 [ 632.197063][ C1] [ 632.197063][ C1] [ 632.197063][ C1] stack backtrace: [ 632.197067][ C1] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted 6.17.0-rc1-virtme #1 PREEMPT(full) [ 632.197071][ C1] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 632.197073][ C1] Call Trace: [ 632.197075][ C1] [ 632.197078][ C1] dump_stack_lvl+0x82/0xc0 [ 632.197100][ C1] print_irq_inversion_bug.part.0+0x206/0x260 [ 632.197104][ C1] mark_lock_irq+0x248/0x400 [ 632.197110][ C1] mark_lock+0x16c/0x2f0 [ 632.197113][ C1] mark_usage+0x11d/0x140 [ 632.197116][ C1] __lock_acquire+0x20b/0x7e0 [ 632.197120][ C1] lock_acquire.part.0+0xb6/0x240 [ 632.197123][ C1] ? console_lock_spinning_enable+0x7d/0xb0 [ 632.197126][ C1] ? rcu_is_watching+0x12/0xb0 [ 632.197133][ C1] ? lock_acquire+0x104/0x160 [ 632.197135][ C1] ? console_lock_spinning_enable+0x7d/0xb0 [ 632.197139][ C1] console_lock_spinning_enable+0x99/0xb0 [ 632.197142][ C1] ? console_lock_spinning_enable+0x7d/0xb0 [ 632.197145][ C1] console_emit_next_record+0x1b2/0x430 [ 632.197148][ C1] ? devkmsg_read+0x460/0x460 [ 632.197152][ C1] ? rcu_is_watching+0x12/0xb0 [ 632.197157][ C1] console_flush_all+0x3b4/0x620 [ 632.197161][ C1] ? console_emit_next_record+0x430/0x430 [ 632.197164][ C1] ? __lock_acquire+0x449/0x7e0 [ 632.197169][ C1] console_unlock+0xba/0x190 [ 632.197172][ C1] ? console_flush_all+0x620/0x620 [ 632.197175][ C1] ? rcu_is_watching+0x12/0xb0 [ 632.197178][ C1] ? irq_work_single+0x108/0x210 [ 632.197181][ C1] ? irq_work_single+0x108/0x210 [ 632.197185][ C1] wake_up_klogd_work_func+0x5b/0x80 [ 632.197189][ C1] irq_work_single+0x108/0x210 [ 632.197192][ C1] irq_work_run_list+0x6a/0x90 [ 632.197195][ C1] ? irq_work_tick+0x9b/0x140 [ 632.197198][ C1] update_process_times+0x136/0x1a0 [ 632.197202][ C1] tick_nohz_handler+0x319/0x4e0 [ 632.197205][ C1] ? rcu_is_watching+0x12/0xb0 [ 632.197209][ C1] __run_hrtimer+0x158/0x370 [ 632.197212][ C1] ? tick_do_update_jiffies64.part.0+0x280/0x280 [ 632.197216][ C1] __hrtimer_run_queues+0x194/0x260 [ 632.197220][ C1] ? hrtimer_interrupt+0x12a/0x850 [ 632.197224][ C1] ? __run_hrtimer+0x370/0x370 [ 632.197227][ C1] ? kvm_clock_get_cycles+0x18/0x30 [ 632.197234][ C1] hrtimer_interrupt+0x339/0x850 [ 632.197241][ C1] __sysvec_apic_timer_interrupt+0xc6/0x150 [ 632.197244][ C1] sysvec_apic_timer_interrupt+0xa3/0xc0 [ 632.197248][ C1] [ 632.197249][ C1] [ 632.197250][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 632.197253][ C1] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 632.197257][ C1] Code: 48 8b 3d 74 4b 05 02 e8 1f 00 00 00 48 2b 05 c8 f4 85 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d b3 ad 0b 00 fb f4 66 0f 1f 00 41 54 55 53 48 89 fb 48 83 ec 10 8b 17 83 e2 fe 41 [ 632.197261][ C1] RSP: 0018:ffffc90000137de8 EFLAGS: 00000202 [ 632.197264][ C1] RAX: 0000000000f5f5ef RBX: 1ffff92000026fc1 RCX: ffffffff9ebd91c5 [ 632.197266][ C1] RDX: 0000000000000000 RSI: ffffffff9f3ea97b RDI: ffffffff9ee57960 [ 632.197267][ C1] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1006116252 [ 632.197269][ C1] R10: ffff8880308b1293 R11: ffff8880308b6958 R12: 0000000000000000 [ 632.197271][ C1] R13: ffff888001ab0040 R14: dffffc0000000000 R15: 0000000000000000 [ 632.197275][ C1] ? ct_kernel_exit.constprop.0+0x105/0x150 [ 632.197297][ C1] default_idle+0x9/0x10 [ 632.197300][ C1] default_idle_call+0x6d/0xa0 [ 632.197304][ C1] cpuidle_idle_call+0x23c/0x370 [ 632.197307][ C1] ? arch_cpu_idle_exit+0x40/0x40 [ 632.197310][ C1] ? tsc_verify_tsc_adjust+0x94/0x310 [ 632.197314][ C1] do_idle+0xe1/0x140 [ 632.197317][ C1] cpu_startup_entry+0x54/0x60 [ 632.197320][ C1] start_secondary+0x1fd/0x290 [ 632.197323][ C1] ? set_cpu_sibling_map+0x1f70/0x1f70 [ 632.197328][ C1] common_startup_64+0x13e/0x148 [ 632.197335][ C1]