[ 725.032642][ C1] sched: DL replenish lagged too much [ 725.033688][ C1] [ 725.033692][ C1] ======================================================== [ 725.033694][ C1] WARNING: possible irq lock inversion dependency detected [ 725.033697][ C1] 6.17.0-rc3-virtme #1 Not tainted [ 725.033700][ C1] -------------------------------------------------------- [ 725.033702][ C1] swapper/1/0 just changed the state of lock: [ 725.033704][ C1] ffffffffb5e77e80 (console_owner){-...}-{0:0}, at: console_lock_spinning_enable+0x7d/0xb0 [ 725.033723][ C1] but this lock took another, HARDIRQ-unsafe lock in the past: [ 725.033725][ C1] (_xmit_ETHER#2){+.-.}-{3:3} [ 725.033731][ C1] [ 725.033731][ C1] [ 725.033731][ C1] and interrupts could create inverse lock ordering between them. [ 725.033731][ C1] [ 725.033732][ C1] [ 725.033732][ C1] other info that might help us debug this: [ 725.033734][ C1] Chain exists of: [ 725.033734][ C1] console_owner --> target_list_lock --> _xmit_ETHER#2 [ 725.033734][ C1] [ 725.033743][ C1] Possible interrupt unsafe locking scenario: [ 725.033743][ C1] [ 725.033745][ C1] CPU0 CPU1 [ 725.033746][ C1] ---- ---- [ 725.033747][ C1] lock(_xmit_ETHER#2); [ 725.033752][ C1] local_irq_disable(); [ 725.033753][ C1] lock(console_owner); [ 725.033756][ C1] lock(target_list_lock); [ 725.033760][ C1] [ 725.033761][ C1] lock(console_owner); [ 725.033764][ C1] [ 725.033764][ C1] *** DEADLOCK *** [ 725.033764][ C1] [ 725.033766][ C1] 2 locks held by swapper/1/0: [ 725.033768][ C1] #0: ffffffffb5f584e0 (console_lock){+.+.}-{0:0}, at: irq_work_single+0x108/0x210 [ 725.033782][ C1] #1: ffffffffb5f58550 (console_srcu){....}-{0:0}, at: console_flush_all+0x12f/0x620 [ 725.033792][ C1] [ 725.033792][ C1] the shortest dependencies between 2nd lock and 1st lock: [ 725.033803][ C1] -> (_xmit_ETHER#2){+.-.}-{3:3} { [ 725.033810][ C1] HARDIRQ-ON-W at: [ 725.033813][ C1] __lock_acquire+0x20b/0x7e0 [ 725.033819][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.033823][ C1] _raw_spin_trylock+0x69/0x80 [ 725.033831][ C1] virtnet_poll_cleantx.isra.0+0x1fe/0x500 [ 725.033838][ C1] virtnet_poll+0xf9/0xad0 [ 725.033842][ C1] __napi_poll.constprop.0+0x9f/0x460 [ 725.033847][ C1] net_rx_action+0x54f/0xda0 [ 725.033851][ C1] handle_softirqs+0x215/0x610 [ 725.033857][ C1] do_softirq+0xad/0xe0 [ 725.033861][ C1] __local_bh_enable_ip+0x101/0x120 [ 725.033865][ C1] virtnet_open+0x228/0xb40 [ 725.033869][ C1] __dev_open+0x224/0x670 [ 725.033873][ C1] __dev_change_flags+0x460/0x6c0 [ 725.033877][ C1] netif_change_flags+0x80/0x160 [ 725.033881][ C1] do_setlink.constprop.0+0x984/0x2640 [ 725.033886][ C1] rtnl_newlink+0x693/0xa60 [ 725.033890][ C1] rtnetlink_rcv_msg+0x709/0xc00 [ 725.033894][ C1] netlink_rcv_skb+0x121/0x340 [ 725.033900][ C1] netlink_unicast+0x4aa/0x780 [ 725.033904][ C1] netlink_sendmsg+0x71d/0xbe0 [ 725.033908][ C1] ____sys_sendmsg+0x3dd/0x890 [ 725.033914][ C1] ___sys_sendmsg+0xed/0x170 [ 725.033919][ C1] __sys_sendmsg+0x10b/0x1a0 [ 725.033924][ C1] do_syscall_64+0xc1/0x370 [ 725.033930][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 725.033935][ C1] IN-SOFTIRQ-W at: [ 725.033938][ C1] __lock_acquire+0x20b/0x7e0 [ 725.033942][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.033946][ C1] _raw_spin_lock+0x33/0x40 [ 725.033951][ C1] virtnet_poll_tx+0x1da/0x6f0 [ 725.033955][ C1] __napi_poll.constprop.0+0x9f/0x460 [ 725.033959][ C1] net_rx_action+0x54f/0xda0 [ 725.033962][ C1] handle_softirqs+0x215/0x610 [ 725.033967][ C1] do_softirq+0xad/0xe0 [ 725.033970][ C1] __local_bh_enable_ip+0x101/0x120 [ 725.033974][ C1] virtnet_napi_tx_enable+0x1b1/0x270 [ 725.033978][ C1] virtnet_open+0x27c/0xb40 [ 725.033982][ C1] __dev_open+0x224/0x670 [ 725.033986][ C1] __dev_change_flags+0x460/0x6c0 [ 725.033990][ C1] netif_change_flags+0x80/0x160 [ 725.033994][ C1] do_setlink.constprop.0+0x984/0x2640 [ 725.033998][ C1] rtnl_newlink+0x693/0xa60 [ 725.034001][ C1] rtnetlink_rcv_msg+0x709/0xc00 [ 725.034005][ C1] netlink_rcv_skb+0x121/0x340 [ 725.034009][ C1] netlink_unicast+0x4aa/0x780 [ 725.034013][ C1] netlink_sendmsg+0x71d/0xbe0 [ 725.034017][ C1] ____sys_sendmsg+0x3dd/0x890 [ 725.034021][ C1] ___sys_sendmsg+0xed/0x170 [ 725.034026][ C1] __sys_sendmsg+0x10b/0x1a0 [ 725.034031][ C1] do_syscall_64+0xc1/0x370 [ 725.034035][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 725.034039][ C1] INITIAL USE at: [ 725.034042][ C1] __lock_acquire+0x20b/0x7e0 [ 725.034046][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.034050][ C1] _raw_spin_trylock+0x69/0x80 [ 725.034055][ C1] virtnet_poll_cleantx.isra.0+0x1fe/0x500 [ 725.034059][ C1] virtnet_poll+0xf9/0xad0 [ 725.034062][ C1] __napi_poll.constprop.0+0x9f/0x460 [ 725.034065][ C1] net_rx_action+0x54f/0xda0 [ 725.034069][ C1] handle_softirqs+0x215/0x610 [ 725.034073][ C1] do_softirq+0xad/0xe0 [ 725.034077][ C1] __local_bh_enable_ip+0x101/0x120 [ 725.034081][ C1] virtnet_open+0x228/0xb40 [ 725.034085][ C1] __dev_open+0x224/0x670 [ 725.034089][ C1] __dev_change_flags+0x460/0x6c0 [ 725.034092][ C1] netif_change_flags+0x80/0x160 [ 725.034095][ C1] do_setlink.constprop.0+0x984/0x2640 [ 725.034100][ C1] rtnl_newlink+0x693/0xa60 [ 725.034104][ C1] rtnetlink_rcv_msg+0x709/0xc00 [ 725.034108][ C1] netlink_rcv_skb+0x121/0x340 [ 725.034112][ C1] netlink_unicast+0x4aa/0x780 [ 725.034116][ C1] netlink_sendmsg+0x71d/0xbe0 [ 725.034120][ C1] ____sys_sendmsg+0x3dd/0x890 [ 725.034124][ C1] ___sys_sendmsg+0xed/0x170 [ 725.034129][ C1] __sys_sendmsg+0x10b/0x1a0 [ 725.034134][ C1] do_syscall_64+0xc1/0x370 [ 725.034138][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 725.034142][ C1] } [ 725.034143][ C1] ... key at: [] netdev_xmit_lock_key+0x10/0x3c0 [ 725.034151][ C1] ... acquired at: [ 725.034152][ C1] __lock_acquire+0x449/0x7e0 [ 725.034156][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.034160][ C1] _raw_spin_lock+0x33/0x40 [ 725.034164][ C1] virtnet_poll_tx+0x1da/0x6f0 [ 725.034168][ C1] poll_napi+0xf8/0x250 [ 725.034173][ C1] netpoll_poll_dev+0x264/0x2f0 [ 725.034177][ C1] __netpoll_send_skb+0x6ba/0x8a0 [ 725.034180][ C1] netpoll_send_skb+0x2a/0x90 [ 725.034184][ C1] send_ext_msg_udp+0x422/0x570 [netconsole] [ 725.034195][ C1] write_ext_msg+0x1c6/0x220 [netconsole] [ 725.034203][ C1] console_emit_next_record+0x20a/0x430 [ 725.034207][ C1] console_flush_all+0x3b4/0x620 [ 725.034212][ C1] console_unlock+0xba/0x190 [ 725.034216][ C1] vprintk_emit+0x290/0x2d0 [ 725.034220][ C1] devkmsg_emit.constprop.0+0xab/0xe0 [ 725.034225][ C1] devkmsg_write+0x193/0x300 [ 725.034229][ C1] vfs_write+0xaae/0x12c0 [ 725.034234][ C1] ksys_write+0xf7/0x1d0 [ 725.034237][ C1] do_syscall_64+0xc1/0x370 [ 725.034240][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 725.034244][ C1] [ 725.034245][ C1] -> (target_list_lock){....}-{3:3} { [ 725.034252][ C1] INITIAL USE at: [ 725.034255][ C1] __lock_acquire+0x20b/0x7e0 [ 725.034259][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.034263][ C1] _raw_spin_lock_irqsave+0x42/0x60 [ 725.034267][ C1] netconsole_netdev_event+0x78/0x63e [netconsole] [ 725.034274][ C1] notifier_call_chain+0xcc/0x150 [ 725.034281][ C1] netif_change_name+0x45a/0x920 [ 725.034285][ C1] do_setlink.constprop.0+0x1ca6/0x2640 [ 725.034289][ C1] rtnl_setlink+0x2ad/0x590 [ 725.034293][ C1] rtnetlink_rcv_msg+0x709/0xc00 [ 725.034297][ C1] netlink_rcv_skb+0x121/0x340 [ 725.034301][ C1] netlink_unicast+0x4aa/0x780 [ 725.034305][ C1] netlink_sendmsg+0x71d/0xbe0 [ 725.034309][ C1] __sys_sendto+0x24b/0x380 [ 725.034314][ C1] __x64_sys_sendto+0xe0/0x1b0 [ 725.034319][ C1] do_syscall_64+0xc1/0x370 [ 725.034323][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 725.034326][ C1] } [ 725.034328][ C1] ... key at: [] target_list_lock+0x18/0x60 [netconsole] [ 725.034336][ C1] ... acquired at: [ 725.034338][ C1] __lock_acquire+0x449/0x7e0 [ 725.034342][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.034345][ C1] _raw_spin_lock_irqsave+0x42/0x60 [ 725.034350][ C1] write_ext_msg+0x8e/0x220 [netconsole] [ 725.034357][ C1] console_emit_next_record+0x20a/0x430 [ 725.034361][ C1] console_flush_all+0x3b4/0x620 [ 725.034365][ C1] console_unlock+0xba/0x190 [ 725.034370][ C1] vprintk_emit+0x290/0x2d0 [ 725.034374][ C1] devkmsg_emit.constprop.0+0xab/0xe0 [ 725.034378][ C1] devkmsg_write+0x193/0x300 [ 725.034382][ C1] vfs_write+0xaae/0x12c0 [ 725.034386][ C1] ksys_write+0xf7/0x1d0 [ 725.034390][ C1] do_syscall_64+0xc1/0x370 [ 725.034394][ C1] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 725.034398][ C1] [ 725.034399][ C1] -> (console_owner){-...}-{0:0} { [ 725.034405][ C1] IN-HARDIRQ-W at: [ 725.034408][ C1] __lock_acquire+0x20b/0x7e0 [ 725.034412][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.034415][ C1] console_lock_spinning_enable+0x99/0xb0 [ 725.034420][ C1] console_emit_next_record+0x1b2/0x430 [ 725.034424][ C1] console_flush_all+0x3b4/0x620 [ 725.034429][ C1] console_unlock+0xba/0x190 [ 725.034433][ C1] wake_up_klogd_work_func+0x5b/0x80 [ 725.034437][ C1] irq_work_single+0x108/0x210 [ 725.034442][ C1] irq_work_run_list+0x6a/0x90 [ 725.034446][ C1] update_process_times+0x136/0x1a0 [ 725.034452][ C1] tick_nohz_handler+0x319/0x4e0 [ 725.034458][ C1] __run_hrtimer+0x158/0x370 [ 725.034463][ C1] __hrtimer_run_queues+0x194/0x260 [ 725.034467][ C1] hrtimer_interrupt+0x339/0x850 [ 725.034472][ C1] __sysvec_apic_timer_interrupt+0xc6/0x150 [ 725.034479][ C1] sysvec_apic_timer_interrupt+0xa3/0xc0 [ 725.034484][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 725.034489][ C1] pv_native_safe_halt+0xf/0x10 [ 725.034494][ C1] default_idle+0x9/0x10 [ 725.034498][ C1] default_idle_call+0x6d/0xa0 [ 725.034502][ C1] cpuidle_idle_call+0x23c/0x370 [ 725.034506][ C1] do_idle+0xe1/0x140 [ 725.034510][ C1] cpu_startup_entry+0x54/0x60 [ 725.034514][ C1] start_secondary+0x1fd/0x290 [ 725.034518][ C1] common_startup_64+0x13e/0x148 [ 725.034523][ C1] INITIAL USE at: [ 725.034526][ C1] } [ 725.034528][ C1] ... key at: [] console_owner_dep_map+0x0/0x60 [ 725.034534][ C1] ... acquired at: [ 725.034536][ C1] mark_lock+0x16c/0x2f0 [ 725.034539][ C1] mark_usage+0x11d/0x140 [ 725.034543][ C1] __lock_acquire+0x20b/0x7e0 [ 725.034547][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.034551][ C1] console_lock_spinning_enable+0x99/0xb0 [ 725.034556][ C1] console_emit_next_record+0x1b2/0x430 [ 725.034560][ C1] console_flush_all+0x3b4/0x620 [ 725.034564][ C1] console_unlock+0xba/0x190 [ 725.034569][ C1] wake_up_klogd_work_func+0x5b/0x80 [ 725.034573][ C1] irq_work_single+0x108/0x210 [ 725.034578][ C1] irq_work_run_list+0x6a/0x90 [ 725.034582][ C1] update_process_times+0x136/0x1a0 [ 725.034586][ C1] tick_nohz_handler+0x319/0x4e0 [ 725.034591][ C1] __run_hrtimer+0x158/0x370 [ 725.034595][ C1] __hrtimer_run_queues+0x194/0x260 [ 725.034605][ C1] hrtimer_interrupt+0x339/0x850 [ 725.034609][ C1] __sysvec_apic_timer_interrupt+0xc6/0x150 [ 725.034614][ C1] sysvec_apic_timer_interrupt+0xa3/0xc0 [ 725.034618][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 725.034621][ C1] pv_native_safe_halt+0xf/0x10 [ 725.034625][ C1] default_idle+0x9/0x10 [ 725.034628][ C1] default_idle_call+0x6d/0xa0 [ 725.034631][ C1] cpuidle_idle_call+0x23c/0x370 [ 725.034633][ C1] do_idle+0xe1/0x140 [ 725.034636][ C1] cpu_startup_entry+0x54/0x60 [ 725.034640][ C1] start_secondary+0x1fd/0x290 [ 725.034644][ C1] common_startup_64+0x13e/0x148 [ 725.034648][ C1] [ 725.034649][ C1] [ 725.034649][ C1] stack backtrace: [ 725.034653][ C1] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted 6.17.0-rc3-virtme #1 PREEMPT(full) [ 725.034659][ C1] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 725.034663][ C1] Call Trace: [ 725.034666][ C1] [ 725.034669][ C1] dump_stack_lvl+0x82/0xc0 [ 725.034677][ C1] print_irq_inversion_bug.part.0+0x206/0x260 [ 725.034685][ C1] mark_lock_irq+0x248/0x400 [ 725.034697][ C1] mark_lock+0x16c/0x2f0 [ 725.034704][ C1] mark_usage+0x11d/0x140 [ 725.034709][ C1] __lock_acquire+0x20b/0x7e0 [ 725.034718][ C1] lock_acquire.part.0+0xb6/0x240 [ 725.034723][ C1] ? console_lock_spinning_enable+0x7d/0xb0 [ 725.034729][ C1] ? rcu_is_watching+0x12/0xb0 [ 725.034736][ C1] ? lock_acquire+0x104/0x160 [ 725.034741][ C1] ? console_lock_spinning_enable+0x7d/0xb0 [ 725.034748][ C1] console_lock_spinning_enable+0x99/0xb0 [ 725.034753][ C1] ? console_lock_spinning_enable+0x7d/0xb0 [ 725.034758][ C1] console_emit_next_record+0x1b2/0x430 [ 725.034765][ C1] ? devkmsg_read+0x460/0x460 [ 725.034772][ C1] ? rcu_is_watching+0x12/0xb0 [ 725.034781][ C1] console_flush_all+0x3b4/0x620 [ 725.034789][ C1] ? console_emit_next_record+0x430/0x430 [ 725.034795][ C1] ? __lock_acquire+0x449/0x7e0 [ 725.034804][ C1] console_unlock+0xba/0x190 [ 725.034809][ C1] ? console_flush_all+0x620/0x620 [ 725.034815][ C1] ? rcu_is_watching+0x12/0xb0 [ 725.034820][ C1] ? irq_work_single+0x108/0x210 [ 725.034826][ C1] ? irq_work_single+0x108/0x210 [ 725.034834][ C1] wake_up_klogd_work_func+0x5b/0x80 [ 725.034840][ C1] irq_work_single+0x108/0x210 [ 725.034846][ C1] irq_work_run_list+0x6a/0x90 [ 725.034851][ C1] ? irq_work_tick+0x9b/0x140 [ 725.034857][ C1] update_process_times+0x136/0x1a0 [ 725.034863][ C1] tick_nohz_handler+0x319/0x4e0 [ 725.034868][ C1] ? rcu_is_watching+0x12/0xb0 [ 725.034875][ C1] __run_hrtimer+0x158/0x370 [ 725.034881][ C1] ? tick_do_update_jiffies64.part.0+0x280/0x280 [ 725.034889][ C1] __hrtimer_run_queues+0x194/0x260 [ 725.034896][ C1] ? hrtimer_interrupt+0x12a/0x850 [ 725.034902][ C1] ? __run_hrtimer+0x370/0x370 [ 725.034907][ C1] ? kvm_clock_get_cycles+0x18/0x30 [ 725.034918][ C1] hrtimer_interrupt+0x339/0x850 [ 725.034925][ C1] ? __common_interrupt+0xa4/0x230 [ 725.034936][ C1] __sysvec_apic_timer_interrupt+0xc6/0x150 [ 725.034942][ C1] sysvec_apic_timer_interrupt+0xa3/0xc0 [ 725.034948][ C1] [ 725.034950][ C1] [ 725.034953][ C1] asm_sysvec_apic_timer_interrupt+0x1a/0x20 [ 725.034958][ C1] RIP: 0010:pv_native_safe_halt+0xf/0x10 [ 725.034965][ C1] Code: 48 8b 3d 74 7b 05 02 e8 1f 00 00 00 48 2b 05 c8 24 86 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa eb 07 0f 00 2d b3 cd 0b 00 fb f4 66 0f 1f 00 41 54 55 53 48 89 fb 48 83 ec 10 8b 17 83 e2 fe 41 [ 725.034969][ C1] RSP: 0018:ffffc90000137de8 EFLAGS: 00000206 [ 725.034974][ C1] RAX: 000000000110923f RBX: 1ffff92000026fc1 RCX: ffffffffb51d71c5 [ 725.034978][ C1] RDX: 0000000000000000 RSI: ffffffffb59eb6fc RDI: ffffffffb5457ae0 [ 725.034981][ C1] RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed1006c56252 [ 725.034984][ C1] R10: ffff8880362b1293 R11: ffff8880362b6958 R12: 0000000000000000 [ 725.034988][ C1] R13: ffff888001ab0040 R14: dffffc0000000000 R15: 0000000000000000 [ 725.034994][ C1] ? ct_kernel_exit.constprop.0+0x105/0x150 [ 725.035004][ C1] default_idle+0x9/0x10 [ 725.035008][ C1] default_idle_call+0x6d/0xa0 [ 725.035012][ C1] cpuidle_idle_call+0x23c/0x370 [ 725.035018][ C1] ? arch_cpu_idle_exit+0x40/0x40 [ 725.035024][ C1] ? tsc_verify_tsc_adjust+0x94/0x310 [ 725.035032][ C1] do_idle+0xe1/0x140 [ 725.035038][ C1] cpu_startup_entry+0x54/0x60 [ 725.035043][ C1] start_secondary+0x1fd/0x290 [ 725.035049][ C1] ? set_cpu_sibling_map+0x1f70/0x1f70 [ 725.035057][ C1] common_startup_64+0x13e/0x148 [ 725.035070][ C1]