[ 38.084944][ T353] GACT probability NOT on [ 39.407540][ C0] [ 39.407751][ C0] ======================================================== [ 39.408002][ C0] WARNING: possible irq lock inversion dependency detected [ 39.408237][ C0] 6.9.0-rc1-virtme #1 Not tainted [ 39.408408][ C0] -------------------------------------------------------- [ 39.408641][ C0] mausezahn/366 just changed the state of lock: [ 39.408833][ C0] ffff888003f42118 (k-slock-AF_INET){+.-.}-{2:2}, at: ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.409130][ C0] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 39.409361][ C0] (&p->alloc_lock){+.+.}-{2:2} [ 39.409365][ C0] [ 39.409365][ C0] [ 39.409365][ C0] and interrupts could create inverse lock ordering between them. [ 39.409365][ C0] [ 39.409941][ C0] [ 39.409941][ C0] other info that might help us debug this: [ 39.410171][ C0] Possible interrupt unsafe locking scenario: [ 39.410171][ C0] [ 39.410410][ C0] CPU0 CPU1 [ 39.410570][ C0] ---- ---- [ 39.410727][ C0] lock(&p->alloc_lock); [ 39.410851][ C0] local_irq_disable(); [ 39.411054][ C0] lock(k-slock-AF_INET); [ 39.411250][ C0] lock(&p->alloc_lock); [ 39.411454][ C0] [ 39.411574][ C0] lock(k-slock-AF_INET); [ 39.411733][ C0] [ 39.411733][ C0] *** DEADLOCK *** [ 39.411733][ C0] [ 39.411965][ C0] 2 locks held by mausezahn/366: [ 39.412128][ C0] #0: ffffffffa57522e0 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1f8/0x670 [ 39.412411][ C0] #1: ffffffffa57522e0 (rcu_read_lock){....}-{1:2}, at: ip_local_deliver_finish+0x209/0x4b0 [ 39.412748][ C0] [ 39.412748][ C0] the shortest dependencies between 2nd lock and 1st lock: [ 39.413024][ C0] -> (&p->alloc_lock){+.+.}-{2:2} { [ 39.413186][ C0] HARDIRQ-ON-W at: [ 39.413309][ C0] __lock_acquire+0x797/0x1570 [ 39.413518][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.413715][ C0] _raw_spin_lock+0x30/0x40 [ 39.413926][ C0] set_mems_allowed+0x1d/0x210 [ 39.414128][ C0] kernel_init_freeable+0x72/0x310 [ 39.414332][ C0] kernel_init+0x20/0x200 [ 39.414534][ C0] ret_from_fork+0x31/0x70 [ 39.414741][ C0] ret_from_fork_asm+0x1a/0x30 [ 39.414950][ C0] SOFTIRQ-ON-W at: [ 39.415074][ C0] __lock_acquire+0x797/0x1570 [ 39.415272][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.415471][ C0] _raw_spin_lock+0x30/0x40 [ 39.415673][ C0] set_mems_allowed+0x1d/0x210 [ 39.415868][ C0] kernel_init_freeable+0x72/0x310 [ 39.416063][ C0] kernel_init+0x20/0x200 [ 39.416269][ C0] ret_from_fork+0x31/0x70 [ 39.416473][ C0] ret_from_fork_asm+0x1a/0x30 [ 39.416671][ C0] INITIAL USE at: [ 39.416792][ C0] __lock_acquire+0x797/0x1570 [ 39.416995][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.417195][ C0] _raw_spin_lock+0x30/0x40 [ 39.417392][ C0] set_mems_allowed+0x1d/0x210 [ 39.417598][ C0] kernel_init_freeable+0x72/0x310 [ 39.417797][ C0] kernel_init+0x20/0x200 [ 39.418001][ C0] ret_from_fork+0x31/0x70 [ 39.418200][ C0] ret_from_fork_asm+0x1a/0x30 [ 39.418403][ C0] } [ 39.418486][ C0] ... key at: [] __key.387+0x0/0x40 [ 39.418734][ C0] ... acquired at: [ 39.418854][ C0] __lock_acquire+0xaf0/0x1570 [ 39.419019][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.419182][ C0] _raw_spin_lock+0x30/0x40 [ 39.419342][ C0] __get_task_comm+0x27/0x70 [ 39.419505][ C0] ref_tracker_alloc+0x2ee/0x490 [ 39.419671][ C0] dst_init+0x84/0x4a0 [ 39.419793][ C0] dst_alloc+0x97/0x150 [ 39.419915][ C0] rt_dst_alloc+0x31/0x3b0 [ 39.420080][ C0] __mkroute_output+0x451/0x1070 [ 39.420237][ C0] ip_route_output_key_hash+0x15d/0x2b0 [ 39.420402][ C0] icmp_route_lookup.constprop.0+0x321/0xb30 [ 39.420602][ C0] __icmp_send+0x8e8/0x1910 [ 39.420758][ C0] ip_fragment.constprop.0+0x183/0x240 [ 39.420920][ C0] ip_output+0x16b/0x4f0 [ 39.421083][ C0] iptunnel_xmit+0x578/0x930 [ 39.421248][ C0] vxlan_xmit_one+0x2a6d/0x4570 [vxlan] [ 39.421441][ C0] vxlan_xmit+0x855/0xf40 [vxlan] [ 39.421613][ C0] dev_hard_start_xmit+0x10e/0x360 [ 39.421776][ C0] __dev_queue_xmit+0xe69/0x16d0 [ 39.421937][ C0] packet_snd+0xd08/0x1840 [ 39.422104][ C0] __sys_sendto+0x37d/0x410 [ 39.422269][ C0] __x64_sys_sendto+0xe0/0x1c0 [ 39.422431][ C0] do_syscall_64+0xc6/0x1e0 [ 39.422598][ C0] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.422806][ C0] [ 39.422888][ C0] -> (k-slock-AF_INET){+.-.}-{2:2} { [ 39.423047][ C0] HARDIRQ-ON-W at: [ 39.423172][ C0] __lock_acquire+0x797/0x1570 [ 39.423375][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.423575][ C0] _raw_spin_lock_bh+0x38/0x50 [ 39.423775][ C0] lock_sock_nested+0x59/0xe0 [ 39.423977][ C0] sock_bindtoindex+0x25/0x50 [ 39.424175][ C0] udp_sock_create4+0x104/0x3f0 [udp_tunnel] [ 39.424417][ C0] vxlan_create_sock+0xbc/0x1d0 [vxlan] [ 39.424670][ C0] vxlan_socket_create+0xfb/0x500 [vxlan] [ 39.424933][ C0] __vxlan_sock_add+0x388/0x8d0 [vxlan] [ 39.425183][ C0] vxlan_open+0x151/0x250 [vxlan] [ 39.425398][ C0] __dev_open+0x221/0x430 [ 39.425599][ C0] __dev_change_flags+0x469/0x6c0 [ 39.425798][ C0] dev_change_flags+0x80/0x160 [ 39.426000][ C0] do_setlink+0x7d6/0x21e0 [ 39.426197][ C0] __rtnl_newlink+0xaa7/0xd80 [ 39.426393][ C0] rtnl_newlink+0x63/0xa0 [ 39.426590][ C0] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 39.426795][ C0] netlink_rcv_skb+0x130/0x360 [ 39.426994][ C0] netlink_unicast+0x449/0x710 [ 39.427199][ C0] netlink_sendmsg+0x723/0xbe0 [ 39.427401][ C0] ____sys_sendmsg+0x7b2/0xa10 [ 39.427598][ C0] ___sys_sendmsg+0xee/0x170 [ 39.427795][ C0] __sys_sendmsg+0xcd/0x170 [ 39.427995][ C0] do_syscall_64+0xc6/0x1e0 [ 39.428199][ C0] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.428434][ C0] IN-SOFTIRQ-W at: [ 39.428552][ C0] __lock_acquire+0x797/0x1570 [ 39.428755][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.428960][ C0] _raw_spin_lock+0x30/0x40 [ 39.429159][ C0] ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.429366][ C0] __udp4_lib_err+0x7f6/0xa50 [ 39.429567][ C0] icmp_unreach+0x2c1/0xba0 [ 39.429767][ C0] icmp_rcv+0x791/0xec0 [ 39.429972][ C0] ip_protocol_deliver_rcu+0x2e5/0x360 [ 39.430262][ C0] ip_local_deliver_finish+0x2c2/0x4b0 [ 39.430455][ C0] ip_local_deliver+0x19d/0x480 [ 39.430658][ C0] ip_rcv+0x564/0x740 [ 39.430821][ C0] __netif_receive_skb_one_core+0x166/0x1b0 [ 39.431162][ C0] process_backlog+0x103/0x670 [ 39.431361][ C0] __napi_poll.constprop.0+0xa2/0x460 [ 39.431561][ C0] net_rx_action+0x49f/0xc00 [ 39.431848][ C0] __do_softirq+0x1f8/0x5df [ 39.432045][ C0] do_softirq+0x4d/0xa0 [ 39.432255][ C0] __local_bh_enable_ip+0xf6/0x120 [ 39.432537][ C0] __dev_queue_xmit+0x6ac/0x16d0 [ 39.432736][ C0] packet_snd+0xd08/0x1840 [ 39.432935][ C0] __sys_sendto+0x37d/0x410 [ 39.433137][ C0] __x64_sys_sendto+0xe0/0x1c0 [ 39.433423][ C0] do_syscall_64+0xc6/0x1e0 [ 39.433626][ C0] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.433866][ C0] INITIAL USE at: [ 39.434073][ C0] __lock_acquire+0x797/0x1570 [ 39.434272][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.434475][ C0] _raw_spin_lock_bh+0x38/0x50 [ 39.434674][ C0] lock_sock_nested+0x59/0xe0 [ 39.435053][ C0] sock_bindtoindex+0x25/0x50 [ 39.435255][ C0] udp_sock_create4+0x104/0x3f0 [udp_tunnel] [ 39.435495][ C0] vxlan_create_sock+0xbc/0x1d0 [vxlan] [ 39.435793][ C0] vxlan_socket_create+0xfb/0x500 [vxlan] [ 39.436041][ C0] __vxlan_sock_add+0x388/0x8d0 [vxlan] [ 39.436251][ C0] vxlan_open+0x151/0x250 [vxlan] [ 39.436546][ C0] __dev_open+0x221/0x430 [ 39.436749][ C0] __dev_change_flags+0x469/0x6c0 [ 39.436947][ C0] dev_change_flags+0x80/0x160 [ 39.437229][ C0] do_setlink+0x7d6/0x21e0 [ 39.437426][ C0] __rtnl_newlink+0xaa7/0xd80 [ 39.437619][ C0] rtnl_newlink+0x63/0xa0 [ 39.437819][ C0] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 39.438112][ C0] netlink_rcv_skb+0x130/0x360 [ 39.438311][ C0] netlink_unicast+0x449/0x710 [ 39.438510][ C0] netlink_sendmsg+0x723/0xbe0 [ 39.438798][ C0] ____sys_sendmsg+0x7b2/0xa10 [ 39.438995][ C0] ___sys_sendmsg+0xee/0x170 [ 39.439194][ C0] __sys_sendmsg+0xcd/0x170 [ 39.439394][ C0] do_syscall_64+0xc6/0x1e0 [ 39.439685][ C0] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.439922][ C0] } [ 39.440002][ C0] ... key at: [] af_family_kern_slock_keys+0x20/0x300 [ 39.440367][ C0] ... acquired at: [ 39.440488][ C0] mark_lock+0x28d/0x3e0 [ 39.440648][ C0] mark_usage+0xd9/0x2a0 [ 39.440806][ C0] __lock_acquire+0x797/0x1570 [ 39.441051][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.441217][ C0] _raw_spin_lock+0x30/0x40 [ 39.441376][ C0] ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.441537][ C0] __udp4_lib_err+0x7f6/0xa50 [ 39.441695][ C0] icmp_unreach+0x2c1/0xba0 [ 39.441939][ C0] icmp_rcv+0x791/0xec0 [ 39.442059][ C0] ip_protocol_deliver_rcu+0x2e5/0x360 [ 39.442216][ C0] ip_local_deliver_finish+0x2c2/0x4b0 [ 39.442373][ C0] ip_local_deliver+0x19d/0x480 [ 39.442639][ C0] ip_rcv+0x564/0x740 [ 39.442765][ C0] __netif_receive_skb_one_core+0x166/0x1b0 [ 39.442964][ C0] process_backlog+0x103/0x670 [ 39.443126][ C0] __napi_poll.constprop.0+0xa2/0x460 [ 39.443372][ C0] net_rx_action+0x49f/0xc00 [ 39.443531][ C0] __do_softirq+0x1f8/0x5df [ 39.443688][ C0] do_softirq+0x4d/0xa0 [ 39.443809][ C0] __local_bh_enable_ip+0xf6/0x120 [ 39.443966][ C0] __dev_queue_xmit+0x6ac/0x16d0 [ 39.444214][ C0] packet_snd+0xd08/0x1840 [ 39.444376][ C0] __sys_sendto+0x37d/0x410 [ 39.444539][ C0] __x64_sys_sendto+0xe0/0x1c0 [ 39.444696][ C0] do_syscall_64+0xc6/0x1e0 [ 39.444948][ C0] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.445145][ C0] [ 39.445223][ C0] [ 39.445223][ C0] stack backtrace: [ 39.445422][ C0] CPU: 0 PID: 366 Comm: mausezahn Not tainted 6.9.0-rc1-virtme #1 [ 39.445749][ C0] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 39.446100][ C0] Call Trace: [ 39.446220][ C0] [ 39.446306][ C0] dump_stack_lvl+0x82/0xd0 [ 39.446662][ C0] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 39.446883][ C0] ? kernel_text_address+0xce/0xe0 [ 39.447046][ C0] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 39.447417][ C0] ? __pfx_usage_skip+0x10/0x10 [ 39.447574][ C0] ? __pfx_usage_match+0x10/0x10 [ 39.447729][ C0] ? arch_stack_walk+0xa2/0xf0 [ 39.447888][ C0] mark_lock_irq+0x4cd/0xa10 [ 39.448048][ C0] ? __pfx_mark_lock_irq+0x10/0x10 [ 39.448294][ C0] ? __pfx_stack_trace_save+0x10/0x10 [ 39.448454][ C0] ? find_held_lock+0x2c/0x110 [ 39.448613][ C0] ? save_trace+0x8f/0x5b0 [ 39.448778][ C0] mark_lock+0x28d/0x3e0 [ 39.448987][ C0] mark_usage+0xd9/0x2a0 [ 39.449107][ C0] __lock_acquire+0x797/0x1570 [ 39.449266][ C0] lock_acquire.part.0+0xeb/0x330 [ 39.449423][ C0] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.449582][ C0] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 39.449824][ C0] ? trace_lock_acquire+0x135/0x1c0 [ 39.449981][ C0] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.450140][ C0] ? lock_acquire+0x32/0xc0 [ 39.450300][ C0] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.450546][ C0] _raw_spin_lock+0x30/0x40 [ 39.450703][ C0] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.450858][ C0] ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.451019][ C0] ? vxlan_err_lookup+0x23f/0x420 [vxlan] [ 39.451279][ C0] ? __pfx_ipv4_sk_update_pmtu+0x10/0x10 [ 39.451436][ C0] ? __udp4_lib_err_encap+0x59c/0x930 [ 39.451601][ C0] __udp4_lib_err+0x7f6/0xa50 [ 39.451763][ C0] icmp_unreach+0x2c1/0xba0 [ 39.452008][ C0] icmp_rcv+0x791/0xec0 [ 39.452129][ C0] ip_protocol_deliver_rcu+0x2e5/0x360 [ 39.452289][ C0] ip_local_deliver_finish+0x2c2/0x4b0 [ 39.452448][ C0] ip_local_deliver+0x19d/0x480 [ 39.452605][ C0] ? __pfx_ip_local_deliver+0x10/0x10 [ 39.452850][ C0] ? ip_rcv_finish_core.constprop.0+0x1ad/0x1300 [ 39.453053][ C0] ip_rcv+0x564/0x740 [ 39.453175][ C0] ? __pfx_ip_rcv+0x10/0x10 [ 39.453335][ C0] ? lock_acquire.part.0+0xeb/0x330 [ 39.453580][ C0] ? process_backlog+0x1f8/0x670 [ 39.453740][ C0] ? __pfx_ip_rcv+0x10/0x10 [ 39.453899][ C0] __netif_receive_skb_one_core+0x166/0x1b0 [ 39.454095][ C0] ? __pfx___netif_receive_skb_one_core+0x10/0x10 [ 39.454383][ C0] ? process_backlog+0x1f8/0x670 [ 39.454546][ C0] ? lock_acquire+0x32/0xc0 [ 39.454706][ C0] ? process_backlog+0x1f8/0x670 [ 39.454877][ C0] process_backlog+0x103/0x670 [ 39.455123][ C0] ? find_held_lock+0x2c/0x110 [ 39.455280][ C0] __napi_poll.constprop.0+0xa2/0x460 [ 39.455442][ C0] net_rx_action+0x49f/0xc00 [ 39.455602][ C0] ? __pfx_net_rx_action+0x10/0x10 [ 39.455845][ C0] ? ktime_get+0xb3/0x190 [ 39.455963][ C0] ? hrtimer_interrupt+0x311/0x770 [ 39.456130][ C0] ? hrtimer_interrupt+0x31d/0x770 [ 39.456289][ C0] __do_softirq+0x1f8/0x5df [ 39.456454][ C0] ? __dev_queue_xmit+0x697/0x16d0 [ 39.456699][ C0] do_softirq+0x4d/0xa0 [ 39.456819][ C0] [ 39.456901][ C0] [ 39.456981][ C0] __local_bh_enable_ip+0xf6/0x120 [ 39.457138][ C0] ? __dev_queue_xmit+0x697/0x16d0 [ 39.457390][ C0] __dev_queue_xmit+0x6ac/0x16d0 [ 39.457550][ C0] ? packet_parse_headers+0x231/0x9d0 [ 39.457708][ C0] ? packet_parse_headers+0x231/0x9d0 [ 39.457865][ C0] ? __pfx_sock_alloc_send_pskb+0x10/0x10 [ 39.458027][ C0] ? __pfx___dev_queue_xmit+0x10/0x10 [ 39.458276][ C0] ? dev_get_by_index+0x177/0x270 [ 39.458435][ C0] packet_snd+0xd08/0x1840 [ 39.458596][ C0] ? __pfx___lock_release+0x10/0x10 [ 39.458759][ C0] ? __pfx_packet_snd+0x10/0x10 [ 39.459007][ C0] ? __might_fault+0xc3/0x170 [ 39.459169][ C0] ? trace_raw_output_mmap_lock_acquire_returned+0x60/0x1e0 [ 39.459403][ C0] ? __might_fault+0x11b/0x170 [ 39.459564][ C0] __sys_sendto+0x37d/0x410 [ 39.459897][ C0] ? __pfx___sys_sendto+0x10/0x10 [ 39.460057][ C0] ? sock_ioctl+0x3bc/0x590 [ 39.460213][ C0] ? __up_read+0x1a7/0x780 [ 39.460376][ C0] ? __pfx___up_read+0x10/0x10 [ 39.460625][ C0] ? do_user_addr_fault+0x4b8/0xbe0 [ 39.460785][ C0] __x64_sys_sendto+0xe0/0x1c0 [ 39.460944][ C0] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 39.461144][ C0] do_syscall_64+0xc6/0x1e0 [ 39.461390][ C0] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.461587][ C0] RIP: 0033:0x7f049191e85a [ 39.461752][ C0] Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89 [ 39.462407][ C0] RSP: 002b:00007ffce7a50a38 EFLAGS: 00000246 ORIG_RAX: 000000000000002c [ 39.462652][ C0] RAX: ffffffffffffffda RBX: 00000000404249d0 RCX: 00007f049191e85a [ 39.462981][ C0] RDX: 00000000000003c5 RSI: 0000000040425022 RDI: 0000000000000005 [ 39.463225][ C0] RBP: 0000000040425022 R08: 00007ffce7a50a40 R09: 0000000000000014 [ 39.463462][ C0] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 39.463788][ C0] R13: 00000000000003c5 R14: 00007ffce7a50a40 R15: 0000000000000000 [ 39.464030][ C0]