[ 39.076421][ T353] GACT probability NOT on [ 40.414461][ C3] [ 40.414598][ C3] ======================================================== [ 40.414828][ C3] WARNING: possible irq lock inversion dependency detected [ 40.415052][ C3] 6.9.0-rc1-virtme #1 Not tainted [ 40.415209][ C3] -------------------------------------------------------- [ 40.415535][ C3] mausezahn/368 just changed the state of lock: [ 40.415719][ C3] ffff888003ecb098 (k-slock-AF_INET){+.-.}-{2:2}, at: ipv4_sk_update_pmtu+0xb3/0xb60 [ 40.415998][ C3] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 40.416218][ C3] (&p->alloc_lock){+.+.}-{2:2} [ 40.416222][ C3] [ 40.416222][ C3] [ 40.416222][ C3] and interrupts could create inverse lock ordering between them. [ 40.416222][ C3] [ 40.416777][ C3] [ 40.416777][ C3] other info that might help us debug this: [ 40.417023][ C3] Possible interrupt unsafe locking scenario: [ 40.417023][ C3] [ 40.417266][ C3] CPU0 CPU1 [ 40.417415][ C3] ---- ---- [ 40.417565][ C3] lock(&p->alloc_lock); [ 40.417679][ C3] local_irq_disable(); [ 40.417862][ C3] lock(k-slock-AF_INET); [ 40.418046][ C3] lock(&p->alloc_lock); [ 40.418239][ C3] [ 40.418354][ C3] lock(k-slock-AF_INET); [ 40.418508][ C3] [ 40.418508][ C3] *** DEADLOCK *** [ 40.418508][ C3] [ 40.418725][ C3] 2 locks held by mausezahn/368: [ 40.418881][ C3] #0: ffffffffac1522e0 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1f8/0x670 [ 40.419164][ C3] #1: ffffffffac1522e0 (rcu_read_lock){....}-{1:2}, at: ip_local_deliver_finish+0x209/0x4b0 [ 40.419476][ C3] [ 40.419476][ C3] the shortest dependencies between 2nd lock and 1st lock: [ 40.419733][ C3] -> (&p->alloc_lock){+.+.}-{2:2} { [ 40.419888][ C3] HARDIRQ-ON-W at: [ 40.420005][ C3] __lock_acquire+0x797/0x1570 [ 40.420203][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.420394][ C3] _raw_spin_lock+0x30/0x40 [ 40.420593][ C3] set_mems_allowed+0x1d/0x210 [ 40.420787][ C3] kernel_init_freeable+0x72/0x310 [ 40.420977][ C3] kernel_init+0x20/0x200 [ 40.421190][ C3] ret_from_fork+0x31/0x70 [ 40.421376][ C3] ret_from_fork_asm+0x1a/0x30 [ 40.421570][ C3] SOFTIRQ-ON-W at: [ 40.421690][ C3] __lock_acquire+0x797/0x1570 [ 40.421872][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.422057][ C3] _raw_spin_lock+0x30/0x40 [ 40.422240][ C3] set_mems_allowed+0x1d/0x210 [ 40.422422][ C3] kernel_init_freeable+0x72/0x310 [ 40.422607][ C3] kernel_init+0x20/0x200 [ 40.422796][ C3] ret_from_fork+0x31/0x70 [ 40.422980][ C3] ret_from_fork_asm+0x1a/0x30 [ 40.423166][ C3] INITIAL USE at: [ 40.423277][ C3] __lock_acquire+0x797/0x1570 [ 40.423467][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.423659][ C3] _raw_spin_lock+0x30/0x40 [ 40.423868][ C3] set_mems_allowed+0x1d/0x210 [ 40.424075][ C3] kernel_init_freeable+0x72/0x310 [ 40.424269][ C3] kernel_init+0x20/0x200 [ 40.424460][ C3] ret_from_fork+0x31/0x70 [ 40.424651][ C3] ret_from_fork_asm+0x1a/0x30 [ 40.424837][ C3] } [ 40.424914][ C3] ... key at: [] __key.387+0x0/0x40 [ 40.425143][ C3] ... acquired at: [ 40.425256][ C3] __lock_acquire+0xaf0/0x1570 [ 40.425421][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.425571][ C3] _raw_spin_lock+0x30/0x40 [ 40.425718][ C3] __get_task_comm+0x27/0x70 [ 40.425869][ C3] ref_tracker_alloc+0x2ee/0x490 [ 40.426021][ C3] dst_init+0x84/0x4a0 [ 40.426133][ C3] dst_alloc+0x97/0x150 [ 40.426250][ C3] rt_dst_alloc+0x31/0x3b0 [ 40.426401][ C3] __mkroute_output+0x451/0x1070 [ 40.426550][ C3] ip_route_output_key_hash+0x15d/0x2b0 [ 40.426708][ C3] icmp_route_lookup.constprop.0+0x321/0xb30 [ 40.426891][ C3] __icmp_send+0x8e8/0x1910 [ 40.427035][ C3] ip_fragment.constprop.0+0x183/0x240 [ 40.427181][ C3] ip_output+0x16b/0x4f0 [ 40.427333][ C3] iptunnel_xmit+0x578/0x930 [ 40.427491][ C3] vxlan_xmit_one+0x2a6d/0x4570 [vxlan] [ 40.427671][ C3] vxlan_xmit+0x855/0xf40 [vxlan] [ 40.427830][ C3] dev_hard_start_xmit+0x10e/0x360 [ 40.427977][ C3] __dev_queue_xmit+0xe69/0x16d0 [ 40.428124][ C3] packet_snd+0xd08/0x1840 [ 40.428273][ C3] __sys_sendto+0x37d/0x410 [ 40.428421][ C3] __x64_sys_sendto+0xe0/0x1c0 [ 40.428575][ C3] do_syscall_64+0xc6/0x1e0 [ 40.428725][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 40.428912][ C3] [ 40.428987][ C3] -> (k-slock-AF_INET){+.-.}-{2:2} { [ 40.429138][ C3] HARDIRQ-ON-W at: [ 40.429250][ C3] __lock_acquire+0x797/0x1570 [ 40.429436][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.429620][ C3] _raw_spin_lock_bh+0x38/0x50 [ 40.429806][ C3] lock_sock_nested+0x59/0xe0 [ 40.429995][ C3] sock_bindtoindex+0x25/0x50 [ 40.430177][ C3] udp_sock_create4+0x104/0x3f0 [udp_tunnel] [ 40.430402][ C3] vxlan_create_sock+0xbc/0x1d0 [vxlan] [ 40.430633][ C3] vxlan_socket_create+0xfb/0x500 [vxlan] [ 40.430875][ C3] __vxlan_sock_add+0x388/0x8d0 [vxlan] [ 40.431110][ C3] vxlan_open+0x151/0x250 [vxlan] [ 40.431345][ C3] __dev_open+0x221/0x430 [ 40.431531][ C3] __dev_change_flags+0x469/0x6c0 [ 40.431803][ C3] dev_change_flags+0x80/0x160 [ 40.431988][ C3] do_setlink+0x7d6/0x21e0 [ 40.432171][ C3] __rtnl_newlink+0xaa7/0xd80 [ 40.432352][ C3] rtnl_newlink+0x63/0xa0 [ 40.432541][ C3] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 40.432724][ C3] netlink_rcv_skb+0x130/0x360 [ 40.432906][ C3] netlink_unicast+0x449/0x710 [ 40.433087][ C3] netlink_sendmsg+0x723/0xbe0 [ 40.433269][ C3] ____sys_sendmsg+0x7b2/0xa10 [ 40.433460][ C3] ___sys_sendmsg+0xee/0x170 [ 40.433645][ C3] __sys_sendmsg+0xcd/0x170 [ 40.433856][ C3] do_syscall_64+0xc6/0x1e0 [ 40.434045][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 40.434267][ C3] IN-SOFTIRQ-W at: [ 40.434378][ C3] __lock_acquire+0x797/0x1570 [ 40.434569][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.434756][ C3] _raw_spin_lock+0x30/0x40 [ 40.434946][ C3] ipv4_sk_update_pmtu+0xb3/0xb60 [ 40.435187][ C3] __udp4_lib_err+0x7f6/0xa50 [ 40.435381][ C3] icmp_unreach+0x2c1/0xba0 [ 40.435624][ C3] icmp_rcv+0x791/0xec0 [ 40.435809][ C3] ip_protocol_deliver_rcu+0x2e5/0x360 [ 40.435994][ C3] ip_local_deliver_finish+0x2c2/0x4b0 [ 40.436177][ C3] ip_local_deliver+0x19d/0x480 [ 40.436365][ C3] ip_rcv+0x564/0x740 [ 40.436517][ C3] __netif_receive_skb_one_core+0x166/0x1b0 [ 40.436737][ C3] process_backlog+0x103/0x670 [ 40.436918][ C3] __napi_poll.constprop.0+0xa2/0x460 [ 40.437102][ C3] net_rx_action+0x49f/0xc00 [ 40.437286][ C3] __do_softirq+0x1f8/0x5df [ 40.437492][ C3] do_softirq+0x4d/0xa0 [ 40.437675][ C3] __local_bh_enable_ip+0xf6/0x120 [ 40.437860][ C3] __dev_queue_xmit+0x6ac/0x16d0 [ 40.438048][ C3] packet_snd+0xd08/0x1840 [ 40.438233][ C3] __sys_sendto+0x37d/0x410 [ 40.438419][ C3] __x64_sys_sendto+0xe0/0x1c0 [ 40.438601][ C3] do_syscall_64+0xc6/0x1e0 [ 40.438795][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 40.439014][ C3] INITIAL USE at: [ 40.439126][ C3] __lock_acquire+0x797/0x1570 [ 40.439309][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.439490][ C3] _raw_spin_lock_bh+0x38/0x50 [ 40.439673][ C3] lock_sock_nested+0x59/0xe0 [ 40.439855][ C3] sock_bindtoindex+0x25/0x50 [ 40.440041][ C3] udp_sock_create4+0x104/0x3f0 [udp_tunnel] [ 40.440268][ C3] vxlan_create_sock+0xbc/0x1d0 [vxlan] [ 40.440474][ C3] vxlan_socket_create+0xfb/0x500 [vxlan] [ 40.440703][ C3] __vxlan_sock_add+0x388/0x8d0 [vxlan] [ 40.440920][ C3] vxlan_open+0x151/0x250 [vxlan] [ 40.441123][ C3] __dev_open+0x221/0x430 [ 40.441315][ C3] __dev_change_flags+0x469/0x6c0 [ 40.441498][ C3] dev_change_flags+0x80/0x160 [ 40.441690][ C3] do_setlink+0x7d6/0x21e0 [ 40.441895][ C3] __rtnl_newlink+0xaa7/0xd80 [ 40.442085][ C3] rtnl_newlink+0x63/0xa0 [ 40.442271][ C3] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 40.442459][ C3] netlink_rcv_skb+0x130/0x360 [ 40.442642][ C3] netlink_unicast+0x449/0x710 [ 40.442845][ C3] netlink_sendmsg+0x723/0xbe0 [ 40.443034][ C3] ____sys_sendmsg+0x7b2/0xa10 [ 40.443220][ C3] ___sys_sendmsg+0xee/0x170 [ 40.443410][ C3] __sys_sendmsg+0xcd/0x170 [ 40.443597][ C3] do_syscall_64+0xc6/0x1e0 [ 40.443787][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 40.444019][ C3] } [ 40.444097][ C3] ... key at: [] af_family_kern_slock_keys+0x20/0x300 [ 40.444361][ C3] ... acquired at: [ 40.444473][ C3] mark_lock+0x28d/0x3e0 [ 40.444625][ C3] mark_usage+0xd9/0x2a0 [ 40.444780][ C3] __lock_acquire+0x797/0x1570 [ 40.445038][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.445189][ C3] _raw_spin_lock+0x30/0x40 [ 40.445337][ C3] ipv4_sk_update_pmtu+0xb3/0xb60 [ 40.445505][ C3] __udp4_lib_err+0x7f6/0xa50 [ 40.445740][ C3] icmp_unreach+0x2c1/0xba0 [ 40.445885][ C3] icmp_rcv+0x791/0xec0 [ 40.446000][ C3] ip_protocol_deliver_rcu+0x2e5/0x360 [ 40.446145][ C3] ip_local_deliver_finish+0x2c2/0x4b0 [ 40.446290][ C3] ip_local_deliver+0x19d/0x480 [ 40.446529][ C3] ip_rcv+0x564/0x740 [ 40.446641][ C3] __netif_receive_skb_one_core+0x166/0x1b0 [ 40.446821][ C3] process_backlog+0x103/0x670 [ 40.446965][ C3] __napi_poll.constprop.0+0xa2/0x460 [ 40.447224][ C3] net_rx_action+0x49f/0xc00 [ 40.447370][ C3] __do_softirq+0x1f8/0x5df [ 40.447518][ C3] do_softirq+0x4d/0xa0 [ 40.447629][ C3] __local_bh_enable_ip+0xf6/0x120 [ 40.447868][ C3] __dev_queue_xmit+0x6ac/0x16d0 [ 40.448016][ C3] packet_snd+0xd08/0x1840 [ 40.448164][ C3] __sys_sendto+0x37d/0x410 [ 40.448311][ C3] __x64_sys_sendto+0xe0/0x1c0 [ 40.448458][ C3] do_syscall_64+0xc6/0x1e0 [ 40.448694][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 40.448880][ C3] [ 40.448956][ C3] [ 40.448956][ C3] stack backtrace: [ 40.449139][ C3] CPU: 3 PID: 368 Comm: mausezahn Not tainted 6.9.0-rc1-virtme #1 [ 40.449544][ C3] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 40.449873][ C3] Call Trace: [ 40.449988][ C3] [ 40.450157][ C3] dump_stack_lvl+0x82/0xd0 [ 40.450312][ C3] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 40.450496][ C3] ? kernel_text_address+0xce/0xe0 [ 40.450653][ C3] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 40.450926][ C3] ? __pfx_usage_skip+0x10/0x10 [ 40.451074][ C3] ? __pfx_usage_match+0x10/0x10 [ 40.451221][ C3] ? arch_stack_walk+0xa2/0xf0 [ 40.451375][ C3] mark_lock_irq+0x4cd/0xa10 [ 40.451614][ C3] ? __pfx_mark_lock_irq+0x10/0x10 [ 40.451773][ C3] ? __pfx_stack_trace_save+0x10/0x10 [ 40.451926][ C3] ? __pfx_validate_chain+0x10/0x10 [ 40.452079][ C3] ? hlock_class+0x4e/0x130 [ 40.452314][ C3] ? save_trace+0x8f/0x5b0 [ 40.452464][ C3] mark_lock+0x28d/0x3e0 [ 40.452575][ C3] mark_usage+0xd9/0x2a0 [ 40.452688][ C3] __lock_acquire+0x797/0x1570 [ 40.452837][ C3] lock_acquire.part.0+0xeb/0x330 [ 40.453125][ C3] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 40.453270][ C3] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 40.453414][ C3] ? trace_lock_acquire+0x135/0x1c0 [ 40.453563][ C3] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 40.453795][ C3] ? lock_acquire+0x32/0xc0 [ 40.453940][ C3] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 40.454098][ C3] _raw_spin_lock+0x30/0x40 [ 40.454256][ C3] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 40.454401][ C3] ipv4_sk_update_pmtu+0xb3/0xb60 [ 40.454642][ C3] ? vxlan_err_lookup+0x23f/0x420 [vxlan] [ 40.454806][ C3] ? __pfx_ipv4_sk_update_pmtu+0x10/0x10 [ 40.454976][ C3] ? __udp4_lib_err_encap+0x59c/0x930 [ 40.455123][ C3] __udp4_lib_err+0x7f6/0xa50 [ 40.455356][ C3] icmp_unreach+0x2c1/0xba0 [ 40.455526][ C3] icmp_rcv+0x791/0xec0 [ 40.455641][ C3] ip_protocol_deliver_rcu+0x2e5/0x360 [ 40.455789][ C3] ip_local_deliver_finish+0x2c2/0x4b0 [ 40.456022][ C3] ip_local_deliver+0x19d/0x480 [ 40.456172][ C3] ? __pfx_ip_local_deliver+0x10/0x10 [ 40.456319][ C3] ? ip_rcv_finish_core.constprop.0+0x1ad/0x1300 [ 40.456503][ C3] ip_rcv+0x564/0x740 [ 40.456613][ C3] ? __pfx_ip_rcv+0x10/0x10 [ 40.456845][ C3] ? lock_acquire.part.0+0xeb/0x330 [ 40.456988][ C3] ? process_backlog+0x1f8/0x670 [ 40.457134][ C3] ? __pfx_ip_rcv+0x10/0x10 [ 40.457279][ C3] __netif_receive_skb_one_core+0x166/0x1b0 [ 40.457551][ C3] ? __pfx___netif_receive_skb_one_core+0x10/0x10 [ 40.457731][ C3] ? process_backlog+0x1f8/0x670 [ 40.457874][ C3] ? lock_acquire+0x32/0xc0 [ 40.458019][ C3] ? process_backlog+0x1f8/0x670 [ 40.458252][ C3] process_backlog+0x103/0x670 [ 40.458398][ C3] ? find_held_lock+0x2c/0x110 [ 40.458543][ C3] __napi_poll.constprop.0+0xa2/0x460 [ 40.458688][ C3] net_rx_action+0x49f/0xc00 [ 40.458925][ C3] ? __pfx_net_rx_action+0x10/0x10 [ 40.459075][ C3] ? ktime_get+0xb3/0x190 [ 40.459187][ C3] ? hrtimer_interrupt+0x311/0x770 [ 40.459333][ C3] ? hrtimer_interrupt+0x31d/0x770 [ 40.459571][ C3] __do_softirq+0x1f8/0x5df [ 40.459727][ C3] ? __dev_queue_xmit+0x697/0x16d0 [ 40.459874][ C3] do_softirq+0x4d/0xa0 [ 40.459985][ C3] [ 40.460060][ C3] [ 40.460134][ C3] __local_bh_enable_ip+0xf6/0x120 [ 40.460370][ C3] ? __dev_queue_xmit+0x697/0x16d0 [ 40.460515][ C3] __dev_queue_xmit+0x6ac/0x16d0 [ 40.460665][ C3] ? packet_parse_headers+0x231/0x9d0 [ 40.460814][ C3] ? packet_parse_headers+0x231/0x9d0 [ 40.461219][ C3] ? __pfx_sock_alloc_send_pskb+0x10/0x10 [ 40.461367][ C3] ? __pfx___dev_queue_xmit+0x10/0x10 [ 40.461528][ C3] ? dev_get_by_index+0x177/0x270 [ 40.461678][ C3] packet_snd+0xd08/0x1840 [ 40.461913][ C3] ? __pfx___lock_release+0x10/0x10 [ 40.462080][ C3] ? __pfx_packet_snd+0x10/0x10 [ 40.462251][ C3] ? __might_fault+0xc3/0x170 [ 40.462398][ C3] ? trace_raw_output_mmap_lock_acquire_returned+0x60/0x1e0 [ 40.462699][ C3] ? __might_fault+0x11b/0x170 [ 40.462845][ C3] __sys_sendto+0x37d/0x410 [ 40.462991][ C3] ? __pfx___sys_sendto+0x10/0x10 [ 40.463144][ C3] ? sock_ioctl+0x3bc/0x590 [ 40.463375][ C3] ? __up_read+0x1a7/0x780 [ 40.463520][ C3] ? __pfx___up_read+0x10/0x10 [ 40.463663][ C3] ? do_user_addr_fault+0x4b8/0xbe0 [ 40.463812][ C3] __x64_sys_sendto+0xe0/0x1c0 [ 40.463957][ C3] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 40.464224][ C3] do_syscall_64+0xc6/0x1e0 [ 40.464369][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 40.464553][ C3] RIP: 0033:0x7ff19005c85a [ 40.464799][ C3] Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89 [ 40.465329][ C3] RSP: 002b:00007fff0476c908 EFLAGS: 00000246 ORIG_RAX: 000000000000002c [ 40.465657][ C3] RAX: ffffffffffffffda RBX: 00000000241d19d0 RCX: 00007ff19005c85a [ 40.465880][ C3] RDX: 00000000000003c5 RSI: 00000000241d2022 RDI: 0000000000000005 [ 40.466098][ C3] RBP: 00000000241d2022 R08: 00007fff0476c910 R09: 0000000000000014 [ 40.466408][ C3] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 40.466626][ C3] R13: 00000000000003c5 R14: 00007fff0476c910 R15: 0000000000000000 [ 40.466846][ C3]