[ 28.876545][ T306] ip (306) used greatest stack depth: 23216 bytes left [ 38.436588][ T354] GACT probability NOT on [ 39.772147][ C3] [ 39.772300][ C3] ======================================================== [ 39.772537][ C3] WARNING: possible irq lock inversion dependency detected [ 39.772764][ C3] 6.9.0-rc1-virtme #1 Not tainted [ 39.772919][ C3] -------------------------------------------------------- [ 39.773156][ C3] mausezahn/368 just changed the state of lock: [ 39.773352][ C3] ffff888003f03858 (k-slock-AF_INET){+.-.}-{2:2}, at: ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.773637][ C3] but this lock took another, SOFTIRQ-unsafe lock in the past: [ 39.773869][ C3] (&p->alloc_lock){+.+.}-{2:2} [ 39.773873][ C3] [ 39.773873][ C3] [ 39.773873][ C3] and interrupts could create inverse lock ordering between them. [ 39.773873][ C3] [ 39.774442][ C3] [ 39.774442][ C3] other info that might help us debug this: [ 39.774673][ C3] Possible interrupt unsafe locking scenario: [ 39.774673][ C3] [ 39.774908][ C3] CPU0 CPU1 [ 39.775064][ C3] ---- ---- [ 39.775218][ C3] lock(&p->alloc_lock); [ 39.775339][ C3] local_irq_disable(); [ 39.775531][ C3] lock(k-slock-AF_INET); [ 39.775732][ C3] lock(&p->alloc_lock); [ 39.775933][ C3] [ 39.776053][ C3] lock(k-slock-AF_INET); [ 39.776212][ C3] [ 39.776212][ C3] *** DEADLOCK *** [ 39.776212][ C3] [ 39.776443][ C3] 2 locks held by mausezahn/368: [ 39.776603][ C3] #0: ffffffff98d522e0 (rcu_read_lock){....}-{1:2}, at: process_backlog+0x1f8/0x670 [ 39.776891][ C3] #1: ffffffff98d522e0 (rcu_read_lock){....}-{1:2}, at: ip_local_deliver_finish+0x209/0x4b0 [ 39.777215][ C3] [ 39.777215][ C3] the shortest dependencies between 2nd lock and 1st lock: [ 39.777485][ C3] -> (&p->alloc_lock){+.+.}-{2:2} { [ 39.777648][ C3] HARDIRQ-ON-W at: [ 39.777769][ C3] __lock_acquire+0x797/0x1570 [ 39.777975][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.778173][ C3] _raw_spin_lock+0x30/0x40 [ 39.778374][ C3] set_mems_allowed+0x1d/0x210 [ 39.778571][ C3] kernel_init_freeable+0x72/0x310 [ 39.778768][ C3] kernel_init+0x20/0x200 [ 39.778968][ C3] ret_from_fork+0x31/0x70 [ 39.779179][ C3] ret_from_fork_asm+0x1a/0x30 [ 39.779380][ C3] SOFTIRQ-ON-W at: [ 39.779498][ C3] __lock_acquire+0x797/0x1570 [ 39.779692][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.779886][ C3] _raw_spin_lock+0x30/0x40 [ 39.780085][ C3] set_mems_allowed+0x1d/0x210 [ 39.780281][ C3] kernel_init_freeable+0x72/0x310 [ 39.780482][ C3] kernel_init+0x20/0x200 [ 39.780680][ C3] ret_from_fork+0x31/0x70 [ 39.780878][ C3] ret_from_fork_asm+0x1a/0x30 [ 39.781075][ C3] INITIAL USE at: [ 39.781193][ C3] __lock_acquire+0x797/0x1570 [ 39.781393][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.781594][ C3] _raw_spin_lock+0x30/0x40 [ 39.781795][ C3] set_mems_allowed+0x1d/0x210 [ 39.781991][ C3] kernel_init_freeable+0x72/0x310 [ 39.782192][ C3] kernel_init+0x20/0x200 [ 39.782392][ C3] ret_from_fork+0x31/0x70 [ 39.782590][ C3] ret_from_fork_asm+0x1a/0x30 [ 39.782791][ C3] } [ 39.782872][ C3] ... key at: [] __key.387+0x0/0x40 [ 39.783108][ C3] ... acquired at: [ 39.783226][ C3] __lock_acquire+0xaf0/0x1570 [ 39.783390][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.783550][ C3] _raw_spin_lock+0x30/0x40 [ 39.783709][ C3] __get_task_comm+0x27/0x70 [ 39.783865][ C3] ref_tracker_alloc+0x2ee/0x490 [ 39.784024][ C3] dst_init+0x84/0x4a0 [ 39.784151][ C3] dst_alloc+0x97/0x150 [ 39.784270][ C3] rt_dst_alloc+0x31/0x3b0 [ 39.784429][ C3] __mkroute_output+0x451/0x1070 [ 39.784586][ C3] ip_route_output_key_hash+0x15d/0x2b0 [ 39.784746][ C3] icmp_route_lookup.constprop.0+0x321/0xb30 [ 39.784941][ C3] __icmp_send+0x8e8/0x1910 [ 39.785102][ C3] ip_fragment.constprop.0+0x183/0x240 [ 39.785258][ C3] ip_output+0x16b/0x4f0 [ 39.785422][ C3] iptunnel_xmit+0x578/0x930 [ 39.785580][ C3] vxlan_xmit_one+0x2a6d/0x4570 [vxlan] [ 39.785767][ C3] vxlan_xmit+0x855/0xf40 [vxlan] [ 39.785937][ C3] dev_hard_start_xmit+0x10e/0x360 [ 39.786095][ C3] __dev_queue_xmit+0xe69/0x16d0 [ 39.786259][ C3] packet_snd+0xd08/0x1840 [ 39.786419][ C3] __sys_sendto+0x37d/0x410 [ 39.786577][ C3] __x64_sys_sendto+0xe0/0x1c0 [ 39.786739][ C3] do_syscall_64+0xc6/0x1e0 [ 39.786899][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.787099][ C3] [ 39.787179][ C3] -> (k-slock-AF_INET){+.-.}-{2:2} { [ 39.787341][ C3] HARDIRQ-ON-W at: [ 39.787460][ C3] __lock_acquire+0x797/0x1570 [ 39.787665][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.787861][ C3] _raw_spin_lock_bh+0x38/0x50 [ 39.788060][ C3] lock_sock_nested+0x59/0xe0 [ 39.788258][ C3] sock_bindtoindex+0x25/0x50 [ 39.788453][ C3] udp_sock_create4+0x104/0x3f0 [udp_tunnel] [ 39.788698][ C3] vxlan_create_sock+0xbc/0x1d0 [vxlan] [ 39.788945][ C3] vxlan_socket_create+0xfb/0x500 [vxlan] [ 39.789240][ C3] __vxlan_sock_add+0x388/0x8d0 [vxlan] [ 39.789494][ C3] vxlan_open+0x151/0x250 [vxlan] [ 39.789705][ C3] __dev_open+0x221/0x430 [ 39.789904][ C3] __dev_change_flags+0x469/0x6c0 [ 39.790102][ C3] dev_change_flags+0x80/0x160 [ 39.790300][ C3] do_setlink+0x7d6/0x21e0 [ 39.790499][ C3] __rtnl_newlink+0xaa7/0xd80 [ 39.790702][ C3] rtnl_newlink+0x63/0xa0 [ 39.790899][ C3] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 39.791099][ C3] netlink_rcv_skb+0x130/0x360 [ 39.791300][ C3] netlink_unicast+0x449/0x710 [ 39.791498][ C3] netlink_sendmsg+0x723/0xbe0 [ 39.791695][ C3] ____sys_sendmsg+0x7b2/0xa10 [ 39.791890][ C3] ___sys_sendmsg+0xee/0x170 [ 39.792091][ C3] __sys_sendmsg+0xcd/0x170 [ 39.792288][ C3] do_syscall_64+0xc6/0x1e0 [ 39.792489][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.792733][ C3] IN-SOFTIRQ-W at: [ 39.792854][ C3] __lock_acquire+0x797/0x1570 [ 39.793053][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.793252][ C3] _raw_spin_lock+0x30/0x40 [ 39.793452][ C3] ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.793646][ C3] __udp4_lib_err+0x7f6/0xa50 [ 39.793845][ C3] icmp_unreach+0x2c1/0xba0 [ 39.794040][ C3] icmp_rcv+0x791/0xec0 [ 39.794238][ C3] ip_protocol_deliver_rcu+0x2e5/0x360 [ 39.794439][ C3] ip_local_deliver_finish+0x2c2/0x4b0 [ 39.794641][ C3] ip_local_deliver+0x19d/0x480 [ 39.794842][ C3] ip_rcv+0x564/0x740 [ 39.795002][ C3] __netif_receive_skb_one_core+0x166/0x1b0 [ 39.795246][ C3] process_backlog+0x103/0x670 [ 39.795443][ C3] __napi_poll.constprop.0+0xa2/0x460 [ 39.795640][ C3] net_rx_action+0x49f/0xc00 [ 39.795843][ C3] __do_softirq+0x1f8/0x5df [ 39.796038][ C3] do_softirq+0x4d/0xa0 [ 39.796236][ C3] __local_bh_enable_ip+0xf6/0x120 [ 39.796435][ C3] __dev_queue_xmit+0x6ac/0x16d0 [ 39.796653][ C3] packet_snd+0xd08/0x1840 [ 39.796851][ C3] __sys_sendto+0x37d/0x410 [ 39.797049][ C3] __x64_sys_sendto+0xe0/0x1c0 [ 39.797261][ C3] do_syscall_64+0xc6/0x1e0 [ 39.797457][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.797694][ C3] INITIAL USE at: [ 39.797815][ C3] __lock_acquire+0x797/0x1570 [ 39.798010][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.798207][ C3] _raw_spin_lock_bh+0x38/0x50 [ 39.798411][ C3] lock_sock_nested+0x59/0xe0 [ 39.798608][ C3] sock_bindtoindex+0x25/0x50 [ 39.798806][ C3] udp_sock_create4+0x104/0x3f0 [udp_tunnel] [ 39.799056][ C3] vxlan_create_sock+0xbc/0x1d0 [vxlan] [ 39.799282][ C3] vxlan_socket_create+0xfb/0x500 [vxlan] [ 39.799525][ C3] __vxlan_sock_add+0x388/0x8d0 [vxlan] [ 39.799739][ C3] vxlan_open+0x151/0x250 [vxlan] [ 39.799949][ C3] __dev_open+0x221/0x430 [ 39.800145][ C3] __dev_change_flags+0x469/0x6c0 [ 39.800343][ C3] dev_change_flags+0x80/0x160 [ 39.800543][ C3] do_setlink+0x7d6/0x21e0 [ 39.800739][ C3] __rtnl_newlink+0xaa7/0xd80 [ 39.800935][ C3] rtnl_newlink+0x63/0xa0 [ 39.801136][ C3] rtnetlink_rcv_msg+0x2fb/0xbf0 [ 39.801333][ C3] netlink_rcv_skb+0x130/0x360 [ 39.801528][ C3] netlink_unicast+0x449/0x710 [ 39.801727][ C3] netlink_sendmsg+0x723/0xbe0 [ 39.801927][ C3] ____sys_sendmsg+0x7b2/0xa10 [ 39.802130][ C3] ___sys_sendmsg+0xee/0x170 [ 39.802328][ C3] __sys_sendmsg+0xcd/0x170 [ 39.802528][ C3] do_syscall_64+0xc6/0x1e0 [ 39.802727][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.802962][ C3] } [ 39.803046][ C3] ... key at: [] af_family_kern_slock_keys+0x20/0x300 [ 39.803322][ C3] ... acquired at: [ 39.803440][ C3] mark_lock+0x28d/0x3e0 [ 39.803597][ C3] mark_usage+0xd9/0x2a0 [ 39.803760][ C3] __lock_acquire+0x797/0x1570 [ 39.803920][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.804078][ C3] _raw_spin_lock+0x30/0x40 [ 39.804236][ C3] ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.804393][ C3] __udp4_lib_err+0x7f6/0xa50 [ 39.804550][ C3] icmp_unreach+0x2c1/0xba0 [ 39.804793][ C3] icmp_rcv+0x791/0xec0 [ 39.804911][ C3] ip_protocol_deliver_rcu+0x2e5/0x360 [ 39.805077][ C3] ip_local_deliver_finish+0x2c2/0x4b0 [ 39.805236][ C3] ip_local_deliver+0x19d/0x480 [ 39.805480][ C3] ip_rcv+0x564/0x740 [ 39.805599][ C3] __netif_receive_skb_one_core+0x166/0x1b0 [ 39.805798][ C3] process_backlog+0x103/0x670 [ 39.805954][ C3] __napi_poll.constprop.0+0xa2/0x460 [ 39.806110][ C3] net_rx_action+0x49f/0xc00 [ 39.806355][ C3] __do_softirq+0x1f8/0x5df [ 39.806512][ C3] do_softirq+0x4d/0xa0 [ 39.806632][ C3] __local_bh_enable_ip+0xf6/0x120 [ 39.806808][ C3] __dev_queue_xmit+0x6ac/0x16d0 [ 39.807052][ C3] packet_snd+0xd08/0x1840 [ 39.807211][ C3] __sys_sendto+0x37d/0x410 [ 39.807369][ C3] __x64_sys_sendto+0xe0/0x1c0 [ 39.807527][ C3] do_syscall_64+0xc6/0x1e0 [ 39.807774][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.807973][ C3] [ 39.808056][ C3] [ 39.808056][ C3] stack backtrace: [ 39.808252][ C3] CPU: 3 PID: 368 Comm: mausezahn Not tainted 6.9.0-rc1-virtme #1 [ 39.808574][ C3] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 39.808922][ C3] Call Trace: [ 39.809050][ C3] [ 39.809145][ C3] dump_stack_lvl+0x82/0xd0 [ 39.809394][ C3] print_irq_inversion_bug.part.0+0x3d9/0x570 [ 39.809594][ C3] ? kernel_text_address+0xce/0xe0 [ 39.809756][ C3] ? __pfx_print_irq_inversion_bug.part.0+0x10/0x10 [ 39.809949][ C3] ? __pfx_usage_skip+0x10/0x10 [ 39.810196][ C3] ? __pfx_usage_match+0x10/0x10 [ 39.810354][ C3] ? arch_stack_walk+0xa2/0xf0 [ 39.810512][ C3] mark_lock_irq+0x4cd/0xa10 [ 39.810668][ C3] ? __pfx_mark_lock_irq+0x10/0x10 [ 39.810998][ C3] ? __pfx_stack_trace_save+0x10/0x10 [ 39.811169][ C3] ? __pfx_validate_chain+0x10/0x10 [ 39.811328][ C3] ? hlock_class+0x4e/0x130 [ 39.811485][ C3] ? save_trace+0x8f/0x5b0 [ 39.811729][ C3] mark_lock+0x28d/0x3e0 [ 39.811849][ C3] mark_usage+0xd9/0x2a0 [ 39.811969][ C3] __lock_acquire+0x797/0x1570 [ 39.812129][ C3] ? usage_skip+0x182/0x190 [ 39.812287][ C3] lock_acquire.part.0+0xeb/0x330 [ 39.812529][ C3] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.812689][ C3] ? __pfx_lock_acquire.part.0+0x10/0x10 [ 39.812845][ C3] ? trace_lock_acquire+0x135/0x1c0 [ 39.813004][ C3] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.813250][ C3] ? lock_acquire+0x32/0xc0 [ 39.813407][ C3] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.813564][ C3] _raw_spin_lock+0x30/0x40 [ 39.813720][ C3] ? ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.813967][ C3] ipv4_sk_update_pmtu+0xb3/0xb60 [ 39.814125][ C3] ? vxlan_err_lookup+0x23f/0x420 [vxlan] [ 39.814293][ C3] ? __pfx_ipv4_sk_update_pmtu+0x10/0x10 [ 39.814450][ C3] ? __udp4_lib_err_encap+0x59c/0x930 [ 39.814608][ C3] __udp4_lib_err+0x7f6/0xa50 [ 39.814854][ C3] icmp_unreach+0x2c1/0xba0 [ 39.815010][ C3] icmp_rcv+0x791/0xec0 [ 39.815134][ C3] ip_protocol_deliver_rcu+0x2e5/0x360 [ 39.815300][ C3] ip_local_deliver_finish+0x2c2/0x4b0 [ 39.815546][ C3] ip_local_deliver+0x19d/0x480 [ 39.815703][ C3] ? __pfx_ip_local_deliver+0x10/0x10 [ 39.815863][ C3] ? ip_rcv_finish_core.constprop.0+0x1ad/0x1300 [ 39.816059][ C3] ip_rcv+0x564/0x740 [ 39.816263][ C3] ? __pfx_ip_rcv+0x10/0x10 [ 39.816420][ C3] ? lock_acquire.part.0+0xeb/0x330 [ 39.816575][ C3] ? process_backlog+0x1f8/0x670 [ 39.816738][ C3] ? __pfx_ip_rcv+0x10/0x10 [ 39.816896][ C3] __netif_receive_skb_one_core+0x166/0x1b0 [ 39.817181][ C3] ? __pfx___netif_receive_skb_one_core+0x10/0x10 [ 39.817379][ C3] ? process_backlog+0x1f8/0x670 [ 39.817538][ C3] ? lock_acquire+0x32/0xc0 [ 39.817788][ C3] ? process_backlog+0x1f8/0x670 [ 39.817976][ C3] process_backlog+0x103/0x670 [ 39.818137][ C3] __napi_poll.constprop.0+0xa2/0x460 [ 39.818298][ C3] net_rx_action+0x49f/0xc00 [ 39.818459][ C3] ? __pfx_net_rx_action+0x10/0x10 [ 39.818705][ C3] ? mark_held_locks+0x9e/0xe0 [ 39.818862][ C3] ? lockdep_hardirqs_on_prepare.part.0+0x14f/0x370 [ 39.819073][ C3] __do_softirq+0x1f8/0x5df [ 39.819231][ C3] ? __dev_queue_xmit+0x697/0x16d0 [ 39.819475][ C3] do_softirq+0x4d/0xa0 [ 39.819597][ C3] [ 39.819679][ C3] [ 39.819762][ C3] __local_bh_enable_ip+0xf6/0x120 [ 39.819921][ C3] ? __dev_queue_xmit+0x697/0x16d0 [ 39.820166][ C3] __dev_queue_xmit+0x6ac/0x16d0 [ 39.820324][ C3] ? packet_parse_headers+0x231/0x9d0 [ 39.820481][ C3] ? packet_parse_headers+0x231/0x9d0 [ 39.820639][ C3] ? __pfx_sock_alloc_send_pskb+0x10/0x10 [ 39.820887][ C3] ? __pfx___dev_queue_xmit+0x10/0x10 [ 39.821066][ C3] ? dev_get_by_index+0x177/0x270 [ 39.821226][ C3] packet_snd+0xd08/0x1840 [ 39.821386][ C3] ? __pfx___lock_release+0x10/0x10 [ 39.821546][ C3] ? __pfx_packet_snd+0x10/0x10 [ 39.821790][ C3] ? __might_fault+0xc3/0x170 [ 39.821950][ C3] ? trace_raw_output_mmap_lock_acquire_returned+0x60/0x1e0 [ 39.822181][ C3] ? __might_fault+0x11b/0x170 [ 39.822428][ C3] __sys_sendto+0x37d/0x410 [ 39.822588][ C3] ? __pfx___sys_sendto+0x10/0x10 [ 39.822761][ C3] ? sock_ioctl+0x3bc/0x590 [ 39.822928][ C3] ? __up_read+0x1a7/0x780 [ 39.823087][ C3] ? __pfx___up_read+0x10/0x10 [ 39.823420][ C3] ? do_user_addr_fault+0x4b8/0xbe0 [ 39.823582][ C3] __x64_sys_sendto+0xe0/0x1c0 [ 39.823743][ C3] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370 [ 39.823946][ C3] do_syscall_64+0xc6/0x1e0 [ 39.824188][ C3] entry_SYSCALL_64_after_hwframe+0x6d/0x75 [ 39.824395][ C3] RIP: 0033:0x7fbecdf3485a [ 39.824558][ C3] Code: d8 64 89 02 48 c7 c0 ff ff ff ff eb b8 0f 1f 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 7e c3 0f 1f 44 00 00 41 54 48 83 ec 30 44 89 [ 39.825217][ C3] RSP: 002b:00007fffb03d8518 EFLAGS: 00000246 ORIG_RAX: 000000000000002c [ 39.825461][ C3] RAX: ffffffffffffffda RBX: 000000000d9629d0 RCX: 00007fbecdf3485a [ 39.825806][ C3] RDX: 00000000000003c5 RSI: 000000000d963022 RDI: 0000000000000005 [ 39.826051][ C3] RBP: 000000000d963022 R08: 00007fffb03d8520 R09: 0000000000000014 [ 39.826292][ C3] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 39.826621][ C3] R13: 00000000000003c5 R14: 00007fffb03d8520 R15: 0000000000000000 [ 39.826869][ C3]