[ 250.959971][ C3] TCP: AO hash mismatch for 10.0.254.1.7011->10.0.1.1.37998 [FP.] L3index: 0
[ 250.960882][ C2] TCP: AO hash mismatch for 10.0.1.1.37998->10.0.254.1.7011 [FP.] L3index: 0
[ 251.411203][ T71]
[ 251.411387][ T71] =====================================================
[ 251.411647][ T71] WARNING: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected
[ 251.411986][ T71] 6.9.0-rc1-virtme #1 Not tainted
[ 251.412214][ T71] -----------------------------------------------------
[ 251.412533][ T71] kworker/u16:1/71 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
[ 251.412837][ T71] ffff888005182c28 (&p->alloc_lock){+.+.}-{2:2}, at: __get_task_comm+0x27/0x70
[ 251.413214][ T71]
[ 251.413214][ T71] and this task is already holding:
[ 251.413527][ T71] ffff88802f83efd8 (&ul->lock){+.-.}-{2:2}, at: rt6_uncached_list_flush_dev+0x138/0x840
[ 251.413887][ T71] which would create a new lock dependency:
[ 251.414153][ T71] (&ul->lock){+.-.}-{2:2} -> (&p->alloc_lock){+.+.}-{2:2}
[ 251.414464][ T71]
[ 251.414464][ T71] but this new dependency connects a SOFTIRQ-irq-safe lock:
[ 251.414808][ T71] (&ul->lock){+.-.}-{2:2}
[ 251.414814][ T71]
[ 251.414814][ T71] ... which became SOFTIRQ-irq-safe at:
[ 251.415323][ T71] __lock_acquire+0x797/0x1570
[ 251.415534][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.415728][ T71] _raw_spin_lock_bh+0x38/0x50
[ 251.415951][ T71] rt6_uncached_list_del+0x77/0x200
[ 251.416146][ T71] ip6_dst_destroy+0x74/0x1e0
[ 251.416349][ T71] dst_destroy+0xba/0x310
[ 251.416531][ T71] rcu_do_batch+0x3c0/0xfb0
[ 251.416665][ T71] rcu_core+0x2be/0x500
[ 251.416787][ T71] __do_softirq+0x1f8/0x5df
[ 251.416925][ T71] irq_exit_rcu+0x97/0xc0
[ 251.417095][ T71] sysvec_apic_timer_interrupt+0x75/0x80
[ 251.417235][ T71] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 251.417421][ T71] default_idle+0xf/0x20
[ 251.417550][ T71] default_idle_call+0x6d/0xb0
[ 251.417762][ T71] cpuidle_idle_call+0x1f4/0x280
[ 251.417963][ T71] do_idle+0xf9/0x160
[ 251.418064][ T71] cpu_startup_entry+0x54/0x60
[ 251.418212][ T71] rest_init+0x14f/0x260
[ 251.418317][ T71] start_kernel+0x318/0x3d0
[ 251.418518][ T71] x86_64_start_reservations+0x18/0x30
[ 251.418722][ T71] x86_64_start_kernel+0xba/0x110
[ 251.418931][ T71] common_startup_64+0x12c/0x138
[ 251.419119][ T71]
[ 251.419119][ T71] to a SOFTIRQ-irq-unsafe lock:
[ 251.419292][ T71] (&p->alloc_lock){+.+.}-{2:2}
[ 251.419300][ T71]
[ 251.419300][ T71] ... which became SOFTIRQ-irq-unsafe at:
[ 251.419723][ T71] ...
[ 251.419725][ T71] __lock_acquire+0x797/0x1570
[ 251.420033][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.420244][ T71] _raw_spin_lock+0x30/0x40
[ 251.420448][ T71] set_mems_allowed+0x1d/0x210
[ 251.420650][ T71] kernel_init_freeable+0x72/0x310
[ 251.420850][ T71] kernel_init+0x20/0x200
[ 251.421052][ T71] ret_from_fork+0x31/0x70
[ 251.421262][ T71] ret_from_fork_asm+0x1a/0x30
[ 251.421469][ T71]
[ 251.421469][ T71] other info that might help us debug this:
[ 251.421469][ T71]
[ 251.421853][ T71] Possible interrupt unsafe locking scenario:
[ 251.421853][ T71]
[ 251.422164][ T71] CPU0 CPU1
[ 251.422365][ T71] ---- ----
[ 251.422573][ T71] lock(&p->alloc_lock);
[ 251.422732][ T71] local_irq_disable();
[ 251.422984][ T71] lock(&ul->lock);
[ 251.423250][ T71] lock(&p->alloc_lock);
[ 251.423523][ T71] <Interrupt>
[ 251.423691][ T71] lock(&ul->lock);
[ 251.423848][ T71]
[ 251.423848][ T71] *** DEADLOCK ***
[ 251.423848][ T71]
[ 251.424309][ T71] 5 locks held by kworker/u16:1/71:
[ 251.424464][ T71] #0: ffff8880018f2948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0xda6/0x1730
[ 251.424759][ T71] #1: ffffc900004cfda0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xde2/0x1730
[ 251.425008][ T71] #2: ffffffffbcb79890 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xbc/0xa20
[ 251.425254][ T71] #3: ffffffffbcb94cc8 (rtnl_mutex){+.+.}-{3:3}, at: default_device_exit_batch+0x81/0x2d0
[ 251.425533][ T71] #4: ffff88802f83efd8 (&ul->lock){+.-.}-{2:2}, at: rt6_uncached_list_flush_dev+0x138/0x840
[ 251.425815][ T71]
[ 251.425815][ T71] the dependencies between SOFTIRQ-irq-safe lock and the holding lock:
[ 251.426090][ T71] -> (&ul->lock){+.-.}-{2:2} {
[ 251.426232][ T71] HARDIRQ-ON-W at:
[ 251.426341][ T71] __lock_acquire+0x797/0x1570
[ 251.426525][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.426704][ T71] _raw_spin_lock_bh+0x38/0x50
[ 251.426881][ T71] rt6_uncached_list_add+0x55/0x1b0
[ 251.427060][ T71] icmp6_dst_alloc+0x2c0/0x450
[ 251.427235][ T71] mld_sendpack+0x529/0xed0
[ 251.427425][ T71] mld_ifc_work+0x32/0x1f0
[ 251.427599][ T71] process_one_work+0xe2c/0x1730
[ 251.427809][ T71] worker_thread+0x587/0xd30
[ 251.428007][ T71] kthread+0x28a/0x350
[ 251.428167][ T71] ret_from_fork+0x31/0x70
[ 251.428342][ T71] ret_from_fork_asm+0x1a/0x30
[ 251.428520][ T71] IN-SOFTIRQ-W at:
[ 251.428621][ T71] __lock_acquire+0x797/0x1570
[ 251.428793][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.428964][ T71] _raw_spin_lock_bh+0x38/0x50
[ 251.429131][ T71] rt6_uncached_list_del+0x77/0x200
[ 251.429299][ T71] ip6_dst_destroy+0x74/0x1e0
[ 251.429475][ T71] dst_destroy+0xba/0x310
[ 251.429648][ T71] rcu_do_batch+0x3c0/0xfb0
[ 251.429817][ T71] rcu_core+0x2be/0x500
[ 251.429985][ T71] __do_softirq+0x1f8/0x5df
[ 251.430155][ T71] irq_exit_rcu+0x97/0xc0
[ 251.430324][ T71] sysvec_apic_timer_interrupt+0x75/0x80
[ 251.430554][ T71] asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 251.430781][ T71] default_idle+0xf/0x20
[ 251.430969][ T71] default_idle_call+0x6d/0xb0
[ 251.431153][ T71] cpuidle_idle_call+0x1f4/0x280
[ 251.431336][ T71] do_idle+0xf9/0x160
[ 251.431477][ T71] cpu_startup_entry+0x54/0x60
[ 251.431645][ T71] rest_init+0x14f/0x260
[ 251.431818][ T71] start_kernel+0x318/0x3d0
[ 251.431989][ T71] x86_64_start_reservations+0x18/0x30
[ 251.432158][ T71] x86_64_start_kernel+0xba/0x110
[ 251.432329][ T71] common_startup_64+0x12c/0x138
[ 251.432501][ T71] INITIAL USE at:
[ 251.432603][ T71] __lock_acquire+0x797/0x1570
[ 251.432775][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.432949][ T71] _raw_spin_lock_bh+0x38/0x50
[ 251.433124][ T71] rt6_uncached_list_add+0x55/0x1b0
[ 251.433291][ T71] icmp6_dst_alloc+0x2c0/0x450
[ 251.433477][ T71] mld_sendpack+0x529/0xed0
[ 251.433653][ T71] mld_ifc_work+0x32/0x1f0
[ 251.433839][ T71] process_one_work+0xe2c/0x1730
[ 251.434051][ T71] worker_thread+0x587/0xd30
[ 251.434219][ T71] kthread+0x28a/0x350
[ 251.434372][ T71] ret_from_fork+0x31/0x70
[ 251.434564][ T71] ret_from_fork_asm+0x1a/0x30
[ 251.434732][ T71] }
[ 251.434804][ T71] ... key at: [<ffffffffbec28cc0>] __key.16+0x0/0x40
[ 251.435014][ T71]
[ 251.435014][ T71] the dependencies between the lock to be acquired
[ 251.435015][ T71] and SOFTIRQ-irq-unsafe lock:
[ 251.435382][ T71] -> (&p->alloc_lock){+.+.}-{2:2} {
[ 251.435531][ T71] HARDIRQ-ON-W at:
[ 251.435636][ T71] __lock_acquire+0x797/0x1570
[ 251.435814][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.435987][ T71] _raw_spin_lock+0x30/0x40
[ 251.436158][ T71] set_mems_allowed+0x1d/0x210
[ 251.436348][ T71] kernel_init_freeable+0x72/0x310
[ 251.436517][ T71] kernel_init+0x20/0x200
[ 251.436694][ T71] ret_from_fork+0x31/0x70
[ 251.436875][ T71] ret_from_fork_asm+0x1a/0x30
[ 251.437043][ T71] SOFTIRQ-ON-W at:
[ 251.437147][ T71] __lock_acquire+0x797/0x1570
[ 251.437316][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.437488][ T71] _raw_spin_lock+0x30/0x40
[ 251.437658][ T71] set_mems_allowed+0x1d/0x210
[ 251.437826][ T71] kernel_init_freeable+0x72/0x310
[ 251.437995][ T71] kernel_init+0x20/0x200
[ 251.438163][ T71] ret_from_fork+0x31/0x70
[ 251.438330][ T71] ret_from_fork_asm+0x1a/0x30
[ 251.438504][ T71] INITIAL USE at:
[ 251.438608][ T71] __lock_acquire+0x797/0x1570
[ 251.438778][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.438965][ T71] _raw_spin_lock+0x30/0x40
[ 251.439135][ T71] set_mems_allowed+0x1d/0x210
[ 251.439307][ T71] kernel_init_freeable+0x72/0x310
[ 251.439480][ T71] kernel_init+0x20/0x200
[ 251.439656][ T71] ret_from_fork+0x31/0x70
[ 251.439825][ T71] ret_from_fork_asm+0x1a/0x30
[ 251.439998][ T71] }
[ 251.440067][ T71] ... key at: [<ffffffffbdbd0d20>] __key.387+0x0/0x40
[ 251.440277][ T71] ... acquired at:
[ 251.440383][ T71] check_prev_add+0x194/0xce0
[ 251.440524][ T71] validate_chain+0x71d/0x9b0
[ 251.440663][ T71] __lock_acquire+0xaf0/0x1570
[ 251.440798][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.440937][ T71] _raw_spin_lock+0x30/0x40
[ 251.441073][ T71] __get_task_comm+0x27/0x70
[ 251.441212][ T71] ref_tracker_alloc+0x2ee/0x490
[ 251.441350][ T71] rt6_uncached_list_flush_dev+0x494/0x840
[ 251.441602][ T71] rt6_disable_ip+0xd9/0x140
[ 251.441741][ T71] addrconf_ifdown.isra.0+0x103/0x1560
[ 251.441879][ T71] addrconf_notify+0xd1/0xd60
[ 251.442025][ T71] notifier_call_chain+0xcd/0x150
[ 251.442169][ T71] dev_close_many+0x2d8/0x650
[ 251.442304][ T71] unregister_netdevice_many_notify+0x3d5/0x1190
[ 251.442491][ T71] default_device_exit_batch+0x234/0x2d0
[ 251.442659][ T71] cleanup_net+0x4f3/0xa20
[ 251.442798][ T71] process_one_work+0xe2c/0x1730
[ 251.442937][ T71] worker_thread+0x587/0xd30
[ 251.443072][ T71] kthread+0x28a/0x350
[ 251.443180][ T71] ret_from_fork+0x31/0x70
[ 251.443419][ T71] ret_from_fork_asm+0x1a/0x30
[ 251.443557][ T71]
[ 251.443631][ T71]
[ 251.443631][ T71] stack backtrace:
[ 251.443800][ T71] CPU: 1 PID: 71 Comm: kworker/u16:1 Not tainted 6.9.0-rc1-virtme #1
[ 251.444218][ T71] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 251.444515][ T71] Workqueue: netns cleanup_net
[ 251.444665][ T71] Call Trace:
[ 251.444848][ T71] <TASK>
[ 251.444924][ T71] dump_stack_lvl+0x82/0xd0
[ 251.445068][ T71] print_bad_irq_dependency+0x957/0xd40
[ 251.445207][ T71] ? __pfx_print_bad_irq_dependency+0x10/0x10
[ 251.445376][ T71] ? __pfx_usage_skip+0x10/0x10
[ 251.445595][ T71] ? __bfs+0x246/0x650
[ 251.445696][ T71] ? __pfx_usage_match+0x10/0x10
[ 251.445829][ T71] check_irq_usage+0x596/0x850
[ 251.445974][ T71] ? arch_stack_walk+0x8c/0xf0
[ 251.446111][ T71] ? __pfx_check_irq_usage+0x10/0x10
[ 251.446245][ T71] ? __bfs+0x246/0x650
[ 251.446346][ T71] ? __pfx_hlock_conflict+0x10/0x10
[ 251.446483][ T71] ? check_path.constprop.0+0x24/0x50
[ 251.446617][ T71] ? check_noncircular+0x14c/0x3e0
[ 251.446755][ T71] ? __pfx_check_noncircular+0x10/0x10
[ 251.446893][ T71] ? check_prev_add+0x194/0xce0
[ 251.447028][ T71] check_prev_add+0x194/0xce0
[ 251.447163][ T71] ? validate_chain+0x130/0x9b0
[ 251.447374][ T71] validate_chain+0x71d/0x9b0
[ 251.447511][ T71] ? mark_lock+0x38/0x3e0
[ 251.447617][ T71] ? __pfx_validate_chain+0x10/0x10
[ 251.447754][ T71] ? hlock_class+0x4e/0x130
[ 251.447887][ T71] ? mark_lock+0x38/0x3e0
[ 251.448073][ T71] __lock_acquire+0xaf0/0x1570
[ 251.448209][ T71] lock_acquire.part.0+0xeb/0x330
[ 251.448351][ T71] ? __get_task_comm+0x27/0x70
[ 251.448489][ T71] ? mark_lock+0x38/0x3e0
[ 251.448666][ T71] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 251.448809][ T71] ? trace_lock_acquire+0x135/0x1c0
[ 251.448944][ T71] ? __get_task_comm+0x27/0x70
[ 251.449088][ T71] ? lock_acquire+0x32/0xc0
[ 251.449228][ T71] ? __get_task_comm+0x27/0x70
[ 251.449438][ T71] _raw_spin_lock+0x30/0x40
[ 251.449580][ T71] ? __get_task_comm+0x27/0x70
[ 251.449721][ T71] __get_task_comm+0x27/0x70
[ 251.449859][ T71] ref_tracker_alloc+0x2ee/0x490
[ 251.450077][ T71] ? __pfx_ref_tracker_alloc+0x10/0x10
[ 251.450215][ T71] ? __lock_release+0x103/0x460
[ 251.450348][ T71] ? rt6_uncached_list_flush_dev+0x494/0x840
[ 251.450523][ T71] ? rt6_disable_ip+0xd9/0x140
[ 251.450735][ T71] ? addrconf_ifdown.isra.0+0x103/0x1560
[ 251.450874][ T71] ? addrconf_notify+0xd1/0xd60
[ 251.451012][ T71] ? notifier_call_chain+0xcd/0x150
[ 251.451150][ T71] ? dev_close_many+0x2d8/0x650
[ 251.451362][ T71] ? unregister_netdevice_many_notify+0x3d5/0x1190
[ 251.451536][ T71] ? default_device_exit_batch+0x234/0x2d0
[ 251.451706][ T71] ? cleanup_net+0x4f3/0xa20
[ 251.451840][ T71] ? process_one_work+0xe2c/0x1730
[ 251.452058][ T71] ? worker_thread+0x587/0xd30
[ 251.452196][ T71] ? kthread+0x28a/0x350
[ 251.452299][ T71] ? ret_from_fork+0x31/0x70
[ 251.452433][ T71] ? ret_from_fork_asm+0x1a/0x30
[ 251.452650][ T71] ? in6_dev_get+0xcc/0x210
[ 251.452788][ T71] rt6_uncached_list_flush_dev+0x494/0x840
[ 251.452963][ T71] rt6_disable_ip+0xd9/0x140
[ 251.453099][ T71] ? __pfx_rt6_disable_ip+0x10/0x10
[ 251.453311][ T71] ? mark_lock+0x38/0x3e0
[ 251.453416][ T71] addrconf_ifdown.isra.0+0x103/0x1560
[ 251.453568][ T71] ? __try_to_del_timer_sync+0xe7/0x140
[ 251.453704][ T71] ? __pfx_addrconf_ifdown.isra.0+0x10/0x10
[ 251.453875][ T71] addrconf_notify+0xd1/0xd60
[ 251.454103][ T71] ? fib_netdev_event+0x186/0x5a0
[ 251.454241][ T71] notifier_call_chain+0xcd/0x150
[ 251.454376][ T71] dev_close_many+0x2d8/0x650
[ 251.454514][ T71] ? __pfx_dev_close_many+0x10/0x10
[ 251.454803][ T71] ? default_device_exit_batch+0x81/0x2d0
[ 251.454942][ T71] ? trace_contention_end+0xdc/0x140
[ 251.455082][ T71] unregister_netdevice_many_notify+0x3d5/0x1190
[ 251.455255][ T71] ? mutex_is_locked+0x17/0x50
[ 251.455467][ T71] ? __pfx_unregister_netdevice_many_notify+0x10/0x10
[ 251.455643][ T71] ? __pfx_unregister_netdevice_queue+0x10/0x10
[ 251.455814][ T71] default_device_exit_batch+0x234/0x2d0
[ 251.455954][ T71] ? __pfx_default_device_exit_batch+0x10/0x10
[ 251.456215][ T71] ? mutex_is_locked+0x17/0x50
[ 251.456354][ T71] ? nexthop_net_exit_batch_rtnl+0x83/0x210
[ 251.456526][ T71] cleanup_net+0x4f3/0xa20
[ 251.456741][ T71] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 251.456874][ T71] ? __pfx_cleanup_net+0x10/0x10
[ 251.457037][ T71] ? trace_lock_acquire+0x135/0x1c0
[ 251.457189][ T71] ? process_one_work+0xde2/0x1730
[ 251.457337][ T71] ? lock_acquire+0x32/0xc0
[ 251.457569][ T71] ? process_one_work+0xde2/0x1730
[ 251.457726][ T71] process_one_work+0xe2c/0x1730
[ 251.457880][ T71] ? __pfx___lock_release+0x10/0x10
[ 251.458032][ T71] ? __pfx_process_one_work+0x10/0x10
[ 251.458262][ T71] ? assign_work+0x16c/0x240
[ 251.458413][ T71] worker_thread+0x587/0xd30
[ 251.458566][ T71] ? lockdep_hardirqs_on_prepare.part.0+0x1af/0x370
[ 251.458760][ T71] ? __pfx_worker_thread+0x10/0x10
[ 251.458992][ T71] ? __pfx_worker_thread+0x10/0x10
[ 251.459143][ T71] kthread+0x28a/0x350
[ 251.459258][ T71] ? __pfx_kthread+0x10/0x10
[ 251.459412][ T71] ret_from_fork+0x31/0x70
[ 251.459641][ T71] ? __pfx_kthread+0x10/0x10
[ 251.459794][ T71] ret_from_fork_asm+0x1a/0x30
[ 251.459950][ T71] </TASK>