Location: Jitsi
Date and Time: Wednesday, February 9, 2022, 5:30pm-8:00pm
Scribe: You
Roll Call: Them
Agenda
- Pick location for next weekly
- Dell R630:
- Debian installed
- VMs copied from magenta
- VMs started and recovered from brokenness
- larger disks are ... slow, I think all three RAID1s are composed of 5400 rpm disks, the 4TB pair seem to stall periodically, about 60MB/s is typical write speed.
- One of the VMs experienced some stalls:
[Sun Feb 6 16:54:42 2022] rcu: INFO: rcu_sched self-detected stall on CPU [Sun Feb 6 16:54:42 2022] rcu: 1-...0: (1 ticks this GP) idle=376/1/0x4000000000000000 softirq=2783997/2783997 fqs=16 [Sun Feb 6 16:54:42 2022] (t=6597 jiffies g=5323225 q=2310) [Sun Feb 6 16:54:42 2022] NMI backtrace for cpu 1 [Sun Feb 6 16:54:42 2022] CPU: 1 PID: 524145 Comm: kworker/1:2 Tainted: G L 5.10.0-11-amd64 #1 Debian 5.10.92-1 [Sun Feb 6 16:54:42 2022] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014 [Sun Feb 6 16:54:42 2022] Workqueue: events drm_fb_helper_dirty_work [drm_kms_helper] [Sun Feb 6 16:54:42 2022] Call Trace: [Sun Feb 6 16:54:42 2022] <IRQ> [Sun Feb 6 16:54:42 2022] dump_stack+0x6b/0x83 [Sun Feb 6 16:54:42 2022] nmi_cpu_backtrace.cold+0x32/0x69 [Sun Feb 6 16:54:42 2022] ? lapic_can_unplug_cpu+0x80/0x80 [Sun Feb 6 16:54:42 2022] nmi_trigger_cpumask_backtrace+0xd7/0xe0 [Sun Feb 6 16:54:42 2022] rcu_dump_cpu_stacks+0xa2/0xd0 [Sun Feb 6 16:54:42 2022] rcu_sched_clock_irq.cold+0x1ff/0x3d6 [Sun Feb 6 16:54:42 2022] update_process_times+0x8c/0xc0 [Sun Feb 6 16:54:42 2022] tick_sched_handle+0x22/0x60 [Sun Feb 6 16:54:42 2022] tick_sched_timer+0x7c/0xb0 [Sun Feb 6 16:54:42 2022] ? tick_do_update_jiffies64.part.0+0xc0/0xc0 [Sun Feb 6 16:54:42 2022] __hrtimer_run_queues+0x12a/0x270 [Sun Feb 6 16:54:42 2022] hrtimer_interrupt+0x110/0x2c0 [Sun Feb 6 16:54:42 2022] __sysvec_apic_timer_interrupt+0x5f/0xd0 [Sun Feb 6 16:54:42 2022] asm_call_irq_on_stack+0x12/0x20 [Sun Feb 6 16:54:42 2022] </IRQ> [Sun Feb 6 16:54:42 2022] sysvec_apic_timer_interrupt+0x72/0x80 [Sun Feb 6 16:54:42 2022] asm_sysvec_apic_timer_interrupt+0x12/0x20 [Sun Feb 6 16:54:42 2022] RIP: 0010:_raw_spin_unlock_irqrestore+0x11/0x20 [Sun Feb 6 16:54:42 2022] Code: e0 4c 39 f0 76 d8 80 4d 00 04 eb 92 cc cc cc cc cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 07 [Sun Feb 6 16:54:42 2022] RSP: 0018:ffffad65404cbc80 EFLAGS: 00000246 [Sun Feb 6 16:54:42 2022] RAX: 0000000000000000 RBX: ffff989c06c45100 RCX: 0000000000000000 [Sun Feb 6 16:54:42 2022] RDX: 000000000000c040 RSI: 0000000000000246 RDI: 0000000000000246 [Sun Feb 6 16:54:42 2022] RBP: ffffad65402a700c R08: 0000000000000300 R09: ffff989c03948e20 [Sun Feb 6 16:54:42 2022] R10: ffff989c029e3980 R11: ffff989d228e5000 R12: ffff989c06c45120 [Sun Feb 6 16:54:42 2022] R13: ffffad65404cbcf8 R14: 0000000000000246 R15: 0000000000000000 [Sun Feb 6 16:54:42 2022] qxl_ring_push+0x8e/0x290 [qxl] [Sun Feb 6 16:54:42 2022] ? ttm_bo_kunmap+0x9f/0xb0 [ttm] [Sun Feb 6 16:54:42 2022] ? ttm_bo_add_mem_to_lru+0x75/0x100 [ttm] [Sun Feb 6 16:54:42 2022] qxl_push_command_ring_release+0x64/0x80 [qxl] [Sun Feb 6 16:54:42 2022] qxl_draw_dirty_fb+0x425/0x450 [qxl] [Sun Feb 6 16:54:42 2022] qxl_framebuffer_surface_dirty+0xf5/0x1c0 [qxl] [Sun Feb 6 16:54:42 2022] drm_fb_helper_dirty_work+0x175/0x1b0 [drm_kms_helper] [Sun Feb 6 16:54:42 2022] process_one_work+0x1b6/0x350 [Sun Feb 6 16:54:42 2022] worker_thread+0x53/0x3e0 [Sun Feb 6 16:54:42 2022] ? process_one_work+0x350/0x350 [Sun Feb 6 16:54:42 2022] kthread+0x11b/0x140 [Sun Feb 6 16:54:42 2022] ? __kthread_bind_mask+0x60/0x60 [Sun Feb 6 16:54:42 2022] ret_from_fork+0x22/0x30
- Noticed that portlandfiber.com, which we host, was busted. Jason recovered a mysql database (or several) and got it working again.
- Node notes:
NodeStreetRoots -- no progress
NodeCascadia -- no progress
Your Item Here
Notes
- None Yet