IT Operations & Networking
Running systems and networks, at work or at home.
ugh. small config change, rolled out to k3s, and now the worker pod is looping: rollback would be easy but i want to understand what happened. restart counter keeps climbing and my on call phone is buzzing. The change was just a couple of environment…
not gonna lie this is driving me nuts third time this quarter, overnight, /var on our web server fills up and things fall over: I clean it, its fine for a few weeks, and then it happens again lmao I want to fix the root cause this time, not just mop teh…
help!! small office, 172.16.10.0/24 network. Since last week the conference room TV disconnects every few hours, and a nearby PC shows this: rebooting fixes it temporarily, nobody admits to changing anything. I suspect something has teh same IP, but how do I…
the marketing lead changed their password on Monday. Since then teh account locks every morning at login: We suspect something is still using the old password, but they have a laptop, a phone, a tablet and who knows what else. How do we narrow it down??…
im literally losing my mind here a service desk tech changed their password on Monday. since then the account locks every time they open Outlook: we suspect something is still using the old password, but they have a laptop, a phone, a tablet and who knows…
Help. The jump host is barely alive. SSH works, but every command complains about writing temp files. Here's the disk: i deleted the biggest log i could find and df didn't budge. am I losing my mind? im pretty suree Im missing something fundamental about how…
curious how others handled the move away from hardware security keys at a place like a school district. who got hardware keys, who stayed on app based approvals, and how did you handle lost devices? im also curious about what happens when someone loses their…
ugh. third time this quarter. after a holiday weekend, /var on our montioring box fills up and things fall over: i clean it, its fine for a few weeks, and then it happens again, i wanna fix the root cause this time, not just mop the floor. Every time, I find…
small office, 172.16.10.0/24 network. since last week a VoIP phone disconnects every few hours, and a nearby PC shows this: rebooting fixes it temporarily.. 🙏 nobody admits to changing anything. I suspect smth has the same IP, but how do I find out what? we…
ugh. were a agency running a handful of web apps 🔥 today its Docker Compose on two VMs, deployed by a script. A new hire is pushing hard for Kubernetes: self healing, rolling updates, industry standard. i worry well spend all our time running the platform…
Working from home today. WireGuard connects fine, teh status is green, but I can't reach the file share: browsing teh regular internet still works. my colleagues in the office are fine. ive reconnected five times, is it me or the VPN? My home setup is a…
new to Kubernetes, please be gentle. my database migration job on EKS wont stay up: the stauts says CrashLoopBackOff... i googled it and it seems to just mean it crashes alot.. :D thanks, Kubernetes... how do i find out WHY? I can see teh pod, I can see the…
im literally losing my mind here help. The jump host is barely alive. SSH works, but every command complains about writing temp files. Here's the disk: I deleted the biggest log I could find and df didn't budge. Am I losing my mind? I'm pretty suree I'm…
help!! help... the file server is barely alive. SSH works, but every command complains about wriitng temp files... heres teh disk: I deleted teh biggest log I could find and df didnt budge. am i losing my mind? im pretty suree Im missing smth fundamental…
guys pls help ive been tasked with moving it to a new host for an old sevrer, now running on VMware ESXi. everything looked fine until i powered it on: The virtual disk is attached and teh right size. The original machine booted fine yesterday. Where did my…
ok weird one my key works on every other server. on a new EC2 instance, from a CI job, nothing: A colleague set up teh account. they swear they pasted my key. I'm satrting to suspect the server, but I don't even know where to look. it's frustrating beacuse i…
Our patching is whenever someone remembers and entirely manual, which means some servers get skipped for months lol i want to fix this without breaking things. How do u structure rings, approvals and reboots?? theres also the reboot question xD some of these…
Setting up a Windows 11 laptop for a new colleague. It joins teh guest network, teh password is accepted, and then: they start tomorrow morning... i tried the troubleshooter, it fixed nothing... any ideas before i just give them my own laptop? the laptop is…
long time lurker first post lol question for endpoint people: were planning a migration with a few hundred laptops. co management works, but were running two systems. is that a stable long term state, or should we commit to one? some ppl on teh team are very…
ok weird one i wrote a simple unit file for our app... running teh command manually works perfectly. runnnig it as backup.service: systemctl status shows sattus=200/CHDIR and thats it. No useful message. Ive reloaded the daemon a dozen times. ive compared…
i raised the limit for the PHP worker to 768Mi after the first OOMKill. Then it happened again: im afraid to just keep doubling it... my team lead asked is it a leak? and i honestly dont know how to tell. i dont want to just throw more memory at the problem…
hmmmm ok our door access controllers at a dental practice get firmware updates maybe once in their lifetime. they sit next to computers with sensitive data. i wanna isolate them, but Im worried about discovery and casting breaking. whats your experience?…
guys pls help genuine question for people running a small cloud footprint: do you enable unattended upgrades in production, or does everything go through a manual window?? 🤷 we patch quarterly today and Im not suree if thats responsible or reckless. part of…
I keep hearing that push based deploys are 'insecure' because CI needs credentials to production. For three engineers running Kubernetes clusters, is that a real concern? What did you gain or lose when switching to pull based? Our auditors have started…
so annoyed rn home lab drama 😭 debian on my old Dell tower got a kernel update and now it panics during boot: it was running for months without a hiccup. is this hardware failing at teh worst possible time, or the update? I havent touched the hardware in…
