System Administration
Servers, identity, patching and day-to-day admin.
Fun problem! After a routine OS upgrade, a command line tool Ive used forever now does this on startup: nothing changed in the tool itself. how do i find out whether it's teh tool, a library, or the system? seriously need help here 😭
our patching is motnhly and entirely manual, which means some servers get skipped for months :D I want to fix this without breaking things. how do you structure rings, approvals and reboots? Theres also the reobot question. Some of these servers havent been…
from experience: Im on a two person IT team... when something breaks at night, whoevers on call often has to work things out from scratch because teh knowledge is in one persons head, weve talked about writing runbooks, but every time we try, they end up…
so annoyed rn our monitoring alerted that / is at 95% on a production VM. I freed some space by clearing the package cache, but its creeping back up. how do you methodically figure out what's growing? and are there…
not gonna lie this is driving me nuts home lab drama. Debian on my cloud instance got a kernel update and now it panics during boot: It was running for months without a hiccup. Is this hardware failing at the worst possible time, or the update? i haven't…
im literally losing my mind here our monitoring box started failing right after a backup run.. lmao services refuse to start, package installs fail, and monitoring shows: df h confirms /var is at 100%. I dont want to just rm things randomly on a production…
our metrics exporter.service keeps flapping. it starts, crashes, restarts, and finally systemd stops trying: i see status=217/USER in the status. The app logs to its own file, which stays empty, sooo it dies before it even gets going. Id like to understand…
not gonna lie this is driving me nuts i wrote a simple unit file for our app. Running the command manually works perfectly. Running it as myapp.service: systemctl status shows status=200/CHDIR and that's it. No useful message. I've reloaded teh daemon a…
ok weird one home lab drama. fedora on my cloud instance got a kernel update and now it panics during boot: it was running for months without a hiccup.. ;) is this hardware failing at teh worst possible time, or the update? i havent touched teh hardware in…
ugh. help. The database VM is barely alive. SSH works, but every command complains about writing temp files. Heres the disk: i deleted the biggest log i could find and df didnt budge... am i losing my mind? im pretty suree Im missing smth fundamental about…
myapp.service was fine until todays server patching. now it wont start: the status line says status=1/FAILURE 👍 i didn't touch the unit file. did teh updtae change smth underneath? the update touched a bunch of pacakges, sooo it could be anything xD ive…
help!! home lab drama. rocky Linux on my physical srever got a kernel update and now it panics during boot: it was running for months without a hiccup. is this hardware failing at teh worst possible time, or the update? I haven't touched the hardware in…
long time lurker first post lol home lab drama. AlmaLinux on my home lab box got a kernel update and now it panics during boot: It was running for months withuot a hiccup. Is this hardware failing at teh worst possible time, or teh update?? i havent touched…
im literally losing my mind here routine patching on our Fedora physical server. Updates installed without errors, I rebooted, and this is what greeted me on the console: the services on this box matter. i have console access but i've never dealt with a…
guys pls help i wrote a simple unit file for our app. Running teh command manually works perefctly. Running it as worker.service: systemctl status shows sattus=1/FAILURE and thats it... no useful message... ive relaoded the dameon a dozen times. Ive compared…
not gonna lie this is driving me nuts api.service was fine until todays server patching. now it wont start: The status line says status=200/CHDIR. i didnt touch the unit file. Did the update change smth underneath? the update touhced a bunch of packages, so…
ok so we look after a dozen on prem servers and patch monthly. management wants faster, the app owners want never on a weekday. im trying to desgin smth sane: staging rings, maintenance windows, auto reboots or not. what actually works for you in practice?…
so annoyed rn our pathcing is whenever someone rmeembers and entirely manual, which means some servers get skipped for months. i want to fix this without breaking things... how do u structure rings, approvals and reboots? theres also the reboot question xD…
applied updates on Debian and the reboot ended in this: It mentions not being able to mount the root filesystem. Nothing chagned with the disks... pls tell me the data is still there. the update included a new kernel, and as far as I can remember there was a…
not gonna lie this is driving me nuts third time this quarter, overnight, /var on our web server fills up and things fall over: I clean it, its fine for a few weeks, and then it happens again lmao I want to fix the root cause this time, not just mop teh…
Help. The jump host is barely alive. SSH works, but every command complains about writing temp files. Here's the disk: i deleted the biggest log i could find and df didn't budge. am I losing my mind? im pretty suree Im missing something fundamental about how…
ugh. third time this quarter. after a holiday weekend, /var on our montioring box fills up and things fall over: i clean it, its fine for a few weeks, and then it happens again, i wanna fix the root cause this time, not just mop the floor. Every time, I find…
im literally losing my mind here help. The jump host is barely alive. SSH works, but every command complains about writing temp files. Here's the disk: I deleted the biggest log I could find and df didn't budge. Am I losing my mind? I'm pretty suree I'm…
help!! help... the file server is barely alive. SSH works, but every command complains about wriitng temp files... heres teh disk: I deleted teh biggest log I could find and df didnt budge. am i losing my mind? im pretty suree Im missing smth fundamental…
Our patching is whenever someone remembers and entirely manual, which means some servers get skipped for months lol i want to fix this without breaking things. How do u structure rings, approvals and reboots?? theres also the reboot question xD some of these…
