Sign in

production environment just nuked my entire sync job... wtf??

backenddeploymenterrorspython2 replies · 6 views · started
Reply
  1. Viktor Duarte
    @siliconfiddler

    so i was just monitoring the logs and everything went completely sideways. the sync job was running fine all night, but suddenly the whole thing just collapsed and now the database is sitting there with half-finished records... im honestly losing my mind right now cuz i havent changed the core logic in weeks.

    PS C:\scripts> python sync_reports.py

    PS C:\scripts> python sync_reports.py
Traceback (most recent call last):
  File "C:\scripts\sync_reports.py", line 3, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
PS C:\scripts> pip install requests
Requirement already satisfied: requests in c:\users\...\python311\lib\site-packages
PS C:\scripts> python sync_reports.py
ModuleNotFoundError: No module named 'requests'

    hmmm, i suspect some kind of environment drift or maybe a ghost dependency got updated in the background? i checked the docker image hash and it looks the same, but the traceback is screaming about a missing module that definitely exists in my local dev container. i tried restarting the service and even nuking the cache but it keeps throwing the same garbage. is it possible someone pushed a change to the base image or is my networking config just acting up again?? i really dont want to have to manually roll back every single record... help lol.

    2
  2. Ibrahim Olsen
    @ibrahim_mutex

    check if your prod environment has restricted outbound access. if that missing module is a third-party lib, the runner might be failing to fetch a sub-dependency at runtime since the connection was blocked. also, check your env vars to make sure the pathing hasn't shifted in the prod container.

    1 0 helpful
  3. Kira Vos
    @kira_early
    in reply to @ibrahim_mutex

    i have to disagree with @ibrahim_mutex on teh runtime fetch theory. if your production environment is set up correctly, it shouldnt be reaching out to fetch dependencies during execution anyway. thats a huge red flag for your build process. id suspect a broken symlink or a volume mount issue in your docker config instead. if the pathing shifted, ur container is just looking at a ghost directory. 🙄 but hey ymmv

    8 2 helpful
Read next · Backend & APIs
nginx says 502, my app says it's fine. who's lying? !!
Solved · 6 replies
Read

Reply to this topic

Sign in or create an account to reply.