You are viewing a single comment's thread from:
RE: The Mystery hunt and an exhausting weekend.
This calls for a basic diagnostics suite to start investigations with :)
We have a tests suite running docker containers and it has become much much more useful ever since we created a bash script trying to do basic analysis of results and giving hints on what failures may mean.
Right now even when I don't have the confidence to automate writing tasks, I spend an extra day or two making scripts to dig around and gather data for me to make a decision. It's such a time saver!
I suspect there are tools that integrate with pcaps to analyse the traffic and detect suspicious symptoms.
Yeah.. well it was a confluence of events really. The fact that the UPS blew out really threw me for a loop because as far as I knew everything was working the night before. Turns out I was soooooooooooooo wrong but only after pulling my hair out at these really weird symptoms. How would you deal with a device that is on a VPN over the internet but not connected to any other devices including those plugged into the same switch.
I suspect that "I would ask ops for help" isn't a good answer to the last question ;)
Not when I am the ops, no ;) Still it is a good war story and I've learned a valuable lesson even after a decade + doing this sort of thing. You cannot take anything for granted no matter how obvious it looks.