cover photo

muppeth

muppeth@hub.disroot.org

Slowly regaining control.

  
Last few days I've closed myself at home and worked on all the pending things to re-gain control over the servers I take care of (personal, disroot, work, etc). In recent months things became a chaotic mess where automation scripts were no longer applied due to undocumented custom changes here and there, some procedures became complex and manual work due to neglect and manual changes etc.

Basically everything became a mess that created situation where I wouldnt touch certain things anymore, or would postpone things forever because instead of simple script to run it, it would require a lot of manual additions and felt like too much to focus on and do.

However, last days brought some stability. I've updated some of the crutial and basic ansible roles, and re-wrote some scripts. I added some extra templates to my zabbix monitoring to have better overview and generally started documenting the work. Its still far from perfect, but i alreaqdy see improvements, I hope by the end of next week, the total control will be regained.
Living on the edge

  
Some people, like to take drugs, base jump, parachute, drive fast cars...

I like to every now and then break production servers :)

During a mariadb update for some reason a database got corrupt, badly. I had to force_recovery to 4, dump all dbs, reinstall and recover all of them. Before that, because I dont like restoring and rather want to fix the issue I spent some time trying different things which failed. Out of stress I realized I'm also restoring test databases (which are even bigger then production ones for not reason as i dont even use them) just to make the process take a bit longer. I feel like I came back home from a good rave and the caffeine is still keeping me awake.

That said I now know what to do so I wont panic next time innodb corruption occurs on my watch.

Image/photo
  
Dev in Prod and skip the tests...?

Sounds like a great idea to safe development time! ;)
  
Sounds like you guyz need a #holiday
<3 !SkyEarth
  
Image/photo