When It’s Hard… Do It More

As the buzz around “devops” starts to decline, we have a few lessons learned that we must not lose. My personal favorite “When it’s hard, do it more”. I stole from the DevOps Handbook. This is a must read and one of my all time favorite work-related books, if you haven’t already done so, read it!

When it’s hard do it more

Every time something unexpected happens during a planned (or unplanned) maintenance operation, I urge the team involved to do it again. Immediately. After there’s time to understand what happened but before the information gets stale. If you take an outage, make it a valuable one! Learn, as an individual how not to do that again. Learn, as a team by updating processes or adding additional logic to trap the error faster. Learn, as an organization by updating documentation and/or SOP’s around what was learned. Finally, do it again! It sounds crazy but the only way to get better is to practice. How else will you know if you’ve updated the process & documentation correctly?

Three areas off the top of my head that are hard, but critical you find the time to do more of.

Roll Credentials. Regularly.

Whether your focus is front end, back end, both, or other… Security is probably the most important thing we can all do for ourselves and for our organizations. Even with managed services or managed secrets (Azure Key Vault, AWS KMS), all organizations have keys that need to be rotated. Rolling these keys is our first line of defense against intruders, both internally and externally. Ideally, we have an automated process that rolls all keys [on demand | daily | weekly | whatever makes sense], but even if the process is all manual, that’s ok!! What’s important is having the process to roll keys in the first place.

Rolling keys should be done regularly. Everyone should be trained in how to roll keys and everyone practice rolling all keys they are responsible for. Rolling keys should be easy, until then… if it’s hard, do it more. It will make the process better, faster, stronger all while improving the security of the organization. With security, there are no second chances!

Write Good Tests

Writing good tests is an art. If you do not have a system set up to test code on commit & push, get it done! Jenkins is free, well-documented, runs on any OS, easy to setup and maintain in a small environment (less than 50 people) and quickly return the investment made to get it working. You will wonder how you ever functioned without it after you’re up and running.

All code should be paired, in some way with an automated test. Many times we’re testing our code in multiple ways (unit, functional, integration, regression, performance). I even see more test code getting tested in an automated fashion to ensure it’s functioning under the current assumptions. Regardless of the technology, the type of testing chosen, or the amount of testing performed, all code should be tested with every commit/push. There are millions of great articles around testing. You can find articles that are targeted for all experience levels and for all languages.

Writing good tests today prevent future you from breaking your stuff. It may be hard to start testing if you have never tested code before, but trust me when I say that it gets easier with practice, and of course, if it’s hard, do it more. 

Give Back

Leaving this bucket generic. I was originally going to call out writing good documentation around your code and processes. But after thinking about it, I’m adding the community to this section. We should all be blogging or contributing to open source projects regularly. Whatever you can spare to do your part to give back to the community. You may think no one wants to read your stuff, but you’d be wrong. If you hit a weird bug somewhere, even if you don’t have the solution, write it down somewhere. Someone may find on a late-night hasty Google search. You know the ones — page 5 of the results because you are desperate. You can contribute to open source projects without knowing how to code! Issues can be filed. Documentation can be updated. In an open source world, the littlest bit from a lot of people drives the entire ecosystem forward.

Writing good documentation is also another art form. Too much documentation and no one will read it. Too little documentation and no one will use your stuff. It takes practice to write good documentation and it requires a feedback cycle of asking the reader how you can improve. This applies to SOP‘s too. It’s never enough to write documentation once and never come back to it. It has to be maintained along with your code so that it’s up to date and trustworthy.

Whether it giving back to the community or it’s writing documentation around your code, processes, tests, whatever…. Give back thru better writing. Give back to yourself, to your team, to your organization, and to the community. It’s the community service of our time. I know that sometimes it’s not always easy to do or clear where to start BUT (for the final time)… if it’s hard, do it more. 


Thank you for reading! These opinions are mine and mine alone. I hope you liked the post, it was a little different than the typical technical, subject focused write up. If you have questions or want to get a hold of me leave a comment below or tweet me (@NickHudacin)!



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s