My grandfather used to tell the following joke -
One of the people living in our town would wake up every morning, go out on his front porch, and let out a blood curdling scream. ?When asked why he did that he said that when he was in Africa the native guide did this every morning to keep elephants away. ?My grandfather said "But there are no elephants within 100 miles of here". ?His friend responded "See how well it works?"
I was reminded of this story when working with a customer who was interested in a particularly complex configuration in their production cluster to ensure that there was no data loss in a node failure scenario. ?However when asked if they had any tests that would show the problem they were concerned about, they did not have one, nor had they experienced the data loss they were trying to avoid.
My main question was "So, after you've gone through this complex configuration and have it in place, how do you prove it is going to work?" ?Basically they were keeping the elephants away. ?Unless you have a test that shows you can experience the problem a solution is designed to solve, you can't really prove your solution has made the situation any better, nor can you tell the business users it was worth the effort.
This goes to a fundamental rule I tell our team all the time. ?The first step in fixing a problem is to ensure you can create a test that shows the problem, or in some way measure the problem. ?Then implement the fix. ?Finally prove the test is working. ?This is true for technical issues as well as process improvement.
Doing anything else is just yelling in order to keep elephants away.