Log Everything – do it for your support people.

I’ve worked on a lot of systems over the past 11 years.  I heard the same discussions.  I heard the same arguments.  I’ve heard the same excuses.

One the issues that always seems to be in contention is how much a web application should log about user activity.  My stand has always been “Log everything you can about what is going on.  The only reason you should cut back is due to storage limitation.”  For some reason, this always causes arguments with my fellow developers.  (It ranks up there with the statement “Our app should be cross-browser compatible.”)

One of my big problems with applications written by junior (and some senior) developers is that they don’t log enough.  On a development machine or server, logging is not really needed—you can set a break point or log onto a server to see what’s going on.  In a production environment with tight change controls, this is not possible.

I’ve looked after third-party software applications in large companies which would routinely fail.  The company would spend a fortune for a software package from a small start-up no one has heard of.  We get this software, install it on our servers and before long, we have our first exception.  Angry users call me.  I call the vendor:

“We are getting an error number x00012012928 when click on the save button.”

“Hmm. What happens when you click on the popup?”, he asks.

“The application crashes.  Now no one can log in.”

“Can you check to see if the server is still up?”, they ask.  This is when I glean that he doesn’t really know what’s going on.  He’ll ask for a reboot next.

“The server is up.  I can still get to the web pages. But no one can save anything.  Can I talk to the developer.”

“I am the developer” the snooty developer replies.

“Okay.  So why is it doing this?”

“Oh.  Umm. . . .  It shouldn’t happen.”  Yeah.  No sh*t.

“What is x00012012928?" I ask trying to move things along.  “It looks like an internally generated error number.  A description of the error would have been nice.  I’ve checked event viewer but see nothing.  It looks like the exception is being caught, but not handled.  What would generate this number?”

“It could be a number of things . . . “  This is the wrong answer.  If you are going to raise your own errors, be as specific as possible.  You can’t anticipate every error, but you can differentiate between db or web error for example.   This conversation goes on and on where eventually I need to get someone with server admin rights to send him an IIS log so he can review it for a few days.  Meanwhile, I have to try to calm down an angry user base and continue to chase the vendor.

I’ve been in the above situation at least a half-dozen times in different companies.  It most definitely is not an exaggeration.

I like to contrast this with a friend of mine who wrote an application where he logged everything.  He kept log files as text files in a web root where he could access them securely with .htaccess rights.  Every day, a new file was generated and files over 30 days old were deleted. 

One day, he got a call from an irate user.

“I’m trying to use this rubbish application and it isn’t working.” angry user yells.

“I see. What is your username?” the developer calmly asks.

“John_Smith”

“I see you logged into the application 3 times today.  Did you get an error every time or just the last time.”  Already the developer with logged activity at his disposal knows more about the user’s problem than the user knows.  If there was an exception, he would have been able to see it in his logs.  If there is no exception listed, he can view user activity.  “I see the last page you accessed was the transaction details page.  Is this where you get the error?  Can you describe it?”

“The button does not show up to save the data I entered,” the calmer user explains.

“I see.  This is a common problem.  You left a mandatory field out.  You should see it highlighted in red.”

“Oh yes.  I see it now.  How did I miss that?  Thanks.”

So the developer had 3 things working for him:

1.  He had more information at his disposal than the user had.  He could identify an exception or user error.

2.  He did not try to make the user feel at fault when he clearly was.  (“This is a common problem” may necessarily be true, but there is no better way to put someone on the defensive than to accuse them of being wrong)

3.  He spoke in a calm, unassuming, and sympathetic voice.  A user can quickly tell whether you are doing everything you can to help him or if he has to try to prove to you that his problem is real.

So, I’m always in favour of extreme logging.  I hear the same excuses every time why this can’t be done.  “It will take up too much space.”  “It will be too difficult to clear down.” “Nothing should go wrong with the application, because we tested it.” “We will log the exceptions.  That’s all we need.”