That’s how long it took me to fix the problem I caused, after crashing the production web servers for a $100M+ multi-national corporation.
And this wasn’t just a marketing site, mind you. This was *the* web presence, online catalog, parts information and primary means of contact for distributors and representatives for several of the largest and most well known brands in that company’s industry. And I crashed every last website the company had, for every brand, for 2 hours.
It’s the year 2002. Microsoft had released their brand new .NET framework, Visual Studio and Visual Basic.NET systems. I dove in head first, setting aside my “classic” ASP (VBScript) and VB6/COM components. In a few short months, I had my first ASP.NET / VB.NET application ready to roll. It was the e-catalog for the company’s distributor and representative network. It replaced a manual upload and link process with a completely automated system that allowed product managers to log in and update the categories, products and PDF files on their own.
Less Than 30 minutes after deploying the application to the production servers, the entire suite of web sites that were hosted on those boxes were down and I was getting panicked calls from product managers and customer service reps. My boss was also getting calls from division and brand presidents / CEOs. It wasn’t a good day for me.
Those Pesky “return” Statements
I was panicked. Full on, heart attack, stomach dropping panic.
I turned to the internet to research the problem and help me solve my problem. Why were my database connections being eaten up? Why weren’t they closing, or recycling in the connection pool the way they were supposed to? I have my database connection “.close()” method right there, just like I always did in my classic ASP pages… right there, after my return statements, where I send the data back out to the page from the data access code.
Yeah, you can probably guess from that last line, that this is what my code looked like:
Oh, the horror when after nearly an hour and a half of searching, I found out that the “return” statement in VB.NET immediately exits the function where it is. How stupid is that? VB6 didn’t do it that way. VBScript (at the time) didn’t do it that way. Why would they change that in VB.NET?! I was horrified! But I was also under the gun to fix it, so I found every database connection “.close()” call and moved them one line above the return statement.
I deployed. It worked. The sites came back up and I slowly started to breathe.
And then my boss called me in to his office…
I sat down and looked at my boss, door closed behind us. I was ready to get fired.
He sat there and looked at me for a moment before asking if the sites were back up. Then he told me how many angry phone calls the customer service reps had received, how many bids we had lost because our reps and distributors couldn’t get the information they needed, and he gave an estimate of the money that we lost in the down time. it was more money than I would make in the next 2 months of work.
Then he asked me what I did wrong and what I learned from the problems and solutions. He wanted to know if I was going to make that mistake again and how I could avoid similar problems in the future.
Lastly – and quite shockingly to me – he told me how the CEO of the company was known to make mistakes of that caliber … every day… and that, while I had caused a lot of damage for a short period of time, it would be washed out by the end of the month.
The important lesson, he told me, was that I owned up to the mistake, dug in and fixed it, and learned how to avoid the problem in the future. I was, in his mind, a better developer at the end of that day. I had survived a catastrophic crash of my own making and I had fixed the problem, learning some very valuable lessons in business down time and in code that day.
Fail Your Way To Success
I’ll never forget that lesson. I was 23 years old, running the public web sites for multiple brands inside of a $100M+ manufacturing company. And my boss taught that that it’s ok to make mistakes – even big ones – as long as we can recover from them and learn from them
He taught me that we don’t need to be afraid of making mistakes. Rather, we need to be afraid of not learning from them.