Episode 23: My New Nuclear Power Plant

Below is the graph of how many clicks we were sending to merchants (again Y axis removed for competitive reasons).  It seems rosy and positive.  However for most of Q1, I felt anything but rosy.

Each evening I’d come home and my wife would try and guess from my frown or my smile if it had been a good revenue day at Like.com.  She would say, “Had a good revenue day?”  I’d say, “Yes”.  She would say, “What caused it.”  I would respond, “I don’t know.”  On a bad day the whole conversation was the same with only one twist.  “Had a good revenue day?”  “No”.  “What caused it.” “I don’t know.” “How are you going to fix it?”  “I don’t know because I don’t know the cause.” “Grrrrrr.”

Launching a new site is like becoming the owner of a brand Russian nuclear power plant.  You have a ton of dials with labels you can’t read.  The only dial you can read is the amount of electricity (in our case revenue) and the core temperature of the nuclear core (in our case number of clicks you are sending to merchants). 

We spent the quarter trying lots of different dials trying to understand the relationships between them.  What happens it we move the filter on the left up.  What happens if we add more price comparison?  What happens if we add more categories?

Some of the changes resulted in what you might expect but frankly more often than not the results were inconclusive.  Why?  One reason was that it was hard to tell which of the three features we had released caused the revenue to move upward or downward.  The second reason was that some changes where not immediate and hence very hard to correlate in time (never mind that correlation doesn’t imply causation).

I hate feeling out of control.  Like everyone else, feeling out of control makes me feel vulnerable and small.  It brings out superstition and dread in all of us.  There was a week in February where we had record numbers and during that week two babies were born at the company.  Someone joked that we should have more babies, quickly.  Someone else joked that released would take at least 9 months to develop…;-)  We all laughed (yes I know we have a nerdy sense of humor... but we like it).  Deep down though,  I had a sense of dread.  As an entrepreneur, I know well that there are things beyond my control, but I still hate it.

Sam our online marketing manager and Beth of Dir of Marketing came to the rescue.  I learned so much from the two of them during this period.  Actually we all did.  Azhar, Burak, and I all were skeptical that some simple UI tweaks could really make such a difference.  Beth proved us wrong and almost overnight we joined the cult of the A/B tester.  Every night Beth would dig into the numbers trying to find correlations.  She knew the numbers cold and helped us to build not just a data dashboard as Peter refers to here but to isolate and double click on each number until we found the cause.  It didn’t always work, but week on week she knew the numbers better than the rest of us on the exec team.  Beth is a very unique marketer, she is literally a poet quant.  She is smart, super aggressive, and never let anything stand in her way.  She was definitely a meat eating dinosaur (vs a vegatarian).  Her insights leds us through this period of uncertainity.

Sam was like a Roman legion, he just kept marching along and building infrastructure.  He used to come to our weekly marketing meeting and want to tell us about this great data warehouse he, Danny, Nikhil, and Pallavi were building.  At first I could have cared less, I would say,  “Sam how were sales this week and what is the plan for next week.”  Overtime I realized that it was his infrastructure that was allowing us to figure out exactly what was and was not working.

After three months we still don’t understand more than about 10% of the dials but with the help of Sam and Beth we are all slowly learning Russian...;-)

Previous Posts in this series

22 - Fast iteration , 21 - Like.com prognosis , 20 -Launch Video and More , 19 -PR Coverage of Like , 18 - Like.com Launches , 17 -Pre launch , 16 , 15 , 14 , 13 , 12 , 11 , 10 , 9 , 8 , 7 , 6 , 5 , 4 , 3 , 2 , 1 .

Comments

[Lloyd Fassett]

Nice approach, but it's not clear you controlled for the growth rate. If you realized 10X growth in clicks while your volume was steady, then it's a home run. That's effeciency. Unless, you also decreased the information displayed so people that were interested had to go to another place to do more research, but not necessarily buy. The growth in the chart look suspiciously do to more traffic though.

[Zaid Farooqui]

Munjal,

Are you guys using a 3rd party A/B testing system or something in-house? Funnily we are at a similar stage: we did a major new release after two months of development and overnight signups increased three times because we put a more distinct sign-up button. If only we can identify other such leaks...

--Zaid

[Zach Coelius]

Great post. Without testing you have nothing in this business