Episode 1: March 21 6am PST to March 28 6am

This is the first episode of 9 of Riya's beta launch. This episode takes place from March 21 6am PST to March 28th 6am PST. All facts are from my memory and hence subject to bias. My original post on this series of posts is here .

After working all night at 6am we finally get the system fully tested and ready to go. Azhar, Ann, Dan, Danny, Neelesh, Ben, Burak, and myself are there in California. Most of the team in India is there. Ann hits the button and tens of thousands of emails go out to all of the people who requested access to the site over the previous 6 months of blogging buzz. Dan, Neelesh, and Azhar break out the cigars and smoke outside in the cold dawn. They are shivering but clearly happy. Arrington blogs about us here . The first photo uploaded is here by Ben Lee our IT super god.

I drive home and goto sleep as that night I am supposed to fly to our India office. I wake up around noon. All seems to be well.

Throughout the day all seems calm. By 9:41am Danny sends out an email saying 30,000 photos have been uploaded. A few bugs popup, the biggest we will only realize a week later is in the core auto-recognition system (more on this later). People are uploading and training Riya to recognize their friends. At 5:55pm PM Danny sends out this graph

I get on an airplane at midnight to goto India via. Hong Kong. I fly Cathay Pacific and there is no Internet in coach. Deven and Vijay are with me. They are going to Hong Kong to visit Nana & Nani (Grandpa and Grandma). Deven sleeps most of the way there and later finds two friends his age (girls mind you.. my son Intl gigglo at age 2 ;-)) to watch DVDs with. All parents break the no TV rule when traveling on long flights... for the good of everyone else in the cabin...;-).

I land 14.5 hours later and see an amazing email: At 6:17am PST on March 22nd so just about 24 hours after we launched Riya crosses 1M photos uploaded. We announce it as taking two days, but I'm not sure why as it was only 17 minutes after the 24 hour mark. The 1M photo is here .

You can see the power of the blogosphere here in our referral user graph. We emailed tens of thousands of people who had requested access and yet an almost equal number came to us directly from blogs. The Techcrunch article got put on Digg and read in thousands of feed readers and viola... the Techcrunch effect begins. Michael's blog is the single more effective vehicle to get the word to the online blogosphere about new technology companies on the planet.

However there is a dark side to this. The servers were straining. Our two front end load balancers were are 90% CPU utilization. Most of our servers were running at 70-80% usage. If you remember we upgraded to almost 64 dual opteron boxes just before launch. This cost almost $300K and a four week delay while we waited for things to be installed. It was an accomplishment that we survived the deluge. At our peak we uploaded 52,000 photos per hour or almost 1.2M photos / day rate. After many years and lots of hardware this is very close to what a major photos site like oFoto or Snapfish or Yahoo photos upload per day. It was impressive ... but

We will soon realize that this delay was not worth it. The unfortunate fact was that the initial hammering we took was just not the reality we would see later. The number of photos uploaded per hour began to fall and then stabilized near the end of the 22nd at around 25,000 photos per hour and would continue to fall for weeks to come.

I land in Bangalore on Thursday night. I head directly to the office despite it being 11pm. I feel nervous and out of touch. Being in plane when the thing you've worked on for years is finally out in the public ... not my best idea. Burak meets me there and we hammer out a few emails on issues we are seeing. He has been in India for the last two weeks to help us coordinate the US and India efforts and is due to go home. On Friday night (Friday morning US time) Burak and I take the India team out to celebrate. We all have fun at a local bar and restarant that serves awesome Afgani food in Bangalore (the name escapes me).

Friday, Saturday, & Sunday - Dan, Neelesh, Ben, & Azhar lead the way in fixing critical issues. They don't sleep much in the next 72 hours. They are truly amazing individuals - all of them.

I read through all of the support emails one by one on Monday and Tuesday and send the following emergency feature requests to Azhar:

"After reviewing the emails I think there is one emergency features we need to try and implement (this week if possible).

- Change album by album from private to public and public to private"

Azhar is one step ahead of me. Sandeep is already working on the fix. He is one of the best leaders and fastest coders we have. At times like this he really shines. We roll out the change by early the following week.

Peter Rip is the first board member to use the product. He gives me gobs of good feedback. Amongst the most fortelling is "I trained my faces and then what do I do?" "There just wasn't enough to do", he tells me. We need to build more stuff to increase the "stickiness of the site". More on this later. Peter shares this photo with me.

Our Giggly server and auto-recognition system still has intermittent issues. We are all freaking out about this. I'm standing at Nikhil's desk many times a day. Nikhil is our rockstar sr. engineer in BLR who built the whole engineering side (vs research side) of our auto-recognition system. Weeks later he will argue the strongest for continuing to focus on it and I against that course of action. He is a very impressive guy and slowly works away at fixing the problem. It will be almost three weeks before all issues are resolved and in that time many big uploaders (2000+ photos) will be impacted, but it was Nikhil whose diligence and dedication got us through it.

I meet with our new controller Kaushik (who used to work with me at Andale) but I barely have time to do more than just dump all of the financial management I was handling on him and saying "so glad you are here." Kaushik is a pro and one of the best controllers I've ever worked with. He says, "Don't worry Munjal - I'll take care of it."

In the middle of all of this with hundreds of emails coming in, my head of customer service tells me he is quiting to take another job. Frackin shit man. Can the timing get any worse. I was really hurt actually. He was a guy I had long admired and had hired early (meaning before we launched) so that we would be ready when the launch deluge hit us. He is the first person to leave the company. He claimed he had a great opportunity back at Andale to run Intl and wanted to eventually move to the UK and thought this a good opportunity. He no longer wanted to do customer service. He is a good guy and I know he will do well. Fortunately he hired a strong team of four customer service managers to who were holding down the fort.

Azhar is in full motion now... he sends out the following email:

>>We need these fixed ASAP. For this week all we are going to do is FIX, Release, Fix, Release,.....

>>Here are the top priority issues for us:

>>1. Nikhil: AutoRec/QA page showing up with Zero results or shows same 2 photos over and over even >>though their are moire photos
>>2. Nikhil: Visual Sig corruption
>>3. Neelesh: Search Index update. Photos are QAed but are not showing up in the people search for >>hours sometime or not at all.
>>4. Tanvir: Any/All remaining issues with uploader, JRE, fatboy returning 0, etc.

>>Let's chat tomorrow morning about this.

If you are ever in a dark alley, he is one guy you want at your side.... what a gem.

March 27th 3:09am - someone uploads 10K porn images...

By the end of the first week - multiple millions of photos have been uploaded and many of the burning issues have been handled. I head out on vacation in Thailand on my way home to California. I blogged about it here .

Tomorrow night I'll put up the next week... March 28th to April 5th

riya
web2.0
riyarocks

Comments

[Shel Israel]

This is very good stuff. You should write a book about it. Seriously.

[Nicholas]

great account, can't wait to hear the rest!

most interesting line... "this [4 week] delay was not worth it"...

[Kunal]

This is great! I'm looking forward to week 2.

[Paul mendoza]

Wow, that sounds pretty crazy. Look forward to hearing more on this. I found the stickyness of the site was lacking as well which is why I find that even though I love the photo recognition, I still use Flickr for most of my photo needs and the blog this feature wasn't working at the time which is something I really wanted. Don't know if it works on your site now though.

[Jason L. Baptiste]

Awesome read Munjal. You're incredibly lucky to have such an awesome team from the PHDs, developers, marketing with Tara, and yourself at the helm. I look forward to the other entries, and the Riya API that is in the works.

Sincerely,
Jason L. Baptiste
CEO of Viral Ventures, parent company of uGather.com

[Munjal Shah]

Jason ... My team is good ... that is the best part about Riya.
Paul ... ya - thing were pretty crazy but in a good sort of way

[sean malone]

Munjal -
Great to see you surfacing for air.
Excellent read.
Thanks for posting.

[Netanel Jacobsson]

This is great! I can really identify myself with this...continue writing!

[Sunil]

This is great!

[BillyWarhol]

Great Story!

i'm interested to hear about the sticky part now!!

as someone mentioned earlier this is definitely movie material*

[user]

Hi Guys,

You are doing an awesome job.

Just an FYI, the Image uploader is not able to connect to the server. I get the message "Problem in connecting to the server. Check the Internet connection"

Thanks,
user.

[Manish Jethani]

>Afgani food in Bangalore

That would be Samarkhand? I love that place ;)

Congrats on the launch!

[Kenny wayne]

This is really amazing. Keep writing and you should do your best to write a book. Good luck.