It is difficult to understand new concepts when they are explained in the abstract. Also, we are tired of dogmatic debates. Let's have a look at how one of the most successful development organizations of our time actually works.

The following case study is the script for a presentation delivered by Jens Meydam (@jmeydam) on March 10, 2011 in Geneva. Many thanks to Don Reinertsen for giving feedback and for providing illustrations from his latest book.

The case study is followed by comments. There are comments by Don Reinertsen, Jeff Sutherland, Alistair Cockburn and Yishan Wong, one of the former Directors of Engineering at Facebook.

People x Process -
How Facebook Became the Hottest Company on the Planet

Changes: March 14, 2011: Added section on Jeff Rothschild. March 16, 2011: Inserted suggested corrections by Stefan Roock and Piaw Na in main text. Expanded introduction to presentations by Adam Mosseri and Julie Zhou: Facebook product design is "data informed, not data driven." March 18, 2011: Added section with comments by Yishan Wong. Added comment by Yishan Wong on Facebook's approach to crowdsourcing. March 26, 2011: Changed order of comments.

Introduction

David Kirkpatrick, the author of The Facebook Effect - The Inside Story of the Company that is Connecting the World, was asked what inspired him to write that book. He gave the following answer:

It was meeting Zuckerberg in September 2006 which began to convince me that this was an unusual phenomenon and company. Then I watched the service grow by the end of 2007 to 50-55 million active users, when it was less than 3 years old and led by a 22-year-old. Nothing like this has ever before happened in the history of business. Many people say my book is overly-sympathetic to the company and to Zuckerberg. But the fact is I have enormous admiration for anyone and any company which can move that fast and accomplish so much in so short of a time. I think any reasonable person should want to understand what it is about this phenomenon and company that has led to such amazing success so fast.

This is my attempt at understanding. :-)

In his answer Kirkpatrick points to two key factors that contributed to Facebook's success: people - in particular Mark Zuckerberg himself - and process - "moving fast." I will try to shed some light on the influence of these two factors. The formula in the title - people times process - has been suggested by Don Reinertsen: If either is zero, then the result is zero.

This presentation has two main parts, the "people part" and the "process part." I will rely on Don Reinertsen's latest work - The Principles of Product Development Flow - throughout the second half of the presentation, the part on process. It is tempting to describe in detail how Facebook operates, but this is not what I am going to focus on. The details are fascinating, but what we really need to do is to gain a better understanding of the underlying forces that are at work when a development organization is moving fast. Few people have thought about this as deeply as Don Reinertsen.

The first half, the "people part", will also give an idea of the evolution of Facebook during the first two years, from the beginning of 2004 until the beginning of 2006. I will focus on a few key moments: the start of Facebook's exponential growth in the first weeks after its launch in February 2004; how, in the fall of 2004, Facebook got the attention of its first real investor, Peter Thiel; and how Facebook started to look like an interesting place to work for some of the best engineers in the industry, leading to the hiring of - among others - Chris Cox, Yishan Wong and the "Microsoft Five" around the beginning of 2006.

I have used primary sources wherever I could. Quora has proven a goldmine. I have also relied on Kirkpatrick's book.

People

Great designs come from great designers, not from great design processes. - Fred Brooks

Mark Zuckerberg

One key to understanding Facebook's success lies in admitting that a lot of it has to do with Mark Zuckerberg.

The movie The Social Network tells a compelling story that is however only partly based on the real characters and events. It popularizes the claim that Zuckerberg "stole" the ideas from the Winklevoss twins and Divya Narendra, who were planning to launch their own site named HarvardConnection - later renamed to ConnectU - and had asked Zuckerberg to work for them.

"Mark is where he is because we approached him to include him in our idea." Tyler Winklevoss told the Times. Both twins are also convinced that ConnectU could have taken the place of Facebook in the world. They give Zuckerberg some credit for "not screwing up", but both believe that it was the idea and not the man that made Facebook a success.

This is view is flawed in three respects.

1. The concept of HarvardConnection was different.

The Winklevoss project combined ideas for a dating site with discounts at the "hottest clubs and lounges in the Boston area" for site members (Kirkpatrick 2010, 81). In contrast, Facebook was initially conceived as really just that, an online version of the "facebooks" at Harvard and other colleges that played such an important part in the social life of the students.

After examining the latest evidence, including instant messages from that period, even the rather sensationalist Business Insider had to conclude:

[...] only a week after beginning development of Harvard Connection, which he referred to as "the dating site," Mark had begun work on a separate project - "the facebook thing." Mark appears to have considered the products as competing for the attention of the same users, but he also appears to have regarded them as different in some key ways.

Zuckerberg did delay the launch of HarvardConnection until after the launch of Thefacebook (the original name of Facebook) in February 2004, but he did not implement their ideas. HarvardConnection launched only three months later in May 2004 and never really took off, neither at Harvard nor elsewhere. The site no longer exists.

2. Neither the idea of a social network nor the idea of an online facebook was new.

Friendster

There are several examples for earlier social networks, one of the most interesting being sixdegrees.com, which was launched in 1997 by Andrew Weinreich. Both the Winklevoss twins and Zuckerberg were definitely aware of Friendster, a popular social network that had been launched a year before Facebook by Jonathan Abrams. Friendster was more oriented towards dating, but was otherwise similar to the first version of Facebook and rapidly growing during the first half of 2003.

When Friendster launched in February 2003 it was an immediate hit. Within months it had several million users. [...] Pretty soon people were talking about Friendster as the "next Google." [...] Friendster seemed to be hitting the big time. Abrams made magazine covers. But by the middle of the year the experience of users started spiraling downward. Millions were joining and Friendster's servers were slowing. It couldn't manage its success. Pages took twenty seconds to load. (Kirkpatrick 2010, 71)

According to Abrams, "the site didn't work well for two years." By then Friendster had been eclipsed by MySpace and Facebook. Interestingly, Abrams does not attribute the initial success of Friendster to big, novel ideas: "The concepts were not new [...] What was new was the vibe of it, the design, the features." (Kirkpatrick 2010, 72)

Club Nexus and houseSYSTEM

There were also predecessors for social networks restricted to a college. The earliest was Club Nexus, launched at Stanford in 2001 by Orkut Buyukkokten, who would later create the networking site Orkut at Google. Membership was restricted to students with a Stanford-issued email address. In September 2003 Aaron Greenspan, a Harvard student, launched a service called houseSYSTEM that had a feature called the "Universal Face Book." According to David Kirkpatrick houseSYSTEM "never got much usage, though hundreds of students signed up to it." (Kirkpatrick 2010, 79)

Why did these earlier college networks not succeed? Club Nexus and houseSYSTEM had similar weaknesses:

Within six weeks Club Nexus had 1,500 members at Stanford, whose student body totaled about 15,000. But after it reached about 2,500, usage leveled off. The service was just too complicated. Buyukkokten was a talented programmer who had loaded it with every interesting feature he could think of. But that made it difficult to use and diffused activity among many different features. You didn't get the sense there were many others in there with you. (Kirkpatrick 2010, 77-78)

Today Zuckerberg won't say much about houseSYSTEM, except that "the trick isn't adding stuff, it's taking away." houseSYSTEM eventually disappeared. Zuckerberg's classmate Sam Lessin [...] recalls it as "a huge sprawling system that could do all sorts of things." By contrast, he says, Thefacebook was almost obsessively minimal. "The only thing you could do immediately was invite more friends. It was that pureness which drove it." (Kirkpatrick 2010, 82)

External Factors

In fact, there was a proliferation of social networks around the time Facebook was launched.

Photos are an almost indispensible feature for a social networking site, and it was only in 2003 that broadband internet access and ownership of digital cameras in the US had reached a level where uploading and viewing digital photos became a viable feature for a mass audience. The time for social networks had come.

The success of Friendster led to many imitations. Naturally, colleges were a particularly fertile ground for experiments in social networking. Open source software significantly lowered the barrier to entry. sixdegrees.com had spent a huge amount of money on database licenses and servers. Facebook and probably most other student websites were built on the LAMP stack and could be operated with minimal cost.

"Electronic facebook for the entire College should be both helpful and entertaining for all"

But there is another obvious reason why the idea to create an online facebook was not original. There was a popular, vocal demand for such a website.

In late 2003 the Ivy League seems to have collectively decided that campus facebooks should go online. Student governments at Cornell, Dartmouth, Princeton, Penn, Yale, and Harvard, among others, were all complaining to college administrators that their campus facebook was not in digital form.

Now that students had seen what was possible with Friendster, they wanted an online facebook. [...] The Crimson [the student newspaper at Harvard] included extensive references to the need to create an online facebook. [...] In a December 11 editorial titled "Put Online a Happy Face: Electronic facebook for the entire College should be both helpful and entertaining for all," its editors practically described how to build one. [...] That fall Zuckerberg took a math class on graph theory. At semester's end everyone in the class went out to dinner and ended up talking about the need for a "universal facebook." So Zuckerberg went home and built one. (Kirkpatrick 2010, 79 and 28)

3. It's the execution.

Why was Zuckerberg so much more successful than others?

An idea is just the beginning. For an idea to become a compelling product, there are so many design decisions and trade-offs that need to be made. People who excel at product development combine two traits:

1. A deep, intuitive understanding of their users and their expressed and latent needs
2. The ability to envision what is possible with the available technology

Zuckerberg possessed both of these traits. By the time he arrived at Harvard, he was already an experienced web developer. It was natural for him to experiment with little projects, increasingly with social features. At Harvard he got noted for having a gift for creating web applications with a "viral" appeal. In fact, the Winklevoss twins and Narendra tried to recruit Zuckerberg because of his proven ability to create a compelling product, an ability they themselves did not have.

Launch

On February 4, 2004 Facebook went live (initially under the name Thefacebook). It was an instant hit.

The software spread quickly from the very beginning. The first users - Zuckerberg's Kirkland House neighbors - sent emails to other students asking them to join and become their friends. [...] Someone suggested sending an email to everyone on the Kirkland House mailing list - about three hundred people. Several dozen signed up almost immediately.

Thus began a viral explosion. By Sunday - four days after launch - more than 650 students had registered. Three hundred more joined on Monday. Thefacebook almost instantly became a main topic of conversation in Harvard dining halls and between classes. People couldn't stop using it. [...]

By the end of the first week, about half of all Harvard undergraduates had signed up, and by the end of February approximately three-fourths. [...] After three weeks Thefacebook had more than 6,000 users. [...]

From even the second week, students at schools other than Harvard were emailing Zuckerberg asking when they could have it, too. Moving beyond Harvard had been in Zuckerberg's mind from the beginning. [...]

It took about half a day to do all the legwork and coding to add each school, but Zuckerberg and Moskovitz started expanding to new ones quickly even though both were still taking a full course load. They opened to students at Columbia on February 25, to Stanford the next day, and to Yale on the 29th. (Kirkpatrick 2010, 31-35)

By the end of March there were 30,000 active users. By the end of May it was operating at thirty-four schools and had almost 100,000 users.

At that point, the continuing success of Facebook depended on Zuckerberg's ability to attract collaborators. Alone he quickly would have been overwhelmed with all the problems and the work that came with rapid growth.

During the first months he could count on the people who were in closest physical proximity to him: his roommates at his residential house at Harvard, in particular Dustin Moskovitz and Chris Hughes. Rather than turning to his fellow students in computer science and convincing them to work on his project, he relied on the collaboration that was naturally emerging in his dorm room.

Dustin Moskovitz

Moskovitz was an economics student and would normally have been one of the least likely people to work on such a project. This is how he got started, according to Zuckerberg:

One of my roommates was "Hey, I'll help you!" I said "Dude! You can't program!" So he went home for the weekend an bought the book PERL for Dummies and said "Now I'm ready." I said "Dude, the site's not written in PERL." (Kirkpatrick 2010, 34)

Moskovitz went on to spearhead the expansion to other colleges and to ensure the site was operating smoothly. According to Zuckerberg, without Moskovitz' contribution, Facebook might not have succeeded.

While Moskovitz basically started at zero, he did over time reach a high level of technical skill. Yishan Wong, one of the former Directors of Engineering at Facebook reflects on Moskovitz' transformation into a top programmer:

At Facebook, I worked on some of Dustin's early code and it wasn't very good. So he didn't start out awesome when Facebook began. But the experience of working 24/7 at Facebook in a key role provided all of the things I listed above, and combined with Dustin's natural intelligence and work ethic, pushed his technical acumen to levels comparable to top CS students - most of whom study and program for a few years before getting to that level, and that's essentially what Dustin did working at Facebook.

In the summer of 2004 Zuckerberg and Moskovitz moved to Palo Alto. First this was supposed to be only for the summer, but in the end they didn't return to Harvard an kept working on Facebook full time.

Sean Parker and Peter Thiel

Next to Moskovitz, the most important early collaborator was Sean Parker, one of the developers of Napster. Parker had met Zuckerberg in New York in April 2004. Parker had heard of Facebook, and Zuckerberg admired Parker for his work on Napster. When Zuckerberg and Moskovitz rented a house in Palo Alto in the summer Parker moved in with them and started to work for Facebook.

Parker was a few years older than Zuckerberg and had experience and connections that proved to be vital for the success of Facebook. Above all, he saw better than anybody else the potential of Facebook. He had already developed a keen interest in social networks and knew several of the key players, among them Jonathan Abrams, the founder of Friendster, and Reid Hoffman, who had founded LinkedIn in May 2003.

From Parker via Reid Hoffman to Thiel

Facebook needed money to keep operating and growing. Parker's connections very quickly proved useful.

Within days after he joined Thefacebook, Sean Parker called his friend Reid Hoffman, the founder of LinkedIn and a big angel investor. [...] Hoffman was impressed with Thefacebook almost immediately but didn't want to be its lead investor, given his involvement in LinkedIn. [...] So he arranged for Parker and Zuckerberg to meet with Peter Thiel [...] who had co-founded and led PayPal and was now a private investor. [...]

[Hoffman had been a colleague of Thiel at PayPal and Thiel had invested in LinkedIn.]

Investors were still generally leery about consumer Internet companies, recalling how much they'd lost when the dot-com bubble burst. "So we felt this was the place to look for opportunity," recalls Thiel. "And within the consumer Internet, social networking seemed to be sort of an incipient trend. But in 2004 social networks were perceived as businesses that were very ephemeral [...] There was a question whether all these companies were just fashions that would only last a very short period of time."

But the message Thiel heard about Thefacebook gave him confidence. Parker, still only twenty-four but already a practiced pitchman, did most of the talking. He explained that while Thefacebook was still relatively small, that was because it required a .edu email address. The potential universe of members was deliberately circumscribed. Only students at select schools could join. What happened once it opened at a new school was what most impressed Thiel. Within days it typically captured essentially the entire student body, and more than 80 percent of users returned to the site daily! Nobody had ever heard of such an extraordinary combination of growth and usage in a start-up Internet company. [...]

Within days [...] Thiel agreed to what may go down in history as one of the great investments of all time. He agreed to loan Thefacebook $500,000, which was intended to eventually convert into a 10.2 percent stake in the company. That would give Thefacebook a valuation of $4.9 million. (Kirkpatrick 2010, 87-88)

Ensuring Independence

Parker had learned some hard lessons about the way venture capital firms quickly dominate a company they invest in, and he was determined not to let this happen with Facebook. Thiel was an ideal investor in this respect, but there would be more rounds of funding. Parker ensured that Zuckerberg would always retain absolute control over Facebook. This, too, is a critical reason for the lasting success of Facebook. Zuckerberg, Moskovitz and Parker were single-mindedly focused on growth. Monetization came as a very distant second. They considered advertizing a necessary evil and blocked anything that would disrupt the layout or the user experience of the site. Had Zuckerberg not been firmly in charge of strategy and product development, Facebook would almost certainly have given in to demands to monetize quickly with intrusive advertizing, to the detriment of the user experience and growth.

Parker also found a way to neutralize Eduardo Saverin, the earliest investor in Facebook (apart form Zuckerberg himself) and one of the co-founders. Saverin had not moved to Palo Alto and got more and more out of touch with developments at Facebook. He did not share Zuckerberg's single-minded pursuit of growth and probably feared that he would never get his investment back. Prompted by his father, he made unreasonable demands and froze Facebook's bank account when his demands were not met. Zuckerberg and his family invested some of their savings intended for his college education to allow Facebook to continue to operate before Parker managed to raise sufficient money.

Parker had to leave Facebook in 2005 after a drug-related incident, although there would never be any official charges against him. Parker and Zuckerberg remained close.

Jeff Rothschild

[In the original script I failed to mention Jeff Rothschild. Several readers saw that as an unforgivable omission. Below is a passage on Jeff Rothschild from Kirkpatrick's book.]

After Accel [a venture capital firm] made its big investment, Kevin Efrusy, who had spearheaded the deal, began visiting regularly to advise Zuckerberg. He proposed bringing on a part-time consultant named Jeff Rothschild, who had co-founded the big business-software company Veritas. Rothschild had both a deep knowledge of data centers and the maturity of a fifty-year-old. Zuckerberg realized that Rothschild could help Thefacebook prevent a Friendster-like breakdown. (Kirkpatrick 2010, 130)

In the end, Zuckerberg persuaded Rothschild to work full-time for Facebook. According to several sources, Rothschild has played an important role in Facebook's success.

Recruitment of Top Engineering Talent

One of the constant fears of Zuckerberg, Moskovitz and Parker at that time was that Facebook might soon share the fate of Friendster: that it would collapse under the weight of its own success.

There were few, if any, companies at that time that had the capability to re-architect and evolve a social networking site to support the kind of growth Facebook was expecting. Even Google ran into performance problems with Orkut. Zuckerberg was aware that hiring top engineering talent was vital for Facebook to keep up its momentum.

Adam D'Angelo

One of Facebook's best engineers ever was Adam D'Angelo. He was also one of the first, although he was initially only helping out and later had a break for a year to finish his degree. Zuckerberg and D'Angelo had been friends since they had studied at Phillips Exeter Academy. They had jointly developed a music suggestion program called Synapse, which got the attention of Microsoft. D'Angelo went on to study computer science at the California Institute of Technology, while Zuckerberg went to Harvard, but the two kept in touch and D'Angelo had been helping with Facebook since its launch.

Just to give an idea what kind of talent Facebook would value, here is what D'Angelo lists on his own LinkedIn profile in the Honors and Awards section:

• Finalist, Topcoder Collegiate Challenge, 2005
• 2nd in North America (team of 3), ACM International Computer Programming Contest, 2004
• 1st in North America (team of 3), ACM International Computer Programming Contest, 2003
• Silver Medal, International Olympiad in Informatics, 2002

This is the person who interviewed job candidates at Facebook in a critical period of its development.

Now he was Thefacebook's top engineer. No matter who else they'd spoken with first, Zuckerberg always wanted candidates for important tech jobs to interview with D'Angelo. If he thought they were smart, they'd get hired. (Kirkpatrick 2010, 131)

After he had finished his B.S. in computer science at Caltech D'Angelo served as Facebook's VP of Engineering and CTO until he left to found Quora in 2008.

Ivy League Graduates

In general, recruitment efforts were targeted at Ivy League computer science students. This is from an article in the Crimson, the Harvard student newspaper, published in November 2005:

Zuckerberg, formerly of the Class of 2006, will forego a Harvard degree to run facebook.com, the second largest online social networking site and the 10th most-trafficked site on the Internet, according to Hughes.

Zuckerberg spent the morning meeting with computer science professors for help recruiting engineers straight out of college.

"The professors can identify who the smart students are," he said, adding that he would prefer to hire younger engineers rather than programming veterans. "The job lends itself to people with raw intelligence rather than industry experience. And if you’re coming out of college, you have a really good idea of what facebook is."

Zuckerberg said he has made similar recruiting trips to Stanford and Berkeley. He will recruit at MIT today before heading back to facebook.com’s headquarters in Palo Alto, Calif.

The "Microsoft Five"

Zuckerberg also tried to recruit former teaching assistants who had impressed him at Harvard and people they were connected with. This led to the recruitment of the "Microsoft Five."

These are the recollections of Yishan Wong:

The "Microsoft Five" joined Facebook in the beginning of [...] 2006 all within a very small span of time. Ironically, they did not all join from Microsoft - the moniker was given to them by a recruiter because they were coming down from Seattle and I guess it stuck. All of them have subsequently gone on to lead significant initiatives at Facebook and have a tremendous impact on the company. [...]

At the time of their joining, Facebook Engineering numbered only about twenty developers. While an earlier wave of apparently incompetent contractors had been wiped out, the specific push to hire top technical talent had only just begun: I arrived a mere two weeks before these guys, [now VP of Product] Chris Cox had started two weeks before me, and Aditya Agarwal (now Director of Engineering) had joined earlier that year.

Even though most of them now hold management positions, each of these people was also an exceptional individual developer. Andrew Bosworth, for example, wrote the entire back-end system for the first version of News Feed. Therefore, a sudden and simultaneous injection of elite engineers now comprising 25% of the entire engineering staff helped to inflect that talent level sharply upwards, raising the standard of quality and practice for the entire engineering team. It also served as a practical demonstration that Facebook could and should strive to only hire top technical talent, and was the beginning of a long road in turning that into an institutionalized practice.

From the Perspective of Charlie Cheever

What went on in the mind of a former Harvard teaching fellow who now had a job at a great company when he was asked to consider working for Facebook?

This is the story of Charlie Cheever, one of the "Microsoft Five" (Cheever had actually worked for Amazon):

I was a few years ahead of Mark and Dustin at Harvard and was a Teaching Fellow for CS51 (the second course in computer science there,) and so they had heard of me, and I had met Mark briefly when he tried out for ultimate frisbee team. Somehow from that, I got an e-mail from a recruiter asking me if I was interested in Facebook. I ignored the e-mail since Facebook didn't seem like a serious company, I figured I could have made the website myself in about 3 days, and I didn't really like the idea of working for someone younger than I was when I was only 24 years old.

The two things that changed my mind were:

1. The launch of Facebook Photos in October 2005. I remember seeing this for the first time, and being annoyed with myself for not having thought of the idea before. My brain tried to think of reasons that it wouldn't take off so I would be less annoyed, but I just knew it was going to be the main way that people found photos of each other.
2. At a birthday party in Belltown (a neighborhood of Seattle,) I ran into Dave Fetterman and Andrew 'Boz' Bosworth who I knew from school, and they had some news: "Guess what? We're quitting our jobs at Microsoft and going down to Silicon Valley to work at Facebook!" I liked Fetterman and Boz and thought they were smart and hearing this made me think that Facebook might actually be a legitimate company where I might find good people to work with. I ended up searching through my Blackberry for the e-mail Facebook had sent me and replying to it from the party.

When I interviewed at Facebook, I remember being especially impressed by Dustin and Adam and the plans they outlined for what could be done next with Facebook, in particular news feed. Also, James Wang interviewed on the same day that I did, and even though I didn't know him very well, I knew he was smart and a lot of my friends really liked him.

I did pretty well in the interviews in part because Fetterman and Boz had given me some tips on the things that I might be asked about, and I got an offer. I thought about it for a while and talked things over with my parents and then I talked to my boss at Amazon and then threw all of my worldly possessions into the back of my car and drove down from Seattle to Palo Alto over New Year's.

"An Unbelievably Great Mix of People, Product, and Purpose"

Facebook was a special place. It had become an engineering power house, and everybody shared a strong sense of mission. This is how Jed Stremel puts it, who worked there from 2005 to 2009:

Especially in the early years (and even today), Facebook had an unbelievably great mix of people, product, and purpose. Many other companies share these traits, but I know few others that share them all. [...]

More specifically from '05, '06, '07:

• Brilliant, visionary founders
• Talented, interesting, and fun coworkers
• Important technical and business challenges
• Sense of purpose; changing the world
• Proving the many doubters wrong

Competing with Google

Almost since the beginning, Facebook saw itself in competition with Google for top engineering talent. Initially this was only about college students. Later, many engineers and managers left Google to work for Facebook, although only when Facebook had already become a relatively mature, successful company. At one point in time, 20 percent of Facebook's employees had previously worked at Google.

One of them is Robert Kieffer:

As a seasoned veteran of Silicon Valley, and current Facebook employee, I have to say Facebook is a pretty cool place to be right now. Google will still be Google in 5 years, probably little changed from what you see today. Facebook, however, is transforming rapidly, and is at the heart of a profound change in our culture. Nobody knows for sure where Facebook will be in five years. Will it be huge - "the next Google!" - or will it flame out, MySpace-fashion? The opportunity to influence that outcome is part of why it's cool to be here.

Process

A bad process will beat a good person every time. - W. Edwards Deming

Moving Fast

Bootcamp

Today, when an engineer starts at Facebook, he first goes through an intensive six-week training called Bootcamp during which he will be:

• Picking up tasks that engineering teams have designated as bootcamp-suitable: small and self-contained.
• Attending onboarding lectures, where other engineers (or non-engineers, in a few cases) give a presentation introducing some area of knowledge (e.g. our code push process, or our service monitoring tools).
• Meeting with managers and engineers from teams they're considering joining.
• Meeting other people - bootcamp mentors (who are senior engineers who volunteer for this function), fellow bootcampers, engineers with whom they have to interact in doing their tasks, etc.

The Bootcamp program is run by one of the veterans of Facebook, Andrew Bosworth from the "Microsoft Five." The aim of the Bootcamp program is not just to get new hires up to speed. It is an immersion experience that impresses on them the key values of Facebook's culture. At the core of this culture is the feeling that it is important to let engineers move fast, even if this means there may be issues in production.

At Facebook, I had checked in code my first day there, and it was pushed to the live site on the next day. There were bugs, and emergency pushes to fix those bugs, but a few days later my code was running in production, and millions of people were benefiting from it. That was 3 years ago, and it's still the case now. In general new-hires have code go live in their first week, maybe first two weeks. Chances are pretty high that your code is directly visible by users, though as the company grows, there is a lot of work on internal tools as well.

[...] you have a unique opportunity to touch 100M's, even potentially 1B's, of peoples' lives. You could be "that guy who broke Facebook last week!" :-) ... and by that I mean, it's scary how easy it is to get code into production here, and to see how it affects people.

Quality

It is not that Facebook doesn't care about quality. Code reviews are mandatory, and there is an emphasis on coding conventions. There are automated tests at all levels of the stack, integrated with continuous integration systems and other tools. The trunk version of the code is constantly tested in actual use internally. Deployment is staged in a way that minimizes risk.

Steve Grimm is Facebook's test engineering tech lead. This is a fragment of his answer on Quora to a question regarding automated testing at Facebook:

For browser-based testing of our Web code, we use the Watir framework. We have Watir tests covering a range of the site's functionality, particularly focused on privacy - there are tons of "user X posts item Y and it should/shouldn't be visible to user Z" tests at the browser level. (Those privacy rules are, of course, also tested at a lower level, but the privacy implementation being rock-solid is a critical priority and warrants redundant test coverage.)

And that is just a small part of what they do in this area.

"Speed Is Something Incredibly Important to Us"

Having said that, Facebook values moving fast more than anything else.

Listen to any Facebook employee in a leadership role, and you won't have to wait long for statements like:

One of the things we most value at Facebook engineering is moving fast. (Robert Johnson, Director of Engineering)

We pride ourselves on our fast release turnaround time. (Steven Grimm, Facebook test engineering tech lead)

Speed is something incredibly important to us. (Adam Mosseri, product design manager)

There is a firm belief at Facebook that nothing should stand in the way of an engineer who wants to release a change or a new feature to production. There is no separate QA team. Engineers are responsible for the quality of their work. Changes are released daily, often multiple times a day. New features are released on a weekly basis.

Facebook positively encourages engineers to err on the side of speed. These are some of the slogans that are posted around Facebook's offices:

• Done is Better Than Perfect
• What would you do if you weren’t afraid?
• Move Fast & Break Things
• Proceed & Be Bold
• HACK

Is this obsession with moving fast just part of Facebook's legacy, more a bug than a feature, or might it be that they actually know what they are doing?

As already mentioned, I will rely heavily on Don Reinertsen's latest work, The Principles of Product Development Flow. The book is organized into principles, over 150 principles in total. The following sections will each start with a list of the principles that I feel are the most relevant. I will quote Reinertsen's explanations of a few of these principles, but we can't go into much detail here. I strongly encourage you to read the book.

Accelerated Learning

• The Fast-Learning Principle:
  Use fast feedback to make learning faster and more efficient. (FF8)
• The Principle of Early Contact:
  Make early and meaningful contact with the problem. (D16)
• The Empowerment Principle of Feedback:
  Fast feedback gives a sense of control. (FF20)
• The Hurry-Up-and-Wait Principle:
  Large queues make it hard to create urgency. (FF21)
• The Batch Size Queueing Principle:
  Reducing batch size reduces cycle time. (B1)
• The Batch Size Feedback Principle:
  Reducing batch sizes accelerates feedback. (B3)
• The Batch Size Overhead Principle:
  Reducing batch size reduces overhead. (B5)
• The Infrastructure Principle:
  Good infrastructure enables small batches. (B19)
• The Repetition Principle:
  Repetition reduces variation. (V8)

Fast Feedback

For Facebook, the most important advantage of moving fast is that it enables fast learning through feedback.

Robert Johnson, Director of Engineering:

(Transcript) You get better results if you can move fast, and the reason you get better results is that, when you're doing something that hasn't been done before, when you're doing something that has a large degree of uncertainty, by far the most important factor to your success is whether you do a good job of figuring out what's going to work, what you should do, and the way you do that is by being able to build a lot of stuff and see if it works, and the way you do that is by being able to build stuff quickly.

This notion of, hey, we should slow down and get it right, doesn't actually make much sense, because unless you have an extremely good idea of what right is, slowing down to get something right just means you're going to take longer before you actually figure out that you're wrong, and, more importantly, why you're wrong, and move on to the next thing.

Batch Size and Feedback

But how do they do it? What do they actually do to move fast? The easiest way to improve flow and to get fast feedback is to reduce batch size, that is, making frequent small releases. Consider this figure:

B1: The Batch Size Queueing Principle: Reducing batch size reduces cycle time.

Figure 5-1 uses a CFD [cumulative flow diagram] to depict a steady arrival rate being processed with different batch sizes. [...] As you can see, queue size is directly proportional to batch size. And, since Little's Formula says average queue size determines average cycle time, this means that we can directly reduce cycle time by reducing batch size.

It is worth pointing out that we have reduced cycle time in Figure 5-1 without making any changes in the average arrival rate or the average departure rate. This means we have reduced cycle time without changing either demand or capacity. This is quite important since throttling demand and increasing capacity can have significant economic cost. Batch size reduction can enable us to shorten cycle time, often by an order of magnitude or more, without changing capacity utilization. (Reinertsen 2009, 112)

Imagine Facebook had a big release once every six months. Changes and new features would have to wait for an average of three months before they would get released, which means feedback would get delayed for an average of three months.

This is how the quote by Robert Johnson continues:

(Transcript) In a traditional environment where you might think something moves pretty fast in software development, where you have say a three or four week release cycle, if you're in that situation you're able to test 10, 20 things a year, and get changes through and get data on that. The thing we do at Facebook is, if I had an idea last night, I coded it up this morning, I get it on the site this afternoon, and tomorrow I have data. And naturally our goal is to be able to get that kind of cycle time going, because when you have that, in a couple of weeks I've actually learned as much as I would have in a year in the other system. And this makes a huge difference because it compounds. The more I learn now the more informed my next set of decisions are, and if I can get data at that kind of rate, then, that's how you get these explosive numbers for things like growth.

There are further benefits to reducing batch size. Frequent, small releases lead to less administrative overhead, for example in the form of status reports. Frequent, small releases also provide a strong incentive to optimize operations for frequent, small releases. This leads to high degree of automation, which leads to a high level of efficiency and reliability. All this, of course, helps you go even faster.

(If you are interested in the details of operations at Facebook, there is a great presentation with the title A Day In the Life of Facebook Operations by Tom Cook.)

Finally, a more subtle point: With slow feedback, learning does not only occur late, much of the learning does in fact not happen at all.

FF8: The Fast-Learning Principle: Use fast feedback to make learning faster and more efficient.

It should be obvious that fast feedback improves the speed of learning. What may be less obvious is that fast feedback also increases efficiency with which we generate information and learn new things. It does this by compressing the time between cause and effect. When this time is short, there are fewer extraneous signals that can introduce noise into our environment. (Reinertsen 2009, 220-221)

Exploiting Variability

• The Principle of Asymmetric Payoffs:
  Payoff asymmetries enable variability to create economic value. (V2)
• The Principle of Variability Consequence:
  Reducing consequences is usually the best way to reduce the cost of variability. (V12)
• The Batch Size Risk Principle:
  Reducing batch size reduces risk. (B4)
• The Newsboy Principle:
  High probability of failure does not equal bad economics. (E20)

Innovative work is inherently risky. An organization like Facebook that is positively revolutionary is creating value not by playing it safe, but by minimizing damage from bad outcomes and exploiting opportunities opened up by good outcomes. This is enabled by fast feedback loops.

Payoff Functions

The following two figures show probability, payoff and expected payoff (the product of probability and payoff) in the case of financial options:

The key insight here is that variability can create economic value. If, under certain circumstances, variability can create value, we must question the widespread assumption that we should always strive to minimize risk and variability.

If we map this reasoning to product development, we may assume that probability behaves in a similar fashion, but the payoff function is different. The following figure displays the graph of a typical payoff function in development:

Payoff-functions in product development are different from those in financial options in two important ways: upside can be limited, and downside can be unlimited. (Reinertsen 2009, 92)

This leads to another important insight. We can try to improve the economic outcome in two ways: by changing variability and by changing the payoff function.

When we alter the consequences of variability, we are changing the economic payoff-function, rather than the probability function. We can think of this payoff-function as having a "left side" and a "right side." We can improve both sides. For example, we can decrease cost on the left side by quickly truncating a bad path. If a feature is much harder to implement than we expected, we can drop it from the product. We can also increase the payoff on the right side. For example, if a feature is easier to implement than we expected, we may choose to overachieve on this feature. (Reinertsen 2009, 103)

Internationalization Project

This kind of thinking is observable in Yishan Wong's account of the internationalization project.

We created an application that would display the extracted strings to translators, allowing them to enter proposed translations as well as vote on these translations. [...]

In those days, crowdsourcing translations was entirely untested, so we had no idea if it would actually work or if it would descend into anarchy and vandalism. Therefore, we actually hired an army of professional translators to stand by in case everything went wrong. [...]

As it happened, the initial run of translation into Spanish produced very good results, with quality superior to professional translation in some cases. This took about two weeks, primarily because we opened up the process and tools slowly. After we verified the quality of results, we opened it up to French and German, which (if I recall correctly) took 24 hours and 48 hours to complete, respectively. The quality of these translations were not initially as good as the Spanish translations, but they came up to par quickly because our system allowed successive refinements to be submitted by users. [...]

One side-effect of this system is that Facebook is able to offer translations in a very large number of languages. While Facebook only has professional support for about 20 major languages, we had smaller highly-motivated populations (e.g. the Basque region of Spain) clamoring almost immediately for us to open up the translation tool in their language. For sufficiently motivated but underrepresented languages, this provided enough community translators to produce a quality translation rivaling those of much larger languages.

Instead of playing it safe and letting professional translators do what they do best, Facebook decided to try a novel approach: The users would be empowered to translate Facebook into their native language via translation features directly built into Facebook (= increasing rather than decreasing variability). They did that in the hope that this would quickly lead to translations that felt natural to the users. They started with relatively familiar languages like Spanish and German. They minimized the risk by hiring translators anyway, just in case things went horribly wrong (= improving the left side of the payoff function). When they found that this approach actually worked well, they were ready to let users translate the site into more exotic languages for which Facebook had no translators to oversee the translation process (= improving the right side).

Piaw Na remarked that in 2003, Google had already tried and tested crowdsourced translation in search and other applications.

Comment on this by Yishan Wong:

We were actually aware of the crowdsourced translations that Google did, but they did them in a totally different (and in our opinion, poorly-designed) manner. Many Googlers were aware of this and tried to point it out to us when we'd interview them for positions relating to internationalization but tended to be limited by the Google experience instead of what we were able to do with our platform. Also, a key piece of our "strategy" involved hiring an internationalization expert from PayPal who I had previously worked with. He managed all of our contract translators as well as provided us with a lifetime of experience with "traditional" internationalization techniques so that, once again, we knew all the old rules before we broke them.

"What Is Going to Happen If Facebook Fails?"

Things rarely go this smoothly.

This is Adam Rifkin's answer to the question: "What is going to happen if Facebook fails?"

Facebook fails regularly as it is.

They've botched several product launches (remember Beacon?), destroyed a once-vibrant developer ecosystem, and have starved their iPhone and Android apps of resources.

The interesting thing about them is, they learn from their failures. In the three above examples, they revamped Facebook ads so brand advertisers are eager to buy them, they shifted developer focus to Facebook Connect and Instant Personalization, and they are aggressively hiring (buying?) mobile and smartphone developers.

They've assembled 500 of the most adaptive engineers on the planet that work so well collectively that they could sustain the losses of both Dustin M[oskovitz] and Adam D['Angelo] without any discernible effect on the quality and speed of product development. That's amazing.

Their product is evolving so quickly, it is hard to imagine a misstep that is an unrecoverable failure.

If Facebook fails, it learns, regroups, adapts, and develops some more.

You know what companies that reminds me of? The Microsoft and Cisco of the 1990s, the Google and Apple of the last decade. Successful companies that pivoted and adapted as they grew.

Decentralized Control and Alignment

• The Main Effort Principle:
  Designate a main effort and subordinate other activities. (D10)
• The Principle of Mission:
  Specify the end state, its purpose, and the minimal possible constraints. (D8)
• The Principle of Peer-Level Coordination:
  Tactical coordination should be local. (D13)
• The Principle of Regenerative Initiative:
  Cultivating initiative enables us to use initiative. (D21)
• The Opportunistic Principle:
  Adjust the plan for unplanned obstacles and opportunities. (D4)
• The Principle of Face-to-Face Communication:
  Exploit the speed and bandwidth of face-to-face communications. (D22)
• The Principle of Colocation:
  Colocation improves almost all aspects of communication. (FF19)
• The Trust Principle:
  Trust is built through experience. (D23)

Decentralized Control

Decentralized control pushes decision making down to the people doing the work.

Agile software development embraces decentralized control and the benefits this brings, in particular with respect to moving fast. It should come at no surprise that Facebook is basically an agile shop, even if you won't hear much about "Scrum at Facebook" or "XP at Facebook." They work in small, interdisciplinary teams, and they deliver high-quality software early and often.

(If you want to learn more about how these teams work, and how product design at Facebook is "data informed, not data driven," have a look at these presentations by Adam Mosseri and Julie Zhou. There is also a short video on the profile team with, among others, product manager Peter Deng and an interview with Facebook's VP of Engineering, Mike Schroepfer.)

Lessons from Advanced Military Doctrine

What Agile software development has traditionally not been good at is achieving alignment with the overall goals of the organization. Reinertsen thinks that the military has developed a deep and sophisticated understanding of how to achieve alignment together with decentralized control, and that product development has a lot to learn from military doctrine.

The military has spent centuries wrestling with the balance between centralized and decentralized control. They have thought carefully about both the advantages and disadvantages of decentralized control. They have tested their ideas in battle, against other smart and motivated thinkers. If their ideas do not work, they die.

[...] the modern military relies upon the initiative of subordinates. It focuses on responding to a fluid battlefield, rather than executing a predetermined plan. It views war as a domain of inherent uncertainty, where the side that can best exploit uncertainty will win. (Reinertsen 2009, 243)

The dark side of decentralized control is misalignment. If all local actors make locally optimum choices, the system level outcome may still be a disaster. Is there a way to achieve alignment without centralized control?

This is the sophisticated heart of maneuver warfare. How do the marines achieve alignment in the presence of uncertainty? How do they opportunistically shift their focus without losing alignment? They do not use elaborately detailed plans and micromanagement. Instead, they use tools like mission orders, doctrine, and lateral communications to combine initiative and alignment. (Reinertsen 2009, 251-252)

Alignment and Mission at Facebook

How is Facebook dealing with this problem?

First, it should be noted that at Facebook, even at the team level control is decentralized. Product managers at Facebook are not stereotypical Scrum Product Owners who figure out what the next steps should be and who reduce the role of the team to rapid execution.

[Stefan Roock remarked that what I call "stereotypical Scrum Product Owners" - Product Owners who reduce the role of the team to rapid execution - are more of an anti-pattern. Product Owners should lead through vision and collaborate with the team - just as in the quote by Jeff Sutherland below.]

Individual engineers and designers have a lot of power at Facebook, to the extent that one might get the (wrong) impression that product managers are "useless" at Facebook.

If you think that a more typical Scrum implementation would be an improvement, consider the following quote by Jeff Sutherland:

In Takeuchi's teams the cross functional team often had to figure out what the product should be. Emergent leadership often occurred to enable this. The best Product Owners I know work with the teams this way.

Remember Takeuchi looks at their teams and it reminded them of the Scrum formation in Rugby. This is intense, focused, and very high energy. In this state, the team works with the product only to get the backlog clear and compelling so they can knock down a great product at high speed.

For the precious few teams that are hyperproductive, it is all about entering into a flow state. People want to watch Tiger Woods hit the ball again so they can experience vicariously the moment when Tiger knows the ball is going in the cup when he is still swinging the club. This flow state merges what people love with the work they do and generates a powerful ecstasy. When the entire team goes into flow it is even more transformative than the individual state.

The Apple management team conveyed some of this flow when they did the iPad launch video. It brought down the internet in Sweden at the time. [...]

At Facebook, this works because team members at Facebook are superbly aligned with regard to the team mission and with regard to the overall mission of the organization.

In maneuver warfare, a key tool for creating alignment is a mission [...]

Mission orders focus on the intent of the operation, rather than constraining how the mission is to be accomplished. [...] It communicates the "why" of the operation, not the "what" or "how."

In maneuver warfare, the mission is commonly described in the section of an operation plan called the "Commander's Intent." The Marines believe orders are incomplete without this information. They believe that intent needs to be understood deeply through the organization. They believe that when intent is clear, Marines will be able to select the right course of action. (Reinertsen 2009, 252-253)

Obviously, Facebook does not use mission commands in the literal sense. However, Mark Zuckerberg's vision is deeply shared and understood not just by the management team, but by every single Facebook employee. Whoever you listen to, everybody at Facebook seems to be excited about the idea of making the world more open and connected and the impact they are having on society.

From Time's Person of the Year story:

"The thing that I really care about is making the world more open and connected," Zuckerberg says. "What that stands for is something that I have believed in for a really long time."

Pressed to define it, Zuckerberg gamely expands. "Open means having access to more information, right? More transparency, being able to share things and have a voice in the world. And connected is helping people stay in touch and maintain empathy for each other, and bandwidth."

It's not just theory to them, they are all passionate users of their own product.

There are more subtle points that I can't cover in detail here. Just think of how important the role of colocation was in Facebook's early history, in Zuckerberg's dorm room at Harvard and later in the first houses they rented in Palo Alto, where Zuckerberg and Moskovitz lived together with Sean Parker.

Also, it is probably not an accident that until recently Facebook had concentrated development at their Palo Alto headquarters, where communication bandwidth is highest. (In August 2010, Facebook opened an engineering center in Seattle.)

Sanity Check: Getting the Economics Right

• The Principle of Marginal Economics:
  Always compare marginal cost and marginal value. (E16)
• The Principle of Quantified Overall Economics:
  Select actions based on quantified overall economic impact. (E1)
• The Principle of Quantified Cost of Delay:
  If you only quantify one thing, quantify the cost of delay. (E3)
• The Principle of Maximum Economic Influence:
  Focus control on project and process parameters with the highest economic influence. (FF1)
• The U-Curve Principle:
  Important trade-offs are likely to have U-curve optimizations. (E6)

So were they right in making "moving fast" their top process priority?

Let's think about it this way: what would have happened if Facebook had moved slower?

They would not have been able to evolve the product as rapidly as they did and make it engaging and relevant for an ever increasing number of users. Growth would have slowed down. Growth, however, was vital for success. Zuckerberg, Moskovitz and Parker were aware of the network effects of what they were creating. The competitor with the most members was likely to take the whole market.

Also, it would have been harder for the engineers to keep up with the explosive growth of the user base. They had to learn fast how to evolve the architecture to make it ever more scalable. It was already an uphill battle as it was. The last thing they needed was a process that slowed them down.

As Bertil Hatt puts it, it was a "tough war of attrition, network effects and speed to scale."

Suppose slowing down would have reduced Facebook's chances to win by 50%. What would have been the economic cost of slowing down, in terms of impact on life cycle profits? At the moment (March 2011), Facebook is valued at $50 billion or more. If slowing down had halved Facebook's chances to win, the cost would have been something like $25 billion.

People Times Process

I will conclude with Andrew Bosworth's account of the history of the "Like button". Much of the development took place in late 2007. This account will complete the picture by giving an impression of some of the patterns of interaction at Facebook - also when things don't go smoothly.

What's the history of the Awesome Button (that eventually became the Like button) on Facebook?

July 13, 2007 - In the initial email discussion about a project codenamed "Props" between me, Justin Rosenstein , Leah Pearlman, Ezra Callahan, and Akhil Wable, we boil down the discussion of symbols to:

• stars (concern that it would translate to "I give this 1 star" which is a bad review)
• plus sign (possibly accompanied with a minus sign, apropos to this discussion)
• thumbs up (concern about internationalization, this is a bad sign in some places)

We also consider lots of language and settle, temporarily, on "awesome."

July 17, 2007 - At a Hackathon, Tom Whitnah, Justin Rosenstein, Olaoluwa 'Ola' Okelola, Rebekah Cox and I build a working initial prototype of the "awesome button," complete with integration into News Feed and Mini Feed.

July 19, 2007 - the longer process of design begins with Aaron Sittig, Rebekah Cox, along with Katie Geminder.

July 31, 2007 - In a larger set of threads, the project builds a huge amount of interest:

• James Wang and platform product marketing are interested because of the potential to use awesome button to filter out bad application stories.
• Feed team interested in improving feed ranking
• Ads team interested in improving CTR

August 17, 2007 - Alexandre Roche delivers comprehensive design mocks.

August 22, 2007 - the word "like" is proposed as a more universal term but receives a lukewarm response. We've become somewhat attached to awesome and, comparatively, like seems bland. Carolyn Abram starts getting involved in language choices.

September 26, 2007 - the project slows to a crawl as we search for a UI that fits in too many places on the site (platform, adds, feed, content).

October 30, 2007 - Friendfeed adds "like" feature. As far as I can tell from my email archives, nobody at FB noticed. =/

November 12, 2007 - Ready to launch and things appear to be all set but final review with Zuck surprisingly doesn't go well. Concerns about the whether the interaction is public or private, cannibalizing from the share feature, and potential conflict with Beacon. Feature development as originally envisioned basically stops.

November 14, 2007 - The feature is temporarily exposed to users due to a bug.

SOMEWHERE IN HERE: The News Feed and Ads teams both launch small scale implementations that accept both positive and negative feedback but apply that information privately to the user and do not share it socially. The news feed system proves ineffective and is ultimately shut down as Tom mentioned earlier.

February 9, 2009 - Facebook launches "like" and is widely accused of stealing the idea from Friendfeed. I believe the credit for getting this out the door goes to Jonathan Pines and Jared Morgenstern as well as designer Soleio

You may have recognized some process aspects. You may also have noticed how important it is to Bosworth to give credit to the right people. This is what "people times process" looks like on the ground.

Facebook as you know it today is the result of countless episodes like this.