Data Scientist forecasts Amazon’s HQ2

by Shawn Berry

The best Data Scientists see in a slew of data what matters, communicate why it matters, and instruct others about how strategy and operations must change to address what matters.

Herein I conjecture the what, why, and how for Amazon as the company evaluates where to build its second headquarters, HQ2, with up to 50,000 employees – a size to rival the near-term size of its original headquarters in Seattle, HQ1.

To my amusement, I see many wild predictions, using unimportant variables, on what city will land HQ2.  For instance, Moody’s Analytics says it analyzed 29 sets of data.  To me, that means Moody’s couldn’t pinpoint what matters.  In contrast, I emphasize Amazon’s own data and then pursue the data found in two leading cities’ bids for HQ2.  I use other published data to confirm or challenge my findings.  I also challenge one well-known piece of published data.

So let’s attempt to deduce what matters the most to Amazon.  Well, first, why does Amazon need a second headquarters?  Good question!  Amazon is starting HQ2 precisely because the company has maxed-out the Seattle IT market.  Hiring 10,000-20,000 new IT employees every year will do that.  Headcount in Seattle is flattening out.  Due to attrition, HQ1 has to hire in droves just to maintain headcount.  But the company sees big growth opportunities worldwide and wants an infusion of new talent.

Ok, what talent exactly?  What types of work does Amazon want people to perform?  And what types of work will Amazon need its employees to perform in the coming years?  We can find some answers by mining data from the Amazon Careers search engine.

Well, today the job in highest demand at Amazon is – surprise, surprise – software engineer!  Software Engineers write computer code that directs your user experience on, Amazon Web Services, Alexa, you name it.  Amazon is currently seeking to add 3,135 software engineers at its Seattle headquarters, HQ1.

Amazon Software Development Greater Seattle 3135 full time postings (cropped)

Other high-skill jobs performed today at Amazon include: Data Scientist (Amazon currently lists 2,040 job postings at HQ1), Technical Program Manager (1,923 job postings at HQ1), IT Support Engineer (1,523 job postings at HQ1), and Project/Operations Manager (1,333 job postings at HQ1).  I could continue but I’m sure you get the idea.

Of course, Amazon’s biggest challenge in continuing its awesome growth is filling so many high-skill jobs with highly qualified workers.  These five job titles alone add up to 10,000 employees that Amazon very much wants to hire – right now, just in Seattle!

We also need to consider the jobs of the future that are growing rapidly today.  Well, you’ve probably heard about Artificial Intelligence and Machine Learning.  The skinny on these fields is that in the near future we won’t need as many Software Engineers to write code to direct our computer interactions.  Instead, humans will be able to train computers to use their deep experience to determine what works where and when.  And so computers will direct your computer interactions.  These jobs of today but especially tomorrow are already in high demand: Amazon lists 117 job postings in AI and 829 jobs in Machine Learning – nearly 1,000 more employees that Amazon sorely wants to hire right now, just in Seattle!

I keep saying “just in Seattle” because Amazon already has up to 20,000 high-skill employees in Software Engineer, Data Scientist, etc. at dozens of other Amazon locations in North America.  To use a baseball analogy, Amazon already has a minor league system throughout North America.  But instead of calling top talent up to the existing Major League team in Seattle, Amazon has wisely decided to start a second Major League team, aka HQ2.

So I investigated how Amazon’s minor league system is functioning.  Of course, Amazon seeks to hire large numbers of high-skill employees in the cities where the company has seen the best job performance.  If you will, Amazon is already betting on certain cities and its choice of HQ2 will be a colossal doubling down on the city with the best demonstrated job performance or, conceivably, the city with the best prospects of job performance.

So on what cities is Amazon betting today?  That is, from what city or cities would Amazon today like to hire thousands of people to perform these high-skill jobs of today and tomorrow?  And what one city’s talent pool is so deep that Amazon can hire not just thousands but tens of thousands of high-skill IT employees?  

The envelope please …

Amazon Postings by Job Title and City

Ok, well, you know, technical difficulties!  Unless you have 20/10 vision, here’s the simpler spreadsheet with just the finalists …


Amazon Postings by Job Title and City Finalists

The simple answer is that Amazon is betting on San Francisco because Amazon wants to hire thousands of high-skill people in the SF Bay Area.  No other North American city has 1,000 such job postings.  Not fair, right?  They always win.  Yes, it’s called a business cluster.  And a business cluster strengthens as long as the region stays on top of its game.

The information technology business cluster nicknamed Silicon Valley (San Francisco in the north to San Jose in the south) has been generations in the making — dating to Fairchild Semiconductor (1957), Hewlett-Packard (1960), Intel (1968), ARPANET (1969), Xerox PARC (1970), Kleiner-Perkins (1972), Sequoia Capital (1972), Apple (1976), Oracle (1977), and literally too many more to mention.

Amazon’s bet for today’s operations is best captured in the first four rows (Software Developer, Project Manager, IT Support, and Program Manager), where San Francisco has roughly 1/5 of HQ1 job postings and Boston and Vancouver each have roughly 1/14 of HQ1 postings.

But notice how Amazon’s bet for tomorrow — AI and Machine Learning — is even more concentrated in two cities — San Francisco (233 job postings, which is 25% of HQ1’s job postings in AI & ML) and Boston (152 job postings, 16% of HQ1).  For Austin, LA, NY, Toronto, and Vancouver, their percentage of combined AI and ML postings relative to HQ1 falls below their overall percentages shown.

Because San Francisco & Boston are Amazon’s top two choices for today and tomorrow, I read, in full, their HQ2 bids:

Bay Area regional proposal for Amazon HQ2 (amazonbid.pdf) can be found at:

Massachusetts’s Amazon HQ2 Bid (Day1Massachusetts_US.pdf) can be found at:

The most interesting nugget from either document relevant to Amazon’s HQ2 was this Tech Job Growth chart from page 106 of San Francisco’s amazonbid.pdf:

Tech job growth 2013-2015

Figure 2 above, a Brookings analysis of Moody’s data, printed in the Wall Street Journal on 9/21/2017, shows that the San Francisco Bay Area — the two Metropolitan Statistical Areas around San Francisco and San Jose — added 60,000 tech jobs in three years.  This +20,000 net tech jobs per year is even more impressive because those two MSAs add to just 6,500,000 people.  New York needs three times the population to generate +8,000 net tech jobs per year.  For every other IT market, +5,000 net tech jobs per year is at or above that labor market’s demonstrated ability to fill technology jobs for all employers combined!  But remember that Amazon’s goal for HQ2 is +5,000 net tech jobs per year for 10 years straight, ultimately reaching 50,000.

At just 25% attrition (75% retention), Amazon HQ2 would need to hire 12,500 people a year in perpetuity to ultimately reach 50,000 employees (Geometric Series: 1 + .75 + .75**2 + … = 1 / (1 — .75) = 4 and 4 * 12,500 = 50,000).  By similar math, at 40% attrition (60% retention), Amazon HQ2 would need to hire 20,000 people a year in perpetuity to eventually reach 50,000 employees.  According to a 2013 survey by PayScale and the 2015 New York Times article ‘Inside Amazon: Wrestling Big Ideas in a Bruising Workplace’, Amazon’s attrition exceeded 25% in both those years.  Of course, a high quality employee is more likely to succeed in a job (less likely to be terminated) so choosing the best IT market will enable Amazon to reach full capacity at HQ2 sooner.

In reading the San Francisco Bay Area Amazon HQ2 bid, I saw the Bay Area emphasize its business success (2nd most US Fortune 500 companies behind New York City, the most leading technology companies, the most recipients of worldwide venture capital, the largest pool of technology talent, and the fasting growing pool of technology talent).  Of course, the SF Bay Area has more leading companies than you can shake a stick at — San Francisco proper has 2,800 tech companies employing 90,000 people — but I was also impressed by how the Fremont City Council unanimously agreed to allow Tesla to double the size of its factory.

The most eye-opening aspect of the SF Bay Area HQ2 bid is the 42 programming school in Fremont that — free of charge to 10,000 people aged 18 to 30 — teaches young people to program in C and other languages.  For people who don’t get accepted to a UC branch, it’s a student-loan free path to a technology career.  For employers, it’s an even deeper talent pool.

In contrast, in reading Massachusetts Amazon HQ2 bid, which I admittedly read second, I saw an overwhelming emphasis on all the colleges and universities in Massachusetts.  Sure, Harvard and MIT are two of the best universities in the world, but so are Stanford and UC Berkeley.  Beyond those two amazing Cambridge universities, half the leading universities in Boston are liberal arts schools that do not send their graduates to STEM (science, technology, engineering, math) careers at Amazon.  The SF bid included: “Stanford and UC Berkeley alone have produced entrepreneurs receiving $9.5 billion in venture funding that helped to found more than 850 companies since 2009.”  Nowhere did the Massachusetts bid quantify the success of Harvard and MIT entrepreneurs in the Boston area.  Yet Amazon’s future success depends on the company continuing to recruit and retain people who think and work like entrepreneurs. 

What I was also looking for in the Massachusetts bid that I did not sufficiently find was a proof of business concept similar to what I saw in the SF bid.  By and large I didn’t see technology companies like Amazon choosing Boston.  I did see Mark Zuckerberg, a former Harvard student, claim that he could have built Facebook in Boston.  Really?  Would a 20-year have received millions in VC as soon as he arrived in town if that town wasn’t Palo Alto?  Outside of Silicon Valley, would a kid turning 21 score $13 million in venture capital long before the site ever went live?  Would that kid have top IT talent working on his business without that influx of money and third party verification?  (No)**3.

Mark Zuckerberg’s revisionist history doesn’t bolster Boston’s case.  Instead the history of Facebook, the world’s fourth most valuable company, demonstrates just how much a business cluster matters.  A brilliant Mark Zuckerberg chose the SF Bay Area.  Maybe he made that “I could’a done it in Beantown” claim while receiving his honorary degree from Harvard.  He and his wife Priscilla Chan do a lot of good so let’s give him the benefit of the doubt.

But speaking about doubt — I recently read MIT Technology Review’s ’50 Smartest Companies (Worldwide)’.  The San Francisco Bay Area is home to 13 of these companies (26% of the list) with a total market capitalization of $2.4 Trillion, including 3 of the 5 highest-valued companies on Earth.  In stunning contrast to SF Bay Area, the Boston Area side of the ledger looks to have been completed by a Boston accountant: 3 nascent private companies, an aging behemoth with no connection to Boston other than a $25 million property tax incentive, and a modest 25-year old company that wouldn’t be highlighted in Silicon Valley.  Clearly, the San Francisco Bay Area’s technology advantage over Boston is far greater than the 13 to 5 score that the MIT Tech Review would have you believe.  A more accurate accounting would be San Francisco 7 — Boston 0.  Given the business ecosystems in which they live, among the 6 prospects in SF and 3 prospects in Boston, would you rather work and own stock in a prospect in San Francisco or one in Boston?


MIT 2017 Tech Review SF vs Boston

If a hot, new IT development takes place (and it always does) and Amazon has taken the New York Times advice to put HQ2 in Denver or Moody’s advice to choose Austin, both fine minor league markets, can Amazon hire the talent (thousands of the newly valuable highly-skilled people) in those remote locations to try to catch up to what is already being accomplished in the San Francisco Bay Area?  Can a company located in Denver or Austin today recruit even just one hundred people with experience in Artificial Intelligence and Machine Learning?  The New York Times October 22, 2017 article on ‘Scarce AI Talent’ clearly says No!  
That is one more rationale for planting a business within that industry’s business cluster — access to top talent as soon as the talent becomes available.  Staying in front of the curve, not falling behind it, is Job 1 in information technology.

The Amazon job posting results for Software Developer, Data Scientist, etc. are seconded by the thorough CBRE 2017 Tech Talent Scorecard, which ranks SF #1 with a score of 81.28.  Seattle ranks a distant #2 at 67.83, NY is #3 at 64.21, and Washington, DC is #4 at 64.13.  No other North American city scores above 60.  For instance, in a component of their rankings, CBRE shows that San Francisco has, by far, the most net technology brain gain — almost 4 times the brain gain that accrues to Seattle.

V BrainGain SF Bay Area

My review of the 50 Companies with the Highest Market Caps (> 120 Billion USD) shows that Information Technology is now central to a company creating wealth for its employees and investors.  Seven of the 11 highly-valuable companies created 1-2 generations ago (25-50 years ago) are IT Companies.  Practically all of the most valuable companies created in the last 25 years are IT Companies.  Expect this pattern to intensify.  Apple, Google, Microsoft, Amazon, Facebook, and Alibaba (China) have the top 6 Market Caps, will expand their IT Operations, and will take business from companies that are not IT Companies.


50 Market Caps by age, IT or not IT

So let’s conduct a final analysis of each of the seven finalists for HQ2:

San Francisco’s labor market ranks first for its demonstrated capacity to perform Amazon work (Software Developer, Data Scientist, etc.).  San Francisco ranks first in North America, by far, at adding tech jobs.  Per my analysis of MIT Tech Review data and my analysis of the 50 Companies with the Highest Market Caps, San Francisco is, by far, the most entrepreneurial IT market on Earth.  The San Francisco Bay Area has some of the best universities in the world: Stanford, UC Berkeley, UC San Francisco, Wharton West, and Carnegie Mellon West, and the free programming school 42.  Further, the San Francisco Bay Area benefits from as much brain gain as the next four North American cities combined.  San Fran’s high quality IT market would guarantee that HQ2 becomes the equal of HQ1, not a stepchild to Seattle, where Jeff Bezos and the S-team work.  Finally, the San Francisco Shipyard and adjoining Candlestick Point (780 acres and up to 4,300,000 square feet) and the Concord Naval Station (2,300 acres and some portion of 6,100,000 square feet) are large, promising sites on which Amazon could build HQ2.  San Francisco’s drawback is that the SF Bay Area is 20% more expensive than Boston and 40% more expensive than Vancouver.

Boston’s labor market ranks a distant second to SF in its demonstrated capacity to perform Amazon work.  Boston has a small advantage vis-a-via the Bay Area in the overall cost of labor and real estate.  Boston has two of the best universities in the world — Harvard and MIT — and its many universities grant STEM degrees in numbers comparable to universities in the SF Bay Area.  So does Boston have any potential upside in IT relative to SF?  Without Amazon’s HQ2, No.  But with Amazon’s HQ2, Yes!  You see, Boston has trouble retaining its graduates.  Look above at the brain drain that Boston suffers!  Amazon’s HQ2 would entice many Boston-area STEM graduates to stay, enabling Boston to dramatically increase its net tech jobs per year. And remember from my analysis of MIT Tech Review data, there is little competition for IT talent in Boston relative to San Francisco (Boston is more biotech than info tech).  These new graduates would cost Amazon less than traditional IT labor and could be guided by existing Amazon employees in Seattle (and San Francisco) who are more experienced Software Engineers, Data Scientists, Project Managers, etc.  So even though Boston didn’t crack the top 10 cities in Tech Job Growth 2013-2015 and ranked only 9th in CBRE’s 2017 Tech Talent Scorecard, Beantown has legs!  And the two keys to unearthing Boston’s potential as HQ2 were mining the right data from Amazon’s Career search engine and seeing Boston’s potential upside in a graphic highlighting SF’s tech brain gain!  Also in Boston’s favor, I know that Jeff Bezos would love to be the only big game in town, maximizing the uptake of his Amazon principles and minimizing employees’ opportunities to jump ship.  Further, Boston is the only top 3 finalist in the Eastern time zone, where half the people in the US and Canada live.  Boston’s bid offers small but centrally located sites at South Boston Waterfront (100+ acres adjacent to South Station), Harrison Albany Corridor (250 acres adjacent to red, green, orange, and silver lines), and Allston/Brighton (a small site near South Station corridor and Harvard).  Finally that Boston gave GE a $25 million property tax incentive for relocating a scant 800 jobs must make Amazon’s mouth water.  At that same rate, Amazon might expect Boston and Massachusetts tax incentives to exceed $1,500,000,000.

Vancouver ranks third for its labor market’s ability to perform the Amazon work, with weakness in tomorrow’s disciplines of Artificial Intelligence and Machine Learning.  But Vancouver has a labor and real estate cost advantage relative to Boston and especially San Francisco — enough to boost Vancouver’s case.  Also Vancouver is among the world’s most livable cities — the most livable in North America — although Boston and San Francisco are often the highest rated US cities, per Mercer and others.  Finally, choosing Vancouver hedges against the political risk that now exists in the United States.  For decades the United States had, by far, more brain gain than any other nation on Earth.  Many citizens of the world, if given half a chance, did whatever they could to become Americans.  Former President Obama expanded the Optional Practical Training program, which allows foreign students earning a STEM degree from a US university to work in the US for up to 3 years, long enough to earn an H-1B visa from an employer.  In contrast, President Trump plans to cut legal US immigration in half.  Per Inside Higher Edthe leading media company serving higher education, in one year the applications to many universities in Canada have risen by 20-40% while the applications from abroad to many universities in the US are down.  This brain gain now flowing to Canada will show up in IT job markets in 2-4 years, as soon as students start to graduate.  Canada is expanding immigration to record levels for each year through 2020.

US STEM programs also face competition for the best Computer Science and Computer Engineering students in China, India, and Korea from a new wave of start-ups in those countries — startups often funded by Silicon Valley-based VCs that have branched out internationally.  Further, as The New York Times details in ‘The Disappearing American Graduate Student’ printed November 3, 2017,  US STEM programs for Masters and Doctorates students have come to rely more and more on international students.  In 1994, 40% of graduate students in Computer Science were international.  Today 66% are.  Faltering international applications won’t damage Carnegie-Mellon, MIT, Stanford, and UC-Berkeley, the four best Computer Science grad schools per US News, but they will damage the remaining 200 Computer Science graduate-degree granting departments in the United States and the ability of US Companies to recruit IT talent.

New York and Los Angeles rank a distant fourth and fifth and even that status is due to their large populations.  Neither has any significant cost advantage to the San Francisco Bay Area and so New York and Los Angeles each lose to San Francisco.

Austin has only 1/4 the Amazon job postings of Vancouver and no cost advantage or any other clear advantage to Vancouver.  So Austin loses to Vancouver.

Toronto has 1/5 the Amazon job postings of Vancouver and roughly the same labor and real estate costs.  Because I predict that Amazon will double-down on a city that the company is already thrilled with — San Fran, Boston, or Vancouver — Toronto loses. 

But Toronto is my Long Shot — Doug Flutie 65-yard miracle pass on the last play — City.  Here’s Toronto’s case: we’re also gaining international brains and we’re much larger than Vancouver so we can gain more brains.  Toronto is home to 2,700,000 City and 5,100,000 Urban while Vancouver has only 600,000 City and 2,100,000 Urban.  The best university in Canada is University of Toronto and half the top 10 Canadian universities are in Ontario, per US News.  The only top 10 Canadian university in Vancouver’s BC is the highly-ranked University of British Columbia.  Also, Toronto is in the Eastern time zone, closer to where half the people in the US and Canada live.  Still, I think that Amazon trusts its considerable experience with its employees performance over the last decade.

So we are down to 3 cities: San Francisco, Boston, and Vancouver.  These three cities more than doubled the other finalist cities in Amazon job postings.  These three cities are remarkably similar to each other and to Seattle, ranking among the most educated, progressive, and livable cities in North America.  Amazon’s diverse work force would feel welcome in any of these cities.

Practically all data points to choosing San Francisco for HQ2.  If Amazon pays six-figures, it might as well hire in San Francisco, the city with the best demonstrated job performance at Amazon work.  But because Amazon could unilaterally benefit from stemming Boston’s brain drain, Amazon is just as likely to choose Boston for HQ2.  That is, Boston may be the city with the best prospects of job performance at Amazon work.  For cost, politics, and quality of life, Vancouver has about half the chance of either Boston or San Francisco.

Forecast:  Amazon is 40% likely to choose Boston for HQ2, 40% likely to choose San Francisco, and 20% likely to choose Vancouver.  Honorable mention: Toronto.

Updated Forecast: Amazon is 40% likely to choose Boston, 10% likely to choose Toronto, and 10% likely to choose Austin (Boston has 4X the positions of either Toronto or Austin).  Also Amazon is 40% likely to make a selection based on a criterion other than Amazon’s history with IT talent.  Jeff Bezos would love to be the only big game in town, maximizing the uptake of his Amazon principles and minimizing employees’ opportunities to jump ship.  Amazon’s goal of being the only big game in town derailed San Francisco.