What I learned about hiring at AWS
This is the second installment in my “what I learned” at AWS series. In my first post, I covered culture and the mechanisms that reinforce and scale it. This time I want to cover hiring. It was one of my bigger adjustments when joining AWS. It was also totally unexpected. I had nearly 20 years of experience when I started at AWS, most of that as an engineering leader. I had directly or indirectly hired hundreds of people and felt successful. This certainly was true in part — I made many great hires in that time. But I also made more not so great hires than I should have.
To understand hiring at AWS you first have to understand ‘the bar’ because every hire must ‘raise the bar’. The bar is the theoretical midpoint of the current employee population in a given role and level. For a candidate to ‘raise the bar’ you have to believe they will outperform 50% of the employees in the same role and level.
To me, this is simple genius. If you don’t believe you are raising the bar, then by definition you must be lowering it. Further, these hiring decisions compound over time. A series of bar lowering hires creates progressively more chance for other bar lowering hires. On the other hand, a series of bar raising hires pushes the bar higher over time.
Ok, raising the bar sounds like a great idea, but how can it work in practice? The bar is abstract and subjective and cannot be precisely defined.
A key part of assessing the bar is a consistent (and thorough!) structure for all interviews. Amazon assesses every candidate against a majority of the 14 Leadership Principles. Leadership principles are assigned ahead of time and each interviewer typically takes 1–2. To assess leadership principles we use a behavioral interviewing technique. These questions almost always start with ‘tell me a time when…’. For example, to assess Insist on the Highest Standards an interviewer might ask “tell me a time when you had to make a choice between quality and delivery”. The scope and impact of the examples that candidates choose help inform about the bar. If the example was about a minor defect in pre-release software, this probably indicates a lower bar. If instead, the example was about a major release with the potential to impact many customers it probably indicates a higher bar. Remember that both examples could still be bar raising because the bar is relative to the level of the role.
The second aspect of the interview is functional skills. Things like coding and software design for software engineers, product definition and pricing for product managers, and project management and technical depth for software managers. Again, functional skills are assessed against the bar for the level. For example, a principal product manager must demonstrate successful product definition on something more ambiguous, large, and impactful than a more junior product manager. Across both LPs and functional skills, each interviewer documents their feedback in detail and explains why they believe the examples do, or do not raise the bar.
Still, though, this interview structure doesn’t solve all the challenges of raising the bar. All interviewers try to assess the candidate against the bar in good faith. But most loops end with different data points — some inclined, some not inclined. This could be because the data points were different based on what each interviewer covered. It’s also possible that interviewers were not well calibrated to the bar.
The mechanism that Amazon created to address these challenges is the Bar Raiser. Bar Raisers have three roles on an interview loop — 1) they are an interviewer, 2) they drive a decision on the loop, and 3) they uphold the bar. Every loop has a Bar Raiser and the Bar Raiser must be from outside the management chain of the hiring manager. This last constraint is an important guard against conflict of interest that can happen when a leader is trying to make hires with interviewers that are in the same reporting chain.
The calibration to the bar and ultimate hire/no-hire decision happens at the interview debrief. Here, the Bar Raiser facilitates a discussion with the interviewers about all of the data points from the interview loop. The Bar Raiser and hiring manager must agree to make a hire. Further, the Bar Raiser can block a hire, even if the rest of the loop is unanimously inclined.
I was a Bar Raiser. I initially chose to become one because I wanted to give back to the broader organization. What I didn’t fully appreciate until I was a Bar Raiser, was just how personally rewarding the experience was. During the Bar Raiser in Training (BRIT) process you participate in 15–20 loops, each time paired with an established Bar Raiser (rarely the same one). Once you demonstrate sufficient proficiency in leading all aspects of the interview loop you graduate as a Bar Raiser.
My learnings as a Bar Raiser and BRIT clearly made me a better assessor of talent and cultural fit. This made me a more effective Hiring Manager and leader. I also gained a unique perspective into different roles and teams around the company. If you work at Amazon today and enjoy interviewing you should absolutely talk to a Bar Raiser or your manager about entering the Bar Raiser In Training process!
I have a few key observations about our hiring practices and the Bar Raisers. Early on as a Hiring Manager, I often felt that it was me vs. the Bar Raiser. In hindsight I was not well calibrated to the bar or the Bar Raiser role. I was frustrated at a string of several interview loops that all came back not inclined. This despite my best efforts to find the right candidates (and despite my often inclined vote!). This sentiment was naive and wrong, but I didn’t fully appreciate that until sometime later.
The real role of the Bar Raiser is to be a partner for the hiring manager and help them make the best decision for the candidate and the team. Bad hires are painful and costly to an organization, exponentially so with more senior roles. The interview process and Bar Raiser role is explicitly intended to avoid bad hires even if it means missing out on a few hires that may have been good ones. In other words, the process at AWS biases for false negative hiring decisions over false positive ones.
Here’s a tip that I feel helps with this notion of raising the bar. Someone that truly raises the bar should excite you as an interviewer. Pay attention to when you really feel that excitement and then articulate why in your feedback. Try to distinguish that from just feeling that a candidate meets the criteria and could do the job. The latter candidate may meet the bar but likely does not raise it and therefore should be a pass.
So, in summary:
- Think about how you try to uphold a hiring bar. Realize that if you aren’t raising the bar then you are lowering it.
- Be purposeful about interview construction. Make sure you are covering both you company core values/leadership principles along with functional skills required for the role.
- Make sure that interviewers have training on how to interview, and especially how to write good feedback
- Consider some type of Bar Raiser as a mechanism to ensure that hiring decisions are ‘bar raising’ ones
P.S. Lest you think the Bar Raisers are a power hungry bunch that relished the power of veto, every Bar Raiser that I ever talked to dreads the use of the veto. I had to use it once and it was quite literally the worst day of my entire tenure at AWS.
I’ll also leave you with my favorite interview question which is ‘What are you most proud of delivering’. It works in virtually every interview setting and tends to establish a good baseline for the type of scope and impact a candidate has shown. A good story about delivery will have numerous different aspects of decision making, quality, and technical depth. I often find that I can cover almost any assigned competency with the right follow up questions on a reasonably scoped example. If you have a favorite interview question, would love to see it in the comment stream!
