Welcome back to another edition of ‘it’s easier to write non-technical posts than technical posts’. I am working on another installment in the open-source circuit design series1, but the topic of today’s post has been top of mind for me recently. I’ve been invested in hiring for my team for a while now and I have Opinions. The proximal cause of this post was this tweet, and my subsequent discussion with James Sanders. Needless to say, I think that it’s possible to make a reasonably good hiring process, but like every (human) system, it’s going to be noisy and subject to other constraints (time, money, etc). It takes serious effort to do a good job and cut through the noise, but I don’t think I’d characterize these processes as ‘just guessing’.
Hiring
There are few skills more critical to any organization than good hiring practices. Unfortunately, in most cases, hiring well is extraordinarily hard to execute2. Conversely, poor hiring, and the obfuscation thereof is a much easier task which only falls apart when reality reasserts itself and your org crumbles to dust3.
The Problem
Let’s pretend that we have started a new, full-stack QC company. Hardware, software, the whole nine yards. The company is called ‘Oraqle’. Ideally, we are trying to fill critical roles in our organization with people who will not only excel at their particular professional responsibilities, but also enhance the performance of their coworkers AND be chill to work with. Keep in mind, we have to work with these people for years (decades?). Our hires absolutely must add business value4, or Oraqle is doomed5. Despite the fact that we’re choosing a co-workers for the next N years, we have to make these decisions after only a few hours of interaction. In some cases the interview process can be lengthy and last several rounds, but a drawn out process risks losing good candidates to faster moving orgs and takes time that otherwise could’ve been spent on technical work.
To recap: we need to hire a competent, more-or-less personable6 people to fill a critical roles in our organization. We have something like 8-127 hours of interview time per candidate in which to determine the scope and quality of the candidate’s skillset, as well as whether they’re capable of just being a normal person for 8ish hours a day.
If we choose poorly, we imperil the future of the company AND the futures of the actually good employees.
Solutions
One of the strange quirks of niche(ish) fields like QC, is that under some circumstances, the hard part of hiring well is just competing with the other major players in the field. Talent identification and evaluation ends up being the trivial part of the problem. You can, in a weekend, literally identify every single graduate student in your field with and enter their names in `hiring_pool.csv`. Chances are, if you are doing a good job maintaining your professional network, you already have the better part of this list in your head.
Unfortunately, if we’re in superconducting qubits, Google and IBM probably have a much fatter hiring budget than us. I wrote a little about salaries in QC back when Quantum Computing Report teamed up with Quantum Futures to post The Chart. You know what most of the feedback was about this chart in 2021? The posted base salaries were too low. My understanding of how these things work is that the actual cost of an employee (to us, the employer) is closer to 50% - 100% greater than the base salary we are paying.
OK, so even though we know exactly which domain experts are graduating in the next 12 months, any exceptional candidate will have no problem getting offers from the largest, most prestigious players in the field. Maybe we can compete on cost of living, or geography, or some third, mysterious thing8.
Even if we could secure a steady supply of new graduates from top academic QC labs, we run into the next problem. Namely, the fact that we could populate `hiring_pool.csv` almost from memory means that there aren’t that many new graduates with the appropriate background each year. My Very Bad Estimate™️ is roughly 130 new PhDs from QC groups in the United States every year. We probably want to hire the best of those, so call it the top 20%, which ends up with 26 solid candidates that everyone wants to hire (split across experiment, theory, superconductors, ions, neutral atoms, etc)9.
My criteria for ‘top’ in the preceding paragraph is simply ‘comes from a top academic group’, which is admittedly crude10. Fortunately for us, widening the net to encompass students from less prestigious groups and candidates from adjacent and not-so-adjacent fields unlocks many more talented individuals. The former are relatively easy to evaluate, as they are fellow QC experts and we can spot bullshitters from a mile away, but what about candidates from outside the field? It is at this point when our evaluation of potential new hires becomes much more noisy. Paradoxically, our ability to lure people from other fields is relatively good. We offer much higher salaries than other physics-centric jobs, and vastly more interesting work than being a Data Gremlin at MindPoison Inc, the hottest new social media startup.
Risk Management
Great! We’ve expanded our candidate search horizons. We get a text from a friend “You should hire Dr. Cortado, she studied blah blah vogon scattering blah blah beamline blah blah crystallography”. Our reply: ‘Sure’, but how the hell do we evaluate whether Dr. Cortado can do the job we have in mind? How do her skills fit into the org?
There’s an argument to be made here about just trusting our friend. Indeed, with certain friends I deeply respect and deeply trust, I would be willing to hire almost on recommendation alone. That said, whenever I refer candidates, I usually want them to be interviewed, because I value my reputational capital.
Homework
If we go with ‘trust, but verify’ then we’re back at the original problem. Dr. Cortado has an impressive array of skills and physics knowledge that is mostly gibberish to us. This is why I really like the ‘homework’ approach to interviewing candidates from outside the field. Problems must be carefully tailored to meet a few criteria:
Test for desired skills- Often this is some sort of domain knowledge, or ability to learn and synthesize quickly. Usually this means the problem can’t be solved without doing at least a little bit of digging through the literature. The homework phrasing should use words that, if entered into Google scholar, would immediately return the necessary papers as the top results. Importantly, we should expect candidates to go read the literature. Half this job is going to be reading papers and at least trying to keep up with the most interesting parts of the arXiv firehose.
Test for advertised skills- Dr. Cortado claims to be an expert in scientific computing. Can we add a little computational twist to the problem?
Make it interesting, and reward clear thinking- The best kinds of these problems will immediately hook desirable candidates. This is also known as the nerd snipe11.
Note that it’s not so important if the candidates don’t actually finish the problem, as long as they show their work. When I was on the receiving end of one of these, my actual answer was quite short, but my explanation of how I came to the answer and what parts I was still uncertain about was about 2/3 of the response length.
This approach has a lot of upfront costs, namely that it is not easy to concoct a series of good problems, but there could be substantial benefits. Not the least of which is that this approach automatically suggests a clear evaluation standard and frees up actual interview time for more relaxed conversation and ‘behavioral’ questions. I’m strongly considering implementing this approach in my own interview flow.
Read a Paper
In the same vein as the homework12 approach, we can ask the candidate to simply read a seminal paper from the field and essentially hold a journal club session, which the candidate leads with a short presentation summarizing the paper followed by discussion. I think this approach isn’t too shabby either, and really amazing candidates would likely be able to draw insights and ask questions that even seasoned veterans hadn’t considered before.
I’ve used this one before and it has been effective at sussing out confident bullshitters. I like to choose short, seminal papers in the field. Something a candidate could read in an hour or two overnight. We give ‘em about a week to prepare.
Seminar
Asking the candidate to present an hour long seminar talk on their work is a standard approach when hiring a PhD candidate. This is risky when the evaluators are narrowly focused, without experience in the candidate’s field, but can be OK if the audience has varied enough knowledge that they can ask interesting and insightful questions (and detect bullshit). I like this approach the most for candidates with experience in the exact field I’m hiring for, or something adjacent. A good seminar will allow the candidate to demonstrate both technical and communications skills using a subject in which she has deep expertise.
On-site Interrogation
Also known as the traditional interview. Software engineering and Finance famously ask many tricky mathematical and/or coding questions during these to detect good candidates. I’m not a huge fan of this approach, largely because I believe13 it lets too many good people slip through. It's not all bad, as candidates who do well on these are usually willing to spend quite a bit of time practicing to prepare for the interview, which is a sign of conscientiousness, a quality we want in our future colleagues.
Limitations
I’ve listed only a few of the options available to us, here. Interviews can span multiple days and incorporate one of more of these elements. Since we are starting from scratch, we are able to fully control the process and define every facet of the candidate funnel and interview pipeline.
In larger, mature orgs, this is not as feasible. If our quantum group were embedded in a much larger multinational corporation, it’s a good bet that our hiring practices might be limited by the guidelines of the overall HR org. It’s possible we could still define much of the interview process, but things like compensation bands, # of interview days, resume filtering, candidate selection, etc could be out of our hands.
Also, much of the above is for hiring people with PhD levels of experience, or more. Hiring undergrads, or Master’s havers is a different beast. These people have less direct experience, but more potential. I’ve written before that it’s actually quite hard for a BS holder with 5ish years experience to match her new PhD-having colleagues without substantial effort and (probably) good mentorship.
I’m not entirely sure on how to detect high-potential candidates coming right out of undergrad. Perhaps one useful metric is whether they’ve been admitted to a PhD program? Maybe the best thing to do is have loose hiring criteria and initial projects where excellence through hard work and initiative becomes immediately obvious. Although graduate programs have a similar problem, us industrial physicists benefit from the attrition, down-selection, and proof-of-work14 that completing a (good) PhD provides.
Long Term Thinking: Metrics and Standardization
One thing we could try that seems pretty rare, is implementing a more rigorous, standardized interview flow where hiring decisions are backed with reports from interviewers, and the HR organization is, in part, tasked with following up in 1 month, 2 months, 6 months, 1 year on the new hire’s performance. The idea behind this would be to eventually tease out answers to questions like 'who has the best nose for good hires?’15 or ‘which interview questions are the most useful? the least?’ As far as I can tell, having good business systems16 in place is an important component in the long term success of Oraqle (and our future prospects of owning a private island).
Of course, in order to propagate our brilliant hiring system, everyone who is assigned to act as an interviewer, or give hiring input, needs to go through our very thorough training course which clearly lays out the logic behind the system and the expectations for all employees engaged in the hiring process.
What I’ve outlined above would require an enormous amount of human effort to architect and implement. Our small startup can’t afford this kind of overhead, but we can at least put in some good practices so the future HR org has good foundations to build from.
Alternatively, we could cut through all of the bullshit and just implement my favorite piece of advice:
Hire good people. Don’t hire bad people.
Sadly, I spent probably 10 hours trying to kluge together conda and pip- installs of various packages, which just ended up not working. I’m taking a little break from that effort until I’m ready to be hurt again.
I suspect this is true of firing well, too.
This is more true in large orgs than small ones.
Acting as a net negative is not automatically fatal in very large organizations, which can absorb a substantial amount of dead weight before finally collapsing.
Your dreams of summering in at you private estate in Tuscany will stay dreams.
At least, not a total monster. Do you REALLY want to hire Fred, the guy who clips his nails in at his desk, even if he’s a genius?
Even relatively short interviews can add up! For rapidly expanding teams, it’s possible to spend entire business weeks interviewing candidates in lieu of doing technical work.
If you’re saying ‘equity’, I will remind you that Google equity is highly liquid and has demonstrated market value. Our equity is highly illiquid and is usually only repriced after a funding round. Also, common advice is value startup equity at essentially $0. For example, see ‘How to value an equity grant’ section of Patrick McKenzie’s blog post on career advice.
My extremely dumb methodology is as follows. Official APS membership for 2023 is 49,701. DQI membership for 2023 is 3,539 people. That means DQI is about 7% of the total APS membership. We also know that, in 2020, about 1,905 Physics PhDs were granted in the United States. So if you stupidly assume the fraction of PhDs from QC labs is proportional to the size of DQI within APS, then you get something like 133 new PhDs in DQI every year. This is already an overestimate, since it’s possible to be a member of multiple APS units, but bear with me. We want students from the top labs, or maybe the top 20% of students? That brings us down to ~26 new PhDs that everyone is going to want to hire.
Please don’t send me email about how there are plenty of smart people working in obscurity in Prof. Schmuckateli’s lab at Big State U. I know.
I actually lost about 5 - 10 minutes to this comic while writing this post before I snapped out of it.
Incidentally, if it strikes you as immoral or unfair to ask candidates to do what could be a substantial amount of work just to get an interview, you could certainly offer to compensate them for their time. I generally sympathize with this argument for less-skilled positions, but if a candidate is applying to a highly compensated position without direct experience with the work, I don’t think it’s unreasonable to expect some time investment on their part.
But can’t prove
We’re reclaiming the phrase.
These people are insanely valuable. I work with someone like this and every bad hire we’ve made has been in opposition to her gut feeling about the candidate. I’ve learned to trust the gut.
It’s the job of executive leadership to make sure the various incentives for the systems and sub-systems don’t become misaligned and lead to dysfunction. So.. we should practice being good at that.