The job market has a matching problem

A few months ago, I thought finding my first industry job after a PhD would mostly be a translation problem.

I knew I would have to explain why work done in particle physics matters outside particle physics. I would need to turn years of CERN-specific terminology into something an engineering manager could understand at a glance. And when I mentioned machine learning, C++, CUDA, GPUs, and real-time systems, I would need to show that I had built things with them, not merely encountered them in a course.

I expected that part.

What I did not expect was how hard it would be to get far enough into the hiring process to have that conversation, usually in the first technical interview after the HR call, if that call ever happens.

The routine is familiar by now: find a position unusually close to my background, tailor my CV, rewrite a cover letter, fill out another application form, and submit it. An automated confirmation arrives. Then, while I wait, I find another position and do the same thing again.

Sometimes the rejection comes a few days later. Sometimes it takes weeks. Sometimes nothing comes at all.

Repeat that often enough and you start debugging yourself. Maybe the problem is the CV, the location, the need for work authorization, the job title, or simply the hundreds of people who clicked “Apply” before you. You rewrite the same bullet point for the tenth time. You wonder whether “GPU real-time trigger” sounds too academic, whether “machine learning systems” is too broad, or whether moving “CUDA” higher on the page will somehow change the outcome.

Eventually, an uncomfortable question emerges. Companies say good technical people are hard to find. Qualified candidates say they cannot get interviews. What is failing between them?

The job market is supposed to connect companies that have problems with people who have the skills to solve them. Instead, we have built a system that is remarkably good at generating, screening, and rejecting applications, but surprisingly bad at creating actual matches.

Applying is cheaper than evaluating

It has never been easier to apply for a job. Candidates can find hundreds of openings through LinkedIn, company career pages, aggregators, mailing lists, and specialist job boards. Many applications take only minutes to submit.

Generative AI has lowered the cost further. It can rewrite a resume, tailor a cover letter, extract keywords from a job description, answer application questions, and produce several versions of the same application with little extra effort.

For an individual candidate, applying more widely is rational. If any one application has a low chance of leading to an interview, submit more of them. But when everyone makes the same calculation, a position that once drew 50 applications can draw hundreds or thousands. Response rates fall, so candidates apply to even more jobs. The loop feeds itself.

I am part of it too. But what other choice do I have? I have to apply for jobs to get interviews, and I have to get interviews to get a job. The probability is straightforward: without connections, the most obvious way to increase the number of interviews is to apply for as many relevant positions as possible. The system, however, is not designed to evaluate that many applications carefully. It is designed to screen them.

Consider some illustrative numbers. Suppose my chance of getting an interview for a given role is 5%, and, once I have an interview, my chance of passing every round and receiving an offer is 10%. If I apply for 200 positions, I can expect 10 interviews and one offer. At 400 applications, that becomes 20 interviews and two offers.

But if a company has the capacity to interview only 10 candidates, the growing application pool pushes in the other direction. My chance of an interview might fall to 2.5%, and my chance of an offer to 0.25%. The numbers are hypothetical, but they illustrate the problem: as the number of applications rises, each individual application becomes less likely to receive serious attention. Candidates respond by applying more widely, which adds even more volume.

I underestimated the transition

I am finishing a PhD in particle physics and trying to move into industry. For several years, I have worked within the LHCb experiment at CERN. Much of that work is closer to software and machine learning engineering than to the stereotypical image of a physicist writing equations on a blackboard, or even analyzing the experimental data we collect at CERN.

I work with C++, CUDA, Python, PyTorch, GPU computing, real-time data processing, scientific software, and machine learning. Part of my research involves integrating a deep-learning primary-vertex reconstruction algorithm into Allen, LHCb’s GPU-based real-time trigger software.

More recently, I have also been researching uses of LLMs in scientific software: fine-tuning, retrieval, agentic workflows, and ways to use language models with large technical codebases.

Outside my research, I contribute to open-source software and spend an unreasonable amount of free time experimenting with Linux, infrastructure, self-hosting, containers, networking, and whichever technical problem has captured my attention that week.

I never assumed that a particle-physics PhD would automatically qualify me as a machine learning or software engineer. I expected to prove that the engineering experience behind the degree was real. I did not expect the harder problem to be reaching a person who could evaluate it.

The application loop

The application itself has become a translation exercise. A phrase such as “developed reconstruction algorithms for a high-energy physics trigger” is meaningful inside particle physics. To someone scanning resumes for an ML systems engineer, it may say very little.

So I reorganize the CV, rewrite the bullet points, draft the cover letter, answer the questions, and submit. The confirmation email arrives immediately: “We have received your application.”

Often, this is the last specific thing I ever hear about the position.

The lack of feedback makes every part of the application feel like a possible cause. Was the CV too academic, or not academic enough for someone coming from a different background? Should I describe the project as machine learning, deep learning, or AI? Should I remove publications and add more software engineering? Did anyone read the cover letter? Did I need visa sponsorship? Or were there 400 applicants, with no guarantee that anyone reached mine at all?

Usually, there is no way to know. Without a useful feedback signal, optimization becomes guesswork.

What about the other side?

It is easy to be frustrated with companies. I often am. But I doubt most of this happens because hiring managers enjoy rejecting qualified people.

Imagine opening a position and receiving 700 applications. Some candidates are excellent. Some have nothing to do with the role. Some meet the requirements but have used generative AI so aggressively that it is hard to tell what they actually did. Some applied because one more click (the LinkedIn Easy Apply button) cost almost nothing. Others may be exactly right for the job but come from a field whose terminology the recruiting pipeline does not recognize.

Someone still has to find the right person in that pile, and nobody can read 700 CVs with equal care. Companies automate because they have to. Applicant tracking systems filter candidates. Recruiting software ranks them. Rules remove people based on location, work authorization, years of experience, education, previous job titles, or keyword matches. Employers are also experimenting with AI systems that summarize resumes, compare profiles with job descriptions, generate screening questions, and help decide whom to reject or advance.

The result is an odd symmetry. One machine helps write the application. Another helps decide whether a human should read it. This may make the process faster, but speed and accuracy are different things. A model can summarize a CV. A ranking system can compare hundreds of candidates. A filter can enforce a hard requirement. None of them can answer the question a company actually cares about: can this person solve our problems?

That answer rarely maps cleanly to keywords, job titles, years of experience, or a similarity score between a CV and a job description. The proxies are imperfect, yet the volume makes some kind of compression unavoidable.

Candidates face the same problem from the opposite direction. We see hundreds of jobs but cannot tell which companies would seriously consider us. Companies see hundreds of candidates but cannot tell which ones are genuinely strong. Both sides automate, and each new layer becomes another place where a good match can disappear.

Ghost postings make the problem worse. Some companies repost the same position again and again, giving candidates false hope that there is a real chance of an interview. I do not know whether these companies are serious about hiring or simply building a talent pool for future positions. Either way, candidates spend time and effort on applications that may never lead anywhere.

Are we optimizing for the wrong thing?

A great deal of technology has made hiring more efficient. The question is: efficient at what?

Job platforms make openings easy to discover. Application systems collect candidates. AI generates tailored materials. Applicant tracking systems sort or reject large numbers of submissions. Generative AI summarizes and evaluates them. Recruiting automation communicates with thousands of candidates.

Yet none of this necessarily makes it easier to answer the simplest question: should these two people talk to each other?

That is now the scarce event, perhaps one in 100 applications on average, or even fewer. Not listings. Not applications. Conversations.

Qualification is not discoverability

This has been one of the more uncomfortable lessons of my current search.

For most of my academic life, the answer to career uncertainty seemed straightforward: get better. Learn another programming language or technical skill. Understand the hardware more deeply. Build another system, publish another paper, contribute to open source, or take on a harder problem.

That instinct runs deep in academia. If you are not competitive enough, improve your qualifications.

Skills still matter. But qualification and discoverability are separate problems. You can have useful experience and fail to communicate it in a resume. You can communicate it clearly and still be application number 368. You can have the right experience but describe it in language a screening system does not associate with the role. You can fit a position whose screening criteria assume a completely different career path.

Or you can encounter someone better.

That last possibility matters. My difficulty finding an industry role does not mean companies are irrational, and it certainly does not mean I am entitled to a job. Many strong people are looking for work, and some roles will attract candidates whose experience fits much better than mine. The competition is real, especially in today’s market.

The feedback loop is broken

Job searching gets into your head because the feedback is both negative and low-information.

When a program crashes, you get a stack trace. When an experiment fails, you inspect the data and running conditions. When an ML model performs poorly, you have metrics. When an application fails, the usual diagnostic is: “We have decided to move forward with other candidates.”

That message cannot tell you whether your CV was weak, your experience was irrelevant, another candidate was stronger, an automated filter removed you, the company had an internal candidate, the role changed, the budget disappeared, a recruiter screened for something you lacked, or nobody meaningfully reviewed the application at all.

Still, it is almost impossible not to take each rejection as evidence about yourself.

You begin reading meaning into tiny signals. A job is reposted after the final interview, this happened to me this week. A recruiter takes three days to answer instead of one. An interview ends ten minutes early. A company views your LinkedIn profile.

Most of this is probably noise. When the outcome matters, noise is easy to overfit.

Even interviews are a noisy sample

Getting an interview changes the experience. Someone has looked at your background and decided that a conversation is worth an hour.

Then the preparation starts. You review algorithms you have not implemented by hand in years. You reread your own projects because familiar details can vanish under pressure. You study system design, practice explaining how your research applies outside academia, and try to guess which part of a broad job description the interviewer actually cares about.

A few questions, a whiteboard problem, or one clumsy explanation may then stand in for years of work.

From the company’s perspective, this is understandable. No employer can work with every candidate for three months before making a decision. At some point, it has to sample. An interview is a sample, but even a necessary measurement can be noisy.

The struggle

The current situation is more complicated than “the market is bad.” Companies continue to say they struggle to find strong engineers, machine learning researchers, infrastructure specialists, GPU programmers, data scientists, and other specialized candidates. People with those skills say they cannot get interviews.

That suggests at least part of the problem is routing, not simply a shortage on one side. The network has nodes, edges, and enormous traffic. The routing is bad.

Generative AI is turning this dynamic into an arms race. Candidates tailor applications to screening systems. Screening systems grow more sophisticated. Candidates get better at optimization. Employers deploy better classifiers and ranking tools.

As both sides improve at this game, the application itself becomes less informative. If every CV mirrors the job description, keyword overlap says less. If every cover letter is polished, polish says less. If every candidate can produce plausible answers to generic questions, those questions say less.

Both sides scale, and the signal-to-noise ratio gets worse.

Employers will naturally look for stronger signals: work samples, technical interviews, public projects, open-source contributions, writing, recommendations, and people they already know. In that sense, a highly automated hiring system may push us back toward something old-fashioned: reputation and human networks. That makes the odds even worse for people like me who rely on cold applications.

Why I’m writing this

There is an obvious reason this topic has been occupying my mind: I am currently looking for a job.

More specifically, I am finishing my PhD and looking for work in machine learning systems, GPU computing, research engineering, scientific software, AI engineering, high-performance computing, or a neighboring field.

I am drawn to software with real constraints: GPUs, performance problems, memory movement, large datasets, and the gap between an elegant idea on paper and a painfully slow implementation. I like machine learning systems where training the model is only part of the challenge, and the real work is making it reliable inside a much larger software system.

That is what attracted me to LHCb’s Allen framework. The LHC produces far more collision data than can simply be written to disk, so decisions have to happen quickly. Modern LHCb trigger processing makes extensive use of GPUs. Bringing machine learning into that environment means dealing with the details a Jupyter notebook lets you ignore: throughput, latency, data layout, memory transfers, inference overhead, GPU utilization, and the algorithm’s interaction with everything around it.

These are the engineering problems I want to keep working on.

I am also increasingly interested in LLM systems for scientific and technical software: fine-tuning, inference, retrieval, agentic systems, evaluation, and ways for these models to interact with large codebases without simply generating convincing nonsense.

The job title matters less to me than the problem. “Machine Learning Engineer,” “Research Engineer,” “AI Research Scientist,” “Scientific Software Engineer,” and “GPU Engineer” all describe parts of the work I would like to do.

But being qualified for these roles does not seem to be enough. I have to be discoverable too, and that is the part I am struggling with.

The job market has a matching problem. Right now, it does not match particularly well for me.