Experiments Illustrated: How We Optimized Premium Listings on Our Nursing Job Board

Also, how georandomization can help clean up spillovers The post Experiments Illustrated: How We Optimized Premium Listings on Our Nursing Job Board appeared first on Towards Data Science.

Mar 10, 2025 - 20:13

Experiments Illustrated: How We Optimized Premium Listings on Our Nursing Job Board

Running experiments is a task that often falls to data scientists. If that’s you, congrats! It can be a rewarding and high-impact area of work, but also requires tools found outside the typical ML-heavy data science curriculum.

Even with the best tools, only a small share of experiments deliver meaningful business value. I’ve been lucky to design and execute many experiments. Of those, I have a few winners. From these, I’m sharing some stories to illustrate key concepts related to experiments.

Multiple Comparisons & How Random Assignment Saved us $1M in Marketing Spend
Choosing what to test & How IntelyCare tested its referral bonus program into existence (coming soon)

Background: I work at a company called IntelyCare. We help connect nurses with various work opportunities (full-time, part-time, contracts, per-diem… the whole menu).

One of our core offerings is a nursing-only job board. If you take a look in the year 2025, you’ll notice two possible ways of sorting jobs by date and by relevance.

Why it matters: The sort-by-relevance feature is our current best lever to guarantee a good experience for paying customers. It also gives us an opportunity to improve the overall efficiency of our job board by steering eyeballs away from low-quality jobs.

Unfortunately, we can’t put every job at the top of a search result. We face a tradeoff between the quantity of top-page listings and the quality of the experience in the form of increased applies.

How it works: “Relevance” doesn’t mean what it normally means. Sorry!

We give each job a score between 0 and 100. When filling a page with jobs, sorting by relevance means we sort the results by that score. That’s it! For brevity, we’ll say any job with a score higher than 0 is “boosted.”

I know what you’re thinking, “This isn’t relevance!” And you’re right, at least in the normal sense of the word. The score doesn’t vary across job-seekers or search terms. A better name would be “relevant to Google.” We’re OK with that because a huge share of our job-board traffic comes from Google, as shown below.

Screenshot of Google search results — “Sort by relevance” here is shorthand for “relevant to Google.” (Image by author)

In Math: We have N jobs. Every day we generate a vector of N integers between 0 and 100. We feed this vector into a black box named Google. If we do a good job, the black box rewards us with many job applications.

By putting the “right” jobs at the top of the page (loaded word there), we can improve upon a chronological sort. Before we can identify the right jobs, we need to know how much Google actually rewards higher-placed jobs.

Day 0: Making progress when you know nothing

Sometimes, just to justify all the simplifying assumptions I’m going to make later, I start a project by writing down the math equation I’d like to solve. I imagine ours looks something like this:

S is our vector of relevancy scores. There are N jobs, so each s_i (an element of S) corresponds to a different job. A function called applies turns S into a scalar. Each day we’d like to find the S that makes that number as large as possible — the relevancy scores that generate the greatest number of job applications for intelycare.com/jobs.
applies is a fine objective function on Day 0. Later on our objective function could change (e.g. revenue, lifetime value). Applies are easy to count, though, and lets me spend my complexity tokens elsewhere. It’s Day 0, people. We’ll come back to these questions on Day 1.
Problem. We know nothing about the applies function until we start feeding it relevancy scores. Read More