We just shipped a new Spark Hire AI feature that has our customers buzzing – AI Video Review.
At a high level, AI Video Review scores candidates’ one-way video interview submissions against several key factors, revealing insights that are correlated with pass rates at this stage in the hiring process.
Given the excitement about this feature and the fact that I’ve talked a lot about our AI narrative, I thought it’d be interesting, and in line with our motto around AI transparency, to share a glimpse at what went into delivering this solution.
But first, a quick TLDR on our AI narrative:
- Our customers are not hiring with AI, they’re using AI to hire.
- AI should be used to highlight what matters.
- Problem-first, not feature-first.
- Compound across features on our solutions.
Okay, let’s get to the good stuff.
Table of contents
The Why: A Problem-First Approach
The Spark Hire AI roadmap is problem-first, not feature-first, and centered around two core, strategic “problem themes”.
Here’s how AI Video Reviews aligns with these themes:
Problem Theme 1: Candidate Screening & Selection
Due to a variety of market forces, our customers are inundated with job applicants, all with unique qualifications and interest levels.
One-way video interviews enable them to “hear the stories” that resumes can’t tell without time, scheduling, and resource constraints.
This actually allows organizations to consider a lot more candidates then if they were spending all day sorting through eerily similar resumes.
Now, let’s say you’re a lean hiring team inviting a lot of candidates to complete one-way video interviews. In this case, you’re probably batching your review process, which means you sit down to review a bunch in chronological order a few times per week.
The challenge is that without prioritizing which submissions you watch first, the best interviews might be waiting for your review. At the same time, those candidates might be progressing quickly with other employers.
Bringing this all back to today’s topic, AI Video Review provides insights on key factors that correlate with one-way video interview “pass-rates”, allowing our customers to prioritize reviewing highly scored interviews so they can move candidates with high potential along faster.
Problem Theme 2: Hiring Collaboration
One-way video interviews typically happen very early in the hiring process.
Sometimes hiring managers evaluate submissions, but other times, managers and other stakeholders don’t participate until much later in the process.
In the latter scenario, hiring stakeholders want context on what’s already happened with the candidates they’re meeting with.
AI Video Review, in conjunction with other features like AI Summary, enables HR and talent acquisition team members to easily share the whole picture of the candidate with the hiring committee, so they’re not wasting their time or the candidates on questions that have already been answered.
This is by no means an extensive list of the challenges we aim to solve with AI Video Review, but they provide a starting point for why we prioritized this feature now
The What: Having a Compounding Impact on Your Screening Process
With these challenges in mind, we turned to: how do we actually solve them?
Yes, we knew directionally that AI Video Review was the feature, but what would AI Video Review actually shine a light on?
This goes back to 3 more principles from our AI narrative:
1. AI should be used to highlight what matters.
With AI Video Review, we wanted to provide signals on factors that were…
- Validated by research
- Relevant to a wide range of jobs
- Rooted in customer needs and requests
- Aligned with the motivational competencies from our Predictive Talent Assessment
While they’re all important, here’s a bit more context on the last point.
Through 400+ research studies, Spark Hire’s Predictive Talent Assessment is proven to help you identify if a candidate has the behavioral competencies to be successful in the role, vs. just looking at a snapshot of the skills they have today.
As Tracy St.Dic from Zapier said in our recent webinar together, “you’re looking for where a candidate is at on a curve, not a single point.”
Once we determined a series of factors we were interested in, validation studies were performed.
We’ll get more into the science behind this a bit later, but the final factors that are included in the initial version of AI Video Review showed statistically significant correlation with one-way video interview “pass rates,” indicating that these factors “mattered.”
2. Spark Hire customers are not hiring with AI, they’re using AI to hire.
The final candidate evaluation and hiring decision are yours to make.
We’re simply giving you more insight so you can make that decision with more confidence.
We’re helping you prioritize so you act faster on top video interview submissions and get back to candidates sooner.
We’re enabling you to collaborate more easily using data.
3. Compound across features on our hiring solutions.
If you’ve been following our journey, you know that Spark Hire is more than what we’ve been historically known for (video interviews).
If you didn’t know, here’s the short version of Spark Hire’s recent history:
- In 2023, we acquired an ATS.
- We then transformed our video interview solution into a multi-assessment offering, which includes our Predictive Talent Assessment and automated reference checks.
- Today, our collaborative hiring ATS is known as Spark Hire Recruit, and our comprehensive assessment solution is Spark Hire Meet (with all of Meet’s capabilities available inside of Meet to provide customers with hiring tools across their entire hiring process).
So, naturally, Spark Hire AI must compound across Meet and Recruit.
What that means is AI features in Meet and Recruit, along with non-AI features, work together to improve hiring outcomes.
At a basic level, AI Video Review insights are fed back into the ATS and captured as part of your overall candidate evaluation.
The How: Making AI Video Review a Reality
Once we knew what we were building, the process of building and shipping AI Video Review to customers was a full-company effort. Here’s how it went down:
First, and at the forefront of everything we do with AI, this feature was developed through a careful, scientific process to ensure accuracy and fairness while upholding data privacy standards.
If some of this sounds really “sciency”, that’s because it is. Our COO, Michael Callans, is our in-house psychologist and leads research at Spark Hire.
Step 1: Defining what to measure
We first selected behavioral competencies and qualities based on these criteria:
- Validated by research
- Relevant to a wide range of jobs
- Rooted in customer needs and requests
- Aligned with the motivational competencies from our Predictive Talent Assessment
- Suitable for consistent scoring from a single one‑way video interview response
We started from a list of about 20 factors, and worked through a process of elimination.
Through the process described below, which included alignment testing of AI scores with human reviewers and with real-world hiring decisions, we narrowed down the list to six factors that provide powerful predictions.
Step 2: Creating a reference dataset without customer information
A set of 500 interview transcripts was assembled and redacted for engineering and initial testing.
No customer data was used.
Before any analysis, the system removed personal identifiers such as names and locations so neither humans nor the AI could view candidate PII.
Step 3: Establishing a human benchmark
We trained reviewers independently, who then scored every interview transcript. A consensus rating was recorded only when at least half of all evaluations for each factor were in agreement, producing a reliable and consistent baseline.
Step 4: Engineering and cross‑validation
Next, a model was developed for rating question and response pairs for each factor. A five-fold cross‑validation protocol (a method that splits data into five parts to validate accuracy) confirmed that AI scores aligned with human consensus across all factors. This ultimately demonstrated that the model could serve as a first‑pass reviewer.
- Across factors, AI matched human ratings with a hit rate above 90% in clear Good/Bad comparisons.
- All factors demonstrated statistically significant relationships between AI predictions and human ratings in mixed-effects regression models, which test for differences across individual candidates. These results confirm that the AI scores consistently contribute meaningful explanatory power.
- Strong correlations were found between AI scores and human raters across multiple factors.
What does all this science mean? The model can reliably approximate human review standards while delivering the speed and consistency benefits of automation.
Step 5: Outcome validation with historical data
To test whether scores actually anticipate hiring evaluations, we utilized aggregated, anonymized interview records, containing real pass/fail outcomes, to test predictive power.
The dataset included 68,236 interview submissions with a pass/fail outcome, and correlations were calculated between AI factor scores and the “Pass” decision.
The results were:
- The factors we decided to use in the initial version of AI Video Review showed positive relationships with Pass outcomes and thus, their reliability was reinforced.
- In an analysis of job families that had significant representation in the dataset, positive correlations were found for all job families. These include:
All this to say, an overall weighted one-way video interview score, built from this data , demonstrated a statistically significant link to hiring outcomes, confirming the model’s value in surfacing candidates who are most likely to succeed.
Step 6: Adhering to fairness and candidate privacy
At Spark Hire, we develop our AI features with fairness in mind, aiming to reduce bias throughout the hiring process.
AI Video Review, as an interview-ranking feature, is designed to evaluate objective information by objective criteria, redacting details like name and the candidate’s location to reduce the risk of bias.
To ensure these features remain fair and aligned with best practices, we regularly conduct bias audits, which evaluate whether the models treat candidates equitably and provide consistent results across different demographic groups.
Our audit for AI Video Review didn’t find any adverse impact or significant differences by ethnicity, age, or gender.
Step 7: Design
During the last year we’ve been hard at work transforming Spark Hire Meet from a video interview platform to a multi-assessment solution. This foundation meant that we already had friendly and intuitive user journeys that help hiring teams review assessments efficiently.
We added AI Video Review to this existing framework, allowing users to filter and prioritize one-way video interview submissions by their AI Video Review score, while still allowing them the transparency to dive into the breakdown of each score.
We also believe in explainability as a core principle in designing AI as part of our AI Design Philosophy and know from our experience building other AI features, such as AI Resume Review, that recruiters and managers want to see the rationale behind each score. With that in mind, we created an option to simply click each score to get key insights that help understand why each rating was given.
Additionally, our Product Designer and UX Writer regularly collaborate to create a friendly design supported by clear and easy-to-read content. Each design is carefully tested in usability tests to make sure it is intuitive to users with different knowledge levels and needs. This is all the more important for a novel AI feature, which would be a new experience for all of our users.
Step 8: From vision to code
Behind every polished feature is an enormous amount of collaboration. Our Product and Engineering teams worked closely to translate research and design into real-world functionality.
From drafting specs and aligning with engineering feasibility, to obsessing over pixel-perfect UI and seamless integration into Spark Hire Meet, it was a full-stack effort.
We kept check-ins frequent, feedback fast, and quality non-negotiable.
And that’s just the build side. We’ll save the Go-To-Market coordination for another post.
The Launch: Product Release and Reviews
We’re super excited to have launched AI Video Review last week and can’t wait for future iterations.
I’m so proud of how our team pulled together to make this happen, and the excitement from our customers is incredibly motivating.
Not the mention, the anticipation about what’s next is thrilling.
And, our commitment to helping our customers use AI to hire is at an all-time high.
I always appreciate a “behind the scenes” look at how something is made, and I hope you enjoyed this peek into how we shipped AI Video Review.
P.S. If you’re interested, below is a video of Spark Hire AI on Meet for one-way video interviews.
