Indeed
Sourcing candidates with AI
The situation
During a 2023 company-wide reorganization process, Indeed’s Small-Medium Business Employer and Enterprise Employer teams re-organized into a single Employer team. This meant redundant products between these two teams had to be merged. This included a newly merged Sourcing team had a lot of pressing needs:
- Clean up redundant SMB and Enterprise pages and make a unified Sourcing product for employers of all sizes.
- Deliver a robust roadmap of new features. Example: retire old candidate matching products and introduce a new tool with robust AI matching features.
- Address a larger audience with varied levels of hiring experience.
- Bring together multiple teams with siloed areas of expertise.
- Resolve a lot of conflicting opinions on terminology.
The challenges
Our stakeholders had strong—but often conflicting—opinions on terminology.
Quotes from stakeholder interviews
Don’t use match because Instant Match is already a branded term
Leadership wants us to stop using the term resumes and start using profiles
Let’s test Smart Sourcing as a new branded term for our product
Small business owners don’t know what technical terms like sourcing mean
Our audience is wary of emerging AI technology like Chat-GPT
UX research
I documented these terminology opinions as verifiable hypotheses in a structured format. I then led a User Understanding workshop with stakeholders from Product, Engineering, and UX to fill in the rest.
The final document had approximately 50 hypotheses, 40 marked as verifiable via UX research.
Ethnographic studies
I partnered with UX researchers to translate hypotheses to research activities. Wrote detailed protocol docs with content-specific questions around terminology and audience sentiment.
Insights from ethnographic sessions
- Employers of all sizes wary of concepts like “automation” or “artificial intelligence.” Negative connotations in the media, concepts associated with job loss.
- But they’re interested in trying out new tools that identifiably make their job more efficient.
- Aware of different types of AI, “AI I can play with, and other AI that works under the hood.” But often confused by specific acronyms like ML, LLM, GPT.
- Smaller business employers do their own research and learn the recruiting lingo. Independently use terms like “sourcing,” “ATS,” “disposition.”
Content Audit
- Audited candidate management experience, approx. 30 pages, 70+ emails and notifications.
- Identified terminology inconsistencies, namely how we talked about people throughout the process.
Insights from the audit
- Terms like “candidate” and “applicant” used interchangeably on the same screen.
- Redundancy in top-level pages, but unique content further down. More content within Enterprise, related to the larger feature offerings.
- Documented against existing content against user mental models. Noted which pages to update or delete.
- Adding for new use cases, including: Indeed's matching algorithm reviews candidate resume before a human does.
Gap Analysis
Insights from the analysis
- Documented against existing content against user mental models. Noted which pages to update or delete
- Add for new use cases, including: Indeed's matching algorithm reviews candidate resume before a human does
- Considering a rejected applicant for a new role
- One job seeker being considered for multiple roles
- Recruiter has resume on Indeed, but hasn’t applied to job
Conceptual Modeling
Existing conceptual model
- I brought in content designers from Qualifications and Job Seeker teams a content modeling workshop, based on Object Oriented UX framework.
- Divided concepts into “core experience” and “emerging technologies” to separate out new generative AI considerations.
Updated conceptual model
Insights
- We identified the existing linear framework (applicant-candidate-employee) built into our product code and content guidelines was insufficient.
- Content designers from other teams had similar concerns and needs to break out of old conceptual models
- Other CDs approved algorithm review as part of candidate, as long as there was human involvement as the next step.
- Content designers can lead key parts of the information architecture with stakeholder support
Content guidelines
I wrote new editorial guidelines for existing entries like “applicant” and “candidate.” If internal algorithms had analyzed their data and highlighted them in our machine for human review, can be considered a candidate.
New definition of “candidate”: a job seeker’s data has been actively reviewed by a human or Indeed algorithm.
Socialized new definitions in presentations with team leadership. Re-wrote style guidelines with example definitions. Led info sessions with Marketing and Support.
Wrote guidelines for new entries including “sourcing,” “generative AI” and “automation.”
Wrote specific content guidelines for Candidate Management products
Solidified algorithmic review as part of candidate process
Incorporated users sentiments towards AI and emerging technology
Reflected existing use cases in sourcing
Differentiated from upcoming generative AI work
Wireframes and prototypes
Iterative design tests with the product manager and UX designers on the team.
Tested out new versions of our text with users to address key concerns, such as clarity of the text, ability to quickly review the page for key actions, and know where to go for more details on individual candidates.
Then, I would update the UI text with clearer terms or additional info text to answer feedback from user tests.
1. New tab to switch views between people who have actively applied and those that have been matched by the new algorithm
2. New column to show the algorithm’s highlighting of key qualifications. Appears in both views to give hiring managers a comparable way to review candidates.
The Results and Key Learnings
Launched MVP Candidate Management page in Q3 2023 with 84% positive CSAT scores (benchmark: 78%)
Updated 12 Support pages, launched within same sprint as MVP
Feedback from product director: “A success case for a small design team working iteratively with limited resources […] the focus on content is high quality resource that other teams should consider.”
Served as the foundation for the updated Smart Sourcing, which was created to serve both enterprise-sized as well as small & medium business employers
Dedicated UX leadership and frequent collaboration throughout the process is key to success