Part 2: If the Pitch is Too Smooth, It Probably Is: Why AI in HR is Difficult

Ernest Ng

Ernest Ng, PhD, serves as the VP of Strategy and Research at HiredScore, specializing in business strategy and AI-driven HR innovations. His extensive experience includes roles at Salesforce, The Walt Disney Company, and the California Department of Education. Additionally, he teaches HR Analytics as an Adjunct Associate Professor at the University of Southern California.

Ready to see what HiredScore can do for your team?

Request a Demo

If you haven’t read part 1 on disclosures, don’t fret. However, I recommend reading it after this one! 

This conference season has been a rich learning experience as I am now on the side of the vendor rather than that of the participant. The hype around Generative AI and AI as a whole is on full display in every HR tech vendor’s keynote including Oracle, Workday, and SAP, as well as industry conferences like Gartner, HR Tech, and Unleash. As someone who managed a data science, engineering, and analytics team in the past, one interesting observation for me is that all the AI use cases were very similar to one another with varying levels of impact and pragmatic application. All involve generating job descriptions, informational articles, goals, skills and learning matching, career pathing, and even attrition prediction as core use cases. So when I encounter that, I begin asking myself questions such as: 

  • Why are they so similar?
  • Why are they for show and not practical?
  • How should we think about implementing AI?

Why is it so similar?: The reason is data

Imagine AI like going on a vacation in a new place. It’s exciting, but also tricky to explore. When you do your research online, all the sites seem to suggest the same things. Tried and true controlled experiences that highlight the place, but is also not a place where locals would ever venture. It’s a tourist trap, an experience to show you’ve been somewhere, but not really experienced that place. AI for HR demos are just like that because certain aspects of AI are new and HR data is messy. So, vendors tend to pick use cases that are more straightforward for new AI functionality to try first, like use cases where the data isn’t a big deal or privacy isn't a big concern. This makes everyone see that AI can be awesome, even though it’s actually just doing something very narrow and not the full experience. 

When most HR professionals are asked the question, “how clean do you think your ATS and HCM data is”, many would answer and assume “fairly clean”. Every candidate has a resume, every employee has a standardized profile in the HCM, and you’re recording transactions regularly. As a result, these demos of AI working like magic are believable. But if you ask anyone with even the briefest experience in People Analytics they will likely tell you that your HR data is probably so disjointed and disconnected that building a quality machine learning model at the individual-level is extremely challenging. Factors like job code changes due to a job architecture project rather than an actual job change, a transfer due to a reorganization instead of an actual internal mobility event, constant manager or team changes due to attrition, tracking contingent workers, and the complexities of data harmonization if you changed/integrated an ATS, HCM, or LMS during the desired period all create machine learning obstacles. 

As someone who has spent over 15 years steeped in this data, I can tell you firsthand that actual structured employee data in most large, complex organizations is complicated, difficult, and so fragmented that applying a generalized model over that data will not produce the same results as promised in a demo. Thus, the demos all are about simple pieces of a greater complex process where the data isn’t dependent or connected to other data sources. 

Unstructured data poses an even more difficult challenge. Resume structures between Japan, China, India, US, etc. all have their own nuances (and, thus, the reason why HiredScore had to build its own globally-valid resume parser), and open-ended text fields are subject to translation complications (why HiredScore AI is built on 70+ languages and not machine translated into English). Simple use cases work well for smaller organizations, but as your business grows and your processes become more intricate, and changes occur within your systems and organizational structure, implementing AI becomes considerably more challenging. Therefore, having a significant track record of building and deploying AI in complex environments is crucial. 

Why is it difficult?: Timing

All these Generative AI use cases are point in time use cases. In other words, they aren’t as dependent on data that happened before or will happen after usage. The output isn’t dependent on the memory of what happened previously, and contextual understanding isn’t necessary. Don’t let generative AI steal your attention away from other AI/ML use cases that have much more tangible outcomes for your business. Just because you can have the AI produce an output, the timing of when that is produced and delivered is equally as, if not more important. 

For example, when it comes to internal mobility recommendations, you can produce the right recommendation to the right employee, but if it’s not delivered at the right time, it’s worthless. If an employee applies to their next perfect role, but there are already 3 candidates in final interview stages, that employee is not going to be considered. It’s all about getting the timing right, and that’s an AI/ML data problem, not a Generative AI use case. 

As an experimental psychologist, I can tell you that cross-sectional studies proliferated the literature over longitudinal studies only because they were cheaper, easier, and faster to run. Monitoring and maintaining a consistent structure over time is costly and difficult. If you’re not being asked by your AI vendor for historical data to tune the model, learn from the changes over time, and challenge your processes, chances are the model isn’t going to work well.   

If it’s easy to do, and the functionality is commoditized swiftly, perhaps it actually doesn’t have much differentiating value to your organization? 

I had a long conversation with an analyst recently who covers Generative AI in HR for a large analyst firm, debating whether an enterprise buyer would have continual use for these Generative AI functionalities to justify a monthly SaaS fee. He struggled to find the justification, and simply noted how exciting the technology was without actually thinking about how it would be applied, practically.  

What is for show and what is practical?: The answer depends on context

Let’s consider the first use case all these vendors mention for Generative AI: writing job descriptions. Here are some questions to ponder: 

  1. How often is a job description created from scratch?
  2. If it is created from scratch, are you going to populate it with the tasks or skills that job requires? (If so, this is a prime use case where Generative AI can help.)
  3. Is this new job description a net new role in the organization, or similar to other roles? How are you going to classify this role in your job architecture to level the compensation correctly?
  4. If its description is similar to other roles, why was it necessary to create it from scratch? If it is indeed a new role, there are many more steps needed to take in order to get the job leveled and budgeted properly before being able to post.

Needless to say, writing job descriptions is a practical example. If you don’t have many controls, like job leveling criteria and position management approval processes in place that prevent managers from simply posting job descriptions as they see fit, generative AI is a no-brainer. However, if you do have all those controls in place, as many large organizations do, is it really a time saver or just a fun party trick that won’t really be used practically? Even more so if you have standardization around prompts you have to add in before allowing it to generate.  

Let’s try another example: summarizing feedback, because I don’t personally think you should outsource the writing of your feedback to an LLM. For one, it will probably be more time consuming to phrase the prompts properly than writing it yourself, and furthermore, you should incorporate your own, original feedback. 

Let’s say you’re a manager with 6 employees and it’s performance review season. You’ve asked each of your employees to give a self evaluation and 3 sets of peer feedback to you. That’s 24 pieces of feedback you’re tasked with reading.

Some questions to ask:

  1. Should you read all 4 pieces of feedback for each employee to give your team the best possible performance evaluation?
  2. If not, what will happen if the employee asks for specific examples when you offer your feedback? 
  3. Does this example resonate with you because this is a real pain point, or do you simply dread the performance review process and want to get it over with as soon as possible?  

So summarizing feedback is one example, but if your company wants people to value feedback as a gift and unique learning opportunity, it should be treated as such.  

How we should be thinking about implementing AI:

The real question you should ask yourself when it comes to AI is what outcomes you want to achieve, and then think about the technology necessary to achieve those outcomes. This is how HiredScore has thought about how we’ve designed our products.

For example:

Iconic Outcome: Capabilities Needed to Achieve Outcome:
Increasing recruiting capacity without increasing headcount • Automated resume scoring and prioritization
• SLA and nudges for recruiters and hiring managers
• Hiring manager self service tools
• Every requisition never starts at zero
Every requisition never starts at zero • Understanding of when a requisition is posted
• Candidate resurfacing at requisition post
• Candidate recommendations take into account relevancy for the role and likelihood of applying
• Automated reach out to apply
Making it easier to get a job internally rather than externally • Internal role recommendations delivered to employee or manager at the ideal time in the employee's life cycle
• Recruiter alert to look at high value internal
• Hiring manager alert to look at high value internal
• Facilitated communication between manager and hiring manager
• Alert to HRBP, Talent Manager, and current manager if employee isn't hired

The key here is unlocking all the personas that impact talent and ensuring that they are all driving towards the same objectives. Our brain can only hold 7 pieces of information in our working memory simultaneously and that limits our ability to understand all the moving parts of a complex system such as an organization. Fortunately, AI has the ability to hold billions of parameters in the model to enhance our ability to drive the best outcomes. But, be warned, not all AI is created equal, and not all AI vendors are the same. 

HiredScore is an AI-driven company. Our sole focus has been AI engineered for HR in the largest, most complex organizations for the past 11 years. We are not a technology platform with AI features that check the box on an RFP. Whether it’s Open AI, Anthropic, Hugging Face, Cohere, Inflection AI, etc., these AI companies are solely building AI models, and not trying to become systems of record or needing their own proprietary system of record to work. As a result, our focus is on using our AI to drive business outcomes for the customer and to maximize your existing HR technology stack. 

Implementing any piece of technology is difficult, but buying and implementing something like AI that will transform your business requires deep thought, real expertise, and a trusted partner. Here are some resources we’ve put together as you’re considering or accelerating your AI journey:

  1. World Economic Forum AI for HR toolkit
  2. Enterprise AI buyers guide

For those interested in better understanding your iconic outcomes, sign up for an Iconic Outcomes Workshop and we can help you prioritize and understand the highest value use cases for AI orchestration. 

Continue Reading
Keeping Up with the Evolution of AI for HR Governance: A Detailed Q&A Recap
How General Motors Delivers the Future of Recruiting with Workday + HiredScore
Webinar details
The contrarian view on skills-based organizations and skills tech

Part 2: If the Pitch is Too Smooth, It Probably Is: Why AI in HR is Difficult

By Ernest Ng
Ready to see what HiredScore can do for you?
Request a demo

If you haven’t read part 1 on disclosures, don’t fret. However, I recommend reading it after this one! 

This conference season has been a rich learning experience as I am now on the side of the vendor rather than that of the participant. The hype around Generative AI and AI as a whole is on full display in every HR tech vendor’s keynote including Oracle, Workday, and SAP, as well as industry conferences like Gartner, HR Tech, and Unleash. As someone who managed a data science, engineering, and analytics team in the past, one interesting observation for me is that all the AI use cases were very similar to one another with varying levels of impact and pragmatic application. All involve generating job descriptions, informational articles, goals, skills and learning matching, career pathing, and even attrition prediction as core use cases. So when I encounter that, I begin asking myself questions such as: 

  • Why are they so similar?
  • Why are they for show and not practical?
  • How should we think about implementing AI?

Why is it so similar?: The reason is data

Imagine AI like going on a vacation in a new place. It’s exciting, but also tricky to explore. When you do your research online, all the sites seem to suggest the same things. Tried and true controlled experiences that highlight the place, but is also not a place where locals would ever venture. It’s a tourist trap, an experience to show you’ve been somewhere, but not really experienced that place. AI for HR demos are just like that because certain aspects of AI are new and HR data is messy. So, vendors tend to pick use cases that are more straightforward for new AI functionality to try first, like use cases where the data isn’t a big deal or privacy isn't a big concern. This makes everyone see that AI can be awesome, even though it’s actually just doing something very narrow and not the full experience. 

When most HR professionals are asked the question, “how clean do you think your ATS and HCM data is”, many would answer and assume “fairly clean”. Every candidate has a resume, every employee has a standardized profile in the HCM, and you’re recording transactions regularly. As a result, these demos of AI working like magic are believable. But if you ask anyone with even the briefest experience in People Analytics they will likely tell you that your HR data is probably so disjointed and disconnected that building a quality machine learning model at the individual-level is extremely challenging. Factors like job code changes due to a job architecture project rather than an actual job change, a transfer due to a reorganization instead of an actual internal mobility event, constant manager or team changes due to attrition, tracking contingent workers, and the complexities of data harmonization if you changed/integrated an ATS, HCM, or LMS during the desired period all create machine learning obstacles. 

As someone who has spent over 15 years steeped in this data, I can tell you firsthand that actual structured employee data in most large, complex organizations is complicated, difficult, and so fragmented that applying a generalized model over that data will not produce the same results as promised in a demo. Thus, the demos all are about simple pieces of a greater complex process where the data isn’t dependent or connected to other data sources. 

Unstructured data poses an even more difficult challenge. Resume structures between Japan, China, India, US, etc. all have their own nuances (and, thus, the reason why HiredScore had to build its own globally-valid resume parser), and open-ended text fields are subject to translation complications (why HiredScore AI is built on 70+ languages and not machine translated into English). Simple use cases work well for smaller organizations, but as your business grows and your processes become more intricate, and changes occur within your systems and organizational structure, implementing AI becomes considerably more challenging. Therefore, having a significant track record of building and deploying AI in complex environments is crucial. 

Why is it difficult?: Timing

All these Generative AI use cases are point in time use cases. In other words, they aren’t as dependent on data that happened before or will happen after usage. The output isn’t dependent on the memory of what happened previously, and contextual understanding isn’t necessary. Don’t let generative AI steal your attention away from other AI/ML use cases that have much more tangible outcomes for your business. Just because you can have the AI produce an output, the timing of when that is produced and delivered is equally as, if not more important. 

For example, when it comes to internal mobility recommendations, you can produce the right recommendation to the right employee, but if it’s not delivered at the right time, it’s worthless. If an employee applies to their next perfect role, but there are already 3 candidates in final interview stages, that employee is not going to be considered. It’s all about getting the timing right, and that’s an AI/ML data problem, not a Generative AI use case. 

As an experimental psychologist, I can tell you that cross-sectional studies proliferated the literature over longitudinal studies only because they were cheaper, easier, and faster to run. Monitoring and maintaining a consistent structure over time is costly and difficult. If you’re not being asked by your AI vendor for historical data to tune the model, learn from the changes over time, and challenge your processes, chances are the model isn’t going to work well.   

If it’s easy to do, and the functionality is commoditized swiftly, perhaps it actually doesn’t have much differentiating value to your organization? 

I had a long conversation with an analyst recently who covers Generative AI in HR for a large analyst firm, debating whether an enterprise buyer would have continual use for these Generative AI functionalities to justify a monthly SaaS fee. He struggled to find the justification, and simply noted how exciting the technology was without actually thinking about how it would be applied, practically.  

What is for show and what is practical?: The answer depends on context

Let’s consider the first use case all these vendors mention for Generative AI: writing job descriptions. Here are some questions to ponder: 

  1. How often is a job description created from scratch?
  2. If it is created from scratch, are you going to populate it with the tasks or skills that job requires? (If so, this is a prime use case where Generative AI can help.)
  3. Is this new job description a net new role in the organization, or similar to other roles? How are you going to classify this role in your job architecture to level the compensation correctly?
  4. If its description is similar to other roles, why was it necessary to create it from scratch? If it is indeed a new role, there are many more steps needed to take in order to get the job leveled and budgeted properly before being able to post.

Needless to say, writing job descriptions is a practical example. If you don’t have many controls, like job leveling criteria and position management approval processes in place that prevent managers from simply posting job descriptions as they see fit, generative AI is a no-brainer. However, if you do have all those controls in place, as many large organizations do, is it really a time saver or just a fun party trick that won’t really be used practically? Even more so if you have standardization around prompts you have to add in before allowing it to generate.  

Let’s try another example: summarizing feedback, because I don’t personally think you should outsource the writing of your feedback to an LLM. For one, it will probably be more time consuming to phrase the prompts properly than writing it yourself, and furthermore, you should incorporate your own, original feedback. 

Let’s say you’re a manager with 6 employees and it’s performance review season. You’ve asked each of your employees to give a self evaluation and 3 sets of peer feedback to you. That’s 24 pieces of feedback you’re tasked with reading.

Some questions to ask:

  1. Should you read all 4 pieces of feedback for each employee to give your team the best possible performance evaluation?
  2. If not, what will happen if the employee asks for specific examples when you offer your feedback? 
  3. Does this example resonate with you because this is a real pain point, or do you simply dread the performance review process and want to get it over with as soon as possible?  

So summarizing feedback is one example, but if your company wants people to value feedback as a gift and unique learning opportunity, it should be treated as such.  

How we should be thinking about implementing AI:

The real question you should ask yourself when it comes to AI is what outcomes you want to achieve, and then think about the technology necessary to achieve those outcomes. This is how HiredScore has thought about how we’ve designed our products.

For example:

Iconic Outcome: Capabilities Needed to Achieve Outcome:
Increasing recruiting capacity without increasing headcount • Automated resume scoring and prioritization
• SLA and nudges for recruiters and hiring managers
• Hiring manager self service tools
• Every requisition never starts at zero
Every requisition never starts at zero • Understanding of when a requisition is posted
• Candidate resurfacing at requisition post
• Candidate recommendations take into account relevancy for the role and likelihood of applying
• Automated reach out to apply
Making it easier to get a job internally rather than externally • Internal role recommendations delivered to employee or manager at the ideal time in the employee's life cycle
• Recruiter alert to look at high value internal
• Hiring manager alert to look at high value internal
• Facilitated communication between manager and hiring manager
• Alert to HRBP, Talent Manager, and current manager if employee isn't hired

The key here is unlocking all the personas that impact talent and ensuring that they are all driving towards the same objectives. Our brain can only hold 7 pieces of information in our working memory simultaneously and that limits our ability to understand all the moving parts of a complex system such as an organization. Fortunately, AI has the ability to hold billions of parameters in the model to enhance our ability to drive the best outcomes. But, be warned, not all AI is created equal, and not all AI vendors are the same. 

HiredScore is an AI-driven company. Our sole focus has been AI engineered for HR in the largest, most complex organizations for the past 11 years. We are not a technology platform with AI features that check the box on an RFP. Whether it’s Open AI, Anthropic, Hugging Face, Cohere, Inflection AI, etc., these AI companies are solely building AI models, and not trying to become systems of record or needing their own proprietary system of record to work. As a result, our focus is on using our AI to drive business outcomes for the customer and to maximize your existing HR technology stack. 

Implementing any piece of technology is difficult, but buying and implementing something like AI that will transform your business requires deep thought, real expertise, and a trusted partner. Here are some resources we’ve put together as you’re considering or accelerating your AI journey:

  1. World Economic Forum AI for HR toolkit
  2. Enterprise AI buyers guide

For those interested in better understanding your iconic outcomes, sign up for an Iconic Outcomes Workshop and we can help you prioritize and understand the highest value use cases for AI orchestration.