How should staffing agencies measure whether AI is actually moving the business forward? In this episode of The Staffing Show, David Folwell is joined by Hilary Baker, Managing Director at StaffingHub, to discuss effective ways of proving AI ROI in staffing. They delve into why usage and time saved alone are not enough, how to connect AI to outcomes such as fill rate, placements, redeployment, and gross profit per recruiter, and where AI tools can help teams prioritize the right workflows. They also explore the role of human judgment, emerging AI-enabled recruiter roles, and why better outcomes matter more than more automation.
[0:01:13] DF: Hello, everyone. Thank you for joining us for another episode of The Staffing Show. Today, I am joined by Hilary Baker, who is my teammate and good friend and who’s deep in the AI game with me. Today, we are excited to jump into something a little bit different. Normally, we’re having on the leaders in the staffing industry. What we’re going to be doing is talking about some of the things we are learning and gaining from using AI and having different people join us and talk about the actual specific use cases and implementation of AI in staffing today. What you’re going to be hearing from us is a lot of examples of how we’re using it in our business, and hopefully, some ways that you can apply to your business to get a better ROI and have it have a bigger impact on your day-to-day work.
[0:02:01] HB: Awesome. Thank you so much for having me, David. I’m not often on the podcast, so this is exciting, and I’m really, really excited to start digging into some of this AI stuff with you. We started this because we are talking about how we’re using AI daily, many times a day on Zoom and then on phone calls and then on text, so we’re like, “Why don’t we get some of this out into the world?” Let’s share this with the people who might get some more value out of it as well.
What we wanted to dig into for this first episode, though, is something that everybody I’m talking to, regardless, honestly, of the industry, but a lot of the staffing leaders are asking, “How on earth am I going to start proving an ROI on AI? I’ve adopted tools, I’ve bought the things, my team is using it, and I know they’re using it, I know I’m using it, but how do I know it’s actually working? How can I connect the dots to how this is impacting my bottom line?” Dave, I wanted to ask you about this because you are building with AI constantly. Every time I talk to you, you’re like, “I’ve maxed out my usage on this one and this one and this one and also this one.” You’re talking to owners of staffing agencies every single week, so what is the most common way you’re seeing these people trying to measure whether or not their AI is working, and where does that measurement start to crumble and fall apart a little bit?
[0:03:30] DF: Yeah, there’s a few things here. One, the most common mistake is that agencies measure usage. To be honest, it’s something that we’ve done a lot of as well, and there’s also this exploratory phase of what works, where am I going to find value? I think that for the leaders out there jumping in and digging into exploring ideas and seeing where it can be impactful is really, really important, because that’s how you can identify new ways to implement your business. At the end of the day, you need to figure out, does this actually make sense? Is there an ROI? Am I just spinning my wheels? I think when it comes to measuring usage, a lot of times, staffing agencies are asking, are people logging in? Are recruiters using the tool? Did it save time? Did people like it? That’s not really an ROI. That is just adoption. An ROI has to show up on a staffing scoreboard. Did the fill rate move? Did time to fill come down? Did submittal-to-interview conversion improve? Or did the GP per recruiter go up?
One of the areas that we’re starting to look at with our platform is: did the placements per recruiter per month go up? Did the number of placements from the network go up over that period after implementing? One of the areas where I think the measurement of this falls apart at most firms is they stop at the team saved time. You hear that a lot. I did it with less time than it took me before. And that’s great, but only if those people are then shifting to do something that is more production, or lower cost.
[0:05:04] HB: Right. Save time for what? What are you going to fill your time with now?
[0:05:08] DF: I mean, I guess, it’s time back to maybe go to the park. Who knows? No, I think that it’s important to pay attention to where you’re saving time and how does that actually impact your bottom line. Because there’s a lot of people, “I can do this 10 times faster now,” and that’s great. What did that allow you to do because of that? Or what did that allow your recruiter to do because of that? I think that’s where the measurement tends to fall off from most of the customers I talk to. It’s hard. There’s no simple way to do it, but there are some tactics that you can take.
[0:05:40] HB: Such as?
[0:05:42] DF: Well, one of them that I think is really easy is thinking about it from a replacement cost of what would you have paid to do this before and how much time would it have taken you to do this previously? For example, we just launched a new website. I look at that website that we launched, and I think that would have taken me, and I’ve launched a lot of websites in my life, because I ran a digital marketing agency. To get to that level of website, one, maybe would have cost 40K to 50K and would have taken a couple months’ worth of work with a team of people. Now we’re doing that in three weeks with a handful of people, and granted, like we’d spend a lot of time building the content and making sure that things were lined up. But the time to deliver and deploy that has shrunk significantly, and it’s something that we did not even as the main project. It was a side project, which is a crazy thing to think about. That’s a specific example.
I think the best measurements of this are when you’re using a holdout group and you’re saying, “We’re going to apply this tool to part of our group, but not to the other. Then we’re going to see how the performance varies.” You have true apples-to-apples and a better version of A/B testing, which I think is the gold standard. Also, much harder to do in practice and frustrating, so then you have to roll it out for one group and then roll it out for another. But if you really want to get detailed on the AI ROI, that can be a really impactful way to do it.
[0:07:13] HB: What do you say to the people who aren’t just measuring time saved, but they’re also, or only measuring how output has changed? Like, we used to send per recruiter 90 outbound emails a week. Now we’re sending 180. Big win. Are you seeing that to be true? Is that an effective measurement, or not so much?
[0:07:33] DF: I actually think that’s a false trap as well. It’s funny, is I think that that’s what everybody’s starting with as well. We can do 10 times the outreach. We have 10 times more contacts to reach out to, and it’s like, that’s great. What’s your actual conversation rate? What is the meaningful signal that you’re dialing in on? I think that all of us that have spent time at some point or another in marketing automation, sales automation, recruiting automation, it used to be like, all right, well, we have these additional messages going out. We know for every message that goes out, there’s value that’s created. That is still true. But AI is increasing the noise. It’s increasing the spam, the junk. I get 10 to 15, clearly AI-generated emails from throwaway email addresses. It looks like burner emails every single day now, along with text messages. I think that we’re moving into an era where if you’re just measuring output, you are going to be in trouble. You need to measure outcome, and you need to look at what is the actual signal that you’re dialing in on.
I think that comes down to meaningful conversations that you’ve had, or replies that had more than – I don’t know exactly what the measurement should be there, but I think we need to start moving down the funnel and away from the traditional activity-based and look at what is the formula for this, because I think it’s about to change pretty rapidly.
[0:09:01] HB: You mentioned quite a few different measurements at the top. Has your fill rate changed? Has your margin moved? What’s your redeployment rate? Where do you, from where you’re sitting, see AI having the biggest impact on those numbers? Where should people be watching first?
[0:09:17] DF: I mean, AI is inherently good at compressing processes, or collapsing systems and processes into simpler, faster, more actionable items. Things that used to take 10 hours can be done in 30 minutes when organized correctly. I do think that the idea of figuring out who to call next is something that maybe used to take 30 minutes. I need to figure out the top five people to go reach out to for this job. That should be shrinking pretty rapidly and allowing people to have more meaningful outreach. I think that you can map internally all of the processes that you have today, look at where you think AI can have a big impact.
I mean, one of the things that I’ve always talked about, recommend over and over and over again is the concept of meta prompting, which is prompting the prompter. I think it’s really important to just lay out, what are all of the challenges? Where’s your team spending all of your time? Just write it all out in a Google doc, or a Word document and list all of the things, like as many of the challenges that you have, as many of the processes that you have, the things that you think are clunky and feed that into whether it’s ChatGPT, or Claude, and say, “Hey, I need your help identifying the areas where AI could have the biggest impact and where we would have a measurable ROI.”
That can be a great way to filter down to exactly what you need to work on next and where AI can have the biggest impact. One of the things we’ve started using is the RICE score, Reach Impact, Competence, and Effort. It’s funny, as I’m like, I use it and I’m like, all right, let’s rank them by RICE score, but I have AI do that over and over and over again. I’m like, here’s my goal. Here are all of the processes that we have in place. Here are the challenges that we have. Rank where AI can have the biggest impact, put a RICE score on it, so that I know where to start, and it will often pick, all right, here’s the thing that I’m really confident I can do for you, and here’s the impact in the effort.
Granted, it has the overconfidence that I think we’ve all probably experienced at this point, where it thinks it can do everything and it doesn’t always work out perfectly. It’s a great way to dip your toes into thinking about where to apply AI strategically and then also, while you’re doing the RICE score, say, what do you suggest the ROI would be for this and how would I measure it? The great part is it will suggest that, too. It’ll give you some specifics on that front, too. I think that a lot of times, people are asking, how do we do these things? A year ago, it was, well, you have to learn how to be an amazing prompter. Now it is, you have to know what questions to ask AI to help it solve you the problem you need, and that meta-prompting skill set is becoming more impactful, because frequently it has – I always think of it as like, you have the smartest person that you’ve ever been next to you ever, but it has very little context on who you are frequently. It’s not always knowing what to solve. But if you can give it enough context and then ask it the right questions, it can actually help narrow down your focus and help you figure out where you can drive that ROI and also how to measure it.
[0:12:28] HB: What are you finding that it’s most important to have that human judgment piece of it come in, though? It gives you this beautiful table, says ranked by RICE score, here are the things that can give you the best ROI starting today. When do you know, yes, that sounds like a great plan, and when do you need to have some human intervention where it’s like, “Let me take this to my team, or let me brainstorm this independently?” Where’s that line for you?
[0:12:52] DF: Everywhere. I feel like the amount of edits that I go through with AI gets 60% to 90% great, but that other 10% to 40% is really, really bad. I think it’s important that you pay attention, and you have to know, I think this is why the human in the loop, the judgment element of this, I think that’s why it’s really important. That’s changing, and it depends on the context, it depends on the prompt, it depends on the task. The more determinant task is, if it’s yes or no, one or zero, and it’s very clear, it’s better at those things, because you can say, here’s an example of what good looks like. Here’s an example of what bad is, and then you have a way to have it measure itself.
But we are finding that the human in the loop, the judgment aspect, is really, really insanely valuable. That said, I also think that you can have AI start running loops, where you have it run for a few hours, or try to run overnight and self-refine and improve to see where you can get it to go from there. Sometimes you end up in a great spot, but we’re trying to refine that and look at better ways to automate our internal processes.
[0:14:01] HB: Just zooming back out to what should staffing agency owners really be keeping an eye on on a daily basis? Aside from usage, we know that everybody’s tracking usage. They know that they’re onboarding recruiters, getting recruiters up to date on, do you have Claude? Do you have ChatGPT, etc.? What else should they be measuring now that wasn’t even a metric, that wasn’t top of mind a year and a half, or two years ago?
[0:14:28] DF: That is a great question. This is one that we’re still trying to figure out internally. I don’t have the laser answer here, but I think the measuring output and outcomes, like are you actually producing more of the thing that you want to produce, whether that’s placements, or valuable conversations, or client conversations, or client leads? I think that having those be a primary thing to measure, I do think that because we are in such new territory with AI, there’s a thought leader that I follow, Nate Jones, who has just amazing content about AI on a daily basis. It’s pretty deep in the game and very product-centric. He, the other day, was saying, there is not a single person that I talk to, regardless of who it is in AI, that doesn’t feel like they’re behind right now.
I think I even saw the creator of Claude Code, Boris, I believe the other day, I saw on TikTok that he said he feels like he’s behind. It’s like there is so many moving parts and it’s a new frontier, so there is no set-in-stone, here’s exactly how you do everything. By the time that happens, a new model is released, and the game changes again. Don’t have exact clarity on what to measure. I do think, if we go back to the fundamentals of what does success mean for your business, what does success mean for your customer, is it, “Hey, we are placing people faster, we’re delivering higher quality people, we’re keeping people on contract at a longer rate.” I think those core business metrics become more important, and then it’s how does AI, it’s a tool, just like many tools that we’ve had over the years, and how does AI serve that outcome? I think that the North Star metrics become more important as you have tools that accelerate a lot of the day-to-day work that used to take so long.
[0:16:18] HB: Do you think that any new roles are going to start popping up within staffing agencies because of this?
[0:16:23] DF: Oh, yeah. If I owned a large staffing agency, I would be hiring at least one or two people, maybe a handful, to be leading the charge in terms of where AI can go and what it can do for my business. If not hiring internally, I would be looking at really strong partners to incorporate, because I think that the gains that we’re going to see from AI are going to be astronomical compared to what we’ve seen over the years. As you said at the beginning of this, I call you with ideas about how to use it, what I’m doing. I am, maybe, I don’t know if it’s a healthy level of obsession, but obsessed is for sure the right way to say it. I think part of that is I just see what is possible with it, and I’m having so many breakthrough moments where I’m like, wow, I didn’t realize I could do that. I didn’t realize I could make that happen. That’s something I’ve thought about for years, but I thought that was a very difficult challenge. It’s fun to see what’s possible, and I think that’s going to happen more and more in the staffing space and just with business as a whole.
I saw the other day; I don’t remember the exact titles, but they were talking about the different roles in software of there’s a designer, there’s the product manager, or the user experience person as well. They’re saying that these are getting compressed as well. There’s now a builder, there might be an idea person that’s just doing MVPs.
[0:17:44] HB: Yeah, I saw like, sweeper, and it’s the person who’s –
[0:17:47] DF: Yeah, sweeper.
[0:17:47] HB: – responsible for optimizing.
[0:17:49] DF: Yeah, exactly. Exactly. I do think that the roles are going to change in the software landscape. I think they’re going to change the staffing, too. I think we are moving into that era, where you’re going to have recruiters that are able to do 10 times what they used to do, because they are AI-enabled recruiters. I think that’s the future. I would also, as a word of caution, more AI is not always better. More messages is usually not the right answer. But I think trying to figure out how to use these tools to be smarter, more strategic, more thoughtful, to help it actually facilitate relationships versus just inundate people with more automation. I think that’s the right path. I think that’s one where we’re starting to see. Be interesting to see what the next few years look like when it comes to the traditional role of recruiter and having to do 100 calls a day. It’s like, maybe that still is the case, but those 100 calls should be a lot more valuable, because you have AI helping facilitate who to call.
[0:18:50] HB: Every episode, I’m going to ask you the same thing. What is one AI development that you’re watching super closely over the next two weeks and why?
[0:18:59] DF: Oh, God. Which one am I not? For any of you that don’t know, Fable 5 was given to us and then taken away, but has been back as of last Wednesday, I believe, and it is the newest release of Claude Code, and it allows you to do – it’s the most advanced Claude Code and AI model out there in the market right now. It’s a precursor to Mythos, which is going to be their future model. The functionality and capabilities are really insane. I mean, is able to do things, has solved problems in a way that I’ve not seen before. That is super exciting.
One of the areas that I just have not stopped thinking about since last summer is the concept of, I don’t know, we call it a second brain, but storing real-time context, storing all of the – I think of it as a data exhaust from your day-to-day work. So, the emails, the calls, the transcripts, keeping track of that and figuring out how to actually implement that into things that give you time back in your day. I’m pretty sure that Jony Ive and OpenAI are going to release something this year, like an earbud or something along those lines that is going to be more of a personal assistant. It’ll be interesting to see what they do. For those of you that don’t know, Jony Ive is the guy that designed the iPhone and OpenAI. Bought his company for, I don’t remember the exact amount, but maybe a couple of billion, or something like that.
It’s going to be interesting to see where that develops. But that’s not near term. We don’t know when that’s going to happen, but there’s going to be some new products that will have, I think, a profound impact on our day-to-day. The one part to note, if you’re listening to this still, you’re probably an early adopter with all of these things. None of them are perfect. They all have their problems that you have to work through. But the impact of each of them can be really profound when you get them dialed in.
[0:20:56] HB: Yeah, absolutely. I would just add to that practice makes progress, not perfect, because that has not yet been defined. But getting in there, just messing around with things is the best way to learn.
[0:21:08] DF: Yeah. Define your outcomes. Think about your ROI. This conversation was a lot about ROI, and candidly, we’re at the earlier stages of it. I see, I’m like, “Oh, this is saving me this much time.” So, I’m jumping in because I can do more than I used to be able to do. I’m trying to work towards a better model internally for how do we measure the outcome. How do we make sure we don’t spin on something for too long? What is the actual time box to put around something, because of what we think the value of the outcome would be on the other side?
[0:21:38] HB: Absolutely. Right. Well, good conversation. We’ll see you guys again next week. Thanks so much, Dave.
[0:21:42] DF: Thanks, Hilary.
