What to do after an ATS demo

Vendors often provide sandbox access if you request it.  Sandbox access gives you a chance to actually use the system yourself, rather than just watching vendor demonstrations. This typically means logging into a demo environment with sample data where you can click through workflows, test features, and get a feel for the user interface.

This hands-on time is valuable because it’s your chance to move beyond the vendor’s carefully choreographed demo script and actually touch the system yourself. However, you need to bear in mind that sandbox environments have limitations. The generic sample data won’t reflect your context, the system won’t be configured for your specific business processes, and you’ll be navigating without the training that actual users would receive.

If you get sandbox access, treat it like a proper test drive. Bring your laptop, log in during the demo, and try to complete actual tasks. This reveals usability issues that don’t show up when you’re just watching someone else drive.

Test the most common tasks your team does daily that relate back to your hiring DNA: posting jobs, reviewing applications, updating candidate statuses, sending communications. Focus on whether the day-to-day tasks feel intuitive or frustrating.

Pay attention to how many clicks simple tasks require. If updating a candidate status involves navigating through three menus, that’s going to annoy your recruiters who do this hundreds of times per week.

Now look at the candidate experience. Walk through the application process yourself from a candidate’s perspective. How intuitive was the process? Was it clear what information was needed? Did you encounter any technical problems or confusing steps?

After you’ve completed sandbox testing or trials with your shortlisted vendors, schedule comprehensive debriefs with each user group separately. These should happen within a few days of finishing all hands-on testing while the experiences are still fresh. Unlike the quick demo debriefs, these sessions should be longer (45-60 minutes) and more detailed, covering your impressions of all the systems you tested.

Recruiter debrief session: Which tasks felt easier than your current system? Which felt harder? What would slow down your daily work? What features would actually save you time?

Hiring manager debrief session: Could you find what you needed quickly? Was it clear how to provide feedback on candidates? Would you be comfortable using this without IT support?

IT debrief session: Are there any security concerns? How complex would implementation be? What ongoing support would this require from your team?

These in-depth debriefs are crucial for making your final decision because they’re based on actual hands-on experience rather than polished demonstrations.

By now you probably have detailed feedback on 3-5 systems. The challenge is comparing them fairly when each has different strengths and weaknesses.

Don’t just add up scores. Weight them according to your real priorities. If candidate experience is crucial to your employer brand, a system that scores poorly there should be penalized heavily, even if it excels elsewhere.

Look for patterns in feedback. If multiple users mention the same concern about a system, take it seriously. If everyone loves a particular feature, that’s worth noting too.

Consider the learning curve. Some systems are more powerful but harder to learn. Others are simpler but less flexible. Match the complexity to your team’s technical comfort level and training appetite.

Now’s the time for detailed reference calls. You’ve narrowed down your options, so you can afford to spend more time on thorough reference checks.

Start by asking vendors for 3-4 reference customers similar to your organization in size, industry, or use case. While these will be their success stories, they’re still valuable for understanding implementation experiences. Supplement these with your own network research and check review sites like G2 or Capterra for broader user feedback and ratings. Look for patterns in reviews—consistent complaints about the same issues across multiple sources are worth investigating.

Ask about the implementation experience specifically. How long did it really take? What went wrong? What would they do differently? This information helps you plan your own implementation and set realistic expectations.

Dig into ongoing support experiences. How responsive is the vendor when things go wrong? Have there been any significant outages or performance issues? How well does the system handle peak hiring periods?

Ask about the vendor relationship overall. Do they feel like a valued customer, or just another license number? How well does the vendor communicate product changes and updates?

Data and scorecards are important, but don’t ignore your instincts. After all the demos and trials, which system feels right for your team?

Consider cultural fit between your organization and the vendor. Do they understand your industry and challenges? Do they seem like a company you’d want to work with for several years?

Think about growth and scalability. Which system feels most capable of growing with your organization? Which vendor seems most likely to continue innovating and improving their product?

Most importantly, which system are your users most excited about? If your team is enthusiastic about learning a new system, half the battle is already won.