Dos and don'ts of running a technology pilot

Piloting a technology system is highly advisable and will increase the probability of it being a success on full rollout

Launching a new ATS system. Across 96 markets. All at the same time. Needless to say it’s a daunting prospect. But this is what greeted Barbara Lee, SVP of global talent acquisition at Nielson, when she joined the firm as a talent consultant back in 1999.

“There are times when you can’t get around it, you just need to launch as a big bang,” she says. But it’s an experience – etched into her mind 20 years on – that only reinforced her sense of the importance of running a pilot when implementing a new HR system. “[Not running a pilot] can seem like a good idea, but it just never is,” she says.

So what do HR professionals need to consider when undergoing this vital process?


Define roles within the HR team

First you need to decide who in your HR team is doing what, says Elisabeth Simpson, head of strategic change management at MHR. “You need to allocate a test manager to co-ordinate your pilot teams and ensure the feedback is provided in a timely and consistent manner,” she says.

“Consider how issues will be handled internally and escalations made to your software provider. Thinking through the roles and processes before you start will prevent friction later on.”

Agree expected outcomes

“It’s all about agreeing outcomes upfront,” says Sarah Mason, chief people officer at Foxtons. She says these vary across projects, but the ultimate desired outcome is often to positively impact business performance. “Sometimes it might be we’re moving from an old legacy system, where we have no choice. In which case the success measure might simply be that people are using it and it’s better than the last system.”

Decide how you will measure success

“Don’t be afraid of anecdotal evidence – data alone won’t always give a full picture,” advises Donna McGrath, a digital learning professional at Deutsche Bank. “When testing out tablets for a learning project [at previous employer Royal Mail] we had a range of measures from simple data on the amount of log-ins, and recording learning and satisfaction with the tool, to gathering anecdotal evidence on ease of use.”

Consider how long the pilot should be

The length of a pilot can vary hugely, explains McGrath: “I’ve worked with pilots lasting a couple of months to in excess of nine months depending on the complexity, usage and sensitivity around using the tech.”

Bespoke projects could take even longer, adds Mason.

Define your test audience carefully

Experts give a somewhat mixed picture of what a test audience should look like. Kath Austin, chief people and marketing officer at Pizza Hut, warns that while the temptation can be to trial new software with “a cohort of your best performers”, this might not be representative.

“For this reason we often achieve better results when the project is rolled out across our whole organisation,” she says. “It is beneficial to involve both supporters and detractors, software gurus and those less tech savvy,” agrees Simpson. Yet, according to McGrath, a small test audience can sometimes be best to accurately measure impact.

Keep in close contact with your vendor

Lee advises that HR stays in contact with the vendor all the way through the process. It’s a sentiment shared by McGrath: “I’ve had some brilliant successes from a vendor changing my perspective with case studies and pushing me to do things differently.”


Forget to manage end-users’ expectations

“You need to say to people it’s normal for it to be a bit rocky,” says Mason. The best strategy to allay any discomfort is communicating the value of what you’re hoping to eventually roll out, she says – something most organisations neglect: “My big tip would be having a social strategy alongside a tech [one].”

“Make sure you have a communications plan ready early,” advises Paul Burrin, VP of Sage People. “One of the main success criteria for any HR software pilot is adoption.”

Create division when selecting test employees

It’s not unusual for employees not chosen for testing to feel hard done by. “Where I’ve seen a staged roll-out work really effectively is in a big global company,” says Mason. “Whereas if everyone’s in the same location, it can feel quite divisive some receiving it and others not. People can say ‘I didn’t hit my sales target because I didn’t have that’.”

Believe everything your tech provider tells you

To get to the bottom of what’s being promised, Lee recommends getting those who will actually be involved with the tech day-to-day involved in the tendering process. “Even if what the vendor has told you about the benefits is accurate, if it’s not going to be used, it’s not really helpful,” she says. “So I’ve tried to get the recruiters and sourcers involved in speaking to the vendors upfront.”

Be afraid to extend the pilot

“We recently ran a pilot that was going to run for six weeks, but we extended that because of holidays,” reports Lee.

“Don’t rush a pilot,” advises Burrin. “Make sure the team has the time to make it successful.”

Aim for perfection in the pilot phase

For Stephen Kelly, VP and CHRO at IBM Global Business Services, HR should follow Agile, design thinking methodology when trialling new tech. “We need HR to become more comfortable with minimum viable products,” he says. “There’s no point spending six months because the business is going faster and we need to get ahead of it.”

Be afraid to scrap the tech “If after 30 days [of working in an Agile way] there’s no progress, it’s either the wrong project or the wrong people,” Kelly continues.

Mason agrees: “For a pilot to be a pilot there has to be a possibility you wouldn’t go for the tech.”

This piece appeared in the October 2019 HR Technology Supplement