Proceedings of the Standing Senate Committee on
National Finance
Issue No. 62 - Evidence - March 28, 2018
OTTAWA, Wednesday, March 28, 2018
The Standing Senate Committee on National Finance met this day at 6:47 p.m. to study such issues as may arise from time to time relating to federal estimates generally, including the public accounts, reports of the Auditor General and government finance (topic: Phoenix pay system problems).
Senator Percy Mockler (Chair) in the chair.
[English]
The Chair: Welcome to this meeting of the Standing Senate Committee on National Finance. My name is Percy Mockler, a senator from New Brunswick and chair of the committee. I wish to welcome all of those who are with us in the room and viewers across the country who may be watching on television or online. As a reminder to those watching, the committee hearings are open to the public and also available online at sencanada.ca.
Now, I would like to ask the senators to introduce themselves, please.
[Translation]
Senator Moncion: Lucie Moncion from Ontario.
[English]
Senator D. Black: Doug Black, Alberta.
Senator Marshall: Elizabeth Marshall, Newfoundland and Labrador.
Senator Eaton: Nicky Eaton, Ontario.
The Chair: I would like to recognize the clerk of the committee, Gaëtane Lemay, and our two analysts, Sylvain Fleury and Alex Smith, who team up to support the work of this committee.
This evening, honourable senators, we continue our study on the Phoenix pay system problems and challenges.
For the first hour, by video conference, we have Michael Walsh from Queensland Health, in Australia. I have been made aware that for you, it’s Thursday morning at 8:50, and here in Canada, it’s 6:50 on Wednesday evening, so thank you for accepting our invitation.
Mr. Walsh was at the forefront when, in March 2010, Queensland Health, a government department in the Australian state of Queensland, went live with an information technology system to manage the payroll and human resources functions for the department’s approximately 78,000 employees. Soon after the launch, significant errors appeared and many employees did not receive their correct pay.
Honourable senators and viewers, to discuss how Queensland Health tackled the issue and solved the problem, and how much it cost, we welcome that organization’s Director General. To share a bit of background information on Mr. Walsh, in 2002 the Queensland government in Australia established the shared services initiative to centralize corporate functions across the whole of government. Mr. Michael Walsh has been the Director General of Queensland Health since July 2015. Queensland Health employs approximately 90,000 people and provides a public health and hospital system for nearly 5 million people.
Mr. Walsh, I have been informed that you have a short presentation, and it will be followed by questions from the senators. On behalf of the Senate of Canada, the Standing Senate Committee on National Finance, I want to recognize the time you have given us.
Michael Walsh, Director General, Queensland Health: Thank you, Senator Mockler. It’s a tradition in Australia to acknowledge the traditional owners of the land upon which we are meeting and their elders past, present and emerging, at the beginning of every meeting. That is done in the spirit of the traditional owners of the land in Australia and also in Canada.
Thank you, senators, for the opportunity to address the committee and then to respond to questions.
As was outlined in the introduction, in 2010, Queensland Health implemented a payroll system that was very problematic, and very quickly after the payroll system was implemented it was clear that there were people not getting paid or people receiving incorrect pay. After the third pay run, I was asked to lead what was called the “payroll stabilization program” in order to ensure that we were able to address these problems as quickly as possible. The commencement of that particular program happened in April 2010 after the implementation of the payroll system in March.
One of the issues that we were able to respond to in terms of the staff was that there was seen, and there was, a very quick response to the issues raised by staff. The first thing we did when we established the program was to go and meet with staff who are affected and to apologize. As you can appreciate, a payroll system in any organization is one of the fundamental relationships that exist between an employer and an employee, and that is a fair day’s pay for a fair day’s work. When an employer doesn’t live up to its side of that relationship and provide the fair day’s pay, you are undermining the fundamental relationship that exists between an employer and employee, and addressing that and reestablishing trust and confidence in the organization by the employees was absolutely critical.
We established a program and set up four streams of work, looking at the business processes, the systems that sit around the payroll system, the actual software application itself and communications. We had four streams of work that ran, and we established a work program in consultation with the staff. We had staff representatives on all of those streams, and we ensured that the development of the program was open and transparent.
The other aspect we undertook at that beginning was to identify what were the measures that would allow us to know when the system was stable — measures such as how many calls were coming into the call centre, how many forms were not completed and entered into the system at the end of each roster cycle, how many ad hoc pays were being made, how many cash advance payments were being made to staff, how many overpayment numbers there were each pay. These data were presented to staff each week so we could track and communicate with staff and we knew when the system was stable.
After about four months, in July 2010, we confirmed that we knew the system was now stable, it was predictable, and the numbers of overpayments and the actions we had taken in the intervening time had meant that it was known how many overpayments and how many ad hoc payments we had, and they weren’t increasing. We then transitioned to the payroll improvement program and worked on reducing those numbers over the coming months and years.
Unfortunately, an implementation as problematic as the one in Queensland Health means we still deal with residual issues today. There are still people repaying overpayments from that implementation, and there are still concerns about the actual software application and the processes, even though the system is operating as well as any system would in an organization. There are trailing residual issues that result from these sorts of programs.
Senator Mockler, that’s my introduction. I’m happy to answer any questions on all the details of what we did at that time.
The Chair: Before turning to questions, I would ask the senators to introduce themselves.
[Translation]
Senator Pratte: André Pratte from Quebec.
[English]
Senator Jaffer: Mobina Jaffer, British Columbia.
Senator Andreychuk: Raynell Andreychuk, Saskatchewan.
The Chair: Thank you. We will now turn to questions.
Senator Marshall: Thank you, Mr. Walsh, for being with us this evening. The situation that you were faced with is similar to the one we are faced with in Canada except that we have more employees that are having payroll problems. We have, it sounds like, a bigger system.
You oversaw the correction, or that’s what you were tasked with. Were you involved in the initial implementation or the design of the system, or did you just come in to oversee the correction?
Mr. Walsh: I wasn’t involved in the implementation of the payroll system and only came in to actually oversee the stabilization and improvement. That was an important aspect of the program. Certainly in communicating and meeting with staff and going around the state, which is similar to a province in Canada, that was one of the questions that people would ask me: Were you involved? Did you create this problem? And the answer is: No, I’m here to actually stabilize and improve it.
Senator Marshall: Here in Canada, with the payroll system, there are about 100 government departments and agencies that are feeding transactions into our payroll system. Can you tell me what has happened and how you are set up? Would you have that many organizations feeding in? I’m trying to get an idea of the magnitude of our problem compared with the magnitude of your problem.
Mr. Walsh: As you indicated, the size of the workforce I understand in Canada is between 250,000 and 300,000 —
Senator Marshall: That’s correct.
Mr. Walsh: Queensland would be comparable to a province in Canada. It’s a state, and it’s a single organization, Queensland Health, and at the time had about 78,000 employees and now 90,000 employees, and they operate under statutory bodies. There are 16 independent statutory bodies, but part of one system, and we run one payroll system to deliver the pay. That means we are able to manage the HR and transfer issues as people move around the system.
There are added complexities in Canada where there are multiple organizations feeding into a single shared service provider, and, therefore, although we have 13 awards that cover the employees of our organization, the situation in Canada is you’ve got more awards, and then you have more potential pay scenarios, I understand about 80,000 in Canada, and here in Queensland about 23,000 scenarios that we would deal with from our awards.
There are similarities, but the situation in Canada is a lot larger and more complex because of the multiple organizations.
Senator Marshall: I’m trying to get a handle now. I think you told us it took four months before it was stabilized because, from what you are saying, you recognized very early that you had a major problem. I take it you were the one who went in right away, within a couple of months, and it took four months to stabilize the situation; is that correct?
Mr. Walsh: That’s correct. After four months, it certainly wasn’t operating optimally, and we still had a large number of overpayments being generated and a large number of forms that still needed to be processed at the end of each roster period, but those numbers weren’t increasing.
Senator Marshall: How long ago were you stabilized? We had somebody testify last night who said that for as long as it took to develop the system and get it going, it is going to take that long again for us to get out of the problems that we’re having. You are saying four months. I’m wondering, how long has it been since the four months has been up, and what has been the experience?
Mr. Walsh: The four months certainly didn’t fix the problem. After four months, we had a handle on it, and the problem wasn’t getting any bigger. Leading up to the four months, it was getting bigger, and that was in July 2010. That is nearly eight years ago. From July 2010, over a period of about four years, significant improvement work was undertaken, including upgrading the system and reconfiguring the actual software application.
So it is true that to actually resolve the issues takes a long time. What we were able to do, though, was to contain the growth of the problem so that we knew that, as we were improving, we were actually delivering better outcomes for people each fortnight.
Senator Marshall: And you still have that payroll system, but it has been improved.
Mr. Walsh: That’s correct, yes.
Senator Moncion: Thank you for your presentation. It is very interesting.
I’m aware that you were not there from the start, but do you know if it was one implementation that was done and the conversion was done in one shot or if there was a pilot project that was done and then there was a gradual conversion?
Mr. Walsh: I should say I’ve read the auditor’s report from Canada about the Phoenix system, as well as the lessons learned. I’m not sure if you’ve seen the Auditor General’s report from Queensland or the commission of inquiry report, but they are all point to very similar problems associated with the implementation.
With the implementation here, one of the issues that was found was, in fact, it was what you would call a one-step go live from the old system to the new system. There wasn’t a pilot or a phased rollout, if you can put it that way, such as a small group of employees moved across from the old system onto the new system and actually monitoring the outcomes of those employees for a period of time and then gradually bringing on other people. That didn’t happen, and it was certainly one of the criticisms of what happened.
There is another approach that can manage a similar issue, and that is you can have parallel processing at the time of go live. It means you continue to run your old system and run your new system in parallel, and you continue to pay people out of the old system but you compare what the pay outcomes would be from the new system for a period of two or three pay periods. That was not done either.
Therefore, it was one step from the old system to the new system, and there was no rollback strategy in place so that you could turn off the new system and turn the old system back on.
Senator Moncion: Thank you. Was staff laid off when the implementation came? What we understand is that there were so many cost efficiencies that you would have once you were on the new system that there was a lot of staff that you would not need any more. Were there cutbacks done on payroll staff before the implementation?
Mr. Walsh: The short answer is no. I’ll just give you the numbers at the time.
At the time in the payroll processing area, the payroll staff numbered about 650. That was up from a normal operating environment of about 550, and that additional 100 was because of all of the work that needed to be done to build, test and design the new system.
Unfortunately, the business and processing steps that were put in around the software application meant that one of the ways that we had to progress to stabilize the system was to continue to put on additional staff after the go live in order to manage the number of forms and to get them processed in time. So we went up to 900 payroll staff in order to manage those processes.
The short answer is no, the staff were not put off prior to the go live. In fact, there were about 100 more to deal with all of the additional work that we needed to have happen at the time, and then we needed to increase the staff to be able to process the forms in time.
Senator Moncion: Thank you. Now that you’ve been on the system for what, six years, do you still have 900 staff, or have you reduced it considerably?
Mr. Walsh: We are in the process of moving the staff from the payroll service out to the hospital and health services. One of the problems that was created in the implementation was the direct link between a person who manages a roster in a hospital or a health service would fill out that roster on paper and then send it into the payroll service who would enter that roster into the system, send a printed output of that back to the roster manager to check that it was correct and then come back into the payroll to make adjustments. Very manual.
We are in the process of putting out the electronic rostering system. Roster managers actually manage their roster directly into the system and can make adjustments and changes as they occur, whether a person is sick, changes shifts or is asked to do overtime in a particular shift. That way, it will reduce the processing that is required in the payroll system, but we are putting the staff out into the hospital and health services to support the roster managers to undertake that work.
Senator Eaton: You weren’t there and you didn’t implement it, you are just tidying it up, but I’m sure you know. When they approached IBM, did they approach IBM with a set of specifics, or did they say to IBM, “Please come in and build a payroll system for us”?
Mr. Walsh: In reading the Auditor General’s report and the commission of inquiry report, one of the issues that they identified in there was that the business requirements for the system changed over the period of the project, and that led to problems associated with the implementation. Some of those changes occurred very close to the go live and weren’t planned well and weren’t tested before they operated.
Senator Eaton: You said there were changes, but was IBM the one that came in and looked and said, “Yes. All right. There are 85,000 people here who need to get paid. There are six different systems. We will build it from scratch,” or did the health system go to IBM and say, “Here are our systems. We have four different tracks. This is what we need”?
I ask this because one of the consultants came as a witness and said that one of the problems was that we went to IBM and said, “Here, this is what we want you to build,” as opposed to saying to IBM, and maybe a couple of other companies, “Come in and have a look at our pay system and build it.” So it was the same thing for you?
Mr. Walsh: It was the same thing. We went out with specifications for what we wanted in the pay system, but they also changed over the period of the implementation project.
Senator Eaton: You sued IBM, I gather?
Mr. Walsh: I’m a public servant in Queensland, so I can’t really talk on behalf of the government. I really can’t comment on that.
Senator Eaton: All right. Over the time you’ve been there tidying up the mess, was there a drop in morale amongst your employees? Was the apology enough, or did you have to do other things to get everybody on side again?
Mr. Walsh: There was a drop in morale and a loss of confidence in the system and the organization. The apology certainly contributed to rebuilding that confidence, but we did a range of other things as well to ensure we were prioritizing the steps we were taking to fix the system and which things needed to occur first. There are so many things you need to do that you can’t do them all at once. We involved the staff in prioritizing those. What was important to them was what we did first. That was important to them.
We also produced a number of reports which we called time and wages reports. People lost confidence in the information that was on their pay slip, so we needed to be able to produce a report that showed their pay transactions over multiple pay periods. That built a lot of confidence because people were then able to look through that and check back with their understanding of their roster to see whether they believed the system was accurately recording what happened to them and what they had done. If there were changes, we would then retroactively make those changes.
There were a lot of things we did, but I would say that fundamentally it’s because we did them with the staff and listened to them and prioritized what they said was important to them.
Senator D. Black: Mr. Walsh, thank you very much for making yourself available. It’s very helpful. I have five very short questions to which I know you’ll be able to provide short answers.
Before I get to the five, I don’t suppose you’re available to come over to Canada for a few months, are you?
Mr. Walsh: I have a job here. I’m happy in my job.
Senator D. Black: You can’t blame us for trying. It might have been longer than a couple of months. We tried.
Which software application did you use in the implementation?
Mr. Walsh: The HR payroll system is SAP and the rostering system is Infor Workbrain.
Senator D. Black: I gather your external consultants were IBM?
Mr. Walsh: That’s correct.
Senator D. Black: Through the process, at any time, can you talk to us about the engagement you or your people had with the Government of Canada when the Government of Canada would reach out to you for advice and problem solving, maybe a shoulder to cry on? How often was that going on?
Mr. Walsh: I haven’t had any contact that I’m aware of, and I’m not aware of any contact in the organization. That doesn’t mean it hasn’t happened, but I certainly haven’t had any contact.
Senator D. Black: You were the person leading the correction process and you’ve never heard from the Government of Canada?
Mr. Walsh: Yes, I was the person leading the correction, and no, I haven’t heard from the Government of Canada.
Senator D. Black: Throughout the process of the work you were doing, did you ever consider dropping the system? Did you ever want to throw up your hands, drop the system and start afresh with a new system? Did that ever enter your contemplation, more than once a day?
Mr. Walsh: The short answer is yes. The frustration of trying to fix a problematic implementation of a payroll system is that it affects peoples’ daily lives. Given the hardship it caused a whole lot of people, you think, “How can we fix this fast?”
But as was pointed out before, there is no real quick fix, and changing a system brings about the same issues as implementing a new system. We certainly looked at whether we could stop the new system and go back to the old system, but again, the problem would be how you would ensure that the transition of the data, the data migration, was done properly. That itself is a long process.
So yes, it was very frustrating, and yes, we certainly wanted to fix it, but no, we didn’t go back.
Senator D. Black: Thank you very much. My last question relates to what the commission of inquiry might have concluded in respect of responsibility. Are you able to summarize, at the end of the day, who, in your view, carried the can in Australia for this failure?
Mr. Walsh: The challenge in all projects is that the project manager, or the person who’s actually commissioned the project, can’t outsource the risk associated with anything that goes wrong. In Canada, as in Queensland, it is the government that ends up having to deal with the risk and it’s also the government that has to solve the issue and commission people. In Queensland, they commissioned me to actually fix that. I think both reports identified that the oversight of the project was not undertaken well, which led to the problems with the implementation, and that really rests with the organization. So that rests with Queensland Health.
Senator D. Black: Are you aware of whether there were any terminations or disciplines or demotions related to that responsibility?
Mr. Walsh: There were some people who left the employ of Queensland Health at the time after the implementation.
Senator D. Black: Connected to the work they had done? Can I connect those dots?
Mr. Walsh: I don’t personally know the details of those, but certainly at the time that was the connection that was made. I don’t know the details of that.
Senator D. Black: Very well. Thank you very much. This has been very helpful.
Senator Andreychuk: Thank you for coming before us. It’s helpful.
It was Queensland Health that you are dealing with. Did any of the other states in Australia go through any similar issues, or was yours unique?
Mr. Walsh: Certainly in Australia, the implementation in Queensland has been the most problematic. Other states have implemented HR and payroll systems, including rostering systems, and have done so successfully. There has not been any other state with the same problems as Queensland.
Senator Andreychuk: Were they after you or before you?
Mr. Walsh: All the ones that have occurred after actually talked to us about lessons learned and what they should do to ensure they’re successful.
Senator Andreychuk: And in the federal government system that you have, has the national government put in any systems and are you aware of any difficulties?
Mr. Walsh: I’m not aware of any difficulties. I’m not specifically aware of any payroll systems the federal government has put in, but there certainly haven’t been any problems with any systems they have put in.
Senator Andreychuk: Is your system based on overtime, shift work and all the complexities of a payroll system that we find in health, corrections and all of these that work on emergencies, varying case loads and hours, which lead to the complexity that our federal system has? Were you confronted with any of those? Was it the system that wasn’t working, or was it aspects of the payroll system that were more complex than you thought they would be?
Mr. Walsh: The answer to your first question is yes. We operate in a 24/7 environment, and we have overtime shifts, allowances, penalties and other payments that get made to staff, all of which are attached to the type of shift and the location of their work and all those sorts of things. So it’s exactly the same issue that you would have in Canada.
In relation to the actual software application itself at the time of go live, it had bugs in it in relation to the calculation of people’s pay. Having said that, all software applications have bugs in them, and we are continually identifying and fixing those. There is no payroll system that’s not continually identifying issues and fixing them. Usually, however, they are relatively minor and don’t affect a large number of people. What we had here, though, was some of them weren’t just minor and they were affecting a large number of people. However, they weren’t the only reason why people’s pay was incorrect.
The processes that were put around the system, namely, how people got their rosters in — and I’ve explained the manual process for roster entry — was also a manual, paper-based system for making any shift adjustment. When you think about 78,000 employees and the number of shifts there are, an average of 10 in a fortnight, and every single shift with a change had a piece of paper, processing that through one centre didn’t make sense.
There was also the disconnection that occurred between the payroll staff and their understanding and knowledge of the local arrangements in a hospital. One of the things we did was reconnected our payroll services and created hubs located across the state to ensure that there was a direct relationship. The head of the payroll hub actually knew the people in the hospital rather than them being an anonymous person and they didn’t know how they undertook their work and organized their services.
We did a whole lot of things to improve the system, all of which contributed.
Senator Andreychuk: You contracted out to get a system in place, previous to you being there. If I’ve misunderstood you, please correct me. You had a contract with IBM for consultancy throughout your time there? If so, what was their role at that time? Was it a separate contract?
Mr. Walsh: At the time of the implementation, Queensland Health had its payroll services undertaken by a whole-of-government shared service entity called CorpTech. CorpTech had the contract with IBM, and my dealings were directly with Corp Tech, so I never dealt with IBM directly in this process.
IBM were the contractors to CorpTech in order to manage the system. They were there to both implement and manage. I was responsible for identifying what changes needed to occur in the system, specifying the exact business requirements that needed to be met, agreeing to the technical specifications that would achieve those business requirements and overseeing the testing before the go live. I never had direct dealings with IBM.
Senator Andreychuk: So IBM was in place for more than Queensland Health; is that correct? It was with the government on a broader basis — is that what you’re saying — and you bought into it?
Mr. Walsh: My understanding is that IBM was contracted for the Queensland Health payroll system. CorpTech was responsible for more than just the Queensland Health payroll system, but IBM was working on the Queensland Health payroll component.
Senator Andreychuk: Okay. Thank you.
Senator Pratte: You mentioned, sir, that you had read the Auditor General of Canada’s report and the lessons learned report and that you found many similarities. Did anything strike you as different from your experience? Was there anything that struck you as really different that happened here in these reports?
Mr. Walsh: You could read them and they could be applied to Queensland Health. I saw them as very similar.
The one thing that did strike me was the length of time between go live and when the reports were done and that things like overpayments, errors and people contacting the payroll service in order to deal with inaccuracies in their pay were continuing to increase. That was happening a long time after the go live.
I think the one difference is that in Queensland it was very quickly responded to. In fact, after the third pay run, we established the stabilization program and commenced a large body of work to actually stabilize the system as quickly as possible. As I said, that happened in about four months, whereas, from what I’ve read in the reports, the problems are still growing rather than having been contained and being addressed and reduced.
Senator Pratte: You mentioned a third pay run. That would mean a few weeks, really?
Mr. Walsh: That’s right. Within about six weeks, yes.
Senator Pratte: Okay. Do you have any idea what could have caused that here, by reading the reports, or what could explain why you reacted so quickly?
Mr. Walsh: I really don’t know. From what I’ve read, the reports really don’t go as to why that’s the case. I also read there is an entity or organization or governance set up to try to address the issues. They are undertaking work to identify and solve the problem, but it doesn’t seem to be reducing, from what I’ve read so far.
Senator Pratte: Since you’ve been in charge, how would you qualify your relationship with IBM?
Mr. Walsh: Absolutely fine. IBM continues to provide support in areas throughout Queensland. That’s not an issue.
Senator Pratte: Okay. Thank you, sir.
Senator Marshall: Are you still working on the payroll system or have you returned to your former job or another job? I’m trying to get an idea how long it took.
Mr. Walsh: I worked on the stabilization program and the improvement program for all of 2010, from April to December. I then actually went into the private sector and worked, then worked in the health system in another state in Australia called New South Wales. In 2015, I applied for and was successful in being the director general of Queensland Health. The role of director general is the chief executive of the health system for the 90,000 employees.
Senator Marshall: So you did what you had to do, you stabilized it, and then you moved on?
Mr. Walsh: Correct, yes.
Senator Marshall: While you were stabilizing the system, were you involved in any of the costing related to what the payroll system was costing in total? The initial implementation costs a certain amount of money. Then there’s another part where you’re trying to clean things up. That also costs money. The last figure I think I saw was around $1 billion. Can you provide a more current figure, or is that it?
Mr. Walsh: Yes, that is the figure, $1.2 billion. That included the actual business as usual running of the payroll service as well. It was not only the operations but also the corrections that needed to occur in order to resolve all the issues. We’re completing the last part of that work in relation to rolling out the rostering system so that roster managers directly manage their rosters in the system and reduce all the paper handling.
Senator Marshall: I read somewhere that there were 10 aborted attempts before it went live. Would you know anything about that?
Mr. Walsh: I don’t know about that, no.
Senator Marshall: That wasn’t within your area. Thank you very much.
The Chair: I have a question, please. With the experience that you have, being settled in supervising and looking at the umbrella of it all, have you retained any data regarding the number of pay errors by type of pay errors that you had in your system?
Mr. Walsh: I would have to go to the submissions that were made to the Commission of Inquiry because there were a lot of reports that were used at the time of the stabilization program that contained that information. I believe that the short answer is yes. I believe they would be available and could be obtained from the Queensland Health Payroll Commission of Inquiry website.
The Chair: Mr. Walsh, on behalf of the Standing Senate Committee on National Finance, we want to thank you very much for sharing your comments with us and your experience. Before I close, do you have any comments to add?
Mr. Walsh: No. I appreciate the invitation to talk with you, and I sincerely hope that it’s helpful for resolving the payroll issues for employees because, at the end of day, as you and I know, it’s actually the employees who are affected by this.
The Chair: As I close, if you have not visited Canada, I hope you will take the time with your family to come and visit Canada.
Mr. Walsh: I will definitely be visiting Canada, yes. Thank you.
The Chair: Honourable senators, we now have the officials from IBM. First, to the witnesses, as chair of the Standing Senate Committee on National Finance, I thank you for accepting our invitation and to be present in order to answer the questions from the senators.
With that, a formal presentation will be from Regan Watts, IBM Canada, Head: Innovation, Citizenship and Government Affairs; and from IBM Services, we have Beth Bell, Vice President and Partner, Canadian Public Sector Leader; and also Bernie Semenjuk, Partner, Oracle Cloud, PeopleSoft Practice Leader (Canada).
I have been informed by the clerk that each and every one will make a short presentation. I will be asking Mr. Watts to make his presentation, followed by Ms. Bell and then Mr. Semenjuk.
Regan Watts, Head, Innovation, Citizenship and Government Affairs, IBM Canada: Thank you, Mr. Chair. We have just one set of remarks where we will be taking turning speaking to the committee.
Thank you to members of the committee for having us here tonight. We deeply appreciate the opportunity to appear before you and we look forward to answering some of your questions in a few minutes.
We will begin our presentation by outlining our view of the transformation project, otherwise known as the transformation of pay administration initiative, also known as Phoenix. We think it is important to review the old business processes and applications and IBM’s specific role in Phoenix as a transformation. We also want to touch briefly on the Queensland Health project. Finally, we will talk about the investments we have made, initiatives at our own experience, that we have brought forward to help the government with the Phoenix transformation. We are here to discuss IBM’s role in the Government of Canada’s transformation project in a transparent and fact-based manner.
There has been some confusion in the media about what exactly Phoenix is or is not. To be clear, the name “Phoenix” was given to the entire pay transformation project by officials from the Government of Canada, including parts of the project that IBM was not responsible for. IBM does not have any products or services that go by the name “Phoenix,” so it is inaccurate to refer to this project as the IBM Phoenix project.
As the chair said, with you today are three senior leaders from IBM Canada. They have already been introduced.
Turning to slide 4, IBM Canada at a glance, we think it’s important the committee understand IBM Canada’s footprint. IBM has been in Canada for 101 years and has been a long-standing and trusted partner for both the federal government as well as other levels of government during this time. We pride ourselves on our public sector expertise, as well as the economic development we drive through our work across the country. We have and continue to make significant annual investments in research and development, our world-class manufacturing facility in Quebec, our software development organization, which is the largest of its kind in Canada, as well as our exports. More than anything else, we’re proud of the 13,000 IBMers located coast to coast who innovate for our clients.
Beth Bell, Vice President and Partner, Canadian Public Sector Leader, IBM Services: Turning to slide 5, we’d like to talk first about the old business processes and systems.
It is my experience that, for a transformation program to be successful, you need to have success in three key areas: people, process and technology. From the people perspective, are the people ready to accept the changes to the way they work? From a process point of view, are the new business processes clear, understood and implemented, and does the technology work?
To put the Phoenix project into context, it’s important to understand where the government initially started, and the key take away is that, in the old environment, entering information into the human resources systems — we will call them HR systems — was less critical as the HR system did not drive payroll outcomes.
Under the old way of doing things, as captured here on the chart, HR departments and departmental compensation advisors often sat in the same office. When things happened to people’s pay, like they were hired, perhaps had to act for someone or went on leave, those events were reported to both the HR and the compensation advisors. The compensation advisors entered the payroll information directly into the old payroll system. The pay amount was then verified by a verification advisor, and a cheque was issued.
On the HR side, HR branch employees would enter HR data into one of dozens of HR systems. There were, in fact, and still remain today, two concurrent systems, a payroll system and multiple human resources systems. Under the old process, HR systems and their data had no impact on payroll, and this was because payroll was a completely separate business process and software system. There was no link between the two systems. Each of the departments had its own HR staff, compensation advisors, business processes, organizational structure and, in fact, their own HR system. Departmental compensation advisors would separately calculate pay and enter those pay transactions into the old payroll system, which would determine the employee payroll. Employee data in these separate HR and payroll systems were often not aligned, meaning there was no single source of data.
The Phoenix transformation project made significant changes to how payroll was generated. While IBM was not involved in the overall design or business case for the Phoenix transformation, we understood that the Government of Canada’s business case was based on three things: eliminating the duplication between the HR and payroll systems; establishing a single source of data; and leveraging well-known commercial off-the-shelf software. For the third element, the Government of Canada chose PeopleSoft, an Oracle product that they did in an early procurement.
If you turn to slide 6, this chart demonstrates the process under the new pay system. The government’s plan for the Phoenix transformation changed the business process in at least two important ways. First, all of the compensation advisors were to be reduced and centralized in Miramichi. Secondly, data from the HR systems and payroll system would be linked for the very first time, and the data, when entered into the HR system, would feed directly into the PeopleSoft payroll system, therefore eliminating the double entry that happened under the old system.
So under this new process, pay events, such as a new hire or a promotion, are reported to local HR staff who could enter it into their departmental HR system. The data then is transferred electronically to the PeopleSoft payroll system. Any additional processing after data entry would have to be handled by one the compensation advisors in Miramichi. Then a cheque is issued. As you can see now, pay depends on managers and human resource professionals entering data into the HR system in a timely and accurate manner. Unlike the old system, there is no verification step after data entry.
If I could ask you to turn to slide 7, I think it’s important for the committee to understand the structure of the Phoenix transformation program. There were three components to the program. First, they wanted to make business process changes; second, PeopleSoft software implementation; and third was the business transformation in Miramichi.
The technical implementation of software was just one component of the overall initiative, and IBM was not involved in the business process or business transformation components of the Phoenix initiative. In fact, we were only responsible for certain elements of the PeopleSoft implementation. This is illustrated at the bottom of chart 7, where you see the IBM logo.
Although we weren’t involved in the business transformation aspect, we understand that the office of the chief human resources officer, who reports under the Treasury Board Secretariat, oversaw changes to the business processes, including the HR processes. With respect to the software implementation, the Government of Canada was still responsible for project management, training, change management and various other aspects of the program.
If you look at slide 8, if you focus on just replacing the legacy system with PeopleSoft, I don’t think we should underestimate the complexity of this PeopleSoft implementation. As mentioned, this was the largest implementation of its kind in Canada. The legacy platform was a 40-year-old customized system that was outdated and needed to be replaced. The government took a major step in moving from this outdated technology to PeopleSoft. What you’ll see in this chart is where the IBM involvement in Phoenix is focused. We were responsible for the installation and the customization of the preselected PeopleSoft software, as per the government’s requirements, and, as I said before, the overall project management remained with the Government of Canada.
With over 100 collective bargaining units, 120 government departments, 80,000 payroll roles, the PeopleSoft software installation was still a massive undertaking. On top of that, the Government of Canada requested change orders or significant customizations to the PeopleSoft system throughout the life of the project. When I say a customization, what I mean is that you have to physically break the PeopleSoft code to insert and code new functionality, and this level of customization, or the level of customization we saw here, is not the normal course of business for a project like this. While there was significant customization requested by the government, we were still able to deliver a software application that works.
I’ll turn it over to Mr. Watts for the remaining slides.
Mr. Watts: Turning to slide 9, this slide shows a simplified version of an HR event-to-pay business process under the new Phoenix transformation. That is, how do you get from an HR event, such as hiring or going on maternity leave or acting status, to receiving a paycheque?
In this instance, an employee or manager reports the event to local human resource staff. Local human resource staff then enter the event into one of the dozens of local HR systems. The data is then transferred from the HR system to the PeopleSoft application. It’s important to note that, up until this point, the PeopleSoft application has no role in the pay process. It is at that point that the PeopleSoft application and/or a compensation advisor calculates pay amount, based on data entered by a human resource professional. An authorization is sent to the bank, and the bank posts the payment to the employee’s account.
To reiterate what Beth said, IBM installed and configured the PeopleSoft application as per the requirements of the contract. As the government has confirmed, the technology is working.
Once HR systems and the payroll application were linked, things that did not previously have consequences on pay under the old system, such as a delay or an error or data entry, now had impact on pay. Let me be clear. We believe the vast majority of the problems related to the Phoenix transformation lay primarily with HR business processes, the amount of training received by employees, including HR and compensation advisors, data entry, including accuracy and timeliness, and a lack of ongoing and root-cause analysis.
Turning to the next slide, I want to talk very briefly about Queensland Health. We had a presentation before this, which I think was very helpful, and there has been talk about the similarities between the two projects, but there are also some differences.
First of all, as was mentioned, they were two completely different payroll systems, both with third-party products. Queensland Health choose SAP. The Government of Canada choose Oracle. Neither product was developed by IBM.
In the case of Queensland, the client believed its old pay system was on the brink of collapse and chose to build an interim system with minimal functionality. They also rushed the procurement, design and build of the system, which caused them to underestimate their business rules and other key aspects of the project. The result was a severely mis-built project. These aren’t points drawn from IBM, but they are from the Queensland Parliament’s official inquiry report. While we don’t agree with everything in the report, what is clear in the Queensland report — much like Goss Gilroy and a report that they did here — no blame was placed on IBM for the pay problems that had occurred.
Contrast this approach to Phoenix, which was replacing an aging system, albeit one that was still functioning. Phoenix was undertaken with a longer-term view as part of a broader business transformation within the public service.
Finally, and turning to the last slide, we want to conclude our remarks outlining some of the investments we have made to help the government with this transformation. The committee has heard testimony about IBM’s ongoing support contract. We have changed the structure of that contract, which has already saved the government millions of dollars. IBM Canada has also invested in helping the government with its Phoenix transformation and will continue to do so. We have been active in our advice and our engagement with the government. We have invested, on our own dime, in helping the government, including providing two project executives, conducting a detailed root-cause analysis, data analytics; and, more recently, we have provided an executive advisor on large payroll systems who is advising the minister and the department.
These investments, which run into the millions of dollars, have been made because we want to help our client in the best way we know how — bringing resources to bear and leveraging technology and people to solve problems.
Lastly, the most important point we want to leave with the committee is this: IBM is fully committed to working with the government in its effort to restore the issues related to Phoenix. We believe the public service deserves to be paid by the government on time and accurately for the work they do on behalf of Canadians, and we will do everything we can to assist the government as they work to fix these problems.
We would be pleased to answer questions.
Senator Marshall: Thank you very much for your presentation.
We were provided with a copy of the contract between IBM and the government. I didn’t have a chance to read it; I just glanced at it. There were a couple of areas there, and I just want to relate what is in the contract to the chart you have on page 7.
My understanding is that, in accordance with the contract, IBM was to provide management and oversight services, implementation and business transformation services, and ad hoc and maintenance coordination services. First, could you talk about the management and oversight services and explain that in relation to the chart that you have on page 7?
Ms. Bell: For the work we did, we, of course, had responsibility to oversee that work, but we weren’t the overall project manager or program manager for the entire transformation program.
Senator Marshall: When you went through the chart, you indicated that for the transformation of the pay system, you were not responsible for it overall. You were responsible, in part, to replace the legacy payroll application with PeopleSoft. You said partly responsible for that, and also that you were responsible for the installation and customization of PeopleSoft software, and that system had been preselected by the government. What would your role have been with these other areas, like the overall common business processes and the business transformation in Miramichi?
Ms. Bell: We had no involvement.
Senator Marshall: And under the contract?
Ms. Bell: No involvement.
Senator Marshall: What about the implementation and business transformation services? Was that restricted to the replace legacy payroll application with PeopleSoft?
Ms. Bell: Can you repeat the question?
Senator Marshall: In the contract, it says IBM was responsible for providing implementation and business transformation services. What exactly is that?
Bernie Semenjuk, Partner, Oracle Cloud, PeopleSoft Practice Leader (Canada), IBM Canada: I was actually with the IBM team, the IBM project executive, that was involved in the Phoenix implementation from about the summer of 2015 through to now. The transformation services that are referred to there would be the change management and training box that you see on slide 8.
Senator Marshall: Were you responsible or a participant?
Mr. Semenjuk: At the beginning of the program, IBM was responsible for change management and training. In 2014, the government took that responsibility back.
Senator Marshall: Do you know why that was?
Mr. Semenjuk: I don’t know.
Senator Marshall: What about the ad hoc and maintenance coordination services? What is that?
Mr. Semenjuk: That would be services that we are providing currently in terms of maintaining the PeopleSoft application. We provide systems maintenance and database maintenance, and we help to operate the daily activities of pay processing.
Senator Marshall: So for all of these processes that had to be carried out before anything went into the payroll system, the payroll application, you are saying you had no involvement in that?
Ms. Bell: Correct.
Mr. Semenjuk: That’s correct.
Ms. Bell: The business process change on the HR side, correct.
Senator Marshall: That was totally under the government’s responsibility?
Ms. Bell: Yes. That would be on the left-hand side, where it says “overhaul common business processes”?
Senator Marshall: Yes, that’s the one I’m referring to.
Ms. Bell: Yes.
Senator Marshall: And the business transformation, Miramichi, you had absolutely no involvement in that at all?
Ms. Bell: Correct.
Senator Marshall: Did I understand correctly that you led this project?
Mr. Semenjuk: From an IBM standpoint, I led the installation and customization portion from approximately the middle of 2015 through to today.
Senator Marshall: Was there somebody else before that?
Mr. Semenjuk: Yes, some of my colleagues prior to me.
Senator Marshall: Are they still with the company?
Mr. Semenjuk: Some of them are, and some of them are no longer with IBM.
Senator Moncion: If I heard you correctly, you said you did some training, but at some point the training was taken away from you?
Mr. Semenjuk: That’s correct. In 2014, the government raised essentially a contract amendment that the government took back training and change management as their responsibility.
Senator Moncion: You knew the software, so were they — it’s unusual. Usually the company will keep up the training until the transformation is done, and then if there are corrections to be made or adjustments to be made.
My other question is a little bit more technical. Oracle is a database. The government had a database that it was using to process its old payroll. How much data conversion was done between the two systems? Was it data that was keyed in or was it a system-to-system transfer that was done?
Mr. Semenjuk: It was an automated conversion, so there were actual programs that did most of the data conversion, but there were also some elements of the data conversion that government employees were responsible to enter.
Senator Moncion: How much control did you have? Because in an IT solution, it’s garbage in, garbage out. You want to know how accurate is the information that is brought from an old system into a new system. How was that handled? That had to be done by you.
Mr. Semenjuk: Not quite. The government was responsible for extracting the information out of the old payroll system, the regional pay system, and cleansing it; and then IBM was responsible for loading the cleansed data into the PeopleSoft application.
Senator Moncion: So you provided them with the type of clean data that needed to be — actually, you gave them the template that needed to be filled so that the information that was going to be brought into Oracle would be the right information?
Mr. Semenjuk: Correct. We provided templates. The government was responsible for populating these templates with cleansed data.
Senator Moncion: The problem could not be, then, with the data entry?
Mr. Semenjuk: The data entry during conversion?
Senator Moncion: Yes.
Mr. Semenjuk: Sorry; I’m not quite following your question.
Senator Moncion: If the data was cleansed and the information that was put in was the right information —
Mr. Semenjuk: We actually did 13 data conversion tests, and in none of those was the data clean that was coming in. So the cleansing of the data that was the government’s responsibility had never passed all of the tests during those 13 data conversion tests that we ran.
Senator Moncion: So you knew that before the conversion?
Mr. Semenjuk: On the final runs, there were a limited number of errors, and we raised those as part of our assessment of the system readiness. Those were deemed acceptable by the government, and they would do some data entry in the new system to correct the remaining errors.
Ms. Bell: That’s for historical data, because then pay events happen once you’re on the new system, and therefore the data entry would be into the HR system as you went forward. This was just to move over the historical data.
Senator Moncion: What kind of pilot project was done? Was there a pilot done? Was there just data that was verified? If so, how long was the pilot done for?
Mr. Semenjuk: There was a pilot in the original plan for go live in 2015. There was a pilot that was planned for July 2015. But in June 2015, the government cancelled that pilot.
Senator Moncion: Do you know why they cancelled it?
Mr. Semenjuk: The overall solution, including the business processes, the training of the people and the software were not ready for that pilot.
Senator Moncion: And they went ahead with doing a conversion, going live, without a pilot? That’s what you’re telling me?
Mr. Semenjuk: Yes, there was no pilot done.
Senator Moncion: Thank you.
Senator Eaton: I just want to follow up on Senator Moncion’s questions.
Why did the government choose an off-the-shelf system? With the complexity of all those ministries and different kinds of pay systems, would it not have been more creative for them to invite you, the private sector, to come in and say, “Listen, our pay system needs upgrading. What do you suggest and how should it work?”
Ms. Bell: Typically, organizations for HR payroll or for financial systems do tend to start with the commercial off-the-shelf products.
Senator Eaton: I don’t think a government this size is a typical issue.
Mr. Watts: Senator, the decision was made by officials of the previous government. We can’t speculate as to why they preselected PeopleSoft.
Senator Eaton: Cheaper?
Mr. Watts: You’d have to speak with those officials who made the decision. We weren’t involved in the process. We came in after PeopleSoft was procured and selected.
Senator Eaton: In other words, IBM came in, and, “Here is a list of specifications: make it work”?
Mr. Watts: Sort of. It was, “We’ve procured software. We need an organization and a team to customize and implement that software. Who can help us do that?”
Ms. Bell: They gave us a very extensive list of specifications.
Senator Eaton: That’s what I’m saying. They did it that way rather than calling people in and saying, “What do you suggest?”
Mr. Watts: Correct.
Senator Eaton: Yesterday, we had Ms. Moir from Goss Gilroy, and she confirmed that the launch could have been delayed even up to the eleventh hour. Do you agree that it could have been possible to delay for a few months? That’s getting to Senator Moncion’s question, to do a pilot. Do you have any idea why it wasn’t delayed, what the rush was?
Mr. Watts: We, as early as July and August 2015, right around when Bernie started, raised concerns about suitability for the go live and continued to have a discussion with the government throughout that fall and into early 2016. Beth can speak to more of the details, but as early as the summer of 2015 and as late as January/February 2016, we had signalled to the government that we didn’t feel the software and the whole transformation was ready.
Senator Eaton: And the training.
Mr. Watts: The training was a part of the transformation.
Senator Eaton: One of the things they remarked yesterday was that perhaps there was a culture in Public Services and Procurement and Treasury Board that they didn’t want bad news.
Ms. Bell: I do think we agree with Goss Gilroy on that. There was a culture at the time that made it difficult to speak truth to power and that bad news didn’t flow uphill.
Senator Eaton: We had the minister here last week. It was very interesting. They’re going to be evolving, I guess, into a new system in five or six or seven years. What advice would you have to give her when they start looking at new systems?
Mr. Watts: Shall I start, Beth?
Ms. Bell: Sure.
Mr. Watts: It’s important to remember that it’s not the software that’s the problem.
Senator Eaton: I’m not saying it is. I’m saying how would you go about it?
Mr. Watts: I want to make sure that much is clear. The software works.
The challenges associated with the project have been around process and people. Beth, who has done these projects across the country, as has Bernie, and IBM, who does these projects around the world, we think of success in these projects as having success with technology, success with people, people management training, change management and business processes.
Senator Eaton: But, Regan, where would you start if you were minister?
Ms. Bell: I would treat it as a transformation program.
Senator Eaton: So let you guys come in and be creative, not give you a whole lot of specifications? Is that what you’re saying?
Ms. Bell: No, I would recognize that you’re not just implementing a payroll system. You’re changing the way employees work. You’re changing the way they generate pay for all of the civil servants across Canada. When you do a big transformation, you have to really govern it and think about it as a transformation as opposed to, “I’m just installing a payroll system.”
Senator Eaton: Thank you very much.
Senator D. Black: Thank you very much. I recognize, of course, that my time is limited, and I’m very respectful of your being here. I would imagine it’s not the place you choose to be on a Wednesday evening.
Ms. Bell: It’s always good to meet new people.
Senator D. Black: Let’s go with that.
I hardly know where to start. I come from a background of being an external consultant, as you two are as well, and I’ve worked on many large projects. I want to explore your relationship with your client.
I think the answer to my question was found in something that Mr. Watts recently said, that you indicated there were problems, you thought, with implementation — let’s hold our horses here and figure it out. Is that accurate?
Mr. Watts: We did express reservations beginning July and August 2015 that we weren’t confident the transformation would be successful.
Senator D. Black: Would you characterize your relationship such that when you had concerns as you moved forward with the project, you were being open with government about what your concerns were?
Ms. Bell: Maybe it would be helpful if I could tell you what happened in the ongoing dialogue during that time.
Senator D. Black: That would be very helpful.
Ms. Bell: As another senator indicated, the pilot was cancelled, but there was a scheduled go live to go live in two phases in 2015, both October and December. Around the July/August time frame, IBM, based on its understanding of the project and the visibility that we had to some of the other aspects of the project, started to signal to the government that we’re not sure that you’re going to make October/December of that year. The executives we were working with asked us, well, when could we make it. We did some work, came back and said that we think July and August 2016, based on the volume of work and changes they still had for us to do, would be a good time frame. Now, that wasn’t a time frame that was great for them. This is part of the ongoing dialogue. They really needed to go live in full by April 2016 because the compensation advisors had already been given their notices.
Senator D. Black: Had you been consulted on that?
Ms. Bell: No, that was outside our purview. They had been given their notices, so they came back to us and said, “Listen, we have to go live in April, fully live.”
Senator D. Black: Because we can’t pay our people?
Ms. Bell: Because the compensation advisors, that part of —
Senator D. Black: I understand. I’m not being critical; I just want to understand.
Ms. Bell: I understand, sir. So we came back to them and said, “Okay, so there’s this much work left to do, and we think that’s going to take until July and August. What of this work could you not do right now so we can go live in April for you?” It was a February/April two-stage go live.
We worked together, gave them some guidance in terms of the amount of work we could do, and they made a selection of what would go in that go live and what functionality would be deferred until after the original go live. We in fact went live in February and April 2016.
Senator D. Black: You would say you went live against your advice?
Ms. Bell: Prior to January 2016, we had some visibility in the governance as to what was going on outside so we could make some assessments. In late January 2016, we didn’t get any more reports telling us what was going on outside, so we were in a position where we couldn’t provide any more advice on the whole transformation program.
Senator D. Black: But why, as a sophisticated consultancy, would you accept that from your client? Why did you take that risk? Why didn’t you resign?
Ms. Bell: Why didn’t we resign?
Mr. Watts: Senator —
Senator D. Black: No, no. Please.
Ms. Bell: So at the time — and maybe I can answer your question this way — we were really put into a position where IBM just focused on the software implementation. We did continue to express our concerns, but the executive at the time told us, “Stick to what you’re responsible for.”
Senator D. Black: Why did you accept that?
Ms. Bell: Because at the end of the day, our responsibility to our client is we knew we could deliver what they wanted within the scope of our work, and that was our responsibility and we delivered on that.
Senator D. Black: Even though — and I’m not being critical of you — you knew at the time that the risk of failure was high?
Ms. Bell: We did not know at the time because we no longer got any of the data on the broader transformation updates after the end of January 2016. So we didn’t know, senator.
Senator D. Black: Did you find that acceptable, that you were cut out of the loop?
Ms. Bell: Not really.
Senator D. Black: Did you express that? And it made no difference?
Ms. Bell: We were asked to focus on the work —
Senator D. Black: “Stick to your knitting; if I want to hear from you, I’ll ask you?”
Ms. Bell: Correct, sir.
The Chair: Mr. Watts, any comments?
Mr. Watts: No, nothing further.
Senator Andreychuk: Somewhat on the same line as Senator Black: Mr. Semenjuk, you said you came in about July 2015?
Mr. Semenjuk: That’s correct.
Senator Andreychuk: Ms. Bell, you were involved from what time?
Ms. Bell: I became involved somewhere around the end of December 2015 in an executive oversight role. Mr. Semenjuk was working day-to-day on the project, and I was working with the assistant deputy minister in the overall governance of our work with the Government of Canada.
Senator Andreychuk: Mr. Watts, when did you become involved with Phoenix?
Mr. Watts: I joined IBM in May 2016.
Senator Andreychuk: Were you involved with Phoenix or just IBM?
Mr. Watts: Just IBM.
Senator Andreychuk: Who is the person, the lead person, for IBM throughout the process, starting and ending? If there was a change, who are they?
Ms. Bell: We had a number of partners, I think two partners, that started in about 2009 and left the project in about 2015 when Bernie and another partner took over from them and continued the project.
Senator Andreychuk: So there were two people in charge? There wasn’t one single lead person that you could point to?
Ms. Bell: They performed different roles. Bernie, you can probably talk to that.
Mr. Semenjuk: Yes. There were two partners that were involved when I was there: myself, sort of what I would call focusing down and into the project, managing day-to-day work activities, and then my colleague who was focused on managing more up and out.
Senator Andreychuk: Who would that be?
Mr. Semenjuk: I don’t know that we’re — are we permitted to —
Mr. Watts: Yes.
Mr. Semenjuk: Gary Hunter.
Senator Andreychuk: Is he still with IBM?
Ms. Bell: No, he resigned. He left IBM.
Senator Andreychuk: When would that have been?
Ms. Bell: Last week.
Senator Andreychuk: So there was not one project. It was functional. Is that what you’re saying? Two people? You did one thing, Mr. Semenjuk, and someone else? Would that have been the system before when Mr. Hunter was there?
Mr. Semenjuk: Mr. Hunter and I worked together. Correct.
Senator Andreychuk: Was there a difference of opinion?
Mr. Semenjuk: Between Mr. Hunter and myself? No. We were very much aligned. I would say that sort of governance structure is fairly typical on these very large ERP transformations where there is a lot of work and it’s a very complex project, so that the responsibility division was very clear.
Senator Andreychuk: On the other side with the government, who did you deal with? Who was the point person with the government?
Mr. Semenjuk: My contact was an individual in PSPC. She was my counterpart. And then Gary had a number of senior leaders within PSPC that were his primary contacts.
Senator Andreychuk: I guess our problem, or at least my problem, is trying to figure out who was in charge on the IBM side and who was in charge from the government side. We’ve heard that the government would, at the political level, get advice. We’re trying to figure out who gave that advice to the first government and the second government. Is there one person that was your point person? If you saw problems, who did you go to on the government side to say, “Hey.” You said you signalled them. Was it, “This isn’t working and this isn’t working,” or did it ratchet up to the point where you said, “This is really a problem the top person needs to get,” and if so, who would that be?
Mr. Semenjuk: The individuals we dealt with on a project leadership level from PSPC were Brigitte Fortin and Rosanna Di Paolo.
Senator Andreychuk: At any time, with your worries or difficulties, did you go beyond those two?
Mr. Semenjuk: Senator, we did not.
Senator Andreychuk: No?
Mr. Semenjuk: No.
Senator Andreychuk: What were the change orders in July/August 2015?
Ms. Bell: Actually, can we correct that? We had a number of meetings at one point in time in early 2016, just to be factual, with Gavin Liddy as well.
Mr. Watts: That was post go live.
Ms. Bell: No, in January.
Senator Andreychuk: Not with any ministers or anything?
Ms. Bell: No.
Senator Andreychuk: Who authorized the change orders that led to some of the difficulties? You said in July/August 2015 you got some change orders to your contracts, I presume?
Mr. Semenjuk: We received change orders throughout the project, over 1,500 change orders. Over and above the original scope of the project, there were 1,500 change orders that either added scope or we talked about in some cases where scope was taken away from IBM and given back to the government in terms of change management and training. Those change orders were all authorized by the leadership within PSPC.
Senator Andreychuk: Having a little to do with corporations at some point, it may have been your contract that you fall back on, I appreciate that, being from a legal background. But at some point, you have to understand the culture you’re working in. Did you come to a conclusion that, “This isn’t going to work for the client and we have some responsibility to alert the client that with all these changes with the culture and being removed from some of it, we can’t possibly deliver with confidence,” rather than sort of saying, “The parameters of the contract are what we’re going to deal with”? It’s within the milieux of corporate responsibility to say, “This isn’t where we started. We’re going in different directions. Should we be here?”
Ms. Bell: I think it’s fair to say — and this happens on every project we’re on where there’s a commercial off-the-shelf software — that in a perfect world, you would just accept the software for what it is and change your business processes to adapt to the software. Okay? So I would say that over the course of the project, as change orders came up, we would have had a discussion on each specific level to say, “Okay, you would like it done this way. PeopleSoft does it this way. Would we want to change the business process that the Government of Canada has to adapt to the way PeopleSoft does it?”
There’s an example we used with the ministers last year, if you would like me to share. It’s quite simple. In the Government of Canada, if you’re a French-speaking person and the job you’re in has a French-speaking requirement, you’re entitled to a French-speaking bonus. Now, if all of those things are true and you flag a person as French speaking and they’re in the French-speaking job, PeopleSoft will pay them a bonus. The application can be configured to do that.
The Government of Canada’s business process is that they will only pay the person the bonus until they’ve been in the job five days. That is a customization to the PeopleSoft system. The discussion we had at the time with them was would they want to change that business process so that they paid the person on day one instead of waiting until after day five, because then we can deliver it without making any changes to PeopleSoft. That’s a really simple example where the Government of Canada said, “No, this is our business process. We need you to make the customization.”
In all these things, we would have compared what they wanted to the way the PeopleSoft system does it and had a discussion on whether we could adapt the business process. Is that fair, Bernie?
Mr. Semenjuk: Yes, that’s very accurate.
Senator Andreychuk: I just have one more question. Having been in business and having clients, at some point, you knew how complex the Canadian government is, its pay systems and all of that.
Ms. Bell: Yes, senator.
Senator Andreychuk: So you knew it wasn’t an easy task at all. At some point, you say, “We’re getting these orders, and they want the customizations. Maybe we need to call a halt to this.” I think that’s a bit of what Senator Black was saying.
Mr. Watts: Senator, that’s precisely why, beginning in July and August 2015, we were raising the concerns we did. As a responsible provider, if you’ve been in the advisory business, you know you give your best advice.
Senator Andreychuk: Would that have been Mr. Semenjuk or Mr. Hunter?
Mr. Semenjuk: Both of us working together to raise those issues. I would have been working more focused with the team to analyze the impact, and Mr. Hunter would have been the individual that largely was communicating those to the senior leadership in PSPC.
Senator Andreychuk: So the critical point was July and August 2015?
Mr. Semenjuk: Correct.
Mr. Watts: Correct.
Senator Andreychuk: Where you said, “Problems.”
Mr. Watts: “We have some concerns.”
Senator Pratte: Beside the size of this project, when you decided to bid for this contract, what was special about it, or was there anything special compared to other projects you were involved in?
Mr. Semenjuk: In terms of the nature of the project to implement a PeopleSoft payroll solution, I’ve done many of those. That’s a very common approach.
I would say the nature of the procurement and the details of how that was done were unique in my experience. We’ve heard a little bit of that where the specifications were very detailed and exact: “Please deliver this. Once you’ve delivered that, we’ll give you another task authorization to do the next thing. Deliver that and that. After that’s completed, we’ll give you another task authorization for the next piece of work.” That was unique in my experience.
Senator Pratte: Was that something you knew when you bid for the contract? Did that cause you to hesitate at all or cause any concern?
Ms. Bell: Neither of us were there. It would have been in 2008. It’s hard for me to speculate what was in the mind of the team who wrote the RFP.
Bernie is correct. We implement. This is quite a normal thing that we do for clients.
Senator Pratte: You might think that from the client’s standpoint, to have a very detailed contract — I’m not in this business at all — it might be a good thing. You know exactly what you want and get exactly what you want. Isn’t that a good thing?
Mr. Semenjuk: It has some benefits in terms of being very precise. The other part is that — and we’ve talked about this a number of times — with these transformation projects, the technology is only one portion of the overall ability to be successful. Having focus also on the business processes and the people in terms of training, education and so on — those three things combined, in my experience, lead to a successful project.
Ms. Bell: This project changed the way people worked, so it’s a big undertaking for that level of transformation.
Senator Pratte: When you were responsible for the training part, or at least when you were involved for the training or the change in training part, what did you do exactly? Up to what point did you achieve anything? Did you actually train people? To what extent did you do that? Or was it just part of the planning? Did you prepare some courses or documents for the training?
Mr. Semenjuk: It was prior to my involvement on the project. In 2014, the government took that functionality back. But my understanding is that IBM would have only been involved in developing the overall change strategy and the training plan.
Senator Pratte: Thank you.
Senator Marshall: First of all, I need clarification on something that Ms. Bell had said. You were advocating changes. You said “leading up to July and August.” Was that July and August 2015 or July and August 2016?
Ms. Bell: It was 2015.
Senator Marshall: Okay.
Let’s talk a bit about what happened in 2016. Based on your testimony, my understanding is that the government started to push you back so that you weren’t so involved any more. What’s your involvement with the government now? Bridge what happened between then and where you’re at now. Do you have any involvement with the government?
Ms. Bell: Yes. We’ll talk specifically on this project, with which we’re still involved. I would say our involvement is quite a bit different with the executives who are in charge today. I meet every two weeks with the associate deputy minister, Mr. Semenjuk and I, and he brings his assistant deputy ministers. We sit around the table and talk about things we’re working on together — different initiatives. I feel comfortable to raise issues and provide advice, and we have discussions on those.
As well, the President of IBM Canada, Mr. Ayman Antoun, meets with Minister Qualtrough every two weeks.
Senator Marshall: What are your terms of reference? What is your involvement? What are you trying to do, or what are you doing for the government now in this capacity?
Ms. Bell: At the point we’re at now, we’re providing advice. We’ve made some investments in data analytics and some advisory personnel. We’ve done some work helping them with root-cause analysis. We’re also maintaining the system and are in the process of taking over the full maintenance of the system. All of our discussions are surrounding all those areas of the project.
Senator Marshall: We’ve had testimony for Mr. Linklater, who is an associate deputy minister.
Ms. Bell: That’s who I meet with every two weeks.
Senator Marshall: Are there other associate deputy ministers or deputy ministers? In previous testimony we’ve gotten, the impression — not the impression; it was in the Goss Gilroy report — that everybody wasn’t involved to the extent they should have been involved. Now that the government has these major problems, are you meeting with people who are involved? Is that problem now solved?
Ms. Bell: I would say — and Bernie, fill in if you think it’s required — largely my interactions are with PSPC. Mr. Linklater is an associate deputy minister. I have met him multiple occasions, as well as the Deputy Minister Lemay and various of his assistant deputy ministers, as well as the CIO for CIOB. IBM is not part of the broader governance.
Senator Marshall: I was going to ask you that.
Ms. Bell: Yes. We’re not part of the broader governance.
Senator Marshall: In your charts, on page 7, are you still restricted to the PeopleSoft software? Are you involved now overall in these common business practices? You’re not involved in any of that.
Ms. Bell: No, we’re not.
Senator Marshall: You still have that very narrow focus?
Ms. Bell: Correct.
Mr. Watts: Correct.
Senator D. Black: I want to follow up where I left off a moment or so ago. This exchange with my colleagues has been very helpful.
Mr. Semenjuk, you indicated, I think very helpfully, that in order to be successful on a big project, the tech, which you did, was one part, and you’re saying we did a fine job of that. But to be successful, there must be coordination on the people front and the process front, and your testimony is that you were frozen out of both of those, for whatever reason. They didn’t like you, they thought you were incompetent, they thought they knew better, who knows the answer, but you were frozen out?
Ms. Bell: We were not involved in the other areas.
Senator D. Black: You were frozen out. You weren’t involved in other areas. And then you tell us you continued to raise concerns, and then at some point I presume you had an internal conversation and you asked, “Are we leaving, or are we just going to hold our nose and drive forward?” You did the latter, obviously. You’re here today.
Ms. Bell: I don’t know if I would characterize it that way.
Senator D. Black: How would you characterize it?
Ms. Bell: We came to an agreed scope for go live that we felt we could execute on. We modified. We did not make all of the customizations that were on the government’s list at the time. Before go live, we said, “If you’d like to go live in February-April, we can do this much work. What would you like to put in this much work?” And they decided on what things they wanted. They prioritized some things and de-prioritized others, which were done at a later date, after go live, and we were comfortable we could deliver that scope within the time frame they needed us to.
Senator D. Black: I didn’t quite understand that, so that’s a helpful clarification. I understood that they wanted to go live on a certain date. You said, “I don’t think we’re going to get that done.” They then said, “And, by the way, we fired our people. We have no people; we have no choice; we have to go live.” That’s what I heard you just say.
Ms. Bell: Let me revisit.
The original go live schedule was October and December 2015. In the July and August time frame, IBM started to say to the Government of Canada that we had concerns with the readiness overall. We voiced those concerns, and the Government of Canada said to us, “Okay, well, if not, October, December, based on what you’re seeing, when would you recommend?” And we said July and August 2016. And they said to us, “That’s not satisfactory to us because we have to be fully live by April 2016 because we’ve already given the CAs their notice and that was the date.” At that point, we said, “Okay. Well, you’ve got this long list of changes you still want done. We can’t do all of that and be live in a judicious way by April 2016,” and I go back to, “We can do this much before then; what would you like us to do?”
Senator D. Black: During this process, were you in touch with your colleagues who implemented the Queensland project?
Ms. Bell: No, we were not. Not to my knowledge.
Mr. Semenjuk: Not to my knowledge.
Senator D. Black: Would you have known that the Queensland project that your firm did had also run into the types of problems that you were forecasting? Did you know that?
Ms. Bell: I don’t think we knew.
Senator D. Black: That’s over there and we’re over here. Interesting now to see that things were kind of similar, but you didn’t know, so fair enough.
I have not had an opportunity to look at the contract yet, but in terms of the way you were paid, was your contract fixed price or did you have certain incentives that you could earn fees depending on where the project was at?
Mr. Semenjuk: The majority of the contract was fixed price based on task authorizations that included a set of deliverables. There were some task authorizations — a very small portion of the overall contract — that were based on time and materials.
Senator D. Black: Would the go live decision have been a financial event for you?
Mr. Semenjuk: There was a deliverable that was the go live readiness report.
Senator D. Black: If you had not gone live, you would have earned less money than if you had gone live.
Mr. Semenjuk: In terms of that deliverable, we would not be able to complete that deliverable and therefore get paid for that deliverable until the go live was executed.
Senator D. Black: How significant would that be in terms of the contract, in your view?
Mr. Semenjuk: In dollar amount, I’m sorry, senator, I don’t have that information offhand.
Senator D. Black: And you can’t give me an order of magnitude?
Mr. Semenjuk: No.
The Chair: Would you provide us with that information on the last question?
Mr. Semenjuk: Sure, I would be happy to.
Senator D. Black: I’m not casting aspersions on anyone. I’m just asking whether or not your decision to move ahead and not tell the government, “If you want to go live, you find another partner” — I’m wondering whether there was a connection between that and the fees you would have earned.
Mr. Semenjuk: No, there was not.
Senator D. Black: Okay. You can see why we might ask the question.
Mr. Semenjuk: Sure.
Senator D. Black: In terms of litigation, have you considered suing the Government of Canada?
Mr. Watts: Senator, we have strong relationships with all our clients. We work very closely with them to achieve their goals.
Senator D. Black: So the answer is no.
Mr. Watts: Suing clients is not something that IBM does on a regular basis.
Senator D. Black: Thank you.
Senator Andreychuk: I want to be sure that I’m correct. You started identifying more serious problems in July, August, 2015, but January 2016, when you were told that they wanted to go live April 1, you said, “Can’t do it?”
Ms. Bell: We started to identify issues in July and August 2015. The discussion on what would go live and what was going to be in the April go live continued through the end of 2015, because I believe the final agreed to go live specification was agreed to at the end of 2015, in December.
Senator Andreychuk: And it was a reduced go live?
Mr. Semenjuk: Reduced scope.
Ms. Bell: Of all the list of things there to do, we eventually did all of them. We only did some of them for the go live. Think of it that way.
Senator Andreychuk: You had a contract, and somebody had to say to you we’re going April 1?
Mr. Semenjuk: Correct.
Senator Andreychuk: Who was that, in January? Who would that have been? You reduced the scope for April but then extended it further, as I understand, to finish it off.
Mr. Semenjuk: The decision to go live, we provided our recommendation from the standpoint of the software application readiness. We provided recommendation for that. The recommendation and assessment of the other components of the project in terms of people readiness, training effectiveness, change management of the business processes and those things, those were outside of our purview.
Senator Andreychuk: I appreciate that, but when you were told they were going to go April and it was reduced, when was that?
Ms. Bell: The determination of that reduced scope culminated in the last week December 2015. We came to agreement on what that reduced scope would be.
Senator Andreychuk: And there was no change thereafter until April?
Ms. Bell: Correct.
Senator Andreychuk: When did you finish the rest?
Mr. Semenjuk: We finished the scope that was deferred in the fall of 2016, and throughout 2016, so there were multiple points where those deferred scope items were delivered throughout 2016, but the last items delivered in the fall of 2016.
Senator Andreychuk: Are you under a new contract, then, from that point until now?
Mr. Semenjuk: The contract that Ms. Bell referred to in terms of the application maintenance work that we do, that is a separate task authorization under our contract.
The Chair: Before we close, I have a question or two.
I’m not putting any doubt on the professionalism of your company nor any of the witnesses this evening, but I believe it’s truth to power and I believe we have to find a solution. There are many of our employees who are hurting out there. If I look at the transactions, there are over 635,000 transactions in which we need to find solutions. We can play the blame game, but one of the key problems of the system is its inability to automatically process retroactive pay issues — and you’ve been there since the beginning — which was scoped out of the project. Did you raise any issues with respect to the lack of retroactivity in the system to the people that you worked with?
Mr. Semenjuk: The retroactive acting that you are referring to was one of the elements that the government determined to defer from the April go live. That was then delivered. So the automation of that was then delivered in the fall of 2016. Our latest statistics show that of the retroactive acting transactions, if they are entered, 93 per cent of those get processed correctly.
The Chair: If we look at the numbers provided to the Finance Committee, we’re looking at what must be voted on in Budget 2018-19. When we do tallies since 2009, we have over a $1 billion problem and we have not fixed anything.
The government announced that it will eventually stabilize and eliminate the Phoenix pay system. I’m sure you are aware of that. How long do you think it will take to stabilize Phoenix and help our people?
My last question is, what would you recommend to the government with the experience that you have?
Mr. Watts: On the first question on the financials, you are a committee studying a budget tabled by the Minister of Finance. It’s not for IBM to comment on the fiscal plan of the Government of Canada. The government has indicated in the recent budget that $461 million will be allocated and presumably spent over the next four or five years to continue fixing the problems associated with Phoenix. We’re going to be a good partner and be with the government throughout that time. We expect the system will continue, and we will continue working with the government on that project.
Your next question was on the fix. There is no silver bullet. This is a complex project with a number of factors. There is no immediate or sweeping action that could be taken to resolve all of the issues with Phoenix. What it will take is strong coordination among technology, people and process, as well as other things we talked about, root-cause analysis, change management, training, data entry and accuracy. Those are all actions that will lead to these problems getting addressed and fixed. From our perspective, we expect and look forward to being a partner with the government as we work through these challenges.
Your last question was about the advice we would give the government. I think we have addressed that earlier, but ensuring all the stakeholders involved in a project, regardless of their role, and taking a seat at the governance table to address those three pillars we talked about is critical. We see more of that happening, and we look forward to continuing to play a role.
The Chair: Given the comments we just heard, would you recommend to the government of the day to eliminate Phoenix?
Mr. Watts: It’s important to remember that Phoenix refers to the whole transformation project: software, people and process. If your question is should we scrap the PeopleSoft software and start fresh, the answer is that that will not solve anything. You heard in the testimony from Queensland prior to now that they have continued with the software. They worked to address the root cause issues, which is advice we would give this committee and we have given the government, and worked through the challenges as they come up. Replacing the software will not wash away the issues.
The Chair: My last question to the officials with experience since 2009 is, when do you think we can fix Phoenix?
Ms. Bell: It’s really hard for us to say because, again, we don’t have the broader governance visibility to understand the full picture of all the issues. We know the issues that we have been disclosed on and provided solutions and/or advice on those, but we don’t sit on the broader governance and we don’t see the full picture, so it would be really hard for us to take a guess.
The Chair: To the three representatives of IBM, thank you very much for sharing our comments with us. It has been very educational and informative. After we finish reading that 1,700-page contract agreement that you have with the government, we might ask you to come back.
On that basis, thank you very much.
(The committee adjourned.)