Call center agents around the globe are trained to communicate effectively with customers, internal and external, from other cultures. A metaphor for this is the movie Analyze This, in which comedian Billy Crystal plays a very serious-minded therapist to tough-guy Robert DiNiro’s character, who has anxiety attacks and is trying to get in touch with his feelings. World-class actors as they are, they did their best to pull it off convincingly. In the end, however, watching them behave in ways that are contrary to their nature was comedic. It made us laugh. So, is it realistic to expect a call center representative to pretend to be someone he or she is not and to pull it off convincingly? Is that not what we do when we put call centers halfway around the globe and ask customer service representatives in India and the Philippines to speak with an American accent and be prepared to chitchat about how the New England Patriots did in the Super Bowl?

The answer to that question is no. There is some level of consensus that customers are not fooled. Customers understand that they are talking to a person who is culturally and linguistically different from themselves. It is, however, possible to train customer service agents to communicate effectively across cultures, allowing customers to have a positive experience. This involves a lot of accent training and a thorough understanding of how customers in different places tend to react to situations.

This article is about applying a methodology to allow you to know how successful your organization is in this area. First, the focus is on customer interaction via the telephone, and then we will explore whether this methodology can also allow you to know how successful your software and website translation are.

Before we explore the evaluation methodology, however, it is useful to take a closer look at the media of interaction we are evaluating and how various dynamics of translation can affect its effectiveness. Context is an important dimension in cross-cultural communication. To demonstrate, let’s look at two scenarios for accomplishing the same thing.

An American executive needs to transfer money from one European account to another in order to make a purchase. He logs onto his German online banking site, reads the menus in English, makes the transfer — no problem. For whatever reason, the transfer does not happen in time. The purchase falls through, and the American executive is very upset.

He picks up the phone and calls the customer service line. The call gets routed to the German bank’s multilingual call center in Hamburg. The call center detects the call as coming from America and provides an English call menu. Entering the customer number confirms the account as an American account, and the call is assigned to an English-speaking German customer service agent. The American executive proceeds to fume over the error. The agent checks the ticket, verifies that all procedures have been followed and reassures the American that everything has been handled correctly. This makes the American furious, which in turn makes the German agent take an even firmer position. The issue goes unresolved, and the American changes banks.

As simplistic as this example is, many things are going on. The German bank’s language menus created no problems, did they? Step one, step two, step three. It is often true that translations that require little contextual meaning work quite well. The reason the transaction did not go through in time could have been translation related. For example, there could have been a holiday in the time period that was not communicated clearly to the customer.

For whatever reason, the problems start when the German and the American interact on the phone. To make a long story short, each evokes the wrong meaning in the other when they speak. The American would have gotten a better response with a less emotionally charged approach, and the German would have appeased the American with a more empathetic and less procedural approach. That is as far as we will go with this example, though there is much more to say. Suffice it to say that it is important to be able to evaluate how effective these international customer situations are, whether in person or in writing.

The Kirkpatrick Four Levels of Evaluation

Regarding our evaluation methodology, let’s start from the beginning. In 1954, the now legendary Donald Kirkpatrick finished his doctorate degree at the University of Wisconsin at Madison. The thesis contained four very simple, yet brilliant, steps that to this day serve as the model for how we evaluate the effectiveness of training programs — the Kirkpatrick Four Levels of Evaluation: Reaction, Learning, Behavior and Results.

A training program is introduced for a reason. We want results. So, we always begin with the results in mind. Results is the fourth level, the ultimate evidence of whether the training intervention worked. In our example, we want customers to have a positive experience when they interact with our overseas call center. To achieve these results, a training intervention is designed — let’s say it is a two-week training program that includes accent training, cultural awareness training, and some actual customer interaction scenarios. The program is designed, and the money and the commitment to do it are all in place. Now, let’s go through the logic of the Four Levels and evaluate how well things are going.

Level One: Reaction. We have all filled out the evaluation sheets at the end of a training program. These are Level One evaluations. It may seem silly, but how the learner reacts is actually quite important. It is hard to learn if the room is cold and the facilitator is unprepared. In our example, we are going to want participants to enjoy learning about the customer’s culture, exercises should be relevant and clear, and the accent training should be interactive and exercises should allow participants to record themselves so they can hear their own improvement. A Level One form usually asks participants to rate materials, exercises, facilitators, and facilities. Most organizations are good at Level One.

Level Two: Learning. Level Two is all about whether participants have gained any change in knowledge, skills or attitude. In our call center training program example, we will be able to detect a little bit of each. Participants will gain knowledge about customer culture, skills in dealing with those cultural traits, and cross-cultural training always involves confronting one’s own attitudes about peoples who are different than you are. The specific knowledge, skills and attitudes that we want to affect with the training intervention are listed in the learning objectives for the program. A pre-test and a post-test are often given to assess whether the program has its intended effect. If you are hiring vendors to do this kind of training, it is a good idea to insist on this kind of testing, and it is also a good idea to be very involved with both the articulation of the learning objectives, Level One analysis and the Level Two learning that the program has had.

Level Three: Behavior. At this point, the training has taken place, and the employee is back on the job taking calls and solving problems. Level Three deals with the extent to which the learner has changed his or her behavior on the job. This is an area where call centers actually do a much better job than many other job settings. The reason is that the customer service agent sits in a very controlled environment, where every minute is accounted for using various forms of technology. One piece of technology allows supervisors to listen in on how the agent handles calls. Using some kind of score card to evaluate the extent to which agents are living up to the best practices presented in the training is a great example of a Level Three training evaluation.

If you think things have gotten too theoretical at this point, let’s look at a specific example of how this kind of training can be evaluated at a Kirkpatrick Level Three. This is a generalization, but some cultures perceive Americans as being pushy at times. In business conversation, Americans will sometimes jump right to the matter at hand without first establishing some rapport through small talk. By creating awareness (knowledge) about this tendency through a training program and by providing some culture-specific conversational strategies (skills), the American agent in this real example was able to reduce the amount of time spent on the phone with a customer. Reducing the amount of time spent on the phone is very important in call center settings because this allows each agent to take more calls, making him or her and the call center as a whole more productive.

The American agent in question was having issues dealing with internal company customers from the United Kingdom. The issues they had to deal with were very high stake. This is another generalization, but many people in the United Kingdom really resent being pushed and prodded. The training program coached the US agent to slow down in the beginning of the conversation and chat about the weather for a minute (another gross generalization, but there is some truth to this one as well). Talking about the weather for a minute or two immediately created empathy and trust, allowing the issues to be resolved more quickly. You see, this colleague in the United Kingdom so resented being pushed that she would actually slow the process down. The magic happened when the US agent transferred learning to behavior. Only when this happens is there an opportunity to see Results.

Level Four: Results. The training intervention was introduced for a reason — to create a better customer experience. In our example, the best way to verify this is to conduct a survey of customers or perhaps a focus group. If you get positive results, the training intervention was very likely worth it. At this point, it is not rocket science to do some cost analysis to compare the cost of the program and estimate the benefits of retaining customers. There are many more factors to include in this analysis depending on the industry you are in and what kind of outsourcing solution we are dealing with, but a straightforward cost-benefit analysis is great for determining if a program was worthwhile. Other metrics in the call center such as reduced talk time and reduced call-backs can also be used to see whether the training has been helpful.

There are so many evaluation programs available today. All evaluation programs try to accomplish the same thing — to make sure that things get done as efficiently as possible in terms of low cost and the least amounts of defects. The most popular, especially in manufacturing settings, is Motorola’s Six Sigma approach. A few years ago it was total quality management. There are other models out there, including return on investment introduced by Jack Philips and value on investment introduced by a consulting company called Booz Allen Hamilton. These are all good if they work in your setting. Frankly, the Kirkpatrick model is the least complex and most intuitive of them all. It has stood the test of time, and it is nearly impossible to poke holes in it or claim that it is somehow invalid. “Reaction, Learning, Behavior and Results” is some really good stuff. In fact, it even works as a model for validating general translation and localization projects. Let’s take a look at how this might work.

The Four Levels in a translation project

Let’s say that you are a localization vendor and that you have just landed a job to localize an English-language intranet portal to five other languages for a company that is experiencing rapid growth. We are going to apply the Kirkpatrick Four Levels to establish whether you are providing the client with the kind of results that he or she wants from translating the content for local markets.

First, we start with the end in mind. You are a company, you have services and you want to meet client needs. What is it that the client wants to accomplish? Think of these as your learning objectives, using the model above. Your objective is to allow employees in five new markets to take advantage of company information as accessible as if they were native English-speaking American employees. They want this for a reason, however. They want employees in these five markets to have a better understanding of products and services, to have a better sense of company history to build citizenship. They probably want better integration of processes and procedures. Later, under Level Four, we will examine what kind of data collection points we might want to keep track of whether results have been attained.

Next, you do the translations. This is a complex process that involves many people. Every company will do this a little differently. Sticking with our Kirkpatrick model, this is the equivalent of developing the program. In fact, you could view the whole translation initiative as a learning intervention. Your client wants its employees to be able to learn about all the products, services, company policies, processes, procedures and everything else by being able to read about it in their native languages. This aim is really not that far removed from a learning initiative such as the one we looked at above. So, here it goes. Let’s analyze how things went using the Kirkpatrick Four Levels.

Level One: Reaction. This level deals with the pre-launch quality validation that most translation companies would go through in a project like this. A company would allow a few test users to read through the content, browse through the menus and in general provide feedback regarding their reaction to the intranet site. If there are any negative reactions, corrective measures would be taken. This might involve awkward sentences, bad color choices, inappropriate pictures or anything else that would not work. Things that don’t work are things that would inhibit users from arriving at Level Two, Learning.

Level Two: Learning. Here, we are concerned with whether employees at the client company are able to gain the knowledge, skills or attitude changes that we intended for them to have. This level is easier to apply in a more controlled learning environment. One could, however, hold a focus group to see whether employees have changed their attitudes about the company, whether they have a better understanding of company processes and procedures, if they have more knowledge about the history of the company, whether they now know more about the company’s products and services. There are a few other ways that we can gain the same Level Two information such as online surveys and even anecdotal evidence.

Level Three: Behavior. Again, Level Three is the key to the kingdom. What changes in behavior did the client want in requesting the translation work? Probably better adherence to policies, increased use of automated processes to reduce cost. There might even be better customer service as employees improve their knowledge and attitudes about the company. You could probably think of many more examples of changes in behavior that can come about in an organization from better access to information.

Level Four: Results. Last, you want to be able to demonstrate to the company that there were results. Here is a list of possible results. The more specifically they can be tied to this specific initiative, the better. But as Dr. Kirkpatrick senior and junior would agree, it is not always possible to have proof. Evidence in the following areas is often good enough when establishing results:

• Increased rate of online automated tools such as human resources benefits, travel arrangements and financial processes
• Improvement in specific areas of employee satisfaction surveys
• Increased use of the intranet in itself can be a desired result
• Improved response times and other performance-related indicators that can be tied to the translation effort

Ending this article where we began, it is not really possible to completely localize direct interpersonal interaction the way we can written language translation. But with the Kirkpatrick Four Levels, we can begin to know how well we are doing in each. M

Erik Granered is the author of Global Call Centers: Achieving Outstanding Customer Service Across Cultures and Time Zones (Nicholas Brealey Publishing, 2004), which introduces culture as a competitive advantage in international customer service settings. Questions or comments? E-mail editor@multilingual.com