Congratulations! If you’re trying to find the best way to test your web and mobile products for localization, it means your application was popular enough to warrant expansion into multiple markets. The need for localization testing is a mark of true success. Unfortunately, it’s also among the most difficult quality-related challenges to solve since it requires highly specialized skills from software-savvy native speakers who live in-market.
Most web, desktop and mobile apps are used under unpredictable, real-world conditions — well outside the sterile confines of any testing lab. And because every country’s language, culture, currency, taxes and standards are different, in the realm of localized apps it’s imperative for companies to prepare their applications to survive in ever-changing foreign markets. With a company’s users being distributed all around the world, it’s crucial that a portion of a company’s app testing and validation is distributed around the globe, too.
This calls for a better way to test, one that meets the “in-the-wild” demands of localization. Enter crowdsourcing. By using a community of diverse and talented professionals, crowdsourced testing provides companies with native speakers worldwide to run complete localization, verifying accuracy and context across currency conversions, dates, special characters and static content.
Localization testing challenges
Let’s say, hypothetically, that you’ve just launched a Mandarin version of your mobile app. Like the rest of your applications, this one was developed and tested using in-house resources. But for this assignment, you need a native Mandarin speaker to confirm that nothing was lost in translation. Luckily, Peter in the marketing department speaks, reads and writes Mandarin — problem solved, right?
Not so fast! While translation validation is indeed part of the localization process, it’s not the only part. First, is Peter fluent enough in Mandarin to fully understand metaphors, similes and slang? Will Peter be pulled from his day-to-day job to validate content revisions, and can he vet every part of your localized application, including your mainstream app content, your error messages and the system-generated e-mails your app sends? If Peter is really going to oversee ever-changing content such as comments, message boards and ratings, it sounds like Peter’s got himself a new full-time job. And what about when the app is launched in German, French or Portuguese (in their various country-specific iterations)? Are you really going to become Rosetta Stone’s best customer in order to keep up?
Assuming you (or your CFO) have already rejected the idea of solving this solely through in-house linguists, what other options do you have? Free translation tools, while innovative, are too often overtly literal and ill-equipped to handle localization on such a huge scale. Other options include outside consulting services specializing in localizing your content, but these often lack an effective third-party mechanism to validate their own work. Putting aside matters of expertise, cost and personnel, there’s still that small matter of localization testing criteria, which includes:
Content. Both static and dynamic content, such as catalogs, search results and metadata.
Dates. Is the date January 1 or 1 January?
Characters. Différent länguages have ðifferent set∫ 亥f characters.
Character display. Arabic and Hebrew languages display right to left, which wreaks havoc with the way some browsers display your content.
Postal codes. In some countries, postal codes contain letters.
Phone numbers. There are different formats for different markets.
Direction. Some languages are written left to right; others are right to left.
Currency conversion. This is especially important for internet retailers.
Tax calculation. VAT, sales tax and other fees vary from country to country.
So now that you have a better idea of what you’re up against, how can you expect to build an in-house staff of software-savvy linguists? Answer: Most companies can’t, at least not without some serious investments in both money and time. But what you can do to overcome the challenge of localization is to leverage a crowdsourced community — one that includes native speakers who live in-market all around the globe.
Traditionally, decision makers had two basic options for completing assigned tasks: through an in-house staff or with an outsourcing firm. With the introduction of crowdsourcing, there’s now a viable third option that’s much less costly and time consuming. Here’s the formal definition of the term, as coined by journalist Jeff Howe in his book, Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business: “[Crowdsourcing is] . . . the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” What’s the significance? Well, for one, it enables you to make decisions on a per-project basis, which comes in handy with localization projects since they are often too infrequent to warrant building out an entire staff of linguists. The opportunity for executives today is to get the most bang for their buck by building in-house strengths around key employees and core competencies, while also tapping into variable resources like crowdsourcing.
Crowdsourcing tips
First, start with why. In many ways, testing is about finding and potentially fixing what doesn’t work. Testers seek out problems because nobody wants to use or sell buggy software. But with localization testing, there’s an added reason for testers to care: You’re protecting the very things that are valuable to an entire culture of people. Cultural identity is an extraordinarily important component of who we are, and bugs that impact cultural identity can often be the most frustrating. For example, take the Cherokee tribe of Native Americans. Like many Native American tribes, their community was displaced by the advancement and settling of Europeans. However, one of their members, a man named Sequoyah, had the foresight to invent a written system for the Cherokee language. With that system, the Cherokee were able to keep detailed records of their history and communicate with other members of their tribe who were now spread across the country.
Today, Cherokee is spoken by thousands of people in the United States and still uses Sequoyah’s complicated writing system. It’s possible for Cherokees to communicate in their language using any modern communication device, including iPads and iPhones. This is because Unicode includes support for the Cherokee syllabary (Figure 1), and any modern device with good Unicode support can handle the Cherokee language.
Second, build a team that matches your product, your task and your users. The best online communities of professionals serve as a perfect complement to in-house teams. Build a team that blends the strengths of your full-time employees (brand, reputation) with those of the community (flexibility, cost-effectiveness, testing coverage across locations, languages). Having this blend of staffing allows you to scale your resources up and down in a fluid manner, meeting tight deadlines during peak periods of development and testing, while controlling costs during slow periods. The end result of this scalable community-driven workforce is faster time-to-market.
This is more an observation than a tip, but independent thought is the lifeblood of creativity. With a diverse community of professionals that transcends location and background, you can avoid the group-think that often stifles internal teams. When a community is following a lead, no matter how large the crowd, it will simply reflect the one leadership voice. The consensus view from inside your company can drown out objections and alternate points of view. This is often an unintended byproduct of strong leadership. Again, a global community brings diverse opinions and experience, as well as fresh eyes, which can result in more complete testing coverage.
Also, to get the most out of crowdsourcing, find communities that enable community members to build their online reputation. This increases accountability and performance by rewarding good actors and punishing bad behavior. Performance ratings and recognition levels enable your firm to choose the right professionals, and promote the desired behavior with members of the community.
Use crowd diversity to mirror real-world conditions. Forget the fact that you are paying these people to write test cases or find bugs. While that’s the primary job to be done, a secondary but powerful benefit is that your product is being validated in the hands and minds of real-world users in real-world conditions. Select the members of the community to best match your customer base, by location, language and expertise. The community working on your app should match the users you hope to target.
Remember that not all crowds are created equal. When selecting a community to work on your products, choose wisely. Some communities tend to be noisy, with lots of opinionating, but not much actual productive work. Choose a community that has proven success and showcases the past performance and reputation of community members. Also look for those communities that can point to clear, demonstrable wins — other customers they’ve worked with to solve real business problems. An important measurement for community sites is whether they include tools that help you manage the ongoing process. It is easy to build a marketplace website that connects an employer with a freelance worker, but this alone does not provide ongoing community-building.
However, you still need to call the shots. Crowdsourcing does not change a fundamental truth of software testing: Effective, detailed communication and project management are both key to any successful project. This is true in managing in-house resources or outsourced partners, and crowdsourcing is no exception. Assign an internal project owner to keep the information flowing and to manage the process.
Though crowdsourcing may be thought of as a solution for startups and small businesses, we find that many large organizations are also turning to the crowd for localization testing. As Google’s senior engineering director Patrick Copeland says, “our global customers have different demands of our products. We want products to ‘feel local,’ and we need to support features that may be unique to specific markets. For instance, in Indic-based languages, using a standard keyboard is difficult, so we develop strategies like virtual keyboards or category browsing for search. As we specialize our products for certain markets, it introduces more challenges for testing. When we can’t find internal talent, community-based testing is an interesting solution to this challenge.”
Mozilla’s quality assurance (QA) director, Matt Evans, also has some thoughts on the subject: “I think the biggest misconception is that it is all lumped under one label and that crowdsourcing is done according to some crowdsourcing manual or standard. It isn’t correct to have crowdsourcing and community-driven projects under the same umbrella. Strict crowdsourced projects tend to be targeted toward tasks that are discrete and distributed in nature. Community-based approaches tend to be longer lived and are typically for public benefit. Membership is considered a privilege, and betterment of the community as a whole is usually at the top of the list of shared goals among the members of the community project. In addition, you find the rise of leaders within community projects that drive the project forward. Community citizenship and passion for the project have been the key success factors for Mozilla’s success, in my opinion.”
When ReviverSoft localized its website and product into 22 languages, the company chose a major localization services company to do the translation work. After the website and product had been localized, Mark Beare, the company’s founding partner, needed a way to test the accuracy of these changes. He decided on an independent third party to validate the accuracy and completeness of its localization efforts.
Beare adopted crowdsourcing to find native speakers who live in-market to ensure the company’s apps were ready for localized success. “We really needed native speakers for this assignment. These are the countries where we’re selling our products, so it was extremely important that the material was accurate and made sense.”
With the help of his dedicated project manager, Beare and his team assembled diverse testers in terms of language and location. With the setup complete, this team of experts spent the next several days reviewing the various localized sites, with a particular focus on the German, French, Danish and Japanese versions. “We had been sending a lot of long files to the translators, so you never know when text is going to be cut off in certain areas of the site, since length often varies,” says Beare. “We needed to make sure that the text rendered correctly, and we needed to know where text should be changed to make it more relevant.”
Within a matter of days, Beare and his team had received complete feedback on all of their recently translated versions. “What we found was that, for the most part, testers were able to use the software with the translations provided, so that was good confirmation. We did, however, find a few issues where the text was not rendered correctly and where certain content needed to be changed completely.”
As ReviverSoft expands into more locations around the globe and as the young company grows in number of employees, Beare expects to continue crowdsourcing its localization testing.
Numerous benefits can be achieved when adopting crowdsourcing for localization testing, including cost containment. Crowdsourcing allows managers to use lower-cost outside support without being tied down by the long-term commitments of outsourcing firms. On a monetary sidenote, companies that venture into localization often have trouble retaining users outside of their home market. Users who feel that they have a hand in the success of a product will have a stronger emotional commitment to the product and the company, creating powerful connections and a potential army of fans.
Today’s customers, no matter where they are, tend to expect apps to be complete and functional from day one. The era of beta-testing in foreign markets as a substitute for QA is long gone. Since localization testing tends to occur during the latter phases of the product’s life cycle, a time when deadlines are fast approaching, crowdsourcing can alleviate delays that often stifle companies around peak release times.
