This reader does not purport to address everything around standards, and was last updated in late 2018. To be informative and of practical use, there must be a focus. This focus is being provided via two limitations. First, we are looking at standards that affect multilingual transformations of content, and at general content life cycle standards only, ad hoc and as far as they have bearing on multilingual transformations. Second, we are looking only at technical standards that are targeting actual technical interoperability — file or data formats and/or communication protocols. Abstract metadata, quality or service level standards are discussed only marginally, and only as long as they have bearing on real machine-to-machine interoperability in localization. Linguistic standardization of particular natural languages (including English) is only mentioned as far as it touches on machine-to-machine interoperability in localization.

BCP 14 (previously also known as RFC 2119, and now includes RFC 8174)

Short Description: BCP stands for Best Current Practice. BCP 14 defines the standardization specific meaning of normative keywords such as MUST, MUST NOT, OPTIONAL, REQUIRED, RECOMMENDED and so on. RFC stands for request for comments, and these are numbered sequentially. RFC 2119 is the most common normative reference in other specifications throughout information technology standardization bodies. RFC 8174 was added to address the ambiguity of uppercase versus lowercase in RFC 2119 keywords. Localization-related standards such as ITS and XLIFF use BCP 14 keywords to make their normative statements that create the basis of conformance statements, testing and verification.

Owner: Internet Engineering Task Force (IETF), a nonmembership standardization body. Contributors are individuals who implicitly commit themselves by contributing without signing any formal contract. IETF creates internet-related technical standards, protocols, processes and nonnormative informational content. IETF is backed by the Internet Society.

Intellectual Property Rights (IPR) Mode: Reasonable and Non-Discriminatory (RAND), an IPR mode that allows owners charging for use of essential patents, provided that the charge is “reasonable” and “non-discriminatory.” This is a bit vague, but 2013 saw some groundbreaking legal development with regard to standards offered under RAND IPR policies. In a Washington District Court, Judge James L. Robart ruled in a RAND case between Microsoft and Motorola that an initial licensing offer from Motorola had not been made in good faith and therefore constituted a breach of contract, in particular the obligation to license essential patents under RAND conditions to all implementers. The methodology of setting the rates in this case was so well explained and logically constructed that it is being used as precedent not only by other US courts but also outside of the United States and by policy makers.

Current version and work in progress: For a very long time (20 years), BCP 14 had only RFC 2119 as its content, so many standard makers treated BCP 14 and RFC 2119 as synonyms. With the addition of RFC 8174 (RFC 2119 Clarification) it became important to distinguish if a standardization work product references BCP 14 as a whole, or just RFC 2119. RFC 8174 makes it clear beyond any reasonable doubt that the IETF notion of normative keywords is typograph-driven. For example, “MUST” or any other of the defined keywords has its normative meaning only as far as it is printed in uppercase, as opposed to the ISO notion of normative keywords that is concept-driven and in fact forbids typographical distinction of normative keywords.

RFC 2119 was released for unlimited distribution in March 1997. RFC 8174 updates RFC 2119 and was released for unlimited distribution in May 2017.

However, it is undesirable to make changes to this BCP because so many normative texts across IETF, W3C, OASIS and so on depend on the meaning of the normative keywords as set out here. It is questionable whether adding RFC 8174 was a good thing, as it deepened the rift between the IETF and ISO notions of normative keywords. It is too early to tell what the impact of the clarification was. Before, if uppercasing was omitted on a keyword in a spec, you could fix it in later publishing stages as it was considered editorial. In theory, it sounds clear to tell that only uppercased keywords are keywords, but in practice, it will make corrections difficult. Changing from plain to uppercase has been made into a material change by this supposed clarification. This clarification has some potential to cause interpretation damage. Until uppercasing was made mandatory, editors were dutifully aiming to avoid the keywords and use non-normative semantic variants. Now, many will stop doing that, citing RFC 8174, causing a lot of confusion in ordinary readers. It may well happen that this clarification will cause more changes to this BCP in the feature.

BCP 47: Tags for Identifying Languages

Short Description: BCP 47 is a normative IETF track that compiles recommendations on how to create a unique language tag from codes defined in several other normative sources, including ISO codes. It is frequently referenced by OASIS, Unicode and W3C standards.

Owner: IETF.

IPR Mode: RAND.

Current version and work in progress: RFC 5646 was released for unlimited distribution in September 2009. RFC 4647, Matching of Language Tags, was released for unlimited distribution in September 2006. BCP 47 is a persistent name that always points to the latest release, no matter what the current RFC number. BCP 47 itself is stable, which is important for backward and forward compatibility. New tags are being continuously registered via registration authorities specified in the standard. Most current developments are connected to Unicode extensions for BCP 47.

BCP 47 Extension T: Transformed Content

Short Description: Extension T is possible via the extensibility mechanism defined in BCP 47 (RFC 5646) itself. Extension T has normative status within the Unicode Consortium, as it is being maintained as part of CLDR (see page 62), which is its major normative deliverable. This extension allows for additional tags specifying from which other language, locale or script the content at hand had been transformed. Extension T is not recommended for usage in structured environments such as XML, where this type of metadata can be specified using markup solutions rather than a single text field. Note that Extension T is appending the information about the originating language or locale with a leading “t,” which means that the BCP tag starts with the target locale and the source locale is appended. This makes sense given the structure of BCP 47 tags, but may be perceived as contrary to the customary listing order of source and target languages, so “EN-t-IT,” for example, actually means that the tagged content is English but was transformed from Italian, not the other way around.

Owner/Maintainer: IETF/Unicode Consortium.

IPR Mode: RAND.

Current version and work in progress: Informational RFC 6497, published in February 2012. Extension T is regularly maintained as part of the CLDR release cycle.

BCP 47 Extension U: Unicode Locale Extension for BCP 47

Short Description: Extension U is possible via the extensibility mechanism defined in BCP 47 (RFC 5646) itself. It has normative status within the Unicode Consortium as it is being maintained as part of CLDR.

Owner/Maintainer: IETF/Unicode Consortium.

IPR Mode: RAND.

Current version and work in progress: Informational RFC 6067 was published in December 2010, and is maintained by the Unicode Consortium as part of CLDR. Extension U is regularly maintained as part of the CLDR release cycle. The Unicode Localization Interoperability Technical Committee (ULI TC) — known since early 2018 as the CLDR Technical Committee Subcommittee (CLDR TC/ULI SC) — delivers the input exception data for sentence-breaking mechanisms for different locales for the periodic CLDR releases. As a result, sentence-breaking behaviors driven by different exception data can be specified through assigned keys under the extension’s U mechanism.

Notably, the canonicalization algorithm included in RFC 6067 is slightly out of date. The current provision is specified in UTS #35. The canonicalization algorithm should be therefore updated in RFC 6067.

CLDR

Short Description: Unicode Common Locale Data Repository, is a standard repository of internationalization building blocks, such as date, time and currency formats, sorting (collation) rules and so on. CLDR is not a standard in a classical sense. It is, as the name suggests, a repository that is being constantly updated and released on a rolling basis following its data release process.

Owner: Unicode Consortium.

IPR Mode: RAND.

Current version and work in progress: Version 34 was released on October 15, 2018. Version 34.1 will be released on or around June 2019.

The submission period for version 35 will start in April 2019. CLDR is being released on a regular semiannual schedule, whereas the cycle starting in the fourth quarter of each year is focused on tooling and bug fixing, and usually skips the public data submission phase.

COmmon Translation Interface (COTI)

Short Description: The Association of German Manufacturers of Authoring and Content Management, shortened to DERCOM in German, created this standard.

DERCOM claims that it had to step up to protect and promote the interests of its membership and, at the same time, to let translation providers use the translation management systems (TMS) of their choice. DERCOM argues that no such interface exists, and it is true that a common interface is better than the present jungle of proprietary APIs on both ends that can only provide interoperability after some custom development on both ends for each new interface. Arguably this can be addressed by an Enterprise Service Bus or messaging architecture, but not every content management system (CMS) owner has the resources or technical muscle to do that.

COTI level 3 provides a sophisticated state machine that provides synchronous support for translation orders, cancellations and even updates of existing orders, which is notoriously tricky. While DERCOM is right that no translation web service interface has been standardized, it is surprising that COTI only standardized the business metadata interface and doesn’t say anything about payload standardization, standardization over a canonical data model such as XLIFF lends itself, but COTI actually does not mandate any data model restrictions with regard to the payload. So actually this specification only solves the CMS part of the equation. The bundle is thrown over the wall and it’s up to the language service provider to figure out what is localizable, what is not and so on.

While German CMS providers might be happy to implement always-synchronous Simple Object Access Protocol (SOAP) web services, the general trend seems to be rather toward RESTful interfaces and microservices, and it might hinder the adoption of COTI that level 3 strictly enforces synchronous SOAP calls. REST refers to Representational State Transfer, a widely known web services architectural best practice that has never been officially standardized. Unlike SOAP, which is a specific standardized protocol, there is no single REST standard or even a single best practice. REST is rather an architectural style for interoperability.

Level 1 is actually the specification of the payload ZIP package, its structure and manifest. Sound familiar? People just keep reinventing this very wheel, albeit each time with subtle differences that don’t allow for automation among those “standard” ZIP packages. Electronic Dossier, Linport, TIPP and now COTI level 1 — all of these just defined a folder structure that they think a translation request and response should contain without talking to each other. Alan Melby and others invested some diplomatic effort into making the Electronic Dossier people talk to the Linport people and merge the Linport project with TIPP. But even before this could bear fruit, DERCOM COTI brought another “standard” folder structure.

COTI level 2 mandates automated exchange of the structured ZIP packages over hot (watched) folders. Whatever is placed in a hot folder is up for grabs at the receiving end. The TMS watches the CMS’s hot folder for new projects and the CMS watches the TMS’s hot folder for responses. Sound like the 1990s?

The interesting contribution to the topic of standardized translation web services is in the level 3 compliance that requires SOAP and doesn’t allow or specify any other admissible bindings, which would certainly be advisable.

Owner: DERCOM. DERCOM is not a standardization body but a German trade association. They nevertheless decided to provide German-producer-centric standardization, and, as they say, “non-proprietary” solution to CMS and TMS interoperability.

IPR Mode: Unclear. The specification, documentation (CHM) and validation artifacts (WSDL, XSD) are available for free, but no restrictions seem to have been specified with regard to licensing of IPR essential to implementing the specification. The specification says to look up “the intellectual property rights section of the technical committee web” at www.dercom.de but there seems to be no such section. This unfortunately means that not even (F)RAND restrictions apply for licensing offers made for essential IPR by their respective owners. This is at least until DERCOM actually comes out with publicly visible IPR information.

Current version and work in progress: COTI version 1.1.1 was put out July 07, 2017, as a minor update from version 1.1 put out April 29, 2016. The 1.0 version was first approved on May 25, 2014, and was subject to only minor editorial fixes by September 2015. Version 1.1 brought some additional material such as the XML Schema, but also changes in the WSDL artifacts.

There doesn’t seem to be a specific plan to further develop the specification. Rather it seems that DERCOM makes changes and fixes as its membership gradually progresses in fulfilling its obligation to implement COTI level 1 through COTI level 3 compliance. The WSDL changes in the version 1.1 were most probably driven by implementation experience of the DERCOM membership.

International Components for Unicode (ICU)

Short Description: Until May 2016, ICU was an IBM-driven open source project, and in fact the most important reference implementation of both Unicode and CLDR. As ICU provides internationalization libraries, it actually consists of two subprojects, ICU4C and ICU4J, that provide libraries for C and C++ and Java respectively. Spectacularly, ICU is the common denominator of both Android and iPhone operating systems. ICU 58 is the first ICU version released under the ICU TC governance.

Owner: Unicode Consortium.

IPR Mode: RAND.

Current version and work in progress: ICU 63 for Unicode 11.0 and CLDR 34 is the current version. ICU 63.1 was released on October 10, 2018. This URL http://icu-project.org/apiref/icu4c always points to the current official version.

ICU (both the Java and C projects) is being developed continually to address bug fixes and dependencies. Major releases are driven by the releases of the Unicode Standard and changes in CLDR data. A new major release should come after stabilization of CLDR 35 data in May 2019. Interestingly, in summer 2018, all ICU infrastructure migrated from SVN to GitHub.

Internationalization Tag Set (ITS)

Short Description: ITS 2.0 comprises 19 metadata categories compared to a mere seven in ITS 1.0. Additionally, ITS 1.0 metadata categories were primarily designed for internationalization of XML content. Nevertheless, as abstract data categories, ITS can be implemented in non-XML environments. Importantly, ITS 2.0 normatively specifies usage of the old and new ITS data categories for XML and HTML 5 content, and new categories have been introduced that explicitly address the localization roundtrip, such as the localization quality assurance related data categories Localization Quality Rating and Localization Quality Issue. Importantly, ITS 2.0 is listed in the JTC 1 Big Data Standards Roadmap (ISO/IEC TR 20547-5) as a key automation enabler for big data and analytics dealing with human language.

Owners of ITS decorated content want their internationalization and localization related metadata to inform the roundtrip and make it to the target content in a meaningfully processed state that allows for drilling down into the process and for reconstructing the audit trail. Localization workflow managers should pay attention to information flows directed by the ITS data categories introduced by their customers up in the tool chain. There is also potential to introduce automated or semiautomated ITS decoration steps before extraction, or to introduce relevant XLIFF mappings of ITS data categories recorded on translations during the localization roundtrip that can be imported back into ITS in XML or HTML on merging back localized content. Categories such as Translate (which had become a native HTML 5 attribute), Elements Within Text, Locale Filter, Target Pointer or External Resource should drive extraction and merging back of localizable content. Terminology and Text Analysis disambiguation markup should be passed on to human and machine translators. Proper interpretation of directionality markup is a must for sound handling of bidirectional content using Arabic or Hebrew scripts. Self-reported machine translation (MT) confidence should be passed on to the content recipients, possibly along with quality assurance related metadata (Localization Quality Issue and Localization Quality Rating).

These considerations are especially valid when hooking up existing localization workflows upward into the tool chain. Existing workflows should introduce mappings of ITS data categories used in source content, so that the metadata flow is not broken throughout the content life cycle.

The current ITS 2.0 categories are Translate (flag indicating translatability or not); Localization Note (for alerts, hints, instructions); Terminology (to identify terms and nonterms and optionally provide definitions); Directionality (manages left to right/right to left display behaviors of content portions); Language Information (BCP47 language tags on relevant content portions); Elements Within Text (shows which elements break flow or not to help encode segmentation); Domain (to identify content theme or topic for better choice of relevant services or creation of training corpora); Text Analysis (for inclusion of automatically provided disambiguation data and term candidates); Locale Filter (indicates which portions of text are relevant for which locales); Provenance (tracks agents — machine or human — involved in content transformations); External Resource (global pointers to external localizable resources such as images or other binary data); Target Pointer (identifies transformation targets in multilingual documents); ID Value (rule to provide unique content identifiers to be maintained throughout transformations); Preserve Space (specifies whitespace handling); Localization Quality Issue (provides a way to mark up and classify specific language quality assurance issues); Localization Quality Rating (quality rating expressed in a single 0-100 score); MT Confidence (provides self-reported MT quality score as a single 0-1 number); Allowed Characters (specifies restrictions on character data using simple regular expressions); and Storage Size (a simple encoding-based restriction mechanism to make sure that translations fit restricted database fields, forms and so on).

Owner/Maintainer: MultilingualWeb-LT Working Group/ITS Interest Group (IG). The World Wide Web Consortium (W3C) ITS IG has been the informal maintainer of ITS 2.0 after the MultilingualWeb-LT Working Group mandate expired; however, interest groups in W3C cannot create normative deliverables. Therefore a new Working Group will have to be formed to commence work on the successor standard, as soon as the ITS IG identifies industry need for a successor version.

However, the status of the ITS IG is unclear, as its charter expired on December 31, 2018, and it is not clear at the moment if the charter can be extended. In case ITS IG lapses, stakeholders of ITS should initiate a W3C community group to look after the adoption and maintenance of ITS 2.0. The main purpose of such a group is to see if there’s a need and momentum to create a new feature or maintenance release of the Internationalization Tag Set.

IPR Mode: Royalty Free (RF), an IPR mode that mandates and guarantees royalty free use of essential patents in order to implement a standard.

Current version and work in progress: The current version is 2.0, published as a full W3C Recommendation on October 29, 2013. Work on another major version is not imminent. That said, ITS 2.0 covered a number of new areas with nonnormative mappings. These included Resource Description Framework (RDF) and XLIFF mappings. See XLIFF for inclusion of the ITS functionality as an XLIFF 2.1 module. Current collaborations and liaisons include OASIS DITA, DockBook, XLIFF and XLIFF OMOS TCs, as well as GALA TAPICC.

ITS IG is working on further elaborating and maintaining other informative mappings, such as NIF and MQM. However, the MQM mapping aspect transitioned to the MQM W3C Community Group. ITS IG also maintains ITS extensions that support categories that could not be normatively specified for various reasons within the ITS 2.0 publishing schedule; for instance, the Readiness data category.

JSON Localization Interchange Fragment Format (JLIFF)

Short Description: JLIFF aims to be compliant with the abstract object model defined in UML diagrams and a prose specification as XLIFF OM. The TC currently works on XLIFF unit JSON representation examples, JSON schema and JSON templates to represent the XLIFF OM UML diagrams and XLIFF unit XML examples.

Owner: OASIS XLIFF Object Model and Other Serializations (OMOS) TC. JLIFF is currently available as a JSON Schema that is reasonably stable but not yet officially published by OASIS.

JLIFF’s main intended use case is the real time interoperability of TMS and CAT tools that support the XLIFF 2 data model. Importantly and unlike XLIFF, JLIFF can support exchange of fragments at unit, group and file levels. For full XLIFF interoperability, it also supports exchange of whole XLIFF Document JLIFF equivalents, but this is not the main use case. It is important to stress that in order to preserve data integrity the lowest interchange level is that of logically separate bitext units and not arbitrary segments.

IPR Mode: Non-Assertion (RF).

Current work in progress: In summer 2018, JLIFF JSON 2.0 and 2.1 schemas were completed by the committee on the XLIFF OMOS TC GitHub repository. The TC started working on a prose specification to be reviewed hopefully in early 2019. JLIFF schema has been attracting implementers even before becoming stable. Notably, Vistatec released an open source implementation of core JLIFF in March 2018 at https://github.com/vistatec/JliffGraphTools. But the TC is aware of at least two early implementations that cannot be publicly disclosed at the moment. It seems that the capability of JLIFF to create fragments compatible with the XLIFF 2 data model is exactly what implementers need for real time API-based interoperability.

Importantly, the JLIFF format will be reused by the GALA TAPICC Track 2, when specifying their real time translation API (JRARTEBU – JLIFF REST API for Real Time Exchange of Bitext Units).

The XLIFF OMOS TC was chartered in December 2015 and started to develop a JSON serialization in parallel to the creation of the abstract object model for the XLIFF 2 family of standards that it is developing as its first priority. The objective is to create such a JSON serialization of XLIFF and XLIFF fragment data categories that would allow lossless interchange between JLIFF and XLIFF based tool stacks. The development of the JLIFF specification and artifacts is being conducted on this GitHub repository: https://github.com/oasis-tcs/xliff-omos-jliff. The repository is public but the contributors need to be or become OASIS XLIFF OMOS TC members. However, anyone can raise issues or comments via the associated issues or wiki at https://github.com/oasis-tcs/xliff-omos-jliff/wiki, as well as minor bug fixes via pull requests. It’s up to the repository maintainers if those pull requests will be merged or not, which is the usual GitHub collaboration workflow.

Multidimensional Quality Metrics (MQM)

Short Description: MQM is primarily a set of error data categories that can be used in language quality assessment, especially in bilingual translation scenarios, but also in monolingual review or authoring scenarios. The aim of MQM is to cover the whole logical space of possible errors and nonconformities related to language and locale-specific presentation of content of any kind.

It is impossible (and even if possible not likely very useful) to use all the error data categories from all levels and sub-levels. Profiling is a key method defined by MQM, so that in any particular project or job a particular profile or subset of relevant error data categories has to be defined based on stakeholder requirements and expectations. Thus MQM concentrates on quality in the sense of fitness for purpose. It specifically rejects the notion of some general or abstract quality without defining the purpose, related requirements and a resulting relevant set of possible error types. Also notable is that MQM doesn’t define any severity levels. There is an option for implementers to predefine severity levels and scoring methods based on any particular MQM subset.

Currently MQM is a proper superset of TQF. MQM and TQF had been developed separately by the QT21 EU project (coordinated by DFKI) and TAUS respectively. The EC funders effectively forced these two projects to reconcile their data models.

For effective exchange of MQM/TQF metadata within projects and jobs, inline capturing and encoding of the recorded errors is critical. This needs to be implemented as the ITS localization quality issue data category in native formats or via the same data category implemented as an XLIFF Version 2.1 module in bitext.

Owners: ASTM International (ASTM), founded in 1898, formerly known as American Society for Testing and Materials, as well as the Worldwide Web Consortium (W3C).

IPR Mode: RAND at ASTM. No IPR mode under W3C, as it is not being developed under the recommendation track, just republishing the ASTM maintained structure of data categories.

Current version and work in progress: Version 1.0 was released December 30, 2015. This URL leads to the latest published version: www.qt21.eu/mqm-definition.

The W3C MQM Community Group is being reactivated to publish updates from ASTM F43 www.astm.org/DATABASE.CART/WORKITEMS/WK46396.htm.

Open Lexicon Interchange Format (OLIF)

Short Description: OLIF is a stable and relatively widely used lexicon interchange format. It has a rich metadata structure and allows for the exchange of complex lexicon entries for various purposes, such as terminology management and MT. OLIF had been designed for use in both monolingual and multilingual context via cross-linking of “mono” elements.

Owner: OLIF Consortium, an ad hoc industry consortium driven by SAP and set up in 2000.

IPR Mode: Unclear. The specifications and schemas are available for free, but no IPR mode seems to have been specified.

Current version and work in progress: Version 2.1 is current. Version 3 has been in beta since 2008; no current work seems to be under way.

Segmentation Rules eXchange (SRX)

Short Description: SRX is an XML vocabulary that facilitates the exchange of segmentation rules between TMX compliant systems. SRX’s relationship to Unicode is not a transparent one, and SRX can be considered incomplete from the engineering point of view. However, its proclaimed goal was not to provide a set of segmentation rules for a number of languages, but rather to provide a mechanism to exchange the rules to improve TMX interoperability. TMX often fails to guarantee its targeted lossless transfer of TM data due to segmentation differences, chief among other issues. The current SRX incarnation works on a closed world assumption, meaning it recreates (and adapts) UAX #29 rules. UAX #29 is referenced and its study encouraged, but the relationship is currently not a maintainable linkage.

According to former Unicode Localization Interoperability Technical Committee (ULI TC) chair Helena Chapman, SRX developers should not take UAX #29 (and its ICU implementation) for granted, and use SRX only for the exchange of rules that differ from the standard UAX #29 behavior. ULI TC has been collecting natural language exceptions to UAX #29 for several major languages and included these in the CLDR release cycle. CLDR, however, uses LDML as its description language, not SRX.

Owner: ETSI ISG LIS. ISO/TC 37/SC 4 was intended as a co-owner of SRX. However, no memorandum of understanding between the groups was signed. This work item expired and was deleted on July 22, 2013, according to the ISO standards publishing policy.

Unicode ULI TC agreed to host the current SRX version after ETSI ISG LIS disbanded as a publicly available specification. However, it is unclear if ULI TC would be legally allowed to produce a new version without formally negotiating IP transfer from ETSI.

IPR Mode: RF for the LISA version; FRAND in ETSI (but wasn’t republished).

Current version and work in progress: A copublication attempt was made with ISO TC37/SC 4 as ISO CD 24621, which successfully passed the committee draft ballot, but no work progressed and the project was deleted on July 22, 2013.

There is no current work in progress due to ETSI ISG LIS dissolution. Version 2.0 was released by LISA OSCAR on April 7, 2008. Based on the published executive summary from November 2011, ETSI ISG LIS scheduled an SRX meeting for March 2012. This meeting happened behind closed doors. There are or should be dependencies with ULI TC on UAX #29 segmentation behavior modifications. SRX has potential as an XML based exchange vehicle for segmentation rules, because there is no ultimate finite solution to the segmentation issue in natural languages. In 2018, ULI TC started a note on segmentation and wordcounting that would build on the principles of UAX #29.

Simple Object Access Protocol (SOAP)

Short Description: SOAP is an XML-based web services protocol. Version 1.0 was submitted to IETF in autumn 1999 as an internet draft but never reached an RFC status, so it actually hadn’t become a standard at IETF. SOAP 1.1 had Note status at W3C and the only SOAP version that ever reached the standard status (W3C Recommendation) is SOAP 1.2.

The SOAP protocol consists of five layers: message format, transfer protocol bindings, message processing models, message exchange patterns and extensibility. Unlike REST, which is strictly HTTP-based, SOAP is protocol-neutral and can work over several lower level protocols such as HTTP (Hypertext Transfer Protocol), SMTP (Simple Mail Transfer Protocol), TCP (Transmission Control Protocol), UDP (User Datagram Protocol) or JMS (Java Message Service).

Owner/ Maintainer: W3C XML Protocol Working Group (closed) / no current maintainer. The SOAP 1.2 multipart recommendation was produced by the XML Protocol Working Group at W3C, and the group was closed on July 10, 2009.

IPR Mode: RF.

Current version and work in progress: 1.2 (Second Edition) was published as a W3C Recommendation on April 27, 2017. The protocol as such is not being developed or maintained at W3C or elsewhere. We list SOAP because it was resurrected in our industry by the COTI level 3 conformance requirement to implement a SOAP-based web service automation.

TBX, also known as ISO 30042

Short Description: TBX is a family of XML-based terminology markup languages that should allow for lossless exchange of terminology-related data and metadata. So far, two more lightweight versions known as TBX Basic (published in 2008 by LISA Terminology Special Interest Group) and TBX-Min (published in 2013 by LTAC Global) have been developed. TBX-Min should be targeting the use case of exchanging terminology with translators in the form of simple glossaries mappable onto UTX. However, TBX Basic is more suitable for mapping between TBX and XLIFF 2.0 with glossary modules.

TBX has been criticized for industry disconnect, for being too heavy on one hand and being too restrictive and not very suitable for MT training on the other. One of the reasons might be that TBX is supposed to be both a representation and exchange format for terminology, but it has been struggling to define a minimum set of terminology metadata suitable for practical interchange in a localization context. Some industry implementations can hardly be considered in the spirit of the standard, as they have not enforced inclusion of even very basic metadata such as part-of-speech or they are not structurally compliant with any of the predefined data structures.

Owner: ISO TC 37/SC3. Previously it was owned by ETSI ISG LIS, an industry specification group that was formed in the spring of 2011 within ETSI to take over the LISA OSCAR standards portfolio, including related LISA intellectual property. ISO TC 37 was a copublisher with LISA.

IPR Mode: Fair, Reasonable and Non-Discriminatory (FRAND) within ETSI; and RAND within ISO. BSD 3 clause licensed open source project at LTAC/TerminOrgs.

Current version and work in progress: 2.0, ISO 30042:2008 is the current version. TBX-Basic 3.1 (the version maintained by LTAC/TerminOrgs) was last updated September 12, 2014. This is the latest version of TBX-Basic compatible with ISO 30042:2008.

Although the revised (second edition) ISO 30042 hasn’t been published yet, it will hopefully become ISO 30042:2019, as it currently is ISO FDIS 30042. It is in a sense the current version, as no further technical work has been possible on this version since August 2018, and it should appear within a month of this issue’s publication.

Even though the underlying ISO standard has not yet been officially published, LTAC has already released industry oriented dialects compliant with ISO FDIS 30042, and thus hopefully ISO 30042:2019 in the very near future. These dialects are available through GitHub.io webpages. Notably TBX-Basic Dialect Version 1.0 is available at https://ltac-global.github.io/TBX-Basic_dialect. LTAC also published a robust developer guide to the public dialects compliant with the new modular TBX design at www.tbxinfo.net/tbx-downloads.

The most important data model change to happen in ISO 30042:2019 is making the inline data model compliant with the XLIFF 2 inline data model. This was implemented by the LTAC/Terminorgs TBX steering committee and TC 37/SC 3, with input from XLIFF and XLIFF OMOS TCs. The modular public dialects (Min, Basic and Linguist) compatible with the second edition of ISO 30042 were already released, also with the new inline data model.

The second ISO edition of TBX accumulated quite a number of breaking changes. But most of those changes are dealing with modernization of the XML tooling. Apart from the inline data model change and related introduction of explicit directionality support, it is important to stress that TBX joined other modern-day standards in leaving its former monolithic design for a new modular design.

This edition of TBX specifies a nonnegotiable core and prescribes that all compliant dialects must include that core. The standard then specifies a modularity mechanism. Outside of ISO, LTAC made sure that the most common dialects are extended from the core using a telescoping principle. The simplest possible dialect is TBX-Core; the TBX-Min is composed from the Core and Min modules. TBX-Basic is TBX-Min plus the Basic module, and TBX-Linguist is TBX-Basic plus the Linguist module. The stakeholders have high hopes that this will vastly improve the so-called “blind” or plug and play interoperability. The second ISO TBX edition also defines TBX agents and provides a compliance clause targeting the document compliance, as well as specific agents’ compliance. Notably, this edition introduces namespace-based modules, albeit only in the DCT (data category as tag name) style, which is a compromise that should not disturb those who are not worried with the DCA (data category as attribute name) style. And vice versa — implementers who are afraid of implementing namespaces can stick to the old DCA style. Helpfully, both DCA and DCT are extended from the same core, and modules specified in either DCA or DCT are semantically equivalent (based on the same sets of additional data categories). DCT is easier to validate and it’s also easier to filter out unsupported modules (as those are in different namespaces) in the DCT style.

ISO 30042:2008 was identical with TBX 2.0 developed at LISA OSCAR. Despite some effort to schedule joint work with ISO, ETSI ISG LIS didn’t manage to renew work on the TBX work item inherited from LISA OSCAR. Instead, LTAC Global and TerminOrgs took to publishing TBX dialects, also called industry specifications, as well as other terminology management related resources based on the current and planned ISO versions of the TBX (Default) standard. GALA and ttt.org still host the latest LISA OSCAR version including the legacy TBX-Basic.

Work in liaison with the OASIS XLIFF TC and later with OASIS XLIFF OMOS TC resulted in development and testing of a TBX-Basic to XLIFF 2 with glossary module mapping. The mapping upgraded to the modular second ISO edition based on TBX-Basic — semantically equivalent with the old TBX-Basic — is now being specified on the standards track within OASIS XLIFF OMOS TC. See the latest editor’s draft at https://tools.oasis-open.org/version-control/browse/wsvn/xliff-omos/trunk/XLIFF-TBX/xliff-tbx-v1.0.pdf.

Translation API Cases and Classes (TAPICC)

Short Description: The first TAPICC symposium, on October 26, 2016, brought a four-way categorization of use cases that were considered in scope of the Translation API (TAPI) standardization: 1) Exchange of a payload blackbox accompanied with rich business or project metadata. In this area, TAPICC wants to abstract common business metadata used in other initiatives past and current including Linport, TIPP, COTI, XLIFF 1.2 project group, OASIS Translation Services TC (closed) and so on. 2) Bidirectional real-time exchange of XLIFF unit data in arbitrary serializations. 3) Enriching of XLIFF units in arbitrary serializations with terminology, translation suggestions, text analysis, process, quality assurance metadata and so on. 4) Exchange of data for layout representation purposes during the bitext management process.

Scenarios 2) and 3) were deemed to be dependent on work in progress within OASIS XLIFF OMOS TC and hence were put on the back burner at TAPICC until XLIFF OMOS TC produces the required abstract data models as well as JSON (JLIFF) and other non-XML serializations. Although scenario 4) was deemed in scope of the TAPICC project, neither the Globalization and Localization Association (GALA) nor XLIFF OMOS TC have resources to tackle the scenario at the moment, therefore volunteers to engage in the API specification for this scenario are sought. The groundwork for this has been done at OASIS XLIFF TC and the provisions of the XLIFF 2 resource data module can be used to address this. Work on non-XML payload exchange would have a dependency on the XLIFF OMOS work in progress for scenarios 2) and 3).

Owner: GALA TAPICC Group.

IPR Mode: The steering committee originally targeted Non-Assertion (RF) but instead opted to be organized as an open source project licensed under the BSD 3 Clause license for code, and Creative Commons 2.0 BY for documentation. GALA decided not to mimic an SDO IPR mode but to donate stable deliverables to OASIS XLIFF or OASIS XLIFF OMOS technical committees, where these would be maintained under RF or Non-Assertion IPR modes respectively.

Current version and work in progress: XLIFF 2 Extraction and Merging Best Practice (XLIFF EMBP) are in version 1.0. https://galaglobal.github.io/TAPICC/T1/WG3/XLIFF-EM-BP-V1.0-LP.xhtml returns the latest published GALA TAPICC version. Several other deliverables have not yet reached stability, but are engaged in a number of public consultations via GALA webinars, for example.

Version 1.0 of the XLIFF EBMP was contributed by GALA to the OASIS XLIFF TC for publication as a TC note.

TAPICC is in a staged development phase, where the Track 1, chartered on February 8, 2017, has four active working groups. T1/WG1 was chartered to develop a consensual set of business metadata that will enable efficient payload exchange. T1/WG 2 was chartered to specify what kind of payload will be allowed to be exchanged via the TAPICC API, and how to resolve potential conflicts between payload encoded metadata and the business level imposed metadata. T1/WG3 was chartered to work on best practices of creating and consuming XLIFF 2 including documenting public libraries and services. T1/WG4 was chartered to design an actual RESTful API that will work with data models specified by the other T1 working groups, most importantly the T1/WG1.

The joint purpose of the T1 groups is to enable asynchronous API exchange within the complex localization supply chain, while relying on an XLIFF 2 based canonical data model.

In October and November 2018, TAPICC chartered and launched a call for volunteers for T2/WG1 — widely publicized in tcworld 2018 in Stuttgart on November 14, 2018. T2/WG1 – JLIFF REST API for Real Time Exchange of Bitext Units (JRARTEBU) has currently been assigned one Work Package: WP1 Unit Exchange. Batch or buffered exchange is out of scope of WP1, although not out of scope of Track 2 or T2/WG1. SOAP and any other protocols or bindings are out of scope of T2/WG1, although not out of scope for TAPICC Track 2. TAPICC Track 2 is going to reuse the OASIS XLIFF OMOS TC specified JLIFF format, the JSON serialization of the XLIFF 2 based object model. This synchronicity of data models will ensure semantic and behavioral interoperability between the asynchronous T1 and real-time transactional T2 bitext interchange.

Translation Memory eXchange (TMX)

Short Description: TMX has been arguably the most important and most widely implemented localization standard format. TMX is a simple XML vocabulary that was designed to provide lossless translation memory (TM) exchange. However, several obstacles prevented TMX from reaching the set goal. Level 1 implementations are too low a common denominator to actually secure lossless interoperability, because of segmentation differences (that should in theory be addressed by SRX) and because of absence of inline markup on Level 1. Level 2 stipulates lossless exchange of native inline codes that are however ignored by many tools and encoded as abstract placeholders. TMX is now far behind industry developments, but it will continue to be important for some time as a legacy format, mainly for collecting MT training corpora from legacy tools and repositories.

Owner: ETSI ISG LIS.

IPR Mode: RF in the LISA published versions, and FRAND in the ETSI ISG LIS version.

Current version and work in progress: There is no current work in progress, as the ETSI ISG LIS group was closed. Unfortunately, OASIS and ETSI were not able to agree on transferring the TMX IP to the OASIS XLIFF OMOS TC that had been chartered to take over the ownership of TMX.

1.4b is the current version with LISA numbering. It is also referred to as ETSI GS LIS 002 V1.4.2 (2013-02). Based on the published executive summary from November 2011, ETSI ISG LIS wished to coordinate TMX 2.0 development with XLIFF 2.0 definition of inline markup codes. During the existence of ETSI ISG LIS there was no technical development on the TMX front. The group only republished the latest LISA version on the ETSI Group Specification template. At FEISGILTT in June 2013 in London the ETSI ISG LIS chair made a public consultation of a possible related work item that ETSI ISG LIS would like to undertake the standardization of fuzzy matching calculations for text segments. At the time of ETSI closing ISG LIS (September 2015), there were informal talks about OASIS XLIFF OMOS TC taking over the TMX ownership, maintenance and development. The XLIFF OMOS TC was chartered and scoped to be able to take over TMX ownership. However, formal IP transfer between ETSI and OASIS has not been negotiated to the present day. Because of the legal matters being stuck, delegates of the June 2016 FEISGILTT and XLIFF Symposium discussed developing an XLIFF 2 profile with mandatory usage of the Translation Candidates module as a maintainable replacement of the TMX functionality.

In 2017, Andrzej Zydroń proposed developing a note within OASIS XLIFF TC that would describe how to use an XLIFF 2 profile instead of the obsolete and unmaintained TMX format. However, this work hasn’t progressed due to lack of interest and technical consensus.

Unicode Bidirectional Algorithm (UAX #9)

Short Description: Default text flow of Arabic and Hebrew scripts is right to left. However, text written in these scripts often contains portions with left-to-right directionality, such as names of companies or products. That is why such text is called bidirectional (bidi). Many characters have strong directionality properties, but there are also characters with weak directionality behavior and neutral characters whose directionality depends on context. In practice, normally invisible control characters (markers) need to be used in order to encode bidi in plain text. Simply put, UAX #9 is a detailed normative account of Unicode bidi behavior (mainly) in plain text.

In theory, the characters that the Unicode Bidirectional Algorithm makes use of to explicitly set text flow direction should not be used within markup context. Instead, the bidi flow control characters should be replaced with appropriate markup controlling the text flow. In practice, many tools ignore directionality markup and apply UAX #9 in full (including the control characters) even in structured and markup environments. This may be due to the fact that UAX #9 has a long tradition (since Unicode 2 in 1996). There is also a standardization gap, as Unicode control and stateful characters are not provided with clear processing requirements to be applied on entering markup environments. UTR #20 unfortunately provides only an abstract guidance rather than clear and unambiguous processing requirements. While XLIFF 2.0 has its own directionality attributes, it does not have attributes corresponding to inline bidi overrides or embeddings, so these are allowed as UAX #9 control characters if needed.

Owner: Unicode Consortium.

IPR Mode: RAND.

Current version and work in progress: Revision 39 was released for Unicode 11.0 on May 9, 2018. UAX #9 is being constantly revised to be up to date with the current Unicode release, and www.unicode.org/reports/tr9 links to the current official version. Additionally, the link www.unicode.org/reports/tr9/proposed.html always points to the latest proposed version if such a proposal exists. However, so far no revision has been proposed for Unicode 12.0 (to appear in May or June 2019; Unicode 12 Beta was authorized on September 21, 2018). Major versions happened between Revision 27 and Revision 29. Revision 29 sent ripples that profoundly influenced handling of bidi text in the world of markup languages. It led to introduction of new direction handling elements and attributes in HTML and XML vocabularies. Because directionality handling in HTML and several XML vocabularies was in flux in 2013, ITS 2.0 does not contain normative provisions for directionality handling. XLIFF 2 does contain up-to-date directionality markup and a valid guidance how to combine it with directionality control characters if necessary when modifying segmentation in compliance with the Revision 35 of UAX #9.

Unicode in XML and other Markup Languages (UTR #20) [Withdrawn]

Short Description: Unicode Technical Reports (UTR) are persistent names that always point to the actual revision. Unicode, as its main target is plain text, contains many control, formatting and other characters. This document gives a normative overview and general guidelines of which characters should and should not be used in markup context. In general, any Unicode character that is XML illegal or would require additional metadata for interpretation should come with a markup handling/replacement recommendation, or processing requirement. Authoring tools, XML editors and browsers are generally encouraged to ignore inappropriate or deprecated Unicode characters, so their preservation on crossing of plain text/markup boundary will often lead to harmful loss of data or metadata. In general, plain text is linear and requires special control characters or specific application behavior to encode metadata and/or styling information that can be handled with structured mark-up in XML or HTML environments.

Owner: W3C (Internationalization Core Working Group). UTR #20 was withdrawn at Unicode in March 2016, and maintenance of the document transferred to W3C.

IPR Mode: W3C (RF).

Current version and work in progress: Although this was last released as a W3C Working Group note on July 13, 2017, the status is still withdrawn. The material is still mostly stale and dated, as it was originally prepared for the base version of Unicode 6.2. The May 2016 update only discusses withdrawal of the UTR status and the transfer of maintenance to W3C. This note is still in need of a major rewrite. Handling of directionality overrides and isolates in XHTML has been described recently.

The newest developments can be followed on the GitHub Editor’s Draft http://w3c.github.io/unicode-xml.

Unicode Locale Data Markup Language (LDML or UTS #35)

Short Description: This specifies an XML vocabulary for encoding locale specific generic data categories — dates, amounts, decimals, units of measure, currency symbols and so on. Its main purpose is to enable the creation and maintenance of CLDR but is also used directly in programming frameworks such as .NET.

Owner: Unicode Consortium.

IPR Mode: RAND.

Above: Six new smileys form part of Unicode 11.0. Below: Parrot, llama and lobster are also new in Unicode 11.0.
Images: Emojipedia Sample Image Collection.

Current version and work in progress: Version 34, Revision 53, was released on October 10, 2018. Since the latest version and revision is for CLDR Version 34, a new release can be expected in about half a year for CLDR Version 35.

Unicode Regular Expressions (UTS #18)

Short Description: Unicode Technical Standards (UTS) are persistent names that always point to the actual revision. UTS #18 gives general guidelines for regular expression engines on how to comply with the Unicode Standard. Three levels are specified, of which two are default (one the minimum feasible for programmers, the other more end-user friendly) and the finest is language specific.

Owner: Unicode Consortium.

IPR Mode: RAND.

Current version and work in progress: Revision 19 was released on October 18, 2016. Revision 19 has Unicode 9 as its baseline. No new revisions have been proposed for Unicode 10, 11 or 12 at the time of writing.

The URL www.unicode.org/reports/tr18 always points to the current official version. Once a new draft is proposed, www.unicode.org/reports/tr18/proposed.html will link to it.

Unicode Standard

Short Description: Unicode is the core standard that allows humanity to encode all written human languages for computer use. Hundreds of thousands of characters covering alphabetic, syllabic and ideographic scripts and more are ordered in planes along with punctuation, control and private use characters. The Unicode standard has been published since October 1991.

Owner: Unicode Consortium.

IPR Mode: RAND.

Current version and work in progress: Unicode 11.0 was published on June 5, 2018. As of Unicode 7.0, there is an aggressive yearly schedule for major releases. Unicode 8.0 appeared in June 2015, Unicode 9.0 was released June 2016, and so on. Unicode 12 is on track for June 2019.

Unicode 11 added seven new scripts and 66 new emoji characters. After adding 684 characters in this release, the new total of Unicode characters stands at 137,374 individual characters. Character groups and whole new scripts continue to be added as per worldwide communities’ requirements. Relatively heavy editorial reshuffle in 7.0 enables the upcoming rapid cycle for major releases. The Unicode core had to adapt to changes introduced in the significant changes in Revision 29 (See UAX #9). Changes in Unicode 11 brought updates to UTS #10 Unicode Collation Algorithm; UTS #39 Unicode Security Mechanisms; UTS #46; and UTS #51 Unicode Emoji.

Unicode Text Segmentation (UAX #29)

Short Description: Unicode Standard Annexes (UAX) are persistent names that always point to the actual revision number. UAX #29 is the key normative source of segmentation rules. Apart from sentence boundaries, which are most relevant for computer-aided translation tools interoperability, it defines more basic grapheme cluster and word boundaries. The segmentation rules are given in more or less natural language as an inductive succession of rules. The specification states itself that the same set of rules can be given using regular expressions. Unfortunately, no finite set of regular expression-based rules can ensure 100% successful sentence segmentation of English text, the main reason being the semantic ambiguity of the full stop. Apart from closing sentences, the same character is being used for closing abbreviations, decimal points and so on. Interestingly, in Hebrew this problem virtually does not exist, as Hebrew does not overload the full stop with abbreviation function.

Although UAX #29 cannot possibly achieve completeness, it is still beneficial to implement it as the basic set of rules, and apply more fine-grained exception rules on top of it. The ULI TC does not plan to influence the default UAX #29 segmentation behavior and releases the locale specific segmentation exceptions as part of CLDR.

Owner: Unicode Consortium.

IPR Mode: RAND.

Current version and work in progress: Revision 33 was released for Unicode 11.0 on May 22, 2018, and www.unicode.org/reports/tr29 always links to the current official version. Additionally, www.unicode.org/reports/tr29/proposed.html always point to the latest proposed version, as long as one exists. The Revision 34 for Unicode 12.0 has not been proposed yet at the time of writing.

ULI exceptions use the UAX #29 behavior as a baseline. UAX #29 contains an informative pointer to CLDR released segmentation exceptions developed by ULI TC that can be used to modify the locale independent baseline segmentation behavior described in UAX #29.

Universal Terminology Exchange (UTX)

Short Description: UTX is a simple bilingual glossary format that was originally targeting MT training. It is a simple tab delimited format, and an XML version also exists, incited by criticism of the localization standardization community. UTX embedding has been specified for XLIFF:doc. TBX-Min mapping was proposed by LTAC Global. Although rather minimal, this tiny terminology exchange standard specifies the part-of-speech field as mandatory and also provides an optional field for term status tracking with predefined values.

Owner: The Asia-Pacific Association for Machine Translation (AAMT). Although the AAMT is not a traditional standardization body, their standardization working groups own and maintain UTX.

IPR Mode: The documentation is available under Creative Commons 4.0. No IPR mode has been specified.

Current version and work in progress: UTX 1.2 Minimal Specification was released February 18, 2018. There are some header changes, and the 1.2 version supports multiple sub-glossaries as part of one glossary. Editing in spreadsheet software became easier. As far as work in progress, the latest developments consisted in the development of TBX-Min mapping by LTAC Global (see TBX).

XLIFF Object Model (XLIFF OM)

Short Description: An abstract object model defined in UML diagrams and a prose specification. Definition of an XLIFF equivalent data model but also of interoperable file, subfile, unit, subunit and subsegment fragments. This requires detailed specification of data integrity dependencies that was not needed in XLIFF 2 and that is always being exchanged as a complete XML file.

The main goal is to be able to transfer the XLIFF inline data model into arbitrary XML and non-XML serializations. As such this is groundwork on an interchange data model that is needed to ground any messaging, bus or webservices architecture or a standard Translation API (see TAPICC).

Owner: OASIS XLIFF OMOS TC.

IPR Mode: Non-Assertion (RF).

Current work in progress: XLIFF OMOS TC was chartered in December 2015 to address the industry need of an abstract object model for the XLIFF 2 family of standards. The objective is to provide a serialization independent description of XLIFF and XLIFF fragment data categories that would allow development of non-XML serializations of the same data model. The goal is to make sure that arbitrary serializations of localization interchange data will be interoperable with XLIFF 2, vice versa and among each other. The object model is being developed on a GitHub repository at https://github.com/oasis-tcs/xliff-omos-om. The repository is public but the contributors need to be or become OASIS XLIFF OMOS TC members. Anyone can raise issues or comments via the associated issues or wiki, as well as minor bug fixes via pull requests. It’s up to the repository maintainers if those pull requests will be merged or not, which is the usual GitHub collaboration workflow. UML diagram development on that repository is powered by the open source Papyrus project.

XML-based Text Memory (xml:tm)

Short Description: xml:tm is a namespace application, which means it is not designed to form an independent document that could exist on its own. Instead, it is designed to be injected as a relatively heavy explicit internationalization apparatus into any well-formed XML document containing human readable language. Unfortunately, it is hardly possible to call this specification a standard due to a very low number of implementations — two, to be exact. The standard is being pushed by only one company without wider industry consensus. It was developed by XTM’s Andrzej Zydroń and donated to LISA, which published it as an OSCAR standard in early 2007. Its failure to become an actual standard should be a memento of the importance of broad consensus building while creating industry standards.

Owner: ETSI ISG LIS.

IPR Mode: RF in LISA published versions and FRAND.

Current version and work in progress: Zydroń has exposed the 2.0 version for public comment on XTM International’s web page. Its ETSI status is unclear from publicly available sources. The last version by LISA, 1.0, was released on February 26, 2007.

XML Localization Interchange File Format (XLIFF)

Short Description: In February 2018, XLIFF TC delivered the first “dot” release of XLIFF 2, which is backwards compatible with XLIFF 2.0. XLIFF 2.1 provides full ITS 2.0 support and advanced validation support. The 2.1 version also deprecated the Change Tracking Module and provided a number of bug fixes in both its core and modules. Interestingly and usefully, XLIFF 2.0 documents are forward-compatible with the XLIFF 2.1 advanced validation artifacts.

The XLIFF 2.0 core covers about 20% of XLIFF 1.2, because in 1.2 everything was core and hence the core was too big to be implemented across the industry.

XLIFF 2.0 and 2.1 were created and approved by a representative group of industry and academic standardizers: big enterprise translation buyers (such as IBM, Microsoft and Oracle), large language service providers (such as SDL and Lionbridge), toolmakers (such as MultiCorpora and ENLASO), industry associations (PSBT, GALA, TAUS), academics (LRC at the University of Limerick) and individuals (notably chair Bryan Schnabel). Many more took interest in the final OASIS organizational approval rounds and during the extensive public reviews.

XLIFF 2 is a lean modular standard that allows for plug and play interoperability in the areas of inline markup, segmentation, glossary exchange, translation and review, engineering quality assurance and so on, effectively catering for an end-to-end mashup of best-of-breed solutions throughout the content value chain.

The translation candidates module allows for local inclusion of translation candidates coming from various sources including translation memories, MT services and various crowdsourcing scenarios. The glossary module allows for local inclusion of relevant terminology in both source and target languages. It supports both monolingual and bilingual scenarios, including terminology life cycle management scenarios and one-to-one mapping with TBX-Basic and UTX is possible. The formatting style module provides two attributes for embedding HTML-encoded preview information that can be used by agents for on-the-fly preview generation to provide context for human translators. The metadata module is a non-namespaced private extensibility mechanism. Custom metadata can be grouped and presented as key-value pairs, which is a predictable way of facilitating interoperable display. The resource data module replaces the binary module capability from 1.2. The change tracking module allows users to log change history, including the provenance of changes. The size and length restriction module provides a generalized way for specifying text size/volume restrictions. This module can cater to simple use cases, such as fitting a database field based on Unicode code point or byte count, and it also caters to complex scenarios, such as fitting specific hardware display restrictions using a specific font. The validation module provides a mechanism for specifying simple localization rules to be checked with relation to source and target, to ensure that a brand name will be included in the target text, for example.

Owner: OASIS XLIFF TC.

IPR Mode: RF on RAND terms in OASIS (freely available). RAND in ISO (sold).

Current version and work in progress: Version 2.1 was published as an OASIS standard on February 13, 2018.

ISO 21720:2017 – XLIFF (XML Localization interchange file format) is identical to the OASIS XLIFF Version 2.0 from August 5, 2014.

After successfully delivering the ITS Module and advanced validation capabilities as part of the 2.1 version, the TC worked on reformatting XLIFF Version 2.1 in ISO style, so that it can again be submitted for publishing in ISO TC 37/SC 5.

The TC also started publicly tracking feature development for the Version 2.2 at https://wiki.oasis-open.org/xliff/FeatureTracking.

The biggest and most interesting feature proposed for Version 2.2 is Rendering Requirements. A detailed proposal of XLIFF Rendering Requirements was presented at the 40th edition of the ASLING Translating and the Computer Conference on November 16, 2018. The idea is to give possibly normative guidance to CAT tool developers on how to display the XLIFF bitext and its rich metadata, but also to lower the technology complexity of rendering bitext and make interactive rendering of XLIFF possible in web browsers. Other proposed features include an attempt to reintroduce the Change Tracking Module that was deprecated between 2.0 and 2.1.

The XLIFF TC is also wrestling with the idea of how to use XLIFF for TMX, since TMX is in legal limbo and cannot be maintained or further developed. Three technical options are under consideration: 1) how to informatively describe an XLIFF 2.0 profile that only uses XLIFF 2 Core and the Translation Candidates Module; 2) describe how to use XLIFF Core storage as a translation memory (TM); 3) develop a new grammar that would reuse the XLIFF 2.0 vocabularies but define a different structure that would be more suitable for bulk exchange of TMs, including in multilingual scenarios.

The work on TBX Basic mapping both to and from XLIFF 2.0 (and higher) was transferred to the XLIFF OMOS TC.