For all the sites for language games , constructed languages , artificial languages, Klingon, palindrome collectors, machine translation jokery, slang rangers and the like now active on the web, has anyone tried to apply decent digital design (DDD) to the task of showing how plain old natural language works?

By DDD, I mean deploying animation, graphics, and powerful multimedia interactivity. For the last couple of millennia, discourse about language (i.e. linguistics, translation theory and even language teaching) has made do with static 2D graphics, word lists, tree diagrams to show relationships between items, and various sets of icons to represent spoken pronunciation. We now have the computing power and associated design practices to represent codified knowledge about language so much more powerfully. Surely it’s time to pull language description as a discipline screaming and kicking into a visual culture of swarms, flows and 3D.

There are two obvious angles of approach for a radically dynamic visual representation of language. One would be to show how languages constantly change through time and space – the geography and history of all human tongues as interweaving kinetic patterns. This is the easy approach, since it is in a sense “external” to language as discourse. One simple attempt I have seen is this ‘animated’ site devoted to showing how various character sets, such as the Latin alphabet, evolved over time. But DDD I’m afraid it ain’t.

The other, far more complex line of attack would be to explore the internal structures and relationships of languages, again as visual displays of fields, waves and particles in constant evolution, not as lists of immutable rules. The problem here is that the analysis of language structure underlying such a presentation is theory-driven rather than an objective given. But you could surely generate a nice digital movie that includes all the different approaches to language analysis as they compete for explanatory supremacy.

The real issue about this DDD approach is that it would itself embody a ‘theory’ about language as flow, rather as our old bookish word culture of language led to a theory of structures that were somehow transformed over time. A dynamic, flowing view of language as endless coagulations of meanings emerging into symbolic form and then disappearing again, contrasts starkly with the mechanical plod of a history based on rules.

The linguist Andrew Wedel recently drew attention to this new linguistics agenda:

“I think there is a big shift from the explanation from a single level, advocated by Noam Chomsky, that one grammar algorithm is coded in our genes, to a more layered set of explanations where structure gradually emerges in layers, over time through many cycles of talking and learning,” he said.

“Languages are the ripples in the dunes and the grains of sand are our conversations, generations talking to each other and learning things and slowly creating these larger ripples in time.”

And Rob Freeman among other linguists is trying to develop computer models of language built out of rules that emerge from utterances, rather than utterances being pre-defined by rules. Now that we can stream our talk and text as bits and bytes that can instantly transform into any kind of multimedia pattern, we have a source of raw data whose own properties – not those imposed by a tradition of lists and rules – could inspire understanding. Or just entertain us.

Under government auspices, a new report has been published by French information and language industry bodies on “Automatic Language Processing in the Information Industry”, researched by the Paris office of Bureau Van Dijk. Well, maybe not such a new report: the data comes from a 2002 survey and I believe was first presented at LangTech 2003.

Here are the key findings as presented in a Paris press conference today:

· The language and speech tech market divides into 80% text and 20% voice. [Not everyone would agree with this].

· The European LT market in 2002 was worth 510 million €, or 33% of the world market.

· UK, France, Germany and Italy accounted for 60% of this market, the UK having the biggest share at 18%, largely due to its flourishing speech tech market.

· In France, the key segments (from document management via translation automation to searching) were content management (search) type applications (25%) and document management (categorization etc) systems (20%). Translation accounted for only 10%.

· Most players estimate growth –to stand at 10-15% a year. Most demand is for document management at 23% and search applications at 21%.

· ROI come mostly from time saving rather than financial returns. Main costs incurred come from feasibility and needs analysis, followed by software development, then customization and maintenance. Multilingual applications in general double the ROI horizon.

· Main trends: shift from supply to demand, as new mobile and desktop platforms in a multilingual search and personalized work space require specific tools.

· The European LT market is growing at 6.9% annually, compared to 9.4% in the USA. The global LT market will be worth 2 billion € in 2005 and should rise to 3 billion in 2007.

I don’t think these findings will surprise anyone. Language technology appears to be but a minor segment of the software or IT industry, with lots of small players (over 300 in Europe alone) that have emerged as spin-offs and startups from the R&D base and are now ripe for M&As. Consolidation is already more striking in the U.S. than in Europe, probably due to such structural brakes as nation/language-based fragmentation in Europe, and an endemic culture of lower investment in risky ventures. This means that current players tend to have small customer footprints in a few selected industries where advanced business intelligence (rather than, say, translation) is a key concern. The scope for real growth and infrastructure investment is often minimal

But if an ambitious global player in a related but external segment – voice technologies, for example – decided to integrate language technologies to provide a better user experience for, say, mobile phone/PDA access and search with built-in multilinguality, then that player might start cherry picking any European players with good technologies for translation, or semantic searching. It happened once before in Flanders Language Valley, though aided by considerable public investment. For the time being though, it looks as if pervasive language technology will continue to be “decades away” as press articles keep telling us.

Useful item from Burritt Sabin on a recent Japanese Language Subcommittee report on kanji, calling for an overhaul of the joyo kanji (the official 1,945 characters used in administrative life and education).

…personal computers, cell phones, and other IT devices may be encoded with as many as 6,355 characters, the total of the first and second levels of the Japanese Industrial Standards (JIS). Simple arithmetic shows the Japanese are using a lot of non-joyo kanji.

What’s more, the use of Japanese and Chinese readings and character styles not in the joyo kanji list is increasing. For example, the old forms of the kanji for “kuni” (country) and “sawa” (valley) have come to be widely used in personal names. As well, many kanji used in place and personal names are not in the joyo kanji list. The reason, according to a Cultural Affairs Agency spokesperson, is that “proper names were expected to be handwritten.” The joyo kanji list was intended as a guideline for printed characters. Even “saka” in Osaka and “oka” in Okayama are not joyo kanji.

See here for more on Japan’s language policy (in English).

Further to my post on IBM releasing numerous language and speech tech patents to the open source (OS) domain, it’s worth mentioning that The PatentCafe’s Patent Search Engine is now offering search and delivery of PDF copies of all the 500 IBM patents for a nominal fee to the OS community.

Common Sense Advisory (CSA) has published its Fourth-Quarter Global Business Confidence Survey on the translation services industry.

The surveys polled buyers and suppliers of language services and technology about their current business situation, plans, and expectations for the near future.

CSA has also just published SDL 2005: Challenges and Opportunities, the second in its series on assessing the (few) publicly traded language service companies.

After IDC and ABI Research and a few others had a go at mapping the language services / localization (software) space during the late 1990s and early 2000s, CSA is now establishing itself as the only research company that is providing sustained close tracking of the translation / localization industry in the context of global business challenges. I should declare an interest here, since I have worked with CSA, though not on these reports. Still, since you only work with companies because you like the way they do things…