Margaret Marks at Translation Blawg rightly wonders what on earth the Webster’s Online Dictionary (WOD) is all about. Although there is quite a lot of background information available on the site, I decided to find out from its creator Phil Parker. Here’s the score.

A Professor at Marketing at INSEAD, the European business school, Philip Parker was born dyslexic. This meant he found reading dictionaries – lists of words and their definitions – much easier than sustained prose, which demanded too much time to decipher. So over the past 30 years he has been collecting dictionaries of all kinds. Around the year 2000, large dictionaries on the web started charging for ‘premium’ words of the sort he needed in his research and that really “pissed him off”. So he decided to leverage the definitions he had collected from his own store, borrowed the out-of-copyright ‘Webster’ badge, and started building WOD, which he intends to make the biggest multilingual dictionary site on the web.

He was lucky since he had loads of help from academic and other assistants, benefited from donations of out of print dictionaries and word lists, and was able to finance the whole thing himself. He even uses a firm in Togo to keyboard in content. This summer he hopes to upgrade the site to feature dictionaries covering 600 languages (10% of the world’s current language population), and in the case of existing site languages such as Spanish, he hopes to increase the entry count from around 100,000 to 600,000 entries.

To give global coverage he is working in a sequences of passes. The first pass was to work by time zones, taking a location such as Europe and collecting dictionary materials for all ‘major’ languages. The second pass, now under way, is to include ‘secondary’ languages (say Maltese in Europe). Next year, he plans to start the third pass by incorporating locally endangered languages, using volunteer help where necessary. One technique is this: he donates a computer and a small stipend to missionary children (e.g. for Tarahumara in Mexico) who then create a local language/English dictionary.

What’s next, once he’s got all these bilingual word lists? Create a total lexical linker, whereby you can click from any word to its equivalent in any other language, using English as the underlying pivot language. An “N-dimensional cube of words in every language to every language,” as he puts it, that will by this summer be the world’s largest compilation of language items ever produced. His content currently weighs in at around one terabyte.

How useful is Phil’s site proving? He reckons it is among the top ten sites used to search Arabic words in Arabic script, since the whole hoard has been programmed for Unicode. And because the Webster word is a synonym for ‘dictionary’ for Americans (as Kodak once was for cameras or Google is for search engines) WOD ranks between 5 and 7 on, well, Google for ‘Webster’ out of about 150 ‘Webster’ sites on the web these days. Probably the best way to appreciate the ambition of Phil Parker’s site is to search the term Webster itself, and see the degree of encyclopedic potential – words, images, statistical findings from corpora, sign language versions, et alia multa – that he is trying to pack into what he calls a hobby. But the definitions don’t include a more recent decomposition – web + ster (as in napster) – a linguistic peer to peer resource.

Is there a ‘best’ automatic translation system? Sure. But you can only know by constantly testing them all across all languages against a vast range of specific tasks. In other words, never. “Best” looks like a Platonic idea, not a statement of fact.

Lots of blogs and sites these days are quoting the Language Weaver tag line – “the best translation systems in the world”. Does this help or hinder perceptions of translation automation, given the highly relative quality of any translation? Or should L W have taken a leaf out of Heineken’s book and added a modifier. The beer advertisement we sometimes see (at least in Europe) has “probably the best beer in the world”. Again, meaningless, but carefully nuanced to engender that vital suspension of disbelief we need when reading ads.

As it happens, Language Weaver has a best rival – an outfit called Delta, which appears to market a Brazilian/Spanish-English system for 125 bucks a shot, a far cry from what L W’s Arabic to English system would cost. Yet when you check to see how many others are trying to best the market, the results are disappointing. According to this Google search, “best machine translation systems” – note the plural – scored 65 hits, and “best translation system” managed 145, most of them from Delta. With a hundred or so citations out of a billion web pages, it clearly doesn’t matter to most people whether any systems are “best” or not.

That plural in Language Weaver is, however, an astute move. Unlike Delta (and many others) the company claims to develop language-specific systems (Arabic, Chinese, Hindi, French, Spanish to English), based on the analysis of statistical patterns of linguistic type phenomena in large bitexts. In other words, it has some sort of “technology” (recently reported on here probably causing the bloggorhea attack) that it deploys in “systems”.

Most of us, however, have tended to think in terms of the X translation system – that is, the underlying engine in X, that somehow has linguistic knowledge hand-coded into it (a dictionary of say 100,000 terms and 3,000 grammar and morphology rules) to drive whatever language pairs are on offer. Perhaps by going superlative about its nonce systems, rather than its core “machinery”, L W is shifting the focus from technology to task diversification. And in the L W case, translation tasks appear to be addressed by leveraging the deciphered output from previous similar tasks.

Let’s hope that a provocative “best” attracts competitors to this space. Plenty of room for more “systems”, even if there are only one or two effective core technologies.

Seems to me there is a knowledge hole in the language technology communication space. There are lots of R&D sites and community events, a few dedicated news sites such as here and here that relay press releases, and one notable publication that is gracious enough to host this blog. What’s missing is any sense of what gets language technologists going. So here’s the first of an occasional series of quickie chats with some lang tech rockers and rollers to find out what’s cooking – and why.

Lingster is an open source initiative to collect and share dictionaries for innovative translation automation systems. It was started by Olga Beregovaya, who works by day as a computational linguist for a large software developer in California, and whose night job is building a multilingual content management system with computer-assisted translation capabilities. Olga is a graduate of St. Petersburg State University, and UC Berkeley.

“I realized I needed lots of lexicons and glossary data. What was available consisted of very plain monolingual word lists, which makes multilingual alignment hard. And they weren’t marked up for morphology or part of speech.

“You can find material on the web but if it’s free and available it’s crap. If it’s decent you have to pay US$ 15,000 or so which is too much for me. I am planning to include 15 language pairs in my application, and I need up to date terms and expressions. So I started lingster as an open source dictionary portal so others could help me and I in turn could help them. 

“The idea is to identify and contribute linguistic data in the form of lexicons and glossaries, idiom lists etc. The next step is to add in part of speech and grammatical information to expressions, and then the R&D community can download the resulting resources for free. This way we all benefit.

“A lot of other people seem to be suffering from the data problem, as the interest in the last few months has been terrific. Lots of linguists see the value of this sort of initiative. Especially since web crawlers programmed to harvest dictionary data end up by hitting Intellectual Property problems. 

“Quality is obviously an issue. We have a filtering system to validate terms using community voting, with a volunteer moderator for each language who throws out unacceptable equivalents. We use a UTF8 control mechanism to stops garbage characters getting into the files, and a conversion mechanism for file formats.

“There is no charge for the software which is all open source, with no hidden dictionary data. We welcome the concept of a community of development, since contributors will be rewarded according to the open source licensing system. Later on, we shall set up a subscription system for those wanting truly premium data.”

And what exactly is Olga’s pet project? An elearning facility that turns any web resource into a potential language learning resource, with instant word lookup. The beta’s due out in a couple of months.