Earlier this week, Meta released Llama 2, its latest large language model (LLM).
Llama 2’s predecessor — Llama — was initially leaked to the public in March; in this most recent version of the LLM, Meta has made an effort to improve transparency in the LLM space by making the tool open source. In a paper published Tuesday, the researchers behind the latest release shared their methodology, noting that Llama 2 appears to be safer — that is, less biased — than other popular models like Google’s PaLM.
“This paper contributes a thorough description of our fine-tuning methodology and approach to improving LLM safety,” reads the paper published by Meta’s researchers on Tuesday. “We hope that this openness will enable the community to reproduce fine-tuned LLMs and continue to improve the safety of those models, paving the way for more responsible development of LLMs.”
The number of publicly available LLMs has skyrocketed in recent months, as developers rush to get their piece of the pie. PaLM, GPT-4, Claude, and now Llama 2 — the list goes on.
Llama 2 is actually a collection of four LLMs, each with a different number of parameters — the smallest with seven billion parameters, the largest with 70 billion. About 90% of the training data was in English, with roughly 9% in unknown languages. The remaining 11% of training data was written in a wide range of languages, including German (0.17%), French (0.16%), and Chinese (0.13%).
Meta notes that the bias in training data toward English means Llama 2 will likely not be suitable for use in multilingual (or even just non-English) contexts.
According to the paper, one of the researchers’ primary focuses in developing Llama 2 was improving safety. Safety — in this context referring to language that is unbiased and nonviolent — has been a challenge in the development of LLMs. That’s in part because these models are trained using millions upon millions of words and texts that are difficult to vet — essentially, they recreate the biases that already exist in human language.
On this measure, the researchers believe Llama 2 fares much better than other LLMs. On a human evaluation of Llama 2 and other LLMs (including PaLM and ChatGPT), Llama 2 appears to be quite safe, with comparatively few safety violations.
“While many companies have opted to build AI behind closed doors, we are releasing Llama 2 openly to encourage responsible AI innovation,” the researchers write. “Based on our experience, an open approach draws upon the collective wisdom, diversity, and ingenuity of the AI practitioner community to realize the benefits of this technology. Collaboration will make these models better and safer.”
