Meta earlier announced a new feature that can handle long text content.Llama 2 Long Large Natural Language Model, based on 32768 sets of tokens and 700 billion parameters, and its overall performance is better than the GPT-3.5-Turbo-16K version that can also process long text content.
Llama 2 Long's strength lies in processing long texts and keeping track of the context, thereby meeting more complex and diverse artificial intelligence interaction processing needs, including natural interaction with chatbots or analysis of documents with large amounts of content.
Previously, large-scale natural language models that could handle long texts were mostly used for commercial applications. Therefore, Meta proposed Llama 2 Long, which will be built onllama 2Based on the open source model itself, it is also provided to more researchers and developers in an open source form.
The training method is based on Llama 2, with an additional 4000 billion markers pre-trained. At the same time, these markers are divided into more smaller sequences. For example, when training a model with 70 billion sets of markers and 130 billion sets of parameters, 32768 sets of marker sequences are used for training. When training models with 300 billion and 700 billion sets of parameters, 16384 sets of markers are used for training.
This allows Llama 2 Long to better perform in contextual correspondence in long texts. Even as the length of the content increases, the range of contextual correspondence will also increase. This allows it to cope with complex program development, content analysis descriptions, or more complex conversational interactions, while also allowing for the training of large natural language models at a relatively low cost.
