We are excited to introduce miniG, a powerful new model designed to explore the frontiers of large-scale synthetic data training.

A New Approach to Training

miniG is a 9B parameter language model (initialized from THUDM/glm-4-9b-chat-1m) complemented by an optional 5B ViT, making it a capable Vision Language Model (VLM). Its foundation is a unique synthesis dataset of over 120 million entries. This dataset was generated using state-of-the-art language models with large context windows, leveraging methodologies akin to retrieval-augmented generation (RAG) and knowledge graph integration.

The entire data synthesis process was conducted within clusters derived from a curated 20 billion token pretraining corpus, with subsequent validation performed by the model itself. Notably, miniG has not undergone thorough alignment with human preferences and is under no obligation to cater to poorly constructed prompts or benchmark clichés.

Core Features

• Supported Modalities: Text and Image. The Vision Language Model has undergone Locked-Image Tuning. For a text-only version, please use the revision=text-only branch at https://huggingface.co/CausalLM/miniG/tree/text-only.
• Massive Context Window: miniG supports a context window of up to 1,000,000 tokens.
• Model Parameters: The model consists of a 9B LLM and an optional 5B ViT.

Usage & Inference Recommendations

Cautionary Notes: We strongly recommend using a standardized implementation like Hugging Face transformers for inference. Accelerated kernels such as vllm or lmdeploy, as well as model quantization, can lead to significant performance degradation and potentially catastrophic effects, especially for vision inference.

Inference Parameters: To achieve results with fewer hallucinations, we advise using sampling with top_p=0.8 and temperature=0.3, or pure temperature sampling at 0.2. A lower temperature is generally required compared to similar models, which we tentatively attribute to overfitting on the vast dataset.

Input Formatting:

1. Ensure the system prompt is not empty. Even a simple "You are a helpful assistant." is sufficient.
2. Always include a newline character \n after each <|role|> tag for proper parsing.

Training & Disclaimer

The final released version was a merge of multiple candidate models. An efficient naïve fine-tuning can be achieved within one day on 16 nodes of 8*A100-80G, with an estimated carbon emission of 700 kg CO2 eq.

Disclaimer: The model was trained on unfiltered internet data and may contain objectionable content. We lack the resources to vet all data or implement RLHF for safety. Users are responsible for performing their own safety checks and filtering model outputs.