graph LR
Data_Preprocessing_Acquisition["Data Preprocessing & Acquisition"]
Tokenizer["Tokenizer"]
Data_Loader["Data Loader"]
Llama_Model_Core["Llama Model Core"]
Training_Module["Training Module"]
Inference_Generation_Module["Inference & Generation Module"]
Main_Application_Entry_Point["Main Application Entry Point"]
Reference_Implementation["Reference Implementation"]
Data_Preprocessing_Acquisition -- "processes and tokenizes data for" --> Tokenizer
Tokenizer -- "provides tokenized data to" --> Data_Loader
Data_Loader -- "supplies batched data to" --> Training_Module
Training_Module -- "updates weights and parameters of" --> Llama_Model_Core
Llama_Model_Core -- "feeds back outputs to" --> Training_Module
Inference_Generation_Module -- "uses" --> Llama_Model_Core
Inference_Generation_Module -- "decodes generated tokens using" --> Tokenizer
Main_Application_Entry_Point -- "initiates construction of" --> Llama_Model_Core
Main_Application_Entry_Point -- "orchestrates" --> Training_Module
Main_Application_Entry_Point -- "initiates" --> Inference_Generation_Module
click Data_Loader href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/nano-llama31/Data_Loader.md" "Details"
click Llama_Model_Core href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/nano-llama31/Llama_Model_Core.md" "Details"
click Training_Module href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/nano-llama31/Training_Module.md" "Details"
click Inference_Generation_Module href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/nano-llama31/Inference_Generation_Module.md" "Details"
click Main_Application_Entry_Point href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/nano-llama31/Main_Application_Entry_Point.md" "Details"
click Reference_Implementation href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/nano-llama31/Reference_Implementation.md" "Details"
The nano-llama31 project is structured around a core Llama 3.1 model, supported by distinct modules for data handling, training, and inference. The Data Preprocessing & Acquisition component is responsible for preparing raw text data, which is then tokenized by the Tokenizer. The Data Loader efficiently batches this tokenized data for consumption by the Training Module. The Llama Model Core encapsulates the fundamental architecture of the Llama model, with the Training Module orchestrating its optimization and learning process. For generating new text, the Inference & Generation Module leverages the trained Llama Model Core and the Tokenizer for decoding. The Main Application Entry Point serves as the central orchestrator, coordinating the setup and execution of the model's training and inference phases. A separate Reference Implementation provides a self-contained, simplified Llama model for comparative purposes, operating independently of the main Llama Model Core. This modular design facilitates clear separation of concerns, enabling efficient data flow and maintainable development.
Handles the initial retrieval, cleaning, and preparation of raw text data, specifically for the "Tiny Stories" dataset.
Related Classes/Methods:
Manages the conversion of text to numerical tokens and vice-versa, essential for model input and output.
Related Classes/Methods:
Data Loader [Expand]
Efficiently loads and batches tokenized data, particularly for training, supporting distributed data handling.
Related Classes/Methods:
Llama Model Core [Expand]
Defines the fundamental architecture and computational logic of the Llama 3.1 model, including its layers and forward pass.
Related Classes/Methods:
Training Module [Expand]
Orchestrates the model training process, including optimizer configuration, loss calculation, and weight updates.
Related Classes/Methods:
Inference & Generation Module [Expand]
Manages the process of generating new text sequences from the trained Llama model.
Related Classes/Methods:
Main Application Entry Point [Expand]
The central control flow for the llama31 application, coordinating model building, training, and inference.
Related Classes/Methods:
Reference Implementation [Expand]
A separate, simplified implementation of Llama model building and text generation, serving as a comparative or demonstrative example.
Related Classes/Methods: