Doing retrieval-augmented generation on mobile device is not easy. This MemeLM is a conversational chat with model that run full local. The RAG is also should be local due to privacy concern. This will hand an extra steps to able to get past context user really meant. Current state is the MemeLM retrieve previous conversation directly to the intial prompt. That will give performance disaster when conversation grow over time.

Refer to this RAG-Driven Generative book, RAG pipeline has three main part:

1. Data collection and preparation: One takes on collecting the data and cleaning it.

2. Data embedding and storage: This state of storing and retrieving the embedding and vector data.

3. Augmented Generation: Last process handles the generating content based on user prompt and retrieval query.

Theorically those process required to make user conversation context more related each of turns. Due to storing conversation data, the Room DB is enough. Room which use SQLite is strong enough and well-integrated with Android environment perfectly.

Memory has two main context: short and long term. Short memory refer to conversational section only and retrieve all context based on active session. Long term memory persist on disk and stay as main-general information about the user itself.

Cleaning data is crucial to get optimal result. The conversation messages is full of unnecessary symbol like emoji. So the OpenNLP will be use only for cleaning the data and tokenize. Preprocessing stage also handle the text normalization before sentence get process with the data cleaning.

Embedding

Embedding mean to transform document to form of vectors. This way make computation faster. Embedding model need to compute before send prompt to the LLM model and after LLM model give result back. The first way is needed because LLM model basically do not have knowledge after cut-date of its dataset. So, any previous conversation barely unknown for the model to process. The MemeLM app use conversation text as document and save in storage as text and vector form. Every message user sent will be kept as text and vector data to future use. Better data, better context and more excelent results will be. Best embedding model for continuesly enlarge data is Embedding Gemma 300m gguf model.

Vector Search

FAISS is a library for efficient similarity search and clustering of dense vectors. Eventough this made for Python, since it made from C++ and able to build, then it is possible to Android to use its features. This library used for storing vector from embedding model with strong performance.

Retrieve context rely on how the communication between embedding model with FAISS. When embedding model give vector result, then the vector storage can search close or similarity value then retrieve the ID of saved message inside local persistance storage. This make computation faster and retrieve only what recent prompt given context.

Because current storage data persistance has two context: chat and global context, we need one context manager to keep everything under control. Single ContextWindowManagement will maintain context retrieval depends on prompt the user give. This is need further detail explanation, we keep that for future topic.

Generation

Augmented generation is when processed data pass to the LLM model. Processed data is ready-to-process from long chain of processing the prompt/message. This is handled by small language model MiniCPM-V4.6 which multimodel along with its Mmproj model. The MemeLM actually my demo app for my main SOTA product which MyZeta AI Avatar (or whatever similar) for implementing RAG on-mobile. So, we have nothing to lose.

The main problem here is about memory. I have not calculate yet real-world how much memory need to process everything on-hand. Technically the app need LLM model, Embedding model, Multimodel model, text cleaning model, vector model (maybe).

All chain is ready to implement, this document is done. The second post will give clear way how do I made it. Wish me luck.

Best,

Dani Zakaria.