- Google Colab is an alternative to Jupyter notebooks that run on their GPUs for free.
RAG - Retrieval Augmented Generation
- It is simply a technique to improve prompts, like a Knowledege Base powered skill, which adds relevant details to the prompt.
- Simplest implementation: when sending a system prefix, add the context dynamically (e.g. collect all data from documents inside of KB for all documents that contain a word from the prompt).
- When testing RAG, remember that the conversation history is always sent to the LLM, so the results for a prompt may be based on a previous answer, and not the RAG itself. Always use a fresh chat to ensure correctness.
- Auto-regressive LLM: predicts a future token from the past.
- Auto-encoding LLM: produces output based on the full input. Example: classication; sentiment analysis; vector embeddings calculation.
- Vectors mathematically represent the “meaning” of an input. It converts unstructured text chunks and queries into numerical arrays that capture semantic meaning rather than exact words. The vectorization result can then be fed as text into a regressive LLM to enhance its context.
- The big idea behind RAG: understand the meaning (via vectorization) of the question/prompt to produce a relevant answer. An encoding LLM is used to vectorize.
- LangChain is a framework for interfacing with many LLMs. It lets you buil simple, linear, step-by-step LLM pipelines.
- Documents should be divided into chunks before vectorization. Results will vary depending on the strategy used to split the documents; therefore, experiment different strategies and evaluate the results.
- An Encoder Model (e.g OpenAI
text embedding 3 small, gemini-embedding-001) turns text into a Vector Embedding. A Vector database like Chroma is where they get stored. The choice of embedding model is more important than the database choice.
- To load data with LangChain: read from disk, divide into chunks, vectorize with encoder model, and store into vector database.
- Visualizing the vectors distribution in 2D/3D can help you analyze the embedding model results.
- LLM Temperature: controls how diverse the output is, like entropy - the higer the value, the less predictable the responses.
- Evaluations (EVALS): curate a test set (example questions with the right context, and reference answers provided); measure retrieval (MRR, nDCG, Recal@k, Precision@k); measure answers (e.g. use LLM to score provided answers).
- Structured outputs can be used when using LLM-as-a-judge to compare the reference answers against the generated answers.
AnswerEval is how the response format is called on LiteLLM
- Evals can be used to validate the chunk heuristics you used.
- RAG Advanced Techniques
- Chunking R&D: experiment with chunking strategies.
- Encoder R&D: select the best encoding model based on a test set.
- Improve prompts: general content, the current date, relevant context and history.
- Document pre-processing: use an LLM to make the chunks or text for encoding.
- Query rewriting: use an LLM to convert the user’s question to a RAG query.
- Query expansion: use an LLM to tunr the question into multiple RAG queries,
- Re-ranking: use an LLM to sub-select from RAG results.
- Hierarchical: use an LLM to summarize at multiple levels.
- Graph RAG: retrieve content closely related to similar documents.
- Agentic RAG: use agents for retrieval, combining with memory and tools such as SQL.
Training and Fine Tunning
- You need to curate the data before training. Plot data to check if the data is skewed, otherwise the distribution will influence the results. Apply weights or penalties if needed. Experiment with the results, and refine.
- Correlation of data points may also influence the results.
- Always start with a baseline model: simple, traditional ML models to have a starting point to compare. E.g.: linear regression.
- LoRA is a parameter-efficient fine-tuning technique for LLMs that enables adaptation of pre-trained models to specific tasks while requiring significantly fewer computational resources and trainable parameters than traditional full model fine-tuning. QLoRA is quantized LoRA, which reduces the precision of the weights.
Agentic AI
- An agent runs tools in a loop to achieve a goal.
- Don’t anthropomorphize and give agents a human role, but rather focvus on solving a business problem.