Mastering Retrieval-Augmented Generation (RAG) with LLMs: A Comprehensive Guide

 RAG Background.

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating retrieval mechanisms with generative capabilities. It improves response accuracy by accessing external databases for relevant information, overcoming LLM limitations in knowledge cut-off and hallucinations. RAG combines the strengths of retrieval (precise, up-to-date data) and generation (contextual, fluent language), making it essential for complex queries, factual correctness, and dynamic knowledge. It optimizes performance, especially in specialized or rapidly evolving fields, ensuring comprehensive, accurate, and contextually relevant outputs, thus significantly enhancing the utility and reliability of LLMs in practical applications.

     In this tutorial, I delve into the key topics and advancements related to Retrieval-Augmented Generation (RAG) using executable codes. Each topic is meticulously explained through video tutorials, offering detailed yet accessible discussions and working demonstrations, accompanied by the corresponding code. Some code segments are sourced from relevant libraries for demonstration purposes. This comprehensive tutorial covers the following topics, providing an in-depth understanding of RAG and its practical applications.

1. Basics of RAG (Retrieval Augmented Generation) with LLM
2. How to use LLM + RAG to Construct Knowledge Graph.
3. How to construct Flow-Diagram by Using LLM + RAG.
4. Graph Based RAG (Retrieval Augmented Generation) Techniques.

Note: In addition to this, it also provides a linked tutorial on a pressing topic: "Use of Long Text Sequences with LLMs Trained on Shorter Text Sequences.". In the future, I will introduce new research advancements in the field of Large Language Models. 

1. Basics of RAG (Retrieval Augmented Generation) with LLM.

Video Tutorial.

Basic RAG Code.

import ollama
import chromadb

documents = [
  "Quantum mechanics is a fundamental theory in physics that describes the behavior of nature at and below the scale of atoms.",
  "It is the foundation of all quantum physics, which includes quantum chemistry, quantum field theory, quantum technology, and quantum information science.",
  "Quantum mechanics can describe many systems that classical physics cannot.",
  "Classical physics can describe many aspects of nature at an ordinary (macroscopic and (optical) microscopic) scale, but is not sufficient for describing them at very small submicroscopic (atomic and subatomic) scales.",
  "Most theories in classical physics can be derived from quantum mechanics as an approximation valid at large (macroscopic/microscopic) scale.",
  "Quantum systems have bound states that are quantized to discrete values of energy, momentum, angular momentum, and other quantities, in contrast to classical systems where these quantities can be measured continuously.",
  "Measurements of quantum systems show characteristics of both particles and waves (wave–particle duality), and there are limits to how accurately the value of a physical quantity can be predicted prior to its measurement, given a complete set of initial conditions (the uncertainty principle)."
]
# Create database
client = chromadb.Client()
collection = client.create_collection(name="docs")
# store each document in a vector embedding database
for i, d in enumerate(documents):
  response = ollama.embeddings(model="mxbai-embed-large", prompt=d)
  embedding = response["embedding"]
  collection.add(
    ids=[str(i)],
    embeddings=[embedding],
    documents=[d]
  )

# an example prompt
prompt = "What are the key benefits of using quantum mechanics over classical physics?"

# generate an embedding for the prompt and retrieve the most relevant doc
response = ollama.embeddings(
  prompt=prompt,
  model="mxbai-embed-large"
)
results = collection.query(
  query_embeddings=[response["embedding"]],
  n_results=1
)
data = results['documents'][0][0]

# generate a response combining the prompt and data we retrieved in step 2
output = ollama.generate(
  model="llama3",
  prompt=f"Using this data: {data}. Respond to this prompt: {prompt}"
)

print(output['response'])

2. How to use LLM + RAG to Construct Knowledge Graph..

Video Tutorial.

Code to Generate Knowledge Graph Triplets.

import ollama
import chromadb

documents = [
  "Quantum mechanics is a fundamental theory in physics that describes the behavior of nature at and below the scale of atoms.",
  "It is the foundation of all quantum physics, which includes quantum chemistry, quantum field theory, quantum technology, and quantum information science.",
  "Quantum mechanics can describe many systems that classical physics cannot.",
  "Classical physics can describe many aspects of nature at an ordinary (macroscopic and (optical) microscopic) scale, but is not sufficient for describing them at very small submicroscopic (atomic and subatomic) scales.",
  "Most theories in classical physics can be derived from quantum mechanics as an approximation valid at large (macroscopic/microscopic) scale.",
  "Quantum systems have bound states that are quantized to discrete values of energy, momentum, angular momentum, and other quantities, in contrast to classical systems where these quantities can be measured continuously.",
  "Measurements of quantum systems show characteristics of both particles and waves (wave–particle duality), and there are limits to how accurately the value of a physical quantity can be predicted prior to its measurement, given a complete set of initial conditions (the uncertainty principle)."
]

# Create database
client = chromadb.Client()
collection = client.create_collection(name="docs")

# store each document in a vector embedding database
for i, d in enumerate(documents):
  response = ollama.embeddings(model="mxbai-embed-large", prompt=d)
  embedding = response["embedding"]
  collection.add(ids=[str(i)], embeddings=[embedding], documents=[d] )

# an example prompt
prompt1 = "What are the key benefits of using quantum mechanics over classical physics?"
prompt2 = "List all entities, and generate the knowledge graph triplets by using all entities."
# Generate Answers - for Prompt-1:
# generate an embedding for the prompt and retrieve the most relevant doc
response1 = ollama.embeddings(
  prompt=prompt1,
  model="mxbai-embed-large"
)
results1 = collection.query(
  query_embeddings=[response1["embedding"]],
  n_results=1
)
data1 = results1['documents'][0][0]

# generate a response combining the prompt and data we retrieved in step 2
output1 = ollama.generate(
  model="llama3",
  prompt=f"Using this data: {data1}. Respond to this prompt: {prompt1}"
)
print("Response for the question -1",output1['response'])
# Generate Answers - for Prompt-2:
# generate an embedding for the prompt and retrieve the most relevant doc
response2 = ollama.embeddings(
  prompt=prompt2,
  model="mxbai-embed-large"
)
results2 = collection.query(
  query_embeddings=[response2["embedding"]],
  n_results=1
)
data2 = results2['documents'][0][0]

# generate a response combining the prompt and data we retrieved in step 2
output2 = ollama.generate(
  model="llama3",
  prompt=f"Using this data: {data2}. Respond to this prompt: {prompt2}"
)
print("Response for the question -2",output2['response'])

Code to Visualize the Knowledge Graph (by using above triplets).

import networkx as nx
import matplotlib.pyplot as plt

# Step 1: Define your triplets
triplets = [
    ("Quantum mechanics", "a fundamental theory", "Theory"),
    ("Theory", "in physics", "Physics"),
    ("Physics", "describes the behavior of", "Nature"),
    ("Nature", "at and below the scale of", "Atoms"),
    ("Atoms", "is related to the scale of", "Scale"),
]

# Step 2: Create a directed graph
G = nx.DiGraph()

# Step 3: Add edges from triplets
for subject, predicate, obj in triplets:
    G.add_edge(subject, obj, label=predicate)

# Step 4: Draw the graph
pos = nx.spring_layout(G, seed=42)  # Position nodes using Fruchterman-Reingold force-directed algorithm

# Draw nodes and edges
nx.draw(G, pos, with_labels=True, node_size=3000, node_color="lightblue", font_size=10, font_weight="bold", arrowsize=20)

# Draw edge labels
edge_labels = nx.get_edge_attributes(G, 'label')
nx.draw_networkx_edge_labels(G, pos, edge_labels=edge_labels, font_color='red')

# Display the graph
plt.title("Knowledge Graph Visualization")
plt.show()

3. How to construct Flow-Diagram by Using LLM + RAG.

Video Tutorial.

Code.

import ollama
import chromadb

documents = [
  "Quantum mechanics is a fundamental theory in physics that describes the behavior of nature at and below the scale of atoms.",
  "It is the foundation of all quantum physics, which includes quantum chemistry, quantum field theory, quantum technology, and quantum information science.",
  "Quantum mechanics can describe many systems that classical physics cannot.",
  "Classical physics can describe many aspects of nature at an ordinary (macroscopic and (optical) microscopic) scale, but is not sufficient for describing them at very small submicroscopic (atomic and subatomic) scales.",
  "Most theories in classical physics can be derived from quantum mechanics as an approximation valid at large (macroscopic/microscopic) scale.",
  "Quantum systems have bound states that are quantized to discrete values of energy, momentum, angular momentum, and other quantities, in contrast to classical systems where these quantities can be measured continuously.",
  "Measurements of quantum systems show characteristics of both particles and waves (wave–particle duality), and there are limits to how accurately the value of a physical quantity can be predicted prior to its measurement, given a complete set of initial conditions (the uncertainty principle)."
]
# Create database
client = chromadb.Client()
collection = client.create_collection(name="docs")

# store each document in a vector embedding database
for i, d in enumerate(documents):
  response = ollama.embeddings(model="mxbai-embed-large", prompt=d)
  embedding = response["embedding"]
  collection.add(ids=[str(i)], embeddings=[embedding], documents=[d] )

# an example prompt
prompt1 = "Generate a Mermaid diagram."
# Generate Answers - for Prompt-1:
# generate an embedding for the prompt and retrieve the most relevant doc
response1 = ollama.embeddings(
  prompt=prompt1,
  model="mxbai-embed-large"
)
results1 = collection.query(
  query_embeddings=[response1["embedding"]],
  n_results=1
)
data1 = results1['documents'][0][0]

# generate a response combining the prompt and data we retrieved in step 2
output1 = ollama.generate(
  model="llama3",
  prompt=f"Using this data: {data1}. Respond to this prompt: {prompt1}"
)
print("Response for the question -1",output1['response'])

4. Graph Based RAG (Retrieval Augmented Generation) Techniques.

Video Tutorial.

Code.

import ollama
import chromadb

documents = [
  "The use of retrieval-augmented generation (RAG) to retrieve relevant information from an external knowledge source enables large language models (LLMs) to answer questions over private and/or previously unseen document collections.",
  "However, RAG fails on global questions directed at an entire text corpus, such as “What are the main themes in the dataset?”, since this is inherently a queryfocused summarization (QFS) task, rather than an explicit retrieval task.",
  "Prior QFS methods, meanwhile, fail to scale to the quantities of text indexed by typical RAG systems.",
  "To combine the strengths of these contrasting methods, we propose a Graph RAG approach to question answering over private text corpora that scales with both the generality of user questions and the quantity of source text to be indexed.",
  "Our approach uses an LLM to build a graph-based text index in two stages: first to derive an entity knowledge graph from the source documents, then to pregenerate community summaries for all groups of closely-related entities.",
  "Given a question, each community summary is used to generate a partial response, before all partial responses are again summarized in a final response to the user.",
  "For a class of global sensemaking questions over datasets in the 1 million token range, we show that Graph RAG leads to substantial improvements over a na¨ıve RAG baseline for both the comprehensiveness and diversity of generated answers."
]
# Create database
client = chromadb.Client()
collection = client.create_collection(name="docs")
# store each document in a vector embedding database
for i, d in enumerate(documents):
  response = ollama.embeddings(model="mxbai-embed-large", prompt=d)
  embedding = response["embedding"]
  collection.add(
    ids=[str(i)],
    embeddings=[embedding],
    documents=[d]
  )

# an example prompt
prompt = "How Graph RAG (Retrieval Augmented Generation, used with Large Language Model) generates a Global Summarization for the given context?"
Entity_1 = "Graph RAG"
Entity_2 = "Global Summarization"
Community_Triplets = [("Graph RAG", "Uses", "Leiden Community Detection Algorithm"),
("LLM", "Extracts", "Entity Knowledge Graph"),
("Graph Index", "Partitioned By", "Community Detection Algorithms"),
("Community Summaries", "Used For", "Global Summarization"),]
summary_text = "Graph RAG - Uses - Leiden Community Detection Algorithm; LLM - Extracts - Entity Knowledge Graph; Graph Index - Partitioned By - Community Detection Algorithms; Community Summaries - Used For - Global Summarization"
# generate an embedding for the prompt and retrieve the most relevant doc
response = ollama.embeddings(
  prompt=prompt,
  model="mxbai-embed-large"
)
summary_text_embedding = ollama.embeddings(
  prompt=summary_text,
  model="mxbai-embed-large"
)
results = collection.query(
  query_embeddings=[summary_text_embedding["embedding"]],
  n_results=1
)
data = results['documents'][0][0]

# generate a response combining the prompt and data we retrieved in step 2
output = ollama.generate(
  model="llama3",
  prompt=f"Using this data: {data}. Respond to this prompt: {prompt}"
)

print(output['response'])

Additional Reference for the topic. 

".Use of Long Text Sequences with LLMs Trained on Shorter Text Sequences." (Or use the direct link: https://www.nirajai.com/home/llm )

Reference.

1. Ding, Yujuan, Wenqi Fan, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. "A Survey on RAG Meets LLMs: Towards Retrieval-Augmented Large Language Models." arXiv preprint arXiv:2405.06211 (2024).
2. Wu, Kevin, Eric Wu, and James Zou. "How faithful are RAG models? Quantifying the tug-of-war between RAG and LLMs' internal prior." arXiv preprint arXiv:2404.10198 (2024).
3. Li, Jiarui, Ye Yuan, and Zehua Zhang. "Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases." arXiv preprint arXiv:2403.10446 (2024).
4. Edge, Darren, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. "From Local to Global: A Graph RAG Approach to Query-Focused Summarization." arXiv preprint arXiv:2404.16130 (2024).

Wasserstein Generative Adversarial Networks (WGANs) - An Easy Tutorial.

Introduction.

The unbeatable aspects of Wasserstein Generative Adversarial Networks (WGANs) come from their significant improvements over traditional GAN architectures. They address critical challenges like mode collapse and enhance the generation of high-quality, diverse samples. The following are the key technical advancements in WGAN architecture that motivate me to create tutorials on WGAN:

1. Wasserstein Distance: Shifts from traditional metrics to the Wasserstein distance for more meaningful training gradients, reducing mode collapse and stabilizing network convergence.
2. Weight Clipping and Lipschitz Constraint: Initially, WGANs used weight clipping to meet the Lipschitz constraint for the Wasserstein distance, but this approach had drawbacks like capacity underuse and gradient problems. The WGAN-GP variant introduced a gradient penalty to overcome these issues, leading to better training stability and sample quality.
3. Gradient Penalty (WGAN-GP): Incorporates a gradient penalty in the loss function, promoting stable training and high-quality output by preventing excessive critic gradients.
4. Critic Role: Unlike traditional GANs' discriminators, WGAN critics assess generated sample quality on a continuous scale, enabling finer quality evaluation and aiding in model training dynamics.
5. Training Protocol: WGANs employ a distinct training method, often involving more frequent training of the critic than the generator to provide effective gradients, ensuring balanced learning and model stability.

     These advancements make WGANs superior for generating realistic samples and ensuring smoother model training, maintaining their unique position in AI research and development.

Video Tutorials.

Part-1

Part-2

Part-3

Code - Training WGAN

# example of training a wgan on mnist
from numpy import expand_dims
import keras
import keras.backend as K
import tensorflow as tf
import numpy as np
from keras import Model
from keras.optimizers import Adam
from keras.layers import Input, Reshape, Flatten
from keras.layers import Dense, BatchNormalization, Conv2D, Conv2DTranspose, LeakyReLU, Dropout
batch_size = 32
input_shape = (28, 28, 1)
latent_dim = 100
img_shape = (28, 28, 1)
class WGAN_1:
  def __init__(self):
    print("welcome to WGAN coding")
  # write code for wasserstein loss.
  def wasserstein_loss(self, y_true, y_pred):
      return K.mean(y_true * y_pred)

  def preprocess_real_part_training_dataset(self):
    # load mnist dataset
    (dataX, dataY), (testDX, testDY) = keras.datasets.fashion_mnist.load_data()
    # Select the first 1000 rows of training data and labels
    dataX = dataX[:1000]
    dataY = dataY[:1000]
    # Add an additional dimension for the grayscale channel by using  expand_dims() from NumPy
    dataX = expand_dims(dataX, axis=-1)
    # convert from unsigned ints to floats and scale from [0,255] to [0,1]
    dataX = dataX.astype(np.float32) / 255.0
    return dataX

  # latent_dim = 100
  # img_shape = (28, 28, 1)
  def define_generator(self, latent_dim, img_shape):
      inputs = Input(shape=latent_dim)
      # Project and reshape the input
      proj = Dense(128 * 7 * 7)(inputs)
      proj = Reshape((7, 7, 128))(proj)
      # Upsample to 14x14
      upsample_1 = Conv2DTranspose(filters=128, kernel_size=4, strides=2, padding='same', activation=LeakyReLU(alpha=0.2),)(proj)
      upsample_1 = BatchNormalization()(upsample_1)
      # Upsample to 28x28
      upsample_2 = Conv2DTranspose(filters=128, kernel_size=4, strides=2, padding='same', activation=LeakyReLU(alpha=0.2),)(upsample_1)
      upsample_2 = BatchNormalization()(upsample_2)
      # Generate output image (28x28x1)
      gen_output = Conv2D(filters=img_shape[2], kernel_size=7, activation='tanh', padding='same')(upsample_2)
      g_model = Model(inputs, gen_output)
      g_model.summary()
      # keras.utils.plot_model(g_model, to_file="g_model.png", show_shapes=True)
      return g_model

  # input_shape = (28, 28, 1)
  def define_critic(self, input_shape):
      inputs = Input(shape=input_shape)
      # convolution layers
      conv1 = Conv2D(filters=64, kernel_size=3, strides=2, activation=LeakyReLU(alpha=0.2), padding='same')(inputs)
      conv1 = Dropout(0.4)(conv1)
      conv1 = Conv2D(filters=128, kernel_size=3, strides=2, activation=LeakyReLU(alpha=0.2), padding='same')(conv1)
      conv1 = Dropout(0.4)(conv1)
      conv1 = Conv2D(filters=256, kernel_size=3, strides=2, activation=LeakyReLU(alpha=0.2), padding='same')(conv1)
      conv1 = Dropout(0.4)(conv1)
      # Flatten Layer
      flatten_layer = Flatten()(conv1)
      critic_decision_layer = Dense(1)(flatten_layer)
      critic_model = Model(inputs, critic_decision_layer)
      # compile model
      optimizer = keras.optimizers.RMSprop(learning_rate=0.00005)
      critic_model.compile(loss=self.wasserstein_loss, optimizer=optimizer, metrics=['accuracy'])
      critic_model.summary()
      # keras.utils.plot_model(critic_model, to_file="critic_model.png", show_shapes=True)
      return critic_model

  def define_wgan(self,latent_dim0, img_shape0):
      # Define the input for the generator
      latent_input = Input(shape=(latent_dim0,))
      # Build the generator
      generator_output = self.define_generator(latent_dim=latent_dim0,img_shape=img_shape0)(latent_input)
      # Build the critic
      critic_input = Input(shape=img_shape0)
      critic_output = self.define_critic(input_shape=img_shape0)(critic_input)
      # Compile the critic
      critic = Model(critic_input, critic_output)
      critic.compile(loss=self.wasserstein_loss, optimizer=Adam(lr=0.0002, beta_1=0.5))
      # Make the critic not trainable
      critic.trainable = False
      # Combine the generator and critic
      gan_output = critic(generator_output)
      wgan_model = Model(latent_input, gan_output)
      # Compile the GAN
      wgan_model.compile(loss=self.wasserstein_loss, optimizer="adam")
      wgan_model.summary()
      # keras.utils.plot_model(wgan_model, to_file="wgan_model.png", show_shapes=True)
      return wgan_model

  def train_save_models(self, clip_value, n_critic, batch_size, input_shape, latent_dim, img_shape, n_epochs=2):
      # manually enumerate epochs
      trainX = self.preprocess_real_part_training_dataset()
      g_model = self.define_generator(latent_dim=latent_dim,img_shape=img_shape)
      critic_model = self.define_critic(input_shape)
      wgan_main = self.define_wgan(latent_dim0=latent_dim,img_shape0=img_shape)
      realY = -tf.ones(shape=(batch_size, 1))
      fakeY = tf.ones(shape=(batch_size, 1))
      for i in range(n_epochs):
          for j in range(len(trainX) // batch_size):
              # generate  random noise as an input  to  initialize the  generator
              noise = tf.random.normal(shape=[batch_size, latent_dim], mean=0, stddev=1)
              for _ in range(n_critic):
                  critic_model.trainable=True
                  # Real samples
                  X_real = trainX[j * batch_size : (j + 1) * batch_size]
                  Y_real = realY
                  d_loss_real = critic_model.train_on_batch(x = X_real,y = Y_real)
                  # fake samples
                  X_fake = g_model.predict_on_batch(noise)
                  Y_fake = fakeY
                  d_loss_fake = critic_model.train_on_batch(x = X_fake, y = Y_fake)
                  # Clip critic weights
                  for l in critic_model.layers:
                      weights = l.get_weights()
                      weights = [np.clip(w, (1-clip_value), clip_value) for w in weights]
                      l.set_weights(weights)
              # Train Generator weights
              critic_model.trainable = False
              g_loss_batch = wgan_main.train_on_batch(x=noise, y=realY)
              print("epoch = ",i,"//",n_epochs," batch = ", j," G_loss_batch ", g_loss_batch)

      g_model.save("g_model.h5")
      critic_model.save("critic_model.h5")
      wgan_main.save("wgan_model.h5")

if __name__ == "__main__":
    print ("Executed when invoked directly")
    input_shape1 = (28, 28, 1)
    img_shape1 = (28, 28, 1)
    latent_dim1 = 100
    n_critic = 5
    clip_value = 0.01
    # Create some dog objects
    wgan1 = WGAN_1()
    critic_model = wgan1.define_critic(input_shape=img_shape1)
    g_model = wgan1.define_generator(latent_dim=latent_dim1, img_shape=img_shape1)
    gan_model = wgan1.define_wgan(latent_dim0=latent_dim1,img_shape0=img_shape1)
    wgan1.train_save_models(n_critic=n_critic, clip_value=clip_value,batch_size=32, input_shape=input_shape1,latent_dim=latent_dim1, img_shape=img_shape1,n_epochs=2)

Code - Testing Generator Model.

# example of loading the generator model and generating images
import numpy as np
from keras.models import load_model
from numpy.random import randn
from keras.models import load_model
from matplotlib import pyplot
import matplotlib.pyplot as plt
# load model
model = load_model('g_model.h5')
# Generate synthetic images
num_images = 10
latent_dim = 100
noise = np.random.normal(0, 1, (num_images, latent_dim))
generated_images = model.predict(noise)

# Plot the generated images
plt.figure(figsize=(10, 10))
for i in range(num_images):
    plt.subplot(1, num_images, i+1)
    plt.imshow(generated_images[i, :, :, 0], cmap='gray')
    plt.axis('off')
plt.show()

Reference:

1. Wasserstein GAN; Martin Arjovsky (Courant Institute of Mathematical Sciences), Soumith Chintala, and Leon Bottou1 (Facebook AI Research) 

2. Ti, Yu. "Gradient Penalty Approach for Wasserstein Generative Adversarial Networks."

3. Kwon, Dohyun, Yeoneung Kim, Guido Montúfar, and Insoon Yang. "Training Wasserstein GANs without gradient penalties." arXiv preprint arXiv:2110.14150 (2021).

4. Guo, Xin, Johnny Hong, Tianyi Lin, and Nan Yang. "Relaxed Wasserstein with applications to GANs." In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3325-3329. IEEE, 2021.

Use of Long Text Sequences with LLM’s Trained on Shorter Text Sequences - ALiBi & RoFORMER

 

Introduction.

Training large language models (LLMs) on longer sequences poses challenges in computational resources, model complexity, gradient propagation, and overfitting. These include increased memory requirements due to self-attention mechanisms, longer training times, difficulty in scaling Transformers for very long sequences, challenges in capturing long-term dependencies, risk of vanishing or exploding gradients, and potential overfitting to training data. Solutions like linear biases, RoFormer, and RoPE improve handling of long-range dependencies, enhance model generalization, and incorporate positional information for better performance in NLP tasks. For Example:

Attention with linear Biases

Improved Handling of Long-Range Dependencies. Traditional attention mechanisms struggle with capturing long-range dependencies in text due to the quadratic increase in computational complexity with sequence length. Linear biases help to mitigate this by effectively incorporating positional information, thus enhancing the model’s ability to maintain context over long distances within the text. 

RoFormer

Improved Model Generalization: By more effectively encoding positional information, RoFormer helps LLMs to generalize better across different tasks and datasets. This results in enhanced performance on a wide range of NLP tasks, including text classification, machine translation, and semantic analysis. 

Enhanced Positional Encoding: RoPE uniquely integrates positional information with the token embeddings, preserving the relative distances between tokens. This method enables the model to better understand and utilize the order of words or tokens, which is crucial for many language understanding and generation tasks.

Video Tutorial -1

Video Tutorial -2

Video Tutorial -3

References.

1. Su, Jianlin, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. "Roformer: Enhanced transformer with rotary position embedding." Neurocomputing 568 (2024): 127063.
2. Press, Ofir, Noah A. Smith, and Mike Lewis. "Train short, test long: Attention with linear biases enables input length extrapolation." arXiv preprint arXiv:2108.12409 (2021).
3. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." Advances in neural information processing systems 30 (2017).

 

Generative Adversarial Network (GAN) , DCGAN, Tutorial and Keras Implementation

 Introduction.

GANs, called Generative Adversarial Networks, are special types of deep learning models that have two main parts: a generator and a discriminator. The generator creates fake data, and the discriminator checks if this data looks real or not compared to real data. By training against each other, GANs get better at making data that looks real, changing how we make new images, expand datasets, and learn without supervision. The following points show the significance of GAN in the area of AI.

• Creative Applications: GANs create realistic images, music, text, and videos, enabling creativity in art, content, and virtual environments.
• Data Augmentation: GANs generate synthetic data to enhance small datasets, improving the performance of machine learning models.
• Defense Against Deepfakes: GANs are used to develop defenses and detect manipulated media content amidst the growth of deepfake technology.
• Drug Discovery and Molecular Design: GANs play an increasing role in drug discovery, producing novel molecular structures with desired properties, potentially transforming the pharmaceutical industry.

Scope.

This article comprises interactive video tutorials and code demonstrations to elucidate the GAN architecture. The discussion covers various topics, culminating in the presentation of straightforwardly designed code.

GAN Part-1

GAN Part-2

GAN Part-3.

Keras implementation of GAN 

The following contains the Kera implementation of Deep Convolution GAN (DCGAN). Please go through the above video tutorials, to properly understand and use the code. 

The system is built using Python 3.10 and relies on several essential library dependencies:

• Tensorflow (version 2.15)
• tqdm (version 4.66.2)
• h5py (version 3.10)
• Keras (version 2.115)

Train DCGAN.

# example of training a gan on mnist
from numpy import expand_dims
from tqdm import tqdm
import keras
import tensorflow as tf
import numpy as np
from keras import Model
from keras.optimizers import Adam
from keras.layers import Input, Reshape, Flatten
from keras.layers import Dense, BatchNormalization, Conv2D, Conv2DTranspose, LeakyReLU, Dropout
batch_size = 32
input_shape = (28, 28, 1)
latent_dim = 100
img_shape = (28, 28, 1)
class GAN_1:
  def __init__(self):
    print("welcome to GAN coding")
  # This code prepares a TensorFlow dataset for training by shuffling the data, batching it into
  # consistent batch sizes, and prefetching batches to optimize data loading during training.
  def preprocess_real_part_training_dataset(self, batch_size):
    # load mnist dataset
    (dataX, dataY), (testDX, testDY) = keras.datasets.fashion_mnist.load_data()
    # Add an additional dimension for the grayscale channel by using  expand_dims() from NumPy
    dataX = expand_dims(dataX, axis=-1)
    # convert from unsigned ints to floats and scale from [0,255] to [0,1]
    dataX = dataX.astype(np.float32) / 255.0
    # testDX = testDX.astype(np.float32) / 255.0
    trainX = tf.data.Dataset.from_tensor_slices(dataX).shuffle(1000)
    # Combines consecutive elements of this dataset into batches.
    trainX = trainX.batch(batch_size, drop_remainder=True).prefetch(1)
    return trainX

  # latent_dim = 100
  # img_shape = (28, 28, 1)
  def define_generator(self, latent_dim, img_shape):
      inputs = Input(shape=latent_dim)
      # Project and reshape the input
      proj = Dense(128 * 7 * 7)(inputs)
      proj = Reshape((7, 7, 128))(proj)
      # Upsample to 14x14
      upsample_1 = Conv2DTranspose(filters=128, kernel_size=4, strides=2, padding='same', activation=LeakyReLU(alpha=0.2),)(proj)
      upsample_1 = BatchNormalization()(upsample_1)
      # Upsample to 28x28
      upsample_2 = Conv2DTranspose(filters=128, kernel_size=4, strides=2, padding='same', activation=LeakyReLU(alpha=0.2),)(upsample_1)
      upsample_2 = BatchNormalization()(upsample_2)
      # Generate output image (28x28x1)
      gen_output = Conv2D(filters=img_shape[2], kernel_size=7, activation='sigmoid', padding='same')(upsample_2)
      g_model = Model(inputs, gen_output)
      # compile model
      g_model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.0002, beta_1=0.5), metrics=['accuracy'])
      g_model.summary()
      return g_model

  # input_shape = (28, 28, 1)
  def define_descriminator(self, input_shape):
      inputs = Input(shape=input_shape)
      # convolution layers
      conv1 = Conv2D(filters=64, kernel_size=3, strides=2, activation=LeakyReLU(alpha=0.2), padding='same')(inputs)
      conv1 = Dropout(0.4)(conv1)
      conv1 = Conv2D(filters=128, kernel_size=3, strides=2, activation=LeakyReLU(alpha=0.2), padding='same')(conv1)
      conv1 = Dropout(0.4)(conv1)
      conv1 = Conv2D(filters=256, kernel_size=3, strides=2, activation=LeakyReLU(alpha=0.2), padding='same')(conv1)
      conv1 = Dropout(0.4)(conv1)
      # Flatten Layer
      flatten_layer = Flatten()(conv1)
      discriminator_decision_layer = Dense(1, activation='sigmoid')(flatten_layer)
      d_model = Model(inputs, discriminator_decision_layer)
      # compile model
      d_model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.0002, beta_1=0.5), metrics=['accuracy'])
      d_model.summary()
      return d_model

  def define_gan(self,latent_dim0, img_shape0):
      # Define the input for the generator
      latent_input = Input(shape=(latent_dim0,))
      # Build the generator
      generator_output = self.define_generator(latent_dim=latent_dim0,img_shape=img_shape0)(latent_input)
      # Build the discriminator
      discriminator_input = Input(shape=img_shape0)
      discriminator_output = self.define_descriminator(input_shape=img_shape0)(discriminator_input)
      # Compile the discriminator
      discriminator = Model(discriminator_input, discriminator_output)
      discriminator.compile(loss="binary_crossentropy", optimizer=Adam(lr=0.0002, beta_1=0.5))
      # Make the discriminator not trainable
      discriminator.trainable = False
      # Combine the generator and discriminator
      gan_output = discriminator(generator_output)
      gan_model = Model(latent_input, gan_output)
      # Compile the GAN
      gan_model.compile(loss="binary_crossentropy", optimizer="adam")
      gan_model.summary()
      return gan_model

  def train_save_models(self, input_shape, latent_dim, img_shape, n_epochs=2, n_batch=256):
      # manually enumerate epochs
      g_model = self.define_generator(latent_dim=latent_dim,img_shape=img_shape)
      d_model = self.define_descriminator(input_shape)
      gan_main = self.define_gan(latent_dim0=latent_dim,img_shape0=img_shape)
      for i in tqdm(range(n_epochs)):
          print()
          print("Epoch {}/{}".format(i + 1, n_epochs))
          # enumerate batches over the training set
          for X_batch in trainX:
              # generate  random noise as an input  to  initialize the  generator
              noise = tf.random.normal(shape=[batch_size, latent_dim])
              generated_images = g_model(noise)
              # print("shape of noise => ",np.shape(noise))
              X_fake_and_real = tf.concat([generated_images, X_batch], axis=0)
              y1 = tf.constant([[0.]] * batch_size + [[1.]] * batch_size)
              d_loss = d_model.train_on_batch(x=X_fake_and_real,y=y1)
              noise1 = tf.random.normal(shape=[batch_size, latent_dim])
              # print("shape of noise1 => ", np.shape(noise1))
              y2 = tf.constant([[1.]] * batch_size)
              gan_loss = gan_main.train_on_batch(noise1, y2)
              print("discriminator loss =>",d_loss, " Gan-Loss => ",gan_loss)
      g_model.save("g_model.h5")
      d_model.save("d_model.h5")
      gan_main.save("gan_model.h5")

if __name__ == "__main__":
    print ("Executed when invoked directly")
    input_shape1 = (28, 28, 1)
    img_shape1 = (28, 28, 1)
    latent_dim1 = 100
    # Create some dog objects
    gan1 = GAN_1()
    trainX = gan1.preprocess_real_part_training_dataset(batch_size=32)
    d_model = gan1.define_descriminator(input_shape=img_shape1)
    # visualkeras.layered_view(d_model)
    # visualkeras.layered_view(d_model, legend=True)
    g_model = gan1.define_generator(latent_dim=latent_dim1, img_shape=img_shape1)
    gan_model = gan1.define_gan(latent_dim0=latent_dim1,img_shape0=img_shape1)
    gan1.train_save_models(input_shape=input_shape1,latent_dim=latent_dim1, img_shape=img_shape1,n_epochs=50,n_batch=32)

Test the trained GAN model.

# example of loading the generator model and generating images
import numpy as np
from keras.models import load_model
from numpy.random import randn
from keras.models import load_model
from matplotlib import pyplot
import matplotlib.pyplot as plt
# load model
model = load_model('g_model.h5')
# Generate synthetic images
num_images = 10
latent_dim = 100
noise = np.random.normal(0, 1, (num_images, latent_dim))
generated_images = model.predict(noise)

# Plot the generated images
plt.figure(figsize=(10, 10))
for i in range(num_images):
    plt.subplot(1, num_images, i+1)
    plt.imshow(generated_images[i, :, :, 0], cmap='gray')
    plt.axis('off')
plt.show()

NOTE: This code is not intended for any commercial use. It is created solely for simple educational purposes. 

Niraj Kumar

Basics of Multivariate and Multi-Step Time Series Forecasting using Keras

Introduction.

Multi-step time series forecasting involves predicting multiple future time steps in a time series sequence. Mathematically, let ${�}_{�}$ represent the value of the time series at time $�$. Multi-step forecasting aims to predict the future values of the time series over a horizon of $ℎ$ time steps. Therefore, the forecasted values can be represented as ${\widehat{�}}_{�+1},{\widehat{�}}_{�+2},\mathrm{.}\mathrm{.}\mathrm{.},{\widehat{�}}_{�+ℎ}$, where ${\widehat{�}}_{�+�}$ denotes the predicted value at time $�+�$ for $�=1,2,\mathrm{.}\mathrm{.}\mathrm{.},ℎ$.

Multi-variate time series forecasting involves predicting the future values of a time series using multiple input variables, where each variable can influence the target time series. Mathematically, let ${�}_{�}^{(1)},{�}_{�}^{(2)},\mathrm{.}\mathrm{.}\mathrm{.},{�}_{�}^{(�)}$ represent $�$ input variables at time $�$, and ${�}_{�}$ represent the target time series. The goal of multi-variate time series forecasting is to predict the future values of the target time series, ${\widehat{�}}_{�+1},{\widehat{�}}_{�+2},\mathrm{.}\mathrm{.}\mathrm{.},{\widehat{�}}_{�+ℎ}$, based on the input variables. This can be represented as a function $�$ such that ${\widehat{�}}_{�+�}=�({�}_{�+�}^{(1)},{�}_{�+�}^{(2)},\mathrm{.}\mathrm{.}\mathrm{.},{�}_{�+�}^{(�)})$ for $�=1,2,\mathrm{.}\mathrm{.}\mathrm{.},ℎ$.

This article contains the following topics with supported video tutorial and code.

• Multivariate Multi-Step Multi-Output Time series Forecasting
• Strategy to prepare dataset.
• Multivariate Single-Step Multi-Output Time series Forecasting
• Strategy to prepare dataset.
• Strategy for the Future Enhancements.

Part-1.

Part-2.

Code-1. [Single-Step Multi-Output]

from keras import Model
from keras.layers import Input, Dense, Bidirectional, LSTM, RepeatVector, TimeDistributed
from sklearn.preprocessing import MinMaxScaler
from numpy import array , hstack
import numpy as np

data_X1 = [21, 26, 31, 36, 41, 46, 51, 56, 61, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115]
data_X2 = [2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59]
data_X3 = [13, 17, 21, 25, 29, 33, 37, 41, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88]

data_X1 = np.array(data_X1)
data_X2 = np.array(data_X2)
data_X3 = np.array(data_X3)

# Step 1 : convert to [rows, columns] structure
data_X1 = data_X1.reshape((len(data_X1), 1))
data_X2 = data_X2.reshape((len(data_X2), 1))
data_X3 = data_X3.reshape((len(data_X3), 1))

print ("data_X1.shape" , data_X1.shape)
print ("data_X2.shape" , data_X2.shape)
print ("data_X3.shape" , data_X3.shape)

data_scalar = MinMaxScaler(feature_range=(0,1))
data_X1_scaled = data_scalar.fit_transform(data_X1)
data_X2_scaled = data_scalar.fit_transform(data_X2)
data_X3_scaled = data_scalar.fit_transform(data_X3)
# Step 3 : horizontally stack columns
dataset_stacked = hstack((data_X1_scaled, data_X2_scaled, data_X3_scaled))
dataset_stacked = np.array(dataset_stacked)
print ("dataset_stacked.shape" , dataset_stacked.shape)
print("final dataset => ",dataset_stacked)

# prepare the dataset
input_timesteps=3
input_features=3
output_timesteps=1
output_features=3
data_Y = []
data_X = []
for i in range(0, ((len(data_X1))-(input_timesteps+output_timesteps))):
   print("---------")
   tmpY2d = []
   for i_row in range(i,(i + output_timesteps)):
       tmpYr = []
       print(i_row)
       for i_col in range(0,output_features):
           tmpYr.append(dataset_stacked[i_row][i_col])
       tmpY2d.append(tmpYr)
   data_Y.append(tmpY2d)
   print("---------")
   tmpX2d = []
   for j_row in range((i+output_timesteps), (i + output_timesteps + input_timesteps)):
       tmpXr = []
       print(j_row)
       for j_col in range(0, input_features):
           tmpXr.append(dataset_stacked[j_row][j_col])

       tmpX2d.append(tmpXr)
   data_X.append(tmpX2d)

data_X = np.array(data_X)
data_Y = np.array(data_Y)
print("shape of input data ",data_X.shape)
print("input data => ",data_X)
print("shape of output data => ", data_Y.shape)
print("Output data => ",data_Y)

def define_model():
  # Define the Input data shape
  encoder_inputs = Input(shape=(input_timesteps, input_features))
  # Use single BiLSTM as Encoder
  # Here we can use bigger network also like one BiLSTM with return_sequences=True and
  # other BiLSTM with return_sequences=False
  # OR CNN, CNN+LSTM and so many
  encoder = Bidirectional(LSTM(units=16, return_sequences=True))(encoder_inputs)
  # Apply RepeatVector to get the result for multiple time steps (here our output_timesteps =2)
  # For this step Decoder operation starts
  # repeat_output = RepeatVector(output_timesteps)(encoder)
  decoder = Bidirectional(LSTM(units=16, return_sequences=False))(encoder)
  # Use TimeDistributed layer to get multiple Output features
  out = Dense(output_features)(decoder)
  # out = TimeDistributed(Dense(output_features))(decoder)
  model = Model(encoder_inputs, out)
  # Compile the model
  model.compile(loss='mae', optimizer='adam', metrics=['mae'])
  model.summary()
  return model

# Call the model
model = define_model()
# Fit the model
model.fit(data_X,data_Y,epochs=4,batch_size=2,verbose=1)
# Take a test data to test the working of the model
test_dataX = []
test_dataX.append(dataset_stacked[0])
test_dataX.append(dataset_stacked[1])
test_dataX.append(dataset_stacked[2])
test_dataX = np.array(test_dataX)
# Reshape the data into 3-D numpy array
test_dataX = np.reshape(test_dataX,(1,input_timesteps,input_features))
print("test dataset => ",test_dataX)
# Run for the prediction output
pred_output = model.predict(test_dataX)
print("prediction output => ",pred_output)
# invert the scaling to get forecast values
inverted_output = data_scalar.inverse_transform(pred_output)
print("obtained prediction => ",inverted_output)

Code-2. [Multi-Step and Multi-Output]

from keras import Model
from keras.layers import Input, Dense, Bidirectional, LSTM, RepeatVector, TimeDistributed
from sklearn.preprocessing import MinMaxScaler
from numpy import array , hstack
import numpy as np

data_X1 = [21, 26, 31, 36, 41, 46, 51, 56, 61, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 115]
data_X2 = [2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59]
data_X3 = [13, 17, 21, 25, 29, 33, 37, 41, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88]

data_X1 = np.array(data_X1)
data_X2 = np.array(data_X2)
data_X3 = np.array(data_X3)

# Step 1 : convert to [rows, columns] structure
data_X1 = data_X1.reshape((len(data_X1), 1))
data_X2 = data_X2.reshape((len(data_X2), 1))
data_X3 = data_X3.reshape((len(data_X3), 1))

print ("data_X1.shape" , data_X1.shape)
print ("data_X2.shape" , data_X2.shape)
print ("data_X3.shape" , data_X3.shape)
# Step 2: Do scaling
data_scalar = MinMaxScaler(feature_range=(0,1))
data_X1_scaled = data_scalar.fit_transform(data_X1)
data_X2_scaled = data_scalar.fit_transform(data_X2)
data_X3_scaled = data_scalar.fit_transform(data_X3)
# Step 3 : horizontally stack columns
dataset_stacked = hstack((data_X1_scaled, data_X2_scaled, data_X3_scaled))
dataset_stacked = np.array(dataset_stacked)
print ("dataset_stacked.shape" , dataset_stacked.shape)
print("final dataset => ",dataset_stacked)

# prepare the dataset
input_timesteps=3
input_features=3
output_timesteps=2
output_features=3
data_Y = []
data_X = []
for i in range(0, ((len(data_X1))-(input_timesteps+output_timesteps))):
   tmpY2d = []
   for i_row in range(i,(i + output_timesteps)):
       tmpYr = []
       for i_col in range(0,output_features):
           tmpYr.append(dataset_stacked[i_row][i_col])
       tmpY2d.append(tmpYr)
   data_Y.append(tmpY2d)
   tmpX2d = []
   for j_row in range((i+output_timesteps), (i + output_timesteps + input_timesteps)):
       tmpXr = []
       for j_col in range(0, input_features):
           tmpXr.append(dataset_stacked[j_row][j_col])
       tmpX2d.append(tmpXr)
   data_X.append(tmpX2d)

data_X = np.array(data_X)
data_Y = np.array(data_Y)
print("shape of input data ",data_X.shape)
print("input data => ",data_X)
print("shape of output data => ", data_Y.shape)
print("Output data => ",data_Y)

def define_model():
  # Define the Input data shape
  encoder_inputs = Input(shape=(input_timesteps, input_features))
  # Use single BiLSTM as Encoder
  # Here we can use bigger network also like one BiLSTM with return_sequences=True and
  # other BiLSTM with return_sequences=False
  # OR CNN, CNN+LSTM and so many
  encoder = Bidirectional(LSTM(units=16, return_sequences=False))(encoder_inputs)
  # Apply RepeatVector to get the result for multiple time steps (here our output_timesteps =2)
  # For this step Decoder operation starts
  repeat_output = RepeatVector(output_timesteps)(encoder)
  decoder = Bidirectional(LSTM(units=16, return_sequences=True))(repeat_output)
  # Use TimeDistributed layer to get multiple Output features
  out = TimeDistributed(Dense(output_features))(decoder)
  model = Model(encoder_inputs, out)
  # Compile the model
  model.compile(loss='mae', optimizer='adam', metrics=['mae'])
  model.summary()
  return model

# Call the model
model = define_model()
# Fit the model
model.fit(data_X,data_Y,epochs=4,batch_size=2,verbose=1)
# Take a test data to test the working of the model
test_dataX = []
test_dataX.append(dataset_stacked[0])
test_dataX.append(dataset_stacked[1])
test_dataX.append(dataset_stacked[2])
test_dataX = np.array(test_dataX)
# Reshape the data into 3-D numpy array
test_dataX = np.reshape(test_dataX,(1,input_timesteps,input_features))
print("test dataset => ",test_dataX)
# Run for the prediction output
pred_output = model.predict(test_dataX)
print("prediction output => ",pred_output)
# invert the scaling to get forecast values
inverted_output = data_scalar.inverse_transform(pred_output[0])
print("obtained prediction => ",inverted_output)

Note. This code is not intended for any commercial use. It is created solely for simple educational purposes. 

Niraj Kumar