rahmaelnabarawy/gptneo125m-finetuned-generate-reviews

GPT-Neo 125M LoRA Fine-Tuned Review Generator

---

Model Overview

This model is a review-generation fine-tuned version of EleutherAI/gpt-neo-125M, trained using LoRA (Low-Rank Adaptation) on the Yelp Review Full dataset.

The model generates structured and natural-sounding restaurant and product reviews when prompted in a conversational format.

Fine-tuning was performed using parameter-efficient training, and LoRA weights were merged into the base model for standalone deployment.

---

Model Details

Model Description

This is a causal language model fine-tuned for review generation.

Expected prompt format:

User: Write a review about a new restaurant.
Assistant:

The model generates a realistic review continuation after Assistant:.

---

Model Type

Causal Language Model (GPT-Neo architecture) fine-tuned using LoRA

Language

English

---

Base Model

EleutherAI/gpt-neo-125M
https://huggingface.co/EleutherAI/gpt-neo-125M

---

Fine-Tuning Method

Low-Rank Adaptation (LoRA)

LoRA enables efficient fine-tuning by training a very small subset of parameters instead of updating the entire model.

After training:

• LoRA weights were merged into the base model
• The final model is standalone (no adapter required)

---

Intended Uses

Direct Use

• Synthetic review generation
• NLP experimentation
• Prompt engineering practice
• Educational purposes (fine-tuning & LoRA practice)

---

Example Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModelForCausalLM.from_pretrained(
    "rahmaelnabarawy/gptneo125m-finetuned-generate-reviews"
).to(device)

tokenizer = AutoTokenizer.from_pretrained(
    "rahmaelnabarawy/gptneo125m-finetuned-generate-reviews"
)

prompt = """User: Write a review about a nearby coffee shop.
Assistant:"""

inputs = tokenizer(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.8,
    top_p=0.95,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Training Details

Dataset

Yelp Review Full
https://huggingface.co/datasets/yelp_review_full

Training subset used:

• 30,000 training samples
• 2,000 validation samples

---

Data Format Used for Training

User: Write a review.
Assistant: <review text>

The conversational structure was intentionally used to guide generation behavior.

---

Training Hyperparameters

Parameter	Value
Base Model	GPT-Neo-125M
LoRA Rank (r)	16
LoRA Alpha	16
LoRA Dropout	0.05
Epochs	3
Batch Size	8
Gradient Accumulation	2
Learning Rate	2e-4
Block Size	128
Optimizer	AdamW
Precision	FP16 (if GPU available)

---

Architecture

GPT-Neo Transformer

• Layers: 12
• Hidden Size: 768
• Attention Heads: 12
• Parameters: 125M
• Objective: Causal Language Modeling

---

Compute Infrastructure

Hardware:

• NVIDIA Tesla T4 GPU (or similar)

Software:

• PyTorch
• Hugging Face Transformers
• PEFT
• Hugging Face Datasets

---

Limitations

• Small model size (125M parameters)
• May generate repetitive or generic text
• Sensitive to prompt format
• Inherits biases from Yelp dataset
• Not optimized for production-level deployment

---

Out-of-Scope Use

This model is NOT intended for:

• Real-world review manipulation
• Deceptive content generation
• High-stakes decision systems