Evaluating ChatGPT as a Recommender System

The evaluation pipeline for assessing ChatGPT's performance as a recommender system.

Project Overview

This project provides the official source code and datasets accompanying the paper “Evaluating ChatGPT as a Recommender System: A Rigorous Approach.” It explores the potential of ChatGPT-3.5 and ChatGPT-4 models in zero-shot recommendation scenarios, offering a comprehensive framework for experimentation and evaluation.

Features

Zero-Shot Recommendations: Leverages ChatGPT models to generate recommendations without prior training on specific datasets.
Baseline Comparisons: Includes implementations of traditional recommender system baselines for performance benchmarking.
Extensive Datasets: Provides datasets necessary for conducting experiments, including MovieLens and Facebook Book datasets.
Evaluation Pipeline: Offers a structured pipeline to facilitate reproducible experiments and evaluations.

Usage

To set up and run the experiments:

Clone the Repository:

git clone https://github.com/sisinflab/Recommender-ChatGPT.git

Set Up the Environment: Ensure Python 3.8.6 or later is installed. Create a virtual environment and install dependencies:
```
conda env create -f conda_env/env.yml
conda activate Recommender-ChatGPT
```
Insert OpenAI API Token: Obtain an API token from OpenAI and insert it into the appropriate configuration file.
Run the Evaluation Pipeline: Execute the main script to start the experiments:
```
python code/main.py
```
Results will be stored in the results folder.

Dependencies

Python: Version 3.8.6 or later
Conda: For environment management
OpenAI API: Access to ChatGPT models
Elliot: Framework for recommender systems evaluation
Pandas: Data manipulation and analysis
NumPy: Numerical computing

Links

For detailed information and updates, please refer to the GitHub repository.