Authors
Alexander Kuhnle, Miguel Aroca-Ouellette, Anindya Basu, Murat Sensoy, John Reid, Dell Zhang
{alexander.kuhnle, miguel.aroca, anindya.basu, murat.sensoy, john.reid, dell.zhang}[at]blueprism[dot]com
Description
There is strong interest in leveraging reinforcement learning (RL) for information retrieval (IR) applications including search, recommendation, and advertising. Just in 2020, the term “reinforcement learning” was mentioned in more than 60 different papers published by ACM SIGIR. It has also been reported that Internet companies like Google and Alibaba have started to gain competitive advantages from their RL-based search and recommendation engines. This full-day tutorial gives IR researchers and practitioners who have no or little experience with RL the opportunity to learn about the fundamentals of modern RL in a practical hands-on setting. Furthermore, some representative applications of RL in IR systems will be introduced and discussed. By attending this tutorial, the participants will acquire a good knowledge of modern RL concepts and standard algorithms such as REINFORCE and DQN. This knowledge will help them better understand some of the latest IR publications involving RL, as well as prepare them to tackle their own practical IR problems using RL techniques and tools.
This tutorial is featured in prestigious conferences SIGIR-2021, ECIR-2021 and SearchSolutions-2020.
Materials
The complete repository for this tutorial can be found at RL Starterpack.
Quick Links:
Presentation Slides
Hands-on Exercises
Schedule
As SIGIR 2021 is pre-recorded, once you have access to the tutorial video you can use the following timestamps to quickly navigate between sections:
- [0:00:00] Intro
- [0:05:18] RL Basics
- [0:14:50] Multi-armed Bandits
- [0:38:40] Tabular Q-Learning
- [1:00:37] Deep Q-Learning
- [1:15:23] IR applications using DQN
- [1:42:36] Policy Gradient (REINFORCE)
- [1:54:58] IR applications using REINFORCE
- [2:50:26] Actor-Critic
- [3:07:34] IR applications using Actor-Critic
- [3:21:09] Recent developments & outlook for research
Contact
Questions? Feedback? Please reach out to one of the tutorial organisers using the emails listed at the top of this page.
Additional Resources
Introductory Materials
- Spinning Up in Deep RL (OpenAI)
- Deep Reinforcement Learning Course (Thomas Simonini)
- Simple Reinforcement Learning with Tensorflow (Arthur Juliani)
- Deep Reinforcement Learning: Pong from Pixels (Andrej Karpathy)
- An Outsider’s Tour of Reinforcement Learning (Ben Recht)
- Lilian Weng:
Critical Voices
- Deep Reinforcement Learning Doesn’t Work Yet (Alex Irpan)
- The Policy of Truth (Ben Recht)
- Lessons Learned Reproducing a Deep Reinforcement Learning Paper (Amid Fish)
Books
- Reinforcement Learning - An Introduction (Sutton & Barto, 2018)
- An Introduction to Deep Reinforcement Learning (Francois-Lavet et al., 30 Nov 2018)
Papers
- DRN: A Deep Reinforcement Learning Framework for News Recommendation (WWW’18)
- Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning (KDD’18)
- Jointly Learning to Recommend and Advertise (KDD’20)
- Reinforcement Learning to Rank with Markov Decision Process (SIGIR’17)
- Adapting Markov Decision Process for Search Result Diversification (SIGIR’17)
- Multi Page Search with Reinforcement Learning to Rank (ICTIR’18)
- Reinforcement Learning to Rank with Pairwise Policy Gradient (SIGIR’20)
- A Reinforcement Learning Framework for Relevance Feedback (SIGIR’20)
- IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models (SIGIR’17)
- Top-K Off-Policy Correction for a REINFORCE Recommender System (WSDM’19)
- Off-policy Learning in Two-stage Recommender Systems (WWW’20)
- Reinforcement Recommendation with User Multi-aspect Preference (WWW’21)
- Self-Supervised Reinforcement Learning for Recommender Systems (SIGIR’20)