Reinforcement Learning for Information Retrieval

Authors

Alexander Kuhnle, Miguel Aroca-Ouellette, John Reid, Dell Zhang, Murat Sensoy

{alexander.kuhnle, miguel.aroca, john.reid, dell.zhang, murat.sensoy}[at]blueprism[dot]com

Description

There is strong interest in leveraging reinforcement learning (RL) for information retrieval (IR) applications including search, recommendation, and advertising. Just in SIGIR-2020, the term “reinforcement learning” is mentioned in 46 different papers. It has also been reported that Internet companies like YouTube and Alibaba have started to gain competitive advantages from their RL-based search and recommendation engines. This full-day tutorial gives IR researchers and practitioners who have little or no experience with RL the opportunity to learn about the fundamentals of modern RL in a practical hands-on setting. Furthermore, some representative applications of RL in IR systems will be introduced and discussed.

This tutorial has been part of the European Conference on Information Retrieval (ECIR) 2021 and Search Solutions 2020.

Material

The complete repository for this tutorial can be found at RL Starterpack.

Quick Links:

• RL Slides
• RL For IR Slides
• TQL Exercise
• DQN Exercise
• PG Exercise
• AC Exercise

Schedule

The ECIR 2021 tutorial is taking place on March 28th, 2021. All times below are in BST (UTC+1).

• 08:00-09:00 RL Basics and Tabular Q-Learning
• 09:00-09:30 Deep Q-Network (DQN) 1/2 (presentation)
• 09:30-10:00 Coffee Break
• 10:00-10:30 Deep Q-Network (DQN) 2/2 (hands-on)
• 10:30-11:15 IR Applications using DQN
• 11:15-13:15 Lunch Break (45 minutes extra)
• 13:15-14:15 Policy Gradient (REINFORCE)
• 14:15-15:15 IR Applications using REINFORCE
• 15:15-15:45 Coffee Break
• 15:45-16:15 Actor Critic
• 16:15-16:45 Outlook
• 16:45-17:15 Final Q&A

Contact

Questions? Feedback? Please reach out to one of the tutorial organisers using the emails listed at the top of this page.

Additional Resources

Introductory Material

• Spinning Up in Deep RL (OpenAI)
• Deep Reinforcement Learning Course (Thomas Simonini)
• Simple Reinforcement Learning with Tensorflow (Arthur Juliani)
• Deep Reinforcement Learning: Pong from Pixels (Andrej Karpathy)
• An Outsider’s Tour of Reinforcement Learning (Ben Recht)
• Lilian Weng:
  • A (Long) Peek into Reinforcement Learning
  • Policy Gradient Algorithms
  • Implementing Deep Reinforcement Learning Models with Tensorflow + OpenAI Gym
  • and more…

Critical Voices

• Deep Reinforcement Learning Doesn’t Work Yet (Alex Irpan)
• The Policy of Truth (Ben Recht)
• Lessons Learned Reproducing a Deep Reinforcement Learning Paper (Amid Fish)

Books

• Reinforcement Learning - An Introduction (Sutton & Barto, 2018)
• An Introduction to Deep Reinforcement Learning (Francois-Lavet et al., 30 Nov 2018)