Demand Estimation and Off-Policy Evaluation for New CSIALife Services

Overview

CSIALife is CSIA's school-wide platform. Before adding new services to it, this study set out to estimate how much they would be used and which way of operating them would work best, without running live experiments on students.

Research Questions

• How much demand should CSIALife expect for seminar-room booking and for the on-site promotion system?
• If the platform changed how seats and seminar rooms are shown to students, would no-shows fall and utilization rise?

Data

• Usage records accumulated in the school's own booking system
• Each record organized by time slot, day of the week, resource type, and whether the booking was a no-show

Method

• Off-policy evaluation: estimating the outcome of an alternative display policy using only data logged under the current policy, with no live trial
• Statistical comparison of the operating policies, together with a check of the conditions under which the estimates can be trusted
• Thompson sampling simulation: a bandit that alternates between policies and gradually converges on the one that performs better

Evaluating a new policy without a live trial

1. 1

   Booking records

   Time slot, day of week, resource type, and no-show status from the school's booking system

2. 2

   Current policy

   Every record was logged under the way seats and seminar rooms are shown today

3. 3

   Off-policy evaluation

   Estimate how an alternative display policy would perform from those logs alone

4. 4

   Reliability check

   Test the conditions under which the estimates can be trusted

5. 5

   Thompson sampling

   A simulated bandit alternates policies and converges on the better one

Note

• The results are forecasts, not usage measured after launch.

Recognition

• School research Bronze Award