Modelling human preferences for ranking and collaborative filtering: a probabilistic ordered partition approach

Abstract

Learning preference models from human generated data is an important task in modern information processing systems. Its popular setting consists of simple input ratings, assigned with numerical values to indicate their relevancy with respect to a specific query. Since ratings are often specified within a small range, several objects may have the same ratings, thus creating ties among objects for a given query. Dealing with this phenomena presents a general problem of modelling preferences in the presence of ties and being query-specific. To this end, we present in this paper a novel approach by constructing probabilistic models directly on the collection of objects exploiting the combinatorial structure induced by the ties among them. The proposed probabilistic setting allows exploration of a super-exponential combinatorial state-space with unknown numbers of partitions and unknown order among them. Learning and inference in such a large state-space are challenging, and yet we present in this paper efficient algorithms to perform these tasks. Our approach exploits discrete choice theory, imposing generative process such that the finite set of objects is partitioned into subsets in a stagewise procedure, and thus reducing the state-space at each stage significantly. Efficient Markov chain Monte Carlo algorithms are then presented for the proposed models. We demonstrate that the model can potentially be trained in a large-scale setting of hundreds of thousands objects using an ordinary computer. In fact, in some special cases with appropriate model specification, our models can be learned in linear time. We evaluate the models on two application areas: (i) document ranking with the data from the Yahoo! challenge and (ii) collaborative filtering with movie data. We demonstrate that the models are competitive against state-of-the-arts.

Appendix

1.1 Computing $C_{k}$

We here calculate the constant $C_{k}$ in Eq. (11). Let us rewrite the equation for ease of comprehension

$$\begin{aligned} \sum _{S\in 2^{R_{k}}}\frac{1}{\left| S\right| }\sum _{x\in S}\phi _{k}\left( x\right) =C_{k}\times \sum _{x\in R_{k}}\phi _{k}(x) \end{aligned}$$

where $2^{R_{k}}$ is the power set with respect to the set $R_{k}$, or the set of all non-empty subsets of $R_{k}$. Equivalently

$$\begin{aligned} C_{k}=\sum _{S\in 2^{R_{k}}}\frac{1}{\left| S\right| }\sum _{x\in S}\frac{\phi _{k}\left( x\right) }{\sum _{x\in R_{k}}\phi _{k}(x)} \end{aligned}$$

If all objects are the same, then this can be simplified to

$$\begin{aligned} C_{k}= & {} \sum _{S\in 2^{R_{k}}}\frac{1}{\left| S\right| }\sum _{x\in S}\frac{1}{N_{k}}=\frac{1}{N_{k}}\sum _{S\in 2^{R_{k}}}1=\frac{2^{N_{k}}-1}{N_{k}} \end{aligned}$$

where $N_{k}=|R_{k}|$. In the last equation, we have made use of the fact that $\sum _{S\in 2^{R_{k}}}1$ is the number of all possible non-empty subsets, or equivalently, the size of the power set, which is known to be $2^{N_{k}}-1$. One way to derive this result is the imagine a collection of $N_{k}$ variables, each has two states: $\mathtt {selected}$ and $\mathtt {notselected}$, where $\mathtt {selected}$ means the object belongs to a subset. Since there are $2^{N_{k}}$ such configurations over all states, the number of non-empty subsets must be $2^{N_{k}}-1$.

For arbitrary objects, let us examine the probability that the object x belongs to a subset of size m, which is $\frac{m}{N_{k}}$. Recall from standard combinatorics that the number of m-element subsets is the binomial coefficient $\left( {{\begin{array}{ll} N_{k}\\ m\\ \end{array}}}\right)$, where $1\le m\le N_{k}$. Thus the number of times an object appears in any m-subset is $\left( {{\begin{array}{ll} N_{k}\\ m\\ \end{array}}}\right) \frac{m}{N_{k}}$. Taking into account that this number is weighted down by m (i.e. |S| in Eq. (11)), the contribution towards $C_{k}$ is then $\left( {{\begin{array}{ll} N_{k}\\ m\\ \end{array}}}\right) \frac{1}{N_{k}}$. Finally, we can compute the constant $C_{k}$, which is the weighted number of times an object belongs to any subset of any size, as follows

$$\begin{aligned} C_{k}= & {} \sum _{m=1}^{N_{k}} \left( {{\begin{array}{ll} N_{k}\\ m\\ \end{array}}}\right) \frac{1}{N_{k}}=\frac{1}{N_{k}}\sum _{m=1}^{N_{k}}\left( {{\begin{array}{ll} N_{k}\\ m\\ \end{array}}}\right) =\frac{2^{N_{k}}-1}{N_{k}} \end{aligned}$$

We have made use of the known identity $\sum _{m=1}^{N_{k}} \left( {{\begin{array}{ll} N_{k}\\ m\\ \end{array}}}\right) =2^{N_{k}}-1$.

1.2 Computing $M_{k}(x)$

We now calculate the constant $M_{k}(x)$ in Eq. (17), which is reproduced here for clarity:

$$\begin{aligned} \sum _{S \in 2^{R_{k}}}\max _{x\in S}\phi (x)=\sum _{x\in R_{k}}M_{k}(x)\phi (x) \end{aligned}$$

(27)

First, we arrange the objects in the decreasing order of worth $\phi (x)$. For notation convenience we assume that the order is $1,2,3,\ldots ,N_{k}$. The largest object will appear in a subset consisting of only itself, and $2^{N_{k}-1}-1$ other subsets. Thus $M_{k}(1)=2^{N_{k}-1}$ since for all subsets to which the largest object belong, the maximum aggregation is the worth of the object, as per definition. Now removing the largest object, consider the second largest one. With the same argument as before, $M_{k}(2)=2^{N_{k}-2}$. Continuing the same line of reasoning, we end up $M_{k}(n)=2^{N_{k}-n}$.

1.3 Pairwise losses

Let $f(x_{i},w)$ be the scoring function parameterised by w that takes the input vector $x_{i}$ and outputs a real value indicating the relevancy of the object i. Let $\delta _{ij}(w)=f(x_{i},w)-f(x_{j},w)$. Pairwise models are quite similar in their general setting. The only difference is the specific loss function:

$$\begin{aligned} \ell (x_{i}\succ x_{j};w)={\left\{ \begin{array}{ll} \log (1+\exp \left\{ -\delta _{ij}(w)\right\} ) &{} \hbox { (RankNet) }\\ \max \left\{ 0,1-\delta _{ij}(w)\right\} &{} \hbox { (Ranking SVM)}\\ (1-\delta _{ij}(w))^{2} &{} \hbox { (Rank Regress)}\\ \exp \left\{ -\delta _{ij}(w)\right\} &{} \hbox { (Rank Boost)} \end{array}\right. } \end{aligned}$$

However, these losses behave quite different from each other. For the RankNet and Rank Boost, minimising the loss would widen the margin between the score for $x_{i}$ and $x_{j}$ as much as possible. The difference is that the RankNet is less sensitive to noise due to the log-scale. The Ranking SVM, however, aims just about to achieve the margin of 1, and the Rank Regress, attempts to bound the margin by 1.

At the first sight, the cost for gradient evaluation in pairwise losses would be $\mathcal {O}(0.5N(N-1)F)$ where F is the number of parameters. However, we can achieve $\max \{\mathcal {O}(0.5N(N-1)),\mathcal {O}(NF)\}$ as follows. The overall loss for a particular query is

$$\begin{aligned} \mathfrak {L}= & {} \sum _{i,j|x_{i}\succ x_{j}}\ell (x_{i}\succ x_{j};w) \end{aligned}$$

Taking derivative with respect to w yields

$$\begin{aligned} \frac{\partial \mathfrak {L}}{\partial w}= & {} \sum _{i,j|x_{i}\succ x_{j}} \frac{\partial \ell (x_{i} \succ x_{j};w)}{\partial \delta _{ij}}\left( -\frac{\partial f_{i}}{\partial w}+\frac{\partial f_{j}}{\partial w}\right) \\= & {} -\sum _{i}\frac{\partial f_{i}}{\partial w}\sum _{j|x_{i}\succ x_{j}}\frac{\partial \ell (x_{i}\succ x_{j};w)}{\partial \delta _{ij}}+\sum _{j}\frac{\partial f_{j}}{\partial w}\sum _{i|x_{i}\succ x_{j}}\frac{\partial \ell (x_{i}\succ x_{j};w)}{\partial \delta _{ij}} \end{aligned}$$

As $\left\{ \frac{\partial \ell (x_{i}\succ x_{j};w)}{\partial \delta _{ij}}\right\} _{i,j|x_{i}\succ x_{j}}$ can be computed in $\mathcal {O}(0.5N(N-1))$ time, and $\left\{ \frac{\partial f_{i}}{\partial w}\right\} _{i}$ in $\mathcal {O}(NF)$ time, the overall cost would be $\max \{\mathcal {O}(0.5N(N-1)),\mathcal {O}(NF)\}$.

1.4 Learning the pairwise ties models

This subsection describes the details of learning the paired ties models discussed in Sect. 6.

1.4.1 Davidson method

Recall from Sect. 2 that in the Davidson method, the probability masses are defined as

$$\begin{aligned} P(x_{i}\succ x_{j};w)= & {} \frac{1}{Z_{ij}}\phi (x_{i});\quad P(x_{i}\sim x_{j};w)=\frac{1}{Z_{ij}}\nu \sqrt{\phi (x_{i})\phi (x_{j})} \end{aligned}$$

where $Z_{ij}=\phi (x_{i})+\phi (x_{j})+\nu \sqrt{\phi (x_{i})\phi (x_{j})}$ and $\nu \ge 0$. For simplicity of unconstrained optimisation, let $\nu =e^{\beta }$ for $\beta \in \mathbb {R}$. Let $P_{i}=P(x_{i}\succ x_{j};w)$, $P_{j}=P(x_{j}\succ x_{i};w)$ and $P_{ij}=P(x_{i}\sim x_{j};w)$.

Taking derivatives of the log-likelihood gives

$$\begin{aligned} \frac{\partial \log P(x_{i}\succ x_{j};w)}{\partial w}= & {} (1-P_{i}-0.5P_{ij})\frac{\partial \log \phi (x_{i},w)}{\partial w}-(P_{i}+0.5P_{ij})\frac{\partial \log \phi (x_{j},w)}{\partial w}\\ \frac{\partial \log P(x_{i}\succ x_{j};w)}{\partial \beta }= & {} -P_{ij}\\ \frac{\partial \log P(x_{i}\sim x_{j};w)}{\partial w}= & {} (0.5-P_{i}-0.5P_{ij})\frac{\partial \log \phi (x_{i},w)}{\partial w}\\&+(0.5-P_{j}-0.5P_{ij})\frac{\partial \log \phi (x_{j},w)}{\partial w}\\ \frac{\partial \log P(x_{i}\sim x_{j};w)}{\partial \beta }= & {} 1-P_{ij}. \end{aligned}$$

1.4.2 Rao-Kupper method

Recall from Sect. 2 that the Rao-Kupper model defines the following probability masses

$$\begin{aligned} P(x_{i}\succ x_{j};w)= & {} \frac{\phi (x_{i})}{\phi (x_{i})+\theta \phi (x_{j})}\\ P(x_{i}\sim x_{j};w)= & {} \frac{(\theta ^{2}-1)\phi (x_{i})\phi (x_{j})}{\left[ \phi (x_{i})+\theta \phi (x_{j})\right] \left[ \theta \phi (x_{i})+\phi (x_{j})\right] } \end{aligned}$$

where $\theta \ge 1$ is the ties factor and w is the model parameter. Note that $\phi (.)$ is also a function of w, which we omit here for clarity. For ease of unconstrained optimisation, let $\theta =1+e^{\alpha }$ for $\alpha \in \mathbb {R}$. In learning, we want to estimate both $\alpha$ and w. Let

$$\begin{aligned} P_{i}= & {} \frac{\phi (x_{i})}{\phi (x_{i})+(1+e^{\alpha })\phi (x_{j})};\quad P_{j}^{*}=\frac{\phi (x_{j})}{\phi (x_{i})+(1+e^{\alpha })\phi (x_{j})};\\ P_{i}^{*}= & {} \frac{\phi (x_{i})}{(1+e^{\alpha })\phi (x_{i})+\phi (x_{j})};\quad P_{j}=\frac{\phi (x_{j})}{(1+e^{\alpha })\phi (x_{i})+\phi (x_{j})}. \end{aligned}$$

Taking partial derivatives of the log-likelihood gives

$$\begin{aligned} \frac{\partial \log P(x_{i}\succ x_{j};w)}{\partial w}= & {} (1-P_{i})\frac{\partial \log \phi (x_{i},w)}{\partial w}-(1+e^{\alpha })P_{j}\frac{\partial \log \phi (x_{j},w)}{\partial w} \\ \frac{\partial \log P(x_{i}\succ x_{j};w)}{\partial \alpha }= & {} -P_{j}e^{\alpha } \\ \frac{\partial \log P(x_{i}\sim x_{j};w)}{\partial w}= & {} (1-P_{i}-(1+e^{\alpha })P_{i}^{*})\frac{\partial \log \phi (x_{i},w)}{\partial w} \\&+\, (1-P_{j}-(1+e^{\alpha })P_{j}^{*})\frac{\partial \log \phi (x_{j},w)}{\partial w} \\ \frac{\partial \log P(x_{i}\sim x_{j};w)}{\partial \alpha }= & {} \left( \frac{2(1+e^{\alpha })}{(1+e^{\alpha })^{2}-1}-P_{i}^{*}-P_{j}^{*}\right) e^{\alpha }. \end{aligned}$$