Deep Neural Networks for YouTube Recommendations: A Paper Review

Deep Neural Networks for YouTube Recommendations

• Authors: Paul Covington, Jay Adams and Emre Sargin (Google)
• Venue: RecSys '16, the 10th ACM Conference on Recommender Systems, Boston, Massachusetts, USA, September 15 to 19, 2016, pages 191 to 198
• DOI: 10.1145/2959100.2959190
• Reviewed: January 15, 2017

This paper describes the system YouTube used to recommend videos in 2016 and the large gains the team saw after moving it to deep learning. At the time, YouTube recommendations helped more than a billion users find videos. The authors name three challenges:

1. Scale. Many recommendation algorithms that work well on small problems fail at YouTube's size.
2. Freshness. Many hours of video are uploaded every second. The system must respond quickly to new uploads and to each user's latest actions.
3. Noise. User behavior is sparse and hard to predict. The system rarely sees a direct measure of user satisfaction, so it learns from implicit signals such as watches.

System overview

The system is built on Google Brain, which Google had recently open sourced as TensorFlow. The models learn about one billion parameters and train on hundreds of billions of examples.

The system has two neural networks, one for candidate generation and one for ranking. Together they form a funnel. Candidate generation narrows a corpus of millions of videos down to a few hundred. Ranking then scores those candidates, and the user sees a few dozen of them.

The candidate generation network takes events from the user's YouTube activity history as input. It provides broad personalization through collaborative filtering. It measures similarity between users with coarse features such as the IDs of watched videos, search query tokens and demographics. The ranking network uses a richer set of features about the video and the user to give each candidate a score. The highest-scoring videos are shown to the user in order of score. The two-stage design also lets YouTube blend in candidates from other sources.

During development, the team used offline metrics such as precision, recall and ranking loss. For the final decision on a model, they relied on live A/B tests, because offline results did not always agree with live results.

Candidate generation

The paper treats recommendation as extreme multiclass classification. The task is to predict which video a user will watch at time t, out of millions of videos (the classes), given the user and their context. The network learns an embedding of the user and context, and a softmax classifier uses it to tell the videos apart. An embedding maps a sparse entity, such as a single video or user, to a dense vector of numbers. The model trains on implicit feedback. A user who completes a video gives a positive example.

A full softmax over millions of classes is slow to train. The authors sample several thousand negative classes for each example and correct for the sampling with importance weighting. For each example, they minimize the cross-entropy loss over the true label and the sampled negatives. This made training more than 100 times faster than a traditional softmax. The authors also tried hierarchical softmax and could not reach the same accuracy with it.

At serving time, the system must pick the top few hundred videos within tens of milliseconds. It does not need calibrated probabilities for this. The problem becomes an approximate nearest neighbor search between the user vector and the video vectors in dot product space.

Architecture. Each video in a fixed vocabulary has a learned embedding. A user's watch history is a variable-length sequence of sparse video IDs. The network averages the embeddings of these videos into one dense vector. Averaging worked better than other options such as summing or taking the component-wise maximum. Search history is handled the same way. Each query is split into unigrams and bigrams, and the token embeddings are averaged. The embeddings are learned jointly with all other model parameters through gradient descent and backpropagation. The averaged vectors are concatenated with other features and passed through several fully connected layers of rectified linear units (ReLU).

Other features. A deep network makes it easy to add other continuous and categorical features. The user's geographic region and device are embedded. Simple binary and continuous features, such as the user's gender, logged-in state and age, enter the network as real values normalized to [0, 1]. Demographic features help the system make sensible recommendations for new users.

Example age. Machine learning systems trained on past data tend to favor the past. The popularity of a video changes quickly. A model trained over a window of several weeks would predict the average popularity over that window. To correct this, the authors feed the age of each training example to the model as a feature. At serving time, they set this feature to zero or slightly negative to show that the model is predicting at the end of the training window. With this feature, the model captures how a video's popularity changes after upload. It also increased watch time on recently uploaded videos in A/B tests.

Choosing labels and context. The authors stress that recommendation solves a surrogate problem. The choice of surrogate has a large effect in A/B tests and is hard to measure offline. They report four choices that helped:

1. Training examples come from all YouTube watches, including videos embedded on other sites. If the model learned only from watches of its own recommendations, new content would struggle to surface.
2. Each user contributes a fixed number of training examples. This weights users equally in the loss and stops a small group of very active users from dominating it.
3. The model is given less information about searches on purpose. A model that knows the user's last search will recommend the videos on that search results page, and those perform poorly as home page recommendations. Representing searches as an unordered bag of tokens, with no sequence information, hides where the label came from.
4. The model predicts the user's next watch using only the actions the user took before it. Many collaborative filtering systems hold out a random watch from the middle of a user's history, which leaks future information into the inputs. Viewing patterns are also asymmetric. People watch episodic series in order and often discover an artist through popular videos before moving to niche ones.

Features and depth. Adding features and hidden layers improved mean average precision on holdout data. In these experiments, the vocabulary held 1 million videos and 1 million search tokens, each embedded in 256 dimensions. Each user contributed up to 50 recent watches and 50 recent searches. The network followed a "tower" pattern, in which each hidden layer has half the units of the layer below it. A network with no hidden layers performed about as well as the earlier matrix factorization system. The authors added width and depth until the gains diminished, up to four hidden layers of 2048, 1024, 512 and 256 ReLU units.

Ranking

Ranking uses impression data to specialize and calibrate the candidate predictions for a particular user interface. For example, a user may like a video in general but be unlikely to click it on the home page because of its thumbnail. Ranking scores only a few hundred videos, so it can use many more features about each video and the user's relationship to it. Ranking also combines candidates from different sources whose scores cannot be compared directly.

The ranking network has an architecture similar to the candidate generation network. It gives each video impression an independent score using weighted logistic regression. The system sorts the videos by this score and returns the list to the user. The authors tune the ranking objective with live A/B tests, and it is generally a simple function of expected watch time per impression. Ranking by click-through rate tends to promote deceptive videos that users do not finish ("clickbait"). Watch time captures engagement better.

Feature engineering. The ranking models use hundreds of features, split roughly evenly between categorical and continuous. The team still spent considerable engineering effort turning raw user and video data into useful features. The most important signals describe the user's past interaction with the video itself and with similar items. One example is how many videos the user has watched from the channel that uploaded the video. These features generalize well across different items. Information from candidate generation is also passed to ranking as features, such as which sources nominated the video and what scores they gave it. Features that count past impressions of a video make the list change between requests. If a user was recently shown a video and did not watch it, the model demotes that video on the next page load.

Embedding categorical features. As in candidate generation, sparse categorical features are mapped to dense embeddings. Each ID space (vocabulary) has its own embedding. Its dimension grows roughly with the logarithm of the number of unique values. Very large vocabularies, such as video IDs and search terms, keep only the top N values by frequency in clicked impressions. Values outside the vocabulary map to a zero embedding. Features in the same ID space share one embedding. For example, the ID of the video being scored and the ID of the last video the user watched use the same video embedding. Each feature still enters the network separately, so the layers above can learn a separate representation for each one. Sharing embeddings improves generalization, speeds up training and reduces memory use.

Normalizing continuous features. Neural networks are sensitive to the scale and distribution of their inputs. The authors found that proper normalization of continuous features was critical for training to converge. They transform each continuous feature with its cumulative distribution so that its values are spread evenly in [0, 1). They approximate this transform by linear interpolation on quantiles computed in one pass over the data before training. They also feed the square and the square root of each normalized feature. These powers let the network form super-linear and sub-linear functions of the feature, and they improved offline accuracy.

Modeling expected watch time. The goal is to predict expected watch time. Each training example is an impression that the user either clicked (positive) or did not click (negative). The model uses logistic regression with a cross-entropy loss. Each positive example is weighted by the time the user spent watching the video, and each negative example has unit weight. Because the click probability is small, the odds the model learns closely approximate the expected watch time of the impression. At serving time, the model uses the exponential function as its final activation to produce these odds.

Hidden layers. The authors measured a "weighted, per-user loss" on next-day holdout data. They score each pair of a clicked and an unclicked impression shown to a user on the same page. If the unclicked impression scores higher, the clicked video's watch time counts as mispredicted. The loss is the mispredicted watch time as a fraction of total watch time.

Hidden layers	Weighted, per-user loss
None	41.6%
256 ReLU	36.9%
512 ReLU	36.7%
1024 ReLU	35.8%
512 ReLU → 256 ReLU	35.2%
1024 ReLU → 512 ReLU	34.7%
1024 ReLU → 512 ReLU → 256 ReLU	34.6%

Wider and deeper networks lowered the loss. They also needed more server CPU time for inference. The network with 1024, 512 and 256 ReLU units gave the best result within the serving budget. In that network, feeding the normalized continuous features without their powers increased the loss by 0.2%. Weighting positive and negative examples equally increased the loss by 4.1%.

Lessons

The authors conclude that their deep collaborative filtering model outperformed the matrix factorization methods YouTube used before. They found that choosing the surrogate problem involves more art than science. Predicting a future watch worked well in live tests because it captures asymmetric co-watch behavior and does not leak future information. Withholding some signals from the classifier was essential to stop the model from overfitting the surrogate problem. The example age feature removed the model's bias towards the past. In ranking, the deep network outperformed YouTube's previous linear and tree-based methods for predicting watch time.

Since 2016

Later YouTube papers built on this design. In 2019, Yi and colleagues described a two-tower retrieval model for YouTube that corrects the sampling bias of in-batch negatives by estimating item frequencies from streaming data. The same year, Zhao and colleagues described a multitask ranking system for recommending what video to watch next, which optimizes several objectives with a Multi-gate Mixture-of-Experts model and reduces selection bias with a Wide & Deep design.