# A Hierarchical Self-attentive Convolution Network for Review Modeling in
Recommendation Systems

Hansi Zeng  School of Computing, University of Utah<EMAIL_ADDRESS>and  Qingyao Ai  School of Computing, University of Utah<EMAIL_ADDRESS>
(2018)

###### Abstract.

Using reviews to learn user and item representations is important for
recommender system. Current review based methods can be divided into two
categories: (1) the Convolution Neural Network (CNN) based models that extract
n-gram features from user/item reviews; (2) the Recurrent Neural Network (RNN)
based models that learn global contextual representations from reviews for
users and items. Despite their success, both CNN and RNN based models in
previous studies suffer from their own drawbacks. While CNN based models are
weak in modeling long-dependency relation in text, RNN based models are slow
in training and inference due to their incapability with parallel computing.
To alleviate these problems, we propose a new text encoder module for review
modeling in recommendation by combining convolution networks with self-
attention networks to model local and global interactions in text together. As
different words, sentences, reviews have different importance for modeling
user and item representations, we construct review models hierarchically in
sentence-level, review-level, and user/item level by encoding words for
sentences, encoding sentences for reviews, and encoding reviews for user and
item representations. Experiments on Amazon Product Benchmark show that our
model can achieve significant better performance comparing to the state-of-
the-art review based recommendation models.

Recommender System, Review Modeling, Self-Attention Network, Convolution
Neural Network

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: Woodstock ’18: ACM Symposium on Neural
Gaze Detection; June 03–05, 2018; Woodstock, NY††booktitle: Woodstock ’18: ACM
Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Review text is considered to be valuable for the learning of effective user
and item representations for recommender systems. Previous studies show that
incorporating user reviews into the optimization of recommender systems can
significantly improve the performance of rating prediction by alleviating data
sparsity problems with user preferences and item properties expressed in
review text (Topic integratedHFT, ; Topic integratedRMR, ; DeepCoNN, ; narre,
). Among different techniques for review modeling in recommendation, deep
learning methods, especially convolution nerual networks (CNN) and recurrent
neural networks (RNN), are the most popular paradigms in state-of-the-art
recommendation models. For example, DeepCoNN(DeepCoNN, ) uses two convolution
nerual networks to learn user and item representations from reviews;
HANN(HANN, ) uses a sequence of gated recurrent unit(GRU) to model the user
and item embeddings from reviews.

Despite their success, both CNN based and RNN based review modeling paradigms
have their own weakness. While the CNN based models are good at capturing
local interactions from text such as n-gram key-phrases, they are weak in
learning global interactions from text (rnn and cnn, ). For example, a
sentence ”this version is not classic like its predecessor, but its pleasures
are still plentiful” could be classified as having negative sentiment by CNN
models because they tend to capture n-grams like “is not classic” but not the
whole meaning of the sentence. While RNN based models are good at learning the
semantic meaning of the whole sentence, they suffer from slow training and
prediction due to their sequential nature. Because RNN has to take inputs one
by one, it is difficult, if not impossible, to use any parallel computing
techniques to speed up their inference in production.

To overcome the shortcomings of CNN and RNN based models, in this paper, we
build a new module to encode text information by combining convolution layer
with self-attention layer (attention, ). Specifically, we use the convolution
layer to capture local interaction between words such as key-phrases, and use
the self-attention layer to learn global interactions from text to make the
whole system friendly to parallel computing. Also, to capture the hierarchical
nature of user/item reviews, we propose to construct multi-layer encoders to
capture the review information hierarchically from sentence-level, review-
level to user/item-level. In sentence-level, we encode a sentence from its
words by stacking a word-level self-attentive convolution layer and an
attention-based aggregation layer to model the contextual representations of
words and their informativeness for the sentence. In review-level, we learn a
review representation from its sentences by stacking the same two layers to
capture the local and global interactions between sentences and their
corresponding importance for semantic modeling. Finally, in user/item-level we
learn user and item representations from their reviews using attention
networks to select and combine review representations according to their
utility for recommendation. Empirical experiments on four datasets from Amazon
Product Benchmark show that our proposed Hierarchical Self-Attentive
Convolution Network (HSACN) can extract more effective user and item
representations from review data comparing to that state-of-the-art review-
based recommendation models.

##  2\. Related Works

Using review text to enhance user and item representations for recommender
system has been widely studied in recent years (Topic integratedHFT, ; Topic
integratedRMR, ). Many works are focus on topic modeling from review text for
users and items. For example, the HFT(Topic integratedHFT, ) uses LDA-like
topic modeling to learn user and item parameters from reviews. The likelihood
from the topic distribution is used as a regularization for rating prediction
by matrxi factorization(MF). The RMR(Topic integratedRMR, ) uses the same LDA-
like model on item reviews but fit the ratings using Guassian mixtures but not
MF-like models. Recently, with the advance of deep learning, many
recommendation models start to combine neural network with review data to
learn user and item representations, including DeepCoNN(DeepCoNN, ),
D-Att(cnndlga, ), NARRE(narre, ), HUITA(HUITA, ), HANN(HANN, ). Despite their
differences, existing work using deep learning models for review modeling can
be roughly divided into two categories – CNN based models and RNN based
models. For example, DeepCoNN (DeepCoNN, ) uses two convolution neural
networks to learn user and item representations from reviews; NARRE (narre, )
extends CNN with an attention network over review-level to select reviews with
more informativeness; HANN (HANN, ) encodes reviews using GRU and uses
attention networks in word-level and review-level, etc. To overcome the
existing problems of review modeling with RNN and CNN models, in this paper,
we propose a text encoding module by stacking convolution and self-attention
layer, where convolution layer captures text local feature and self-attention
layer learns text global feature. Also, we use the encoding module
hierarchically to flexible pay varying attentions to reviews for user/item
representations.

##  3\. Proposed Method

In this section, we introduce our proposed recommendation model based on local
convolution and global self-attention, see Figure 1. There are three major
encoders to encode information: a sentence encoder that learns the embedding
of a sentence from its words; a review encoder learning the representation of
a review from its sentences; and a user/item encoder which learns user/item
representation from its reviews. Specifically, we first introduce the sequence
encoding and aggregating modules shared by sentence and review encoders, and
then the details about other model architectures.

Figure 1. Architecture of the proposed method

###  3.1. Sequence Encoding & Aggregating Module

Sequence Encoding Module learns a contextual representation for each element
in the sequence by considering the local and global relations between them. It
is a stack of a convolution layer, a multihead self-attention layer, and a
feed-forward layer. Formally, the input of the module is a sequence of
embeddings $\\{\bm{x_{1}},\ldots,\bm{x_{t}}\\}$, and the output is
$\\{\bm{z_{1}},\ldots,\bm{z_{t}}\\}$, where $\bm{x_{i}}\in\mathbb{R}^{d}$,
$\bm{z_{i}}\in\mathbb{R}^{d_{k}}$, $d$ is the dimension of input embeddings,
and $d_{k}$ is the hidden dimension of the module. First, we use the
convolution network to learn n-gram features from the input sequence.
Mathematically, for a sequence of embeddings
$\\{\bm{x_{1}},\ldots,\bm{x_{t}}\\}$, we denote $\bm{c}_{i}\in\mathbb{R}^{dn}$
as a concatenation of embeddings
$\bm{x}_{i-\lfloor\frac{n-1}{2}\rfloor},\ldots,\bm{x}_{i+\lfloor\frac{n-1}{2}\rfloor}$,
where $n$ is the width. Then we can learn the n-gram feature: $\bm{q}_{i}$,
$\bm{k}_{i}$, $\bm{v}_{i}$ from
$\bm{x}_{i-\lfloor\frac{n-1}{2}\rfloor},\ldots,\bm{x}_{i+\lfloor\frac{n-1}{2}\rfloor}$
by using the convolution weights
$\\{\bm{W}_{Q},\bm{W}_{K},\bm{W}_{V}\\}\in\mathbb{R}^{d_{k}\times nd}$ and
biases $\\{\bm{b}_{Q},\bm{b}_{K},\bm{b}_{V}\\}\in\mathbb{R}^{d_{k}}$
respectively:

(1) $\displaystyle\bm{q}_{i}$
$\displaystyle=\text{Relu}(\bm{W}_{Q}\bm{c}_{i}+\bm{b}_{Q})$ (2)
$\displaystyle\bm{k}_{i}$
$\displaystyle=\text{Relu}(\bm{W}_{K}\bm{c}_{i}+\bm{b}_{K})$ (3)
$\displaystyle\bm{v}_{i}$
$\displaystyle=\text{Relu}(\bm{W}_{V}\bm{c}_{i}+\bm{b}_{V})$

After the extraction of n-gram features, we take them as input to the
multihead self-attention network . We use self-attention since it can capture
the long-range dependencies of the sequence which is the complementary to
local n-gram feature. Formally, we split
$(\bm{q}_{i},\bm{k}_{i},\bm{v}_{i})_{i=1}^{t}$ into H heads, denoted as
$(\bm{q}_{i}^{h},\bm{k}_{i}^{h},\bm{v}_{i}^{h})_{i=1}^{t}$ for each head,
where
$\\{\bm{q}_{i}^{h},\bm{k}_{i}^{h},\bm{v}_{i}^{h}\\}\in\mathbb{R}^{d_{h}}$ and
$d_{h}$ is the subspace dimension for each head. We apply the same operation
to each head. Besides we use relative position embeddings (relative attention,
) $\bm{p}^{V}$, $\bm{p}^{K}$ to encode the pairwise distance between input
sequences. Formally, each head’s output $\bm{z}_{i}^{h}$ is the weighted sum
of transformed inputs with a learnable vector $\bm{p}_{ij}^{V}\in R^{d_{h}}$
encoding the distance between element $i$ and element $j$,

(4)
$\mathbf{z}_{i}^{h}=\displaystyle\sum_{j=1}^{t}a_{ij}^{h}(\bm{v}^{h}_{j}+\bm{p}^{V}_{ij})$

each weighted coefficient is computed using a softmax function,

(5) $a_{ij}^{h}=\frac{\exp{(e_{ij}^{h})}}{\sum_{l=1}^{t}\exp{(e_{il}^{h}})}$

And $e_{ij}^{h}$ is computed as the inner product between two elements,

(6)
$e_{ij}^{h}=\frac{<\bm{q}_{i}^{h},\bm{k}_{j}^{h}+\bm{p}^{K}_{i,j}>}{\sqrt{d_{h}}}$

where $\bm{p}^{K}_{i,j}\in\mathbb{R}^{d_{h}}$ is a learnable parameter. Noted
that the $\bm{p}^{V}_{ij}$ and $\bm{p}^{K}_{i,j}$ are different parameters
since one encodes the positional information between values
$(\bm{v}_{i}^{h})_{i=1}^{t}$, and the other encodes the positional information
between keys $(\bm{k}_{i}^{h})_{i=1}^{t}$, but they can be shared across
attention heads. Finally, we concatenate each head’s output $\bm{z}_{i}^{h}$
to get $\bm{z}_{i}$ and apply a one layer feed-forward network with weights
$\bm{W}_{f}$, bias $\bm{b}_{f}$ to learn a more flexible representation,

(7) $\bm{z}_{i}=\text{Relu}(\bm{W}_{f}\bm{z}_{i}+\bm{b}_{f})$

where $\bm{W}_{f}\in\mathbb{R}^{d_{k}\times d_{k}}$,
$\bm{b}_{f}\in\mathbb{R}^{d_{k}}$.

Sequence Aggregating Module takes the output of the sequence encoding module
as input, and distills their information to form a single vector using
attention network. We then treat the vector as the representation of the
sequence. Formally, let $\\{\bm{z_{1}},\ldots\bm{z_{t}}\\}$ be the input
sequence, then the $i-th$ element importance score is computed as,

(8) $\displaystyle a_{i}$
$\displaystyle=\bm{h}^{T}\text{Relu}(\bm{W}_{p}\bm{z}_{i}+\bm{b}_{p})$ (9)
$\displaystyle a_{i}$
$\displaystyle=\frac{\exp(a_{i})}{\sum_{j=1}^{t}\exp(a_{j})}$

where $\bm{W}_{p}\in\mathbb{{R}}^{d_{p}\times d_{k}}$,
$\bm{b}_{p}\in\mathbb{R}^{d_{p}}$, $\bm{h}\in\mathbb{R}^{d_{p}}$ are learnable
parameters, and in our model, we set $d_{p}=\frac{d_{k}}{2}$. Then the final
output $\bm{l}$ is the weighted sum of contextual representations of each
element,

(10) $\bm{l}=\displaystyle\sum_{i=1}^{t}a_{i}\bm{z}_{i}$

###  3.2. Model Architecture

Sentence Encoder. The sentence encoder learns a sentence representation from
words, and there are three modules in the encoder. The first one is a word
embedding layer that maps a sequence of words from a sentence to their
corresponding dense vectors. Mathematically, assume a sentence $s$ consists of
words $\\{w_{1},\ldots w_{t}\\}$ with size of $t$. The embedding layer
transforms the word sequence into dense vectors
$\\{\bm{e_{1}},\ldots\bm{e_{t}}\\}$, where $\bm{e_{i}}\in\mathbb{R}^{d}$. We
denote the word embedding matrix as $\mathbf{E}\in\mathbb{R}^{V\times d}$,
where $V$ is the word vocabulary size, $d$ is the dense vector dimension.

The second one is a word-level sequence encoding module we defined in section
3.1, and we denote it as a function $f_{s}$. The input of the function is word
embeddings $\\{\bm{e_{1}},\ldots\bm{e_{t}}\\}$, and the output is their
corresponding contextual representations
$\\{\bm{z^{w}_{1}},\ldots\bm{z^{w}_{t}}\\}$, which are computed as,

(11)
$\\{\bm{z^{w}_{1}},\ldots\bm{z^{w}_{t}}\\}=f_{s}(\\{\bm{e_{1}},\ldots\bm{e_{t}}\\},d_{w})$

where $d_{w}$ is the hidden dimension of the module, and
$\bm{z_{i}^{w}}\in\mathbb{R}^{d_{w}}$.

The third module is a word-level sequence aggregating module defined in
section 3.1. We denote it as a function $g_{s}$, and it takes contextual
representations $\\{\bm{z^{w}_{1}},\ldots\bm{z^{w}_{t}}\\}$ as input to get
the final sentence embedding,

(12) $\bm{s}=g_{s}(\\{\bm{z^{w}_{1}},\ldots\bm{z^{w}_{t}}\\})$

Review Encoder. The review encoder learns a review representation from its
sentences, which consists of two major modules. The first module is a
sentence-level sequence encoding module, and we treat it as a function
$f_{r}$. For each review, the function takes a sequence of sentence embeddings
$\\{\bm{s}_{1},\ldots\bm{s}_{n}\\}$ as input and get their corresponding
contextual representations $\\{\bm{z}^{s}_{1},\ldots\bm{z}^{s}_{n}\\}$, which
are computed as,

(13)
$\\{\bm{z^{s}_{1}},\ldots\bm{z^{s}_{n}}\\}=f_{r}(\\{\bm{s_{1}},\ldots\bm{s_{n}}\\},d_{s})$

where $d_{s}$ is the hidden dimension of the the module, and
$\bm{z^{s}_{i}}\in\mathbb{R}^{d_{s}}$.

The second module of the review encoder is the sentence-level aggregating
module, and we denote it as $g_{r}$. The input of the function is
$\\{\bm{z}^{s}_{1},\ldots\bm{z}^{s}_{n}\\}$, and the output is the review
representation $r$,

(14) $\bm{r}=g_{r}(\\{\bm{z^{s}_{1}},\ldots\bm{z^{s}_{n}}\\})$

User/item Encoder. We learn a user representation using review-level
aggregating module, denoted as $g_{u}$, to select their reviews with more
informativeness. Formally, we assume a user $u$ writes $m$ reviews
$r_{1},\ldots r_{m}$, then the final user representation can be computed as,

(15) $\bm{u}^{r}=g_{u}(\\{r_{1},\ldots r_{m}\\})$

Also, the representation of item $i$ can be computed in similar way, and we
denote it as $\bm{i}^{r}$.

Prediction layer. We first project user representation learned from reviews
$u^{r}$ into the latent space using one layer feed-forward network,

(16) $\overline{\bm{u}^{r}}=\text{Relu}(\bm{W}_{u}u^{r}+\bm{b}_{u})$

where $\bm{W}_{u}\in\mathbb{R}^{d_{l}\times d_{s}}$,
$\bm{b}_{u}\in\mathbb{R}^{d_{l}}$. Similarly using the equation above, we can
get the projected item representation $\overline{\bm{i}^{r}}$. Also, we
introduce another set of embeddings $\overline{\bm{u}^{d}}$,
$\overline{\bm{i}^{d}}$, with the same dimension as $\overline{\bm{u}^{r}}$
and $\overline{\bm{i}^{r}}$, for each user and item, and then the interaction
between user and item can be modeled as,

(17)
$\bm{h}=(\overline{\bm{u}^{d}}+\overline{\bm{u}^{r}})\odot(\overline{\bm{i}^{d}}+\overline{\bm{i}^{r}})$

The final rating prediction can be computed as,

(18) $\hat{R}_{uv}=\bm{w}_{f}^{T}\bm{h}+b_{u}+b_{v}+b_{g}$

where $\bm{w}_{f}\in\mathbb{R}^{d_{l}}$, $b_{u},b_{v},b_{g}\in\mathbb{R}$. The
$b_{u}$ represents user bias, $b_{v}$ is item bias, and $b_{g}$ is the global
bias.

When training model, we minimize the loss between ground truth rating $R_{uv}$
with predicted rating $\hat{R}_{uv}$. We use the mean square error as the loss
function,

(19)
$\mathcal{L}=\frac{1}{N}\displaystyle\sum_{i=1}^{N}(R_{uv}-\hat{R}_{uv})^{2}$

where $N$ is the number of user item pairs in the training dataset.

##  4\. Experiment

Datasets and evluation metrics. We conduct our experiment on four different
categories of 5-core Amazon product review datasets (amazon dataset, ). The
domains and statistics of these four categories are shown in the Table 1. For
each dataset, We randomly split user-item pairs into $80\%$ training set,
$10\%$ validation set, and $10\%$ testing set. And we use NLTK(NLTK, ) to
tokenize sentences and words of reviews. Since the length and the number of
reviews have a long tail effect, we only keep the number of reviews cover
$90\%$ and length of reviews cover $70\%$ users and items for the balance of
efficiency and performance. Following(narre, ), we adopt Root Mean Square
Error(RMSE) as the main metric to evaluate the performance of our model.

Table 1. Statistical details of the datasets.

| Video Games | Toys and Games | Kindle Store | Office Products
---|---|---|---|---
users | 24,303 | 19,412 | 68,223 | 4,905
items | 10,672 | 11,924 | 61,935 | 2,420
reviews | 231,780 | 167,597 | 982,619 | 53,258

Baselines. To evaluate the performance of our method we compare it with
several baseline models. (1) NeurMF: CF based model combines linearity of GMF
and non-linearity of MLPs for modeling user and item latent
representations(neurMF, ); (2) HFT: topic modeling based model combines the
ratings with reviews via LDA(Topic integratedHFT, ); (3) DeepCoNN: CNN based
model uses two convolution neural network to learn user and item
representations (DeepCoNN, ); (4) D-ATT: CNN based model extends DeepCoNN by
using dual-attention layer over the word-level before convolution(cnndlga, );
(5) NARRE: CNN based model modifies the DeepCoNN by using the attention
network over review-level to select reviews with more informativeness(narre,
).

Parameter settings . We use 300-dimensional pretrained word embeddings from
Google News(googlenews, ), and employ the Adam for optimization with an inital
learning rate $0.0002$. We set the hidden dimension of word-level and
sentence-level sequence encoding module as $150$, and the latent dimension of
the prediction layer as $32$. Also, the convolution kernel size is $3$, and
number of head for each self-attention layer is $2$. We apply dropout after
the word embedding layer, after each feed forward layer in sequence encoding
modules, and before the prediction layer with rate $[0.3,0.5,0.5]$. We set the
hyper-parameters of baseline models following the settings of their original
papers.

Results and analysis . The RMSE results of compared models are shown in Table
2. Several observations can be made from the results. Firstly, models with
review information generally outperform models without review information. It
validates that utilizing rich information from reviews for user and item
representations can boost the performance of recommender systems. Secondly,
the CNN based models with attention network like NARRE and D-ATT can
outperform DeepCoNN consistently. It proves that using attention network to
distill important information from reviews can improve the final performance.
Finally, our model, namely HSACN, shows consistently improvement over all
baseline models on all datasets. The relative improvement against NARRE are
$1.1\%$ in Video Games, $0.5\%$ in Toys and Games, $2\%$ in Kindle Store and
$1.2\%$ in Office Products. The reason might be that our model consists of
convolution layer with self-attention layer, which can both extract the local
and global information of reviews. Also, the hierarchical model architecture
helps select information more robust and effective.

Table 2. The RMSE scores of the different methods.

Dataset | Video Games | Toys and Games | Kindle Store | Office Products
---|---|---|---|---
NeurMF | 1.0950 | 0.9359 | 0.8246 | 0.8809
HFT | 1.0834 | 0.9075 | 0.7980 | 0.8452
DeepCoNN | 1.0737 | 0.8950 | 0.7938 | 0.8412
D-ATT | 1.0667 | 0.8914 | 0.7905 | 0.8335
NARRE | 1.0630 | 0.8932 | 0.7881 | 0.8331
HSACN | 1.0511 | 0.8871 | 0.7721 | 0.8230

Case study. We conduct a case study to check if our model can select
informative words and sentences. The visualization of the attention weights in
word-level and sentence-level aggregating module are shown in Figure 2. The
ground-truth rating is $4.0$. We can see that the word-level aggregating
module can not only effectively select the positive words like ”appreciate”,
”loves”, ”wonderful”, ”nice”, but also sentiment negation words like
”criticism”, ”but”. And with the help of self-attention layer of our model
that can capture global interactions of text, the predicted rating is $4.23$
which is roughly a positive rating but considers the effect of negative
sentiment words. Also the important sentences can be effectively selected by
the sentence-level aggregation module. For example, the sentence ”I appreciate
the materials used to the puzzle and our three year old loves the picture that
is formed when the puzzle is complete” is assigned higher score, since it can
more effectively represent the user preferences and item properties. In
contrast, the sentence ”I have added an image to the product page” has lower
attention score since it contains limit information of users and items.

Figure 2. Randomly select a (user,item) pair on the Toys and Games dataset,
then visualize a most important review chosen by review-level sequence
aggregating module. Orange boxes to the left represent the sentence important
score, and blue boxes on each individual word represent the word important
score. Darker color means higher important score.

##  5\. Conclusion

In this paper, we propose a hierarchical self-attentive convolution network
(HSACN) to learn user and item representations from reviews in recommendation
systems. We propose to use a text encoder combining convolution networks with
self-attention networks to learn local and global interactions in text
together, and construct our model hierarchically in sentence-level, review-
level and user/item level. Experiments and case studies show that our model
can achieve better performance and extract more effective user and item
representations from reviews compared with other state-of-art review based
recommendation models. For future work, we want to conduct more analysis on
the performance and efficiency of our model in other recommendation scenarios
such as ranking and diversification.

## References

  * (1) Lei Zheng, Vahid Noroozi, Philip S. Yu.  Joint Deep Modeling of Users and Items Using Reviews for Recommendation  In WSDM 2017. 425–434.
  * (2) Chong Chen, Min Zhang, Yiqun Liu, Shaoping Ma.  Neural Attentional Rating Regression with Review-level Explanations.  In WWW ’18. Pages 1583–1592.
  * (3) Dawei Cong, anyan Zhao, Bing Qin, Yu Han, Murray Zhang  Hierarchical Attention based Neural Network for Explainable Recommendation  In CMR ’19, June 10–13, 2019, Ottawa, ON, Canada
  * (4) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin  Attention Is All You Need  In 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
  * (5) Peter Shaw, Jakob Uszkoreit, Ashish Vaswani  Self-Attention with Relative Position Representations  In Proceedings of NAACL-HLT 2018, pages 464–468
  * (6) Sungyong Seo, Jing Huang, Hao Yang, Yan liu.  Interpretable Convolutional Neural Networks with Dual Local and Global Attention for Review Rating Prediction  In RecSys ’17. 297–305.
  * (7) Chuhan Wu, Fangzhao Wu, Junxin Liu, and Yongfeng Huang.  Hierarchical User and Item Representation with Three-Tier Attention for Recommendation  In Proceedings of NAACL-HLT 2019, pages 1818–1826
  * (8) Steffen Rendle.  Factorization Machines  In ICDM ’10: Proceedings of the 2010 IEEE International Conference on Data Mining December 2010 Pages 995–1000
  * (9) Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schutze.  Comparative study of cnn and rnn for natural ¨ language processing.  arXiv preprint arXiv:1702.01923.
  * (10) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, J. Dean  Distributed representations of words and phrases and their compositionality  In NIPS’13: Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2 December 2013 Pages 3111–3119
  * (11) R. He, J. McAuley  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering  WWW, 2016
  * (12) Julian McAuley and Jure Leskovec  Hidden factors and hidden topics: understanding rating dimensions with review text  In RecSys 165–172 2013
  * (13) Guang Ling, Michael R Lyu, and Irwin King  Ratings meet reviews, a combined approach to recommend  In RecSys 105–112 2014
  * (14) Yang Bao, Hui Fang, Jie Zhang  TopicMF: Simultaneously Exploiting Ratings and Reviews for Recommendation  In AAAI 2014, Pages: 2-8
  * (15) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua.  Neural collaborative filtering.  In Proceedings ofthe 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 173–182.
  * (16) Edward Loper , Steven Bird  NLTK: The Natural Language Toolkit (2002)  In ACL’2002