# MASKER: Masked Keyword Regularization for Reliable Text Classification

Seung Jun Moon1, Sangwoo Mo∗1, Kimin Lee2, Jaeho Lee1, Jinwoo Shin1
Equal contributionWork was done while the author was at KAIST

###### Abstract

Pre-trained language models have achieved state-of-the-art accuracies on
various text classification tasks, e.g., sentiment analysis, natural language
inference, and semantic textual similarity. However, the reliability of the
fine-tuned text classifiers is an often underlooked performance criterion. For
instance, one may desire a model that can detect out-of-distribution (OOD)
samples (drawn far from training distribution) or be robust against domain
shifts. We claim that one central obstacle to the reliability is the over-
reliance of the model on a limited number of keywords, instead of looking at
the whole context. In particular, we find that (a) OOD samples often contain
in-distribution keywords, while (b) cross-domain samples may not always
contain keywords; over-relying on the keywords can be problematic for both
cases. In light of this observation, we propose a simple yet effective fine-
tuning method, coined masked keyword regularization (MASKER), that facilitates
context-based prediction. MASKER regularizes the model to reconstruct the
keywords from the rest of the words and make low-confidence predictions
without enough context. When applied to various pre-trained language models
(e.g., BERT, RoBERTa, and ALBERT), we demonstrate that MASKER improves OOD
detection and cross-domain generalization without degrading classification
accuracy. Code is available at https://github.com/alinlab/MASKER.

##  1 Introduction

Text classification (Aggarwal and Zhai 2012) is a classic yet challenging
problem in natural language processing (NLP), having a broad range of
applications, including sentiment analysis (Bakshi et al. 2016), natural
language inference (Bowman et al. 2015), and semantic textual similarity
(Agirre et al. 2012). Recently, Devlin et al. (2019) have shown that fine-
tuning a pre-trained language model can achieve state-of-the-art performances
on various text classification tasks without any task-specific architectural
adaptations. Thereafter, numerous pre-training and fine-tuning strategies to
improve the classification accuracy further have been proposed (Liu et al.
2019; Lan et al. 2020; Sanh et al. 2019; Clark et al. 2020; Sun et al. 2019;
Mosbach, Andriushchenko, and Klakow 2020; Zhang et al. 2020). However, a vast
majority of the works have focused on evaluating the accuracy of the models
only and overlooked their reliability (Hendrycks et al. 2020), e.g.,
robustness to out-of-distribution (OOD) samples drawn far from the training
data (or in-distribution samples).

Figure 1:  Out-of-distribution (OOD) and cross-domain examples, where class
‘Apple’ is the original domain. The OOD sample contains the word ‘apple’ (red)
but in a different context. The cross-domain sample does not share the words
(e.g., ‘Tim Cook’) with the original domain, but it still contains some clues
(yellow) to guess the correct class.

(a) In-distribution

(b) Out-of-distribution

(c) Cross-domain

Figure 2:  Frequency of the keywords selected from the source class (x-axis)
in the target class (y-axis). (a) Both the source and target classes are in-
distribution, (b) source and target distributions are in- and out-of-
distribution, respectively, and (c) source and target distributions are
identical but of multiple domains. In (b), one can see that OOD classes often
contain the same keywords from the (similar but different) in-distribution
classes, e.g., see ‘Iron’ and ‘Autos.’ In (c), one can see that the keywords
in one domain do not perfectly align to the other domain, e.g., see ‘IMDB
(neg)’ and ‘Food (pos).’

While pre-trained language models are known to be robust in some sense
(Hendrycks et al. 2020), we find that fine-tuned models suffer from the over-
reliance problem, i.e., making predictions based on only a limited number of
domain-specific keywords instead of looking at the whole context. For example,
consider a classification task of ‘Apple’ visualized in Figure 1. If the most
in-distribution samples contain the keyword ‘Apple,’ the fine-tuned model can
predict the class solely based on the existence of the keyword. However, a
reliable classifier should detect that the sentence “I ate an apple this
morning” is an out-of-distribution sample (Hendrycks and Gimpel 2017; Shu, Xu,
and Liu 2017; Tan et al. 2019). On the other hand, the sentence “Tim Cook said
that …” should be classified as the topic ‘Apple’ although it does not contain
the keyword ‘Apple’ and the keyword ‘Tim Cook’ is not contained in the
training samples. In other words, the reliable classifier should learn
decision rules that generalize across domains (Fei and Liu 2015; Bhatt,
Semwal, and Roy 2015; Bhatt, Sinha, and Roy 2016).

This problematic phenomenon frequently happens in real-world datasets. To
verify this, we extract the keywords from Amazon 50 class reviews (Chen and
Liu 2014) dataset and sentiment analysis datasets (IMDB (Maas et al. 2011);
SST-2 (Socher et al. 2013); Fine Food (McAuley and Leskovec 2013)), following
the attention-based scheme illustrated in Section 2.1. Figure 2 shows the
frequency of the keywords selected from the source class in the target class.
Figure 2(a) shows that the keywords are often strongly tied with the class,
which leads the model to learn a shortcut instead of the context. Figure 2(b)
shows the results where the source and target classes are different classes of
the Amazon reviews dataset. Here, OOD classes often contain the same keywords
from the in-distribution classes, e.g., the class ‘Autos’ contains the same
keywords as the class ‘Iron.’ On the other hand, Figure 2(c) shows the results
where both source and target classes are sentiments (‘pos’ and ‘neg’) classes
in IMDB, SST-2, and Fine Food datasets. While the same sentiment shares the
same keywords, the alignment is not perfect; e.g., ‘IMDB (neg)’ and ‘Food
(pos)’ contain the same keywords.

###  1.1 Contribution

We propose a simple yet effective fine-tuning method coined masked keyword
regularization (MASKER), which handles the over-reliance (on keywords) problem
and facilitates the context-based prediction. In particular, we introduce two
regularization techniques: (a) masked keyword reconstruction and (b) masked
entropy regularization. First, (a) forces the model to predict the masked
keywords from understanding the context around them. This is inspired by
masked language modeling from BERT (Devlin et al. 2019), which is known to be
helpful for learning context. Second, (b) penalizes making high-confidence
predictions from “cut-out-context” sentences, that non-keywords are randomly
dropped, in a similar manner of Cutout (DeVries and Taylor 2017) used for
regularizing image classification models. We also suggest two keyword
selection schemes, each relying on dataset statistics and attention scores. We
remark that all proposed techniques of MASKER can be done in an unsupervised
manner.

We demonstrate that MASKER, applied to the pre-trained language models: BERT
(Devlin et al. 2019), RoBERTa (Liu et al. 2019), and ALBERT (Lan et al. 2020),
significantly improves the OOD detection and cross-domain generalization
performance, without degrading the classification accuracy. We conduct OOD
detection experiments on 20 Newsgroups (Lang 1995), Amazon 50 class reviews
(Chen and Liu 2014), Reuters (Lewis et al. 2004), IMDB (Maas et al. 2011),
SST-2 (Socher et al. 2013), and Fine Food (McAuley and Leskovec 2013)
datasets, and cross-domain generalization experiments on sentiment analysis
(Maas et al. 2011; Socher et al. 2013; McAuley and Leskovec 2013), natural
language inference (Williams, Nangia, and Bowman 2017), and semantic textual
similarity (Wang et al. 2019) tasks. In particular, our method improves the
area under receiver operating characteristic (AUROC) of BERT from 87.0% to
98.6% for OOD detection under 20 Newsgroups to SST-2 task, and reduce the
generalization gap from 19.2% to 10.9% for cross-domain generalization under
Fine Food to IMDB task.

###  1.2 Related Work

Distribution shift in NLP. The reliable text classifier should detect
distribution shift, i.e., test distribution is different from the training
distribution. However, the most common scenarios: OOD detection and cross-
domain generalization are relatively under-explored in NLP domains (Hendrycks
et al. 2020; Marasović 2018). Hendrycks et al. (2020) found that pre-trained
models are robust to the distribution shift compared to traditional NLP
models. We find that the pre-trained models are not robust enough, and we
empirically show that pre-trained models are still relying on undesirable
dataset bias. Our method further improves the generalization performance,
applied to the pre-trained models.

Shortcut bias. One may interpret the over-reliance problem as a type of
shortcut bias (Geirhos et al. 2020), i.e., the model learns an easy-to-learn
but not generalizable solution, as the keywords can be considered as a
shortcut. The shortcut bias is investigated under various NLP tasks (Sun et
al. 2019), e.g., natural language inference (McCoy, Pavlick, and Linzen 2019),
reasoning comprehension (Niven and Kao 2019), and question answering (Min et
al. 2019). To our best knowledge, we are the first to point out that the over-
reliance on keywords can also be a shortcut, especially for text
classification. We remark that the shortcut bias is not always harmful as it
can be a useful feature for in-distribution accuracy. However, we claim that
they can be problematic for unexpected (i.e., OOD) samples, as demonstrated in
our experiments.

Debiasing methods. Numerous debiasing techniques have been proposed to
regularize shortcuts, e.g., careful data collection (Choi et al. 2018; Reddy,
Chen, and Manning 2019), bias-tailored architecture (Agrawal et al. 2018), and
adversarial regularization (Clark, Yatskar, and Zettlemoyer 2019; Minderer et
al. 2020; Nam et al. 2020). However, most prior work requires supervision of
biases, i.e., the shortcuts are explicitly given. In contrast, our method can
be viewed as an unsupervised debiasing method, as our keyword selection
schemes automatically select the keywords.

##  2 Masked Keyword Regularization

We first introduce our notation and architecture setup; then propose the
keyword selection and regularization approaches in Section 2.1 and Section
2.2, respectively.

Notation. The text classifier $f:\mathbf{x}\mapsto y$ maps a document
$\mathbf{x}$ to the corresponding class $y\in\\{1,\dots,C\\}$. The document
$\mathbf{x}$ is a sequence of tokens $t_{i}\in\mathcal{V}$, i.e.,
$\mathbf{x}=[t_{1},\dots,t_{T}]$ where $\mathcal{V}$ is the vocabulary set and
$T$ is the length of the document. Here, the full corpus
$\mathcal{D}=\\{(\mathbf{x},y)\\}$ is a collection of all documents, and the
class-wise corpus $\mathcal{D}_{c}=\\{(\mathbf{x},y)\in\mathcal{D}\mid y=c\\}$
is a subset of $\mathcal{D}$ of class $c$. The keyword set
$\mathcal{K}\subset\mathcal{V}$ is the set of vocabularies which mostly
affects to the prediction.111Chosen by our proposed keyword selection (Section
2.1). The keyword $\mathbf{k}=[k_{1},\dots,k_{L}]$ of the document
$\mathbf{x}$ is given by $\mathbf{k}=[t_{i}\in\mathbf{x}\mid
t_{i}\in\mathcal{K}]$, where $L\leq T$ is the number of keywords in the
document $\mathbf{x}$.

Architecture. We assume the pre-trained language model follows the bi-
directional Transformer (Vaswani et al. 2017) architecture, widely used in
recent days (Devlin et al. 2019; Liu et al. 2019; Lan et al. 2020). They
consist of three components: embedding network, document classifier, and
token-wise classifier. Given document $\mathbf{x}$, the embedding network
produces (a) a document embedding (for an entire document), and (b) token
embeddings, which correspond to each input token. The document and token-wise
classifier predict the class of document and tokens, respectively, from the
corresponding embeddings. For the sake of simplicity, we omit the shared
embedding network and denote the document and token-wise classifier as
$f_{\mathtt{doc}}:\mathbf{x}\mapsto y$ and
$f_{\mathtt{tok}}:\mathbf{x}=[t_{1},\dots,t_{T}]\mapsto\mathbf{s}=[s_{1},\dots,s_{T}]$,
respectively, where $s_{i}\in\mathcal{V}$ is a target token corresponds to
$t_{i}$.

Figure 3:  Top 10 keywords chosen from the frequency-based and attention-based
selection schemes under the Amazon 50 class reviews dataset. The frequency-
based scheme chooses uninformative words (e.g., ‘305’), while the attention-
based scheme chooses more informative ones (e.g., ‘watch’).

(a) Masked keyword reconstruction

(b) Masked entropy regularization

Figure 4:  Illustration of two portions of our proposed method, MASKER: (a)
_Masked keyword reconstruction_ masks keyword tokens in input sentences and
forces the model to predict the original words in masked tokens. (b) _Masked
entropy regularization_ masks non-keyword tokens in input sentences and forces
the model to print uniform distribution, as regarding it as OOD.

###  2.1 Keyword Selection Schemes

We consider two keyword selection schemes, based on the dataset statistics
(model-free) and trained models. While the former is computationally cheaper,
the latter performs better; hence, one can choose its purpose.

Frequency-based. We first choose the keywords using the relative frequency of
the words in the dataset. Specifically, we use the term frequency-inverse
document frequency (TF-IDF; Robertson (2004)) metric, which measures the
importance of the token by comparing the frequency in the target documents
(term frequency) and the entire corpus (inverse document frequency). Here, the
keywords are defined as the tokens with the highest TF-IDF scores. Formally,
let $\mathbf{X}_{c}$ be a large document that concatenates all tokens in a
class-wise corpus $\mathcal{D}_{c}$, and
$\mathbf{D}=[\mathbf{X}_{1},\dots,\mathbf{X}_{C}]$ be a corpus of such large
documents. Then, the frequency-based score of token $t$ is given by

$\displaystyle
s^{\mathtt{freq}}(t)=\max_{c\in\\{1,\dots,C\\}}~{}\mathtt{tf}(t,\mathbf{X}_{c})\cdot\mathtt{idf}(t,\mathbf{D})$
(1)

where $\mathtt{tf}(t,\mathbf{X})=0.5+0.5\cdot
n_{t,\mathbf{X}}/\max\\{{n_{t^{\prime},\mathbf{X}}}:t^{\prime}\in\mathbf{X}\\}$,
$\mathtt{idf}(t,\mathbf{D})=\log(|\mathbf{D}|/|\\{\mathbf{X}\in\mathbf{D}:t\in\mathbf{X}\\}|)$,
and $n_{t,\mathbf{x}}$ is number of token $t$ in document $\mathbf{x}$. Note
that the frequency-based selection is model-agnostic and easily computed, but
does not reflect the contribution of the words to the prediction.

Attention-based. We also choose the keywords using the model attention as it
is a more direct and effective way to measure the importance of words on model
prediction. To this end, we first train a model with a standard approach using
the cross-entropy loss $\mathcal{L}_{\mathtt{CE}}$, which leads the model to
suffer from the over-reliance (on keywords) issue. Our idea is to use the
attention values of the model for choosing the keywords. Here, the keywords
are defined as the tokens with the highest attention values. Formally, let
$\textbf{a}=[a_{1},\dots,a_{T}]\in\mathbb{R}^{T}$ be attention values of the
document embedding, where $a_{i}$ corresponds to the input token $t_{i}$.
Then, the attention-based score of token $t$ is given by

$\displaystyle
s^{\mathtt{attn}}(t)=\sum_{(\mathbf{x},y)\in\mathcal{D}}\frac{1}{n_{t,\mathbf{x}}}\sum_{i\in\\{1,\dots,T\\}}\mathbb{I}(t_{i}=t)\cdot\frac{a_{i}}{\lVert\textbf{a}\rVert}$
(2)

where $\mathbb{I}$ is an indicator function and $\lVert\cdot\rVert$ is
$\ell_{2}$-norm.

We choose the keywords by picking the top $K$ tokens according to the scores
in Eq. (1) and Eq. (2) for each selection scheme, respectively. We also test
the class-balanced version, i.e., pick the top $K/C$ tokens for each class,
but the class-agnostic one performed better.

Comparison of the selection schemes. We observe that the frequency-based
scheme often selects uninformative keywords that uniquely appears in some
class. In contrast, the attention-based scheme selects more general keywords
that actually influence the prediction. Figure 3 shows the keywords chosen by
both selection schemes: the frequency-based scheme chooses uninformative words
such as ‘305’ and ‘forerunner,’ while the attention-based scheme chooses more
informative ones such as ‘watch’ or ‘chips.’

###  2.2 Regularization via Keyword Masking

Using the chosen keywords, we propose two regularization techniques to reduce
the over-reliance issue and facilitate the model to look at the contextual
information.

Masked keyword reconstruction. To enforce the model to look at the surrounding
context, we guide the model to reconstruct the keywords from keyword-masked
documents. Note that it resembles the masked language model (Devlin et al.
2019), but we mask the keywords instead of random words. Masked keyword
reconstruction only regularizes sentences with keywords, and we omit the loss
for ones without any keywords. Formally, let $\tilde{\mathbf{k}}$ be a random
subset of the full keyword $\mathbf{k}$ (selected as in Section 2.1), that
each element is chosen with probability $p$ independently. We mask
$\tilde{\mathbf{k}}$ from the original document $\mathbf{x}$ and get the
masked document $\tilde{\mathbf{x}}=\mathbf{x}-\tilde{\mathbf{k}}$. Then, the
masked keyword reconstruction (MKR) loss is

$\displaystyle\mathcal{L}_{\mathtt{MKR}}(\tilde{\mathbf{x}},v):=\sum_{i\in\mathtt{index}(\tilde{\mathbf{k}})}\mathcal{L}_{\mathtt{CE}}(f_{\mathtt{tok}}(\tilde{\mathbf{\mathbf{x}}})_{i},v_{i})$
(3)

where $\mathtt{index}(\tilde{\mathbf{k}})$ is the index of the keywords
$\tilde{\mathbf{k}}$ with respect to the original document $\mathbf{x}$,
$v_{i}$ is the index of the keywords with respect to the vocabulary set. We
remark that the reconstruction part is essential; we also test simply
augmenting the masked documents, i.e.,
$\mathcal{L}_{\mathtt{CE}}(f_{\mathtt{doc}}(\tilde{\mathbf{x}}),y)$, but it
performed worse. Choosing proper keywords is also crucial; attention-based
keywords performs better than frequency-based or random keywords, as shown in
Table 1 and Table 3.

Masked entropy regularization. Furthermore, we regularize the prediction of
the context-masked documents, that context (non-keyword) words are randomly
dropped. The model should not classify the context-masked documents correctly
as they lost the original context. Formally, let $\widehat{\mathbf{c}}$ be a
randomly chosen subset of the full context words
$\mathbf{c}=\mathbf{x}-\mathbf{k}$, where each element is chosen with
probability $q$ independently. We mask $\widehat{\mathbf{c}}$ from the
original document $\mathbf{x}$ and get the context-masked document
$\widehat{\mathbf{x}}=\mathbf{x}-\widehat{\mathbf{c}}$. Then, the masked
entropy regularization (MER) loss is

$\displaystyle\mathcal{L}_{\mathtt{MER}}(\widehat{\mathbf{x}}):=D_{\mathrm{KL}}(\mathcal{U}(y)||f_{\mathtt{doc}}(\widehat{\mathbf{x}}))$
(4)

where $D_{\mathrm{KL}}$ is the KL-divergence and $\mathcal{U}(y)$ is a uniform
distribution. We remark that MER does not degrade the classification accuracy
since it regularizes non-realistic context-masked sentences, rather than full
documents. Table 1 shows that MER does not drop the classification accuracy in
original domain, while Table 3 and Table 4 show that MER improves the cross-
domain accuracy. On the other hand, MER differs from the prior sentence-level
objectives, e.g., next sentence prediction (Devlin et al. 2019), as our goal
is to regularize shortcuts, not learning better in-domain representation.

To sum up, the final objective is given by

$\displaystyle\mathcal{L}_{\mathtt{total}}=\mathcal{L}_{\mathtt{CE}}+\lambda_{\mathtt{MKR}}\mathcal{L}_{\mathtt{MKR}}+\lambda_{\mathtt{MER}}\mathcal{L}_{\mathtt{MER}}$
(5)

where $\lambda_{\mathtt{MKR}}$ and $\lambda_{\mathtt{MER}}$ are
hyperparameters for the MKR and MER losses, respectively. Figure 4 visualizes
the proposed losses, and the overall procedure is in Appendix B.

Method | Classifier |  Keyword | MKR | MER | AUROC $\uparrow$ | EER $\downarrow$ |  Detection Accuracy $\uparrow$ |  Classification Accuracy $\uparrow$
---|---|---|---|---|---|---|---|---
OC-SVM | - | - | - | - | 57.01$\pm$1.08 | 41.12$\pm$0.90 | 72.99$\pm$4.69 | -
OpenMax | - | - | - | - | 53.02$\pm$3.74 | 55.21$\pm$2.03 | 74.01$\pm$0.58 | 79.01$\pm$0.51
DOC | - | - | - | - | 75.12$\pm$3.06 | 25.55$\pm$4.12 | 80.21$\pm$0.76 | 83.14$\pm$0.82
BERT | Multi-class | - | - | - | 78.46$\pm$1.16 | 28.30$\pm$1.11 | 81.19$\pm$0.62 | 84.76$\pm$0.35
1-vs-rest | - | - | - | 80.56$\pm$0.84 | 25.71$\pm$0.71 | 80.78$\pm$0.48 | 84.66$\pm$0.26
BERT +MASKER (ours) | 1-vs-rest | Random | ✓ | - | 80.52$\pm$0.72 | 25.42$\pm$0.38 | 80.83$\pm$0.33 | 85.06$\pm$0.14
1-vs-rest | Random | - | ✓ | 81.51$\pm$0.27 | 24.17$\pm$0.87 | 82.13$\pm$0.58 | 84.89$\pm$0.71
1-vs-rest | Frequency | ✓ | - | 81.32$\pm$0.78 | 25.12$\pm$1.18 | 81.33$\pm$0.67 | 83.35$\pm$1.77
1-vs-rest | Frequency | - | ✓ | 82.54$\pm$0.54 | 23.88$\pm$0.76 | 82.72$\pm$0.58 | 85.25$\pm$0.70
1-vs-rest | Attention | ✓ | - | 83.33$\pm$0.44 | 24.55$\pm$0.67 | 82.11$\pm$0.68 | 85.27$\pm$0.33
1-vs-rest | Attention | - | ✓ | 82.60$\pm$0.51 | 25.02$\pm$0.83 | 80.99$\pm$0.65 | 85.02$\pm$0.29
1-vs-rest | Attention | ✓ | ✓ | 85.48$\pm$0.54 | 22.30$\pm$1.20 | 85.02$\pm$0.55 | 85.15$\pm$0.95
Table 1:  Ablation study on OOD detection under the Amazon 50 class reviews. We use 25% of classes as in-distribution, and the rest as OOD. The reported results are averaged over five trials, subscripts denote standard deviations, and the best results are highlighted in bold. All components of our method contribute to the OOD detection performance (%).  ID | OOD | OC-SVM | DOC | Vanilla / Residual / MASKER
---|---|---|---|---
BERT | RoBERTa | ALBERT
Newsgroup | Amazon | 62.1 | 84.1 | 85.4/86.7/87.0(+1.9%) | 85.3/85.9/87.2(+2.3%) | 86.7/85.4/89.4(+3.1%)
Reuters | 53.9 | 60.0 | 91.8/93.0/97.7(+6.4%) | 93.1/92.1/93.9(+0.8%) | 93.3/92.0/94.7(+1.6%)
IMDB | 59.8 | 88.6 | 94.6/95.7/98.5(+4.1%) | 95.2/95.9/97.7(+2.7%) | 94.5/92.6/96.6(+2.2%)
SST-2 | 63.0 | 88.1 | 87.0/97.0/98.6(+13.3%) | 94.7/94.9/98.2(+3.6%) | 95.8/95.6/98.1(+2.4%)
Fine Food | 62.8 | 81.3 | 85.3/87.3/93.4(+9.5%) | 88.7/89.4/92.9(+4.7%) | 77.6/86.6/91.6(+18.1%)
Amazon | Newsgroup | 61.3 | 81.3 | 84.8/83.9/87.2(+2.8%) | 87.9/86.6/91.0(+3.5%) | 87.3/85.0/88.4(+1.3%)
Reuters | 55.5 | 79.8 | 89.7/89.7/93.5(+4.2%) | 92.3/92.6/93.6(+1.4%) | 93.1/93.5/94.5(+1.5%)
IMDB | 66.2 | 89.6 | 93.3/92.8/95.2(+2.0%) | 90.1/87.0/93.3(+3.5%) | 89.9/88.6/95.6(+6.4%)
SST-2 | 60.9 | 91.5 | 93.0/88.9/95.6(+2.8%) | 92.4/94.8/96.4(+4.4%) | 93.4/91.9/96.9(+3.7%)
Fine Food | 51.1 | 66.8 | 78.5/77.7/84.9(+8.2%) | 74.9/80.0/80.7(+7.8%) | 82.6/86.3/87.3(+5.7%)
Table 2:  AUROC (%) on various OOD detection scenarios. The reported results
are averaged over three trials, and the best results are highlighted in bold.
Bracket denotes the relative gain of MASKER over the vanilla model.

Method | Classifier | Keyword | MKR | MER | Dataset (Train $\to$ Test)
---|---|---|---|---|---
IMDB $\to$ SST-2 |  IMDB $\to$ Food |  SST-2 $\to$ IMDB |  SST-2 $\to$ Food |  Food $\to$ SST-2 |  Food $\to$ IMDB
OpenMax | - | - | - | - |  79.55$\pm$0.78 (-8.12) |  75.41$\pm$1.20 (-12.25) |  75.30$\pm$0.44 (-7.61) |  62.19$\pm$3.06 (-20.72) |  61.85$\pm$0.63 (-31.70) |  67.50$\pm$1.50 (-26.04)
DOC | - | - | - | - |  77.90$\pm$1.22 (-10.06) |  78.33$\pm$1.52 (-9.64) |  76.88$\pm$0.70 (-6.23) |  64.47$\pm$2.52 (-18.63) |  62.00$\pm$0.86 (-31.27) |  67.31$\pm$1.28 (-25.96)
BERT | Multi-class | - | - | - |  85.92$\pm$1.92 (-7.57) |  92.90$\pm$2.47 (-0.60) |  85.74$\pm$0.56 (-6.74) |  87.57$\pm$1.13 (-4.91) |  67.55$\pm$5.27 (-28.92) |  77.31$\pm$2.09 (-19.16)
1-vs-rest | - | - | - |  84.28$\pm$0.23 (-8.92) |  87.81$\pm$3.91 (-5.39) |  85.34$\pm$0.63 (-7.46) |  84.35$\pm$1.48 (-8.45) |  64.57$\pm$1.27 (-32.15) |  81.34$\pm$0.78 (-12.16)
BERT +MASKER (ours) | 1-vs-rest | Random | ✓ | - |  87.29$\pm$1.48 (-6.29) |  90.52$\pm$1.28 (-3.06) |  86.57$\pm$0.87 (-7.60) |  78.00$\pm$0.86 (-16.17) |  78.79$\pm$1.15 (-17.51) |  84.56$\pm$1.59 (-12.91)
1-vs-rest | Random | - | ✓ |  86.84$\pm$1.76 (-5.97) |  90.27$\pm$1.25 (-2.54) |  87.18$\pm$0.81 (-8.52) |  85.91$\pm$1.04 (-9.79) |  79.50$\pm$0.46 (-17.03) |  84.61$\pm$0.41 (-11.92)
1-vs-rest | Frequency | ✓ | - |  86.52$\pm$1.40 (-6.04) |  88.41$\pm$1.72 (-4.15) |  87.06$\pm$1.14 (-7.37) |  79.99$\pm$1.88 (-14.44) |  74.72$\pm$1.90 (-23.73) |  80.94$\pm$2.80 (-17.51)
1-vs-rest | Frequency | - | ✓ |  86.38$\pm$0.88 (-6.72) |  84.20$\pm$2.59 (-8.90) |  85.31$\pm$2.31 (-13.55) |  88.43$\pm$1.85 (-10.43) |  75.34$\pm$1.54 (-20.66) |  85.34$\pm$0.99 (-10.66)
1-vs-rest | Attention | ✓ | - |  87.50$\pm$1.49 (-5.91) |  92.03$\pm$2.97 (-5.74) |  87.78$\pm$1.34 (-4.92) |  90.12$\pm$2.68 (-2.58) |  75.57$\pm$4.02 (-20.82) |  79.32$\pm$5.09 (-17.07)
1-vs-rest | Attention | - | ✓ |  87.71$\pm$0.71 (-5.59) |  90.39$\pm$0.39 (-2.63) |  84.92$\pm$2.52 (-7.64) |  87.21$\pm$1.01 (-5.35) |  75.80$\pm$1.84 (-20.87) |  82.13$\pm$2.39 (-14.54)
1-vs-rest | Attention | ✓ | ✓ |  88.02$\pm$1.31 (-5.44) |  93.58$\pm$2.63 (+0.12) |  88.43$\pm$0.38 (-3.89) |  89.21$\pm$0.40 (-3.11) |  80.02$\pm$1.52 (-16.44) |  85.57$\pm$0.28 (-10.90)

Table 3:  Ablation study on cross-domain generalization under sentiment
analysis task. The reported results are averaged over five trials, subscripts
denote standard deviations, bracketed numbers denote the generalization gap
from the training domain accuracy, and the best accuracies are highlighted in
bold. All components of our method contribute to the cross-domain accuracy
(%).

Train | Test | OpenMax | DOC | Vanilla / Residual / MASKER
---|---|---|---|---
BERT | RoBERTa | ALBERT
IMDB | IMDB | 87.7 | 88.0 | 93.5/92.8/93.5(+x.x%) | 95.3/94.0/95.6(+x.x%) | 91.6/91.4/90.8(+x.x%)
SST-2 | 79.6 | 77.9 | 85.9/86.9/88.1(+2.6%) | 89.7/90.2/91.8(+2.3%) | 89.8/89.0/89.9(+0.1%)
Fine Food | 75.4 | 78.3 | 92.9/92.5/93.6(+0.8%) | 92.6/87.7/93.0(+0.4%) | 87.1/87.8/92.1(+5.7%)
SST-2 | SST-2 | 82.9 | 83.1 | 92.5/90.3/92.3(+x.x%) | 94.5/92.0/94.3(+x.x%) | 91.9/90.9/91.5(+x.x%)
IMDB | 75.3 | 76.9 | 85.7/86.2/88.4(+3.2%) | 86.0/85.7/87.3(+1.5%) | 83.0/83.0/83.8(+1.0%)
Fine Food | 62.2 | 64.5 | 87.6/88.2/89.2(+1.8%) | 86.5/87.8/89.6(+3.6%) | 79.0/80.6/84.9(+7.5%)
Fine Food | Fine Food | 93.6 | 93.3 | 96.5/94.8/96.5(+xx.x%) | 96.9/95.7/97.1(+x.x%) | 95.5/95.4/96.6(+xx.x%)
IMDB | 67.5 | 67.3 | 77.3/81.1/85.6(+10.7%) | 84.1/84.6/86.6(+2.9%) | 74.7/80.4/84.8(+13.5%)
SST-2 | 61.9 | 62.0 | 67.6/67.8/80.0(+18.3%) | 78.5/80.1/83.8(+6.8%) | 71.7/73.2/83.3(+16.2%)
(a) Sentiment analysis

Model | Telephone
---|---
Telephone (ID) |  Letters (OOD) |  Face-to-Face (OOD)
BERT | 80.5 | 77.4 | 77.5
+MASKER | 80.2 |  80.4 (+3.9%) |  78.5 (+1.3%)
RoBERTa | 84.8 | 83.6 | 83.5
+MASKER | 86.0 |  85.9 (+2.8%) |  83.8 (+0.4%)
ALBERT | 82.9 | 82.8 | 80.9
+MASKER | 82.5 |  84.0 (+1.5%) |  86.8 (+7.3%)
(b) Natural language inference

MSRvid | Images
---|---
MSRvid (ID) |  Images (OOD) |  Headlines (OOD) |  Images (ID) |  MSRvid (OOD) |  Headlines (OOD)
91.5 | 82.0 | 61.7 | 88.0 | 89.7 | 73.9
91.2 |  84.3 (+2.8%) |  66.7 (+8.1%) | 88.1 |  91.6 (+2.1%) |  75.3 (+1.9%)
94.2 | 88.0 | 80.3 | 91.8 | 92.9 | 84.1
93.7 |  88.0 (+0.0%) |  84.0 (+4.6%) | 91.3 |  94.1 (+1.3%) |  85.3 (+1.4%)
92.6 | 81.2 | 60.6 | 90.4 | 90.9 | 69.8
93.3 |  82.6 (+1.7%) |  68.8 (+13.5%) | 90.5 |  92.0 (+1.2%) |  78.4 (+12.3%)
(c) Semantic textual similarity

Table 4:  Accuracy (%) of original domain and cross-domain on (a) sentiment
analysis, (b) natural language inference, and (c) semantic textual similarity
tasks, respectively. The reported results are averaged over three trials for
sentiment analysis and semantic textual similarity, and a single trial for
natural language inference. Bold denotes the best results among the three
methods, and bracket denotes the relative gain of MASKER over the vanilla
model.

##  3 Experiments

We demonstrate the effectiveness of our proposed method, MASKER. In Section
3.1, we describe the experimental setup. In Section 3.2 and 3.3, we present
the results on OOD detection and cross-domain generalization, respectively.

###  3.1 Experimental setup

We demonstrate the effectiveness of MASKER, applied to the pre-trained models:
BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019) and ALBERT (Lan et al.
2020). We choose $10\times C$ keywords in a class agnostic way, where $C$ is
the number of classes. We drop the keywords and contexts with probability
$p=0.5$ and $q=0.9$ for all our experiments. We use
$\lambda_{\mathtt{MKR}}=0.001$ and $\lambda_{\mathtt{MER}}=0.001$ for OOD
detection, and same $\lambda_{\mathtt{MKR}}=0.001$ but
$\lambda_{\mathtt{MER}}=0.0001$ for cross-domain generalization, as the
entropy regularization gives more gain for reliability than accuracy (Pereyra
et al. 2017). We modify the hyperparameter settings of the pre-trained models
(Devlin et al. 2019; Liu et al. 2019), specified in Appendix A.

1-vs-rest classifier. Complementary to MASKER, we use 1-vs-rest classifier
(Shu, Xu, and Liu 2017) as it further improves the reliability (see Table 1
and Table 3). Intuitively, 1-vs-rest classifier can reject all classes (all
prediction scores are low); hence detect OOD samples well.

Baselines. We mainly compare MASKER with vanilla fine-tuning of pre-trained
models (Hendrycks et al. 2020), with extensive ablation study (see Table 1 and
Table 3). Additionally, we compare with residual ensemble (Clark, Yatskar, and
Zettlemoyer 2019), applied to the same pre-trained models. Residual ensemble
trains a debiased model by fitting the residual from a biased model. We
construct a biased dataset by subsampling the documents that contain keywords.
To benchmark the difficulty of the task, we also report the classic non-
Transformer models, e.g., one-class support vector machine (OC-SVM, Schölkopf
et al. (2000)), OpenMax (Bendale and Boult 2016), and DOC (Shu, Xu, and Liu
2017).

###  3.2 OOD Detection

We use the highest softmax (or sigmoid) output of the model as confidence
score for OOD detection task. We use 20 Newsgroups (Lang 1995) and Amazon 50
class reviews (Chen and Liu 2014) datasets for in-distribution, and Reuters
(Lewis et al. 2004), IMDB (Maas et al. 2011), and SST-2 (Socher et al. 2013)
datasets for out-of-distribution.

Table 1 shows an ablation study on MASKER under the Amazon reviews dataset
with a split ratio of 25%. All components of MASKER contribute to OOD
detection. Note that MASKER does not degrade the classification accuracy while
improving OOD detection. Also, the attention-based selection performs better
than the frequency-based or random selection, which implies the importance of
selecting suitable keywords. Recall that the attention-based scheme selects
the keywords that contribute to the prediction, while the frequency-based
scheme often chooses domain-specific keywords that are not generalizable
across domains.

Table 2 shows the results on various OOD detection scenarios, comparing the
vanilla fine-tuning, residual ensemble, and MASKER. Notably, MASKER shows the
best results in all cases. In particular, MASKER improves the area under
receiver operating characteristic (AUROC) from 87.0% to 98.6% on 20 Newsgroups
to SST-2 task. We find that residual ensemble shows inconsistent gains: it
often shows outstanding results (e.g., Newsgroup to SST-2) but sometimes fails
(e.g., Amazon to Fine Food). In contrast, MASKER shows consistent improvement
over the vanilla fine-tuning.

In Figure 5(a) and Figure 5(b), we visualize the t-SNE (Maaten and Hinton
2008) plots on the document embeddings of BERT and MASKER, under the Amazon
reviews dataset with a split ratio of 25%. Blue and red points indicate in-
and out-of-distribution samples, respectively. Unlike the samples that are
entangled in the vanilla BERT, MASKER clearly distinguishes the OOD samples.

(a) BERT

(b) MASKER (ours)

(c) BERT

(d) MASKER (ours)

Figure 5:  t-SNE plots on the document embeddings of BERT and MASKER, on (a,b)
OOD detection (Amazon 50 class reviews with split ratio 25%), and (c,d) cross-
domain generalization (Fine Food to SST-2). (a,b) Blue and red dots indicate
the in- and out-of-distribution samples, respectively. (c,d) Blue and red dots
indicate the samples from the same classes (‘negative’) from training and test
domains, respectively. MASKER better distinguishes OOD samples and entangles
cross-domain samples.

###  3.3 Cross-domain Generalization

We conduct the experiments on sentiment analysis (IMDB (Maas et al. 2011);
SST-2 (Socher et al. 2013); Fine Food (McAuley and Leskovec 2013)), natural
language inference (MNLI, Williams, Nangia, and Bowman (2017)), and semantic
textual similarity (STS-B, Wang et al. (2019) dataset) tasks, following the
settings of Hendrycks et al. (2020).

Table 3 shows an ablation on MASKER study under the sentiment analysis task.
The results are consistent with OOD detection, e.g., all components contribute
to cross-domain generalization. Notably, while MER is not helpful for the
original domain accuracy (see Table 1), it improves the cross-domain accuracy
for most settings. In particular, MASKER improves the cross-domain accuracy
from 75.6% to 80.0% for Fine Food to SST-2 task. We analyze the most
influential keywords (see Appendix D) and find that MASKER extracts the
sentiment-related (i.e., generalizable) keywords (e.g., ‘astonishing’) while
the vanilla BERT is biased to some domain-specific words (e.g., ‘moonlight’).

Table 4 presents the results on sentiment analysis, natural language
inference, and semantic textual similarity tasks. We compare MASKER with the
vanilla fine-tuning and residual ensemble. The residual ensemble helps cross-
domain generalization, but the gain is not significant and often degrades the
original domain accuracy. This is because the keywords can be useful features
for classification. Hence, naiv̈ely removing (or debiasing) those features may
lose the information. In contrast, MASKER facilitates contextual information
rather than removing the keyword information, which regularizes the over-
reliance in a softer manner.

In Figure 5(c) and Figure 5(d), we provide the t-SNE plots on the document
embeddings of BERT and MASKER, under the Fine Food to STS-2 task. Blue and red
points indicate original and cross-domain samples, respectively. MASKER better
entangles the same classes in training and test datasets (of the different
domains) while BERT fails to do so.

##  4 Conclusion

The reliability of text classifiers is an essential but under-explored
problem. We found that the over-reliance on some keywords can be problematic
for out-of-distribution detection and generalization. We propose a simple yet
effective fine-tuning method, coined masked keyword regularization (MASKER),
composed of two regularizers and keyword selection schemes to address this
issue. We demonstrate the effectiveness of MASKER under various scenarios.

## Acknowledgements

This work was supported by Center for Applied Research in Artificial
Intelligence(CARAI) grant funded by Defense Acquisition Program
Administration(DAPA) and Agency for Defense Development(ADD) (UD190031RD).

## References

  * Aggarwal and Zhai (2012) Aggarwal, C. C.; and Zhai, C. 2012.  A survey of text classification algorithms.  In _Mining text data_ , 163–222. Springer.
  * Agirre et al. (2012) Agirre, E.; Cer, D.; Diab, M.; and Gonzalez-Agirre, A. 2012.  Semeval-2012 task 6: A pilot on semantic textual similarity.  In _* SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , 385–393.
  * Agrawal et al. (2018) Agrawal, A.; Batra, D.; Parikh, D.; and Kembhavi, A. 2018.  Don’t just assume; look and answer: Overcoming priors for visual question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4971–4980.
  * Bakshi et al. (2016) Bakshi, R. K.; Kaur, N.; Kaur, R.; and Kaur, G. 2016.  Opinion mining and sentiment analysis.  In _2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom)_ , 452–455. IEEE.
  * Bendale and Boult (2016) Bendale, A.; and Boult, T. E. 2016.  Towards open set deep networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1563–1572.
  * Bhatt, Semwal, and Roy (2015) Bhatt, H. S.; Semwal, D.; and Roy, S. 2015.  An Iterative Similarity based Adaptation Technique for Cross-domain Text Classification.  In _Proceedings of the Nineteenth Conference on Computational Natural Language Learning_ , 52–61. Beijing, China: Association for Computational Linguistics.  doi:10.18653/v1/K15-1006.  URL https://www.aclweb.org/anthology/K15-1006.
  * Bhatt, Sinha, and Roy (2016) Bhatt, H. S.; Sinha, M.; and Roy, S. 2016.  Cross-domain Text Classification with Multiple Domains and Disparate Label Sets.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1641–1650. Berlin, Germany: Association for Computational Linguistics.  doi:10.18653/v1/P16-1155.  URL https://www.aclweb.org/anthology/P16-1155.
  * Bowman et al. (2015) Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 632–642.
  * Chen and Liu (2014) Chen, Z.; and Liu, B. 2014.  Mining topics in documents: standing on the shoulders of big data.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 1116–1125.
  * Choi et al. (2018) Choi, E.; He, H.; Iyyer, M.; Yatskar, M.; Yih, W.-t.; Choi, Y.; Liang, P.; and Zettlemoyer, L. 2018.  Quac: Question answering in context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2174–2184.
  * Clark, Yatskar, and Zettlemoyer (2019) Clark, C.; Yatskar, M.; and Zettlemoyer, L. 2019.  Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 4069–4082.
  * Clark et al. (2020) Clark, K.; Luong, M.-T.; Le, Q. V.; and Manning, C. D. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186.
  * DeVries and Taylor (2017) DeVries, T.; and Taylor, G. W. 2017.  Improved regularization of convolutional neural networks with cutout.  _arXiv preprint arXiv:1708.04552_ .
  * Fei and Liu (2015) Fei, G.; and Liu, B. 2015.  Social media text classification under negative covariate shift.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2347–2356.
  * Geirhos et al. (2020) Geirhos, R.; Jacobsen, J.-H.; Michaelis, C.; Zemel, R.; Brendel, W.; Bethge, M.; and Wichmann, F. A. 2020.  Shortcut Learning in Deep Neural Networks.  _arXiv preprint arXiv:2004.07780_ .
  * Hendrycks and Gimpel (2017) Hendrycks, D.; and Gimpel, K. 2017.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks.  In _International Conference on Learning Representations_.
  * Hendrycks et al. (2020) Hendrycks, D.; Liu, X.; Wallace, E.; Dziedzic, A.; Krishnan, R.; and Song, D. 2020\.  Pretrained Transformers Improve Out-of-Distribution Robustness.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2744–2751.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_.
  * Lan et al. (2020) Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 2020.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_.
  * Lang (1995) Lang, K. 1995.  Newsweeder: Learning to filter netnews.  In _Machine Learning Proceedings 1995_ , 331–339. Elsevier.
  * Lewis et al. (2004) Lewis, D. D.; Yang, Y.; Rose, T. G.; and Li, F. 2004.  Rcv1: A new benchmark collection for text categorization research.  _Journal of machine learning research_ 5(Apr): 361–397.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ .
  * Maas et al. (2011) Maas, A. L.; Daly, R. E.; Pham, P. T.; Huang, D.; Ng, A. Y.; and Potts, C. 2011\.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , 142–150.
  * Maaten and Hinton (2008) Maaten, L. v. d.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9(Nov): 2579–2605.
  * Marasović (2018) Marasović, A. 2018.  NLP’s generalization problem, and how researchers are tackling it.  _The Gradient_ .
  * McAuley and Leskovec (2013) McAuley, J. J.; and Leskovec, J. 2013.  From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews.  In _Proceedings of the 22nd international conference on World Wide Web_ , 897–908.
  * McCoy, Pavlick, and Linzen (2019) McCoy, T.; Pavlick, E.; and Linzen, T. 2019.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 3428–3448.
  * Min et al. (2019) Min, S.; Wallace, E.; Singh, S.; Gardner, M.; Hajishirzi, H.; and Zettlemoyer, L. 2019.  Compositional questions do not necessitate multi-hop reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 4249–4257.
  * Minderer et al. (2020) Minderer, M.; Bachem, O.; Houlsby, N.; and Tschannen, M. 2020.  Automatic Shortcut Removal for Self-Supervised Representation Learning.  In _International Conference on Machine Learning_.
  * Mosbach, Andriushchenko, and Klakow (2020) Mosbach, M.; Andriushchenko, M.; and Klakow, D. 2020.  On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines.  _arXiv preprint arXiv:2006.04884_ .
  * Nam et al. (2020) Nam, J.; Cha, H.; Ahn, S.; Lee, J.; and Shin, J. 2020.  Learning from Failure: Training Debiased Classifier from Biased Classifier.  _arXiv preprint arXiv:2007.02561_ .
  * Niven and Kao (2019) Niven, T.; and Kao, H.-Y. 2019.  Probing neural network comprehension of natural language arguments.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 4658–4664.
  * Pereyra et al. (2017) Pereyra, G.; Tucker, G.; Chorowski, J.; Kaiser, Ł.; and Hinton, G. 2017.  Regularizing neural networks by penalizing confident output distributions.  In _ICLR Workshop_.
  * Reddy, Chen, and Manning (2019) Reddy, S.; Chen, D.; and Manning, C. D. 2019.  Coqa: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ 7: 249–266.
  * Robertson (2004) Robertson, S. 2004.  Understanding inverse document frequency: on theoretical arguments for IDF.  _Journal of documentation_ .
  * Sanh et al. (2019) Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  In _5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019_.
  * Schölkopf et al. (2000) Schölkopf, B.; Williamson, R. C.; Smola, A. J.; Shawe-Taylor, J.; and Platt, J. C. 2000.  Support vector method for novelty detection.  In _Advances in neural information processing systems_ , 582–588.
  * Shu, Xu, and Liu (2017) Shu, L.; Xu, H.; and Liu, B. 2017.  Doc: Deep open classification of text documents.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2911–2916.
  * Socher et al. (2013) Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A. Y.; and Potts, C. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , 1631–1642.
  * Sun et al. (2019) Sun, C.; Qiu, X.; Xu, Y.; and Huang, X. 2019.  How to fine-tune bert for text classification?  In _China National Conference on Chinese Computational Linguistics_ , 194–206. Springer.
  * Tack et al. (2020) Tack, J.; Mo, S.; Jeong, J.; and Shin, J. 2020.  CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances.  In _Advances in Neural Information Processing Systems_.
  * Tan et al. (2019) Tan, M.; Yu, Y.; Wang, H.; Wang, D.; Potdar, S.; Chang, S.; and Yu, M. 2019.  Out-of-Domain Detection for Low-Resource Text Classification Tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 3566–3572.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , 5998–6008.
  * Wang et al. (2019) Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _International Conference on Learning Representations_.
  * Williams, Nangia, and Bowman (2017) Williams, A.; Nangia, N.; and Bowman, S. R. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_ .
  * Zhang et al. (2020) Zhang, T.; Wu, F.; Katiyar, A.; Weinberger, K. Q.; and Artzi, Y. 2020.  Revisiting Few-sample BERT Fine-tuning.  _arXiv preprint arXiv:2006.05987_ .

##  Appendix A Experimental Details

###  A.1 Training Details

Following Devlin et al. (2019); Liu et al. (2019), we select the best
hyperparameters from the search space below. We choose learning rate from
{$1\mathrm{e}{-5}$, $2\mathrm{e}{-5}$, $5\mathrm{e}{-5}$} and batch size from
{$16$,$32$}. We halve the learning rate for the embedding layers of MASKER
since the regularizer for fits to the classifier, and directly updating the
embedding layers can be unstable. We also use the batch size of 4 for random
word reconstruction due to the large vocabulary size. We use the Adam (Kingma
and Ba 2015) optimizer for all experiments. We train vanilla BERT and ALBERT
for 3$\sim$4 epochs, and RoBERTa for 10 epochs following Devlin et al. (2019)
and Liu et al. (2019), respectively. For MASKER, we train BERT+MASKER,
ALBERT+MASKER for 6$\sim$8 epochs, and train RoBERTa+MASKER for 12 epochs. We
remark that all the models are trained until convergence. Since MER cannot be
directly applied to the regression tasks (e.g., STS-B), we only use MKR for
such settings.

###  A.2 Dataset Details

We use the pre-defined train and test splits if they exists: for IMDB (Maas et
al. 2011), SST-2 (Socher et al. 2013), MNLI (Williams, Nangia, and Bowman
2017), and STS-B (Wang et al. 2019) datasets. If pre-defined splits not
exists, we randomly divide the dataset with 70:30 ratio, using them for train
and test splits, respectively: for 20 Newsgroups (Lang 1995), Amazon 50 class
reviews (Chen and Liu 2014), and Fine Food (McAuley and Leskovec 2013)
datasets. We do not use any pre-processing methods, e.g., removing headers.

###  A.3 Evaluation Metrics

Let TP, TN, FP, and FN denotes true positive, true negative, false positive,
and false negative, respectively. We use the following metrics for OOD
detection:

  * •

Area under the receiver operating characteristic curve (AUROC). The ROC curve
is a graph plotting true positive rate (TPR) = TP / (TP+FN) against the false
positive rate (FPR) = FP / (FP+TN) by varying a threshold. AUROC measures the
area under the ROC curve.

  * •

Equal error rate (EER). EER is the error rate when the confidence threshold is
located where FPR is the same with the false negative rate (FNR) = FN /
(TP+FN).

  * •

Detection accuracy. Measures the maximum classification probability among all
possible threshold sets.

  * •

TNR at TPR 80%. Measures true negative rate (TNR) = TN / (FP+TN) when TPR =
80%.

AUROC measures the overall performance varying thresholds, and the other three
metrics measure the performance for some fixed threshold.

##  Appendix B Overall Procedure of MASKER

Figure 7 visualizes the overall procedure of MASKER, including the keyword
selection scheme (attention-based selection using a vanilla model), masked
keyword reconstruction, and masked entropy regularization.

##  Appendix C Additional Experimental Results

Table 5 presents the additional OOD detection results, including the split
settings of 20 Newsgroups and Amazon reviews datasets (Shu, Xu, and Liu 2017).
Our method consistently outperforms the baseline methods. Table 6 presents the
classification accuracy of the baselines and MASKER, which validates that
MASKER does not degrade the accuracy. Table 7 presents the other cross-domain
results under the STS-B dataset, shows the effectiveness of our method. We
also try to regularize attention weights to be uniform directly, but it harms
both classification accuracy and OOD detection performance for vanilla models.

##  Appendix D Analysis on Attention Scores

In Figure 6, we visualize the most influential keywords measured by the
attention scores. Our method makes predictions based on the generalizable
keywords, e.g., sentiment-related keywords for sentiment analysis tasks.

Figure 6:  Top 10 keywords according to the attention scores, chosen by BERT
and ours trained on Fine Food and tested on SST-2. The sentiment-related
keywords are highlighted.

Figure 7:  The overall procedure of MASKER using the attention-based keywords.
Vanilla training and our proposed method are presented as the red and blue
boxes, respectively. Parameter sharing without back-propagation is presented
as a dashed arrow.

ID | OOD | Split Ratio | AUROC $\uparrow$ | EER $\downarrow$ |  Detection Accuracy $\uparrow$ |  TNR at TPR 80% $\uparrow$ |  Classification Accuracy $\uparrow$
---|---|---|---|---|---|---|---
|  |  | DOC / BERT / BERT+MAKSER (ours)
Newsgroup | Newsgroup | 10% | 83.7/85.4/87.0 | 23.1/22.9/21.0 | 81.0/90.9/91.5 | 61.0/67.4/75.2 | 98.7/99.0/98.9
25% | 86.1/89.0/91.0 | 18.7/19.1/17.0 | 82.7/83.0/86.9 | 78.5/80.5/84.8 | 93.9/95.3/94.9
50% | 80.4/82.4/83.0 | 28.2/25.4/25.0 | 72.3/75.0/76.8 | 65.0/68.0/71.8 | 89.9/94.4/94.0
Amazon | 100% | 84.1/86.0/96.8 | 23.3/19.5/8.3 | 90.1/87.2/94.7 | 74.5/86.7/95.5 | 86.9/90.4/90.1
Reuter | 60.0/91.8/97.7 | 41.1/14.7/6.7 | 75.2/85.7/93.6 | 21.3/84.5/96.4
IMDB | 88.6/94.6/98.5 | 19.1/11.5/5.1 | 88.4/93.4/96.3 | 81.7/87.7/98.2
SST-2 | 88.1/87.0/98.6 | 18.7/18.9/5.1 | 86.6/88.5/96.0 | 81.8/92.4/98.4
Fine Food | 81.3/85.3/93.4 | 25.7/19.8/10.9 | 74.8/82.7/90.5 | 67.6/85.9/95.2
Amazon | Amazon | 10% | 80.7/82.8/86.0 | 22.9/22.1/21.0 | 79.2/92.3/93.0 | 74.0/75.7/80.1 | 92.7/93.5/93.3
25% | 75.1/78.5/85.5 | 25.6/28.3/22.3 | 80.2/81.2/85.0 | 66.0/66.3/74.1 | 83.1/84.8/85.2
50% | 74.6/75.4/77.7 | 29.0/28.9/27.9 | 69.4/70.2/71.1 | 60.0/60.7/61.5 | 78.8/81.7/80.6
Newsgroup | 100% | 81.3/84.8/87.2 | 23.0/21.6/20.0 | 80.0/81.5/83.5 | 72.7/77.0/80.4 | 66.6/70.8/70.0
Reuter | 79.8/89.7/93.5 | 30.8/18.1/12.2 | 82.5/87.1/89.9 | 60.0/83.2/90.7
IMDB | 89.6/93.3/95.2 | 18.1/13.9/10.5 | 82.4/87.0/90.7 | 83.0/89.2/92.9
SST-2 | 91.5/93.0/95.6 | 15.8/14.1/9.5 | 84.5/89.6/92.9 | 87.1/88.2/93.9
Fine Food | 66.8/78.5/84.9 | 38.0/30.0/19.5 | 69.7/73.9/80.7 | 55.3/62.7/80.8
Table 5:  AUROC (%) on various additional OOD detection scenarios. The reported results are averaged over three trials, and the best results are highlighted in bold. MASKER outperforms the baselines in all cases.  Dataset | Openmax | DOC | Vanilla/Residual/MASKER
---|---|---|---
BERT | RoBERTa | ALBERT
Newsgroups | 85.8 | 86.9 | 90.4/89.5/90.1 | 90.9/88.1/90.7 | 89.3/89.7/89.7
Amazon | 63.0 | 66.6 | 70.8/70.3/70.0 | 71.0/69.1/71.2 | 68.7/64.2/68.6
Table 6:  Classification accuracy (%) of the datasets used for OOD detection. MASKER shows a comparable accuracy with the vanilla models, while residual ensemble shows a marginal drop.  Model | MSRvid | Images
---|---|---
MSRvid | Images | MSRpar | Headlines | Images | MSRvid | MSRpar | Headlines
BERT | 91.5 | 82.0 | 38.2 | 61.7 | 88.0 | 89.7 | 50.8 | 73.9
+MASKER | 91.2 | 84.3 | 40.9 | 66.7 | 88.1 | 91.6 | 52.5 | 75.3
RoBERTa | 94.2 | 88.0 | 66.0 | 80.3 | 91.8 | 92.9 | 68.4 | 84.1
+MASKER | 93.7 | 88.0 | 67.1 | 84.0 | 91.3 | 94.1 | 70.1 | 85.3
ALBERT | 92.6 | 81.2 | 39.4 | 60.6 | 90.4 | 90.9 | 44.9 | 69.8
+MASKER | 93.3 | 82.6 | 39.8 | 68.8 | 90.5 | 92.0 | 45.2 | 78.4
Model | Headlines | MSRpar
Headlines | MSRvid | Images | MSRpar | MSRpar | MSRvid | Images | Headlines
BERT | 86.1 | 83.2 | 81.1 | 69.9 | 74.2 | 74.1 | 71.9 | 67.1
+MASKER | 86.8 | 88.0 | 83.6 | 75.8 | 77.6 | 79.1 | 75.9 | 67.7
RoBERTa | 90.7 | 93.3 | 90.1 | 75.5 | 86.4 | 88.2 | 85.8 | 85.4
+MASKER | 88.2 | 90.3 | 90.7 | 70.9 | 84.8 | 90.2 | 85.9 | 86.4
ALBERT | 86.8 | 89.3 | 87.1 | 63.5 | 78.5 | 82.4 | 80.8 | 69.2
+MASKER | 87.0 | 90.4 | 87.1 | 67.5 | 76.7 | 82.7 | 81.7 | 75.8
Table 7:  Total Pearson correlation (%) of four genres in the STS-B dataset.
The reported results are averaged over 3 trials.


# Further Improvements to the Chevalley-Warning Theorems

David B. Leep  Department of Mathematics, University of Kentucky, Lexington,
KY, 40506<EMAIL_ADDRESS>and  Rachel L. Petrik  Department of Mathematics,
Rose-Hulman Institute of Technology, Terre Haute, IN, 47803  petrik@rose-
hulman.edu

###### Abstract.

We study lower bound estimates for the number of solutions of systems of
equations over finite fields. Heath-Brown improved the lower bounds given by
the classical _Chevalley-Warning Theorems_ by excluding systems of equations
whose solutions form an affine space. We improve each of Heath-Brown’s results
and demonstrate sharpness in several cases.

###### Key words and phrases:

Chevalley-Warning Theorem, Finite Fields, Homogeneous Forms, Number of Zeros

###### 2010 Mathematics Subject Classification:

11E76, 11T06

Acknowledgements here.

##  1\. Introduction

The goal of this paper is to study lower bounds on the number of solutions to
systems of polynomial equations over finite fields. Given a general system of
polynomials, a classical question has been to determine the existence of a
solution. For systems over finite fields, the key results on the existence of
solutions are the _Chevalley–Warning theorems_ [2, 4] (see Theorem 1.1 below).
In particular, Warning provided a lower bound in [4] for the number of
solutions. Easy examples exist to show that Warning’s result is best possible.
Avoiding these easy cases, Heath-Brown [3] (see Theorem 1.3) improved the
lower bound given in [4]. In this paper, we improve each of Heath-Brown’s
lower bounds (see Theorem 1.4) by separating out the various cases more
clearly and computing the estimates more precisely. In addition, we simplify
the proof of Theorem 1.3 (2) for $q\geq 4$ and extend it to $q=3$. In Section
4, we provide examples showing many of our bounds are optimal.

Let $\mathbb{F}_{q}$ denote the finite field of order $q=p^{k}$, where $p$ is
a prime and $k\in\mathbb{Z}_{>0}$. We write
$\textit{{f}}=\\{f_{1},\dots,f_{r}\\}$ to denote a system of polynomials in
$\mathbb{F}_{q}[x_{1},\dots,x_{n}]$. Let $d_{i}=\deg(f_{i})$ for $1\leq i\leq
r$. We define the degree of f to be $d\coloneqq d_{1}+\dots+d_{r}$. Let
$Z(\textit{{f}},\mathbb{F}_{q}^{n})$ be the set of zeros of f over
$\mathbb{F}_{q}$ and let
$N(\textit{{f}},\mathbb{F}_{q}^{n})=|Z(\textit{{f}},\mathbb{F}_{q}^{n})|$.

######  Theorem 1.1.

Let $f_{1},\dots,f_{r}\in\mathbb{F}_{q}[x_{1},\dots,x_{n}]$,
$\deg(f_{i})=d_{i}$, $1\leq i\leq r$. Assume $n>d_{1}+d_{2}+\cdots+d_{r}$.
Then the following statements hold.

  1. (1)

_(Chevalley,[2])_ If $Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is nonempty, then
$N(\textit{{f}},\mathbb{F}_{q}^{n})\geq 2$.

  2. (2)

_(Warning,[4])_ $N(\textit{{f}},\mathbb{F}_{q}^{n})\equiv 0\bmod{p}$.

  3. (3)

_(Warning,[4])_ If $Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is nonempty, then
$N(\textit{{f}},\mathbb{F}_{q}^{n})\geq q^{n-d}$.

  4. (4)

_(Ax,[1])_ $N(\textit{{f}},\mathbb{F}_{q}^{n})\equiv 0\bmod{q}$.

A polynomial $f$ in $n$ variables is defined over $\mathbb{F}_{q}$ if
$f\in\mathbb{F}_{q}[x_{1},\dots,x_{n}]$. A polynomial
$f\in\mathbb{F}_{q}[x_{1},\dots,x_{n}]$ is a _(homogeneous) form of degree_
$d$, for some $d\geq 0$, if $f\neq 0$ and each monomial in $f$ has degree $d$.
We say that $(a_{1},\dots,a_{n})\in\mathbb{F}_{q}^{n}$ is a zero of f if
$f_{i}(a_{1},\dots,a_{n})=0$, $1\leq i\leq r$. We say f is a _homogeneous
system_ if each $f_{i}$ is a homogeneous form. If f is a homogeneous system
with $\deg(f_{i})\geq 1$ for each $i$, then $f_{i}(0,\dots,0)=0$ for each $i$.
A zero $(a_{1},\dots,a_{n})\in\mathbb{F}_{q}^{n}$ of a homogeneous system f is
a _nontrivial zero_ if $(a_{1},\dots,a_{n})\neq(0,\dots,0)$.

If $f_{1},\dots,f_{r}$ are homogeneous forms and $n>d$, then the system has a
nontrivial zero by Theorem 1.1 (1) and, therefore, there are at least
$q^{n-d}\geq 2$ zeros by Theorem 1.1 (3). If $P$ is the number of projective
zeros defined over $\mathbb{F}_{q}$, then
$N(\textit{{f}},\mathbb{F}_{q}^{n})=P(q-1)+1$. This implies

(1) $N(\textit{{f}},\mathbb{F}_{q}^{n})\equiv 1\bmod q-1.$

Further improvements require additional hypotheses, as examples exist (e.g.
norm forms) that show the bound in Theorem 1.1 (3) is best possible. In these
examples the set of zeros forms a subspace of dimension $n-d$ in
$\mathbb{F}_{q}^{n}$. In fact, Heath-Brown showed (see Theorem 1.3 below) that
all examples where Theorem 1.1 (3) is sharp have the property that the set of
zeros forms an affine space of $\mathbb{F}_{q}^{n}$.

######  Definition 1.2.

An affine space is a coset of a subspace. We say that two affine spaces are
parallel if they are cosets of the same subspace. The dimension of an affine
space is the dimension of the corresponding subspace. We let
$\mathbb{A}^{t}(\mathbb{F}_{q})$ denote any affine space of dimension $t$ over
$\mathbb{F}_{q}$.

Since we work entirely in $\mathbb{F}_{q}^{n}$, it follows that
$\mathbb{A}^{n}(\mathbb{F}_{q})=\mathbb{F}_{q}^{n}$. When
$Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is not an affine space, Heath-Brown ([3])
provided the following improvement to Theorem 1.1. If $d=1$, then
$Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is always an affine space, which is
excluded. Thus, without loss of generality, we can always assume $d\geq 2$.

######  Theorem 1.3 (Heath–Brown, [3], Theorem 2).

Suppose that $n>d$, $Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is non-empty, and
$Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is not an affine space of
$\mathbb{F}_{q}^{n}$. Then the following statements hold.

  1. (1)

For any $q$, we have $N(\textit{{f}},\mathbb{F}_{q}^{n})>q^{n-d}$.

  2. (2)

If $q\geq 4$, we have $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq 2q^{n-d}$.

  3. (3)

For any $q$, we have
$N(\textit{{f}},\mathbb{F}_{q}^{n})\geq\dfrac{q^{n+1-d}}{(n+2-d)}$ provided
that the polynomials $f_{1},\dots,f_{r}$ are homogeneous.

The main result of this paper, Theorem 1.4, is to show that each part of
Theorem 1.3 can be improved as follows.

######  Theorem 1.4.

Suppose that $n>d$, $Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is non-empty, and
$Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is not an affine space of
$\mathbb{F}_{q}^{n}$. Then the following statements hold.

  1. (1)

For $q=2$, $N(\textit{{f}},\mathbb{F}_{2}^{n})\geq 2^{n-d}+2$.

  2. (2)

For $q\geq 3$, $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq 2q^{n-d}$.

  3. (3)

For $q\geq 2$,
$N(\textit{{f}},\mathbb{F}_{q}^{n})>\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}$
provided that the polynomials $f_{1},\dots,f_{r}$ are homogeneous.

  4. (4)

For $q\geq 3$, $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq 2q^{n-d}+(q-2)q$
provided that the polynomials $f_{1},\dots,f_{r}$ are homogeneous.

Moreover, the bounds in (1), (2), and (4) are sharp under these hypotheses.

In Section 4, we state more precisely our results on the sharpness of these
bounds. When $n>d$, the bound given in Theorem 1.4 (3) is better than the
result given in Theorem 1.3 (3) because

$\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}>q^{n-d}\cdot\dfrac{q}{n+2-d}=\dfrac{q^{n+1-d}}{n+2-d}.$

##  2\. Assumed Results from Heath-Brown’s Paper

We need the following results, which are proven in [3], to prove Theorem 1.4.

######  Theorem 2.1 (Heath–Brown, [3], Theorem 1).

For any two parallel affine spaces
$L_{1},L_{2}\subseteq\mathbb{A}^{n}(\mathbb{F}_{q})$ of dimension at least d,
we have

$N(\textit{{f}},L_{1})\equiv N(\textit{{f}},L_{2})\bmod{q}.$

######  Lemma 2.2 (Heath–Brown, [3], Lemma 1).

Let $L_{0}\subseteq\mathbb{A}^{n}(\mathbb{F}_{q})$ be an affine space. Choose
an affine space $L$ of maximal dimension $k$ such that $L\supseteq L_{0}$ and
$N(\textit{{f}},L)=N(\textit{{f}},L_{0})$. Suppose $L^{\prime}\supset L$ is an
affine space of dimension $k+1$ such that $N(\textit{{f}},L^{\prime})$ is
minimal. Then $N(\textit{{f}},L^{\prime})>N(\textit{{f}},L)$ and

$N(\textit{{f}},\mathbb{A}^{n}(\mathbb{F}_{q}))\geq
N(\textit{{f}},L)+\dfrac{q^{n-k}-1}{q-1}(N(\textit{{f}},L^{\prime})-N(\textit{{f}},L)).$

For the next lemma, we require the following definition.

######  Definition 2.3.

Let $\mathbb{K}$ be a field. A set of points in $\mathbb{K}^{t}$ is in general
position if no $n$ of them lie in an $(n-2)$-dimensional affine space for
$2\leq n\leq t+1$.

######  Lemma 2.4 (Heath–Brown, [3], Lemma 2).

Let $S\subseteq\mathbb{A}^{t}(\mathbb{F}_{q})$ be a set containing $t+1$
points in general position. Then the following statements hold.

  1. (1)

If $q=2$, and if there is no 2-dimensional affine space
$L\subseteq\mathbb{A}^{t}(\mathbb{F}_{q})$ meeting $S$ in exactly 3 points,
then $S=\mathbb{A}^{t}(\mathbb{F}_{q})$.

  2. (2)

If $q\geq 3$, and if $l\subseteq S$ for every line $l$ meeting $S$ in at least
two points, then $S=\mathbb{A}^{t}(\mathbb{F}_{q})$.

  3. (3)

If $q\geq 4$, and if $|S\cap l|$ $\geq q-1$ for every line $l$ meeting $S$ in
at least two points, then $\mathbb{A}^{t}(\mathbb{F}_{q})\setminus S$ is
contained in a hyperplane.

  4. (4)

If $m\geq 2$ is an integer, and if $|S\cap l|$ $\geq m+1$ for every line $l$
meeting $S$ in at least two points, then $|S|$ $\geq\dfrac{m^{t+1}-1}{m-1}$.

Our proof of Theorem 1.4 does not depend on Lemma 2.4 (3), the most difficult
part of Lemma 2.4, whereas the proof provided by Heath-Brown requires this
statement.

Heath-Brown did not explicitly state the following lemmas, however, the proofs
of these lemmas appear within the proof of his main result in [3]. We include
the proofs for the convenience of the reader.

######  Lemma 2.5 (Heath–Brown, [3], page 431).

Let $L$ be an affine space of dimension $d$. If $n\geq d$ and
$N(\textit{{f}},L)=v$ where $1\leq v\leq(q-1)$, then
$N(\textit{{f}},L^{*})\geq v$ for every affine space $L^{*}$ of dimension $d$
parallel to $L$. As a result, $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq
vq^{n-d}$.

###### Proof.

By Theorem 2.1, $N(\textit{{f}},L^{*})\equiv N(\textit{{f}},L)=v\bmod q$.
Since $1\leq v\leq(q-1)$, we must have $N(\textit{{f}},L^{*})\geq v$. Since
there are $q^{n-d}$ affine spaces $L^{*}$ of dimension $d$ parallel to $L$, we
have $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq vq^{n-d}$. ∎

######  Lemma 2.6 (Heath–Brown, [3], page 432).

Let $L\subseteq\mathbb{F}_{q}^{n}$ be an affine space. Suppose $2\leq
N(\textit{{f}},L)\leq q-1$ and $\deg(\textit{{f}})=d$.

  * (1)

Then $\dim(L)\leq d$.

  * (2)

If f is a homogeneous system, then $\dim(L)\leq d-1$.

###### Proof.

(1) If $\dim(L)>d$, then $q\mid N(\textit{{f}},L)$ by Theorem 1.1 (4) and
Theorem 2.1, which is a contradiction.

(2) By (1), assume $\dim(L)=d$. First suppose $\boldsymbol{0}\in L$ and thus,
$L$ is a subspace. Since f is a homogeneous system, we have $q-1\mid
N(\textit{{f}},L)-1$ by Equation (1). Thus, either $N(\textit{{f}},L)=1$ or
$N(\textit{{f}},L)\geq q$, which is a contradiction.

Now suppose $\boldsymbol{0}\notin L$. Let $L^{\prime}$ be the
$(d+1)$-dimensional subspace containing $L$ and $\boldsymbol{0}$. Since f is a
homogeneous system, distinct zeros of $Z(\textit{{f}},L)$ correspond
bijectively to distinct projective zeros of $Z(\textit{{f}},L^{\prime})$.
Therefore, $N(\textit{{f}},L^{\prime})=1+(q-1)N(\textit{{f}},L)$ by Equation
(1). Theorem 1.1 (4) implies that $q\mid N(\textit{{f}},L^{\prime})$. Thus,
$N(\textit{{f}},L)\equiv 1\bmod q$, and so either $N(\textit{{f}},L)=1$ or
$N(\textit{{f}},L)\geq q+1$, which is a contradiction. ∎

##  3\. Proof of Theorem 1.4

The following inequality is stated without proof in a slightly weaker form
(for $q\geq 2v+3$) in [3]. Recall that $\lfloor x\rfloor$ denotes the greatest
integer less than or equal to $x$.

######  Lemma 3.1.

Suppose $v\in\mathbb{Z}$. Let $v\geq 2$ and $q\geq 2v+1$, then
$\Big{\lfloor}\dfrac{qv}{v+1}\Big{\rfloor}^{v}>\dfrac{q^{v}}{v+1}$ with the
exception of $v=2$, $q=5,7$.

###### Proof.

First we demonstrate that it is sufficient to show

$v+1>\Big{(}1+\dfrac{1}{q-1}\Big{)}^{v}\Big{(}1+\dfrac{1}{v}\Big{)}^{v}.$

Suppose

$v+1>\Big{(}1+\dfrac{1}{q-1}\Big{)}^{v}\Big{(}1+\dfrac{1}{v}\Big{)}^{v}=\Big{(}\dfrac{q}{q-1}\Big{)}^{v}\Big{(}\dfrac{v+1}{v}\Big{)}^{v}.$

Taking the $v^{\text{th}}$ root of both sides yields

$(v+1)^{\frac{1}{v}}>\Big{(}\dfrac{q}{q-1}\Big{)}\Big{(}\dfrac{v+1}{v}\Big{)}.$

Rearranging these terms through multiplication, we find

$\dfrac{q}{(v+1)^{\frac{1}{v}}}<\dfrac{(q-1)v}{v+1}=\dfrac{qv}{v+1}-\dfrac{v}{v+1}\leq\Big{\lfloor}\dfrac{qv}{v+1}\Big{\rfloor},$

which gives the result of the lemma after raising both sides to the
$v^{\text{th}}$ power.

Assume $q\geq 2v+1$. Note that since $e^{y}>1+y$ for $y>0$, it follows that

(2) $e^{x}>\Big{(}1+\frac{x}{v}\Big{)}^{v},$

for all $x>0$, $v>0$. Then

$1+\dfrac{1}{q-1}\leq 1+\dfrac{1}{2v}=1+\dfrac{\frac{1}{2}}{v}$
$\Big{(}1+\dfrac{1}{q-1}\Big{)}^{v}\leq\Big{(}1+\dfrac{\frac{1}{2}}{v}\Big{)}^{v}<e^{\frac{1}{2}}.$

Applying (2)

$\displaystyle\Big{(}1+\dfrac{1}{q-1}\Big{)}^{v}\Big{(}1+\dfrac{1}{v}\Big{)}^{v}<e^{\frac{1}{2}}e=e^{\frac{3}{2}}<5\leq
v+1\text{ for }v\geq 4.$

Now let $v=3$. Then

$\Big{(}1+\dfrac{1}{q-1}\Big{)}^{v}\Big{(}1+\dfrac{1}{v}\Big{)}^{v}<e^{\frac{1}{2}}\Big{(}\dfrac{4}{3}\Big{)}^{3}=e^{\frac{1}{2}}\dfrac{64}{27}<3.91<4=v+1.$

Now let $v=2$. Notice that

$\displaystyle\Big{(}\dfrac{q}{q-1}\Big{)}^{v}\Big{(}1+\dfrac{1}{v}\Big{)}^{v}$
$\displaystyle=\dfrac{9}{4}\Big{(}\dfrac{q}{q-1}\Big{)}^{2}.$

We want the values of $q$ satisfying the inequality
$\dfrac{9}{4}\Big{(}\dfrac{q}{q-1}\Big{)}^{2}<3.$ This inequality is satisfied
if and only if $\Big{(}\dfrac{q}{q-1}\Big{)}^{2}<\dfrac{4}{3}$, which is true
if and only if $q\geq 8$.

∎

######  Lemma 3.2.

Assume $n>d$ and $Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is nonempty. If
$Z(\textit{{f}},\mathbb{F}_{q}^{n})$ is not an affine space of
$\mathbb{F}_{q}^{n}$, then $Z(\textit{{f}},\mathbb{F}_{q}^{n})$ contains at
least $n+2-d$ points in general position.

###### Proof.

By Theorem 1.1 (3), $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq q^{n-d}$. Suppose
that a maximal set of points of $Z(\textit{{f}},\mathbb{F}_{q}^{n})$ in
general position has at most $n+1-d$ points. Then this maximal set of points
lies in some affine space $\mathbb{A}^{n-d}(\mathbb{F}_{q})$. Then
$Z(\textit{{f}},\mathbb{F}_{q}^{n})\subseteq\mathbb{A}^{n-d}(\mathbb{F}_{q})$,
for otherwise we would have a larger set of points in general position. Since
$N(\textit{{f}},\mathbb{F}_{q}^{n})\geq q^{n-d}$ and
$|\mathbb{A}^{n-d}(\mathbb{F}_{q})|=q^{n-d}$, it follows that
$N(\textit{{f}},\mathbb{F}_{q}^{n})=q^{n-d}$ and
$Z(\textit{{f}};\mathbb{F}_{q}^{n})$ forms an affine space, which is a
contradiction. ∎

We now begin the proof of Theorem 1.4. The proof of Theorem 1.4 (1), (2), and
(3) closely follows the ideas of Heath–Brown’s proof in [3].

###### Proof of Theorem 1.4.

Since we must check many cases, we give a convenient labeling to easily track
where we are in the proof. Case IIB2(b) means we are in the proof of (2), Case
B, subcase 2, subsubcase b.

By Lemma 3.2, $Z(\textit{{f}},\mathbb{F}_{q}^{n})$ contains a maximal set of
$t+1\geq n+2-d$ points in general position. Then $n+1-d\leq t\leq n$. Let
$\mathbb{A}^{t}(\mathbb{F}_{q})$ be the $t$-dimensional affine space spanned
by these $t+1$ points. Note that
$|Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{q}))|\leq|Z(\textit{{f}},\mathbb{F}_{q}^{n})|$.

(1) Case I Assume $q=2$ and let
$S=Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{2}))$.

Case IA Suppose the hypotheses of Lemma 2.4 (1) are satisfied. Then
$Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{2}))=\mathbb{A}^{t}(\mathbb{F}_{2})$
and

$\displaystyle N(\textit{{f}},\mathbb{F}_{2}^{n})$ $\displaystyle\geq
N(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{2}))=|Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{2}))|=|\mathbb{A}^{t}(\mathbb{F}_{2})|=2^{t}$
$\displaystyle\geq 2^{n+1-d}=2^{n-d}+2^{n-d}\geq 2^{n-d}+2.$

Case IB Suppose the second hypothesis of Lemma 2.4 (1) is not satisfied. Then
there exists a 2-dimensional affine space
$L_{0}\subseteq\mathbb{A}^{t}(\mathbb{F}_{2})$ with $N(\textit{{f}},L_{0})=3$.

Take $L$ and $L^{\prime}$ as in Lemma 2.2. Then
$N(\textit{{f}},L)=N(\textit{{f}},L_{0})=3$. Note dim$(L)=k\leq d$, because if
dim$(L)>d$, then $2\mid N(\textit{{f}},L)$ by Theorem 1.1 (2) and Theorem 2.1,
which is a contradiction. Then Lemma 2.2 yields the following bound

$\displaystyle N(\textit{{f}},\mathbb{F}_{2}^{n})$ $\displaystyle\geq
N(\textit{{f}},L)+\dfrac{2^{n-k}-1}{2-1}(N(\textit{{f}},L^{\prime})-N(\textit{{f}},L))\geq
3+2^{n-k}-1\geq 2^{n-d}+2.$

Remark: If we use Theorem 1.1 (2) and Theorem 1.3 (1), then the result follows
more easily because $2^{n-d}+2$ is the smallest integer greater than $2^{n-d}$
that is divisible by $2$. However, our intermediate estimates above give
potentially better results.

(2) Case II Assume $q\geq 3$ and let
$S=Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{q}))$.

Case IIA Suppose the second hypothesis of Lemma 2.4 (2) is satisfied. Then
$Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{q}))=\mathbb{A}^{t}(\mathbb{F}_{q})$
and

$N(\textit{{f}},\mathbb{F}_{q}^{n})\geq
N(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{q}))=|Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{q}))|=|\mathbb{A}^{t}(\mathbb{F}_{q})|=q^{t}\geq
q^{n+1-d}>2q^{n-d}.$

Case IIB Suppose the second hypothesis of Lemma 2.4 (2) is not satisfied. Then
there is a line $L_{0}\subseteq\mathbb{A}^{t}(\mathbb{F}_{q})$ such that
$2\leq|L_{0}\cap S|$ and $2\leq N(\textit{{f}},L_{0})\leq q-1$. Take $L$ and
$L^{\prime}$ as in Lemma 2.2. Then $2\leq
N(\textit{{f}},L)=N(\textit{{f}},L_{0})\leq q-1$, and so $\dim(L)=k\leq d$ by
Lemma 2.6. Then there are three subcases.

Case IIB(1) Suppose $k\leq d-2$. By Lemma 2.2,

$\displaystyle N(\textit{{f}},\mathbb{F}_{q}^{n})$ $\displaystyle\geq
N(\textit{{f}},L)+\dfrac{q^{n-k}-1}{q-1}(N(\textit{{f}},L^{\prime})-N(\textit{{f}},L))$
$\displaystyle>\dfrac{q^{n-k}-1}{q-1}\geq\dfrac{q^{n+2-d}-1}{q-1}>q^{n+1-d}\geq
3q^{n-d}>2q^{n-d}.$

Case IIB(2) Suppose $k=d$. Then $2\leq N(\textit{{f}},L)\leq q-1$. Then by
Lemma 2.5, we find $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq 2q^{n-d}$.

Case IIB(3) Suppose $k=d-1$. Then either

  * (a)

$N(\textit{{f}},L^{\prime})-N(\textit{{f}},L)\geq 2$

  * (b)

$N(\textit{{f}},L^{\prime})-N(\textit{{f}},L)=1$, and thus $3\leq
N(\textit{{f}},L^{\prime})\leq q$, or

Case IIB3(a) Suppose (a) holds. Define $L^{\prime}$ as in Lemma 2.2. Then
Lemma 2.2 gives the following bound.

$\displaystyle N(\textit{{f}},\mathbb{F}_{q}^{n})$ $\displaystyle\geq
N(\textit{{f}},L)+\dfrac{q^{n-k}-1}{q-1}(N(\textit{{f}},L^{\prime})-N(\textit{{f}},L))$
$\displaystyle>\dfrac{q^{n-k}-1}{q-1}(N(\textit{{f}},L^{\prime})-N(\textit{{f}},L))\geq
2\cdot\dfrac{q^{n+1-d}-1}{q-1}>2q^{n-d}.$

Case IIB3(b) Suppose (b) holds. We have $\dim(L^{\prime})=d$. First assume
$2\leq N(\textit{{f}},L)\leq q-2$. Then $3\leq N(\textit{{f}},L^{\prime})\leq
q-1$. By Theorem 2.1, we have $N(\textit{{f}},L^{*})\equiv
N(\textit{{f}},L^{\prime})\bmod{q}$ for every affine space $L^{*}$ of
dimension $d$ parallel to $L^{\prime}$. Then
$N(\textit{{f}},L^{*})\notin\\{0,1,2\\}$ because
$N(\textit{{f}},L^{\prime})\not\equiv 0,1,2\bmod{q}$. Thus,
$N(\textit{{f}},L^{*})\geq 3$. Since there are $q^{n-d}$ affine spaces $L^{*}$
of dimension $d$ that are parallel to $L^{\prime}$, it follows that
$N(\textit{{f}},\mathbb{F}_{q}^{n})\geq 3q^{n-d}>2q^{n-d}$.

Now assume $N(\textit{{f}},L)=q-1$. Then $N(\textit{{f}},L^{\prime})=q$. Let
$v\in Z(\textit{{f}},L)$, let $w\in Z(\textit{{f}},L^{\prime})\backslash L$,
and let $L^{\prime}_{0}$ be the line containing $v,w$. Then
$L^{\prime}_{0}\subset L^{\prime}$ and $N(\textit{{f}},L^{\prime}_{0})=2$.
Thus $|L^{\prime}_{0}\cap S|=2$. This means that we can assume from the start
that $N(\textit{{f}},L)=2$. If $q\geq 4$, then $N(\textit{{f}},L)=2\leq q-2$
and we finish using the argument in the previous paragraph.

It remains to consider the case that $q=3$. For the first part of the
following argument, it is more convenient to consider general values of $q$.
At the end of the argument, we specialize to $q=3$.

Let $L^{\prime\prime}$ be an affine subspace of dimension $d+1$ containing
$L$. Since $N(\textit{{f}},L^{\prime})-N(\textit{{f}},L)\geq 1$ for any affine
space of dimension $d$ with $L\subset L^{\prime}\subset L^{\prime\prime}$,
Lemma 2.2 implies

$\displaystyle N(\textit{{f}},L^{\prime\prime})$ $\displaystyle\geq
N(\textit{{f}},L)+\dfrac{q^{(d+1)-(d-1)}-1}{q-1}(N(\textit{{f}},L^{\prime})-N(\textit{{f}},L))\geq
N(\textit{{f}},L)+\dfrac{q^{2}-1}{q-1}$
$\displaystyle=(q-1)+\dfrac{q^{2}-1}{q-1}=(q-1)+(q+1)=2q.$

Suppose that $N(\textit{{f}},L^{\prime\prime})\geq 2q+1$ for all affine spaces
$L^{\prime\prime}$ having dimension $d+1$ and containing $L$. Then
$N(\textit{{f}},L^{\prime\prime})\geq 3q$ for all such affine spaces
$L^{\prime\prime}$ by Theorem 1.1 (4) and Theorem 2.1. Since
$N(\textit{{f}},L^{\prime\prime})-N(\textit{{f}},L^{\prime})\geq 3q-q=2q$,
Lemma 2.2 implies that

$\displaystyle N(\textit{{f}},\mathbb{F}_{q}^{n})$ $\displaystyle\geq
N(\textit{{f}},L^{\prime})+\dfrac{q^{n-d}-1}{q-1}(N(\textit{{f}},L^{\prime\prime})-N(\textit{{f}},L^{\prime}))\geq
q+\dfrac{q^{n-d}-1}{q-1}\cdot 2q$
$\displaystyle=q+2q(q^{n-d-1}+\cdots+q+1)\geq q+2q^{n-d}>2q^{n-d}.$

Thus we can assume that there exists $L^{\prime\prime}$ as above such that
$N(\textit{{f}},L^{\prime\prime})=2q$. Now assume that $q=3$, and so
$N(\textit{{f}},L^{\prime\prime})=6$.

Let $Z(\textit{{f}},L^{\prime\prime})=\\{P_{0},\ldots,P_{5}\\}$. We can assume
that $Z(\textit{{f}},L)=\\{P_{0},P_{1}\\}$ and
$Z(\textit{{f}},L^{\prime})=\\{P_{0},P_{1},P_{2}\\}$. We can assume that
$P_{0}=(0,\ldots,0)$ and $P_{1}=(1,0,0,\ldots,0)$. Since $P_{2}\notin L$, we
can assume that $P_{2}=(0,1,0,\ldots,0)$. Since $P_{3}\notin L^{\prime}$, we
can assume that $P_{3}=(0,0,1,0,\ldots,0)$. Since $P_{4},P_{5}\in
L^{\prime\prime}\backslash L^{\prime}$, we can assume that
$P_{4}=(a_{1},a_{2},\ldots,a_{n})$ and $P_{5}=(b_{1},\ldots,b_{n})$ where some
$a_{i}\neq 0$ with $i\geq 3$ and some $b_{j}\neq 0$ with $j\geq 3$. We have
$3\leq\dim(\operatorname{Span}\\{P_{1},\ldots,P_{5}\\})\leq 5$.

We will find an affine space $\widehat{L}$ of dimension $n-1$ such that
$N(\textit{{f}},\widehat{L}\cap L^{\prime\prime})\equiv 2\bmod 3$. Then since
$\widehat{L}\cap L^{\prime\prime}$ is nonempty and
$N(\textit{{f}},\widehat{L}\cap L^{\prime\prime})\neq 6$, it follows that
$\dim(\widehat{L}\cap L^{\prime\prime})=(d+1)-1=d$. Then Lemma 2.5 implies
that $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq 2q^{n-d}$.

Suppose first that $\dim(\operatorname{Span}\\{P_{1},\ldots,P_{5}\\})=5$. Then
we can assume that

$P_{4}=(0,0,0,1,0,\ldots,0)\text{ and }P_{5}=(0,0,0,0,1,0,\ldots,0).$

Let $\widehat{L}:x_{1}+x_{2}+\cdots+x_{5}=1$, an affine space of dimension
$n-1$. Then $Z(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})=\\{P_{1},P_{2},\ldots,P_{5}\\}$. Thus
$N(\textit{{f}},\widehat{L}\cap L^{\prime\prime})=5\equiv 2\bmod 3$.

Now suppose that $\dim(\operatorname{Span}\\{P_{1},\ldots,P_{5}\\})=4$. Then
we can assume that

$P_{4}=(0,0,0,1,0,\ldots,0)\text{ and
}P_{5}=(b_{1},b_{2},b_{3},b_{4},0,\ldots,0).$

Suppose that $b_{i}=0$ for some $i\leq 4$. Let $\widehat{L}:x_{i}=0$. Then
$N(\textit{{f}},\widehat{L}\cap L^{\prime\prime})=5\equiv 2\bmod 3$. Now
suppose that each $b_{i}\neq 0$. Suppose that $b_{i}=1$ for some $i\leq 4$.
Let $\widehat{L}:x_{i}=1$. Then $N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})=2$. Now suppose that $b_{1}=\cdots=b_{4}=2$. Let
$\widehat{L}:x_{1}-x_{2}+x_{3}-x_{4}=0$. Then $N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})=2$.

Suppose that $\dim(\operatorname{Span}\\{P_{1},\ldots,P_{5}\\})=3$. Then we
can assume that

$P_{4}=(a_{1},a_{2},a_{3},0,\ldots,0)\text{ and
}P_{5}=(b_{1},b_{2},b_{3},0,\ldots,0).$

Suppose that $a_{i}=1$. If $b_{i}\in\\{0,2\\}$, then let
$\widehat{L}:x_{i}=1$, which gives $N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})=2$. Thus we can assume that $b_{i}=1$. Suppose that
$a_{i}=0$. If $b_{i}=0$, then let $\widehat{L}:x_{i}=0$, which gives
$N(\textit{{f}},\widehat{L}\cap L^{\prime\prime})=5$. If $b_{i}=1$, then let
$\widehat{L}:x_{i}=1$, which gives $N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})=2$. Thus we can assume that $b_{i}=2$. Suppose that
$a_{i}=2$. If $b_{i}=2$, then let $\widehat{L}:x_{i}=2$, which gives
$N(\textit{{f}},\widehat{L}\cap L^{\prime\prime})=2$. If $b_{i}=1$, then let
$\widehat{L}:x_{i}=1$, which gives $N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})=2$. Thus we can assume that $b_{i}=0$.

It follows that we can assume $b_{i}\equiv 2-a_{i}\bmod 3$. That is,

$P_{4}=(a_{1},a_{2},a_{3},0\ldots,0)\text{ and
}P_{5}=(2-a_{1},2-a_{2},2-a_{3},0,\ldots,0).$

Then $(a_{1},a_{2},a_{3})\neq(0,0,0),(1,1,1),(2,2,2)$ because $P_{4}\neq
P_{5}$ and $P_{4},P_{5}$ are distinct from $P_{0}$.

Suppose that $(a_{1},a_{2},a_{3})=(0,1,2)$ or some permutation of these
coordinates. Then $(b_{1},b_{2},b_{3})=(2,1,0)$ and we let
$\widehat{L}:x_{1}+x_{3}=2$. Then $N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})=2$. Suppose that $(a_{1},a_{2},a_{3})=(0,0,2)$. Then
$(b_{1},b_{2},b_{3})=(2,2,0)$ and we take $\widehat{L}:x_{1}+x_{3}=2$. Then
$N(\textit{{f}},\widehat{L}\cap L^{\prime\prime})=2$. Suppose that
$(a_{1},a_{2},a_{3})=(1,1,0)$. Then $(b_{1},b_{2},b_{3})=(1,1,2)$ and we take
$\widehat{L}:x_{1}+x_{2}=2$. Then $N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})=2$. Suppose that $(a_{1},a_{2},a_{3})=(2,2,1)$. Then
$P_{5}=(b_{1},b_{2},b_{3})=(0,0,1)=P_{3}$, which is a contradiction. This
covers all possible cases and the proof of Case II is complete.

Remark: Some additional information can be obtained from the proof of Case
IIB. Suppose that $q=3$, $\dim(L^{\prime\prime})=d+1$, and
$N(\textit{{f}},L^{\prime\prime})=6$, as above. Let $\widehat{L}$ be an affine
space of dimension $n-1$ such that $N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})\equiv 2\bmod 3$. Then, as noted above, $\dim(\widehat{L}\cap
L^{\prime\prime})=d$. We can write $L^{\prime\prime}=(\widehat{L}\cap
L^{\prime\prime})\cup(\widehat{L}_{1}\cap
L^{\prime\prime})\cup(\widehat{L}_{2}\cap L^{\prime\prime})$ where
$\widehat{L}\cap L^{\prime\prime},\widehat{L}_{1}\cap
L^{\prime\prime},\widehat{L}_{2}\cap L^{\prime\prime}$ are parallel affine
spaces each having dimension $d$, and thus the union is disjoint. Then

$6=N(\textit{{f}},L^{\prime\prime})=N(\textit{{f}},\widehat{L}\cap
L^{\prime\prime})+N(\textit{{f}},\widehat{L}_{1}\cap
L^{\prime\prime})+N(\textit{{f}},\widehat{L}_{2}\cap L^{\prime\prime}).$

By Theorem 2.1, each term on the right is congruent to 2 modulo 3 and thus
each term is at least $2$. Since the sum equals $6$, it follows that each term
equals $2$.

As a consequence, it follows that
$\dim(\operatorname{Span}\\{P_{1},\ldots,P_{5}\\})\neq 5$ because in that case
above, we found an affine space $\widehat{L}$ such that
$N(\textit{{f}},\widehat{L}\cap L^{\prime\prime})=5$.

(3) Case III We begin with the case $q=2$. Since

$\frac{2^{n+1-d}-1}{2^{n-d}}=2-\frac{1}{2^{n-d}}\leq\begin{cases}\frac{3}{2}\leq\frac{n+2-d}{2}&\text{
if $n-d=1$}\\\ 2\leq\frac{n+2-d}{2}&\text{ if $n-d\geq 2$,}\end{cases}$

it follows that $\dfrac{2^{n+1-d}-1}{2-1}\cdot\dfrac{2}{n+2-d}\leq
2^{n-d}<2^{n-d}+2$, which is the lower bound in Theorem 1.4 (1).

Now assume $q\geq 3$. Let $v=n+1-d$, let
$m=\Big{\lfloor}\dfrac{qv}{v+1}\Big{\rfloor}$, and let
$S=Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{q}))$. Then $v\geq 2$ and $2\leq
m\leq q-1$.

Case IIIA Suppose the hypothesis of Lemma 2.4 (4) is satisfied. Then
$|Z(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{q}))|\geq\dfrac{m^{t+1}-1}{m-1}$.
Thus,

$\displaystyle N(\textit{{f}},\mathbb{F}_{q}^{n})$ $\displaystyle\geq
N(\textit{{f}},\mathbb{A}^{t}(\mathbb{F}_{q}))\geq\dfrac{m^{t+1}-1}{m-1}\geq\dfrac{m^{n+2-d}-1}{m-1}.$

We will now show that
$\dfrac{m^{n+2-d}-1}{m-1}>\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}$. This
is equivalent to showing

$\dfrac{m^{v+1}-1}{m-1}>\dfrac{q^{v}-1}{q-1}\cdot\dfrac{q}{v+1},$

which is equivalent to showing

$m^{v}+m^{v-1}+\dots+m+1>\dfrac{q^{v}+q^{v-1}+\dots+q}{v+1}.$

First assume $v\geq 2$, $q\geq 2v+1$, and $(q,v)\neq(5,2),(7,2)$. Then by
Lemma 3.1, $m^{v}>\dfrac{q^{v}}{v+1}$. Now observe that

$m=\Big{\lfloor}\dfrac{qv}{v+1}\Big{\rfloor}\leq\dfrac{qv}{v+1}<q.$

Since $\dfrac{q}{m}>1$, we have for $1\leq i\leq v$ that

$m^{i}>\dfrac{q^{v-i}}{m^{v-i}}\cdot\dfrac{q^{i}}{v+1}\geq\dfrac{q^{i}}{v+1}.$

Therefore, $m^{v}+m^{v-1}+\dots+1>\dfrac{q^{v}+q^{v-1}+\dots+q}{v+1}$.

Suppose $(q,v)=(5,2)$. Since f is a homogeneous system,
$N(\textit{{f}},\mathbb{F}_{5}^{n})\equiv 1\bmod 4$ by Equation (1). Since
$n-d=v-1=1$, Theorem 1.4 (2) implies $N(\textit{{f}},\mathbb{F}_{5}^{n})\geq
10$. Thus,

$N(\textit{{f}},\mathbb{F}_{5}^{n})\geq
13>10=\dfrac{5^{2}-1}{5-1}\cdot\dfrac{5}{3}=\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}.$

Now suppose $(q,v)=(7,2)$. Since f is a homogeneous system,
$N(\textit{{f}},\mathbb{F}_{7}^{n})\equiv 1\bmod 6$ by Equation (1). Since
$n-d=1$, Theorem 1.4 (2) implies $N(\textit{{f}},\mathbb{F}_{7}^{n})\geq 14$.
Thus,

$N(\textit{{f}},\mathbb{F}_{7}^{n})\geq
19>\dfrac{56}{3}=\dfrac{7^{2}-1}{7-1}\cdot\dfrac{7}{3}=\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}.$

Thus, if $q\geq 2v+1$, then the proof is finished in this case.

Remark: If we use Theorem 1.4 (4), proved later, then we obtain a better bound
in the cases $(q,v)=(5,2),(7,2)$. See the beginning of Section 5.

Now suppose $q\leq 2v$. Then $2v+2\geq q+2$. Thus,

$2\geq\dfrac{q+2}{v+1}>\dfrac{q+1+\frac{1}{q-1}}{v+1}=\dfrac{q^{2}}{(v+1)(q-1)}.$

Multiply both sides by $q^{n-d}$, substitute $n+1-d$ for $v$, and apply
Theorem 1.4 (2) to obtain

$\displaystyle N(\textit{{f}},\mathbb{F}_{q}^{n})$ $\displaystyle\geq
2q^{n-d}>\dfrac{q^{n+2-d}}{(v+1)(q-1)}>\dfrac{q^{n+2-d}-q}{(v+1)(q-1)}=\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}.$

Remark: Heath-Brown’s proof of this part omitted the case when $q\leq 2v$. The
case when $q\leq 2v$ appears again in Corollary 5.2.

Case IIIB Suppose the hypothesis of Lemma 2.4 (4) is not satisfied. Then there
is a line $L_{0}\subseteq\mathbb{A}^{t}(\mathbb{F}_{q})$ with $2\leq
N(\textit{{f}},L_{0})\leq m$. Take $L$ and $L^{\prime}$ as in Lemma 2.2. Then
$2\leq N(\textit{{f}},L)=N(\textit{{f}},L_{0})\leq m\leq q-1$, and so
$\dim(L)=k\leq d-1$ by Lemma 2.6 (2). Then we have the following two subcases.

Case IIIB(1) Suppose $k\leq d-2$. Then $n-k\geq n+2-d$ and Lemma 2.2 implies
that

$\displaystyle N(\textit{{f}},\mathbb{F}_{q}^{n})$ $\displaystyle\geq
N(\textit{{f}},L)+\dfrac{q^{n+2-d}-1}{q-1}(N(\textit{{f}},L^{\prime})-N(\textit{{f}},L))>\dfrac{q^{n+2-d}-1}{q-1}$
$\displaystyle>\dfrac{q(q^{n+1-d}-1)}{q-1}>\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}.$

Case IIIB(2) Suppose $k=d-1$. Then $\dim(L^{\prime})=d$. Since
$N(\textit{{f}},L^{\prime})\geq N(\textit{{f}},L)+1\geq 3$, Lemma 2.6 (2)
implies that $N(\textit{{f}},L^{\prime})\geq q$. Since $N(\textit{{f}},L)\leq
m$, we have $N(\textit{{f}},L^{\prime})-N(f,L)\geq q-m$. Since
$m=\Big{\lfloor}\dfrac{qv}{v+1}\Big{\rfloor}$, Lemma 2.2 implies that

$\displaystyle N(\textit{{f}},\mathbb{F}_{q}^{n})$
$\displaystyle>\dfrac{q^{n-k}-1}{q-1}(N(\textit{{f}},L^{\prime})-N(\textit{{f}},L)\geq\dfrac{q^{n+1-d}-1}{q-1}(q-m)$
$\displaystyle\geq\dfrac{q^{n+1-d}-1}{q-1}\Big{(}q-\dfrac{qv}{v+1}\Big{)}=\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{v+1}=\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}.$

(4) By Theorem 1.4 (2), $N(\textit{{f}},\mathbb{F}_{q}^{n})\geq 2q^{n-d}$. By
Theorem 1.1 (4), $N(\textit{{f}},\mathbb{F}_{q}^{n})=2q^{n-d}+aq$ for some
$a\in\mathbb{Z}_{\geq 0}$. Since $N(\textit{{f}},\mathbb{F}_{q}^{n})\equiv
1\bmod q-1$, we know $2+a\equiv 1\bmod q-1$, which implies $a\equiv-1\bmod
q-1$. Since $a\in\mathbb{Z}_{\geq 0}$, we have $a\geq q-2$, which gives the
desired result.

We will demonstrate that bounds (1), (2) and (4) in Theorem 1.4 are sharp
under these hypotheses in Section 4, which will complete the proof. ∎

##  4\. Optimality of Theorem 1.4

Sharpness of Theorem 1.4 (1) We will show that Theorem 1.4 (1) is optimal for
every $d\geq 2$ and $n\in\\{d+1,d+2\\}$.

For each $d\in\mathbb{Z}$, $d\geq 1$, let
$g_{d}\in\mathbb{F}_{2}[x_{1},\dots,x_{d}]$ be a homogeneous form of degree
$d$ having no nontrivial zero over $\mathbb{F}_{2}.$ For example, $g_{d}$
could be chosen to be a norm form.

######  Proposition 4.1.

Let $d\in\mathbb{Z}$, $d\geq 2$, and let $n=d+1$. Then there exists a
homogeneous form $f_{d}\in\mathbb{F}_{2}[x_{1},\dots,x_{n}]$ of degree $d$
such that $|Z(f_{d},\mathbb{F}_{2}^{n})|=4=2^{n-d}+2$ and
$Z(f_{d},\mathbb{F}_{2}^{n})$ is not an affine space in $\mathbb{F}_{2}^{n}$.

###### Proof.

For $d=2,$ let $f_{2}=x_{1}x_{2}-x_{3}^{2}$. Then
$|Z(f_{2},\mathbb{F}_{2}^{3})|=4$ because if $x_{1}=0$, then the zeros over
$\mathbb{F}_{2}$ are $(0,0,0)$ and $(0,1,0)$, while if $x_{1}=1$, then the
zeros over $\mathbb{F}_{2}$ are $(1,0,0)$ and $(1,1,1)$. These four zeros do
not form a subspace of $\mathbb{F}_{2}^{3}$ and thus do not form an affine
space in $\mathbb{F}_{2}^{3}$.

Let $d\geq 3$ and suppose by induction that
$f_{d-1}\in\mathbb{F}_{2}[x_{1},\dots,x_{d}]$ is a homogeneous form of degree
$d-1$ such that $|Z(f_{d-1},\mathbb{F}_{2}^{d})|=4$ and
$Z(f_{d-1},\mathbb{F}_{2}^{d})$ is not an affine space in
$\mathbb{F}_{2}^{d}$. Let

$f_{d}=x_{d+1}^{d}-f_{d-1}(x_{1},\dots,x_{d})x_{d+1}+g_{d}(x_{1},\dots,x_{d}).$

Then $f_{d}$ is a homogeneous form of degree $d$ in
$\mathbb{F}_{2}[x_{1},\dots,x_{d+1}]$. Let
$a_{1},\dots,a_{d}\in\mathbb{F}_{2}$. First suppose
$f_{d-1}(a_{1},\dots,a_{d})\neq 0$. Then $(a_{1},\dots,a_{d})\neq 0$ and so
$g_{d}(a_{1},\dots,a_{d})\neq 0$. Then for each $a_{d+1}\in\mathbb{F}_{2}$,

$f_{d}(a_{1}\dots,a_{d+1})=a_{d+1}^{d}-a_{d+1}+g_{d}(a_{1},\dots,a_{d})=g_{d}(a_{1},\dots,a_{d})\neq
0.$

Now suppose $f_{d-1}(a_{1},\dots,a_{d})=0.$ Then there exists a unique
$a_{d+1}\in\mathbb{F}_{2}$ such that $a_{d+1}^{d}+g_{d}(a_{1},\dots,a_{d})=0.$
Thus,
$|Z(f_{d},\mathbb{F}_{2}^{n})|=|Z(f_{d},\mathbb{F}_{2}^{d+1})|=|Z(f_{d-1},\mathbb{F}_{2}^{d})|=4.$

Let $\pi:\mathbb{F}_{2}^{d+1}\rightarrow\mathbb{F}_{2}^{d}$ be the projection
map defined by $\pi(a_{1},\dots,a_{d+1})=(a_{1},\dots,a_{d})$. Then
$\pi(Z(f_{d},\mathbb{F}_{2}^{d+1}))=Z(f_{d-1},\mathbb{F}_{2}^{d}).$ Since
$Z(f_{d-1},\mathbb{F}_{2}^{d})$ is not an affine space in
$\mathbb{F}_{2}^{d}$, it follows that $Z(f_{d},\mathbb{F}_{2}^{d+1})$ is not
an affine space in $\mathbb{F}_{2}^{d+1}=\mathbb{F}_{2}^{n}.$ ∎

######  Proposition 4.2.

Let $d\in\mathbb{Z}$, $d\geq 2$, and let $n=d+2$. Then there exists a
homogeneous form $f_{d}\in\mathbb{F}_{2}[x_{1},\dots,x_{n}]$ of degree $d$
such that $|Z(f_{d},\mathbb{F}_{2}^{n})|=6=2^{n-d}+2$ and
$Z(f_{d},\mathbb{F}_{2}^{n})$ is not an affine space in $\mathbb{F}_{2}^{n}$.

###### Proof.

Any subset of $\mathbb{F}_{2}^{n}$ with cardinality $6$ is not an affine space
in $\mathbb{F}_{2}^{n}$ because $6$ is not a $2$-power.

For $d=2,$ let $f_{2}=x_{1}^{2}+x_{1}x_{2}+x_{2}^{2}+x_{3}x_{4}.$ Then
$|Z(f_{2},\mathbb{F}_{2}^{4})|=6$ because there are $4$ zeros in
$\mathbb{F}_{2}^{4}$ if $x_{3}=1$ and there are $2$ zeros in
$\mathbb{F}_{2}^{4}$ if $x_{3}=0$.

Let $d\geq 3$ and suppose by induction that
$f_{d-1}\in\mathbb{F}_{2}[x_{1},\dots,x_{d+1}]$ is a homogeneous form of
degree $d-1$ with $|Z(f_{d-1},\mathbb{F}_{2}^{d+1})|=6.$ By an invertible
linear change of variables, we can assume that $f_{d-1}(0,0,\dots,0,1)=0$.

Let
$f_{d}=x_{d+2}^{d}-f_{d-1}(x_{1},\dots,x_{d+1})x_{d+2}+g_{d}(x_{1},\dots,x_{d}).$
Then $f_{d}$ is a homogeneous form of degree $d$ in
$\mathbb{F}_{2}[x_{1},\dots,x_{d+2}].$ Let
$a_{1},\dots,a_{d+1}\in\mathbb{F}_{2}$. First suppose
$f_{d-1}(a_{1},\dots,a_{d+1})\neq 0$. If $a_{1}=\dots=a_{d}=0,$ then
$f_{d-1}(0,0,\dots,0,a_{d+1})=0$ for $a_{d+1}\in\mathbb{F}_{2}$ because
$f_{d-1}(0,\dots,0,1)=0$ by assumption. Thus $(a_{1},\dots,a_{d})\neq 0$ and
so $g_{d}(a_{1},\dots,a_{d})\neq 0$. Then

$f_{d}(a_{1},\dots,a_{d+1},a_{d+2})=a_{d+2}^{d}-a_{d+2}+g_{d}(a_{1},\dots,a_{d})=g_{d}(a_{1},\dots,a_{d})\neq
0.$

Now suppose $f_{d-1}(a_{1},\dots,a_{d+1})=0$. Then there exists a unique
$a_{d+2}\in\mathbb{F}_{2}$ with $a_{d+2}^{d}+g_{d}(a_{1},\dots,a_{d})=0$.
Thus,
$|Z(f_{d},\mathbb{F}_{2}^{n})|=|Z(f_{d},\mathbb{F}_{2}^{d+2})|=|Z(f_{d-1},\mathbb{F}_{2}^{d+1})|=6$.

∎

Sharpness of Theorem 1.4 (2) Consider the polynomial

$g(x_{1},x_{2},x_{3})=x_{1}x_{2}+x_{3}^{2}+1$

over $\mathbb{F}_{3}$. Since $|Z(g,\mathbb{F}_{3}^{3})|=6$ and $6$ is not a
power of 3, $Z(g,\mathbb{F}_{3}^{3})$ does not form an affine space in
$\mathbb{F}_{3}^{3}$. The lower bound given in Theorem 1.4 (2) is
$2q^{n-d}=2(3)^{3-2}=6$.

Sharpness of Theorem 1.4 (4) An example with $d=2$, $n=3$ is given by the
homogeneous form

$h(x_{1},x_{2},x_{3})=x_{1}x_{2}-x_{3}^{2}$

over $\mathbb{F}_{q}$ where $q\geq 3$. There are exactly $(q-1)q+q=q^{2}$
zeros of this equation. The lower bound given in Theorem 1.4 (4) is
$2q^{n-d}+(q-2)q=2q+(q-2)q=q^{2}$. If $Z(h,\mathbb{F}_{q}^{3})$ is an affine
space, then $Z(h,\mathbb{F}_{q}^{3})$ is a subspace since $\boldsymbol{0}\in
Z(h,\mathbb{F}_{q}^{3})$. This is not the case because $(1,0,0),(0,1,0)\in
Z(h,\mathbb{F}_{q}^{3})$, but, their sum, $(1,1,0)\notin
Z(h,\mathbb{F}_{q}^{3})$.

A second example with $d=3$, $n=4$ is given by

$j(x_{1},x_{2},x_{3},x_{4})=x_{1}^{3}-x_{2}^{2}x_{1}+c(x_{2},x_{3},x_{4}),$

where $c(x_{2},x_{3},x_{4})\in\mathbb{F}_{3}[x_{2},x_{3},x_{4}]$ is a cubic
form having no nontrivial zero over $\mathbb{F}_{3}$, such as a norm form. Let
$(a_{1},a_{2},a_{3},a_{4})\in\mathbb{F}_{3}^{4}$ be a zero of $j$. If
$a_{2}\neq 0$, then $a_{1}^{3}-a_{2}^{2}a_{1}=a_{1}^{3}-a_{1}=0$, but
$c(a_{2},a_{3},a_{4})\neq 0$. Therefore, $j(a_{1},a_{2},a_{3},a_{4})\neq 0$.
Thus, $a_{2}=0$. Let $a_{3},a_{4}\in\mathbb{F}_{3}$ be arbitrary. Then there
exists a unique $a_{1}\in\mathbb{F}_{3}$ such that
$a_{1}^{3}=-c(a_{2},a_{3},a_{4})$. Therefore,
$N(j,\mathbb{F}_{3}^{4})=9=q^{2}=2q^{n-d}+(q-2)q$, which is the lower bound
given in Theorem 1.4 (4). If $Z(j,\mathbb{F}_{3}^{4})$ is an affine space,
then $Z(j,\mathbb{F}_{3}^{4})$ is a subspace because $\boldsymbol{0}\in
Z(j,\mathbb{F}_{3}^{4})$. By the pigeonhole principle, there exist two
distinct nonzero pairs
$(a_{3},a_{4}),(a_{3}^{\prime},a_{4}^{\prime})\in\mathbb{F}_{3}^{2}$ such that
$a_{1}=a_{1}^{\prime}$. Then
$(a_{1},0,a_{3},a_{4})-(a_{1}^{\prime},0,a_{3}^{\prime},a_{4}^{\prime})=(0,0,a_{3}-a_{3}^{\prime},a_{4}-a_{4}^{\prime})$.
This is not a zero of $j$ because $c$ has no nontrivial zero over
$\mathbb{F}_{3}$.

A third example with $d=2$ and $n=4$ is given by

$h(x_{1},x_{2},x_{3},x_{4})=x_{1}x_{2}+x_{3}^{2}+x_{4}^{2}\in\mathbb{F}_{3}[x_{1},x_{2},x_{3},x_{4}].$

Then $h$ has exactly $(q-1)q^{2}+q=q^{3}-q^{2}+q=21$ zeros over
$\mathbb{F}_{3}$. The lower bound given in Theorem 1.4 (4) is
$2q^{2}+(q-2)q=3q^{2}-2q=21$. Since $|Z(h,\mathbb{F}_{q}^{4})|=21$ is not a
power of $3$, it follows that $Z(h,\mathbb{F}_{q}^{4})$ is not an affine
space.

##  5\. Comparing (3) and (4) of Theorem 1.4

We now compare the statements in Theorem 1.4 for the case of homogeneous
forms.

We already observed at the beginning of the proof of Case III that when $q=2$
the bound in Theorem 1.4 (3) is strictly less than the bound in Theorem 1.4
(1).

Now assume that $q\geq 3$. If $n-d=1$, then

$2q^{n-d}+(q-2)q=q^{2}>\frac{q^{2}+q}{3}=\frac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}.$

In this case, Theorem 1.4 (4) gives a larger lower bound than Theorem 1.4 (3).

Next suppose $n-d\geq 2$. For real numbers $A$ and $B$, let $A\
\mathbin{\stackanchor[11pt]{\stackanchor[7pt]{<}{=}}{>}}\ B$, denote the three
statements $A<B$, $A=B$, $A>B$.

######  Proposition 5.1.

Let $q\geq 3$, $n-d\geq 2$, and $v=n+1-d$.

  1. (1)

If
$q>\Big{(}2+\dfrac{q^{2}-2q}{q^{n-d}}\Big{)}v+1+\dfrac{q^{2}-2q}{q^{n-d}}-\Big{(}\dfrac{1}{q}+\dfrac{1}{q^{2}}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}$,
then Theorem 1.4 (3) gives a larger lower bound than Theorem 1.4 (4).

  2. (2)

If
$q<\Big{(}2+\dfrac{q^{2}-2q}{q^{n-d}}\Big{)}v+1+\dfrac{q^{2}-2q}{q^{n-d}}-\Big{(}\dfrac{1}{q}+\dfrac{1}{q^{2}}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}$,
then Theorem 1.4 (4) gives a larger lower bound than Theorem 1.4 (3).

###### Proof.

We have

$\displaystyle\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}=\dfrac{q^{n+1-d}+q^{n-d}+\dots+q}{v+1}=\dfrac{q^{n-d}}{v+1}\Big{(}q+1+\dfrac{1}{q}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}.$

We now begin the proof.

$\displaystyle q\ $
$\displaystyle\mathbin{\stackanchor[11pt]{\stackanchor[7pt]{<}{=}}{>}}\
\Big{(}2+\dfrac{q^{2}-2q}{q^{n-d}}\Big{)}v+1+\dfrac{q^{2}-2q}{q^{n-d}}-\Big{(}\dfrac{1}{q}+\dfrac{1}{q^{2}}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}$

if and only if

$\displaystyle
q+1+\Big{(}\dfrac{1}{q}+\dfrac{1}{q^{2}}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}\ $
$\displaystyle\mathbin{\stackanchor[11pt]{\stackanchor[7pt]{<}{=}}{>}}\
\Big{(}2+\dfrac{q^{2}-2q}{q^{n-d}}\Big{)}(v+1)$

if and only if

$\displaystyle\dfrac{q^{n-d}}{v+1}\Big{(}q+1+\dfrac{1}{q}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}\
$ $\displaystyle\mathbin{\stackanchor[11pt]{\stackanchor[7pt]{<}{=}}{>}}\
q^{n-d}\Big{(}2+\dfrac{q^{2}-2q}{q^{n-d}}\Big{)}=2q^{n-d}+q^{2}-2q$

if and only if, using the first equation,

$\displaystyle\dfrac{q^{n+1-d}-1}{q-1}\cdot\dfrac{q}{n+2-d}\ $
$\displaystyle\mathbin{\stackanchor[11pt]{\stackanchor[7pt]{<}{=}}{>}}\
2q^{n-d}+(q-2)q.$

∎

######  Corollary 5.2.

Let $q\geq 3$, $n-d\geq 2$, and $v=n+1-d$.

  1. (1)

If $q\geq 3v+2$, then the hypothesis in Proposition 5.1 (1) holds.

  2. (2)

If $q\leq 2v$, then the hypothesis in Proposition 5.1 (2) holds.

###### Proof.

(1) Since $n-d\geq 2$ and $q\geq 3$, we have $0<\frac{q^{2}-2q}{q^{n-d}}<1$,
and so

$2<\Big{(}2+\dfrac{q^{2}-2q}{q^{n-d}}\Big{)}<3.$

This gives

$3v+2>\Big{(}2+\dfrac{q^{2}-2q}{q^{n-d}}\Big{)}v+1+\dfrac{q^{2}-2q}{q^{n-d}}-\Big{(}\dfrac{1}{q}+\dfrac{1}{q^{2}}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}.$

(2) We have
$\Big{(}\dfrac{1}{q}+\dfrac{1}{q^{2}}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}<\frac{1}{q-1}\leq
1$. Thus

$1-\Big{(}\dfrac{1}{q}+\dfrac{1}{q^{2}}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}>0.$

This gives

$2v<\Big{(}2+\dfrac{q^{2}-2q}{q^{n-d}}\Big{)}v+1+\dfrac{q^{2}-2q}{q^{n-d}}-\Big{(}\dfrac{1}{q}+\dfrac{1}{q^{2}}+\dots+\dfrac{1}{q^{n-d-1}}\Big{)}.$

∎

## References

  * [1] J. Ax.  Zeroes of polynomials over finite fields.  Amer. J. Math., 86:255–261, 1964.
  * [2] C. Chevalley.  Démonstration d’une hypothèse de M. Artin.  Abh. Math. Sem. Univ. Hamburg, 11(1):73–75, 1935.
  * [3] D. R. Heath-Brown.  On Chevalley-Warning theorems.  Uspekhi Mat. Nauk, 66(2(398)):223–232, 2011.
  * [4] E. Warning.  Bemerkung zur vorstehenden Arbeit von Herrn Chevalley.  Abh. Math. Sem. Univ. Hamburg, 11(1):76–83, 1935.