# CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for
Programming Language Attend Code Structure

Nuo Chen , Qiushi Sun11footnotemark: 1 , Renyu Zhu11footnotemark: 1 , Xiang Li
, Xuesong Lu, and Ming Gao
School of Data Science and Engineering, East China Normal University,
Shanghai, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Equal contribution, authors are listed
alphabetically. Corresponding author.

###### Abstract

Code pre-trained models (CodePTMs) have recently demonstrated significant
success in code intelligence. To interpret these models, some probing methods
have been applied. However, these methods fail to consider the inherent
characteristics of codes. In this paper, to address the problem, we propose a
novel probing method CAT-probing to quantitatively interpret how CodePTMs
attend code structure. We first denoise the input code sequences based on the
token types pre-defined by the compilers to filter those tokens whose
attention scores are too small. After that, we define a new metric CAT-score
to measure the commonality between the token-level attention scores generated
in CodePTMs and the pair-wise distances between corresponding AST nodes. The
higher the CAT-score, the stronger the ability of CodePTMs to capture code
structure. We conduct extensive experiments to integrate CAT-probing with
representative CodePTMs for different programming languages. Experimental
results show the effectiveness of CAT-probing in CodePTM interpretation. Our
codes and data are publicly available at
https://github.com/nchen909/CodeAttention.

##  1 Introduction

(a) A Python code snippet with its U-AST

(b) The attention matrix (filtered)

(c) The distance matrix (filtered)

Figure 1: Visualization on the U-AST structure, the attention matrix generated
in the last layer of CodeBERT (Feng et al., 2020) and the distance matrix. (a)
A Python code snippet with its corresponding U-AST. (b) Heatmaps of the
averaged attention weights after attention matrix filtering. (c) Heatmaps of
the pair-wise token distance in U-AST. In the heatmaps, the darker the color,
the more salient the attention score, or the closer the nodes. In this toy
example, only the token “.” between “tmpbuf” and “append” is filtered. More
visualization examples of filtering are given in Appendix D.

In the era of “Big Code” (Allamanis et al., 2018), the programming platforms,
such as GitHub and Stack Overflow, have generated massive open-source code
data. With the assumption of “Software Naturalness” Hindle et al. (2016), pre-
trained models Vaswani et al. (2017); Devlin et al. (2019); Liu et al. (2019)
have been applied in the domain of code intelligence.

Existing code pre-trained models (CodePTMs) can be mainly divided into two
categories: structure-free methods Feng et al. (2020); Svyatkovskiy et al.
(2020) and structure-based methods (Wang et al., 2021b; Niu et al., 2022b).
The former only utilizes the information from raw code texts, while the latter
employs code structures, such as data flow (Guo et al., 2021) and flattened
AST111Abstract syntax tree. (Guo et al., 2022), to enhance the performance of
pre-trained models. For more details, readers can refer to Niu et al. (2022a).
Recently, there exist works that use probing techniques Clark et al. (2019a);
Vig and Belinkov (2019); Zhang et al. (2021) to investigate what CodePTMs
learn. For example, Karmakar and Robbes (2021) first probe into CodePTMs and
construct four probing tasks to explain them. Troshin and Chirkova (2022) also
define a series of novel diagnosing probing tasks about code syntactic
structure. Further, Wan et al. (2022) conduct qualitative structural analyses
to evaluate how CodePTMs interpret code structure. Despite the success, all
these methods lack quantitative characterization on the degree of how well
CodePTMs learn from code structure. Therefore, a research question arises: Can
we develop a new probing way to evaluate how CodePTMs attend code structure
quantitatively?

In this paper, we propose a metric-based probing method, namely, CAT-probing,
to quantitatively evaluate how CodePTMs Attention scores relate to distances
between AST nodes. First, to denoise the input code sequence in the original
attention scores matrix, we classify the rows/cols by token types that are
pre-defined by compilers, and then retain tokens whose types have the highest
proportion scores to derive a filtered attention matrix (see Figure 1(b)).
Meanwhile, inspired by the works (Wang et al., 2020; Zhu et al., 2022), we add
edges to improve the connectivity of AST and calculate the distances between
nodes corresponding to the selected tokens, which generates a distance matrix
as shown in Figure 1(c). After that, we define CAT-score to measure the
matching degree between the filtered attention matrix and the distance matrix.
Specifically, the point-wise elements of the two matrices are _matched_ if
both the two conditions are satisfied: 1) the attention score is larger than a
threshold; 2) the distance value is smaller than a threshold. If only one
condition is reached, the elements are _unmatched_. We calculate the CAT-score
by the ratio of the number of matched elements to the summation of matched and
unmatched elements. Finally, the CAT-score is used to interpret how CodePTMs
attend code structure, where a higher score indicates that the model has
learned more structural information.

Our main contributions can be summarized as follows:

  * •

We propose a novel metric-based probing method CAT-probing to quantitatively
interpret how CodePTMs attend code structure.

  * •

We apply CAT-probing to several representative CodePTMs and perform extensive
experiments to demonstrate the effectiveness of our method (See Section 4.3).

  * •

We draw two fascinating observations from the empirical evaluation: 1) The
token types that PTMs focus on vary with programming languages and are quite
different from the general perceptions of human programmers (See Section 4.2).
2) The ability of CodePTMs to capture code structure dramatically differs with
layers (See Section 4.4).

##  2 Code Background

###  2.1 Code Basics

Each code can be represented in two modals: the source code and the code
structure (AST), as shown in Figure 1(a). In this paper, we use Tree-sitter
222github.com/tree-sitter to generate ASTs, where each token in the raw code
is tagged with a unique type, such as “identifier”, “return” and “=”. Further,
following these works Wang et al. (2020); Zhu et al. (2022), we connect
adjacent leaf nodes by adding data flow edges, which increases the
connectivity of AST. The upgraded AST is named as U-AST.

###  2.2 Code Matrices

There are two types of code matrices: the attention matrix and the distance
matrix. Specifically, the attention matrix denotes the attention score
generated by the Transformer-based CodePTMs, while the distance matrix
captures the distance between nodes in U-AST. We transform the original
subtoken-level attention matrix into the token-level attention matrix by
averaging the attention scores of subtokens in a token. For the distance
matrix, we use the shortest-path length to compute the distance between the
leaf nodes of U-AST. Our attention matrix and distance matrix are shown in
Figure 1(b) and Figure 1(c), respectively.

##  3 CAT-probing

###  3.1 Code Matrices Filtering

As pointed out in Zhou et al. (2021), the attention scores in the attention
matrix follow a long tail distribution, which means that the majority of
attention scores are very small. To address the problem, we propose a simple
but effective algorithm based on code token types to remove the small values
in the attention matrix. For space limitation, we summarize the pseudocodes of
the algorithm in Appendix Alg.1. We only keep the rows/cols corresponding to
frequent token types in the original attention matrix and distance matrix to
generate the selected attention matrix and distance matrix.

###  3.2 CAT-score Calculation

After the two code matrices are filtered, we define a metric called CAT-score,
to measure the commonality between the filtered attention matrix $\mathbf{A}$
and the distance matrix $\mathbf{D}$. Formally, the CAT-score is formulated
as:

$\text{CAT-
score}=\frac{\sum_{C}\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbbm{1}_{\mathbf{A}_{ij}>\theta_{A}\text{
and
}\mathbf{D}_{ij}<\theta_{D}}}{\sum_{C}\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbbm{1}_{\mathbf{A}_{ij}>\theta_{A}\text{
or }\mathbf{D}_{ij}<\theta_{D}}},$ (1)

where $C$ is the number of code samples, $n$ is the length of $\mathbf{A}$ or
$\mathbf{D}$, $\mathbbm{1}$ is the indicator function, $\theta_{A}$ and
$\theta_{D}$ denotes the thresholds to filter matrix $\mathbf{A}$ and
$\mathbf{D}$, respectively. Specifically, we calculate the CAT-score of the
last layer in CodePTMs. The larger the CAT-score, the stronger the ability of
CodePTMs to attend code structure.

##  4 Evaluation

###  4.1 Experimental Setup

#### Task

We evaluate the efficacy of CAT-probing on code summarization, which is one of
the most challenging downstream tasks for code representation. This task aims
to generate a natural language (NL) comment for a given code snippet, using
smoothed BLEU-4 scores (Lin and Och, 2004) as the metric.

#### Datasets

We use the code summarization dataset from CodeXGLUE (Lu et al., 2021) to
evaluate the effectiveness of our methods on four programming languages (short
as PLs), which are JavaScript, Go, Python and Java. For each programming
language, we randomly select $C=3,000$ examples from the training set for
probing.

#### Pre-trained models

We select four models, including one PTM, namely RoBERTa (Liu et al., 2019),
and three RoBERTa-based CodePTMs, which are CodeBERT (Feng et al., 2020),
GraphCodeBERT (Guo et al., 2021), and UniXcoder (Guo et al., 2022). All these
PTMs are composed of $12$ layers of Transformer with $12$ attention heads. We
conduct layer-wise probing on these models, where the layer attention score is
defined as the average of $12$ heads’ attention scores in each layer. The
comparison of these models is introduced in Appendix B. And the details of the
experimental implementation are given in Appendix C.

In the experiments, we aim to answer the three research questions in the
following:

  * •

RQ1(Frequent Token Types): What kind of language-specific frequent token types
do these CodePTMs pay attention to?

  * •

RQ2(CAT-probing Effectiveness): Is CAT-probing an effective method to evaluate
how CodePTMs attend code structure?

  * •

RQ3(Layer-wise CAT-score): How does the CAT-score change with layers?

(a) Go (b) Java (c) JavaScript (d) Python

Figure 2: Visualization of the frequent token types on four programming
languages.

###  4.2 Frequent Token Types

Figure 2(a)-(d) demonstrates the language-specific frequent token types for
four PLs, respectively. From this figure, we see that: 1) Each PL has its
language-specific frequent token types and these types are quite different.
For example, the Top-3 frequent token types for Java are “public”, “s_literal”
and “return”, while Python are “for”, “if”, “)”. 2) There is a significant gap
between the frequent token types that CodePTMs focus on and the general
perceptions of human programmers. For instance, CodePTMs assigned more
attention to code tokens such as brackets. 3) Attention distribution on Python
code snippets significantly differs from others. This is caused by Python
having lesser token types than other PLs; thus, the models are more likely to
concentrate on a few token types.

(a) Go

(b) Java

(c) JavaScript

(d) Python

Figure 3: Comparisons between the CAT-score and the performance on code
summarization task.

###  4.3 CAT-probing Effectiveness

To verify the effectiveness of CAT-probing, we compare the CAT-scores with the
models’ performance on the test set (using both best-bleu and best-ppl
checkpoints). The comparison among different PLs is demonstrated in Figure 3.
We found strong concordance between the CAT-score and the performance of
encoder-only models, including RoBERTa, CodeBERT, and GraphCodeBERT. This
demonstrates the effectiveness of our approach in bridging CodePTMs and code
structure. Also, this result (GraphCodeBERT > CodeBERT > RoBERTa) suggests
that for PTMs, the more code features are considered in the input and pre-
training tasks, the better structural information is learned.

In addition, we observe that UniXcoder has completely different outcomes from
the other three CodePTMs. This phenomenon is caused by UniXcoder utilizing
three modes in the pre-training stage (encoder-only, decoder-only, and
encoder-decoder). This leads to a very different distribution of learned
attention and thus different results in the CAT-score.

###  4.4 Layer-wise CAT-score

We end this section with a study on layer-wise CAT-scores. Figure 4 gives the
results of the CAT-score on all the layers of PTMs. From these results, we
observe that: 1) The CAT-score decreases in general when the number of layers
increases on all the models and PLs. This is because attention scores
gradually focus on some special tokens, reducing the number of matching
elements. 2) The relative magnitude relationship (GraphCodeBERT > CodeBERT >
RoBERTa) between CAT-score is almost determined on all the layers and PLs,
which indicates the effectiveness of CAT-score to recognize the ability of
CodePTMs in capturing code structure. 3) In the middle layers (4-8), all the
results of CAT-score change drastically, which indicates the middle layers of
CodePTMs may play an important role in transferring general structural
knowledge into task-related structural knowledge. 4) In the last layers
(9-11), CAT-scores gradually converge, i.e., the models learn the task-
specific structural knowledge, which explains why we use the score at the last
layer in CAT-probing.

(a) Go

(b) Java

(c) JavaScript

(d) Python

Figure 4: Layer-wise CAT-score results.

##  5 Conclusion

In this paper, we proposed a novel probing method named CAT-probing to explain
how CodePTMs attend code structure. We first denoised the input code sequences
based on the token types pre-defined by the compilers to filter those tokens
whose attention scores are too small. After that, we defined a new metric CAT-
score to measure the commonality between the token-level attention scores
generated in CodePTMs and the pair-wise distances between corresponding AST
nodes. Experiments on multiple programming languages demonstrated the
effectiveness of our method.

##  6 Limitations

The major limitation of our work is that the adopted probing approaches mainly
focus on encoder-only CodePTMs, which could be just one aspect of the inner
workings of CodePTMs. In our future work, we will explore more models with
encoder-decoder architecture, like CodeT5 (Wang et al., 2021b) and PLBART
(Ahmad et al., 2021), and decoder-only networks like GPT-C (Svyatkovskiy et
al., 2020).

## Acknowledgement

This work has been supported by the National Natural Science Foundation of
China under Grant No. U1911203, the National Natural Science Foundation of
China under Grant No. 62277017, Alibaba Group through the Alibaba Innovation
Research Program, and the National Natural Science Foundation of China under
Grant No. 61877018, The Research Project of Shanghai Science and Technology
Commission (20dz2260300) and The Fundamental Research Funds for the Central
Universities. And the authors would like to thank all the anonymous reviewers
for their constructive and insightful comments on this paper.

## References

  * Ahmad et al. (2021) Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021.  Unified pre-training for program understanding and generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2655–2668, Online. Association for Computational Linguistics.
  * Allamanis et al. (2018) Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018.  A survey of machine learning for big code and naturalness.  _ACM Computing Surveys (CSUR)_ , 51(4):81.
  * Clark et al. (2019a) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019a.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Clark et al. (2019b) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019b.  What does bert look at? an analysis of bert’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A pre-trained model for programming and natural languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1536–1547, Online. Association for Computational Linguistics.
  * Guo et al. (2022) Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022.  UniXcoder: Unified cross-modal pre-training for code representation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7212–7225, Dublin, Ireland. Association for Computational Linguistics.
  * Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021.  GraphCodeBERT: Pre-training code representations with data flow.  In _International Conference on Learning Representations_.
  * Hindle et al. (2016) Abram Hindle, Earl T. Barr, Mark Gabel, Zhendong Su, and Premkumar T. Devanbu. 2016\.  On the naturalness of software.  _Commun. ACM_ , 59(5):122–131.
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  Codesearchnet challenge: Evaluating the state of semantic code search.  _CoRR_ , abs/1909.09436.
  * Kanade et al. (2020) Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020.  Learning and evaluating contextual embedding of source code.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 5110–5121. PMLR.
  * Karmakar and Robbes (2021) Anjan Karmakar and Romain Robbes. 2021.  What do pre-trained code models know about code?  In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , pages 1332–1336. IEEE.
  * Kim et al. (2020) Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. 2020.  Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction.  In _International Conference on Learning Representations_.
  * Lin and Och (2004) Chin-Yew Lin and Franz Josef Och. 2004.  ORANGE: a method for evaluating automatic evaluation metrics for machine translation.  In _COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics_ , pages 501–507, Geneva, Switzerland. COLING.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021.  Codexglue: A machine learning benchmark dataset for code understanding and generation.  _CoRR_ , abs/2102.04664.
  * Niu et al. (2022a) Changan Niu, Chuanyi Li, Bin Luo, and Vincent Ng. 2022a.  Deep learning meets software engineering: A survey on pre-trained models of source code.  _CoRR_ , abs/2205.11739.
  * Niu et al. (2022b) Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, and Bin Luo. 2022b.  Spt-code: Sequence-to-sequence pre-training for learning the representation of source code.  _arXiv preprint arXiv:2201.01549_.
  * Sontakke et al. (2022) Ankita Nandkishor Sontakke, Manasi Patwardhan, Lovekesh Vig, Raveendra Kumar Medicherla, Ravindra Naik, and Gautam Shroff. 2022.  Code summarization: Do transformers really understand code? In _Deep Learning for Code Workshop_.
  * Svyatkovskiy et al. (2020) Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020.  Intellicode compose: code generation using transformer.  _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_.
  * Troshin and Chirkova (2022) Sergey Troshin and Nadezhda Chirkova. 2022.  Probing pretrained models of source code.  _arXiv preprint arXiv:2202.08975_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Vig and Belinkov (2019) Jesse Vig and Yonatan Belinkov. 2019.  Analyzing the structure of attention in a transformer language model.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 63–76, Florence, Italy. Association for Computational Linguistics.
  * Wan et al. (2022) Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu, and Hai Jin. 2022.  What do they capture? - A structural analysis of pre-trained language models for source code.  _CoRR_ , abs/2202.06840.
  * Wang et al. (2020) Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. 2020.  Detecting code clones with graph neural network and flow-augmented abstract syntax tree.  In _2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , pages 261–271.
  * Wang et al. (2021a) Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, and Xin Jiang. 2021a.  Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation.  _arXiv preprint arXiv:2108.04556_.
  * Wang and Li (2021) Yanlin Wang and Hui Li. 2021.  Code completion by modeling flattened abstract syntax trees as graphs.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14015–14023.
  * Wang et al. (2021b) Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021b.  CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8696–8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2021) Sheng Zhang, Xin Zhang, Weiming Zhang, and Anders Søgaard. 2021.  Sociolectal analysis of pretrained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4581–4588, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhou et al. (2021) Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 11106–11115.
  * Zhu et al. (2022) Renyu Zhu, Lei Yuan, Xiang Li, Ming Gao, and Wenyuan Cai. 2022.  A neural network architecture for program understanding inspired by human behaviors.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5142–5153, Dublin, Ireland. Association for Computational Linguistics.
  * Zügner et al. (2021) Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. 2021.  Language-agnostic representation learning of source code from structure and context.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.

##  Appendix A Frequent Token Types Filtering Algorithm

Algorithm 1 describes the procedure to generate frequent token types.

Algorithm 1 Frequent Token Type Selection

1: Language $lang$

2: Frequent token type list $type\\_list$

3:rank = len(token types) * [0] $\triangleright$ Initialize rank for each
token type

4:for $t$ in token types do

5: for $m$ in CodePTM models do

6: confidence[$t$,$m$] = 0

7: for $c$ in code cases do

8: $att$ = get_att($m$,$lang$,$c$) $\triangleright$ Get attention matrix

9: $mask\\_theta$ = is_gt_theta($att$) $\triangleright$ Set $att$ position
greater than $\theta_{A}$ to 1, otherwise 0

10: $mask\\_type$ = is_type_t($att$) $\triangleright$ Set $att$ position is
type $t$ to 1, otherwise 0

11: $part$ = sum_mat($mask\\_theta\&mask\\_type$) $\triangleright$ Sum all
elements of the matrix

12: $overall$ = sum_mat($mask\\_type$)

13: confidence[$t$,$m$] $\leftarrow$ confidence[$t$,$m$] + $part$ / $overall$
$\triangleright$ Compute confidence

14: end for

15: confidence[$t$,$m$] $\leftarrow$ confidence[$t$,$m$] / len($c$)
$\triangleright$ Average confidence

16: rank[$t$] $\leftarrow$ rank[$t$] + get_rank(confidence,$m$)
$\triangleright$ Rank confidence for $m$, and sum rank for $t$

17: end for

18:end for

19: token type list includes those $t$ with rank[$t$]<40

##  Appendix B Comparison of CodePTMs

Table 2 gives the comparison of the PTMs used in our experiments from three
perspectives: the inputs of the model, the pre-training task, and the training
mode.

##  Appendix C Experimental Implementation

We keep the same hyperparameter setting for all CodePTMs. The detailed
hyperparameters are given in Table 1.

Our codes are implemented based on PyTorch. All the experiments were conducted
on a Linux server with two interconnected NVIDIA-V100 GPUs.

Hyperparameter | value
---|---
Batch Size | 48
Learning Rate | 5e-5
Weight Decay | 0.0
Epsilon | 1e-8
Epochs | 15
Max Source Length | 256
$\theta_{A}$ | third quartile of values in $A$
$\theta_{D}$ | first quartile of values in $D$
Table 1:  Hyperparameters for CAT-probing  Models | Inputs | Pre-training Tasks | Training Mode
---|---|---|---
RoBERTa | Natural Language (NL) | Masked Language Modeling (MLM) | Encoder-only
CodeBERT | NL-PL Pairs | MLM+Replaced Token Detection (RTD) | Encoder-only
GraphCodeBERT | NL-PL Pairs & AST | MLM+Edge Prediction+Node Alignment | Encoder-only
UniXcoder | NL-PL Pairs & Flattened AST | MLM | Encoder &
ULM (Unidirectional Language Modeling) | Decoder &
Denoising Objective (DNS) | Encoder-decoder
Table 2: The comparison of different language models mentioned in this paper.

##  Appendix D Case Study

In addition to the example visualized in Figure 1, we have carried out three
new examples to show the effectiveness of the filtering strategy in Section
3.1, The visualizations are shown in Table 3.

Source Code | Attention Heatmap | Attention Heatmap with
---|---|---
Token Type Selection
|  |
|  |
|  |
Table 3: Heatmaps of the averaged attention weights in the last layer before
and after using token selection, including Go, Java, and JavaScript code
snippets (from top to bottom).