# Explainable Graph Neural Networks for Observation Impact Analysis in
Atmospheric State Estimation

Hyeon-Ju Jeon1, Jeon-Ho Kang1, In-Hyuk Kwon1, O-Joun Lee2 Corresponding
author: Hyeon-Ju Jeon (Tel.: +82-2-6480-6425) Hyeon-Ju Jeon†, Jeon-Ho Kang,
In-Hyuk Kwon
Data Assimilation Group
Korea Institute of Atmospheric Prediction Systems
Seoul, Republic of Korea
<EMAIL_ADDRESS>&O-Joun Lee
Dept. of Artificial Intelligence
The Catholic University of Korea
Bucheon, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

This paper investigates the impact of observations on atmospheric state
estimation in weather forecasting systems using graph neural networks (GNNs)
and explainability methods. We integrate observation and Numerical Weather
Prediction (NWP) points into a meteorological graph, extracting $k$-hop
subgraphs centered on NWP points. Self-supervised GNNs are employed to
estimate the atmospheric state by aggregating data within these $k$-hop radii.
The study applies gradient-based explainability methods to quantify the
significance of different observations in the estimation process. Evaluated
with data from 11 satellite and land-based observations, the results highlight
the effectiveness of visualizing the importance of observation types,
enhancing the understanding and optimization of observational data in weather
forecasting.

††footnotetext: † Correspondence<EMAIL_ADDRESS>Tel.: +82-2-6480-6497

_K_ eywords Explainable Graph Neural Networks $\cdot$ Impact Analysis $\cdot$
Atmospheric State Estimation $\cdot$ Multi-modal Observation Data

##  1 Introduction

Weather forecasting, a critical component in industries like transportation
and manufacturing, relies heavily on Numerical Weather Prediction (NWP)
systems, which are based on 3D physical models and dynamical equations [1, 2].
For NWP systems to predict future atmospheric states effectively, they require
accurate current atmospheric states as initial values. This necessity
underscores the importance of a data assimilation (DA) system, which
approximates the true atmospheric states by merging observations with
prediction results from dynamical models [3]. The integration of a wide range
of observations, from sources like aircraft, radiosondes, and satellites, is
crucial for enhancing the DA system’s accuracy [4].

Traditional methods to assess the impact of observations on weather forecasts
include forecast sensitivity to observation (FSO) and its variations, such as
ensemble FSO and hybrid FSO [2, 5, 6]. These methods compute the gradient of
the forecast with respect to the assimilated observations within the DA system
but are limited by their dependency on the system’s structural changes.

Our study proposes a novel approach using Graph Neural Networks (GNNs) [7, 8,
9] to estimate the impact of observations independently of the system’s
structure. GNNs have been increasingly employed in meteorological predictions,
including solar radiation and sea surface temperature predictions, by
capturing variable interactions in neighboring regions [10, 11, 12]. The
GraphCast [13] model, for instance, transforms the NWP system’s 3D grid into a
hierarchical graph to capture long-range spatial interactions using GNNs.
However, it does not incorporate the latest observations in its predictive
model. To our knowledge, no existing model fuses observations with NWP grids
in a graph format for current atmospheric state estimation using GNNs.

Additionally, we apply explainability methods to evaluate the impact of
observations on current atmospheric state estimation [14]. These methods,
including model gradient analysis and input perturbation, have been previously
used for validating deep learning models in Earth system science [15, 16, 17,
18]. Our study extends the use of these methods to feature analysis in
atmospheric science, providing a novel perspective.

The contributions of this paper include:

  * •

Defining a meteorological graph that includes real atmospheric state and
observational data, addressing the challenge of unstructured data.

  * •

Developing a self-supervised graph convolutional network (GCN) model for
atmospheric state estimation, demonstrating superior performance over other
baseline models.

  * •

Using explainability methods to estimate and visually represent the impact of
observations on accurate atmospheric state predictions.

##  2 Atmospheric State Estimation

###  2.1 Composing Meteorological Contexts

Weather forecasting heavily relies on both local and global weather
conditions, but estimation of the current atmospheric state is predominantly
influenced by nearby observations. In the Korean Integrated Model (KIM),
observations impact NWP grid points within a 50km radius. This study treats
atmospheric state estimation as a node-level regression task, aiming to
estimate the states of NWP grid points at time $t$ using data from NWP grid
points at time $t-1$ and observations at time $t$.

A meteorological graph, $\mathcal{N}=(V,E,\lambda_{v})$, was constructed,
comprising observation points and NWP grid points as nodes, where $V\ni v_{i}$
represents the nodes, $E$ the edges, and $\lambda_{v}$ a function mapping
nodes to their types. Adjacency between nodes is determined by a 50km radius
proximity.

In this graph $\mathcal{N}$, various weather variables are assigned as node
attributes $a_{i}$ to each node $v_{i}$. The type and quantity of these
variables differ based on node types (observation or NWP points) and
observation types (e.g., IASI, GK2A, etc.). To standardize different feature
vector sizes, a projection layer $p(\mathcal{N},v_{i})_{\lambda_{v}}$ maps
each node $v_{i}$ to a fixed size vector $h_{i}\in\mathbb{R}^{d}$, with $d$
being the embedding dimension.

Given the extensive scale of the 3D NWP grid points, which are uniformly
distributed across the globe at 50km intervals, the meteorological graph is
large, posing challenges for GNNs application. In addition, local observations
have more influence on the estimation of the atmospheric state than the
broader global weather context. Therefore, the study extracts ego-centric
subgraphs centered on NWP points from the main graph. Each subgraph
$g_{m}(v_{i})$ comprises $k$-hop neighbors of $v_{i}$ and their
interconnecting edges. Utilizing these subgraphs as individual samples, we
effectively aggregate local weather contextual information within the $k$-hop
range. This approach allows for the application of GNNs to atmospheric state
estimation while managing computational costs.

###  2.2 Pre-training with Node Feature Reconstruction

Observational data in meteorology, such as temperature, can have varying
implications depending on the surrounding weather conditions. For instance, a
temperature of 305K might signify different things in tropical regions
compared to mid-latitude areas. To interpret these variations, GCNs are pre-
trained on a node attribute reconstruction task. This process enables the
determination of the specific meanings of observations in diverse
meteorological contexts.

The GCN-based graph encoder takes a meteorological context subgraph with node
initial feature vectors ($h_{i}$) passed through the projection layer and
composes final vector representations of nodes ($H_{i}$). This pre-training
phase involves training node representations in meteorological contexts
alongside the attribute reconstruction task. The graph encoder, utilizing GCN
layers, concurrently learns node features and graph structures. The graph
encoder is formulated as:

$\displaystyle
H^{(l+1)}=\sigma(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}H^{(l)}W^{(l)}),$
(1)

where $W^{(l)}$ indicates a weight matrix in $l^{th}$ layer, $H^{(l)}$ is the
feature matrix generated by the $l^{th}$ layer. In addition,
$\tilde{D}_{ii}=\sum_{j}\tilde{A}_{ij}$, and $\tilde{A}=A+I_{N}$ is the
adjacency matrix of the context subgraph $g_{m}$, where $I_{N}$ indicates the
identity matrix. Also, $H^{(0)}$ is the set of $h_{i}$.

The objective of this pre-training is to approximate the reconstructed node
attributes to the actual attributes as closely as possible, minimizing
reconstruction error. This error is quantified using the L2 loss, with the
objective function defined as:

$\displaystyle L_{ssl}=\lVert a,\hat{a}\rVert_{2}^{2}+\psi\lVert
W\rVert_{2}^{2},$ (2)

where $a$ and $\hat{a}$ are the actual and predicted node attributes,
respectively, and $\psi$ is a weight for the regularization term.

###  2.3 Estimating Current Atmospheric States

In our approach to atmospheric state estimation, node representations from the
graph encoder are transformed into representations of ego-centric subgraphs
through graph pooling. Subsequently, a Multi-Layer Perceptron (MLP) is
employed to estimate the current atmospheric states of the central nodes,
which are NWP points, based on these subgraph representations. This process
can be formulated as:

$\displaystyle Z=\textrm{MLP}(\textrm{POOL}(H^{(n)})),$ (3)

where $H^{(n)}=\textrm{GCN}(h,A)$ is the final node representations from the
GCN layers. $\textrm{POOL}(\cdot)$ indicates the graph pooling layer, which
conducts average pooling for node representations in context subgraphs.
Thereby, subgraph representations reflect weather context within the $k$-hop
radius of NWP points. Finally, the $\textrm{MLP}(\cdot)$ layer maps high
dimensional subgraph representations to the weather variables and predicts the
current conditions.

The objective of the fine-tuning is to accurately predict the current
atmospheric states, aligning them as closely as possible with their actual
values. During training, the regression errors over all subgraphs and weather
variables are computed using the L2 loss, which is defined as:

$\displaystyle L_{reg}=\lVert Z,\hat{Z}\rVert_{2}^{2}+\psi\lVert
W\rVert_{2}^{2},$ (4)

where $Z$ and $\hat{Z}$ are the true and predicted atmospheric states,
respectively.

##  3 Observation Impact Analysis

Estimating the impact of observations in meteorology involves understanding
the contribution of a node $v_{j}$ within a weather context $g_{m}(v_{i})$ to
the predicted atmospheric states $\hat{Z}$. The sensitivity of node $v_{j}$ to
the prediction $\hat{Z}_{g_{m}(v_{i})}$, represented as
$S_{i,j}(H^{(n)}_{j},\hat{Z}_{g_{m}(v_{i})})$, is used to estimate this
impact. The overall importance of observation $v_{j}$ in the meteorological
graph $\mathcal{N}$ is then calculated by averaging sensitivities across
different subgraphs:

$\displaystyle S_{j}=\frac{1}{i}\sum_{i}S_{i,j}.$ (5)

This approach allows us to aggregate the importance of each observation type,
thereby determining the impact of each observation type on estimating
atmospheric states.

To quantitatively measure the impact, we use gradients from the prediction
model, commonly applied in graph reasoning. Three methods are used to provide
explanations based on the gradients and are empirically compared. The
contrastive gradient-based saliency map (SA) [15] utilizes these gradients to
indicate how changes in the input could lead to variations in the output. This
can be formulated as:

$\displaystyle L_{Gradient}=\textrm{ReLU}\left(\frac{\partial\hat{Z}}{\partial
H^{(n)}}\right).$ (6)

Additionally, the Grad-CAM method [15] focuses on the last graph convolutional
layers rather than the input space, identifying node importance using back-
propagation gradients. This method computes weights $\alpha_{n}$ as the
average of the gradients and represents node importance as a weighted sum of
feature maps:

$\displaystyle L_{Grad-
CAM}=\textrm{ReLU}\left(\sum_{n}\alpha_{n}H^{(n)}\right).$ (7)

Layer-wise Relevance Propagation (LRP) [19] provides another perspective by
reverse-propagating the prediction from the output of the graph convolutional
layers back to the input features. It decomposes the prediction score into
neuron importance scores based on hidden features and weights. The relevance
score propagation at the neuron level is given by:

$\displaystyle
R_{a}=\sum_{b}\frac{h_{a}^{(n)}w_{ab}}{\sum_{k}h_{k}^{(n)}w_{kb}}R_{b},$ (8)

where $h_{a}^{(n)}$ is the activation of the $a^{th}$ hidden neuron in the
$n^{th}$ layer, and $w_{ab}$ is the weight connecting the $a^{th}$ neuron to
the $b^{th}$ neuron. This method intuitively assigns a larger fraction of the
target neuron score to neurons contributing more significantly to the target
neuron activation.

##  4 Experimental Results and Discussion

In this section, we validate the proposed atmospheric state estimation system
and visualize the importance of observation. We collected observation data and
KIM data used in the Korea Meteorological Administration (KMA). We used the
output (i.e., u-component of wind (U), v-component of wind (V), temperature
(T), relative humidity (Q)) of the DA system as the true value since the
output was assumed to be the actual atmospheric state in the NWP system. Also,
the observations were preprocessed by the KMA system. AIRCRAFT (U, V, T),
GPSRO (banding angle (BA)), SONDE (U, V, T, Q), AMV (brightness temperature
(TB)), AMSU-A (TB), AMSR2 (TB), ATMS (TB), CrIS (TB), GK2A (TB), IASI (TB),
and MHS (TB), a total of 11 satellite and ground observations [4] that have
different variables (U, V, T, Q, TB, and BA) were used in the experiment. We
have restricted the region to 500 hPa over East Asia. The observation and NWP
data (20 April 2021 to 30 April 2021) were used as the training dataset. We
then evaluated the proposed model using the next ten days (1 May 2021 to 10
May 2021).

###  4.1 Effectiveness of the Proposed Estimation Model

To explore the contribution of the graph-based data structure, we first
compare the prediction performance of our model with fully connected networks
(FCN). In addition, we also evaluate the influence of the self-supervised
learning and attention mechanisms through ablation tests.

Model | Variables | $RMSE$ | $MAE$ | $R^{2}$ | $var$
---|---|---|---|---|---
FCN | U(m/s) | 0.20 | 0.17 | 0.60 | 0.60
V(m/s) | 0.13 | 0.11 | 0.34 | 0.34
T(K) | 0.23 | 0.20 | 0.83 | 0.83
Q(kg/kg) | 0.07 | 0.07 | 0.37 | 0.37
GCN | U(m/s) | 0.20 | 0.17 | 0.64 | 0.64
V(m/s) | 0.10 | 0.07 | 0.56 | 0.56
T(K) | 0.22 | 0.20 | 0.88 | 0.88
Q(kg/kg) | 0.05 | 0.05 | 0.53 | 0.53
GAT | U(m/s) | 0.18 | 0.16 | 0.67 | 0.67
V(m/s) | 0.10 | 0.07 | 0.56 | 0.56
T(K) | 0.22 | 0.19 | 0.88 | 0.88
Q(kg/kg) | 0.04 | 0.04 | 0.56 | 0.56
Proposed w/ GCN | U(m/s) | 0.16 | 0.13 | 0.73 | 0.74
V(m/s) | 0.09 | 0.06 | 0.73 | 0.73
T(K) | 0.19 | 0.16 | 0.93 | 0.93
Q(kg/kg) | 0.03 | 0.03 | 0.64 | 0.64
Proposed w/ GAT | U(m/s) | 0.17 | 0.14 | 0.72 | 0.72
V(m/s) | 0.09 | 0.07 | 0.71 | 0.72
T(K) | 0.20 | 0.18 | 0.90 | 0.90
Q(kg/kg) | 0.04 | 0.04 | 0.62 | 0.62
Table 1: A performance comparison of the proposed model with the baseline
models.

In Table 1, GCN, Graph Attention Network (GAT), and the proposed models have
higher accuracy than the FCN model for all evaluation metrics which are widely
used in the previous study [10]. Among the evaluation metrics, the accuracy of
$var$ and $R^{2}$ show similar results, but they are calculated with different
variances. Therefore, if the two values are the same, we can assess that the
error of the model is unbiased. We have verified that graph-structured
meteorological data can improve the performance of current atmospheric state
prediction. The proposed model based on self-supervised learning achieved
significantly better performance than GCN and GAT. A node representation that
has been pre-trained on the node feature reconstruction considering
meteorological contexts is more effective in estimating current atmospheric
states than the cases without opportunities to learn spatial correlations
between weather variables. In particular, the performance of GAT, which
considers nodes’ importance based on their attention scores, is slightly
better than GCN for all meteorological variables. However, the U and T
variables show very small performance deviations. We assume that for
estimating certain weather variables, all the closely located observations can
have uniformly high (or low) importance.

On the other hand, GCN achieved better performance than GAT, as the backbone
of our model. The proposed model based on GCN increased the accuracy by
14.06%, 30.36%, 5.68%, and 20.75% over vanilla GCN, while the proposed model
based on GAT increased the accuracy by 7.46%, 26.79%, 2.27%, and 10.71% over
vanilla GAT for each variable, respectively. Pre-training the feature
reconstruction task enables GNN models to understand correlations between
weather variables. Then, the accuracy of GCN could be significantly improved,
since GCN models merely aggregate neighboring nodes’ features with mean
aggregator.

###  4.2 Stability of the Explainability Methods

The results of the explainability methods provide insights into correlations
between weather variables in a human-understandable way. However, it is
difficult to evaluate these methods from this perspective due to the lack of
ground truth and criteria for human understandability. Also, comparing
different explainability methods requires a lot of time and human resources to
investigate the results for each meteorological context subgraph. Therefore,
evaluation metrics should evaluate the results from the perspective of the
model, such as whether the explanations are faithful to the model. We define
$Fidelity+$ as the difference in accuracy obtained by occluding fixed
percentages of input features assessed as important. In addition, $Fidelity-$
indicates the difference between the predictions obtained by masking
unimportant input features while retaining the important features. We applied
the explanation method to the proposed model based on GCN, which has the best
performance in atmospheric state estimation.

Methods | 10% | 20%
---|---|---
$Fidelity+$ | $Fidelity-$ | $Fidelity+$ | $Fidelity-$
SA | 0.17 | 0.09 | 0.35 | 0.19
Grad-CAM | 0.20 | 0.08 | 0.36 | 0.17
LRP | 0.25 | 0.09 | 0.39 | 0.16
Table 2: Fidelity comparisons among explainability methods.

In Table 2, we occlude $10\%$ and $20\%$ of the input features and then
compare their fidelity scores. In the $Fidelity+$ metric, LRP outperforms the
other explainability methods, showing that the propagation-based technique is
more desirable than traditional gradient-based techniques for GNNs. We can
assume that SA and Grad-CAM have not made a sophisticated transition from the
image domain to the graph-specific design, which is the main reason for their
lower performance than the propagation-based LRP methods. The $Fidelity-$
scores are similar to the $Fidelity+$ results, but the performance differences
between the models are small. For low importance nodes, all methods perform
consistently in estimating the node importance. When occluding nodes in the
bottom 20% of importance, the $Fidelity+$ score is significantly large due to
the loss of a large number of nodes. However, masking nodes in the bottom 10%
of importance does not have a significant impact on accuracy.

Figure 1: The averaged impact of each observation type.

As a result, the averaged effect of each type of observation on the estimation
of the current state of the atmosphere can be visualized as shown in Figure 1.
The variation of importance by the observation type assessed by SA is small
compared to other methods. Node importance is widely distributed across
observation types because the SA method has explainable noise and poor
localization performance [15]. The Grad-CAM, with its improved localization
performance, has the largest variation in importance by observation type.
Therefore, the method has a high potential for use in cases where $n$
observation types with very large impacts or $n$ observation types with very
small impacts need to be selected. Although the LRP method estimates different
degrees of impact for different observation types, the ranking of importance
observation is similar.

(a) SA

(b) Grad-CAM

(c) LRP

Figure 2: Time series of observations impact.

Figure 2 shows the impact by observation type evaluated for each time period,
and observation types with an importance of 0 are those for which there are no
observations exist in the target region (i.e., East Asia) at that time. The
time series pattern of the impact by observation type evaluated by SA and
Grad-CAM in Figure 2 does not change significantly. In addition, compared to
the SA method, the Grad-CAM method assigns relatively greater importance to
high-impact observation types, and the high-impact observation types are
emphasized. The LRP method calculates the relative importance for each
observation type, and the impact of each observation is evaluated differently
for each meteorological context. The property of conservation in the LRP
method is significant in interpreting the physical attributions in the graph-
level prediction task.

##  5 Conclusion

In this paper, we propose an atmospheric state estimation model based on self-
supervised graph neural networks. Then, we applied explainability methods to
analyze the impact of observations on the current atmospheric state and
visualize the importance of observation type.

## Acknowledgments

This work was supported in part by the R&D project “Development of a Next-
Generation Data Assimilation System by the Korea Institute of Atmospheric
Prediction System (KIAPS)”, funded by the Korea Meteorological Administration
(KMA2020-02211) (H.-J.J.) and in part by the National Research Foundation of
Korea (NRF) Grant funded by the Korea government (MSIT) (No. 2022R1F1A1065516
and No. 2022K1A3A1A79089461) (O.-J.L.).

## References

  * [1] Ivana Štulec, Kristina Petljak, and Dora Naletina.  Weather impact on retail sales: How can weather derivatives help with adverse weather deviations?  Journal of Retailing and Consumer Services, 49:1–10, July 2019.
  * [2] Shunji Kotsuki, Kenta Kurosawa, and Takemasa Miyoshi.  On the properties of ensemble forecast sensitivity to observations.  Quarterly Journal of the Royal Meteorological Society, 145(722):1897–1914, May 2019.
  * [3] In-Hyuk Kwon, Hyo-Jong Song, Ji-Hyun Ha, Hyoung-Wook Chun, Jeon-Ho Kang, Sihye Lee, Sujeong Lim, Youngsoon Jo, Hyun-Jun Han, Hanbyeol Jeong, Hui-Nae Kwon, Seoleun Shin, and Tae-Hun Kim.  Development of an operational hybrid data assimilation system at kiaps.  Asia-Pacific Journal of Atmospheric Sciences, 54(S1):319–335, June 2018.
  * [4] Jeon-Ho Kang, Hyoung-Wook Chun, Sihye Lee, Ji-Hyun Ha, Hyo-Jong Song, In-Hyuk Kwon, Hyun-Jun Han, Hanbyeol Jeong, Hui-Nae Kwon, and Tae-Hun Kim.  Development of an observation processing package for data assimilation in kiaps.  Asia-Pacific Journal of Atmospheric Sciences, 54(S1):303–318, June 2018.
  * [5] Eugenia Kalnay, Yoichiro Ota, Takemasa Miyoshi, and Junjie Liu.  A simpler formulation of forecast sensitivity to observations: application to ensemble kalman filters.  Tellus A: Dynamic Meteorology and Oceanography, 64(1):18462, December 2012.
  * [6] Mark Buehner, Ping Du, and Joël Bédard.  A new approach for estimating the observation impact in ensemble–variational data assimilation.  Monthly Weather Review, 146(2):447–465, January 2018.
  * [7] Hyeon-Ju Jeon and Jason J Jung.  Discovering the role model of authors by embedding research history.  Journal of Information Science, 49(4):990–1006, August 2021.
  * [8] Van Thuy Hoang, Hyeon-Ju Jeon, Eun-Soon You, Yoewon Yoon, Sungyeop Jung, and O-Joun Lee.  Graph representation learning and its applications: A survey.  Sensors, 23(8):4168, April 2023.
  * [9] O-Joun Lee, Hyeon-Ju Jeon, and Jason J. Jung.  Learning multi-resolution representations of research patterns in bibliographic networks.  Journal of Informetrics, 15(1):101126, February 2021.
  * [10] Hyeon-Ju Jeon, Min-Woo Choi, and O-Joun Lee.  Day-ahead hourly solar irradiance forecasting based on multi-attributed spatio-temporal graph convolutional network.  Sensors, 22(19):7179, sep 2022.
  * [11] Minbo Ma, Peng Xie, Fei Teng, Bin Wang, Shenggong Ji, Junbo Zhang, and Tianrui Li.  HiSTGNN: Hierarchical spatio-temporal graph neural network for weather forecasting.  Information Sciences, 648:119580, nov 2023.
  * [12] Yuting Yang, Junyu Dong, Xin Sun, Estanislau Lima, Quanquan Mu, and Xinhua Wang.  A cfcc-lstm model for sea surface temperature prediction.  IEEE Geoscience and Remote Sensing Letters, 15(2):207–211, February 2018.
  * [13] Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Peter Battaglia.  Learning skillful medium-range global weather forecasting.  Science, page eadi2336, November 2023.
  * [14] Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji.  Explainability in graph neural networks: A taxonomic survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5782–5799, 2023.
  * [15] Phillip E. Pope, Soheil Kolouri, Mohammad Rostami, Charles E. Martin, and Heiko Hoffmann.  Explainability methods for graph convolutional neural networks.  In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019). IEEE, jun 2019.
  * [16] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec.  Gnnexplainer: Generating explanations for graph neural networks.  In Proceedings of the 32th Advances in Neural Information Processing Systems (NeurIPS 2019), 8-14 December 2019, Vancouver, BC, Canada, pages 9240–9251, 2019.
  * [17] Minh N. Vu and My T. Thai.  Pgm-explainer: Probabilistic graphical model explanations for graph neural networks.  In Proceedings of the 33th Advances in Neural Information Processing Systems (NeurIPS 2020), 6-12 December 2020, virtual, 2020.
  * [18] E. A. Irvine, S. L. Gray, J. Methven, and I. A. Renfrew.  Forecast impact of targeted observations: Sensitivity to observation error and proximity to steep orography.  Monthly Weather Review, 139(1):69–78, January 2011.
  * [19] Federico Baldassarre and Hossein Azizpour.  Explainability techniques for graph convolutional networks.  arXiv preprint arXiv:1905.13686, 2019.


11institutetext: L3S Research Center, Leibniz University Hannover 30167,
Hannover, Germany
11email<EMAIL_ADDRESS>22institutetext: TIB—Leibniz Information Centre for
Science and Technology, Germany 22email<EMAIL_ADDRESS>
# Scholarly Knowledge Extraction
from Published Software Packages

Muhammad Haris 11 0000-0002-5071-1658 Markus Stocker 2211 0000-0001-5492-3212
Sören Auer 2211 0000-0002-0698-2864

###### Abstract

A plethora of scientific software packages are published in repositories,
e.g., Zenodo and figshare. These software packages are crucial for the
reproducibility of published research. As an additional route to scholarly
knowledge graph construction, we propose an approach for automated extraction
of machine actionable (structured) scholarly knowledge from published software
packages by static analysis of their (meta)data and contents (in particular
scripts in languages such as Python). The approach can be summarized as
follows. First, we extract metadata information (software description,
programming languages, related references) from software packages by
leveraging the Software Metadata Extraction Framework (SOMEF) and the GitHub
API. Second, we analyze the extracted metadata to find the research articles
associated with the corresponding software repository. Third, for software
contained in published packages, we create and analyze the Abstract Syntax
Tree (AST) representation to extract information about the procedures
performed on data. Fourth, we search the extracted information in the full
text of related articles to constrain the extracted information to scholarly
knowledge, i.e. information published in the scholarly literature. Finally, we
publish the extracted machine actionable scholarly knowledge in the Open
Research Knowledge Graph (ORKG).

###### Keywords:

Analyzing Software Packages Open Research Knowledge Graph Code Analysis
Abstract Syntax Tree Scholarly Communication Machine Actionability

##  1 Introduction

A variety of general and domain-specific knowledge graphs have been proposed
to represent (scholarly) knowledge in a structured manner [7, 26]. General
purpose knowledge graphs include DBpedia111https://www.dbpedia.org [18],
Wikidata222https://www.wikidata.org/wiki/Wikidata:Main_Page [25], YAGO [23],
etc., whereas domain-specific infrastructures include approaches in Cultural
Heritage [13], KnowLife in Life Sciences [9], Hi-Knowledge in Invasion
Biology333https://hi-knowledge.org [10, 8], COVID-19 Air Quality Data
Collection444https://covid-aqs.fz-juelich.de, Papers With Code in Machine
Learning555https://paperswithcode.org, Cooperation Databank in Social
Sciences666https://cooperationdatabank.org [21], among others. In addition,
knowledge graph technologies have also been employed to describe software
packages in a structured manner [16, 2].

Extending the state-of-the-art, we propose an approach for scholarly knowledge
extraction from published software packages by static analysis of package
contents, i.e., (meta-)data and software (in particular, Python scripts), and
represent the extracted knowledge in a knowledge graph. The main purpose of
this knowledge graph is to capture information about the materials and methods
used in scholarly work described in research articles.

We address the following research question: Can structured scholarly knowledge
be automatically extracted from published software packages? Our approach
consists of the following steps:

  1. 1.

Mining software packages deposited in Zenodo777https://zenodo.org using its
REST API888https://developers.zenodo.org and analyzing the API response to
extract the linked metadata information, i.e, associated scholarly articles.
We complement the approach by leveraging the Software Metadata Extraction
Framework (SOMEF) to parse the README files and extract other related metadata
information (i.e., software name, description, used programming languages).

  2. 2.

Perform static code analysis to extract information about the procedures
performed on data. We utilize Abstract Syntax Tree (AST) representations to
statically analyze program code and identify operations performed on data.

  3. 3.

Identify scholarly knowledge by performing keyword-based search of extracted
information in article full text. Thus, among all the information extracted
from software packages we identify that which is scholarly knowledge.

  4. 4.

Construct a knowledge graph of scholarly knowledge extracted from software
packages. For this purpose, we leverage the Open Research Knowledge Graph
(ORKG)999https://www.orkg.org/orkg/ [14], a production research infrastructure
that supports producing and publishing machine actionable scholarly knowledge.

##  2 Related Work

Several approaches have been suggested to retrieve metadata from software
repositories. Mao et al. [19] proposed the Software Metadata Extraction
Framework (SOMEF) to extract metadata from software packages published on
GitHub. Specifically, the framework employs machine learning-based methods to
extract repository name, software description, citations, reference URLs, etc.
from README files and to represent the metadata in structured formats (JSON-
LD, JSON and RDF). SOMEF was later extended to extract additional metadata and
auxiliary files (e.g., Notebooks, Dockerfiles) from software packages [16].
Moreover, the extended work also supports creating a knowledge graph of parsed
metadata, thus improving search of software deposited in repositories.
Abdelaziz et al. [1] proposed CodeBreaker, a knowledge graph with information
about 1.3 million Python scripts published on GitHub. The graph was embedded
in an IDE to recommend code functions while writing software. Similarly,
GraphGen4Code [2] is a knowledge graph with information about software
included in GitHub repositories. It was generated by analyzing the
functionalities of Python scripts and linking them with the natural language
artefacts (documentation and forum discussions on StackOverflow and
StackExchange). The knowledge graph contains 2 billion triples. Several other
machine learning-based approaches for searching [11] software scripts and
summarization [3, 12] have been proposed. The Pydriller [22] and
GitPython101010https://github.com/gitpython-developers/GitPython frameworks
were proposed to mine information from GitHub repositories, including source
code, commits, branch differences, etc. Similarly, ModelMine [20] mines and
analyzes models included in repositories. Vagavolu et al. [24] presented an
approach that leverages Code2vec [17] and includes semantic graphs with
Abstract Syntax Tree (AST) for performing different software engineering
tasks. [4] presented an AST based-approach for code representation and
considered code data flow mechanisms to suggest code improvements.

##  3 Methodology

In this section, we present our methodology for automatically extracting
_scholarly_ knowledge from software packages and building a knowledge graph
from the extracted meta(data). Figure 1 provides an overview of the key
components.

Figure 1: Pipeline for constructing a knowledge graph of scholarly knowledge
extracted from software packages: 1) Mining software packages from the Zenodo
repository using its REST API; 2) Extracting software metadata by analyzing
the Zenodo API results as well as the GitHub API, using SOMEF; 3) Performing
static code analysis using AST representations of software to extract code
semantics, in particular operations on data; 4) Performing keywords-based
search in article full texts to identify scholarly knowledge; 5) Knowledge
graph construction with scholarly knowledge extracted from software packages.

###  3.1 Mining Software Packages

We mine software packages from the Zenodo repository by leveraging its REST
API. The metadata of each package is analyzed to retrieve its DOI and metadata
about related versions and associated scholarly articles. The versions of
software packages are retrieved by interpreting relation: isVersionOf
metadata, whereas the DOI of the linked article, if available, is fetched
using the relation: cites or relation: isSupplementTo metadata. We also
leverage the Software Metadata Extraction Framework (SOMEF) and GitHub API to
extract additional metadata from software packages, in particular software
name, description, used programming languages, GitHub URL. Since not all
software packages include the cites or isSupplementTo relations in metadata,
we utilize SOMEF to parse the README files of software packages as an
additional approach to extract the DOI of the related scholarly article.

Figure 2: Static code analysis: Exemplary Python script (shortened) included
in a software package. The script lines highlighted with same color show
different procedural changes that a particular variable has undergone.

####  3.1.1 Static Code Analysis

We utilize Abstract Syntax Tree (AST) representations for static analysis of
Python scripts included in software packages. AST provides structured
representations of scripts, omitting unnecessary syntactic details (e.g.,
semicolons, commas, and comments). Our goal is to extract information about
the data used in scripts and the procedures performed on that data. Our
developed Python-based module sequentially reads the scripts contained in
software packages and generates the AST. The implemented procedures and
variables are tokenized and represented as nodes in the tree, which
facilitates the analysis of the code flow. Thus, by traversing the tree we
extracts the information about the data used in the scripts, the procedures
performed on the data and, if available, the output data.

Figure 3: Abstract Syntax Tree (AST) of the script shown in Fig. 2. For
simplicity, the AST is shown only for Lines 1, 10 and 16.

Fig. 2 shows the Python script included in the software
package111111https://zenodo.org/record/5874955. The script shows an example in
which Sample.csv and Reference.csv used as input data, then the operation
LinearSVR is performed on the data, and finally the resulting data score.csv
is generated.

Fig. 3 shows the AST of the Python script (Fig. 2) created using a suitable
Python library121212https://docs.python.org/3/library/ast.html. For
simplicity, we show the AST of lines 1, 10, and 16. In the tree structure, the
name of the node represents the functionality of each line of the script. For
example, line 1 performs a task that reads data and assigns it to a variable.
Therefore, the relevant node in the tree is labelled Assign. We retrieve all
leaf nodes since they represent variables, their values, and procedures.
Analyzing these script semantics, we can then find the flow of data between
procedures. We investigate the flow of variables that contain the input data,
i.e., examining which operations used a particular variable as a parameter.

###  3.2 Identifying Scholarly Knowledge

Not all information extracted from software packages and AST-analyzed program
code is scholarly knowledge. Information is scholarly knowledge if it is
included in a scholarly article. Hence, we filter the information extracted
from software packages for information referred to in the article citing the
software package. For this, we employ keyword-based search. Specifically, we
search for the terms extracted in AST-analyzed program code in the related
article full text. Assuming that the DOI of the related article has been
identified, we fetch the PDF version of the article by utilizing the Unpaywall
REST
<EMAIL_ADDRESS>We make use of the Unpaywall API because, contrary to DOI metadata, it
provides the URL to the PDF version of scholarly articles. In our example
(Fig. 2), the extracted terms (Sample, Reference, read_csv, LinearSVR,
svr.fit, and to_csv) are searched in the PDF and we find Sample, Reference and
LinearSVR are cited in the scholarly article. We thus assume that the
extracted information is scholarly knowledge.

###  3.3 Knowledge Graph Construction

We now construct the knowledge graph with the scholarly knowledge obtained in
the analysis of software packages. For this, we leverage the Open Research
Knowledge Graph (ORKG) [14]. The ORKG aims to represent scholarly articles in
a machine actionable and structured form. Abstractly speaking, the ORKG
represents research contributions describing key results, the materials and
methods used to obtain the results, and the addressed research problem.

Figure 4: Knowledge graph depicting the scholarly knowledge extracted from a
software package related to an article, describing key aspects (e.g., method
used) of a research contribution of the work described in the article.

The scholarly information extracted from software packages in organized in
triples and ingesting into ORKG using its REST API. Fig. 4 shows the resulting
knowledge graph for a paper and its research
contribution141414https://orkg.org/paper/R209873. The figure also shows the
metadata of corresponding software package151515https://orkg.org/content-
type/Software/R209880.

##  4 Results and Discussion

At the time of writing, there are more than 80,000 software packages available
on Zenodo. To expedite the execution process, we discard packages larger than
400 MB. We thus consider 52,236 software packages. We further process only
those software packages that are also available on GitHub, that is 40,239
packages. We analyze the metadata of the software packages and the respective
README files and find a total of 6221 research articles, of which 642 articles
are associated with the related software packages in metadata through the
cites or isSupplementTo relations. The remaining 5579 articles are extracted
by analyzing the README files of the software packages using SOMEF. We only
analyze software packages that include Python scripts and have linked
scholarly articles, that is 2172 packages. Table 1 summarizes the statistics.

Table 1: Statistics about the (scholarly) information extracted from software packages. Entity | Total
---|---
Software package | 52236
Paper |  Explicit links in metadata: 642; SOMEF-based link extraction: 5579 (Total: 6221)
GitHub URL | 40,239
Python-based software packages, linked with articles | 2172
Analyzed Python scripts | 67,936

Out of 6221 articles, 4328 are described in ORKG because for the remaining
articles the DOIs in README files are not parsed correctly. The articles added
to ORKG include ORKG research contribution descriptions linking the software
package and including information about computational methods and data used in
research extracted by analyzing the software packages.

##### Software semantics and Named Entity Recognition (NER) models

There exist numerous approaches for the extraction of scholarly knowledge from
articles using machine learning and natural language processing, including
scientific named entity recognition [15, 6] and sentence classification [5].
These approaches process the entire text to extract the essential entities in
scholarly articles, which is costly in terms of data collection and training.
Moreover, the approaches require large training data to achieve acceptable
performance. We argue that extracting scholarly knowledge from software
packages as proposed here is a significant step towards automated and cheap
construction of scholarly knowledge graphs. Instead of extracting scholarly
entities from full texts using machine learning models, the scholarly
knowledge is extracted from related software packages with more structured
data.

##### Future directions

In future work, we aim to develop a pipeline that will automatically execute
the software packages that contain scholarly knowledge. Such an approach can
be integrated into software repositories (zenodo, figshare) to automatically
execute the published software and determine whether the (extracted) scholarly
knowledge is reproducible.

##  5 Conclusions

Our work is an important step towards automated and scalable mining of
scholarly knowledge from published software packages and creating the
knowledge graph using the extracted data. The resulting knowledge graph holds
the links between articles and software packages, as well as and most
interestingly descriptions of the computational methods and materials used in
research work presented in articles. Evaluated on zenodo, our approach can be
extended to other repositories, e.g., figshare, as well as software in
languages other than Python, e.g., R, Java, Javascript, and C++—potentially
further increasing the number of articles and related scholarly knowledge
added to ORKG.

## Acknowledgment

This work was co-funded by the European Research Council for the project
ScienceGRAPH (Grant agreement ID: 819536) and TIB–Leibniz Information Centre
for Science and Technology.

## References

  * [1] Abdelaziz, I., Srinivas, K., Dolby, J., McCusker, J.P.: A demonstration of codebreaker: A machine interpretable knowledge graph for code. In: SEMWEB (2020)
  * [2] Abdelaziz, I., Dolby, J., McCusker, J., Srinivas, K.: A toolkit for generating code knowledge graphs. In: Proceedings of the 11th on Knowledge Capture Conference. p. 137–144. K-CAP ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3460210.3493578, https://doi.org/10.1145/3460210.3493578
  * [3] Ahmad, W., Chakraborty, S., Ray, B., Chang, K.W.: A transformer-based approach for source code summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4998–5007. Association for Computational Linguistics (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.449
  * [4] Allamanis, M., Brockschmidt, M., Khademi, M.: Learning to represent programs with graphs. arXiv preprint arXiv:1711.00740 (2017)
  * [5] Brack, A., Hoppe, A., Buschermöhle, P., Ewerth, R.: Cross-domain multi-task learning for sequential sentence classification in research papers. In: Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries. JCDL ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3529372.3530922, https://doi.org/10.1145/3529372.3530922
  * [6] Brack, A., Müller, D.U., Hoppe, A., Ewerth, R.: Coreference resolution in research papers from multiple domains. In: Hiemstra, D., Moens, M.F., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds.) Advances in Information Retrieval. pp. 79–97. Springer International Publishing, Cham (2021)
  * [7] Chao, K., Tao, L., Li, M., Guoyu, J., yuchao, W., Yu, Z.: Construction and application research of knowledge graph in spacecraft launch. Journal of Physics: Conference Series 1754, 012180 (02 2021). https://doi.org/10.1088/1742-6596/1754/1/012180
  * [8] Enders, M., Havemann, F., Ruland, F., Bernard-Verdier, M., Catford, J.A., Gómez-Aparicio, L., Haider, S., Heger, T., Kueffer, C., Kühn, I., et al.: A conceptual map of invasion biology: Integrating hypotheses into a consensus network. Global Ecology and Biogeography 29(6), 978–991 (2020)
  * [9] Ernst, P., Meng, C., Siu, A., Weikum, G.: Knowlife: A knowledge graph for health and life sciences. pp. 1254–1257 (03 2014). https://doi.org/10.1109/ICDE.2014.6816754
  * [10] Heger, T., Pahl, A.T., Botta-Dukát, Z., Gherardi, F., Hoppe, C., Hoste, I., Jax, K., Lindström, L., Boets, P., Haider, S., et al.: Conceptual frameworks and methods for advancing invasion ecology. Ambio 42(5), 527–540 (2013)
  * [11] Husain, H., Wu, H.H., Gazit, T., Allamanis, M., Brockschmidt, M.: Codesearchnet challenge: Evaluating the state of semantic code search (June 2020), https://www.microsoft.com/en-us/research/publication/codesearchnet-challenge-evaluating-the-state-of-semantic-code-search/
  * [12] Iyer, S., Konstas, I., Cheung, A., Zettlemoyer, L.: Summarizing source code using a neural attention model. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2073–2083. Association for Computational Linguistics (Aug 2016). https://doi.org/10.18653/v1/P16$-$1195
  * [13] Jain, N.: Domain-specific knowledge graph construction for semantic analysis. In: Harth, A., Presutti, V., Troncy, R., Acosta, M., Polleres, A., Fernández, J.D., Xavier Parreira, J., Hartig, O., Hose, K., Cochez, M. (eds.) The Semantic Web: ESWC 2020 Satellite Events. pp. 250–260. Springer International Publishing, Cham (2020)
  * [14] Jaradeh, M.Y., Oelen, A., Farfar, K.E., Prinz, M., D’Souza, J., Kismihók, G., Stocker, M., Auer, S.: Open research knowledge graph: Next generation infrastructure for semantic scholarly knowledge. In: Proceedings of the 10th International Conference on Knowledge Capture. p. 243–246. K-CAP ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3360901.3364435
  * [15] Jiang, M., D’Souza, J., Auer, S., Downie, J.S.: Improving scholarly knowledge representation: Evaluating bert-based models for scientific relation classification. In: Ishita, E., Pang, N.L.S., Zhou, L. (eds.) Digital Libraries at Times of Massive Societal Transition. pp. 3–19. Springer International Publishing, Cham (2020)
  * [16] Kelley, A., Garijo, D.: A framework for creating knowledge graphs of scientific software metadata. Quantitative Science Studies 2(4), 1423–1446 (12 2021). https://doi.org/10.1162/qss_a_00167
  * [17] Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International conference on machine learning. pp. 1188–1196. PMLR (2014)
  * [18] Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P., Hellmann, S., Morsey, M., Van Kleef, P., Auer, S., Bizer, C.: Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web Journal 6 (01 2014). https://doi.org/10.3233/SW-140134
  * [19] Mao, A., Garijo, D., Fakhraei, S.: Somef: A framework for capturing scientific software metadata from its documentation. In: 2019 IEEE International Conference on Big Data (Big Data). pp. 3032–3037 (2019). https://doi.org/10.1109/BigData47090.2019.9006447
  * [20] Reza, S.M., Badreddin, O., Rahad, K.: ModelMine: A Tool to Facilitate Mining Models from Open Source Repositories. Association for Computing Machinery, New York, NY, USA (2020), https://doi.org/10.1145/3417990.3422006
  * [21] Spadaro, G., Tiddi, I., Columbus, S., Jin, S., Ten Teije, A., Team, C., Balliet, D.: The cooperation databank: Machine-readable science accelerates research synthesis. Perspectives on Psychological Science p. 17456916211053319 (2020)
  * [22] Spadini, D., Aniche, M., Bacchelli, A.: Pydriller: Python framework for mining software repositories. p. 908–911. ESEC/FSE 2018, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3236024.3264598, https://doi.org/10.1145/3236024.3264598
  * [23] Suchanek, F., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. pp. 697–706 (01 2007). https://doi.org/10.1145/1242572.1242667
  * [24] Vagavolu, D., Swarna, K.C., Chimalakonda, S.: A mocktail of source code representations. In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). pp. 1296–1300 (2021). https://doi.org/10.1109/ASE51524.2021.9678551
  * [25] Vrandečić, D., Krötzsch, M.: Wikidata: A free collaborative knowledgebase. Commun. ACM 57(10), 78–85 (sep 2014). https://doi.org/10.1145/2629489, https://doi.org/10.1145/2629489
  * [26] Zhao, Z., Han, S.K., So, I.M.: Architecture of knowledge graph construction techniques (2018)