#  Synthetic Embedding of Hidden Information in Industrial Control System
Network Protocols for Evaluation of Steganographic Malware
Technical Report

Tom Neubert, Bjarne Peuker, Laura Buxhoidt,
Eric Schueler, Claus Vielhauer

Brandenburg University of Applied Sciences
Department of Computer Science & Media
Brandenburg, Germany
<EMAIL_ADDRESS>
## Abstract

For the last several years, the embedding of hidden information by
steganographic techniques in network communications is increasingly used by
attackers in order to obscure data infiltration, exfiltration or command and
control in IT (information technology) and OT (operational technology)
systems. Especially industrial control systems (ICS) and critical
infrastructures have increased protection requirements. Currently, network
defense mechanisms are unfortunately quite ineffective against novel attacks
based on network steganography. Thus, on the one hand huge amounts of network
data with steganographic embedding is required to train, evaluate and improve
defense mechanisms. On the other hand, the real-time embedding of hidden
information in productive ICS networks is crucial due to safety violations.
Additionally it is time consuming because it needs special laboratory setup.
To address this challenge, this work introduces an embedding concept to
generate synthetic steganographic network data to automatically produce
significant amounts of data for training and evaluation of defense mechanisms.
The concept enables the possibility to manipulate a network packet wherever
required and outperforms the state-of-the-art in terms of embedding pace
significantly.

##  1 Introduction

Nowadays, Industrial Control Systems (ICS) play an essential role in our
economy and in our every day life. In various cyber-physical systems they help
to control and automate processes. The field of supported automation processes
is significant from rather simple traffic light controls to complex networks
of ICS (e.g. in nuclear power plants). Since the well known Stuxnet-Attack
[4], the IT-security community is aware that it is of great importance to
ensure the further development of security mechanisms of ICS against cyber
attackers. In particular with the rise of Ethernet connection and TCP/IP based
protocols in ICS, the boundaries between OT (operational technology) and IT
(information technology) are blurred. Due to these blurred boundaries, similar
threats and attacking trends in the IT-domain can be observed in the OT-domain
(e.g. Stuxnet-Attack [4], Ukrainian Power Grid Attack [7] and attacks on
safety systems [3]).
Thanks to the continuous research and development of defense mechanisms in
ICS, there is a rather reliable and precise functionality against conventional
attack vectors. Thus, it is a current trend among attackers to use
steganographic techniques to hide information in network communication [9] to
conceal relevant attack procedures (especially) in advanced persistent threats
(APTs), like data in- and exfiltration, command & control communication or
lateral movement for as long as possible. Due to the mentioned blurred
boundaries between OT and IT, attackers could use hidden channel network
communication in ICS as shown in [11, 1, 6]. For these purposes, current
defense mechanisms are unfortunately inadequate to counter such novel attack
vectors with sophisticated hidden channel network communication and urgently
need further development.
For the elaboration and evaluation of novel defense mechanisms, diverse and
heterogeneous data for training and test purposes is essential. However, the
generation of network data with hidden information based on steganographic
techniques in ICS is challenging, because each steganographic embedding needs
a complex, sophisticated and very specific experimental setup and
implementation which is very time consuming, plus raises various security and
safety issues.
Due to this, in [10] an approach is presented which enables the possibility to
generate network data with steganographic embedding artificially, based on
network data recordings. The approach from [10] (see Section 2.2) is limited
to a very specific steganographic embedding technique based on TCP-timestamps
and additionally it has a proportionate low embedding pace which limits the
amount of synthetic data, that can be produced in a given amount of time.
Thus, this work introduces an advanced concept for the synthetic embedding of
hidden information in network data, which offers the possibility to embed
hidden information literally everywhere in (uncompromized) network packet
recordings with a significantly faster embedding pace than the state-of-the-
art approach from [10]. From now on, we refer to it as Synthetic
Steganographic Embedding (SSE) concept. The SSE-concept has two synthetic
embedding options, one option focuses on a high embedding pace and the other
option on a more comfortable and easier to handle embedding.

##  2 State-of-the-Art

In this Section, we introduce the term network steganography in the context of
this technical report (Section 2.1) and present related work regarding the
generation of synthetic network data in Section 2.2.

###  2.1 Network Steganography in ICS

According to [12] with a focus on network steganography the term steganography
can be described as follows: “Steganography is the art and science of
concealing the existence of information transfer and storage”. Thus, network
steganography targets the embedding of hidden data in network communication
i.e. traffic. From an attacker’s perspective, a warden (e.g. intrusion
detection system) observes the network traffic and the embedding should be
inconspicuous in a sense that he would not be able to differ between genuine
and steganographic communication [2]. This can be realized for example by
manipulating the packets payload on least significant bits, by using unused
space in headers or by artificially produced timing delays to modulate time
intervals between specific packets [8].
Network steganography in the ICS network communication domain is specific due
to the lower amount of available data for potential embedding, because
transmitted network packets are usually smaller since only meta-data or few
(sensor) values are transmitted to keep the communication lean and simple.
Additionally, ICS specific protocols like OPC-
UA111https://opcfoundation.org/about/opc-technologies/opc-ua/ or Modbus-
TCP222https://www.modbus.org/docs/Modbus_Messaging_Implementation_Guide_V1_0b.pdf
are very commonly encapsulated in actual transport protocols such as TCP/IP,
which enables the opportunity for utilizing the data fields of the ICS
specific protocols in addition to TCP/IP protocol headers. It is also not
uncommon for the ICS-specific payload to be transmitted unencrypted.
Relevant and exemplary attack vectors with steganographic embedding techniques
in ICS are presented in [2, 11, 5, 1].
Potential network steganographic embedding patterns and a related terminology
is summarized in [13]. A generic taxonomy with the intention of a unified
understanding of terms and their applicability for steganographic methods can
be found in detail in [12].

###  2.2 Synthetic Steganographic Network Data Generation

In 2021 an initial artificial steganographic network data (ASND) generation
concept was introduced in [10] due to the lack of available training and test
data to elaborate and evaluate defense mechanisms in ICS against
steganographic attack vectors. To counter this research gap, the ASND
generation concept promises an easy and fast generation of artificial network
data, which simulates a steganographic embedding in TCP-timestamps. It is
stated that most important factors to be simulated would be:

  1. 1.

the physical network including layout and components,

  2. 2.

the network traffic including types of flows, directions, protocols used,
typical payloads, etc. and

  3. 3.

the type and characteristics of the steganographic hidden channel.

In [10] and in this work only the last aspect (3) of this list is simulated,
the rest is taken directly from a uncompromized recording of a physical
laboratory setup, which ensures a plausible environment. For the network data
capturing the work uses Wireshark333https://www.wireshark.org. For the
steganographic embedding and the manipulation of the network packet captures,
a processing pipeline with tools from the Wireshark family (editcap, mergecap
tshark) is used.
However, the original ASND generation concept is limited to a very specific
embedding only for TCP-timestamps and has quite low embedding pace with 0.86
seconds per packet on commodity hardware. This paper aims to overcome these
shortcomings with a concept that enables the opportunity to embed hidden
information literally everywhere in a network packet (plausible or not) with
significantly higher embedding pace (instant manipulation of thousands of
packets).

##  3 SYNTHETIC STEGANOGRAPHIC EMBEDDING (SSE)-CONCEPT

Our novel Synthetic Steganographic Embedding (SSE)-Concept for network data is
based on the idea to enable the opportunity to create appropriate synthetic
steganographic network data in an straightforward, comprehensible, fast and
easy way for the training and evaluation of defense mechanisms against modern
attack vectors which use steganographic techniques to hide information within
the network traffic to stay undetected (comparable to [10]). The novel and
advanced SSE-Concept shall offer a significantly faster and unrestricted
embedding in network data than the state-of-the-art.
Basically, the concept does not synthesize the complete network traffic, it
takes recorded network traffic from an uncompromized ICS (laboratory) setup
and selects packets and parts of a packet’s payload to synthesize only a
chosen and specific part of the payload which is well suited for a
steganographic embedding of hidden information (e.g. sensor values or
timestamps). The rest of the network traffic stays untouched with complete
integrity, this ensures a plausible environment so that potential defense
mechanisms can focus on the impact of the used steganographic embedding (as
mentioned in Section 2.2). The SSE-concept is visualized in Figure 1 and has
two synthetic embedding options (SEOA and SEOB). SEOA focuses on a very fast
and efficient embedding without accessing structural elements of a packet, the
processing is based on parsing through the hexdump of packets. The other
embedding option SEOB delivers a much more comfortable embedding with an
easier access to structural elements of a network packet based on json-
objects, but the process is more complex and thus more time consuming.
The concept and its four segments:

  * •

Segment I: Record and Pre-Processing

  * •

Segment II: SEOA

  * •

Segment III: SEOB and

  * •

Segment IV: Retrieval

are described in detail in the following subsections.

Figure 1: Novel Synthetic Steganographic Embedding Concept for Network Data in
ICS

###  3.1 Segment I: Record and Pre-Process Network Data

The concept starts with Segment I and its first segment element $SE_{I.1}$
where initially network data is captured from an uncompromized laboratory
ground truth (ICS-) setup. The network data can be recorded with different
tools, we use Wireshark for the recording. The output should be delivered in
pcap or pcapng format444https://www.tcpdump.org/ for further processing. These
file formats are common and widespread logging protocols which are well suited
for the structural recording of network data over a specific time period.
In the following step $SE_{I.2}$ a pre-processing is performed in order to
identify relevant communication between selected partners of a specific
communication where a steganographic embedding shall be performed. For
$SE_{I.2}$, we use Wireshark filters. After the recording and pre-processing
of the network data the resulting pcap\- or pcapng-file with the packet
capturing can now be used for synthetic embedding and forms our steganographic
cover. Based on the created cover, the embedding can be performed with the
synthetic embedding options SEOA (see Section 3.2) or SEOB (see Section 3.3).

###  3.2 Segment II: Synthetic Embedding Option (SEO) A

Segment II contains SEOA (Synthetic Embedding Option) where the network
traffic can be artificially manipulated based on txt-hexdump files. The
segment has four elements ($SE_{II.1}$ \- $SE_{II.4}$). The first element
$SE_{II.1}$ converts the captured network recording to hexdump and saves the
hexdump to txt-file format. For the conversion, the Wireshark console
application tshark can be used with the following command:

    tshark --hexdump frames -P -t ad -o
    gui.column.format:
    "Time","%t" -r filepath.pcap > output.txt

The command has the following meaning: the term `--hexdump` `frames` means
that only all packets bytes are converted to hexdump without secondary data
sources like decrypted data to reduce the errors when reconverting the
manipulated hexdump back into pcap-format.
The term `-P -t ad -o gui.column.format:` `"Time","%t"` defines the timestamp
format (if not set, packets will have incorrect timestamps) and `-r` is the
read option to import a source file. After the conversion into txt-data
format, relevant packets have to be selected ($SE_{II.2}$) for embedding
($SE_{II.3}$) and therefore, a script for automation has to be elaborated.
In detail, by analyzing the previously recorded and pre-processed network
traffic, suitable packets for a potential embedding have been selected
($SE_{II.2}$) now. Suitable packets contain payload like sensor values,
timestamps or other numeric values with at best many rather irrelevant digits
which make an embedding as unobtrusive as possible. The packet selection
($SE_{II.2}$) of suitable packets for potential embedding can be done by
identifying unique features (binary signatures) of these packets and finding
the position of these packets with the resulting hexdump signature in the
pcap-file. Wireshark offers the possibility to interpret structural packet
elements of the encapsulated ICS protocol (like OPC-UA, Modbus-TCP) to
interpret the pcap(ng)-files. We use this functionality to look up related
hexdump signature sequences in order to identify and select suitable packets
for potential embedding (`embedding_` `packet_feature`).
In the same manner, we identify the actual position of the structural element
for embedding ($SE_{II.3}$) of the payload (`embedding_value`) within the
previously selected packets.
The steps $SE_{II.2}$ and $SE_{II.3}$ are performed in a loop with the
elaborated automation script which selects the packets and manipulates the
hexdump txt-file of the packets until the embedding of the hidden message is
complete. This results in a manipulated txt-(hexdump) file. In very simplified
pseudo code structure these steps could be described as follows:

    for hexdump_line in txt_file:
        if hexdump_line.contains(embedding_packet_feature):
            hexdump_line.replace(value, embedding_value)

After the completed embedding, the resulting txt-file has to be reconverted
($SE_{II.4}$) into a plausible, unconspicous and synthetic network data file.
This can be done with the Wireshark console application text2pcap with the
following command:

    text2pcap -F pcap -N Eth1 -t "%F %T,%f"  in.txt  out.pcap

where `-F pcap` sets the output file type and `-N Eth1` defines the synthetic
interface name of the .pcap(ng)-file. `-t "%F %T,%f"` defines the date/time
coding, it has to be consistent with the coding from segment element
$SE_{II.1}$. The result of Segment II (SEOA) is synthetic network data with
hidden information embedded by steganographic techniques.

###  3.3 Section III: Synthetic Embedding Option (SEO) B

Segment III includes SEOB which offers the possibility to manipulate network
packets based on json-data. SEOB is comparable to the embedding process of
[10] but instead of the Wireshark tool editcap it uses json file format to
manipulate network packets, because editcap is limited to TCP-timestamps (as
mentioned). Here, steps from $SE_{III.1}$ to $SE_{III.4}$ are carried out in a
loop (in contrast to SEOA) until the embedding of the hidden message is
complete. Therefore, a script for automation for these steps has to be
elaborated (we use Python). Before the start of SEOB, a list with identifiers
(`packet_ID`) of suitable packets has to be generated. Suitable packets here
have the same characteristics as in SEOA and can be chosen with tshark
filters. The script needs access to this created list. SEOB starts with
selecting a single suitable packet ($SE_{III.1}$) for embedding by isolating
it with:

    editcap -r input.pcap isolated_packet.pcap
                packet_ID

where the term `-r` is the read option followed by the output path and
`packet_ID` is the index number of a selected suitable packet for embedding.
Now the isolated packet has to be removed from the initial input file to
replace it later with the synthetic packet with:

    editcap -V input.pcap input.pcap packet_ID

After the selection and the removal of the isolated packet from the input, the
single packet is converted into json file format ($SE_{III.2}$) with:

    tshark -r isolated_packet.pcap -T jsonraw >> packet.json

because converting the complete network traffic into json-data creates a file
that is difficult to process due to a potentially huge file size. Now, a
single network packet is represented as ajson-object and thus it can be
processed and manipulated with self-written source code anywhere. The term
`-r` defines the read option, the term -T defines the format of the text
output.
As in SEOA, we recommend to select suitable packet payload for embedding with
for example numeric values with many heterogeneous digits which make an
embedding as unobtrusive as possible ($SE_{III.3}$). After embedding the
information into a single packet json-data, the json-packet with synthetic
embedding is now reconverted with the external Wireshark console application
json2pcap and merged back into the input with mergecap, therefore the
following commands can be used:

    json2pcap -i packet.json -o stego_packet.pcap
    mergecap -w output.pcap input.pcap stego_packet.pcap

where `json2pcap` reconverts the manipulated packet back into pcap network
data and `mergecap` merges it back into the initial input network data cover
($SE_{III.4}$).

###  3.4 Segment IV: Retrieval

In the last Segment IV, the embedded message has to be retrieved from the
synthetic network data in order to find out if the embedding was successful
($SE_{IV.1}$). To do so, a retrieval script has to be elaborated with access
to the hidden message based on the secret information (from the embedding
process like steganographic embedding key) that are necessary for retrieval.
Additionally, the generated synthetic network data with steganographic
embedding has to be structurally and syntactically analyzed with Wireshark to
detect and mark obvious structural error in the network data. If the analysis
results in no error detections, the synthetic steganographic embedding is
successful and completed.
In the next Sections (4 & 5) the SSE-Concept is validated by an exemplary
embedding and an analysis.

##  4 Conclusion and Future Work

In this technical report we present a novel synthetic steganograhic embedding
(SSE) concept with two synthetic embedding options (SEOA is fast, SEOB is more
flexible in terms of accessing dedicated elements of the encapsulated
protocol, both options are equally precise) to generate substantial amounts of
steganographic network data in ICS to train and evaluate defense mechanisms in
the future work. The novel SSE-concept clearly outperforms the state-of-the-
art approach from [10] in terms of embedding pace and delivers the novel
opportunity to manipulate a network packet anywhere. The SSE-concept forms the
foundation for our future work, to analyze steganographic data in ICS networks
to elaborate suitable defense mechanisms.

## Acknowledgements

The presented work is funded by the German Federal Ministry for the
Environment, Nature Conservation, Nuclear Safety and Consumer Protection
(BMUV, project no. 1501666B) in the framework of the German reactor safety
research program.

## References

  * [1] M. Hildebrandt, R. Altschaffel, K. Lamshoeft, M. Lange, M. Szemkus, T. Neubert, C. Vielhauer, Y. Ding, and J. Dittmann.  Threat analysis of steganographic and covert communication in nuclear i&c systems.  In Proceedings of IAEA ICONS 2020: International Conference on Nuclear Security: Sustaining and Strengthening Efforts, 10-14 February 2020, Vienna, Austria, https://event.do/iaea/a/#/events/3301/f/29007, 2020.
  * [2] M. Hildebrandt, K. Lamshoeft, J. Dittmann, T. Neubert, and C. Vielhauer.  Information hiding in industrial control systems: An opc ua based supply chain attack and its detection.  pages 115–120, 2020.
  * [3] Dragos Inc.  Trisis malware - analysis of safety system targeted malware.  https://dragos.com/wp-content/uploads/TRISIS-01.pdf, 2018\.
  * [4] D. Kushner.  The Real Story of Stuxnet.  IEEE Spectrum; DOI: 10.1109/MSPEC.2013.6471059, 2013.
  * [5] K. Lamshoeft, C. Kraetzer, J. Dittmann, T. Neubert, and C. Vielhauer.  Information hiding in cyber physical systems: Challenges for embedding, retrieval and detection using sensor data of the swat dataset.  In Proceedings of the 2021 ACM Workshop on Information Hiding and Multimedia Security (IHMMSec ’21), June 22–25, 2021, Virtual Event, Belgium. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3437880.3460413, 2021.
  * [6] K. Lamshoeft, T. Neubert, M. Lange, R. Altschaffel, M. Hildebrandt, Y. Ding, C. Vielhauer, and J. Dittmann.  Novel challenges for anomaly detection in i&c networks: Strategic preparation for the advent of information hiding based attacks.  Atw. Atomwirtschaft, 65:504–508, 10 2020.
  * [7] R. M. Lee, M. J. Assante, and T. Conway.  Analysis of the cyber attack on the ukrainian power grid.  Technical report, SANS Institute, 2016.
  * [8] W. Mazurczyk, S. Wendzel, and K. Cabaj.  Towards Deriving Insights into Data Hiding Methods Using Pattern-based Approach. ARES 2018, 13th International Conference on Availability, Reliability and Security; Hamburg, Germany, August 27 - August 30, ISBN: 978-1-4503-6448-5, 2018.
  * [9] MITRE and ATTACK.  Data obfuscation: Steganography.  https://attack.mitre.org/versions/v14/techniques/T1001/002/, 2020.
  * [10] T. Neubert, C. Kraetzer, and C. Vielhauer.  Artificial steganographic network data generation concept and evaluation of de- tection approaches to secure industrial control systems against stegano- graphic attacks.  In The 16th International Conference on Availability, Relia- bility and Security (ARES 2021), August 17–20, 2021, Vienna, Austria. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3465481.3470073, 2021.
  * [11] T. Neubert and C. Vielhauer.  Kill chain attack modeling for hidden channel attack scenarios in industrial control systems.  21st IFAC World Congress, Berlin, Germany, July 11-17, Submission 1475, 2020.
  * [12] S. Wendzel, L. Caviglione, W. Mazurczy, A. Mileva, J. Dittmann, C. Krätzer, K. Lamshöft, C. Vielhauer, L. Hartmann, J. Keller, T. Neubert, and S. Zillien.  A generic taxonomy for steganography methods.  July 2022.
  * [13] S. Wendzel, L. Caviglione, W. Mazurczyk, A. Mileva, J. Dittmann, C. Krätzer, K. Lamshöft, C. Vielhauer, L. Hartmann, J. Keller, and T. Neubert.  A revised taxonomy of steganography embedding patterns.  The 16th International Conference on Availability, Reliability and Security (ARES 2021), August 17–20, 2021, Vienna, Austria. ACM, New York, NY, USA, 12 pages. https://doi.org/10. 1145/3465481.3470069, 2021.


# Estimating the false discovery risk of (randomized) clinical trials in
medical journals based on published $p$-values

Ulrich Schimmack1 and František Bartoš2,3
1 Department of Psychology, University of Toronto Mississauga, Mississauga,
Canada
2 Department of Psychological Methods, University of Amsterdam, Amsterdam, the
Netherlands
3 Institute of Computer Science, Czech Academy of Sciences, Prague, Czech
Republic

Correspondence concerning this article should be addressed to Ulrich Schimmack
at<EMAIL_ADDRESS>

## Abstract

The influential claim that most published results are false raised concerns
about the trustworthiness and integrity of science. Since then, there have
been numerous attempts to examine the rate of false-positive results that have
failed to settle this question empirically. Here we propose a new way to
estimate the false positive risk and apply the method to the results of
(randomized) clinical trials in top medical journals. Contrary to claims that
most published results are false, we find that the traditional significance
criterion of $\alpha=.05$ produces a false positive risk of 13%. Adjusting
$\alpha$ to .01 lowers the false positive risk to less than 5%. However, our
method does provide clear evidence of publication bias that leads to inflated
effect size estimates. These results provide a solid empirical foundation for
evaluations of the trustworthiness of medical research.

## Introduction

Many sciences are facing a crisis of confidence in published results [1].
Meta-scientific studies have revealed low replication rates, estimates of low
statistical power, and even reports of scientific misconduct [2].

Based on assumptions about the percentage of true hypotheses and statistical
power to test them, Ioannidis [3] arrived at the conclusion that most
published results are false. It has proven difficult to test this prediction.
First, large scale replication attempts [4, 5, 6] are inherently expensive and
focus only on a limited set of pre-selected findings [7]. Second, studies of
meta-analyses have revealed that power is low, but rarely lead to the
conclusion that the null-hypothesis is true [8, 9, 10, 11, 12, 13, 14, 15, 16]
(but see [17, 18]).

So far, the most promising attempt to estimate the false discovery rate has
been Jager and Leek’s [19] investigation of $p$-values in medical journals.
They extracted 5,322 $p$-values from abstracts of medical journals and found
that only 14% of the statistically significant results may be false-positives.
This is a sizeable percentage, but it is inconsistent with the claim that most
published results are false. Although Jager and Leek’s article was based on
actual data, the article had a relatively minor impact on discussions about
false-positive risks, possibly due to several limitations of their study [20,
21, 22, 23].

One problem of their estimation method is the problem to distinguish between
true null-hypotheses (i.e., the effect size is exactly zero) and studies with
very low power in which the effect size may be very small, but not zero. To
avoid this problem, we do not estimate the actual percentage of false
positives, but rather the maximum percentage that is consistent with the data.
We call this estimate the false discovery risk (FDR). To estimate the FDR, we
take advantage of Soric’s [24] insight that the false discovery risk is
maximized when power to detect true effects is 100%. In this scenario, the
false discovery rate is a simple function of the discovery rate (i.e., the
percentage of significant results). Thus, the main challenge for empirical
studies of FDR is to estimate the discovery rate when selection bias is
present and inflates the observed discovery rate. To address the problem of
selection bias, we developed a selection model that can provide an estimate of
the discovery rate before selection for significance. The method section
provides a detailed account of our method and compares it to Jager and Leek’s
[19] approach.

## Methods

To estimate the false discovery rate, Jager and Leek [19] developed a model
with two populations of studies. One population includes studies in which the
null-hypothesis is true ($\mathcal{H}_{0}$). The other population includes
studies in which the null-hypothesis is false; that is, the alternative
hypothesis is true ($\mathcal{H}_{1}$). The model assumes that the observed
distribution of significant $p$-values is a mixture of these two populations,
modelled as a mixture of truncated beta distributions of the $p$-values. The
first problem for this model is that it can be difficult to distinguish
between studies in which $\mathcal{H}_{0}$ is true and studies in which
$\mathcal{H}_{1}$ is true, but they had low statistical power. The second
problem is that published studies are heterogeneous in power which might not
be properly captured when modelling distribution of $p$-values under
$\mathcal{H}_{1}$ with a truncated beta distribution. To avoid the first
problem, we refrain from estimating the actual false discovery rate, which
requires a sharp distinction between true and false null-hypotheses (i.e., a
distinction between zero and non-zero effect sizes). Instead, we aim to
estimate the maximum false discovery rate that is consistent with the data. We
refer to this quantity as the false discovery risk. To avoid the second
problem, we use z-curve which models the distribution of observed
$z$-statistics as a mixture of truncated folded normal distributions. This
mixture model allows for heterogeneity in power of studies when
$\mathcal{H}_{1}$ is true [25, 26].

### False discovery risk

To estimate the false discovery risk, we take advantage of Soric’s [24]
insight that the maximum false discovery rate is limited by statistical power
to detect true effects. When power is 100%, all non-significant results are
produced by testing false hypotheses ($\mathcal{H}_{0}$). As this scenario
maximizes the number of non-significant $\mathcal{H}_{0}$, it also maximizes
the number of significant $\mathcal{H}_{0}$ tests and the false discovery
rate. For example, if 100 studies produce 30 significant results, the
discovery rate is 30%. And when the discovery rate is 30%, the maximum false
discovery risk with $\alpha$ = 0.05 is $\approx$ 0.12. In general, the false
discovery risk is a simple transformation of the discovery rate, such as

$\text{false discovery risk}=\frac{1-\text{discovery rate}}{\text{discovery
rate}}\times\frac{\alpha}{1-\alpha}.$

If all conducted hypothesis tests were reported, the false discovery risk
could be determined simply by computing the percentage of significant results.
However, it is well-known that journals are more likely to publish
statistically significant results than non-significant results. This selection
bias renders the observed discovery rate in journals uninformative [26, 25].
Thus, a major challenge for any empirical estimates of the false discovery
risk is to take selection bias into account.

### Z-curve 2.0

Brunner and Schimmack [25] developed a mixture model to estimate the average
power of studies after selection for significance. Bartoš and Schimmack [26]
recently published an extension of this method that can estimate the expected
discovery rate before selection for significance on the basis of the weights
derived from fitting the model to only statistically significant $p$-values.
The selection process assumes that the probability of a study to be published
is proportional to its power [25]. Extensive simulation studies have
demonstrated that z-curve produces good large-sample estimates of the expected
discovery rate [26]. Moreover, these simulation studies showed that z-curve
produces robust confidence intervals with good coverage. As the false
discovery risk is a simple transformation of the expected discovery rate,
these confidence intervals also provide confidence intervals for estimates of
the false discovery risk.

Z-curve obtains the expected discovery rate estimate from estimating the mean
power of studies before selection of significance. Z-curve leverages
distributional properties of $z$-statistics—studies homogeneous in power
produce normally distributed $z$-statistics centered at a $z$-transformation
of the true power (e.g., Equations 3 in Bartoš and Schimmack [26]). Z-curve
incorporates the selection on statistical significance via a truncation at the
corresponding $\alpha$ level, similarly to the Jager and Leek’s approach.
However, in contrast to other approaches, z-curve acknowledges that studies
are naturally heterogeneous in power (i.e., studies are investigating
different effects with different sample sizes) and estimates the mean power
via a mixture model,

$f(z;\theta)=\sum_{j=1}^{J}\pi_{j}f_{j[a,b]}(z;\theta_{j}),$

where $z$ correspond to statistically significant $z$-statistics modelled via
$J$ mixtures of truncated folded normal distributions, $f(z;\theta)$, where
$\theta$ denotes parameters of the truncated folded normal
distributions.111The fold is introduced by our lack of knowledge of the
expected direction of the effect. See Figure 1 in in Bartoš and Schimmack [26]
for visual intuition.

We extended the z-curve model to incorporate rounded $p$-values and $p$-values
reported as inequalities (i.e., $p<0.001$). We did so in the same way as the
Jager and Leek’s approach; we implementing interval censored (for rounded
$p$-values) and right censored (for $p$-values reported as inequalies)
likelihoods versions of the z-curve model,

$f^{[l,u]}(z;\theta)=\sum_{j=1}^{J}\pi_{j}f^{[l,u]}_{j[a,b]}(z;\theta_{j}),$

where $l$ and $u$ correspond to the lower and upper censoring points on a
mixture of truncated folded normal distributions. For example, a $p$-value
reported as inequality, $p<0.001$, has only a lower censoring point at
$z$-score corresponding to $p=0.001$, and rounded $p$-value, $p=0.02$, has a
lower censoring point at a $z$-score corresponding to $p=0.025$ and upper
censoring point at a $z$-score corresponding to $p=0.015$.

## Simulation study

We performed a simulation study that extends the simulations performed by
Jager and Leek [19] in several ways. We did not simulate $\mathcal{H}_{1}$
$p$-values directly (i.e., we did not use a right skewed beta-distribution as
a Jagger and Leek). Instead, we simulated $\mathcal{H}_{1}$ $p$-values from
two-sided $z$-tests. We drew power of each simulated $z$-test (the only
required parameter) from a distribution based on Lamberink and colleagues’
[16] empirical estimate of medical clinical trials’ power (excluding all power
estimates based on meta-analyses with non-significant results). This allowed
us to assess the performance of the methods under heterogeneity of power to
detect $\mathcal{H}_{1}$ corresponding to the actual medical literature. To
simulate $\mathcal{H}_{0}$ $p$-values, we used a uniform distribution.

We manipulated the true false discovery rate from 0 to 1 with a step size of
0.01 and simulated 10,000 observed significant $p$-values by changing the
proportion of $p$-values simulated from $\mathcal{H}_{0}$ and
$\mathcal{H}_{1}$. Similarly to Jager and Leek [19], we performed four
simulation scenarios with an increasing percentage of imprecisely reported
$p$-values. Scenario A used exact $p$-values, scenario B rounded $p$-values to
three decimal places (with $p$-values lower than 0.001 censored at 0.001),
scenario C rounds 20% $p$-values to two decimal places (with $p$-values
rounded to 0 censored at 0.01), and scenario D first rounds 20% $p$-values to
two decimal places and further censors 20% $p$-values at on of the closest
ceilings (0.05, 0.01, or 0.001).

Figure 1: Comparison of z-curve and Jager and Leek’s false discovery risk
estimates using a simulation study. Estimated false discovery rate (FDR,
y-axis) for z-curve (black-filled squares) and Jager and Leek’s swfdr (empty
circles) vs. the true false discovery rate (FDR, x-axis) across four
simulation scenarios (panels). Scenario A uses exact $p$-values, scenario B
rounds $p$-values to three decimal places, scenario C rounds 20% $p$-values to
two decimal places, and scenario D rounds 20% $p$-values to two decimal places
and censors 20% $p$-values at on of the closest significance levels (0.05,
0.01, or 0.001).

Fig. 1 displays the true ($x$-axis) vs. estimated ($y$-axis) false discovery
rate for Jager and Leek’s [19] swfdr method and the false discovery risk for
z-curve across the different scenarios (panels). When precise $p$-values are
reported (panel A in the upper left corner), z-curve handles the heterogeneity
in power well across the whole range of false discovery rate and produces
accurate estimates of false discovery risk. Higher estimate than the actual
false discovery rates are expected because the false discovery risk is an
estimate of the maximum false discovery rate. Discrepancies are especially
expected when power of true hypothesis tests is low. For the simulated
scenarios, the discrepancies are less than 20 percentage points and decrease
as the true false discovery rate increases. Even though Jager and Leek’s [19]
method aims to estimate the true false discovery rates, it produces higher
estimates than z-curve. This is problematic because the method produces
inflated estimates of the true false discovery rate. Even if the estimates
were interpreted as maximum estimates, the method is less sensitive to the
actual variation in the false discovery rate than the z-curve method.

Panel B shows that the z-curve method produces similar results when $p$-values
are rounded to three decimals. The Jager and Leek’s [19] method however
experiences estimation issues, especially in the lower spectrum of the true
false discovery rate since the current swfdr implementation only allows to
deal with rounding to two decimal places (we also tried specifying the
$p$-values as a rounded input; however, the optimizing routine failed with
several errors).

Panel C shows a surprisingly similar performance of the two methods when 20%
of $p$-values are rounded to two decimals, except for very high levels of true
false discovery rates, where Jager and Leek’s [19] method starts to
underestimate the false discovery rate. Despite the similar performance, the
results have to be interpreted as estimates of the false discovery risk
(maximum false discovery rate) because both methods overestimate the true
false discovery rate for low false discovery rates.

Panel D shows that both methods have problems when 20% of $p$-values are at
the closest ceiling of .05, .01, or .001 without providing clear information
about the exact $p$-value. Underestimation of true false discovery rates over
40% is not as serious problem because any actual false discovery rate over 40%
is unacceptably high. One potential solution to the underestimation in this
scenario might by exclusion of the censored $p$-values from the analyses,
assuming that censoring is independent the size of the $p$-value.

Table 1: Root mean square error and bias of z-curve and Jager and Leek’s false
discovery risk estimates in a simulation study.

Method (RMSE) | A | B | C | D
---|---|---|---|---
z-curve | 0.07 (0.00) | 0.12 (0.01) | 0.10 (0.01) | 0.11 (0.01)
swfdr | 0.21 (0.01) | 0.21 (0.01) | 0.13 (0.01) | 0.17 (0.01)
Method (bias) | A | B | C | D
z-curve | 0.04 (0.00) | 0.09 (0.01) | 0.07 (0.01) | -0.05 (0.01)
swfdr | 0.17 (0.01) | 0.02 (0.02) | 0.04 (0.01) | -0.03 (0.02)

Root mean square error (RMSE) and bias + standard errors (in brackets) for the
estimated false discovery rate (FDR) by z-curve and Jager and Leek’s swfdr
[19] across four simulation scenarios (columns). Scenario A uses exact
$p$-values, scenario B rounds $p$-values to three decimal places, scenario C
rounds 20% $p$-values to two decimal places, and scenario D rounds 20%
$p$-values to two decimal places and censors 20% $p$-values at on of the
closest significance levels (0.05, 0.01, or 0.001).

Root mean square error and bias of the false discovery rate estimates for each
scenario summarized in Table 1 show that z-curve produces estimates with
considerably lower root mean square error. The results for bias show that both
methods tend to produce higher estimates than the true false discovery rate.
For z-curve this is expected because it aims to estimate the maximum false
discovery rate. It would only be a problem if estimates of the false discovery
risk were lower than the actual false discovery rate. This is only the case in
Scenario D, but as shown previously, underestimation only occurs when the true
false discovery rate is high.

To summarize, our simulation confirms that Jager and Leek’s [19] method
provides meaningful estimates of the false discovery risk and that the method
is more likely to overestimate rather than underestimate the true false
discovery rate. Our results also show that z-curve improves over the original
method and that the modifications can handle rounding and imprecise reporting
when the false discovery rates are below 40%. Only if estimates exceed 40%, it
is possible that most published results are false positive results.

## Application to medical journals

We developed an improved abstract scraping algorithm to extract $p$-values
from abstracts in major medical journals accessible through
https://pubmed.ncbi.nlm.nih.gov/. Our improved algoritm addressed multiple
concerns raised in commentaries to the Jager and Leek’s [19] original article.
Specifically, (1) we extracted $p$-values only from abstracts labeled as
“randomized controlled trial” or “clinical trial” as suggested by [20, 23,
21], (2) we improved the regex script for extracting $p$-values to cover more
possible notations as suggested by [23], (3) we extracted confidence intervals
from abstracts not reporting $p$-values as suggested by [23, 22]. We further
scraped $p$-values from abstracts in “PLoS Medicine” to compare the false
discovery rate estimates to a less-selective journal as suggested by [20].
Finally, we randomly subset the scraped $p$-values to include only a single
$p$-value per abstract in all analyses, thus breaking the correlation between
the estimates as suggested by [20]. Although there are additional limitations
inherent to the chosen approach, these improvements, along with our improved
estimation method, make it possible to test the prediction by several
commentators that the false discovery rate is well above 14%.

We executed the scraping protocol on July 2021 and scraped abstracts published
since 2000 (see Table 2 for a summary of the scraped data). Interactive
visualization of the individual abstracts and scraped values can be accessed
at https://tinyurl.com/zcurve-FDR.

Table 2: Summary of the scraped data

Journal | Abstracts | CT or RCT | Scrapeable |  $p$-values
---|---|---|---|---
Lancet | 7638 | 2099 | 1556 | 5403
BMJ | 4480 | 939 | 643 | 1625
NEJM | 4912 | 2767 | 2304 | 7137
JAMA | 5491 | 1452 | 1274 | 4484
PLoS | 2944 | 346 | 279 | 1102
Total | 25465 | 7603 | 6056 | 19751

The “Abstracts” column contains the total number of accessed abstracts, the
“CT or RCT” column contains the number of abstracts labels as either “clinical
trials” or “randomized controlled trials”, the “Scrapeable” column contains
the number of further abstracts that contained at least one automatically
scrapeable $p$-value or a confidence interval, and the “$p$-values” column
contains the total number of $p$-values extracted. PLoS stands for the PLoS
Medicine and NEJM stands for the New England journal of medicine.

Fig. 2 visualizes z-curve and Jager and Leek [19] method’s false discovery
risk estimates based on scraped abstracts from clinical trials and randomized
controlled trials and further divided by journal and whether the article was
published before (and including) 2010 (left) or after 2010 (right). We see
that, in line with the simulation results, Jager and Leek’s [19] method
produces slightly higher false discovery rate estimates. Furthermore, z-curve
produced considerably wider bootstrapped confidence intervals, suggesting that
the confidence interval reported by Jager and Leek [19] ($\pm$ 1 percentage
point) was too narrow.

Figure 2: Estimates of false discovery risk across decades and journals.
Estimated false discovery rate (FDR, y-axis) with z-curve (black-filled
squares) and Jager and Leek’s [19] swfdr (empty circles) based on clinical
trials and randomized control trials divided by journal (x-axis, with
information about unique articles in brackets) and whether the article was
published before (and including) 2010 (left) or after 2010 (right).

A comparison of the false discovery estimates based on data before (and
including) 2010 and after 2010 shows that confidence intervals overlap,
suggesting that false discovery rates have not changed. Separate analyses
based on clinical trials and randomized controlled trials also showed no
significant differences (see Fig. 3). Therefore, to reduce the uncertainty
about the false discovery rate, we estimate the false discovery rate for each
journal irrespective of publication year. The resulting false discovery rate
estimates based on z-curve and Jager and Leek’s [19] method are summarized in
Table 3. We find that all false discovery rate estimates fall within a .05 to
.30 interval. Finally, further aggregating data across the journals provides a
false discovery rate estimate of 0.13, 95% [0.08, 0.21] based on z-curve and
0.19, 95% [0.17, 0.20] based on Jager and Leek’s [19] method. This finding
suggests that Jager and Leek’s [19] extraction method slightly underestimate
the false discovery rate, whereas their model overestimated the false
discovery rate. Finally, our improved false discovery risk estimate based on
z-curve closely matches the original false discovery rate estimate.

Table 3: Combined and journal specific false discovery risk estimates.

Journal | z-curve | swfdr
---|---|---
Lancet | 0.07 [0.05, 0.11] | 0.21 [0.18, 0.24]
BMJ | 0.23 [0.08, 0.54] | 0.28 [0.23, 0.32]
NEJM | 0.18 [0.10, 0.30] | 0.14 [0.11, 0.18]
JAMA | 0.29 [0.15, 0.55] | 0.26 [0.21, 0.31]
PLoS | 0.12 [0.05, 0.54] | 0.21 [0.13, 0.32]
Combined | 0.13 [0.08, 0.21] | 0.19 [0.17, 0.20]

False discovery risk (FDR) estimates and 95% confidence interval for each of
the analyzed journal and combined data set based on clinical trials and
randomized controlled trials published since 2000 with z-curve and Jager and
Leek’s [19] swfdr method.

Figure 3: Estimates of false discovery risk across decades, journals, and
study types. Estimated false discovery rate (FDR, y-axis) for randomized
controlled trials (RCT, black-filled squares) and clinical trials (CT, empty
squares) with z-curve and divided by journal (x-axis, with information about
unique articles in brackets, first line for RCT and second line for CT) and
whether the article was published before (and including) 2010 (left) or after
2010 (right).

## Additional z-curve results

So far, we used the expected discovery rate only to estimate the false
discovery risk, but the expected discovery rate provides valuable information
in itself. Ioannidis’s [3] predictions of the false discovery rate for
clinical trials were based on two scenarios. One scenario assumed high power
(80%) and a 1:1 ratio of true and false hypothesis. This scenario implies a
true discovery rate of .5*.8 + .5*.05 = 42.5%. The second scenario assumed 20%
power and a ratio of 1 true hypothesis for every 5 false hypotheses. This
scenario implies a true discovery rate of .17*.20 + .83*.05 = 7.5%. Scenarios
also varied in terms of bias that would inflate the observed discovery rates.
However, z-curve corrects for this bias. Thus, it is interesting to compare
z-curve’s estimates of the discovery rate with Ioannidis’s assumptions about
the discovery rates in clinical trials. The z-curve estimate of the EDR was
30% with a 95% confidence interval from 20% to 41%. This finding suggests that
most published results from clinical trials in our sample were well-powered
and likely to test a true hypothesis. This conclusion is also consistent with
the finding that our estimate of the false discovery risk is similar to
Ioannidis’s predictions for high quality clinical trials, FDR = 15%. Thus, a
simple explanation for the discrepancy between Ioannidis’s claim and our
results is that Ioannidis’s underestimated the proportion of well-powered
studies in the literature.

The expected discovery rate also provides valuable information about the
extent of selection bias in medical journals. While the expected discovery
rate is only 30%, the observed discovery rate (i.e., the percentage of
significant results in abstracts) is more than double (69.7%). This
discrepancy is visible in Figure 4. The histogram of observed non-significant
z-scores does not match the predicted distribution (blue curve). This evidence
of considerable selection bias implies that reported effect sizes are inflated
by selection bias. Thus, follow-up studies need to adjust effect sizes when
planning the sample sizes via power analyses. Moreover, meta-analyses need to
correct for selection bias to obtain unbiased estimates of the average effect
size [27].

Figure 4: Z-curve of the combined data set. Distribution of all significant
$p$-values converted into $z$-statistics with fitted z-curve density (solid
blue line) and 95% point-wise confidence bands (doted blue line).

Z-curve also provides information about the expected replication rate of
statistically significant results in medical abstracts, defined as the
probability of obtaining a statistically significant result in a close
replication study with the same sample size and significance criterion (see
e.g., [28, 29, 30] for alternatives). The expected replication rate is not a
simple complement of the false discovery risk, as a true discovery might fail
to replicate due to low power of the replication study and the false discovery
might “falsely” replicate with a probability corresponding to $\alpha$. The
expected replication rate would be 65% with a confidence interval ranging from
61% to 69%. It is important to realize that this expected replication rate
includes replications of true and false hypotheses. Although the actual false
positive rate is not known, we can use the z-curve estimate to assume that 14%
of the significant results are false positives. Under this assumption the
power of replication studies of true hypotheses is 75%. In contrast, the
probability of obtaining a false positive result again is only 0.7%. Thus,
actual replication studies even with the same sample sizes successfully reduce
the false positive risk. However, the risk of false negatives is substantial
and 44% of non-significant results in replication studies test a true
hypothesis.

Furthermore, z-curve estimates of the replication rate are higher than actual
replication rates [26]. One reason could be that exact replication studies are
impossible and changes in population will result in lower power due to
selection bias and regression to the mean. In the worst case, the actual
replication rate might be as low as the expected discovery rate. Thus, our
results predict that the success rate of actual replication studies in
medicine will be somewhere between 30% and 65%. To ensure a high probability
of replicating a true result, it is therefore necessary to increase sample
sizes of replication studies. When this is not possible, it is important to
interpret an unsuccessful replication outcomes with caution and to conduct
further research. A single failed replication study is inconclusive.

Ioannidis [23] suggested that an FDR of 14% is a desirable goal. However, many
researchers falsely assume that the $\alpha$ = .05 ensures a false discovery
risk of only 5%. Moreover, the longevity of $\alpha$ = .05 suggests that many
researchers consider an error rate of 5% acceptable. We showed that $\alpha$ =
.05 entails an FDR of 14%. To lower the FDR to 5% or less, it is necessary to
lower $\alpha$. Z-curve analysis can help to find an $\alpha$ level that meets
this criterion. With $\alpha$ = .01, the expected discovery rate decreases to
20% and the false discovery risk decreases to 4%. Based on these results, it
is possible to use $\alpha$ = .01 as a criterion to reject the null-hypothesis
while maintaining a false positive risk below 5%. While journals may be
reluctant to impose a lower level of $\alpha$, readers can use this criterion
to maintain a reasonably low risk of accepting a false hypothesis.

## Discussion

Like many other human activities, science relies on trust. Ioannidis’s (2005)
article suggested that trust in science is unwarranted and that readers are
more likely to encounter false than true claims in scientific journals. In
response to these concerns, a new field of meta-science emerged to examine the
credibility of published results. Our article builds on these efforts by
introducing an improved method to estimate the false discovery risk in medical
journals, with a focus on clinical trials. Contrary to Ioannidis’s suggestion
that most of these results may be false positives, we found that the false
discovery risk is 14% with the traditional criterion for statistical
significance ($\alpha$ = .05) and 4% with a lower threshold of $\alpha$ = .01.
Thus, our results provide some reassurance that medical research is more
robust than Ioannidis predicted.

At the same time, our results also show some problems that could be easily
addressed by journal editors. The biggest problem remains publication bias in
favor of statistically significant results. The selection for significance has
many undesirable consequences. Although medicine has responded to this problem
by demanding preregistration of clinical trials, our results suggest that
selection for significance remains a pervasive problem in medical research. A
novel contribution of z-curve is the ability to quantify the amount of
selection bias. Journal editors can use this tool to track selection bias and
implement policies to reduce it. This can be achieved in several ways. First
editors may publish more well-designed replication studies with non-
significant results. Second, editors may check more carefully whether
researchers followed their preregistration. Finally, editors can prioritize
studies with high sample sizes.

Despite our improvements over Jager and Leel’s study, our study has a number
of limitations that can be addressed in future research. One concern is that
results in abstracts may not be representative of other results in the results
section. Another concern is that our selection of journals is not
representative. This are valid concerns that limit our conclusions to the
particular journals that we examined. It would be wrong to generalize to other
scientific disciplines or other medical journals. We showed that z-curve is a
useful tool to estimate the false positive risk and to adjust $\alpha$ to
limit the false positive risk to a desirable level. This tool can be used to
estimate the false discovery risk for other disciplines and research areas and
set $\alpha$ levels that are consistent with the discovery rates of these
fields. We believe that empirical estimates of discovery rates, false positive
risks, selection bias, and replication rates can inform scientific policies to
ensure that published results provide a solid foundation for scientific
progress.

## Data and code availability

Supplementary Materials including data and R scripts for reproducing the
simulations, data scraping, and analyses are available from
https://osf.io/y3gae/. The zcurve R package is available from
https://cran.r-project.org/package=zcurve.

## References

  * [1] Baker M. Reproducibility crisis. Nature 2016; 533(26): 353–66.
  * [2] Fanelli D. Opinion: Is science really facing a reproducibility crisis, and do we need it to?. Proceedings of the National Academy of Sciences 2018; 115(11): 2628–2631.
  * [3] Ioannidis JP. Why most published research findings are false. PLoS Medicine 2005; 2(8): e124.
  * [4] Open Science Collaboration . Estimating the reproducibility of psychological science. Science 2015; 349(6251).
  * [5] Camerer CF, Dreber A, Forsell E, et al. Evaluating replicability of laboratory experiments in economics. Science 2016; 351(6280): 1433–1436.
  * [6] Camerer CF, Dreber A, Holzmeister F, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour 2018; 2(9): 637–644.
  * [7] Coyne JC. Replication initiatives will not salvage the trustworthiness of psychology. BMC Psychology 2016; 4(1): 1–11.
  * [8] Fanelli D, Costas R, Ioannidis JP. Meta-assessment of bias in science. Proceedings of the National Academy of Sciences 2017; 114(14): 3714–3719.
  * [9] Wooditch A, Fisher R, Wu X, Johnson NJ. P-value problems? An examination of evidential value in criminology. Journal of Quantitative Criminology 2020; 36(2): 305–328.
  * [10] Barnes J, TenEyck MF, Pratt TC, Cullen FT. How powerful is the evidence in criminology? On whether we should fear a coming crisis of confidence. Justice Quarterly 2020; 37(3): 383–409.
  * [11] Nuijten MB, Assen vMA, Augusteijn HE, Crompvoets EA, Wicherts JM. Effect sizes, power, and biases in intelligence research: a meta-meta-analysis. Journal of Intelligence 2020; 8(4): 36.
  * [12] Stanley TD, Carter EC, Doucouliagos H. What meta-analyses reveal about the replicability of psychological research.. Psychological Bulletin 2018; 144(12): 1325.
  * [13] Ioannidis JPA, Stanley TD, Doucouliagos H. The power of bias in economics research. The Economic Journal 2017; 127(605): F236-F265.  doi: 10.1111/ecoj.12461
  * [14] Van Aert RC, Wicherts JM, Van Assen MA. Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis. PloS One 2019; 14(4): e0215052.
  * [15] Mathur MB, VanderWeele TJ. Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers. Research Synthesis Methods 2021; 12(2): 176–191.
  * [16] Lamberink HJ, Otte WM, Sinke MR, et al. Statistical power of clinical trials increased while effect size remained stable: an empirical analysis of 136,212 clinical trials between 1975 and 2014. Journal of Clinical Epidemiology 2018; 102: 123–128.
  * [17] Bartoš F, Gronau QF, Timmers B, Otte WM, Ly A, Wagenmakers EJ. Bayesian model-averaged meta-analysis in medicine. Statistics in Medicine 2021; 40(30): 6743-6761.
  * [18] Bartoš F, Maier M, Wagenmakers EJ, et al. Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics. Preprint at: https://arxiv.org/abs/2208.12334; 2022
  * [19] Jager LR, Leek JT. An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 2014; 15(1): 1–12.
  * [20] Goodman SN. Discussion: An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 2014; 15(1): 13–16.
  * [21] Gelman A, O’Rourke K. Difficulties in making inferences about scientific truth from distributions of published p-values. Biostatistics 2014; 15(1): 18–23.
  * [22] Benjamini Y, Hechtlinger Y. Discussion: An estimate of the science-wise false discovery rate and applications to top medical journals by Jager and Leek. Biostatistics 2014; 15(1): 13–16.
  * [23] Ioannidis JP. Discussion: Why “An estimate of the science-wise false discovery rate and application to the top medical literature” is false. Biostatistics 2014; 15(1): 28–36.
  * [24] Sorić B. Statistical “discoveries” and effect-size estimation. Journal of the American Statistical Association 1989; 84(406): 608–610.
  * [25] Brunner J, Schimmack U. Estimating population mean power under conditions of heterogeneity and selection for significance. Meta-Psychology 2020; 4.
  * [26] Bartoš F, Schimmack U. Z-curve 2.0: Estimating replication rates and discovery rates. Meta-Psychology 2021\.
  * [27] Bartoš F, Maier M, Wagenmakers EJ, Doucouliagos H, Stanley TD. Robust Bayesian meta-analysis: Model-averaging across complementary publication bias adjustment methods. Research Synthesis Methods 2022\.
  * [28] Held L, Micheloud C, Pawel S. The assessment of replication success based on relative effect size. The Annals of Applied Statistics 2022; 16(2): 706 – 720.
  * [29] Pawel S, Held L. The sceptical Bayes factor for the assessment of replication success. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2022; 84(3): 879–911.
  * [30] Ly A, Etz A, Marsman M, Wagenmakers EJ. Replication Bayes factors from evidence updating. Behavior Research Methods 2019; 51(6): 2498–2508.