# Did You Mean…? Confidence-based Trade-offs in Semantic Parsing

Elias Stengel-Eskin
Johns Hopkins University
<EMAIL_ADDRESS>Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
###### Abstract

We illustrate how a calibrated model can help balance common trade-offs in
task-oriented parsing. In a simulated annotator-in-the-loop experiment, we
show that well-calibrated confidence scores allow us to balance cost with
annotator load, improving accuracy with a small number of interactions. We
then examine how confidence scores can help optimize the trade-off between
usability and safety. We show that confidence-based thresholding can
substantially reduce the number of incorrect low-confidence programs executed;
however, this comes at a cost to usability. We propose the DidYouMean system
(cf. Fig. 1) which better balances usability and safety.

##  1 Introduction

Task-oriented dialogue systems (Gupta et al., 2018; Cheng et al., 2020;
Semantic Machines et al., 2020) represent one path towards achieving the long-
standing goal of using natural language as an API for controlling real-world
systems by transforming user requests into executable programs, i.e.
translating natural language to code. Central to the systems’ success is the
ability to take rational actions under uncertainty (Russell and Norvig, 2010).
When model confidence is low and the system is unlikely to succeed, we would
prefer it defer actions and request clarification, while at high confidence,
clarification requests may annoy a user. Relying on model confidence requires
it to be well-correlated with accuracy, i.e. it requires a _calibrated_ model.

Figure 1: The DidYouMean system. At high confidences, we simply execute the
predicted parse. At low confidences, DidYouMean rephrases the query based on
the predicted program and asks a user to confirm the paraphrase. The program
is executed if the user accepts.

Recent work has focused on the calibration of semantic parsing models.
Specifically, Stengel-Eskin and Van Durme (2022) benchmarked the calibration
characteristics of a variety of semantic parsing models, finding some of them
to be well-calibrated, especially on parsing for task-oriented dialogue. Given
the relatively well-calibrated nature of these models, we first examine how
they could be used in an annotation interface, with a view to balancing the
trade-off between _annotation cost_ and _correctness_. We simulate a human-in-
the-loop (HITL) experiment where high-confidence tokens are automatically
annotated and low-confidence tokens trigger a dialogue with an oracle
annotator, who either picks the correct token from a top-K list or manually
inserts it. With a small number of interactions we substantially boost
annotator accuracy.

A similar trade-off exists between _usability_ and _safety_ in task-oriented
user interfaces. We examine how sequence-level model confidence scores can be
used to balance this trade-off by reducing the number of incorrect programs
executed while also minimizing the number of follow-up user interactions and
their cognitive burden. We find that thresholding outputs based on model
confidence (i.e. rejecting outputs falling below a tuned threshold) reduces
the number of incorrect programs executed by $76\%$ compared to the baseline.
However, this comes at a cost to usability, where roughly half the correctly-
predicted parses are also rejected. To strike a balance between safety and
usability, we introduce the DidYouMean system (cf. Fig. 1), which rephrases
the input conditioned on the predicted parse and asks users to confirm the
accuracy of the paraphrase. In a user study, we obtain an $36\%$ improvement
in usability over the thresholded system while maintaining a $58\%$ reduction
in the number of incorrect programs executed.

##  2 Related Work

Our experiments in Section 4 involve a predictive model for human-in-the-loop
coding: similar models have been integrated into IDEs, e.g. Chen et al.
(2021). DidYouMean relates to the interactive semantic parsing domain (Li and
Jagadish, 2014; Chaurasia and Mooney, 2017; Su et al., 2018), where humans are
included in the semantic parsing loop. In this domain, Yao et al. (2019)
introduce a confidence-based interactive system in which a parsing agent can
ask users for clarification. Our work follows in this spirit, but asks the
user to confirm a parse rather than generating questions for the user to
answer. DidYouMean also relates broadly to selective prediction, where a model
is expected to abstain from making decisions at low confidence (Chow, 1957;
El-Yaniv et al., 2010; Varshney et al., 2022; Xin et al., 2021; Whitehead et
al., 2022). Our system extends beyond selective prediction’s setting by
including a human-in-the-loop. Finally, DidYouMean shares a motivation with
Fang et al. (2022), who introduce a method for reliably summarizing programs.
Their work provides post-hoc action explanations while we focus on resolving
misunderstandings _before_ execution.

##  3 Methods

Datasets Our data is drawn from the SMCalFlow (Semantic Machines et al., 2020)
task-oriented dialogue dataset, which contains Lisp-like programs (cf.
Appendix A). We follow the same preprocessing as Platanios et al. (2021), and
use the SMCalFlow data splits given by Roy et al. (2022): $108,753$ training,
$12,271$ validation, and $13,496$ testing dialogue turns.

Models We use MISO (Zhang et al., 2019a, b), a well-calibrated model from
Stengel-Eskin and Van Durme (2022). Rather than predict the SMCalFlow surface
form, including syntactic tokens like parentheses, MISO directly predicts the
underlying execution graph. The graph can deterministically be “de-compiled”
into its surface form, and vice-versa. The fact that MISO predicts an
underlying graph makes it attractive for applications which require confidence
scores, as it only predicts content tokens (i.e. functions, arguments) rather
than syntactic tokens (parentheses). For details on MISO’s architecture, see
Zhang et al. (2019b) and Stengel-Eskin et al. (2022).

For token confidence estimation, we use the maximum probability across the
output vocabulary at each timestep. This has been shown to be a relatively
robust confidence estimator in classification (Hendrycks and Gimpel, 2016;
Varshney et al., 2022). For sequence-level scores, we follow Stengel-Eskin and
Van Durme (2022) and take the minimum over token-level confidence scores.

##  4 Human-in-the-Loop Simulation

Production datasets like SMCalFlow are constantly evolving as new
functionalities are added. The expensive and time-consuming nature of
annotating data can be mitigated by the use of predictive parsing models which
suggest speculative parses for new utterances. However, the model’s output can
be incorrect, especially given out-of-distribution inputs. We need to ensure
that annotators are not introducing errors by overly trusting the model.

If the model is well-calibrated, we can use the confidence to reduce such
errors. For example, we can alert annotators to low confidence predictions and
ask them to intervene (Lewis and Gale, 1994). Using a threshold, we can
prioritize time or correctness: a higher threshold would result in more
annotator-model interactions, decreasing the speed but increasing program
correctness (reducing the need for debugging) while a lower threshold would
increase speed but also lower the accuracy.

Since we do not have access to expert SMCalFlow annotators, we simulate an
oracle human-in-the-loop (HITL) annotator who always provides a correct answer
by using the gold annotations provided in the dataset. Specifically, for a
given input, we decode the output tokens of a predicted program $o_{0},\ldots
o_{n}$ normally as long as predictions are confident (above a given
threshold). If at time $t$ the confidence $p(o_{t})$ falls the threshold, we
attempt to match the decoded prefix $o_{0},\ldots,o_{t-1}$ to the gold prefix
$g_{0},\ldots g_{t-1}$. If the prefixes do not match, we count the example as
incorrect. If they do match, we replace $o_{t}$ with $g_{t}$, the gold
prediction from our oracle annotator, and continue decoding. We consider three
metrics in this experiment: (1) The exact match accuracy of the decoded
programs (higher is better). (2) The percentage of total tokens for which we
have to query an annotator (lower is better). (3) The percentage of uncertain
tokens (below the threshold) for which the gold token $g_{t}$ is in the top 5
predictions at timestep $t$. Here, higher is better, as selecting a token from
a candidate list is typically faster than producing the token.

Results and Analysis Fig. 2 shows our three metrics as the threshold is
increased in increments of $0.1$. We see that accuracy grows exponentially
with a higher threshold, and that the percentage of tokens for which an
annotator intervention is required grows at roughly the same rate. The
exponential growth reflects the distribution of token confidences, with most
tokens having high confidence. Finally, we see that while at low confidence,
most tokens must be manually inserted, the rate at which they are chosen from
the top 5 list rapidly increases with the threshold. Thus, the increased
number of annotator interactions required at higher thresholds may be offset
by the fact that many of these interactions are a choice from the top-5 list.

Figure 2: Simulated annotator-in-the-loop results across increasing confidence
thresholds.

##  5 User Correction via DidYouMean

Section 4 showed that token-level confidence scores can be used to balance
speed and correctness in an annotation interface. We see a similar trade-off
between safety and usability in user interfaces using semantic parsing models.
Here, we define safety as rejecting unsuccessful programs _before executing
them_. This strict definition is motivated by physical domains: imagine that
rather than controlling a digital assistant, a user is guiding a robot via
language commands (e.g. Winograd (1972); Lynch and Sermanet (2020); Stengel-
Eskin et al. (2021); Lynch et al. (2022); Nair et al. (2022)). In this
setting, actions may have irreversible consequences, so determinine safety
before execution is key. Safety considerations need to be balanced with
usability of the system: an unplugged agent would be very safe but unusable.
To increase usability, an agent might make follow-up requests to a user, like
asking for clarification or confirmation. The types of requests the agent
makes have varying cognitive load on the user: for example, providing
confirmation takes less effort than rephrasing.

We measure how well we can reject incorrect programs _before_ executing them.
Following past work in selective prediction, we measure success by coverage
and risk, as well as F score w.r.t. program correctness. Coverage is the
percentage of inputs for which a program is executed and risk is the
percentage of executed programs that were _incorrect_. Precision is inverse
risk, and recall is the percentage of correct programs which were accepted. We
additionally consider F1 and F0.5, which upweights precision (safety) by a
factor of 2. A low-coverage, low-risk system may be safer but have more false
negatives, i.e. reject more correct programs, decreasing its usability. A
high-coverage, high-risk system is more usable at the cost of false positives,
i.e. executing incorrect programs. We do not commit to setting an optimal
threshold for this trade-off, since it is task-specific.

We consider 3 systems. As a baseline, we consider a system that executes
everything it predicts (accept); this will result in the highest-possible
coverage, but also high risk. We can also use MISO’s calibrated nature to
improve safety outcomes by tuning a sequence-level confidence threshold for
rejecting programs (tuned). We tune on the full validation set using F1; we
explore the range $[0.0,1.0)$ in increments of $0.01$, finding $0.40$ to be
optimal. Finally, we introduce the DidYouMean system for filtering low-
confidence programs. For a given utterance, DidYouMean shows the user a
paraphrase of the input; the user then decides to accept the parse based on
this paraphrase. This allows correctly-predicted low-confidence programs to be
accepted and executed, while reducing the user load: making a binary choice to
accept a paraphrase is a receptive task, while rephrasing an instruction is a
more costly productive task.

Glossing Model Since users are typically unfamiliar with formats like Lisp, we
need to present the user with a natural language paraphrase – or _gloss_ – of
the candidate parse. To train a glossing model, we modify Roy et al. (2022)’s
seq2seq BenchCLAMP framework: rather than using the user utterance with the
previous turn’s context $(\mathcal{U}_{0},\mathcal{A}_{0},\mathcal{U}_{1})$ as
input and a program $\mathcal{P}$ as output, we take the context and _program_
$(\mathcal{U}_{0},\mathcal{A}_{0},\mathcal{P})$ as the input and user
instruction $\mathcal{U}_{1}$ as the output. We use the BART-large
architecture (Lewis et al., 2020). See Appendix A for model details.

DidYouMean System When a low-confidence parse is detected, DidYouMean triggers
a dialogue with the user in order to recover some usability over simply
rejecting all low-confidence parses. Fig. 1 shows the system workflow.
DidYouMean shows the original utterance $\mathcal{U}_{1}$ and the gloss
$\hat{\mathcal{U}}^{*}$ to the user, who determines whether they are identical
or not. If they accept the gloss, we optionally re-parse the gloss
$\hat{\mathcal{U}}^{*}$ rather than the original utterance $\mathcal{U}_{1}$;
this can remove typos and other idiosyncracies. We call this the re-parsed
setting, while choosing the original prediction $\hat{\mathcal{U}}$ is the
chosen setting. We predict that allowing users to accept and reject glosses
will improve the balance between safety and usability (i.e. F1) over the
threshold system by allowing them to accept correct low-confidence parses. In
other words, adding human interaction will allow us to achieve a balance which
cannot be attained given the tradeoffs resulting from thresholding.

User Study We conduct a static user study of DidYouMean using examples from
the SMCalFlow validation set. We sample 100 MISO predictions with a minimum
confidence below $0.6$ (to ensure that the set contains a number of mistakes).
This sample is stratified across 10 equally-spaced bins with 10 samples per
bin. MTurk annotators were shown the dialogue history, the user utterance, and
the gloss, and asked to confirm that the gloss was correct. The template and
instructions can be seen in Appendix B. We obtained 3 judgments per example.

Annotation Statistics 8 annotators completed at least one judgement, with 4
completing the majority. All 3 annotators agreed on $79\%$ examples,
indicating the task is well-formulated. For the remaining $21\%$, we use the
majority decision to accept or reject. After majority voting, annotators
accepted $68/100$ glosses and rejected $32$.

Results

Setting | Cov. $\uparrow$ | Risk $\downarrow$ | FP $\downarrow$ | F1 $\uparrow$ | F0.5 $\uparrow$
---|---|---|---|---|---
Accept | 1.00 | 0.67 | 67 | 0.50 | 0.38
Tuned | 0.32 | 0.50 | 16 | 0.49 | 0.50
Chosen | 0.68 | 0.54 | 37 | 0.61 | 0.51
Re-parsed | 0.68 | 0.41 | 28 | 0.66 | 0.62
Table 1: Coverage, risk, number of false positives (FP), and F measures for
accepting correct parses and rejecting incorrect parses.

Table 1 shows the results of the user study. In addition to standard selective
prediction metrics like coverage (the percentage of inputs for which a program
is executed) and risk (the percentage of executed programs that are incorrect)
we report the number of false positives (incorrect programs executed) and F1
and F0.5 scores. Tuning a threshold results yields better safety outcomes than
accepting everything, with lower risk. However, this safety comes at a cost to
the usability of the system; a coverage of only 0.32 indicates that only
$32\%$ of inputs have their programs executed. The “tuned” system’s low
usability is reflected in the F1 and F0.5 scores, which balance precision and
recall. The “chosen” system, while better in F1, is comparable to the “tuned”
system in F0.5, which takes both usability and safety into account but
prioritizes safety at a 2:1 ratio. Users are able to recover some usability
(as measured by coverage) in this setting but also add to the risk, which is
higher for “chosen” than “tuned”. The number of incorrect programs executed
increases when glosses are chosen (as compared to the tuned threshold). When
the accepted glosses are re-parsed, we see a shift back towards a system
favoring safety, with fewer incorrect programs being executed than in the
“chosen” setting; this is reflected in a lower risk score. For both F1 and
F0.5, the “re-parsed” system best balances usability and safety.

These results show that a calibrated model can be used with a threshold to
greatly improve safety, reducing the number of incorrect programs accepted by
$76\%$. DidYouMean allows users to recover some low-confidence programs by
accepting and rejecting programs based on their glosses, resulting in the best
aggregated scores. Note also that the threshold was tuned on F1 score on the
entire dev set. This means that the F1 performance of that tuned system is as
high as possible for confidence-threshold-based system. Thus, DidYouMean
achieves a balance outside what can be achieved by tuning: simply increasing
the threshold would decrease safety and result in a lower F1 than the current
threshold of 0.40.

##  6 Conclusion

We examine two common trade-offs in semantic parsing, and how a well-
calibrated model can be used to balance them. In Section 4 we illustrated how
token-level model confidences could be used in a simulated HITL task-oriented
parsing annotation task. Our experiments in Section 5 extended these results
to sequence-level confidences and non-expert users; we found that model
confidence could be used to improve the usability-safety trade-off and
introduced DidYouMean, which improved usability by asking users to accept
predictions.

##  7 Limitations

Our study is limited by the models, datasets, and languages we consider.
Firstly, we examine only English datasets, limiting the impact of our results.
We also only consider one task-oriented parsing dataset, and focus on one
model architecture.

We make several limiting assumptions in Section 4 and Section 5. Foremost
amongst these is the assumption of access to an oracle annotator in Section 4;
clearly, no such annotator exists. Our results may vary when real annotators
are brought into the loop. For one, we do not know exactly how choosing from
the top-k list will compare to insertion w.r.t. speed. We also do not know how
automation bias (Cummings, 2004) would affect the top-k list: given that the
correct answer is often in the list, real annotators might overly rely on the
list and prefer it to token insertion, resulting in incorrect programs.

The experiments in Section 5 rely on a glossing model to translate predicted
programs into natural language (NL). We approached with a neural Lisp-to-NL
model; this has several limitations. Neural text generation models often
hallucinate outputs, i.e. generated glosses may not be faithful to their
corresponding programs. Unlike Fang et al. (2022), who use a grammar-based
approach for response generation, we do not assume access to a grammar but
note that our method is compatible with grammar-based constraints. Our
annotators in Section 5 face the additional challenge of interpreting and
choosing glosses. SMCalFlow programs are nuanced and slight input variations
can result in different programs. These nuances are often obscured by the
glossing model, resulting in two different programs glossing to semantically
equivalent utterances; we explore this further in Appendix C. Annotators might
mistakenly accept glosses from incorrect programs or reject correct glosses;
this would be difficult to address even with a faithful translation method.

## Acknowledgements

We would like to thank Anthony Platanios, Subhro Roy, Zhengping Jiang, Kate
Sanders, Yu Su, and Daniel Khashabi for their feedback on an earlier draft.
This work was supported by NSF $\\#1749025$, and Elias Stengel-Eskin is
supported by an NSF Graduate Research Fellowship.

## References

  * Chaurasia and Mooney (2017) Shobhit Chaurasia and Raymond Mooney. 2017.  Dialog for language to code.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 175–180.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Cheng et al. (2020) Jianpeng Cheng, Devang Agrawal, Héctor Martínez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li, Dhivya Piraviperumal, Jason D. Williams, Hong Yu, Diarmuid Ó Séaghdha, and Anders Johannsen. 2020.  Conversational semantic parsing for dialog state tracking.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8107–8117, Online. Association for Computational Linguistics.
  * Chow (1957) Chi-Keung Chow. 1957.  An optimum character recognition system using decision functions.  In _IRE Transactions on Electronic Computers_ , 4, pages 247–254. IEEE.
  * Cummings (2004) Mary Cummings. 2004.  Automation bias in intelligent time critical decision support systems.  _AIAA 1st Intelligent Systems Technical Conference_.
  * El-Yaniv et al. (2010) Ran El-Yaniv et al. 2010.  On the foundations of noise-free selective classification.  _Journal of Machine Learning Research_ , 11(5).
  * Fang et al. (2022) Hao Fang, Anusha Balakrishnan, Harsh Jhamtani, John Bufe, Jean Crawford, Jayant Krishnamurthy, Adam Pauls, Jason Eisner, Jacob Andreas, and Dan Klein. 2022.  The whole truth and nothing but the truth: Faithful and controllable dialogue response generation with dataflow transduction and constrained decoding.  _arXiv preprint arXiv:2209.07800_.
  * Gupta et al. (2018) Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018.  Semantic parsing for task oriented dialog using hierarchical representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2787–2792.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Lewis and Gale (1994) David D Lewis and William A Gale. 1994.  A sequential algorithm for training text classifiers.  In _SIGIR’94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University_ , pages 3–12. Springer.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Li and Jagadish (2014) Fei Li and Hosagrahar V Jagadish. 2014.  Constructing an interactive natural language interface for relational databases.  _Proceedings of the VLDB Endowment_ , 8(1):73–84.
  * Lynch and Sermanet (2020) Corey Lynch and Pierre Sermanet. 2020.  Language conditioned imitation learning over unstructured data.  _arXiv preprint arXiv:2005.07648_.
  * Lynch et al. (2022) Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. 2022.  Interactive language: Talking to robots in real time.  _arXiv preprint arXiv:2210.06407_.
  * Nair et al. (2022) Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. 2022\.  Learning language-conditioned robot behavior from offline data and crowd-sourced annotation.  In _Conference on Robot Learning_ , pages 1303–1315. PMLR.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Platanios et al. (2021) Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, and Dan Klein. 2021.  Value-agnostic conversational semantic parsing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3666–3681, Online. Association for Computational Linguistics.
  * Roy et al. (2022) Subhro Roy, Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Eisner, and Benjamin Van Durme. 2022.  Benchclamp: A benchmark for evaluating language models on semantic parsing.  _arXiv preprint arXiv:2206.10668_.
  * Russell and Norvig (2010) Stuart Russell and Peter Norvig. 2010.  _Artificial intelligence a modern approach_.  Pearson Education, Inc.
  * Semantic Machines et al. (2020) Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020.  Task-oriented dialogue as dataflow synthesis.  _Transactions of the Association for Computational Linguistics_ , 8:556–571.
  * Stengel-Eskin et al. (2021) Elias Stengel-Eskin, Andrew Hundt, Zhuohong He, Aditya Murali, Nakul Gopalan, Matthew Gombolay, and Gregory D. Hager. 2021.  Guiding multi-step rearrangement tasks with natural language instructions.  In _5th Annual Conference on Robot Learning_.
  * Stengel-Eskin et al. (2022) Elias Stengel-Eskin, Emmanouil Antonios Platanios, Adam Pauls, Sam Thomson, Hao Fang, Benjamin Van Durme, Jason Eisner, and Yu Su. 2022.  When more data hurts: A troubling quirk in developing broad-coverage natural language understanding systems.  _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Stengel-Eskin and Van Durme (2022) Elias Stengel-Eskin and Benjamin Van Durme. 2022.  Calibrated interpretation: Confidence estimation in semantic parsing.  _arXiv preprint arXiv:2211.07443_.
  * Su et al. (2018) Yu Su, Ahmed Hassan Awadallah, Miaosen Wang, and Ryen W White. 2018.  Natural language interfaces with fine-grained user interaction: A case study on web apis.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 855–864.
  * Varshney et al. (2022) Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022.  Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1995–2002.
  * Whitehead et al. (2022) Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. 2022.  Reliable visual question answering: Abstain rather than answer incorrectly.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Winograd (1972) Terry Winograd. 1972.  Understanding natural language.  _Cognitive psychology_ , 3(1):1–191.
  * Xin et al. (2021) Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021.  The art of abstention: Selective prediction and error regularization for natural language processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1040–1051.
  * Yao et al. (2019) Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019.  Model-based interactive semantic parsing: A unified framework and a text-to-sql case study.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5447–5458.
  * Zhang et al. (2019a) Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a.  AMR parsing as sequence-to-graph transduction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 80–94, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2019b) Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019b.  Broad-coverage semantic parsing as transduction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3786–3798, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Model and Data

### Data

We use version 2.0 of SMCalFlow, which contains language inputs paired with
Lisp-like programs. Fig. 3 shows an example input and output for SMCalFlow.
Unlike many other task-oriented formalisms, SMCalFlow contains salience
operators like refer, which allow for deixis in the input.

Figure 3: Input-output example for SMCalFlow

### Input Representation

We follow Stengel-Eskin et al. (2022) and Roy et al. (2022) in using the
previous dialogue turn as input if available. Thus, each datapoint consists of
an input $X=(\mathcal{U}_{0},\mathcal{A}_{0},\mathcal{U}_{1})$ and an output
program $\mathcal{P}$, where $\mathcal{U}_{0}$ is the previous user utterance
(if it exists), $\mathcal{A}_{0}$ is an automatically-generated agent response
to the previous utterance, and $\mathcal{U}_{1}$ is the current user
utterance. Paraphrasing To find a good paraphrase of each predicted parse, we
generate $N$ glosses $\hat{\mathcal{U}}_{1},\ldots,\hat{\mathcal{U}}_{N}$ via
beam search and take the one that yields the highest probability of the
predicted parse $\hat{\mathcal{P}}$, i.e.
$\hat{\mathcal{U}}^{*}=\text{argmax}_{i}\>P_{MISO}(\hat{\mathcal{P}}|\mathcal{U}_{0},\mathcal{A}_{0},\hat{\mathcal{U}}_{i})$.

### Evaluating the glossing model

Instead of evaluating the glossing model using string metrics such as BLEU
(Papineni et al., 2002) which can be noisy, we choose to evaluate the output
programs using cycle-consistency. Specifically, we evaluated trained models by
glossing programs in the gold test set and then parsing those glosses with a
fixed MISO parser. We explore T5-base, T5-large, BART-base, and BART-large
architectures. All accuracy scores are reported in Fig. 4 along with the
baseline accuracy obtained by parsing the gold test inputs. The best-
performing gloss model is BART-large. Note that all glossing models outperform
MISO without glosses. This can be explained by the fact that we gloss the
inputs from the gold program, which we then evaluate on, allowing for
information leakage. We also hypothesize that the gloss model, having been
trained on the entire dataset of utterances, averages out many annotator-
specific idiosyncracies which may make inputs hard to parse. This result does
_not_ imply that glosses generated from _predicted_ programs would yield
better performance when parsed than the user input.

Figure 4: Exact match for programs parsed from glossed inputs from the
SMCalFlow test set. Performance without glosses is overlaid in black.

##  Appendix B User study template

Fig. 5 shows the template for the confirmation HIT. The instructions asked
users to read the paraphrase produced by the agent and determine whether the
agent had correctly understood the user.

Figure 5: Template for the confirmation HIT.

Annotators were recruited from a list of qualified annotators on Amazon
Mechanical Turk and paid $\$0.05$ per example, averaging to roughly $\$16$ per
hour.

##  Appendix C Selection HIT

In Section 5 we presented the DidYouMean system, which allowed users to
confirm or reject a potential gloss. The gloss was chosen from a larger set of
candidates; this naturally raises the question of whether users can directly
choose a gloss from the candidate list, rather than accepting or rejecting a
single gloss at a time. Here, we examine to what extent it is helpful to users
to choose glosses from a list of options. We take a sample of validation
programs for SMCalFlow, stratified across 10 evenly-spaced bins from 0.0 to
1.0. Note that this differs from the setting in Section 5, where the maximum
bin was 0.6. We then present the top predictions using nucleus sampling
(Holtzman et al., 2019); the number of predictions depends on the confidence
of the model. Specifically, we set a cutoff of 0.85 and add predictions until
the sum of the minimum token probabilities across predictions is greater than
0.85. We limit ourselves to a maximum of 10 predictions, even if the cutoff is
not reached.

The template is seen in Fig. 6; each example gives the gloss as well as a
rounded confidence score for the predicted program. Annotation was run with
the same group of annotators as the experiments in Section 5; annotators were
paid $\$0.11$ per example, or about $\$16$ per hour of annotation. Each
example was annotated by a single annotator, all of whom had been vetted in a
pilot annotation task. Annotators were instructed to help a robot named
SvenBot, who had recently learned English and was not confident about its
understanding, disambiguate between several options. The interface contained a
text box where annotators could optionally manually re-write the input; this
was only to be done in cases where _none_ of the options reflected the
intended meaning of the original utterance.

Figure 6: Template for the selection HIT.

Of the 100 examples sampled, annotators manually rewrote 7 and chose from the
top list for the other 93. Ignoring the rewritten examples, 39 model
predictions were incorrect and 54 were correct; by choosing glosses,
annotators correct 5 incorrect predictions. However, they also inadvertently
changed 4 correct predictions to incorrect glosses. Figure Fig. 7 shows the
exact match accuracy before and after annotators selected glosses at each
confidence bin. At low confidence, we see very minor increases on the order of
a single program being corrected. At high confidences, annotators generally
have only one or two options, and are able to choose the correct one,
resulting in similar performance to nucleus decoding. However, at medium
confidences, annotators often choose the wrong gloss, leading to lower
performance. Qualitatively, these incorrect choices are primarily driven by
glosses that appear to be paraphrases of each other, but in fact correspond to
subtly different Lisp programs. Taken together, these results suggest that
accepting and rejecting glosses is more promising than choosing them.

Figure 7: Selection experiment. Annotators sometimes select accurate glosses,
but often have trouble deciding between seemingly invariant glosses, lowering
performance especially at medium confidences.


# Area (or entropy) products for Newman-Unti-Tamburino class of black holes

Parthapratim<EMAIL_ADDRESS>
Department of Physics
Hiralal Mazumdar Memorial College For Women
Dakshineswar, Kolkata-700035, India

###### Abstract

We compute area (or entropy) product formula for Newman-Unti-Tamburino (NUT)
class of black holes. Specifically, we derive the area product of outer
horizon and inner horizon (${\mathcal{H}}^{\pm}$) for Taub-NUT, Euclidean
Taub-NUT black hole, Reissner-Nordström–Taub-NUT black hole, Kerr-Taub-NUT
black hole and Kerr-Newman-Taub-NUT black hole under the formalism developed
very recently by Wu et al. [1] [PRD 100, 101501(R) (2019)]. The formalism is
that a generic four dimensional Taub-NUT spacetime should be described
completely in terms of three or four different types of thermodynamic hairs.
They are defined as the Komar mass ($M=m$), the angular momentum
($J_{n}=m\,n$), the gravitomagnetic charge ($N=n$), the dual (magnetic) mass
$(\tilde{M}=n)$. After incorporating this formalism, we show that the area (or
entropy) product of both the horizons for NUT class of black holes are _mass-
independent_. Consequently, the area product of ${\mathcal{H}}^{\pm}$ for
these black holes are _universal_. Which was previously known in the
literature that the area product of said black holes are _mass-dependent_.
Finally, we can say that this universality is solely due to the presence of
_new conserved charges $J_{N}=M\,N$_ which is closely analogue to the Kerr
like angular momentum $J=a\,M$.

##  1 Introduction

Probably, black hole (BH) is most intriguing as well as thermal objects [2] in
the universe. The thermal object by virtue of it has both temperature and
entropy [3]. This entropy is again proportional to the area of the event (or
outer) horizon (EH) and Cauchy (or inner) horizon (CH) i.e. ${\cal
S}_{\pm}=\frac{{\cal A}_{\pm}}{4}$, where, ${\cal S}_{\pm}$ is the Bekenstein-
Hawking entropy (in units in which $G=\hbar=c=k_{B}=1$) and ${\cal A}_{\pm}$
is area of both the horizons. Analogously, the temperature is proportional to
the surface gravity of EH (${\cal H}^{+}$) and CH (${\cal H}^{-}$) i.e.
$T_{\pm}=\frac{\kappa_{\pm}}{2\pi}$, where $T_{\pm}$ is Hawking temperature
evaluated on ${\cal H}^{\pm}$ and $\kappa_{\pm}$ is surface gravity of the BH
evaluated on ${\cal H}^{\pm}$.

Recently, it has been argued that a generic four dimensional Taub-NUT BH
should be interpreted as in terms of three or four different types of
thermodynamic hairs [1]. They are defined as the Komar mass ($M=m$), the
angular momentum ($J_{n}=m\,n$), the gravitomagnetic charge ($N=n$), the dual
(magnetic) mass $(\tilde{M}=n)$. The authors also derived consistent
thermodynamic first law and Bekenstein-Smarr formula, Christodoulou-Ruffini
mass formula and other thermodynamic properties. More appropriately, they
solved long standing various problems of Taub-NUT (both spherically symmetric
and axisymmetric) geometry.

In the Refs. [4, 5, 6], it has been demonstrated that the area (or entropy)
products for NUT class of BHs (Taub-NUT, Kerr-Taub-NUT and Kerr-Newman-Taub-
NUT) have “mass-dependent” feature. This indicates that the area product of
${\cal H}^{\pm}$ has no nice universal feature nor any quantization feature.
However in this work, we will prove that the area product of ${\cal H}^{\pm}$
for NUT class of BHs both in spherically symmetric cases and axisymmetric
cases are indeed _mass-independent_. Therefore we can argue that the area
product of ${\cal H}^{\pm}$ is _“universal”_ in nature. They depends only on
the quantized charges and quantized angular momentum. Moreover, we argue that
this universality occurs due to inclusion of _spinning conserved charges
$J_{n}$_. Alternatively, we can demand that the universality takes place due
to multihair attributes of the NUT parameter (both rotation-like and
electromagnetic charge like feature) or

$\frac{J_{n}}{m}=n\equiv N=\frac{J_{N}}{M}$

.

Area products of multi-horizon BHs are nowadays a fascinating topic of
research for both general relativity community and string theory community due
to the seminal work of Ansorg [7] and Cvetic [8] [See also [9, 10, 11, 12, 13,
14, 15, 16, 17, 18] ]. The main focus therein is that the area (or entropy)
product of both the horizons namely the EH (${\cal H}^{+}$) and the CH (${\cal
H}^{-}$). For instance, it is well-known that for Kerr-Newman class of BH
having EH and CH, the area product formula of ${\cal H}^{\pm}$ becomes

$\displaystyle{\cal A}_{+}{\cal A}_{-}$ $\displaystyle=$
$\displaystyle(8\pi)^{2}\left[J^{2}+\frac{Q^{4}}{4}\right]~{}.$ (1)

The new physics here is that due to stationarity there is an absence of matter
between the EH and CH [10].

What is the origin of Bekenstein-Hawking [3, 2] entropy
$S_{\pm}=\frac{A_{\pm}}{4}$ at the microscopic level? This is a genuine topic
of research in quantum gravity. Here $A_{\pm}$ is the area of the BH EH and
CH. There has been a significant success in four and five dimensions for
supersymmetric Bogomol’ni-Prasad-Sommerfield (BPS) class of BHs [19]. Where it
is demonstrated that the microscopic degrees of freedom should be explained in
terms of 2D conformal-field-theory (CFT). If there exists BH entropy ($S_{+}$)
for outer horizon then it is also equally true that there exists BH entropy
($S_{-}$) for inner horizon. For maximally spinning spacetime, the microscopic
entropy could be found in [20]. However, the detailed structure of the
microscopic entropy of non-extremal BH still to be unknown to us. Some hopeful
progress have been reported in [21].

For BPS class of BHs, the area product formula is computed to be [8]

$\displaystyle{\cal A}_{+}{\cal A}_{-}$ $\displaystyle=$
$\displaystyle(8\pi)^{2}\left[\sqrt{N_{1}}+\sqrt{N_{2}}\right]\left[\sqrt{N_{1}}-\sqrt{N_{2}}\right]=(8\pi)^{2}N,\,\,N\in{\mathbb{N}},N_{1}\in{\mathbb{N}},N_{2}\in{\mathbb{N}}~{}.$
(2)

Note that here the area product is synonymous to entropy product. The integers
$N_{1}$ and $N_{2}$ should be interpreted as the excitation numbers of the
left and right moving modes of a weakly-coupled two-dimensional conformal
field theory. The parameters $N_{1}$ and $N_{2}$ are depended explicitly on
all the BH parameters. This implies that the product must be quantized [22,
23, 24, 25] and it is an integer quantity.

However in this work, we will be focus on area product formula for NUT
geometry. Particularly, we compute the product of all horizons areas for Taub-
NUT BH, Euclidean Taub-NUT BH, Reissner-Nordström–Taub-NUT BH, Kerr-Taub-NUT
BH and Kerr-Newman-Taub-NUT BH. Remarkably, our calculation shows that the
area products for both the physical horizons are _mass-independent
(universal)_. They depend only the various charges, angular momentum ($J$) and
new conserved charges ($J_{n}$). This universal behaviour arises mainly due to
the presence of new conserved charges $J_{N}$ which is closely analogue to the
angular momentum $J=a\,M$ of Kerr class of BHs.

In the next section, we will compute the area product formula for Taub-NUT BH
and show that the product is a universal quantity. Analogously, we derive the
area products for Euclidean-Taub-NUT BH in Sec. (3). In Sec. (4), we derive
the area product formula for charged Taub-NUT BH and prove that the product is
mass-independent. In Sec. (5) and Sec. (6), we compute the area product
formula for spinning Taub-NUT BH and charged spinning Taub-NUT BH
respectively. We prove that the product is indeed mass-independent for NUT
class of BHs. Finally, in Sec. (7), we have given the conclusions of the work.

##  2  Area product formula of Taub-NUT BH

First, we consider the Lorentzian Taub-NUT BH [26, 27, 28, 29]

$\displaystyle ds^{2}$ $\displaystyle=$ $\displaystyle-{\cal
B}(r)\,\left(dt+2n\,\cos\theta\,d\phi\right)^{2}+\frac{dr^{2}}{{\cal
B}(r)}+\left(r^{2}+n^{2}\right)\left(d\theta^{2}+\sin^{2}\theta
d\phi^{2}\right)~{},$ (3)

where the function ${\cal B}(r)$ is defined by

$\displaystyle{\cal B}(r)$ $\displaystyle=$
$\displaystyle\frac{1}{r^{2}+n^{2}}\left[r^{2}-n^{2}-2mr\right]$ (4)

Under new formalism [1] for NUT class of BHs, the global conserved charges are
defined as

$\displaystyle m$ $\displaystyle=$ $\displaystyle M~{}\mbox{(Komar mass)}$
$\displaystyle n$ $\displaystyle=$ $\displaystyle N~{}\mbox{(gravitomagnetic
charge)}$ $\displaystyle m\,n$ $\displaystyle=$ $\displaystyle
J_{n}~{}\mbox{(angular momentum)}~{}.$ (5)

Taking cognizance of Eq. (5) then the metric can be re-written as

$\displaystyle ds^{2}$ $\displaystyle=$ $\displaystyle-{\cal
B}(r)\,\left(dt+2N\,\cos\theta d\phi\right)^{2}+\frac{dr^{2}}{{\cal
B}(r)}+\left(r^{2}+N^{2}\right)\left(d\theta^{2}+\sin^{2}\theta
d\phi^{2}\right)~{},$ (6)

where the function ${\cal B}(r)$ is given by

$\displaystyle{\cal B}(r)$ $\displaystyle=$
$\displaystyle\frac{1}{r^{2}+N^{2}}\left(r^{2}-N^{2}-2Mr\right)$ (7)

The BH horizons are located at

$\displaystyle r_{\pm}=M\pm\sqrt{M^{2}+N^{2}}\,\,\mbox{and}\,\,r_{+}>r_{-}$
(8)

$r_{+}$ is called EH and $r_{-}$ is called CH. Thus the area of the BH is
computed as

$\displaystyle{\cal A}_{\pm}$ $\displaystyle=$
$\displaystyle\int^{2\pi}_{0}\int^{\pi}_{0}\sqrt{g_{\theta\theta}\,g_{\phi\phi}}{\mid}_{r=r_{\pm}}d\theta\,d\phi=8\pi\left[M^{2}+N^{2}\pm\sqrt{M^{4}+J_{n}^{2}}\right]~{}.$
(9)

Thus the area product of ${\cal H}^{\pm}$ is computed to be

$\displaystyle{\cal A}_{+}{\cal A}_{-}$ $\displaystyle=$
$\displaystyle(8\pi)^{2}\left(J_{n}^{2}+N^{4}\right)~{}.$ (10)

It implies that the product is indeed independent of the Komar mass.
Consequently, the product is _universal_. From the area product formula it
follows that this expression is truly universal solely due to

$J_{n}=m\,n=M\,N=J_{N}$

which is quite analogue to the Kerr like angular momentum. Alternatively, we
can say that the universality comes due to the multihair attributes of NUT
parameter. Except this new input the area product of NUT class of BHs never be
universal.

The BH temperature of both the horizons are computed via surface gravity on
the horizons defined by

$T_{\pm}=\frac{\kappa_{\pm}}{2\pi}=\frac{{\cal
B}^{\prime}(r_{\pm})}{4\pi\left(r_{\pm}^{2}+N^{2}\right)}=\frac{r_{\pm}-M}{2\pi\left(r_{\pm}^{2}+N^{2}\right)}=\frac{1}{4\pi\,r_{\pm}},$
(11)

It should be noted that besides the area of ${\cal H}^{\pm}$, the surface
gravity of ${\cal H}^{\pm}$ is also another important quantity of the
thermodynamics. Therefore it may play crucial role to understanding the
entropy at the microscopic level. Almost all the other thermodynamic
properties like first law of thermodynamics, Smarr formula etc. have discussed
in Ref. [1] so we have not repeated them here.

##  3  Area products for Euclidean Taub-NUT BH

The Euclidean [30, 31, 32, 33, 34] class of Taub-NUT BH should be obtain by
putting the condition $t=i\,\tau$ and $N=i\,N$ in Eq. (6). Then the function
${\cal B}(r)$ becomes

$\displaystyle{\cal B}(r)$ $\displaystyle=$
$\displaystyle\frac{1}{r^{2}-N^{2}}\left(r^{2}+N^{2}-2Mr\right)$ (12)

Therefore the BH horizons are situated at

$\displaystyle r_{\pm}=M\pm\sqrt{M^{2}-N^{2}}\,\,\mbox{and}\,\,r_{+}>r_{-}$
(13)

Where $r_{+}$ is called EH and $r_{-}$ is called CH. Thus the area of the BH
is computed as

$\displaystyle{\cal A}_{\pm}$ $\displaystyle=$
$\displaystyle\int^{2\pi}_{0}\int^{\pi}_{0}\sqrt{g_{\theta\theta}\,g_{\phi\phi}}{\mid}_{r=r_{\pm}}d\theta\,d\phi=8\pi\left[M^{2}-N^{2}\pm\sqrt{M^{4}-J_{N}^{2}}\right]~{}.$
(14)

Therefore the area product of ${\cal H}^{\pm}$ becomes

$\displaystyle{\cal A}_{+}{\cal A}_{-}$ $\displaystyle=$
$\displaystyle(8\pi)^{2}\left(N^{4}-J_{n}^{2}\right)~{}.$ (15)

Thus it implies that the product is mass-independent and consequently it is
universal.

The BH temperature is defined by

$T_{\pm}=\frac{\kappa_{\pm}}{2\pi}=\frac{{\cal
B}^{\prime}(r_{\pm})}{4\pi\left(r_{\pm}^{2}-N^{2}\right)}=\frac{r_{\pm}-M}{2\pi\left(r_{\pm}^{2}-N^{2}\right)}=\frac{1}{4\pi\,r_{\pm}},$
(16)

##  4  Area product formula of Reissner-Nordström–Taub-NUT BH

Now we can extend the area product formula to the charged case. Therefore the
function ${\cal B}(r)$ in this case would be

$\displaystyle{\cal B}(r)$ $\displaystyle=$
$\displaystyle\frac{1}{r^{2}+N^{2}}\left(r^{2}-N^{2}-2Mr+Q^{2}\right)$ (17)

where $Q$ is the purely electric charge. Now the BH horizons are located at

$\displaystyle
r_{\pm}=M\pm\sqrt{M^{2}-Q^{2}+N^{2}}\,\,\mbox{and}\,\,r_{+}>r_{-}$ (18)

Now the area of the BH is

$\displaystyle{\cal A}_{\pm}$ $\displaystyle=$
$\displaystyle\int^{2\pi}_{0}\int^{\pi}_{0}\sqrt{g_{\theta\theta}\,g_{\phi\phi}}{\mid}_{r=r_{\pm}}d\theta\,d\phi=4\pi\left[2\left(M^{2}+N^{2}\right)-Q^{2}\pm
2\sqrt{M^{4}-M^{2}Q^{2}+J_{N}^{2}}\right]~{}.$ (19)

Hence the area product of ${\cal H}^{\pm}$ is

$\displaystyle{\cal A}_{+}{\cal A}_{-}$ $\displaystyle=$
$\displaystyle(8\pi)^{2}\left[J_{n}^{2}+\left(\frac{Q^{2}}{2}-N^{2}\right)^{2}\right]~{}.$
(20)

Therefore the product of ${\cal H}^{\pm}$ is mass-independent as well as
universal.

##  5  Area product formula of Kerr-Taub-NUT BH

Now we turn to the spinning NUT BH. The metric for spinning Taub-NUT BH [35]
in Boyer-Lindquist like coordinates $(t,r,\theta,\phi)$ is

$\displaystyle ds^{2}$ $\displaystyle=$
$\displaystyle-\frac{\Delta}{\rho^{2}}\,\left[dt-
Pd\phi\right]^{2}+\frac{\sin^{2}\theta}{\rho^{2}}\,\left[(r^{2}+a^{2}+N^{2})\,d\phi-a\,dt\right]^{2}+\rho^{2}\,\left[\frac{dr^{2}}{\Delta}+d\theta^{2}\right]~{}.$
(21)

where

$\displaystyle a$ $\displaystyle\equiv$
$\displaystyle\frac{J}{M},\,\rho^{2}\equiv r^{2}+(N+a\,\cos\theta)^{2}$ (22)
$\displaystyle\Delta$ $\displaystyle\equiv$ $\displaystyle
r^{2}-2Mr+a^{2}-N^{2}$ (23) $\displaystyle P$ $\displaystyle\equiv$
$\displaystyle a\,\sin^{2}\theta-2N\,\cos\theta~{}.$ (24)

where the global conserved charges for this spacetime are the Komar mass $M$,
angular momentum $J=aM$ and gravitomagnetic charge or dual mass or NUT
parameter $N$.

The radii of the horizon is computed by the function

$\Delta|_{r=r_{\pm}}=0$

which implies that

$\displaystyle r_{\pm}$ $\displaystyle\equiv$ $\displaystyle
M\pm\sqrt{M^{2}-a^{2}+N^{2}}$ (25)

Now after introducing the new conserved charge $J_{N}=MN$, the area of both
the horizons (${\cal H}^{\pm}$) is

$\displaystyle{\cal A}_{\pm}$ $\displaystyle=$
$\displaystyle\int^{2\pi}_{0}\int^{\pi}_{0}\sqrt{g_{\theta\theta}\,g_{\phi\phi}}\,d\theta\,d\phi$
(26) $\displaystyle=$ $\displaystyle
8\pi\left[(M^{2}+N^{2})\pm\sqrt{M^{4}+J_{N}^{2}-J^{2}}\right]~{}.$ (27)

Now we can calculate the area product for Kerr-Taub-NUT BH as

$\displaystyle{\cal A}_{+}{\cal A}_{-}$ $\displaystyle=$
$\displaystyle(8\pi)^{2}\left(J^{2}+J_{N}^{2}+N^{4}\right)~{}.$ (28)

The area product is mass-independent as well as universal.

##  6  Area product formula of Kerr-Newman-Taub-NUT BH

Finally, we consider most general class of BH without introducing the
cosmological constant. The metric is of the form like Eq. (21) and the horizon
function [35] is defined to be

$\displaystyle\Delta$ $\displaystyle\equiv$ $\displaystyle
r^{2}-2Mr+a^{2}+Q^{2}-N^{2}~{}.$ (29)

Here the conserved charges are Komar mass $M$, angular momentum $J=aM$ and
gravitomagnetic charge or dual (magnetic) mass or NUT charge $N$. The horizons
are located at

$\displaystyle r_{\pm}$ $\displaystyle\equiv$ $\displaystyle
M\pm\sqrt{M^{2}-a^{2}-Q^{2}+N^{2}}$ (30)

Analogously, the BH horizon area of ${\cal H}^{\pm}$ is derived to be

$\displaystyle{\cal A}_{\pm}$ $\displaystyle=$
$\displaystyle\int^{2\pi}_{0}\int^{\pi}_{0}\sqrt{g_{\theta\theta}\,g_{\phi\phi}}d\theta
d\phi$ (31) $\displaystyle=$ $\displaystyle 4\pi\left[2(M^{2}+N^{2})-Q^{2}\pm
2\sqrt{M^{4}+J_{N}^{2}-J^{2}-M^{2}Q^{2}}\right]~{}.$ (32)

Finally, the product of area of both the horizons for most general class of
NUT BH without the cosmological constant

$\displaystyle{\cal A}_{+}{\cal A}_{-}$ $\displaystyle=$
$\displaystyle(8\pi)^{2}\left[J^{2}+J_{N}^{2}+\left(\frac{Q^{2}}{2}-N^{2}\right)^{2}\right]~{}.$
(33)

The product is universal as well as quantized.

##  7  Conclusions

It has recently been stated that a generic 4D Taub-NUT BH could be completely
described in terms of three or four different types of thermodynamic hairs.
They must be defined as the Komar mass ($M=m$), the angular momentum
($J_{n}=m\,n$), the gravitomagnetic charge ($N=n$), the dual (magnetic) mass
$(\tilde{M}=n)$. Under these circumstances, we demonstrated that the area (or
entropy) product of multi-horizons for NUT class of BH is _mass-independent_.
That means the mass-independent features of NUT class of BHs are interestingly
shown to be _universal_ which was earlier known as “mass-dependent”
characteristics in the literature.

We have explicitly checked here for Lorentzian Taub-NUT BH, Euclidean Taub-NUT
BH, Reissner-Nordström–Taub-NUT BH, Kerr-Taub-NUT BH and Kerr-Newman-Taub-NUT
BH. In each case, we showed that the area product of both physical horizons is
universal in nature after the inclusion of angular momentum like parameter
$J_{N}=M\,N$ with the generic parameters $M$ and $N$ as a global conserved
charges. This universal feature of area (or entropy) products that we have
derived for NUT class of BHs further provide any hints for potential
explanation of the microscopic structure of the BHs in four dimensions.

## References

  * [1] S. Q. Wu and D. Wu, “Thermodynamical hairs of the four-dimensional Taub-Newman-Unti-Tamburino spacetimes”  Phys. Rev. D 100, 101501(R) (2019) and references therein.
  * [2] J. M. Bardeen et al., “The Four laws of black hole mechanics”,  Commun. Math. Phys. 31, 161 (1973).
  * [3] J. D. Bekenstein, “Black holes and entropy”, Phys. Rev. D 7, 2333 (1973).
  * [4] P. Pradhan, “Thermodynamic product formula for a Taub-NUT black hole”,  JETP 122 ,113 (2016) and references therein.
  * [5] P. Pradhan, “Surface area products for Kerr-Taub-NUT space-time”,  Euro. Phys. Lett 115, 30003 (2016) and references therein.
  * [6] P. Pradhan, “Area product and mass formula for Kerr-Newman-Taub-NUT spacetime”, Mod. Phys. Lett. A  30, 1550170 (2015) and references therein .
  * [7] M. Ansorg and J. Hennig, “The Inner Cauchy horizon of axisymmetric and stationary black holes with surrounding matter in Einstein-Maxwell theory”, Phys. Rev. Lett. 102, 221102 (2009).
  * [8] M. Cvetic, G. W. Gibbons and C. N. Pope, “Universal Area Product Formulae for Rotating and Charged Black Holes in Four and Higher Dimensions”, Phys. Rev. Lett. 106, 121301 (2011).
  * [9] S. Detournay, “Inner Mechanics of 3d Black Holes ”, Phys. Rev. Lett.  109, 031101 (2012).
  * [10] M. Visser, “Area products for stationary black hole horizons”, Phys. Rev. D 88, 044014 (2013).
  * [11] A. Castro, M. J. Rodriguez, “Universal properties and the first law of black hole inner mechanics ”,  Phys. Rev. D 86, 024008 (2012).
  * [12] M. Cvetič et al., “Entropy-product rules for charged rotating black holes”, Phys. Rev. D 88, 044046 (2013).
  * [13] P. Pradhan, “Black Hole Interior Mass Formula”,  Euro. Phys. J. C 74, 2887 (2014).
  * [14] P. Pradhan, “Thermodynamic product formula for Hořava-Lifshitz black hole”, Phys. Lett. B 747, 64 (2015).
  * [15] D. N. Page and A. A. Shoom, “Universal area product for black holes: A heuristic argument”, Phys. Rev. D 92, 044039 (2015).
  * [16] P. Pradhan, “Entropy Product Formula for spinning BTZ Black Hole”,  JETP Lett.  102, 481 (2015).
  * [17] P. Pradhan, “Area (or entropy) product formula for a regular black hole”,  Gen. Rel. Grav.  48, 19 (2016).
  * [18] P. Pradhan, “Area functional relation for 5D-Gauss-Bonnet-AdS black hole”, Gen. Rel. Grav.  48, 116 (2016).
  * [19] A. Strominger and C. Vafa, “Microscopic origin of the Bekenstein-Hawking entropy”, Phys. Lett. B. 379, 99 (1996).
  * [20] M. Guica et al., “The Kerr/CFT correspondence ”, Phys. Rev. D 80 124008 (2009).
  * [21] A. Castro et al., “Hidden conformal symmetry of the Kerr black hole ”, Phys. Rev. D 82, 024008 (2010).
  * [22] F. Larsen, “A String model of black hole microstates ”, Phys. Rev. D 56, 1005 (1997).
  * [23] M. Cvetič and D. Youm, “Entropy of nonextreme charged rotating black holes in string theory”,  Phys. Rev. D 54, 2612 (1996).
  * [24] M. Cvetič and F. Larsen, “Greybody factors for rotating black holes in four dimensions”,  Nucl. Phys. B 506, 107 (1997).
  * [25] M. Cvetič and F. Larsen, “General rotating black holes in string theory: Greybody factors and event horizons”,  Phys. Rev. D 56, 4994 (1997).
  * [26] C. W. Misner, “The Flatter regions of Newman, Unti and Tamburino’s generalized Schwarzschild space”, J. Math. Phys. 4, 924 (1963).
  * [27] E. Newman, L. Tamburino, T. Unti, “Empty‐Space Generalization of the Schwarzschild Metric”,  J. Math. Phys. 4, 915 (1963).
  * [28] J. G. Miller, M. D. Kruskal, B. B. Godfrey, “Taub-NUT (Newman, Unti, Tamburino) Metric and Incompatible Extensions”,  Phys. Rev. D4, 2945 (1971).
  * [29] G. Holzegel, “A Note on the instability of Lorentzian Taub-NUT-space”,  Class. Quant. Grav. 23, 3951 (2006).
  * [30] C. J. Hunter, “Action of instantons with a nut charge”, Phys. Rev. D 59, 024009 (1998).
  * [31] S. W. Hawking and C. J. Hunter, “Gravitational entropy and global structure”, Phys. Rev. D 59, 044025 (1999).
  * [32] S. W. Hawking, C. J. Hunter, D. N. Page, “NUT charge, anti-de Sitter space, and entropy”,  Phys. Rev. D 59, 044033 (1999).
  * [33] A. Chamblin et al., “Large N phases, gravitational instantons and the nuts and bolts of AdS holography”,  Phys. Rev. D 59, 064010 (1999).
  * [34] R. Emparan et al. “Surface terms as counterterms in the AdS/CFT correspondence”,  Phys. Rev. D 60, 104001 (1999).
  * [35] J. G. Miller, “Global analysis of the Kerr-Taub-NUT metric”,  J. Math. Phys. 14, 486 (1973).