# Thermodynamic witness of quantum probing

H.Dong  Institute of Theoretical Physics,Chinese Academy of Science, Beijing
100080, China  X.F. Liu  Department of Mathematics, Peking University, Beijing
100871, China  C.P. Sun<EMAIL_ADDRESS>http://power.itp.ac.cn/~suncp/
Institute of Theoretical Physics,Chinese Academy of Science, Beijing 100080,
China

###### Abstract

The thermodynamic influence of quantum probing on an object is studied. Here,
quantum probing is understood as a pre-measurement based on a non-demolition
interaction, which records some information of the probed object, but does not
change its energy state when both the probing apparatus and the probed object
are isolated from the environment. It is argued that when the probing
apparatus and the probed object are immersed in a same equilibrium
environment, the probing can affect the effective temperature of the object or
induce a quantum isothermal process for the object to transfer its energy.
This thermodynamic feature can be regarded as a witness of the existence of
quantum probing even if the quantum probing would not disturb the object if
the environment were not present.

###### pacs:

03.65.Ta, 03.65.Yz , 05.30.Ch

Introduction\- The Landauer’s principle that the erasure of one bit of
information requires a minimum heat generation of $k_{B}T\ln 2$, which is
based on the second law of thermodynamics landaure ; book_maxwelldeamon2 ,
underlies the thermodynamics of information processing. This principle
eventually resolves the Maxwell’s demon paradox: why a demon can assist a
binary thermal medium to do extra work book_maxwelldeamon2 . For the science
and technology of quantum information, the Landauer’s principle is undoubtedly
crucial since it gives a physical limitation on the spatial-time scales of
logical devices on chips.

However, in the arguments for and against the Landauer’s principle, the
conventional question in the thermodynamics of information processing that
whether the measurement process requires a cost of heat generation has never
been convincingly answered. Surely, earlier authors had touched on this
problem, but it has not been clarified yet because the measurement process is
not defined properly. Particularly, in the quantum approach of measurement,
people can not be unanimous for some fundamental problems, such as whether or
not there exist wave function collapse zurek . In fact, to clarify the
situation, we need to answer the following subtle questions in an unambiguous
way: when can we say a system is performing a measurement on another system
and what kind of measurement can dissipate information?

In this paper, we refer quantum probing to a pre-measurement book_measurement
; sun  based on the non-demolition coupling of the probed system $S$ to the
apparatus $A$, which is only a unitary process to produce entanglement between
$S$ and $A$, and does not concerns the subtle, seemingly philosophical
arguments, such as wave function collapses. Generally speaking, when the
energy state of the measured system $S$ is not influenced by the coupling to
the measuring apparatus $A$, but $A$ can record some information of the system
$S$, the measurement performed by $A$ on $S$ is called a pre-measurement. The
so called non-demolition (pre-) measurement book_measurement  is an ideal
measurement under some circumstances. In the case of non-demolition
measurement, the system-apparatus coupling $V_{SA}$ commutes with the
Hamiltonian $H_{S}$ of the measured system, but does not commute with the
Hamiltonian $H_{A}$ of the measuring apparatus. When studying non-demolition
measurement it is usually assumed that both $A$ and $S$ are isolated from an
environment. In this paper, we will investigate the effect of environment on
the measured system in a non-demolition measurement. In the following
discussion, we will use the name quantum probing or just probing when
regarding such non-demolition measurement. In the presence of environment, the
practical thermalization efftem  of the total system $A+S$ will have
thermodynamic effects on $S$. These effects can be regarded as the
thermodynamic witness of quantum probing.

When the existence of an environment $E$ is considered, quantum probing is
understood in two steps(illustrated in Fig. 1) :

  * •

Due to the extreme-weakness of the coupling between $S+A$ and $E$, it can be
neglected within the dephasing time $\tau_{2}$ of the system, the dephasing
being the result of the interaction between $S$ and $A$. In the end of this
step, the initial factorized state of $S+A$ becomes a state assuming the form
of an ideal Schmidt decomposition $\sum
c_{n}\left|n\right\rangle\otimes\left|D_{n}\right\rangle$sun , with
$\left\langle D_{m}\right|\left.D_{n}\right\rangle=\delta_{mn}$.

  * •

If the probing apparatus continues to probe the state of the system $S$, the
non-demolition coupling should hold for a longer time, and the communication
with the environment will result in the thermalization of the total system
$S+A$. In this step, the information of the initial state should be erased
totally, but the correlation between $A$ and $S$ needs to exist to leave the
witness of quantum probing.

As shown as follows, it is the comment environment of the probed system and
the probing apparatus that selects a special set of ideal entanglement states
of $S+A$ to be thermalized, so that these witnesses of quantum probing are of
thermodynamics, and thus observable in the classical or macroscopic level.

Figure 1: Time scale of the non-demolition pre-measurement and thermalization.
(1) In the time interval $\left(0,\tau_{2}\right)$, the apparatus makes the
pre-measurement: $\sum
c_{n}\left|n\right\rangle\otimes\left|D\right\rangle\rightarrow\sum
c_{n}\left|n\right\rangle\otimes\left|D_{n}\right\rangle$ and the environment
does not play the role, since its coupling to $S+A$ is weaker than that
between $S$ and $A$. (2) In the time interval
$\left(\tau_{2},\tau_{1}\right)$, the effect of thermalization due to the
environment becomes prominent. The total system $S+A$ is finally thermalized
in the canonical state $\rho_{\mathrm{can}}\left(T\right)$ with temperature
$T$.

Universal Setup for quantum probing apparatus\- Let $A$ be a general apparatus
weakly coupling to the system $S$ to be probed. We require that the energy
spectrum of $A$ be denser than that of $S$. The following heuristic argument
may help to justify this requirement: to measure the spatial scale of an
object, the ruler should have a much finer graduation than the size of this
object.

To be precise, let $H_{A}=\sum_{k}\epsilon_{k}\left|k\right\rangle\left\langle
k\right|$ be the spectrum decomposition of the Hamiltonian of $A$ and
$H_{S}=\sum_{n}E_{n}\left|n\right\rangle\left\langle n\right|$ the spectrum
decomposition of the Hamiltonian of $S$, the requirement then can be expressed
as $\min\\{|E_{n}-E_{n+M}|\\}>>\max\\{|\epsilon_{k}-\epsilon_{k+1}|\\}$. Here,
$\left|k\right\rangle$ is the eigenvector of $H_{A}$ corresponding to the
eigenvalue $\epsilon_{k}$, and $\left|n\right\rangle$ is the eigenvector of
$H_{S}$ corresponding to the eigenvalue $E_{n}$ Let $V_{AS}$ be a weak
coupling between $S$ and $A$. The weakness of $V_{AS}$ means that its effect
on the dynamics of the total system $S+A$ can be well studied by the
perturbation method frohlich .

To investigate the behavior of the total system $S+A$ immersed in an
environment, we consider the partition function

$Z=\mathrm{Tr}\left(e^{-\beta H}\right)=\mathrm{Tr}\left(W^{\dagger}e^{-\beta
H}W\right)$

Here, as a trick, we have introduced a unitary transformation
$W=\exp\left(-S\right)$, defined by an anti-Hermitian operator $S$, which is a
perturbation quantity of the same order as $V_{AS}$. If

$V_{AS}+\left[H_{A}+H_{S},S\right]=0,$

then the partition function can be approximated as
$Z=\mathrm{Tr}\exp\left(-\beta H_{\mathrm{eff}}\right)$ where the effective
Hamiltonian

$H_{\mathrm{eff}}=H_{A}+H_{S}+\frac{1}{2}\left[V_{AS},S\right],$ (1)

is just the Frohlich-Nakajima Hamiltonian in solid state physics frohlich .
Here, we re-derive it to justify its applicability in thermodynamics.

Since the minimal energy level spacing of the system is much larger than the
energy level spacing of the apparatus, the effective interaction
$V_{\mathrm{eff}}$ can be obtained as
$V_{\mathrm{eff}}=\sum_{n}\mathcal{H}\left(n\right)\left|n\right\rangle\left\langle
n\right|$ where

$\mathcal{H}\left(n\right)=\sum_{k}G(n)\left|k\right\rangle\left\langle
k^{\prime}\right|$ (2)

is a branched effective Hamiltonian of the apparatus corresponding to the
situation that the system is prepared in the state $\left|n\right\rangle$ with
the coupling

$G(n)=\frac{\left\langle
nk\right|V_{AS}\left|nk^{\prime}\right\rangle}{\left(\epsilon_{k}-\epsilon_{k^{\prime}}\right)}.$
(3)

The above obtained effective interaction $V_{\mathrm{eff}}$ satisfies
$\left[H_{S},V_{\mathrm{eff}}\right]=0$. Thus the total effective Hamiltonian
$H_{\mathrm{eff}}=H_{S}+H_{A}+V_{\mathrm{eff}}$ describes a non-demolition
measurement without the presence of an environment. In this case, the
factorized initial state
$\left|\varphi\left(0\right)\right\rangle=\sum_{n}c_{n}\left|n\right\rangle\otimes\left|D\right\rangle$
will evolve into
$\left|\varphi\left(t\right)\right\rangle=\sum_{n}c_{n}\left|n\right\rangle\otimes\left|D_{n}\left(t\right)\right\rangle$
where $\left|D_{n}\left(t\right)\right\rangle$ takes the form
$\left|D_{n}\left(t\right)\right\rangle=\exp\left[-iG\left(n\right)t\right]\left|D\right\rangle.$
If $\left\\{\left|D_{n}\left(t\right)\right\rangle\right\\}$ becomes an
orthogonal set when $t$ approaches infinity, then the time evolution
$\left|\varphi\left(t\right)\right\rangle$ represents a process of ideal pre-
measurement.

Thermodynamic effects of measurements\- Now we study the change in the
thermodynamic features of the system caused by the apparatus. We assume that
both the probed system and the measuring apparatus are immersed in the same
thermal bath with temperature $T$ or inverse temperature $\beta=1/k_{B}T$.
After or during the measuring process, the total system $S+A$ will reach the
state with the same temperature $T$, if the total system is non-degenerate.
Then we can calculate the reduced density matrix
$\rho_{S}=\mathrm{Tr_{A}}\left[\exp\left(-\beta\left(H_{S}+H_{A}+V_{\mathrm{eff}}\right)\right)\right]$,
obtaining

$\rho_{S}=\frac{1}{Z_{S}^{\prime}}\sum_{n}e^{-\beta
E_{n}}\xi\left(n\right)\left|n\right\rangle\left\langle n\right|,$ (4)

where $Z_{S}^{\prime}=\sum_{n}\exp\left(-\beta E_{n}\right)\xi\left(n\right)$
and

$\xi\left(n\right)=\mathrm{Tr_{A}}\left\\{\left\langle
n\right|\exp\left[-\beta\left(H_{A}+V_{\mathrm{eff}}\right)\right]\left|n\right\rangle\right\\}$
(5)

is a formal factor depending on the system state $\left|n\right\rangle$ and
vanishing trivially when no coupling exists.

Here, we consider a manipulation process. (i) Initially, no probing apparatus
is coupled to the system, which is in equilibrium with the heat bath with
temperature $T$. (ii) Then, the apparatus begin to probe the system at time
$t=0$. As the evolution described in Fig. 2, the total system $S+A$ reaches
the state with the same temperature $T$. We observe that the thermodynamic
effect of measurement implied in Eq. (6) allows two interpretations. These two
interpretations are illustrated in Fig. 2 as those in Ref. q1 ; quanPRE2007 .

Figure 2: Two interpretations. (i) Isometric process ($A\rightarrow C$, the
solid line): the inverse temperature $\beta$ is fixed in the process. (ii)
Isothermal process ($A\rightarrow B$, the dotted line): The level spacing
$\Delta$ is fixed in the process.

The first interpretation goes as follows. The change from the close thermal
state of the system $\rho_{Sc}=\exp\left(-\beta H_{S}\right)/Z_{S}$ to the
modified thermal state $\rho_{S}$ is understood as a quantum isometric
process, represented by the solid line between the points $A$ and $C$ in Fig.
2. In this process, the temperature keeps unchanged, but the energy level
spacings are alternated by the interaction with the apparatus. Accordingly, we
define $\xi\left(n\right)=\exp\left(-\beta\Delta E_{n}\right)$, then
$\rho_{S}$ can be written in the form with clear physical meaning:

$\rho_{S}=\frac{1}{Z^{\prime}_{S}}\sum_{n}e^{-\beta\left(E_{n}+\Delta
E_{n}\right)}\left|n\right\rangle\left\langle n\right|.$ (6)

In the present case, it is rather natural to regard the inner energy change
$\Delta
U=\mathrm{Tr_{S}}\left[H_{\mathrm{S}}\left(\rho_{S}-\rho_{Sc}\right)\right]$
as a witness of the thermodynamic role of measurement.

In the second interpretation the change from $\rho_{Sc}$ to $\rho_{S}$ is
understood as a quantum isothermal process, represented by the dotted line
between the points $A$ and $B$ in Fig. 2. In this process, the energy level
spacings are fixed. To justify our using the term “isothermal process” here,
at least to some extent, we define the effective temperature efftem  via
$\beta\left(n\right)=\ln(P_{n}/P_{n+1})/\Delta_{n}$ or

$\beta\left(n\right)=\beta+\frac{1}{\Delta_{n}}\ln\frac{\xi\left(n\right)}{\xi\left(n+1\right)},$
(7)

where $\Delta_{n}=E_{n+1}-E_{n}$ is the $n-th$ energy level spacing. We notice
that generally this generalized temperature cannot be regarded as an effective
temperature since it depends on the energy levels of the system. But if
$\beta\left(n+1\right)-\beta\left(n\right)=0,$ i.e.,

$\left[\frac{\xi\left(n+1\right)}{\xi\left(n+2\right)}\right]^{\Delta_{n}}=\left[\frac{\xi\left(n\right)}{\xi\left(n+1\right)}\right]^{\Delta_{n+1}},$
(8)

then $\beta\left(n\right)$ becomes a well defined thermodynamic parameter
independent of $n$, which we denote by $\beta_{\mathrm{eff}}$. In this case,
the above defined generalized temperature allows the physical interpretation
of effective temperature, and we have
$\rho_{S}=\sum_{n}\exp\left(-\beta_{\mathrm{eff}}E_{n}\right)\left|n\right\rangle\left\langle
n\right|/Z_{S}^{\prime}$, and the inner energy change

$\Delta U=\sum_{n}E_{n}\left[e^{-\beta_{\mathrm{eff}}E_{n}}-e^{-\beta
E_{n}}\right]$ (9)

reflects the thermalization effect.

The quantum role of measurement and renormalization\- As an explicit
illustration, we model the apparatus with weak couplings to the probed system
as a collection of harmonic oscillators. According to the results of Ref.
leggett  and Ref. suncp98pra  (where he arguments are carried out for the bath
modeling, but can work well for our setup for the apparatus), the coupling of
the system to the apparatus is linear with respect to the coordinates of the
bath harmonic oscillators. Let $b_{j}^{\dagger}\left(b_{j}\right)$ be the
creation(annihilation) operator of the $j-th$ mode of the bath with eigen-
frequency $\omega_{j}$, and $\lambda_{n}g_{j}$ the coupling coefficient of the
system state $\left|n\right\rangle$ to the $j-th$ mode. Then, the total
Hamiltonian is obtained as

$\displaystyle H$ $\displaystyle=$
$\displaystyle\sum_{n}E_{n}\left|n\right\rangle\left\langle
n\right|+\sum_{k}\omega_{k}b_{k}^{\dagger}b_{k}$ (10)
$\displaystyle+\sum_{n}\lambda_{n}\left|n\right\rangle\left\langle
n\right|\sum_{k}\left(g_{k}a_{k}^{\dagger}+\mathrm{h.c}\right).$

In this case, the above defined generalized inverse temperature reads

$\beta\left(n\right)=\beta\left(1-\frac{|\lambda_{n+1}|^{2}-|\lambda_{n}|^{2}}{E_{n+1}-E_{n}}\varepsilon\right),$
(11)

where

$\varepsilon=\sum_{k}\frac{|g_{k}|^{2}}{\omega_{k}}=\int\rho\left(\omega\right)\frac{|g_{k}|^{2}}{\omega_{k}}dk$
(12)

represents the self-energy of the apparatus, which causes the Lamb shift of
its coupled system. As pointed above, to define reasonably an effective
temperature for the system, the generalized inverse temperature
$\beta\left(n\right)$ should be independent of the energy level. Here is a
simple example satisfying this condition: the system is a harmonic oscillator
with the energy level $E_{n}=\left(n+1/2\right)\omega$ and the coupling
strength $\lambda_{n}=\sqrt{n}$. In this example, the well defined effective
inverse temperature is

$\beta_{\mathrm{eff}}=\beta(1-\sum_{k}\frac{|g_{k}|^{2}}{\omega\omega_{k}}),$
(13)

and the corresponding effective temperature
$T_{\mathrm{eff}}=\left(k_{B}\beta_{\mathrm{eff}}\right)^{-1}$ of the system
is higher than that of the environment $T=\left(k_{B}\beta\right)^{-1}$.

For a two-level system with the excited state $\left|e\right\rangle$ and the
ground state $\left|g\right\rangle$ and the energy level spacing $\Delta$, the
effective inverse temperature is also well defined. It reads

$\beta_{\mathrm{eff}}=\beta+\frac{1}{\Delta}\ln\frac{\xi_{g}}{\xi_{e}}.$ (14)

If the apparatus is a single mode cavity with frequency $\omega_{b}$ and the
two-level system is coupled to it by the dipole interaction with coupling
strength $g$, then in the large detuning case $\omega_{b}\gg\Delta$, the
formal factor $\xi_{e}$ and $\xi_{g}$ can be explicitly calculated as follows:

$\displaystyle\xi_{e}$ $\displaystyle=$
$\displaystyle\sum_{n}\exp\left\\{-\beta\left[\omega_{b}n+\frac{\left|g\right|^{2}}{\omega_{b}-\Delta}\left(n+1\right)\right]\right\\},$
$\displaystyle\xi_{g}$ $\displaystyle=$
$\displaystyle\sum_{n}\exp\left\\{-\beta\left[\omega_{b}n+\frac{\left|g\right|^{2}}{\omega_{b}-\Delta}n\right]\right\\}.$
(15)

Since $\xi_{g}>\xi_{e}$, we are led to the conclusion that the measurement
will decrease the temperature of the system by

$\Delta
T=\frac{\frac{1}{\Delta}\ln\frac{\xi_{g}}{\xi_{e}}}{\beta+\frac{1}{\Delta}\ln\frac{\xi_{g}}{\xi_{e}}.}T.$
(16)

Figure 3: Fidelity. (a) Fidelity $\mathcal{F}$ vs the level shift $\lambda$.
(b)Fidelity $\mathcal{F}$ vs the temperature shift $\Delta T$. Here, we take
$\omega=1$ and $\beta=1$.

Finally, to quantitatively evaluate the thermalized state resulting from the
interaction with the environment, let us check the fidelity fed  $\mathcal{F}$
of such a state to the ordinary canonical state
$\rho_{\mathrm{can}}=\exp\left(-\beta
H_{S}\right)/\mathrm{Tr}\left[\exp\left(-\beta H_{S}\right)\right]$. Generally
for the reduced density matrix in Eq. 4, the fidelity between the initial
state and the final state reads

$\mathcal{F}=\frac{\sum_{n}e^{-\beta
E_{n}}\sqrt{\xi\left(n\right)}}{\left(\sum_{n}e^{-\beta
E_{n}}\right)^{1/2}\left(\sum_{n}e^{-\beta
E_{n}}\xi\left(n\right)\right)^{1/2}}.$ (17)

When the probed system is also a harmonic oscillator discussed above, the
fidelity can be analytically obtained as

$\mathcal{F}=\frac{\sqrt{\sinh\frac{\beta\omega}{2}\sinh\frac{\beta\left(\omega-\lambda\right)}{2}}}{\sinh\frac{\omega-\lambda/2}{2}},$
(18)

where $\lambda=\sum_{k}|g_{k}|^{2}/\omega_{k}$ characterizes the shift of the
energy level of the harmonic oscillator, which reflects the effect due to the
coupling to the apparatus. In Fig. 3(a), the fidelity is plotted against the
level shift. As the coupling between the apparatus and the system is turn on,
the energy level for the system is effective
$E^{\prime}_{n}=n(\omega-\lambda)$ based on the first interpreting of the
reduced density matrix. The fidelity between the canonical thermal and the
reduced density matrix decreases as the coupling becomes strong. Therefore,
the system gradually deviates from canonical thermal state, leaving an
evidence of witness of the apparatus. For the second interpreting, we plot the
fidelity $\mathcal{F}$ as the function of the temperature shift $\Delta
T=1/\beta_{\mathrm{eff}}-1/\beta=\left[\beta(\omega/\lambda-1)\right]^{-1}$ in
Fig. 3(b).

Conclusion\- In summary, for the weak coupling case, we show from the
generalized approach of Frohlich-Nakajima transformation the universality of a
non-demolition Hamiltonian in connection with the probing process. Based on
this general non-demolition measurement, we investigate the probing effect on
the system when the total measurement happens in a reservoir. It is concluded
that the probing of the system can be witted through the change of the
effective temperature, even though there is no direct energy exchange between
the system and the detector. To characterize the change of state for the
system, we evaluate the fidelity of the modified canonical thermal state to
the original canonical state without being probed.

The work is supported by National Natural Science Foundation of China and the
National Fundamental Research Programs of China under Grant No. 10874091 and
No. 2006CB921205.

## References

  * (1) R. Landauer, IBM J. Res. Dev. 5, 183 (1961).
  * (2) H.S. Leff, & A.F. Rex(eds), Maxwell’s Demon 2: Entropy, Classical and Quantum Information, Computing. Institute of Physics. (2003).
  * (3) W. H. Zurek, Rev. Mod. Phys. 75,715 (2003).
  * (4) V. B. Braginsky and F. Y. Khalili, Quantum Measurement, Camebridge University Press, (1992).
  * (5) C.P. Sun, Phys. Rev. A 48, 898 (1993).
  * (6) H. Dong, S. Yang, X. F. Liu, and C. P. Sun, Phys. Rev. A 76, 044104 (2007); H. T. Quan, P. Zhang, and C. P. Sun, Phys. Rev. E 73, 036122 (2006).
  * (7) H. Fröhlich, Phys. Rev. 79, 845(1950); S. Nakajima, Adv. Phys. 4, 363(1953).
  * (8) H. T. Quan, P. Zhang, and C. P. Sun, Phys. Rev. E 72, 056110 (2005)
  * (9) H.T. Quan, Yu-xi Liu, C. P. Sun and F. Nori, Phys. Rev. E 76, 031105 (2007); H.T. Quan, Phys. Rev. E 79, 041129 (2009)
  * (10) A. O. Caldeira and A. J. Leggett, Ann. Phys. (N.Y.) 149, 374 (1983); A. J. Leggett, S. Chakravarty, A. T. Dosey, M. P. A. Fisher, and W. Zwerger, Rev. Mod. Phys. 59, 187 (1987).
  * (11) C.P. Sun, H. Zhan and X.F. Liu, Phys.Rev. A 58, 1810 (1998).
  * (12) P. Zanardi, H. T. Quan, X. Wang, C. P. Sun Phys. Rev. A 75, 032109 (2007) and references therein.


0 Research please specify

# Expectation Versus Reality:
The Failed Evaluation of a Mixed-Initiative Visualization System

Sunwoo Ha Ha is with Washington University in St. Louis<EMAIL_ADDRESS>Adam
Kern Kern is with MIT Lincoln Laboratory<EMAIL_ADDRESS>Melanie
Bancilhon Bancilhon is with Washington University in St. Louis:
<EMAIL_ADDRESS>and Alvitta Ottley Ottley is with Washington University
in St. Louis<EMAIL_ADDRESS>
###### Abstract

Our research aimed to present the design and evaluation of a mixed-initiative
system that aids the user in handling complex datasets and dense visualization
systems. We attempted to demonstrate this system with two trials of an online
between-groups, two-by-two study, measuring the effects of this mixed-
initiative system on user interactions and system usability. However, due to
flaws in the interface design and the expectations that we put on users, we
were unable to show that the adaptive system had an impact on user
interactions or system usability. In this paper, we discuss the unexpected
findings that we found from our “failed” experiments and examine how we can
learn from our failures to improve further research.

An overview of the system with visualization components labelled. Component 1
corresponds to the hover tooltip; component 2 corresponds to the information
card; component 3 corresponds to the label and filter.

Introduction

Evaluation in visualization studies show how visual aides can support
analysts, researchers, and all who must interact with data of all types in
understanding, synthesizing, and communicating that data. Historically, this
has been accomplished using controlled, in-person laboratory experiments
following practices from the psychology and broader HCI community. With calls
for more ecologically valid evaluations that examine actions from a more
representative user study population, there has been a relatively recent
tendency to collect data via crowdsourced platforms in place of an
undergraduate student population.

Researchers have successfully replicated pioneering studies using platforms
such as Amazon Mechanical Turk [3], and have produced more generalizable
findings that involve a more extensive and diverse population (see [6] for a
comprehensive survey of the prior work). However, there are a few caveats.
Mechanical Turk data is notoriously noisy, and researchers typically need to
use techniques such as attention checks, ground truths, and interaction
analysis for quality assurance [12].

In this paper, we report the expectations, findings, and lessons learned from
two “failed” Mechanical Turk experiments in which we aimed to evaluate a
mixed-initiative visualization system. The research agenda was motivated by
the need to improve data exploration for “small” but crowded data
visualizations. In particular, for high-density data, visualizing every data
point can lead to overplotting and information overload. Although, there are
several methods for improving visual clutter such as filtering and sampling
[15], these methods largely focus on “big” data. Information overload can
still occur in the small data settings, and naïve data reduction methods
applied to a small samples can remove elements that are important to the user
or exaggerate irrelevant points.

Our research aimed to manage overplotting by presenting a mixed-initiative
information visualization system. The design uses a hidden Markov model
algorithm, developed by Ottley et. al [18], to capture and predict user
attention. The visualization then responds by emphasizing the points that
likely fits the user’s interest. After conducting two trials of a large scale
crowd-sourced experiment to study the effect of the system described in this
paper, we found no evidence to support our hypotheses. Furthermore, our
analysis revealed two significant and unexpected findings:

  * •

Participants just wanted to hover. We used clicking to trigger the
visualization adaptation. However, the subjects in our online experiment
overwhelmingly interacted with the visualization by hovering.

  * •

Open-ended tasks were not appropriate for our online studies. A vast majority
of our online participants failed to provide the quality of feedback that we
expected and that were produced by our in-person pilot studies.

##  1 Mixed-Initiative System Overview

Our proposed mixed-initiative system was straightforward. We utilized the
algorithm presented in Ottley et al. [18] that captures and predicts users’
attention, to promote potential points of interest to the user’s interface.
Consider for example, a visualization with large amounts of occlusion where
datapoints overlap and partially or fully obscure each other. The user would
typically use zooming to handle this occlusion, but this solution has its
problems and limitations. When the visualization is completely zoomed out,
many of the datapoints are partially or entirely occluded, making large trends
hard to see. When the visualization is completely zoomed in, focal points can
be seen in their entirety, but such a close view can cause the analyst to lose
context [11]. Our visualization system responds by adaptively re-drawing
datapoints, bringing datapoints that the user is likely to be interested in to
the foreground, and sending “uninteresting” datapoints to the background (as
seen in Figure 1). In doing so, we hoped to create a system that allows for an
informative overview, encourages easier exploration, and reduces the need for
visual transformations.

Figure 1: The changing z-order of the pins as the user clicks, with a
particular interest in Mexican restaurants (coded in brown).

###  1.1 Interface Design

The system’s interface is displayed in Figure Expectation Versus Reality: The
Failed Evaluation of a Mixed-Initiative Visualization System. Our experiment
used two datasets, Toronto restaurants [21] and St. Louis crimes [16]. We use
pins the location of each data point on the map and color-coded then according
to categories. For the Toronto dataset, the color of indicates the main
cuisine for a given restaurant, and the color of the pins for the St. Louis
crime data indicates the type of crime. Details of points are provided in two
forms: (1) a tooltip that appear on hovered, and (2) “information cards” are
adding to the sidebar on click. The hover tooltip, as seen in segment 1 of
Figure Expectation Versus Reality: The Failed Evaluation of a Mixed-Initiative
Visualization System, shows just a few details, such as the restaurant’s name,
rating, and price, or the crime’s type and description. The information cards,
on the other hand, show all the attributes of a given datapoint, as seen in
segment 2 Figure Expectation Versus Reality: The Failed Evaluation of a Mixed-
Initiative Visualization System. The information cards also provide two more
modes of interaction. The first is “View”, which transitions the viewport to
center and zoom in on the datapoint corresponding to that information card and
temporarily enlarges the selected datapoint to bring the user’s attention to
the pin. The second is “Delete”, which removes the card from the sidebar. This
allows the user to keep a running list of datapoints of interest, and refer
back to them on the map on demand. A legend at the bottom right of the screen,
also serves as a filter, as seen in segment 3 of Figure Expectation Versus
Reality: The Failed Evaluation of a Mixed-Initiative Visualization System.

##  2 Testing Our System

We conducted two online experiments to determine the effect of a real-time
adaptive system on user interactions and system usability. For each study, we
recruited 200 participants via Amazon’s Mechanical Turk. Our experiment used
the two datasets introduced in the previous section and two conditions
(responsive and unresponsive) to study the effects of adaptive systems,
creating four groups:

  1. 1.

Responsive Toronto, then Unresponsive St. Louis

  2. 2.

Responsive St. Louis, then Unresponsive Toronto

  3. 3.

Unresponsive Toronto, then Responsive St. Louis

  4. 4.

Unresponsive St. Louis, then Responsive Toronto

If a user is on the responsive session of the experiment, the system adapts
the visualization to the user’s interests. In the unresponsive session, the
system will not change the visualization at all as the user interacts.

###  2.1 Task Design

Task design is critical to the success of an evaluation [17]. As a result, we
carefully considered the evaluation of our system and explored a variety of
task taxonomies (e.g., [2] and [22]). Ultimately, we wanted to focus and
exploratory data analysis. Specifically, we distinguish between bottom-up
exploration and top-down exploration. Bottom-up explorations “are driven in
reaction to the data” [1] or “may be triggered by salient visual cues” [14].
Top-down explorations, on the other hand, are based on a high-level goals or
hypothesis [5, 14]. We settled on an open-ended task because we wanted to
observe the users’ instinctual behavior. We conducted a series of in-person
pilot studies to determine the best phrasing for open-ended tasks prompts.

###  2.2 Procedure and Data Collection

At the start of the experiment, the participants were randomly assigned to one
of the four groups. Inspired by a laboratory study on latency by Liu and Heer
[14], the participants were first asked to “take some time to interact with
the dataset in front of [them], exploring the data and gathering insights”.
Once the participants felt that they were familiar with the dataset, they were
given the opportunity to write down as many (or as few) insights as they would
like. The participants were primed for interaction and insight gathering.
Before the experiment started, examples of insights were shown to users (e.g.,
“There are more kid-friendly coffee shops Downtown than there are Uptown”).
The goal of this priming was to introduce the participants to the
visualization and make explicit the idea of an “insight” without biasing the
user during either segment of the experiment. In this way, users were free to
perform exploratory data analysis without guidance or restriction, creating a
general-purpose task that can demonstrate the flexibility of the adaptive
system. Additionally, the reward structure of the experiment was designed to
encourage this insight-gathering: participants were awarded $\$1$ for
participating, and $\$0.50$ for every insight gathered. After recording all
the insights that they found, the participants completed the System Usability
Scale (SUS) [8], a widely-used, “robust and versatile tool for usability
professionals” [4], with an added comments section at the end of each survey
for general comments from users. Upon completion of the survey, the
participants continued onto the second condition/visualization which follows
the same procedure as the first.

As the participants interacted with the system, we captured every mouse
interaction: clicks, hovers, zooms, pans, views, deletes, and filter toggles.
To separate intentional from unintentional hovers, we only recorded hovers
with duration of at least 250 milliseconds. Additionally, we captured all
insights, interaction time, and survey responses.

##  3 What We Expected

Before publishing our study on Mechanical Turk, there were some expectations
that we put on our users.

  * •

One of the main ways that users would interact with our system was through
clicking on data points.

  * •

The users would be able to provide useful insights about the data.

  * •

The responsive system would elicit fewer zooms and more insights.

Overall, we hoped to see the mixed-initiative system have a significant
positive impact on the users’ interactions and subjective feedback.

##  4 The Reality of Study 1

In our first study, hovering over a data point revealed a tooltip as shown in
Figure Expectation Versus Reality: The Failed Evaluation of a Mixed-Initiative
Visualization System. Clicking on a point adds a card to the sidebar that
shows a historical log of click interactions.

Figure 2: A graph from our first experiment showing how often users clicked on
the visualization within a condition. As the number of clicks increases on the
$x$-axis, the percentage of users who clicked at least $x$ times decreases.
The dotted line represents the users whose click interactions were within the
top 20% of their cohort.

Participants spent, on average, 27 minutes exploring the two datasets and
recording their insights.

People wanted to hover. Unsurprisingly, we observed a large variance in the
number of interactions. However, on average, participants clicked on 9 out of
2915 points on the Toronto map and 7 out of 1951 points in the St. Louis map.
Figure 2 plots the percentage of users by the number click performed during a
session. Since, we used clicks as input to the machine learning algorithm that
tracks and predicts future interactions, our responsive conditions were
largely ineffective. We found no evidence that the adaptive system impacted
analysis, and there was no indication that participants even noticed the
adaptation. For further analysis, we narrowed the dataset to a group of users
whose interactions met the expectations of the experimental design (41
participants remained). Table 1 displays the average number of each type of
interaction, spread across datasets and conditions.

| Toronto | St. Louis
---|---|---
| Responsive | Unresponsive | Responsive | Unresponsive
Clicks | 7.9 | 12.6 | 11.8 | 11.0
Hovers | 52.0 | 41.0 | 46.4 | 38.6
Zooms | 57.4 | 25.4 | 45.8 | 35.6
Insights | 5.3 | 4.8 | 5.5 | 6.2
Table 1: Average values for interaction data frequency from the first trial.

Insights were shallow.  We coded insights and categorized them as deep and
shallow based on the amount of information that they contain. A shallow
insight is an observations attained only through minimal interactions. For
example,

  * ID893: “There is a large concentration of Chinese restaurants in one area.”

  * ID950: “A majority of the crimes committed are theft-related.”

A deep insight requires building knowledge of the dataset through
interactions. For example,

  * ID821: “There must be a Chinatown, or Asian-American populated area down Spadina Avenue between and including Dundas Street West, College Street, and Beverley Street.”

  * ID795: “There are incidences of larceny scattered all over the area. Most have to do with automobiles, but burglaries near the river tend to involve burglaries in businesses and buildings.”

Participants in our study entered a total of 779 insights, of which 747 were
shallow and 32 were classified as deep. We believed that the lack of
differences between the conditions were due to a flaw in the system design:
the information given on-hover was nearly identical to the information given
on-click. It is possible that this information parity gave no incentive for
users to click on data points, other than to keep their information in a
persistent state on the sidebar. This was particularly problematic in the
context of our experiment that relied on user clicks to trigger the
experimental condition.

##  5 The Reality of Study 2

Due to the flaws in the interface design and the overall null results, we
rerun the experiment and address the design missteps causing the discrepancy
between expectations and reality. We kept much of the experiment design from
Experiment 2, but made two minor changes:

  1. 1.

We removed the tooltip on hover. This meant that participants only saw details
on click and clicking also triggered in the system adaptation.

  2. 2.

We added additional guidance for formulating insights. In addition to the
examples detailed in Section 2.2, the instruction dissuaded shallow insights
by stating “Obvious insights like ‘There are a lot of coffee shops’ will not
be rewarded the bonus.”

In this second study, participants spent an average of 29 minutes exploring
the two datasets and recording their insights.

People did not want to click. We observed a moderate increase in the number of
clicks. On average, participants clicked on 40 out of 2915 points on the
Toronto map and 23 out of 1951 points on the St. Louis map. However, we
observed a pattern similar to Experiment 1. 110 out of 200 participants
clicked on fewer than 5 data points. An analysis of the remaining 90
participants revealed inconclusive results (see Table 2).

| Toronto | St. Louis
---|---|---
| Responsive | Unresponsive | Responsive | Unresponsive
Clicks | 36.5 | 63.0 | 31.0 | 21.2
Hovers | 50.8 | 67.6 | 48.1 | 41.2
Zooms | 58.5 | 54.6 | 38.6 | 49.2
Insights | 4.8 | 5.3 | 5.6 | 4.7
Table 2: Average values for interaction data frequency from the second trial.

Insights were shallow again. Similar to the click finding, we observed an
increase in the number of insights and moderate improvement in the quality of
insights. Participants in the second study entered a total of 1045 insights,
of which 996 were shallow and 79 were classified as deep.

##  6 Lessons Learned

The finding that people may not interact with visualization in the way that we
expect them to is not new. In the storytelling realm, Boy et al. [7] found the
participants in the web-based field experiments did not engage with
visualization as expected. Reports from New York Times suggest that users
prefer scrolling as a means of interaction, which as led them to reconsider
their investment in interactive visualization [20]. These are only anecdotal
results, however, along with the findings of the “failed” user studies in this
paper, they echo the sentiments of Lam [13] who encourages designers to weigh
the cost against potential gains of interaction.

Many researchers believe the “the purpose of visualization is insight” [9]. In
our studies, we opted for open ended tasks and captured insights in addition
to quantitative measures. Although there is prior work that define [10] and
characterize [19] insights, insight-based evaluative methods, especially for
online studies, are not clear. Many of the existing studies (e.g., [15] and
[19]) captured insights in laboratory settings. In addition, the open-ended
nature of our tasks made it difficult to the filter participants who were
clicking through to get paid. This potentially highlights an important
limitation of online studies and provides suggestive evidence that open-ended
tasks may not be appropriate for this experiment setting.

Not all of our findings were negative. The fact that users found the
visualization well-designed, yet were overwhelmed by the size and density of
the datasets, indicates that the chosen datasets worked well to induce a need
for an assistive or mixed-initiative agent to help users make sense of the
data. When designing this adaptive system, we were nervous that constantly
updating the visualization would be disorienting to the user, especially given
failed examples of adaptive systems like the notorious Microsoft Office
Clippy. However, the usability results for both responsive and non-responsive
conditions were overwhelmingly positive and we saw no impact on usability for
the responsive conditions. Overall, these lessons learned are motivating and
we believe that we can move forward to create and evaluate a more rigorous and
robust mixed-initiative system that actively supports the user during
exploration.

##  7 Future Directions

The failures we discussed opens up next directions for our adaptive system.

  1. 1.

In both experiments, we saw that a majority of the users liked to hover to
interact and gain insights. Like clicks, hovers could provide us a better
understanding of the users’ interests in real-time. A possible solution to
improve the system would be to incorporate hovers along with clicks to the
algorithm presented in Ottley et. al [18].

  2. 2.

Again, in both experiments, we had difficulty in obtaining insights that were
deep and in good quality. It would be interesting to see if a mix of closed
and open-ended tasks will help increase the quality of insights from
Mechanical Turk users. The closed-ended tasks would be asked first to get the
users familiar with interacting with the system. Then, the users would be free
to explore and complete the open-ended tasks.

##  8 Concluding Remarks

We designed a mixed-initiative system and attempted to investigate the effect
of the system on user interaction and system usability. It is tempting to say
that results we found supports the alternative hypothesis and conclude that
“the adaptive system did nothing”. However, it is more accurate to say that we
do not have enough evidence to reject the null hypothesis. Why is this a
crucial distinction? Because it does not destroy any hope of an effective
mixed-initiative system. It is important to note that these results should not
dissuade researchers from further work on mixed-initiative systems like the
one we designed. Although we were unsuccessful in achieving the data we
expected, we found significant and unexpected findings from the users. When
developing these systems, it is easy to assume that the users will interact
the way that we want them to. Since we made an assumption that users would
mainly show interest by clicking on the points, our system was unable to aide
the users when their interactions did not meet our expectations. We have
learned from our failures and hope that the VIS community can also learn from
them too.

###### Acknowledgements.

This work was supported in part by the National Science Foundation under Grant
No. 1755734.

## References

  * [1] S. Alspaugh, N. Zokaei, A. Liu, C. Jin, and M. A. Hearst.  Futzing and moseying: Interviews with professional data analysts on exploration practices.  IEEE Transactions on Visualization and Computer Graphics, 25(1):22–31, 2018. doi: 10 . 1109/TVCG . 2018 . 2865040
  * [2] R. Amar, J. Eagan, and J. Stasko.  Low-level components of analytic activity in information visualization.  IEEE Symposium on Information Visualization, p. 7, 2005.
  * [3] Amazon.  Mechanical Turk, 2020.
  * [4] A. Bangor, P. T. Kortum, and J. T. Miller.  An Empirical Evaluation of the System Usability Scale.  International Journal of Human–Computer Interaction, 24(6):574–594, 2008. doi: 10 . 1080/10447310802205776
  * [5] L. Battle and J. Heer.  Characterizing exploratory visual analysis: A literature review and evaluation of analytic provenance in tableau.  Computer Graphics Forum, 38(3):145–159, 2019. doi: 10 . 1111/cgf . 13678
  * [6] R. Borgo, L. Micallef, B. Bach, F. McGee, and B. Lee.  Information visualization evaluation using crowdsourcing.  In Computer Graphics Forum, vol. 37, pp. 573–595. Wiley Online Library, 2018.
  * [7] J. Boy, F. Detienne, and J.-D. Fekete.  Storytelling in information visualizations: Does it engage users to explore data?  In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1449–1458, 2015.
  * [8] J. Brooke.  SUS - A quick and dirty usability scale.  Usability evaluation in industry, 189(194):4–7, 1996.
  * [9] M. Card.  Readings in information visualization: using vision to think.  Morgan Kaufmann, 1999.
  * [10] R. Chang, C. Ziemkiewicz, T. M. Green, and W. Ribarsky.  Defining insight for visual analytics.  IEEE Computer Graphics and Applications, 29(2):14–17, 2009.
  * [11] M. R. Jakobsen and K. Hornbæk.  Interactive visualizations on large and small displays: The interrelation of display size, information space, and scale.  IEEE Transactions on Visualization and Computer Graphics, 19(12):2336–2345, 2013. doi: 10 . 1109/TVCG . 2013 . 170
  * [12] R. Kosara and C. Ziemkiewicz.  Do mechanical turks dream of square pie charts?  In Proceedings of the 3rd BELIV’10 Workshop: Beyond time and errors: Novel evaluation methods for information visualization, pp. 63–70, 2010\.
  * [13] H. Lam.  A framework of interaction costs in information visualization.  IEEE transactions on visualization and computer graphics, 14(6):1149–1156, 2008.
  * [14] Z. Liu and J. Heer.  The effects of interactive latency on exploratory visual analysis.  IEEE Transactions on Visualization and Computer Graphics, 20(12):2122–2131, 2014. doi: 10 . 1109/TVCG . 2014 . 2346452
  * [15] Z. Liu, B. Jiang, and J. Heer.  imMens: Real-time visual querying of big data.  Computer Graphics Forum, 32(3):421–430, 2013. doi: 10 . 1111/cgf . 12129
  * [16] Metropolitan Police Department of St. Louis.  St. Louis Crime Data, 2017.
  * [17] T. Munzner.  A nested model for visualization design and validation.  IEEE Transactions on Visualization and Computer Graphics, 15(6):921–928, 2009. doi: 10 . 1109/TVCG . 2009 . 111
  * [18] A. Ottley, R. Garnett, and R. Wan.  Follow The Clicks: Learning and Anticipating Mouse Interactions During Exploratory Data Analysis.  Computer Graphics Forum, 38(3):41–52, 2019. doi: 10 . 1111/cgf . 13670
  * [19] P. Saraiya, C. North, and K. Duca.  An insight-based methodology for evaluating bioinformatics visualizations.  IEEE transactions on visualization and computer graphics, 11(4):443–456, 2005.
  * [20] A. Tse.  Why we are doing fewer interactives.  The New York Times, 2016.
  * [21] Yelp.  Yelp Open Dataset, 2018.
  * [22] J. S. Yi, Y. a. Kang, J. Stasko, and J. Jacko.  Toward a deeper understanding of the role of interaction in information visualization.  IEEE Transactions on Visualization and Computer Graphics, 13(6):1224–1231, 2007. doi: 10 . 1109/TVCG . 2007 . 70515