The different paths to entropy

In order to undestand how the complex concept of entropy emerged,we propose a trip towards the past reviewing the works of Clausius, Boltzmann, Gibbs and Planck. In particular, since the Gibbs's work is not very well known, we present a detailed analysis, recalling the three definitions of the entropy that Gibbs gives. May be one of the most important aspect of the entropy is to see it as a thermodynamic potential like the other thermodynamic potentials as proposed by Callen. We close with some remarks on entropy and irreversibility.

1.Introduction
The concept of entropy is not easy to grasp (even for physicists) and frequently entropy is seen as very mysterious quantity. It received a very large number of interpretations, explications, applications. The literature on entropy is huge and it is possible in any work on entropy to mention only a very small fraction of the already published works on the subject [1]. Only for the curiosity of the reader one can mention that not all these applications have a real rational basis connected firmly with physical laws [2].
The difficulty to understand the concept of entropy has a long history as one has testimonies from the beginning of the twentieth century until today One can quote James Swinburne who initiated with John Perry a debate on entropy from 1902 to 1907 [3]. It results a series of articles in electrical engineering journals. Swinburne himself wrote a book on Thermodynamics: Entropy or Thermodynamics from an Engineer's Standpoint and the Reversibility of Thermodynamics. Swinburne wrote: As a young man I tried to read thermodynamics, but I always came up against entropy as a brick wall that stopped my further progress. I found the ordinary mathematical explanation, of course, but no sort of physical idea underlying it. No author seemed even to try to give any physical idea. Having in those days great respect for textbooks, I concluded that the physical meaning must be so obvious that it needs no explanation, and that I was especially stupid on the particular subject.
Recently Ben Naim [4] published three books on entropy with the explicit goal to make understanding entropy easy for all. The simple fact that he needs three books for this is a clear sign of the complexity and the difficulty of this task.
In this paper, I propose a kind of trip through time, coming back to the original works on entropy. It may be illuminating to analyze how the concept was born and how it evolved through the works of Clausius, Boltzmann, Gibbs and Planck. There are already several works on the history of entropy (see for example ref. 5) but it appears that they are far from being complete: very often only the works of Clausius and Boltzmann are described, and further developments are forgotten.
Firstly I intend to go to the initial appearance of the concept by Clausius in 1864 and afterward to follow two other important developments in the definition of entropy: Boltzmann and Gibbs. Though the fundamental work of Gibbs is frequently mentioned, it remains not very well known. Who does remember that Gibbs proposed three definitions for the entropy? This is the reason why it seemed that there is interest in exposing the Gibbs's work in some detail. But I shall be more concise about Boltzmann who is much well known.
On the way I shall also mention Planck and his work on the black body radiation since the use of the Boltzmann entropy was essential for his pioneering work. I shall go on to the definition of entropy in Quantum Mechanics. Finally I shall discuss how the entropy developed in the modern view of Thermodynamics.

2.Clausius, the father of the Entropy
Clausius published a series of papers (in German) on the theory of heat and its applications to engines, from 1850 to 1864. The papers were translated to English and published all together in a book in 1872, The Mechanical Theory of Heat with its Applications to Steam-Engine and to the Physical Properties of Bodies. [6] The book is divided into 9 memoirs and I shall mention only those which are relevant to the subject of entropy.
To understand the development of the Claudius's ideas, one has to come back to the work of Sadi Carnot Réflexions sur la puissance du feu et sur les machines propres à développer cette puissance published in 1824 [7]. The important result of his work is that it is not possible to transform completely into mechanical work a given quantity of heat extracted from a hot reservoir in a cyclic process. A part of the heat must be transferred to a cold heat reservoir. This is true even in a case of an engine working reversibly. In this case, the engine can be used in the reverse direction and a quantity of heat may be transferred from the cold reservoir to the hot reservoir but a mechanical work must be provided to the engine, work which is transformed into heat. Clausius formulated the Second law of Thermodynamics in saying that heat cannot pass from a cold reservoir to a hot one by itself.
Thus the Second law of the Thermodynamic is simply the impossibility to transform heat completely into mechanical work in a cyclic process. The impossibility for heat to pass spontaneously from a cold reservoir to a hot reservoir can be deduced from this formulation of the Second Law. We have to add that it is not possible to make an analogy with a water fall since the heat is not a conserved quantity but a form of energy.
We note two things. Apparently, this formulation based on the Carnot engine or the Carnot cycle does not introduce the notion of non-reversible process since the engine can work in the two directions. Secondly, this formulation does not introduce entropy.
However, irreversibility is hidden in that there is asymmetry in this formulation of the Second Law: in a cyclic process, it is possible to transform completely work into heat when the contrary is not possible. Moreover, heat passes spontaneously from a hot reservoir to a cold reservoir while the contrary is possible only in investing work, i.e. non-spontaneously.
The spontaneous process is therefore irreversible.
In the fourth memoir (published in 1854), Clausius proceeds to an original analysis of the Carnot cycle and sees in it two transformations: the first is the transfer of heat from the cold to the hot reservoir (or from the hot to the cold) and the second is the transformation of work into heat (or heat into work). He considers these two transformations as equivalent, not in the meaning that one can replace the other but rather that one transformation cannot appear without the other. Now he proposes to introduce a new concept that he calls the equivalentvalue of these transformations. If Q is the heat transformed in work (or the work into the heat Q) from the hot reservoir at temperature T h , the equivalent-value is Now if Q' is the heat transferred from one reservoir to the other (the cold one having a temperature T c ) the equivalent-value is Following Clausius, since these transformations are equivalent their equivalent-values are equal. Supposing that the Carnot cycle is working as an engine giving work to the surrounding, one has Q > 0 and Q' < 0. Equating (1) and (2) give the well-known relationship between the heat expelled from the hot reservoir (Q -Q'), that received by the cold reservoir (-Q') and the temperatures of the reservoirs Generalizing to a cyclic process as the limit of infinity of elementary Carnot cycles, Clausius gets that in a reversible cyclic process, one has ʃdQ/T = 0 the integral being calculated along the cycle. Or, in other words, the differential is an exact differential and by this way Clausius introduces a new variable S which depends only on the state of the system. In fact by this way one can only calculate the difference of S between two different states (A and B) of the system as following a reversible path But it is remarkable that at this stage he does give the name "entropy" to this new variable.
Considering an irreversible cyclic process, Clausius finds that the equivalent values of the two transformations are not equal but the positive transformation dominates. About a irreversible cyclic process, he finds that ʃdQ/T ≥ 0 Here also the integral is calculated along the cycle.
In an adiabatic irreversible process dQ = 0 and one has the fundamental relation between entropy and irreversibility: In a closed system (adiabatic transformation) the entropy cannot decrease, it remains constant or increases.
Suddenly (two pages before the end of his text), Clausius (under the influence of british scientists like Thomson and Rankine as he admits) states that "if all the changes of condition occurring in universe the transformations in one definite direction exceed in magnitude those in the opposite direction, the entire condition of universe must always continue to change in that first direction, and the universe must consequently approach incessantly a limiting condition," He is clearly less pessimistic than Thomson [8] who predicts in 1852 that gives an explanation of irreversibility. It is a highly controversial paper and I do not bring here the different points of view about this theorem [9].
To prove this theorem, Boltzmann first introduced in a paper (also published in 1872) his equation called today by his name: "the Boltzmann equation". It is an integro-differential equation concerning the distribution function f(v,t) such that f(v,t)d 3 v gives the relative number of molecules of a gas between v and v + d 3 v at time t (v being the vector velocity).
The H-theorem states that the quantity defined as decreases with time or dH/dt ≤ 0 (10) until that dH/dt = 0 at equilibrium.
It is tempting to associate the quantity H with -S as Boltzmann wrote (quoted in reference 10 ) Thus, one may prove that, because of the atomic movement in systems consisting of arbitrarily many material points, there always exists a quantity which, due to these atomic movements, cannot increase, and this quantity agrees, up to a constant factor, exactly with the value that I found in [a paper published in 1871] for the well-known integral ∫dQ/T.
The rationale of Boltzmann was to find a quantity which reaches an extremum in the equilibrium state. However, as mentioned above, the H-theorem was severely criticized and until today it is object of discussions [9]. Contrarily to the Boltzmann's opinion, −H is not considered as the entropy.
The second important contribution of Boltzmann concerns his famous formula S = k B Ln W based on his H theorem, in a paper published in 1877. I shall survey the essential argument of this article and shall not enter into the details of the derivation. The principal point is the definition of a microstate. A microstate corresponds to a macroscopic state of the system (in equilibrium or not) and it is defined by the positions and the momenta of all the particles of the system. Clearly a large number of microstates corresponds to this particular macroscopic state. One can imagine that following the time the system passes from one microstate to another. A fundamental hypothesis concerning a closed system is that the probability of occurrence of a particular microstate is identical for all the microstates of the system.
Consequently the equilibrium state is that with the largest possible number of microstates since it corresponds to the largest probability for the system itself.
If a closed system is not in an equilibrium state, it will evolve to states with larger and larger number of microstates until it reaches the equilibrium state with the largest number of microstates corresponding to its energy. But one has to refine this picture: even being in the equilibrium state, the system may pass to another macroscopic state with a smaller number of microstates. However, the probability that the system reaches such a state is very, very low such it is never observed. Here is the important point: the introduction of the notion of the probability for the occurrence of a state of the system. As mentioned above the equilibrium state is the one with the largest probability and once it reaches this state it goes backward only with a very, very small probability. (A more detailed analysis shows that some fluctuations around the equilibrium state exist but one does not consider them for the time.

See below)
The following step is to identify the largest probability with entropy and to write the proportionality of S with the logarithm of the number W of microstates The Second Law of the Thermodynamics takes now a probabilistic character. The irreversibility appearing when a system out of equilibrium comes back to equilibrium is explained in terms of probability which increases. Clearly, it is not the end of the story but an important step is made. For details about the works of Boltzmann see refs. [10] and [11].
But until now an important aspect of the problem was neglected. We have admitted as evident that microstates can be distinguished. How is it possible to differentiate microstates?
What is the parameter which defines different microstates? There is no problem if the It is clear that Planck needed this discretization in order to get the good result. One can ask an apparently innocent question In order to emphasize the influence of the discretization, consider the following problem: N independent particles can take energies form 0 to ∞, in contact with a thermal reservoir at temperature T. What are the energy and the entropy of the system as functions of T? (Here we do not introduce a second variable as usual like the volume . We suppose that the volume is constant). We shall do that following two ways. In the first the energy is continuous as proposed. In the second, we shall suppose that the particle energy can be divided into discrete quantities E = Kn with n = 0, 1,2,3 ,,,, and K is a constant. We shall calculate E and S as functions of T and K and after wad we shall do K→0 to recover the continuity of the individual energies.
In the first case the total energy of the system is The entropy can be calculated from the expression (∂S/∂T) = (1/T)(∂E/∂T) or One gets It is necessary to introduce an integration constant, the temperature T 0 , since T cannot go to zero (this gives S → ∞). The expression (15) has meaning only for T > T 0 . However it is always possible to calculate the entropy difference ΔS between two states at temperatures T 1 and T 2 . One has ΔS = N k B Ln(T 2 /T 1. ).
In the second method, we calculate the one particle partition function Z: The average energy of the system is The entropy is obtained from S = − (∂F/∂T) where F is the free energy And finally It is important to note that this expression is correct at all temperatures and in particular S tends to zero if T decreases towards zero. Now we shall let K going to zero. For the energy we get E = N k B T as above (13). But for the entropy the things are different. For K → 0 , one has In other word there is no limit for S when K → 0. However the entropy difference is as above ΔS = N k B Ln(T 2 /T 1. ). One sees that the limit K → 0 corresponds to the classical limit at high temperature.
This simple calculation gives us an important lesson. With the help of the entropy (and not the energy) we discover that at low temperature, the energy of particles must be discrete: if not the entropy cannot be calculated at low temperature (see 15 and 23).
The conclusion is that the entropy is the gate to the world of discreteness or Quantum Mechanics. Wishing to find the thermal properties of a particular system of a large number of particles, Gibbs uses the trick of an ensemble. He considers a very large number of identical systems, the ensemble and supposes that each system in the ensemble is in a particular situation i.e. that in each system the particles are in different positions with different momenta or in a different microstate. Gibbs defines the density D in the extension in phase (following the words of Gibbs) or in the phase space with dimension 2n = 6 times the number of particles in one system as follows. It is the number of systems having the position of their particles described by the coordinates q 1 , q 2 , q 3 … q i … q n , and momenta p 1 , p 2 . p 3 … p i … p n in a small box (size given by dq 1 dq 2 …dq .i ...dq n dp 1 dp 2 dp 1 dp 2 …dp i …dp n ) around these positions and momenta values. One already sees the need to make a discretization of the quantity D since the quantities dq's and dp's are very small but have necessarily some finite values.

5.Gibbs and Statistical Mechanics
Gibbs gives a demonstration of the well-known Liouville theorem which states that if the forces acting on the particles of one system are function only of the coordinates (with or without the time), the density-in-phase D is constant in time. Instead of using D, Gibbs defines the probability to find one system in a given box of the phase space by P = D / N where N is the number of systems in the ensemble. From the knowledge of P, it is possible to calculate all the average or macroscopic properties of the system. We shall call P the distribution probability. One notes that this definition of P does not imply that the systems are in equilibrium and thus one can define P also for situations out of equilibrium.
Gibbs considers what he calls the "canonical ensemble" i.e ensemble of identical systems controlled by the temperature and other external parameters such as the volume. The differential of the work made by these external parameters is Σ -A i da i . For this ensemble Gibbs proposes (apparently without justification, but one can see that as a hypothesis) that where Ψ and Θ are constants and ε is the energy of one system. Since one system is controlled by the temperature and external parameters, the energy is not constant but fluctuates. P dε gives the distribution of one system i.e. the probability to find a system with energy between ε and ε + dε. The logarithm of P is linear with the energy η = Ln P = (Ψ -ε)/Θ Gibbs calls the quantity η index of probability. By definition of P, one has ʃ P dq 1 …dq n dp 1 ....dp n = 1 ʃ exp[(Ψ -ε)/Θ] dq 1 …dq n dp 1 ....dp n = 1 and the average value of a quantity B is given by B av = ʃ B P dq 1 …..dp n = ʃB exp[(Ψ -ε)/Θ] dq 1 …dq n dp 1 ....dp n 5.1The first definition of entropy Differentiating the following expression (relatively to Θ), obtained from (27) exp(− Ψ/Θ) = ʃ exp( -ε /Θ) dq 1 …dq n dp 1 ....dp n and taking the averages, Gibbs gets (see details in Appendix A) dε av = -Θ dη av -Σ (A i ) av da i with η av = ʃ η P dq 1 …..dp n = ʃ P Ln P dq 1 …..dp n.
Equation (30) is similar to the well-known thermodynamic equation giving the differential of the internal energy U dU = T dS -Σ A i da i Thus Gibbs concludes that the parameter Θ is proportional to the temperature and that the entropy is proportional to S ≈ -η av = -ʃ P Ln P dq 1 …..dp n . (32) In other words the entropy is the average of the logarithm of the distribution described by P (with a change in the sign in order to make the entropy positive) [p. 45, ref.13 ]. This definition is general even if P is different from the expression (24).
It is the most important expression of the entropy and it seems that Gibbs was the first to find it.

5.2The second definition of entropy
For the second definition of the entropy, Gibbs introduced the function V(ε) defined as the "volume" of the phase space with energy smaller than a given value of the energy. In other words, in the framework of the canonical ensemble, V is the number of boxes in the phase space (i.e in which there are coordinates and momenta of at least one system) for which the total energy is less or equal than a given value of energy: V = ʃ dq 1 …dq n dp 1 ....dp n the integral being calculated for values of q 1 , q 2 ….q n , p 1 , p 2 …..p n corresponding to energies smaller than a given value ε.
The derivative (dV/dε) can be seen as the energy density of the space phase since the quantity (dV/dε)dε indicates how many boxes there are with energy between ε and ε + dε. It results now that in the above integrals the differential form dq 1 ……dp n can be replace by (dV/dε)dε. Thus the integral (28) giving the average value of the quantity B is A new function is now defined and Gibbs shows two following important results (see Appendix B for details of the calculations); 1. The average of the derivative (dΦ/dε) av is equal to 1/Θ.
2. The value of (dΦ/dε) for the most probable value of the energy ε 0 is also equal to 1/Θ.
Consequently, since Θ was already identified with the temperature, Gibbs concludes that Φ o , the value of Φ for the most probable value of the energy, is proportional to the entropy: In the case of the of the microcanonical ensemble (with a definite of the energy) the quantity (dV/dε)dε calculated for this specific value of the energy is the "volume" of the shell of thickness dε. More simply (dV/dε) is the "surface" of the "volume" V(ε) and it gives the number of microstates.
One can see that the second definition (for the microcanonical ensemble) of the entropy is However, the definition (36) corresponds to the macrocanonical ensemble and one concludes that the average of the "surface" of the "volume" V(ε) is the entropy and is equal to its value for a given energy in the microcanonical ensemble.
The procedure of Gibbs has the advantage to be valid even with continuous quantities i.e. at not too low temperatures.
Consequently (LnV) av is also a candidate for the entropy. Since Now one has Ln V proportional to (3n/2) Ln ε and dV/dε proportional to (3n/2 −1 )Ln ε. If n is much larger than 1, the dependence of Ln ε and that of dV/dε with ε is the same. Thus the means of Ln ε and dV/dε differ only by some constants.
In the chapter XIV of his book, Gibbs discusses the different forms of the entropy: η av , Φ and Ln V. The important point is that the three expressions are equal in the limit of an infinite number of degrees of freedom i.e. an infinite number of particles. He discusses what could be the best expression for the entropy and he hesitates between the second or the third definition.

States out of equilibrium and irreversibility
A large part of Gibbs's book is devoted to equilibrium states but he envisages also situations out of equilibrium, when the distribution probability P is not equal to that of equilibrium. In particular he demonstrates a series of inequalities [chapter XI] which show the particular character of the entropy as a quantity which in certain circumstances can only increases or dos not change.
An important case is given by the theorem I of chapter XI. If an ensemble of systems has its distribution probability P different from that of the canonical ensemble i.e. that given by (12) above, the quantity η av is larger than for the canonical ensemble. An ensemble with a distribution probability different from that of the canonical is not in equilibrium. Thus when the ensemble comes back to the equilibrium, its η av is less than that of the ensemble out of equilibrium. Since η av is proportional to -S, this implies that when the ensemble comes back to equilibrium, its entropy increases. In other words an arrow of time is present in the real word due to its probabilistic character.
To conclude this survey of the Gibbs's work in Statistical Mechanics, one has to emphasize that the starting point of Gibbs is not to look for a quantity which must be maximum at the equilibrium state. In other words the problem of the irreversibility is not his starting point but only a consequence of his analysis.

6.Entropy in Quantum Mechanics
The well-known expression for the entropy in Quantum Mechanics is S = -k B Σρ n Ln ρ n = Tr[ρ Ln ρ] where ρ is the density matrix (k B is the Boltzmann constant). One notes already the similitude with the Gibbs expression (32). However following Landau and Lifshitz [14], it is not possible to separate the two statistical aspects: that due to Quantum Mechanics and that due to Statistical Mechanics. They introduce a statistical matrix in place of the density matrix.

I think that is important to understand the development of the notion of entropy in
Quantum Mechanics and how one can get (41 ). For that I present the derivation given by Landau and Lifshitz in their book Statistical Physics [14]. Interesting enough to see that one recovers all the above expressions but in the frame of the Quantum Mechanics. If one speaks about one subsystem, we can remove the subscript n. If the number of quantum states with energy less than a given value E is described by the function Γ(E), the number of states between E and E + dE is given by (dΓ(E)/dE)dE . Thus the probability that the energy of the system is between the values E and E + dE is the product of the probability w(E) by the number of states with energy between E and E and E + dE with the condition ʃ W(E)dE = 1.
The function W(E) is in general peak shaped , since the macroscopic value of the energy given by its average E av is also equal to the most probable value E m or E m = E av .

Entropy today
The preceding sections showed us that there are several definitions of the entropy. One can put them into two classes. In the first, the entropy is a characteristic of systems with temperature i.e. with a very large number of particles. The thermodynamic version is the entropy with its differential dS = dQ/T. The statistical version is given by the expressions S ≈ -η av = -ʃ P Ln P dq 1 …..dp n (32) S = -k B Σ w n Ln w n (58) The second class is related to the property of the entropy as a maximum in a closed system.
The thermodynamics version states that in a process taking place adiabatically, the entropy either remains constant or increases. The statistical version is the Boltzmann formula and the expression (36) of Gibbs:

7.1.Entropy as a tool
Following Clausius one can see entropy as a tool: it is a function of the state of the system and in order to calculate the entropy change between two states one has to choose a reversible path. It is very useful in order to solve problems in Thermodynamics. The conjunction of the two fundamental laws of thermodynamic permits to get numerous important results that one can find in all textbooks on thermodynamics. dE = dQ -PdV and dS = dQ/T It seems that the first to follow this way was Clausius who in fact discovered the two fundamental laws.

Entropy as a potential
An another thermodynamic potential The concept of thermodynamics potentials is very well known. Classically they are the Helmholtz free energy F = U -TS (U is the internal energy), the Gibbs free energy G = U -TS + PV (P is pressure and V the volume) and the grand potential -PV. When expressed as function of T, V and N (the number of particles in the system), F is a minimum for processes taking place at constant T, N. And when G is expressed as a function of T, P and N, it is a minimum for processes taking place at constant T, N. The same holds for the grand potential Ψ expressed a function of T, V and the chemical potential. Now one can include the entropy in the category of the thermodynamic potentials and considers -S as a potential for processes taking place at constant energy, constant V and constant N. When one says that in process taking place in a closed system, the entropy is maximal, one can say that: in a process taking place at constant U, V and N, -S is a minimum. This gives a very simple meaning for the entropy associating it with the notion of thermodynamic potential.
To see the entropy as a potential for an isolated system permits to remove the mysterious aspect of the entropy. As in Mechanics the equilibrium is characterized by a minimum of the potential energy, in a closed system the thermodynamic equilibrium is characterized by the minimum of -S. As in Mechanics, the derivatives of the potential energy give the forces, the derivatives of the entropy give various thermodynamics quantities. This point of view was adopted by Callen [15] who puts the entropy on the same grounds than the Helmholtz free energy, the Gibbs free energy or the grand potential.
I shall take a very simple (even trivial) example. In spite of its simplicity, this shows how S can be seen as a potential. One considers an isolated box of volume V with an ideal gas inside and a mass hanged by a spring at the top of the box. One supposes that the heat capacity C G of the gas and that of the mass (C M ) are independent of the temperature. The mechanical energy of the spring is (1/2)Kx 2 when K is the spring constant, independent of the temperature and x is the displacement of the mass from the equilibrium position. One supposes also the internal energies to the gas and of the mass are independent of the volume: U G = C G T and U M = C M T. The system is in equilibrium: the temperature of the gas and the mass is T 1 and the mass is maintained at the position x 0 . One can liberate the mass from this position and after some time it will come to the position x = 0 because of the friction with the gas. The mechanical energy of the spring is transformed in heat and the temperature of the system increases.
For the sake of simplicity one can suppose that the mass will move very slowly without oscillations. One has two equations: Conservation of the energy U 0 = (C G + C M )T + 1/2 Kx 2 (46) One neglects the gravitational potential energy to simplify the problem but its inclusion gives the same conclusion.
which is a standard result of the thermodynamic fluctuations theory.

Entropy out of equilibrieum
Landau and Lifshitz [14] propose a definition of the entropy for states out of equilibrium.
Suppose that the system with relaxation time Δt is not in an equilibrium state. One can divide (a thought experiment) the system into numerous small (but macroscopic) subsystems with relaxation time much smaller than Δt. It is possible to define for each these subsystem its entropy and the total entropy is the sum of all these partial entropies.
Furthermore there is a long tradition among those who use the microcanonical ensemble to calculate the entropy via the formula S = k B Ln W. The quantity W can be well defined even for states out of equilibrium and the final stage is to find the conditions for W to be a maximum. It is the standard method exposed in several textbooks to obtain the Maxwell-Boltzmann, the Fermi-Dirac and the Bose-Einstein distributions.

Entropy and irreversibility.
About irreversibility, the work of Planck [18] is well known. For him, the notion of process, in particular irreversible process, is essential for the definition of the entropy: Furthermore irreversibility [16] is not a simple concept and needs special attention and clarification. We take a very simple example of the free expansion of gas in a close vessel. In the final state the entropy is larger than in the initial state. It is often considered that this is the signature of an irreversible process. But one can see the things in another manner. The entropy is a state function and becomes larger because the parameters of the final state are such that the entropy increases. The fact it is possible to pass also from the initial state to the final state in a reversible way could be an indication that the increase of the entropy is not necessarily related to an irreversible process. This is why Planck proposes his own definition of irreversible process. A process is irreversible if, in order to come back to the initial state of the system, it is necessary to induce a change in the state of environment of the system. In the case of the free expansion, in order to come back to the initial state of the gas, first the volume of the gas must be decreased doing work from the environment and then the temperature of the gas must be brought to its initial value by thermal contact with the a reservoir. Finally the environment did work on the gas and exchanged heat with the gas. In other words, in the complete process of free expansion of a gas and its return to the initial state, the environment did not return to its initial state.
Introducing the entropy through irreversible processes is a very standard method to define the entropy. However this method presents some difficulties. First, because in the development of Thermodynamics, one uses derivatives and integrals and this is possible only if all the states of a system are equilibrium states using the concept of reversible quasistatic processes. In this part of Thermodynamics called "Thermostatics" (in contrast to Thermodynamics of irreversible process) one never considers irreversible processes. There is some paradox to introduce entropy using the concept of irreversibility to study phenomena in which irreversible processes never appear. Another paradox is as follows: in order to calculate the entropy which increases in an irreversible process, one has to find a reversible path between the initial and the final states. Reversible processes are necessary to characterize irreversible processes!

Conclusion
In this trip towards entropy, I tried to find aspects of the entropy which are relatively easy to understand: entropy as a tool, entropy as a potential, entropy as indication of distribution and fluctuations, in the microcanonical ensemble through the Boltzmann formula and in the canonical ensemble through the Gibbs formula, entropy as the gate to the Quantum Mechanics.
The property of the entropy to be a maximum in some circumstances has always excited the imagination.. Already from the beginning, there is the famous sentence of Clausius, (the last line of his text): The entropy of the universe tends to a maximum I wonder about the impact of this notion of a maximum. What is this particular quantity which can only increase? On other hand, it appears so "natural" to associate an equilibrium state with a minimum of some quantity like the potential energy that associating it with a maximum seems a paradox. May be a part of the mystery comes from this association of equilibrium and maximum of entropy.
The intriguing aspect of entropy is, above all, due to the richness of the concept. In this trip toward the past, it was not possible to discuss all the facets of the concept nor the different methods to expose the Second Law. There are numerous works on these other aspects and I refer the reader to ref. 16. Most recent developments (Tsallis) are also absent since my intention was to investigate the emergence of the concept.


Precision-biased Parsing and High-Quality Parse Selection

We introduce precision-biased parsing: a parsing task which favors precision over recall by allowing the parser to abstain from decisions deemed uncertain. We focus on dependency-parsing and present an ensemble method which is capable of assigning parents to 84% of the text tokens while being over 96% accurate on these tokens. We use the precision-biased parsing task to solve the related high-quality parse-selection task: finding a subset of high-quality (accurate) trees in a large collection of parsed text. We present a method for choosing over a third of the input trees while keeping unlabeled dependency parsing accuracy of 97% on these trees. We also present a method which is not based on an ensemble but rather on directly predicting the risk associated with individual parser decisions. In addition to its efficiency, this method demonstrates that a parsing system can provide reasonable estimates of confidence in its predictions without relying on ensembles or aggregate corpus counts.

Introduction and Methodology
Parsing technology has made great progress over the last decade, and current state-of-the-art parsers for English have reported accuracies in the low 90%'s. Current parsing systems are designed to provide a complete parse to every sentence, and are evaluated based on their average number of correct decisions over a test corpus. Such evaluation, however, does not tell a complete story, as the mistakes are never uniformly distributed across sentences. In practice, parsers usually perform very well on some sentences, but also perform very poorly on others. In addition, parsers do not provide confidence estimates in their predictions, making it hard for downstream applications to rely on parsers' output, even if the average parsing quality is high.
We advocate a modification to the parsing task, which we term precision-biased parsing. Rather than providing a complete parse to every sentence, we advocate providing partial analyses to some sentences, while trying to guarantee that the structures that are provided are of high quality. In other words, we advocate parsing systems which are able to trade recall for precision. The trade-off can occur either at the sentence level (abstaining from providing a parse to some sentences) or at the individual attachment level (abstaining from attaching some words or phrases to the rest of the sentence in case the attachment is uncertain).
Such trade-off is useful for tasks which rely on precise structures. 1 These include information extraction and question answering systems, sentence simplification and summarization systems and syntax-based translation, as well as linguisticallyoriented tasks such as learning selectional preferences, case frames or lexical ontologies. In addition, partial but precise output may prove useful for selftraining (McClosky et al., 2006), uptraining (Petrov et al., 2010 and active-learning setups. Some previous efforts (Alexander Yates and Etzioni, 2006;Reichart and Rappoport, 2007) attempt to identify high-quality parse trees among parser's output. This is an instance of trading recall for precision at the sentence level. Here, we focus on trading recall for precision at the individual attachment level. As discussed in section 3, solving the problem at the individual attachment level entails a natural solution to the problem at the sentence level as well. Moreover, we believe there is benefit in solving the problem at the attachment level -useful information can be extracted from a partial parse tree, even when some attachment decisions are missing or marked as unreliable.
The precision-biased task was explored in the past in the context of parsers based on manuallydeveloped grammars (Carroll and Briscoe, 2002;Watson et al., 2005). However, state-of-the-art datadriven statistical parsers do not allow trading recall for precision. In this paper, we focus on data-driven dependency parsing.
We begin by defining the precision-biased parsing task and its evaluation measures (Section 2), and provide a strong baseline based on parse-ensembles (Section 5). While effective, the ensemble system requires substantial computational effort, and is of little theoretical interest as it is well known that committee (dis)agreement is a good indicator of confidence. We propose another method based on parsermodeling in Section 6. We train a probabilistic classifier to try and predict the risk associated with attachment decisions in the parser's output. The classifier learns the error patterns of the parser, and assigns a reliability score to parse edges. By thresholding these reliability scores, we can effectively trade recall for precision. The method comes close to the ensemble-based baseline in terms of precision and coverage, while running faster and providing a straightforward way to control the recall/precision tradeoff. More importantly, it demonstrates that reasonable confidence estimates of the correctness of parser predictions can be attained without relying on committee, diversity, or aggregate corpus-based counts of recurring structures. Inspecting the behavior of the learned model on PP-attachment, a classic case of syntactic ambiguity, reveal that it is not judged by the model as being categorically hard. Rather, some PP-attachment cases are marked as unreliable, while others are not.

Precision-biased Dependency Parsing
In the traditional dependency parsing task, the input is a set of sentences (the test corpus), and we are interested in the most precise analysis for each sentence. The task performance is measured based on the average number of tokens that got assigned a correct parent over the entire test corpus. Crucially, the parsing process in this task must assign a parent to each of the tokens in the test corpus.
In contrast, in the precision-biased task we are concerned with precision more than recall. We allow the parsing system to abstain from providing an analysis to some of the input by skipping some decisions. That is, we require the parser to assign parents to as many of the input tokens as possible, but allow it to leave the parents of some tokens unassigned. Metrics Let T be the set of input tokens, A be the set of tokens that got assigned a parent, and S be the set of tokens that were not assigned a parent (T = A∪S, A ∩ S = ∅). Let C ⊂ A ⊂ T be the set of tokens that got assigned a correct parent. Then: By requiring complete coverage of the input tokens (coverage = |A| |T | = 1) we get A = T , and then precision = recall = accuracy, where accuracy is the traditional dependency parsing accuracy.
The precision-biased setting allows coverage < 1. The aim is to maximize precision, while still retaining sufficient coverage of the input tokens. 2 Parse-selection Previous work addresses the parseselection task: selecting a subset of the input sentences for which we have high-accuracy parses. This is an instance of precision-biased parsing in which abstaining on a token requires abstaining also on all the other tokens in the same sentence. When discussing the parse selection task we distinguish between token coverage which is identical to coverage as defined above, and sentence coverage which is the number of selected sentences divided by the total number of input sentences |{sent|sent⊂A}| |{sent|sent⊂T }| . The definitions of precision and recall remain as above.
We tackle precision-biased parsing by defining a riskiness function on individual attachment decisions. The riskiness R(tok, par) of a token/parent pair is the inverse of our confidence in the attachment decision. A high riskiness indicates uncertainty in the decision, and low riskiness indicates that we believe the decision to be correct. We then set a riskiness-threshold, and abstain from any parent assignment for which the riskiness is above the threshold. Setting a low riskiness-threshold results in higher precision (considering even low-risk decisions as too risky), and setting a high riskinessthreshold results in higher coverage.
Parse-selection Having defined a risk function and a risk-threshold, we get a natural selection criteria for the parse-selection task: high-quality parses are those for which at most K attachments are above the riskiness threshold. The precision/coverage balance can be controlled either by changing the risk threshold, or by changing K.

Related Work
While little research attention was dedicated to the precision-biased task (Carroll and Briscoe, 2002;Watson et al., 2005), several studies address the parse-selection task. Yates et.al. (2006) perform parse-selection by filtering out parses containing "semantically implausible" relations, where semantic-plausibility is estimated by high cooccurrence of the words in relation in a large corpora (i.e., the web).
Reichart and Rappoport (2007) perform committee based selection of high-quality constituencyparses by calculating an agreement measure between 20 copies of a lexicalized parser trained on different subsets of a training corpus. Sagae and Tsujii (2007) select high-quality dependency parses by using two dependency parsers and selecting only sentences on which both parses agree on the entire parse.
Kawahara and Uchimoto (2008) identify highquality dependency parses by training a classifier based on sentence level features: sentence length, average dependency length, number of unknown words, number of commas and conjunctions, and corpus frequencies of sentence words.
Reichart and Rappoport (2009) identify highquality parses of an unsupervised parser by looking for parses with many reliable constituents, where reliability of a constituent is calculated based on the number of times its POS-sequence appears in the automatically parsed text. Finally, Dell'Orletta et.al, (2011) assign quality-scores to dependencyparses using a metric which measures various syntactic properties of the parse tree and compares them the aggregate measurements over the entire parsed corpora.
To summarize, there are three lines of work addressing the parse-selection task: selecting parses based upon agreement between a committee of parsers, selecting parses based on agreement between the parses and aggregate counts over a large corpora (either lexicalized "semantic" agreement or syntactic agreement), and selecting parses based on sentence-specific features (length, vocabulary, number of commas and so on).
In contrast, we are primarily interested in selecting high-quality edges rather than complete parses. We view parse-selection as an extension of the precision-biased parsing task, and perform parseselection based on the number of risky attachment decisions. Our assessment of the riskiness or reliability of a particular decision is not based on aggregate corpus counts nor on global features of the input sentence (though such kinds of information may be integrated in the future). In our first method, we adopt a committee-based approach, but apply it primarily for edge-selection. In the second method we present below, we investigate features which may help the parser assess edge riskiness. We note that the marginal edge probabilities obtained from a loglinear parsing model as in (Smith and Smith, 2007) are not reliable predictors of edge riskiness: indeed, the pruning procedure used in (Carreras et al., 2008) consider edges with marginal scores of up to 10 −6 of the highest scoring edge as possible candidates in order to ensure sufficient coverage, indicating that such models may greatly overestimate the marginals of incorrect edges, while underestimating the marginal values of correct edges.
Kawahara (2001) present an automatic method for Case-Frame dictionary construction for Japanese. Their method identify verb case-frames by identifying reliable syntactic constructions, where the re-liability is learned using a hand-crafted heuristic and aggregate corpus counts. This demonstrates the usefulness of identifying reliable instances of specific constructions. Our proposal is to try and identify reliable instances of many different constructions, without relying on hand-crafted heuristics.

Data
Our experiments are based on the dependencyversion of the Penn WSJ corpus, as converted using the Penn2Malt 3 software with Collins' head-rules. The data is POS-tagged using the HMM-based Hunpos tagger 4 .
While our work is not directly comparable to any previous work 5 , this presents an opportunity to stop following the standard train/test/dev splits, and in particular to stop testing only on section 23. Instead, we adopt a setup in which we use sections 2-11 (about 18k sentences) for training the parser(s), sections 12-15 (8900 sentences) for training the riskiness-estimator (where appropriate), section 16 (2780 sentences) for development and sections 17-21 (9500 sentences) for testing. This setup leaves a reasonable amount of training data for the statistical models (the parser training set is roughly the same size as the one used in the CoNLL shared task), while retaining a much larger test set than the standard setting.

Parser-ensemble Riskiness Estimation
Our first method of estimating the riskiness function is using an ensemble method. Ensemble methods have been shown to provide good results for dependency parsing (Sagae and Lavie, 2006;Hall et al., 2007), as well as for parse-selection as discussed above. Here, we use ensembles to estimate the riskiness of individual edges in a dependency tree. To estimate the riskiness, we parse the input sentence using k different parsers and take the intersection of their predictions. The riskiness of a token/parent pair is 0 if all k parsers agree on that prediction, and We use an ensemble of 3 parsers: a lineartime shift-reduce parser as described in (Huang et al., 2009) (SHIFTREDUCE), the globally optimized first-order projective dependency parser of (McDonald et al., 2005) (MST1), and the easy-first parser of (Goldberg and Elhadad, 2010) (EASYFIRST). Such ensemble was shown in (Goldberg and Elhadad, 2010) to provide good oracle accuracies, as well as state-of-the-art accuracies in a non-oracle setting due to the diversity among its parsers. The runtime of this ensemble is dominated by the O(n 2 ) feature extraction stage and the O(n 3 ) inference of the globally optimized MST1 parser.

Precision-biased Scores
The precision-biased scores of the ensemble system on the test-set are 96% precision with a coverage of 84.2% (recall of 80.8%). By not providing an analysis for about 15% of the input tokens, we get an impressive gain in precision. Parse-selection Scores As discussed above, we reduce parse-selection to risk-based precision-biased parsing by selecting parses with at most K risky attachments. Table 5.1 shows the precision and sentence-coverage on the development and test set for various values of K. With a K value of 0 (forcing the parsers to agree on all edges) achieves a precision of 97.5 while covering about a quarter of the sentences in the test set. By allowing one disagreement between the parsers the precision drops to 95.0, but we gain a better sentence-coverage -about 36%. Increasing the value of K decreases the selected parses accuracy while increasing their quantity. While the ensemble method is effective at the precision-biased task, it has two shortcomings: (1) it takes a long time to run due to the runtime complexity of the MST1 parser, and (2) it does not provide a way of tuning the precision/coverage balance.
Here we take a different route. We use a single parser (we use the EASYFIRST parser for its balance between speed and accuracy and its incremental parse construction), and train a discriminative probabilistic classifier to predict the risk associated with its predictions.
EASYFIRST is a greedy parser that work by incrementally adding dependency edges in a bottom-up fashion (see (Goldberg and Elhadad, 2010) for the details). It is trained to take easy decisions before harder ones, but does not provide confidence in its predictions: at a given step, the highest scoring action can still be very ambiguous, yet easier than the alternatives. The two measures of the best possible ("easiest") action and the riskiness of an action are interrelated, but not identical. The easiest action, at a given stage, may still be risky. For example, consider the case in which the parser sees a configuration consisting of [Verb Noun Prep]. This is a PP attachment ambiguity, where the Prep should be the child of either the Noun or the Verb. Concretely, the parser should choose to either attach Noun under Verb and then Prep under Verb+Noun, or to first attach Prep under Noun and then Noun+Prep under Verb. At this stage, the two possible attachments (Verb+Noun and Noun+Prep) are risky (even though the Verb+Noun edge will turn out in the final parse in any case), but the parser should nevertheless choose one of them. It will choose, based on its training experience and on the specific properties of the VP, NP and PP at hand, the action which it finds is most correct. This would be the least-risky attachment at a given stage, but it does not reflect directly on the objective riskiness of the decision at large.
In the precision-biased setting, we are interested in assessing the objective riskiness of various parser decisions. Riskiness Predictor We train a separate classifier to assess the riskiness involved in each prediction. We interpret the riskiness as a probability function: risk(context) = P r(decision is wrong|context).
In words: the riskiness is the probability of the parser making a wrong choice in a given situation (context). The riskiness function does not necessarily depend on the actual decision (i.e., it should be interpreted as "when faced with situation X you are likely to make a mistake" rather than as "attaching t i below t j in situation X is likely to be wrong"), but the decision can be encoded in the context if desired.
We treat riskiness prediction as a binary classification task, and fit a Maximum Entropy model 6 based on training data as described in Section 7.1 below.
We experimented with several alternative interpretations of the riskiness function, capturing different kinds of information (features).

Riskiness of parser actions An interesting question
that arises is whether the information available during parsing is sufficient for determining the risk associated with a parsing decision, and which kinds of information are most useful.
The first set of experiments attaches risk to parser actions. These aim to answer the question "can the parser assess the quality of its own actions". Note that parser actions are not equivalent to attachment decisions: the easy-first parser may choose to attach a token to its correct parent and still be wrong, because it is not yet the correct time to do so (because the child node is not yet saturated), and this action, while resulting in one correct edge, prevents future correct edges from being added (consider the Verb+Noun edge in the PP-attachment example above). Thus, this set of experiments can be used only for the parse-selection task (selecting as good parses those for which there were less than K risky actions), and not for the precision-biased parsing task.
We experimented with the following feature sets: Process-based features: action process is a minimal set of 5 numerical features which relate only to the parsing process itself. These include: sentence-length, current number of parent-less tokens, score of the best action (to be applied), score of the second best action, and the difference between the best and second-best actions. State-based features: action state is a set of features which relate only to what the parser sees.
Here, we use the exact same feature set which is used by the parser for predicting the scores of the various actions. Riskiness of predicted edges The second set of experiments associates riskiness with edge predictions. That is, riskiness is interpreted as "what is the probability of this particular predicted edge to be wrong". In contrast to the previous experiments, this definition of riskiness addresses the full precision-biased parsing problem, by abstaining from providing (or ruling-out) attachments for edges that are considered too risky.
We experimented with the following feature sets: State-based features: as above, the edge state feature set encodes exactly what the parser sees when making an attachment decision, i.e., the feature set used by the parser when scoring actions. However, here wrong parser actions which result in a correct edge are considered as correct (non-risky) examples. Edge-factored features: the edge factored feature set is not related to the parsing process, and can be extracted from the parse tree in a post-processing step. Here, we use the same features as used in Ryan McDonald's first-order edge-factored MST parser (McDonald et al., 2005). Higher-order features: the edge higher featureset does not depend on the parsing process, and uses more information than the edge-factored one: the features of a (token,parent) pair include information on the token and the parent, as well as on the siblings of the token, siblings of the parent, children of the token, and parent of the parent.
edge state has only a negligible effect on the parsing time (as above, the features are already extracted by the parser), while edge factored and edge higher have a noticeable (though still small) effect on the parsing time by adding n feature extraction and scoring operations.

Training
We followed the following procedure: 1. Train the easy-first parser on the parser-training set, and use it to parse the rest of the data (riskinesstraining, test and dev, see Sec. 4) while keeping track of the parser's predictions. 8 2. Extract correct and incorrect decisions and their corresponding features (according to the definitions above) from the automatically parsed data. 3. Train a MaxEnt binary classifier on the riskinesstraining set.

Evaluation
ROC We begin by plotting the ROC curves for identifying risky decisions using the different MaxEnt risk predictors with varying riskthresholds. Figure 2 presents the results. The curves are not entirely comparable: the two lower curves (action process and action state) identify risky parser actions, while the higher curves identify risky edge attachments. There is a clear hierarchy between the different predictors, but even the simplest ones are quite effective at identifying risky decisions. The predictors that attach riskiness to edges are more effective than those that attach riskiness to parsing actions, even when the same feature-set is used (edge state vs. action state), and the two predictors that use information external to the parser (edge factored and edge higher) are better than those using information internal to the parser. Still, it is interesting to note that the exact same feature-set which is available to the parser during parsing is sufficient to assess the riskiness of many of the decisions which are based on the same feature-set.
There is only a small difference between the edge factored feature-set and the one including higher-order features (edge higher): while the extra contextual information does help, most of the riskiness associated with an edge can already be determined based on the edge itself and sentence-level properties (without considering proposed surrounding edges). Precision-biased Scores We now turn to evaluating the precision-biased results for the various predictors. As discussed above, these can be calculated only for the three edge-based riskiness predictors. For the precision-biased results, the parser abstains from predicting edges with an associated risk- iness above a certain riskiness-threshold. Figure 1 plots the precision and coverage of the parser for varying riskiness-thresholds, using the three different riskiness-predictors. The third plot in the figure plots precision against coverage for the same predictors. The overall trends are similar to those observed in the ROC curves, though here the difference between edge factored and edge higher is somewhat more pronounced.
With appropriate riskiness thresholds we could achieve a coverage as high as 95%, or precision of above 97%. Unfortunately, we cannot get both: higher precisions mean lower coverages and viceversa. Compared to the ensemble-based riskiness estimation (96% precision with 84% coverage) the single-parser results are not as strong. The same level of coverage (84.6%) results in a precision of around 94.2%, and a precision of 96.5% leads to a coverage of 70%. A riskiness-threshold of 0.15 (edge higher predictor) strikes a nice balance of just over 95% precision with 80% coverage. A coverage of 90% gets us a precision of above 92%, still substantially higher than the 88.4% of the fullcoverage baseline parser. The numbers are practically the same for the development and test sets. Parse-selection scores The parse-selection task has two tunable parameters: the riskiness-threshold R, and the number of risky decisions (K) above which we regard the complete parse-tree as unreliable. For each predictor we performed a grid-search over these parameters with R ranging in value from 0 to 0.5 with increments of 0.01, and K ranging from 0 to 4. For each point we recorded the precision and the sentence-coverage over the development set. We then chose the parameters yielding the  best sentence-coverage for a given precision level (we varied the precision levels from 89 to 99 with increments of 0.5). We processed the test-set with the selected parameter-values. Figure 3 plots precision against sentence-coverage on the test-set using the (K,R) obtained on the development set. to be risky? A complete analysis of the behaviour is well beyond the scope of this paper, but we provide a glimpse into what is possible by inspecting the model's behaviour on a particular case of syntactic ambiguity: PP attachment. Not surprisingly, attaching a preposition to its parent is judged very risky. When sorting POS-tags by the number of times their parent-attachment is judged to be risky, prepositions are at the top of the list. But do the models learn that all PP attachment decisions are risky and abstain from attaching any PP to their parent? Or maybe some kinds of prepositions riskier than others? Table 2 shows the confusion matrix for prepositions as judged by the edge higher model with a riskthreshold of 0.15.  Interestingly, while PP attachment is the most risky phenomena, most PP attachment cases are correctly judged by the model to be non-risky (4302 cases). 326 other PP attachment cases are judged by the model to be safe, but are incorrect. Finally, the model marks 2314 PP-attachment cases as risky, and 961 of these are indeed parsing mistakes.
When breaking down by preposition type, we can observe that "of" is by far the least ambiguous (1457 cases, or 93%, are correctly marked as safe), "in","for","on","as","at" are the most am-  biguous (marked as risky about half of the time) while "by","that","from" are in between (25-35% of the cases are judged to be risky).

Discussion
We advocate a modified version of the parsing task -precision-biased parsing -which favors precision over recall by allowing the parser to abstain from decisions about which it is uncertain. In our view, partial but highly accurate structural information is in many cases more valuable than complete but less accurate structural information. The precision-biased parsing problem is related to confidence estimation, that is, attaching reliability scores to model predictions.
In order to address the precision-biased parsing task we introduce the notion of riskiness of parser decisions. On the basis of riskiness assessment, the parser can abstain from risky predictions. This gives rise to a natural solution to the parse-selection task: reliable parse-trees are those associated with few risky actions.
After verifying that disagreement in a parserensemble is a good indicator of risky edges, we presented a novel approach that does not rely on a parser-ensemble, but instead learns to predict the riskiness involved with individual actions of a single parser. While the method sacrifices more coverage than the parser-ensemble in order to achieve the same level of accuracy, the results are encouraging and demonstrate that a single parsing system can monitor the confidence of its own predictions.
Single parser riskiness assessment turns out to be a good indicator of confidence on aggregate: the single-parser system is as capable as the ensemblebased one at selecting high-quality complete parses.


s2fieldsofstudy:
Computer Science
Engineering

text:
Statistical Sensitivity Analysis for Training Feedforword Neural Networks by B-Splines Weight Functions

Based on the new architecture and algorithm of training neural networks by B-splines weight functions, the statistical sensitivity analysis for neural networks using B-splines weight functions is discussed in this paper. The sensitivity formula of B-splines weight function neural networks is derived. Finally, the correctness of theoretical sensitivity formula is verified by simulation.

Introduction
Sensitivity refers to how a system output is influenced by its input perturbations. Sensitivity analysis for neural networks can be dated back to the 1960s. Many scholars have studied sensitivity or the sensitivity of neural networks [1,2]. In 1962, Hoff used an n-dimensional hypersphere to model Adaline for sensitivity analysis. Winter (1989) was the first one to derive an analytical expression for the probability of error in Madaline caused by weight perturbations. Stevenson continued Winter's work, established the sensitivity of Madaline to weight error, defined the mathematical model of sensitivity as hypersphere and analyzed the sensitivity of ADALINE. Piché [3] used a statistical approach to relate the output error to the change of weights for an ensemble of Madalines, with several activation functions such as linear, sigmoid, and threshold. On the related literature, other sensitivity function has been defined, such as output sensitivity, trajectory sensitivity, function sensitivity, etc.
By early algorithms (such as backpropagation (BP) or radial basis function (RBF) algorithm), the trained weights were constant data (constant weights). For the constant weights, let us assume that the connection weight vector changes from W* to W=W*+∆W, where W* is the matrix of constant weights for the global minima of neural networks, and ∆W indicates the weight perturbations. Under the assumption of statistical weight perturbations, the statistical sensitivity can be defined as follows [2].
It is well known that artificial neural networks have been widely used in many fields [4]. BP algorithm is trained by the steepest descent-like algorithms and can not find global minima in many applications. Many techniques have been introduced to improve the performance of the steepest descent-like algorithms. Those studies mainly focus on: improving the learning speed [5,6], changing the network's architecture, growing and pruning algorithm [7,8], optimizing the initialized weights or some other values [9], convergence of online training procedure [10], using more additional parameters, deterministic convergence, online gradient method [11], tuning network's parameters, neural network's algorithms using genetic or evolutionary methods [12], hybrid neural networks and so on. Unfortunately, the drawbacks can not be overcome completely arising from the steepest descent-like algorithms, such as the local minima, the slow convergence speed and the limited scale of problems. Therefore, the statistical sensitivity defined in (1) can hardly be used to find correct results.
In recent years, a new class of algorithms for training feedforward neural networks has been proposed by Zhang [13], who changes the weights from constant data to the cubic spline functions (weight functions), and the arguments of the cubic spline functions are input patterns.
The new algorithms proposed by [13] not only simplify the architecture of neural networks, but also overcome the drawbacks by using early algorithms, such as local minima, slow convergence and difficult to obtain the global optimal point. Distinct form [13], a new algorithm for training neural networks using B-spline weight functions are introduced in this paper and the mathematic expression for statistical sensitivity is also obtained, in which the definition of statistical sensitivity is different from (1). Finally, the simulation examples show that the correctness of theoretical sensitivity formula derived from this paper.

Fundamentals of training neural networks by B-splines Weight Functions
, the interpolate nodes can be expressed as . .
Any spline function ( ) b x can be expressed as where i c is a constant.

Advanced Engineering Forum Vols. 6-7 1215
For the interval[ ] , a b ，the spline space is N k + dimensional, and the function ( ) w x has N k + interpolation nodes, let the interpolation points be The k-th spline interpolation satisfies the following conditions ( ) Where i y is the output value, and ( ) b x satisfied (10) is called the k-th spline interpolation function.
If we take 0 1 , , , N x x x ⋅⋅⋅ as spline interpolation nodes, from (10) and (8) we have We can solve the linear system (11) for finding the coefficient i c .

Statistical Sensitivity Analysis of B-splines Weight Function Neural Networks
Different from (1), the definition of statistical sensitivity can be described as: When noise x ∆ is embedded, the theoretic error of B-splines weight function neural networks can be expressed as Where … represents some kinds of norms. Generally speaking, the theoretical noise errors of neural networks contain model error and approximation noise error. , N x x + can get the maximum values, we have where p is the index of subinterval Suppose the input without perturbation is , then the approximation error will be represent approximate output and target output respectively, the output perturbation is

Simulations
To show the results proposed in this paper, an example is given below. The architecture of the network is 3-4, and the learning cure [13] is Then we can get B-splines weight functions by the training patterns. The statistical sensitivity of B-splines weight function neural network can be calculated by (22). For the trained neural network, suppose the output without disturbance is * Y , and the disturbed output is Y , then relative error can be got by the formula as follows.
Advanced Engineering Forum Vols. 6-7 When the input perturbation is very small, theoretical noise sensitivity matches approximate noise sensitivity, see Fig. 2 Fig. 3 illustrates that the output relative error is accordingly very small (close to zero), so we can conclude that the network has generation ability. As can be seen from Fig. 2 and Fig.  3, when the input perturbations increase to some extent, the sensitivity changes quite so obvious, i.e. the sensitivity may reduce, may increase, or increase and then decrease. But the output relative error increases rapidly at the same time. The network doesn't remain stable, very sensitive to the input perturbations and the output deviates from the target values.

Conclusions
Although some results of statistical sensitivity analysis have been achieved using early algorithms, further analysis of the method proposed in this paper is necessary for improved insight into its effectiveness. The sensitivity formula of B-splines weight function neural networks is derived and the correctness of theoretical sensitivity formula is verified by simulations. It can be seen that the theoretical value of sensitivity is determined by several factors. The sensitivity depends not only on the weight functions, but also on the input patterns, which could be determined by measuring the sensitivity of the network.


extfieldsofstudy:
Physics
Computer Science
Mathematics

text:
Capacities of classical compound quantum wiretap and classical quantum compound wiretap channels

We determine the capacity of the classical compound quantum wiretapper channel with channel state information at the transmitter. Moreover we derive a lower bound on the capacity of this channel without channel state information and determine the capacity of the classical quantum compound wiretap channel with channel state information at the transmitter.

I. INTRODUCTION
The compound channel models transmission over a channel that may take a number of states, its capacity was determined by [6]. A compound channel with an eavesdropper is called a compound wiretap channel. It is defined as a family of pairs of channels {(W t , V t ) : t = 1, · · · , T } with common input alphabet and possibly different output alphabets, connecting a sender with two receivers, one legal and one wiretapper, where t is called a state of the channel pair (W t , V t ). The legitimate receiver accesses the output of the first channel W t in the pair (W t , V t ), and the wiretapper observes the output of the second part V t in the pair (W t , V t ), respectively, when a state t governs the channel. A code for the channel conveys information to the legal receiver such that the wiretapper knows nothing about the transmitted information. This is a generalization of Wyner's wiretap channel [15] to the case of multiple channel states.
We will be dealing with two communication scenarios. In the first one only the transmitter is informed about the index t (channel state information (CSI) at the transmitter), while in the second, the legitimate users have no information about that index at all (no CSI).
The compound wiretap channels were recently introduced in [9]. A upper bound on the capacity under the condition that the average error goes to zero and the sender has no knowledge about CSI is obtained. The result of [9] was improved in [5] by using the stronger condition that the maximal error should go to zero. Furthermore, the secrecy capacity for the case with CSI was calculated. This paper is organized as follows.
In Section II we present some known results for classical compound wiretap channel which we will use for our result's proof.
In Section III we derive the capacity of the classical compound quantum wiretap channel with CSI and give a lower bound of the capacity without CSI. In this channel model the wiretapper uses classical quantum channels.
In Section IV we derive the capacity of the classical quantum compound wiretap channel with CSI. In this model both the receiver and the wiretapper use classical quantum channels, and the set of the states can be both finite or infinite. Here we will use an idea which is similar to the one used in [2].

II. CLASSICAL COMPOUND WIRETAP CHANNELS
Let A,B, and C be finite sets, P (A), P (B), and P (C) be the sets of probability distributions on A, B and C, respectively. Let θ := {1, · · · , T }. For every t ∈ θ let W t be a channel A → P (B) and V t be a channel A → P (C). We call (V t , W t ) t∈θ a compound wiretap channel. W n t and V n t stand for the n-th memoryless extensions of stochastic matrices W t and V t .
Here the first family represents the communication link to the legitimate receiver while the output of the latter is under control of the wiretapper.
Let X be a discrete random variable on a finite set {x 1 , · · · , x n }, with probability distribution function p i := P r(x i ) for i = 1, · · · , n, then the Shannon entropy is defined as Let X be a discrete random variable on a finite set X with probability distribution function P X , let Y be a discrete random variable on a finite set Y with probability distribution function P Y , and let P XY be their joint probability distribution, then the mutual information between X and Y is defined as Let N (x|x n ) be the number of occurrences of the symbol x in the sequence X n . For a probability distribution P ∈ P (A) and δ ≥ 0 let typical sequences and conditional typical sequence be defined as : An (n, J n ) code for the compound wiretap channel (V t , W t ) t∈θ consists of stochastic encoders {E} : {1, · · · , J n } → P (A n ) and a collection of mutually disjoint sets {D j ⊂ B n : j ∈ {1, · · · , J n }} (decoding sets).
A non-negative number R is an achievable secrecy rate for the compound wiretap channel (W t , V t ) in the case with CSI if there is a collection of (n, J n ) codes ({E t : t ∈ θ}, {D j : j = 1, · · · , J n }) such that where J is an uniformly distributed random variable with value in {1, · · · , J n }, and Z n t are the resulting random variables at the output of wiretap channels V n t . A non-negative number R is an achievable secrecy rate for the compound wiretap channel (W t , V t ) in the case without CSI if there is a collection of (n, J n ) codes (E, {D j : j = 1, · · · , J n }) such that Remark 1: A weaker and widely used security criterion is obtained if we replace (2), respectively (4), with lim n→∞ max t∈θ 1 n I(J; Z n t ) = 0 .
In case with CSI, let p ′ t (x n ) := p n t (x n ) p n t (T n p t ,δ ) , if x n ∈ T n pt,δ 0 , else and X (t) := {X (t) j,l } j∈{1,··· ,Jn},l∈{1,··· ,Ln,t} be a family of random matrices whose entries are i.i.d. according to p ′ t . It was shown in [5] that for any ω > 0, if we set where µ is a positive constant which does not depend on j, t, and can be arbitrarily small when ω goes to 0, then there are such {D j : j = 1, · · · , J n } that for all t ∈ θ P r Since here only the error of the legitimate receiver is analyzed, so for the result above just the channels V t , but not those of the wiretapper, are regarded.
In view of (5), one has (see [5]) the largest achievable rate, called capacity, of the compound wiretap channel with CSI at the transmitter C S,CSI , is given by where B t are the resulting random variables at the output of legal receiver channels. Z t are the resulting random variables at the output of wiretap channels. Analogously, in case without CSI, the idea is similar to the case with CSI: Let p ′ (x n ) := if x n ∈ T n p,δ 0 else and X n := {X j,l } j∈{1,··· ,Jn},l∈{1,··· ,Ln} be a family of random matrices whose components are i.i.d. according to p ′ .
For any ω > 0, define where µ is a positive constant which does not depend on j, t, and can be arbitrarily small when ω goes to 0, then there are such {D j : j = 1, · · · , J n } that for all t ∈ θ P r Using (7) one can obtain (see [5]) that the secrecy capacity of the compound wiretap channel without CSI at the transmitter C S is lower bounded as follows,

III. CLASSICAL COMPOUND QUANTUM WIRETAP CHANNELS
Let A and B be finite sets, and let H be a finite-dimensional complex Hilbert space. Let P (A) and P (B) be the sets of probability distributions on A and B respectively, and S(H) be the space of self-adjoint, positive-semidefinite bounded linear operators with trace 1 on H. Let θ := {1, · · · , T } and for every t ∈ θ let W t be a channel A → P (B) and V t be a classical quantum channel, i.e., a map A → S(H): For a state ρ, the von Neumann entropy is defined as Let P be a probability distribution over a finite set J, and Φ := {ρ(x) : x ∈ J} be a set of states labeled by elements of J. Then the Holevo χ quantity is defined as ) .
A non-negative number R is an achievable secrecy rate for the classical compound quantum wiretap channel where J is an uniformly distributed random variable with value in {1, · · · , J n }. Z t are the sets of states such that the wiretapper will get. A non-negative number R is an achievable secrecy rate for the classical compound quantum wiretap channel Theorem 1: The largest achievable rate (secrecy capacity) of the classical compound quantum wiretap channel (W t , V t ) t∈θ in the case with CSI C S,CSI at the transmitter is given by Respectively, in the case without CSI, the secrecy capacity of the classical compound quantum wiretap channel (W t , V t ) t∈θ C S is lower bounded as follows where B t are the resulting random variables at the output of legal receiver channels, and Z t are the resulting random states at the output of wiretap channels. Proof: 1) Lower bound Let p ′ t , X (t) , and D j be defined like in classical case. Then (5) still holds since the sender transmits through a classical channel to the legitimate receiver. We abbreviate X := {X (t) : t ∈ θ}.
Analogously, in the case without CSI, let p ′ X n and D j be defined like in classical case, then (7) still holds.
For ρ ∈ S(H) and α > 0 there exists an orthogonal subspace projector Π ρ,α commuting with ρ ⊗n and satisfying where a := #{A}, and K is a constant which is in polynomial order of n.
For P ∈ P (A), α > 0 and x n ∈ T n P there exists an orthogonal subspace projector Π V,α (x n ) commuting with V ⊗n x n and satisfying: where a := #{A}, d := dim H, and K is a constant which is in polynomial order of n (see [13]).
where α will be defined later.
Lemma 1 (see [14]): Let ρ be a state and X be a positive operator with X ≤ id (the identity matrix) and With the Lemma 1, (11), (17), and the fact that Π P Vt,α √ a and Π Vt,α (x n ) are both projection matrices, for any t and x n it holds: We set Θ t := x n ∈T n p t ,δ p ′ n t (x n )Q t (x n ). For given z n and t, z n |Θ t |z n is the expected value of z n |Q t (x n )|z n under the condition x n ∈ T n pt,δ . Lemma 2 (see [3]): Let V be a finite dimensional Hilbert space, X 1 , · · · , X L be a sequence of i.i.d. random variables with values in S(V) such that X i ≤ µ · id V for all i ∈ {1, · · · , L}, and ǫ ∈]0, 1[. Let p be a probability distribution on {X 1 , · · · , X L }, ρ = i p(X i )X i be the expected value of X i , and Π ′ ρ,λ be the projector onto the subspace spanned by the eigenvectors of ρ whose corresponding eigenvalues are greater than λ dim V , then Let V be the image of Π P,α √ a . By (12), we have The first inequality follows from (16). The second inequality holds because Π Vt,α and Π P Vt,α √ a are projection matrices. The third inequality holds because Π P Vt,α √ a is a projection matrix onto V.
Let L n,t = 2 n(χ(P,Zt)+2ζ) and n be large enough, then by (22) for all j it holds where υ is some positive suitable constant which does not depend on j and t.
Analogously, in the case without CSI, let L n = 2 n maxt(χ(P,Zt)+δ) and n be large enough, then we can find some positive constant υ so that for all j.

Remark 2:
Since exp(−2 nζ ) converges to zero double exponentially faster, the inequality (24) remains true even if T depends on n and is exponentially large over n, i.e., we can still achieve exponentially small error.
From (5) and (24), it follows: For any ǫ > 0, if n is large enough then the event has a positive probability. This means that we can find a realization x j,l with a positive probability such that for all t ∈ θ, we have where E t is the random output of (X (t) j,l ) l . Choose a sufficiently large but fixed α in (19) so that for all j it holds V ⊗n j,l ) < ǫ . In this case, for any given j ′ ∈ {1, · · · , J n } we have and E j Ln,t l=1 1 j,l ) − Θ t ≤ ǫ for any probability distribution uniformly distributed on {1, · · · , J n }. [14]): Let X and Y be two states in a d-dimensional complex Hilbert space and X − Y ≤ µ < 1 e , then

Lemma 3 (Fannes inequality
If J is a probability distribution uniformly distributed on {1, · · · , J n }, then from the inequality (28) and Lemma 3 we have We have lim Analogously, in the case without CSI, we can find a realization x n j,l of X (t) j,l with a positive probability such that: For all t ∈ θ, we have For any γ > 0 let for any probability distribution J uniformly distributed on {1, · · · , J n } in the case without CSI.

2) Upper bound for case with CSI
Considering (C n ) is a sequence of (n, J n ) code such that where lim n→∞ ǫ 1,n = 0 and lim n→∞ ǫ 2,n = 0, J denotes the random variable which is uniformly distributed on the message set {1, . . . , J n }. Let C(V t , W t ) denote the secretey capacity of the wiretap channel (V t , W t ) in the sense of [13]. Choose t ′ ∈ θ such that It is well-known, in information theory, that even in the case without wiretapper (we have only one classical channel W t ′ ), the capacity cannot exceed I(J; B t ′ )+ξ for any constant ξ > 0. So the capacity of a quantum wiretap channel (V t ′ , W t ′ ) cannot be greater than for any ǫ > 0.
Since we cannot exceed the secrecy capacity of the worst wiretap channel, we have

IV. CLASSICAL QUANTUM COMPOUND WIRETAP CHANNEL WITH CSI
Let H be a finite-dimensional complex Hilbert space. Let S(H) be the space of self-adjoint, positive-semidefinite bounded linear operators on H with trace 1. For every t ∈ θ let W t respectively V t be quantum channels, i.e., completely positive trace preserving maps S(H) → S(H).
An (n, J n , λ) code for the classical quantum compound wiretap channel (W t , V t ) t∈θ consists of a family of vectors w := {w(j) : j = 1, · · · , J n } ⊂ S(H ⊗n ) and a collection of positive semi-definite operators {D j : j ∈ {1, · · · , J n }} ⊂ S(H ⊗n ) which is a partition of the identity, i.e.
A non-negative number R is an achievable secrecy rate for the classical quantum compound wiretap channel (W t , V t ) t∈θ with CSI if there is an (n, J n , λ) code ( w t := {w t (j) : j} : t , {D j : j}) such that where J is a uniformly distributed random variable with value in {1, · · · , J n }, and Z t are the sets of states such that the wiretapper will get. Theorem 2: The largest achievable rate (secrecy capacity) of the classical quantum compound wiretap channel in the case with CSI is given by where B t are the resulting random states at the output of legal receiver channels, and Z t are the resulting random states at the output of wiretap channels. Proof: Our idea is to send the information in two parts, firstly, we send the state information with finite blocks of finite bits with a code C 1 to the receiver, and then, depending on t, we send the message with a code C (t) 2 in the second part.

1) Sending channel state information with finite bits
For the first part, we don't require that the first part should be secure against the wiretapper, since we assume that the wiretapper already has the full knowledge of the CSI.
By ignoring the security against the wiretapper, we have only to look at the compound channel (W t ) t∈θ . Let W = (W t ) t be an arbitrary compound classical quantum channel. Then by [4], for each λ ∈ (0, 1) the λ-capacity C(W, λ) equals If min t max p χ(p, W t ) > 0 holds, then the sender can build a code C 1 such that the CSI can be sent to the legal receiver with a block with length l ≤ log T mint maxp χ(p,Wt) . We need to do nothing because in this case the right hand side of (38) is zero.
Let c = 1 − λ, then for any required upper bound δ = 2 −c ′ , with given c ′ > 0, the sender can repeat sending this block log c · c ′ times, and the legal receiver simply picks out the state that he receives most frequently to find out t with a error probability ≤ δ.
The first part is of length l · log c · c ′ = O(1), which is negligible compared to the second part.

2) Message transformation when both the sender and the legal receiver Know CSI
If both the sender and the legal receiver have the full knowledge of t, then we only have to look at the single wiretap channel (W t , V t ).
In [7] and [8], it is shown that there exists an (n, J n , λ) code for the quantum wiretap channel (W, V ) with log J n = max P,w (χ(P, B ⊗n ) − χ(P, Z ⊗n )) − ǫ , for any ǫ > 0, where B is the resulting random variable at the output of legal receiver's channel and Z the output of the wiretap channel.
When the sender and the legal receiver both know t, they can build an (n, J n,t , λ) code C Thus, Remark 3: For the construction of the second part of our code, we use random coding and request that the randomization can be sent (see [7]). However, it is shown in [5] that the randomization could not always be sent if we require that we use one unique code which is secure against the wiretapper and suitable for every channel state, i.e., it does not depend on t. This is not a counterexample to our results above, neither to the construction of C 1 nor to the construction of C (t) 2 , because of following facts.
The first part of our code does not need to be secure. For our second part, the legal transmitters can use the following strategy: At first they bulid a code C 1 = (E, {D j : j = 1, · · · , J n }) and a code C for every t ∈ θ. If the sender wants to send the CSI t ′ ∈ θ and the message j, he encodes t ′ with E and j with E (t ′ ) , then he sends both parts together through the channel. After receiving both parts, the legal receiver decodes the first part with {D j : j}, and chooses the right decoders {D j : j} : t ∈ θ to decode the second part. With this strategy, we can avoid using one unique code which is suitable for every channel state.

3) Upper bound
For any From [7] and [8], we know that the capacity of the quantum wiretap channel (W t ′ , V t ′ ) cannot be greater than Since we cannot exceed the capacity of the worst wiretap channel, we have This together with (42) completes the proof of Theorem 2.
Thus if for every t ∈ θ and n ∈ N, holds for all P ∈ P (A) and {w t (j) : j = 1, · · · , J n } ⊂ S(H ⊗n ), we have So far, we assumed that |θ|, the number of the channels, is < ∞. Now we look at the case where |θ| can be arbitrary. where · 1 stands for the trace norm. It is well known [11] that this norm is multiplicative, i.e. W ⊗ W ′ ♦ = W ♦ · W ′ ♦ . A τ -net in the space of the completely positive trace preserving maps is a finite set W (k) K k=1 with the property that for each W there is at least one k ∈ {1, · · · , K} with W − W (k) ♦ < τ . Lemma 4 (τ −net [10]): For any τ ∈ (0, 1] there is a τnet of quantum-channels W If |θ| is arbitrary, then for any ξ > 0 let τ = ξ − log ξ . By Lemma 4 there exists a finite set θ ′ with |θ ′ | ≤ ( 3 τ ) 2d 4 and τ -nets Since by [7], the error of the code C (t ′ ) 2 decreases exponentially to its length, we can find an N = O(− log ξ) such that for all t ′ ∈ θ ′ it holds Then, if the sender obtains the state information "t" , he can send with finite bits "t ′ " to the legal receiver in the first part, and then they build a code C (t ′ ) 2 that fulfills (46) and (47) to transmit the message.
N τ tends to zero when ξ goes to zero, since N = O(− log ξ).
By (48)  The bits that the sender uses to transform the CSI is large but constant, so it is still negligible compared to the second part. We obtain C CSI > lim n→∞ inf t∈θ max P,wt 1 n (χ(P, B ⊗n t ) − χ(P, Z ⊗n t )) . (50) The proof of the converse is similar to those given in the proof of Theorem 2, where we consider a worst t ′ .
Remark 5: For Theorem 2 and Theorem 3, we have only required that the probability that the legal receiver does not obtain the correct message tends to zero when the code length goes to infinity. We have not specified how fast it should tends to zero. If we analyze the relation between the error probability ε and the code length, then we have the following facts.
In the case of finite θ, let ε 1 denote the probability that the legal receiver does not obtain the correct CSI, and let ε 2 denote the probability that the legal receiver, having CSI, does not obtain the correct message. Since the length of first part of the code is l · log c · c ′ = O(log ε 1 ), as we defined in Section IV, we have ε −1 1 is O(exp(l · log c · c ′ )) = O(exp(n)), where n stands for the length of first part. And for the second part of the code, ε 2 decreased exponentially to the length of the second part, as proven in [7]. Thus, the error probability ε = max{ε 1 , ε 2 } decreases exponentially to the code length in the case of finite θ.
If θ is infinite, let ε 1 denote the probability that the legal receiver does not obtain the correct CSI. Then we have to build two τ -nets, each contains O(( − log ε1 ε1 ) −2d 4 ) channels. If we want to send the CSI of these τ -nets, l, as defined in Section IV, will be O(−2d 4 · log(ε 1 log ε 1 )), this means here ε −1 1 will be O(exp( n 4d 4 )) = O(exp(n)), where n stands for the length of first part. So we can still achieve that the error probability decreases exponentially to the code length in case of infinite θ.

ACKNOWLEDGMENT
We thank Igor Bjelakovic and Holger Boche for useful discussions. Support by the Bundesministerium für Bildung und Forschung (BMBF) via grant 01BQ1052 is gratefully acknowledged.