* [103] S. Chung and L. Abbott, “Neural population geometry: An approach for understanding biological and artificial neural networks,” _Current opinion in neurobiology_ , vol. 70, pp. 137–144, 2021.
  * [104] M. JORDAN and D. RUMELHART, “Forward models: Supervised learning with a distal teacher,” _COGNITIVE SCIENCE_ , vol. 16, p. 307354, 1992.
  * [105] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” _IEEE transactions on knowledge and data engineering_ , vol. 35, no. 1, pp. 857–876, 2021.
  * [106] M. Farrell, S. Recanatesi, and E. Shea-Brown, “From lazy to rich to exclusive task representations in neural networks and neural codes,” _Current opinion in neurobiology_ , vol. 83, p. 102780, 2023.
  * [107] I. Rish _et al._ , “An empirical study of the naive bayes classifier,” in _IJCAI 2001 workshop on empirical methods in artificial intelligence_ , vol. 3, no. 22, 2001, pp. 41–46.
  * [108] A. Clark, “An embodied cognitive science?” _Trends in cognitive sciences_ , vol. 3, no. 9, pp. 345–351, 1999.
  * [109] A. A. Russo, S. R. Bittner, S. M. Perkins, J. S. Seely, B. M. London, A. H. Lara, A. Miri, N. J. Marshall, A. Kohn, T. M. Jessell _et al._ , “Motor cortex embeds muscle-like commands in an untangled population response,” _Neuron_ , vol. 97, no. 4, pp. 953–966, 2018.
  * [110] S. Vyas, M. D. Golub, D. Sussillo, and K. V. Shenoy, “Computation through neural population dynamics,” _Annual review of neuroscience_ , vol. 43, pp. 249–275, 2020.
  * [111] T. Ito and J. D. Murray, “Multitask representations in the human cortex transform along a sensory-to-motor hierarchy,” _Nature Neuroscience_ , vol. 26, no. 2, pp. 306–315, 2023.
  * [112] J. C. Whittington, W. Dorrell, S. Ganguli, and T. Behrens, “Disentanglement with biological constraints: A theory of functional cell types,” in _The Eleventh International Conference on Learning Representations_ , 2022\.
  * [113] T. Ito, T. Klinger, D. Schultz, J. Murray, M. Cole, and M. Rigotti, “Compositional generalization through abstract representations in human and artificial neural networks,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 32 225–32 239, 2022.
  * [114] M. Wilson, “Six views of embodied cognition,” _Psychonomic bulletin & review_, vol. 9, pp. 625–636, 2002.
  * [115] D. V. Buonomano and M. M. Merzenich, “Cortical plasticity: from synapses to maps,” _Annual review of neuroscience_ , vol. 21, no. 1, pp. 149–186, 1998\.
  * [116] Y. Ma, D. Tsao, and H.-Y. Shum, “On the principles of parsimony and self-consistency for the emergence of intelligence,” _Frontiers of Information Technology & Electronic Engineering_, vol. 23, no. 9, pp. 1298–1323, 2022.
  * [117] A. H. Schoenfeld, _How we think: A theory of goal-oriented decision making and its educational applications_. Routledge, 2010.
  * [118] H. A. Simon, “The architecture of complexity,” _Proceedings of the American philosophical society_ , vol. 106, no. 6, pp. 467–482, 1962.
  * [119] A. Treisman, “The binding problem,” _Current opinion in neurobiology_ , vol. 6, no. 2, pp. 171–178, 1996.
  * [120] B. L. McNaughton, F. P. Battaglia, O. Jensen, E. I. Moser, and M.-B. Moser, “Path integration and the neural basis of the’cognitive map’,” _Nature Reviews Neuroscience_ , vol. 7, no. 8, pp. 663–678, 2006.
  * [121] A. S. Etienne and K. J. Jeffery, “Path integration in mammals,” _Hippocampus_ , vol. 14, no. 2, pp. 180–192, 2004.
  * [122] A. Samsonovich and B. L. McNaughton, “Path integration and cognitive mapping in a continuous attractor neural network model,” _Journal of Neuroscience_ , vol. 17, no. 15, pp. 5900–5920, 1997.
  * [123] H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve _et al._ , “Hopfield networks is all you need,” _arXiv preprint arXiv:2008.02217_ , 2020.
  * [124] M. T. Ullman, “A neurocognitive perspective on language: The declarative/procedural model,” _Nature reviews neuroscience_ , vol. 2, no. 10, pp. 717–726, 2001.
  * [125] ——, “Contributions of memory circuits to language: The declarative/procedural model,” _Cognition_ , vol. 92, no. 1-2, pp. 231–270, 2004.
  * [126] E. C. Tolman, “Cognitive maps in rats and men.” _Psychological review_ , vol. 55, no. 4, p. 189, 1948.
  * [127] T. E. Behrens, T. H. Muller, J. C. Whittington, S. Mark, A. B. Baram, K. L. Stachenfeld, and Z. Kurth-Nelson, “What is a cognitive map? organizing knowledge for flexible behavior,” _Neuron_ , vol. 100, no. 2, pp. 490–509, 2018.
  * [128] A. Dedieu, N. Gothoskar, S. Swingle, W. Lehrach, M. Lázaro-Gredilla, and D. George, “Learning higher-order sequential structure with cloned hmms,” _arXiv preprint arXiv:1905.00507_ , 2019.
  * [129] R. L. Klatzky, “Allocentric and egocentric spatial representations: Definitions, distinctions, and interconnections,” in _Spatial cognition: An interdisciplinary approach to representing and processing spatial knowledge_. Springer, 1998, pp. 1–17.
  * [130] R. M. Murray, Z. Li, and S. Shankar Sastry, “A mathmematical introduction to robotic manipulation,” 1994.
  * [131] C. F. Moss and S. R. Sinha, “Neurobiology of echolocation in bats,” _Current opinion in Neurobiology_ , vol. 13, no. 6, pp. 751–758, 2003.
  * [132] S. Toledo, D. Shohami, I. Schiffner, E. Lourie, Y. Orchan, Y. Bartan, and R. Nathan, “Cognitive map–based navigation in wild bats revealed by a new high-throughput tracking system,” _Science_ , vol. 369, no. 6500, pp. 188–193, 2020.
  * [133] L. Harten, A. Katz, A. Goldshtein, M. Handel, and Y. Yovel, “The ontogeny of a mammalian cognitive map in the real world,” _Science_ , vol. 369, no. 6500, pp. 194–197, 2020.
  * [134] J. O’keefe and L. Nadel, “Précis of o’keefe & nadel’s the hippocampus as a cognitive map,” _Behavioral and Brain Sciences_ , vol. 2, no. 4, pp. 487–494, 1979.
  * [135] J. Steckel and H. Peremans, “Batslam: Simultaneous localization and mapping using biomimetic sonar,” _PloS one_ , vol. 8, no. 1, p. e54076, 2013.
  * [136] M. H. Raibert, _Legged robots that balance_. MIT press, 1986.
  * [137] L. Pessoa, E. Thompson, and A. Noë, “Finding out about filling-in: A guide to perceptual completion for visual science and the philosophy of perception,” _Behavioral and brain sciences_ , vol. 21, no. 6, pp. 723–748, 1998.
  * [138] D. H. Ballard, “Generalizing the hough transform to detect arbitrary shapes,” _Pattern recognition_ , vol. 13, no. 2, pp. 111–122, 1981.
  * [139] J. Hawkins, M. Lewis, M. Klukas, S. Purdy, and S. Ahmad, “A framework for intelligence and cortical function based on grid cells in the neocortex,” _Frontiers in neural circuits_ , vol. 12, p. 121, 2019.
  * [140] T. DeWolf and C. Eliasmith, “The neural optimal control hierarchy for motor control,” _Journal of neural engineering_ , vol. 8, no. 6, p. 065009, 2011\.
  * [141] R. Bellman, “Dynamic programming,” _Science_ , vol. 153, no. 3731, pp. 34–37, 1966.
  * [142] T. Poggio and E. Bizzi, “Generalization in vision and motor control,” _Nature_ , vol. 431, no. 7010, pp. 768–774, 2004.
  * [143] L. G. Ungerleider and J. V. Haxby, “‘what’and ‘where’in the human brain,” _Current opinion in neurobiology_ , vol. 4, no. 2, pp. 157–165, 1994\.
  * [144] G. Taylor and L. Kleeman, _Visual perception and robotic manipulation: 3D object recognition, tracking and hand-eye coordination_. Springer, 2008, vol. 26.
  * [145] B. Amirikian and A. P. Georgopoulos, “Modular organization of directionally tuned cells in the motor cortex: is there a short-range order?” _Proceedings of the National Academy of Sciences_ , vol. 100, no. 21, pp. 12 474–12 479, 2003.
  * [146] G. E. Loeb, I. E. Brown, and E. J. Cheng, “A hierarchical foundation for models of sensorimotor control,” _Experimental brain research_ , vol. 126, pp. 1–18, 1999.
  * [147] K. Fukushima, “A hierarchical neural network model for associative memory,” _Biological cybernetics_ , vol. 50, no. 2, pp. 105–113, 1984.
  * [148] D. Krotov, “Hierarchical associative memory,” _arXiv preprint arXiv:2107.06446_ , 2021.
  * [149] J. J. DiCarlo and D. D. Cox, “Untangling invariant object recognition,” _Trends in cognitive sciences_ , vol. 11, no. 8, pp. 333–341, 2007.
  * [150] O. Sacks, “The man who mistook his wife for a hat and other clinical tales,” 1970\.
  * [151] D. H. Ballard, “Animate vision,” _Artificial Intelligence_ , vol. 48, no. 1, pp. 57–86, 1991.
  * [152] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” _arXiv preprint physics/0004057_ , 2000.
  * [153] J. Cui, Y. N. Wu, and T. Han, “Learning hierarchical features with joint latent space energy-based prior,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 2218–2227.
  * [154] S. Hedayati, R. E. O’Donnell, and B. Wyble, “A model of working memory for latent representations,” _Nature Human Behaviour_ , vol. 6, no. 5, pp. 709–719, 2022.
  * [155] B. e. a. Millidge, “Universal hopfield networks: A general framework for single-shot associative memory models,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 15 561–15 583.
  * [156] V. Di Lollo, “The feature-binding problem is an ill-posed problem,” _Trends in cognitive sciences_ , vol. 16, no. 6, pp. 317–321, 2012.
  * [157] J. Theeuwes, “Visual selective attention: A theoretical analysis,” _Acta psychologica_ , vol. 83, no. 2, pp. 93–154, 1993.
  * [158] G. R. Mangun, “Neural mechanisms of visual selective attention,” _Psychophysiology_ , vol. 32, no. 1, pp. 4–18, 1995.
  * [159] L. Itti and C. Koch, “Computational modelling of visual attention,” _Nature reviews neuroscience_ , vol. 2, no. 3, pp. 194–203, 2001.
  * [160] A. Thiele and M. A. Bellgrove, “Neuromodulation of attention,” _Neuron_ , vol. 97, no. 4, pp. 769–785, 2018.
  * [161] R. Khalitov, T. Yu, L. Cheng, and Z. Yang, “Sparse factorization of square matrices with application to neural attention modeling,” _Neural Networks_ , vol. 152, pp. 160–168, 2022.
  * [162] J. M. Wolfe and T. S. Horowitz, “Five factors that guide attention in visual search,” _Nature Human Behaviour_ , vol. 1, no. 3, p. 0058, 2017.
  * [163] S. Pinker and P. Bloom, “Natural language and natural selection,” _Behavioral and brain sciences_ , vol. 13, no. 4, pp. 707–727, 1990.
  * [164] M. H. Fischer and R. A. Zwaan, “Embodied language: A review of the role of the motor system in language comprehension,” _Quarterly journal of experimental psychology_ , vol. 61, no. 6, pp. 825–850, 2008.
  * [165] A. M. Glenberg and M. P. Kaschak, “Grounding language in action,” _Psychonomic bulletin & review_, vol. 9, no. 3, pp. 558–565, 2002.
  * [166] D. Pecher and R. A. Zwaan, “The role of perception and action in memory, language, and thinking,” _Grounding cognition: The role of perception and action in memory, language, and thought_ , 2005.
  * [167] B. Hommel, J. Müsseler, G. Aschersleben, and W. Prinz, “The theory of event coding (tec): A framework for perception and action planning,” _Behavioral and brain sciences_ , vol. 24, no. 5, pp. 849–878, 2001.
  * [168] Y. K. Shin, R. W. Proctor, and E. J. Capaldi, “A review of contemporary ideomotor theory.” _Psychological bulletin_ , vol. 136, no. 6, p. 943, 2010\.
  * [169] V. Gallese and A. Goldman, “Mirror neurons and the simulation theory of mind-reading,” _Trends in cognitive sciences_ , vol. 2, no. 12, pp. 493–501, 1998.
  * [170] M. Jeannerod, “Neural simulation of action: a unifying mechanism for motor cognition,” _Neuroimage_ , vol. 14, no. 1, pp. S103–S109, 2001.
  * [171] R. A. Zwaan and L. J. Taylor, “Seeing, acting, understanding: motor resonance in language comprehension.” _Journal of Experimental Psychology: General_ , vol. 135, no. 1, p. 1, 2006.
  * [172] J. Merel, M. Botvinick, and G. Wayne, “Hierarchical motor control in mammals and machines,” _Nature communications_ , vol. 10, no. 1, p. 5489, 2019.
  * [173] N. Chomsky, “Syntactic structures,” 1957.
  * [174] A. Pascual-Leone and F. Torres, “Plasticity of the sensorimotor cortex representation of the reading finger in braille readers,” _Brain_ , vol. 116, no. 1, pp. 39–52, 1993.
  * [175] S. Schaal, “Is imitation learning the route to humanoid robots?” _Trends in cognitive sciences_ , vol. 3, no. 6, pp. 233–242, 1999.


Efficient Experimental Design for Regularized Linear Models

C. Devon Lin

Department of Mathematics and Statistics

Queen’s University, Canada

Peter Chien

Department of Statistics

University of Wisconsin-Madison

Xinwei Deng

Department of Statistics

Virginia Tech

###### Abstract

Regularized linear models, such as Lasso, have attracted great attention in
statistical learning and data science. However, there is sporadic work on
constructing efficient data collection for regularized linear models. In this
work, we propose an experimental design approach, using nearly orthogonal
Latin hypercube designs, to enhance the variable selection accuracy of the
regularized linear models. Systematic methods for constructing such designs
are presented. The effectiveness of the proposed method is illustrated with
several examples.

Keywords: Design of experiments; Latin hypercube design; Nearly orthogonal
design; Regularization, Variable selection.

##  1 Introduction

In statistical learning and data sciences, regularized linear models have
attracted great attention across multiple disciplines (Fan, Li, and Li, 2005;
Hesterberg et al., 2008; Huang, Breheny, and Ma, 2012; Heinze, Wallisch, and
Dunkler, 2018). Among various regularized linear models, the Lasso is one of
the most well-known techniques on the $L_{1}$ regularization to achieve
accurate prediction with variable selection (Tibshirani, 1996). Statistical
properties and various extensions of this method have been actively studied in
recent years (Tibshirani, 2011; Zhao and Yu 2006; Zhao et al., 2019; Zou and
Hastie, 2015; Zou, 2016). However, there is sporadic work on constructing
efficient data collection for regularized linear models. In this article, we
study the data collection for the regularized linear model from an
experimental design perspective.

First, we give a brief description of the Lasso procedure. Consider a linear
model

$\displaystyle y=\textbf{x}^{T}\mbox{\boldmath$\beta$\unboldmath}+\epsilon,$
(1)

where $\textbf{x}=(x_{1},\ldots,x_{p})^{T}$ is the vector of $p$ continuous
predictor variables, $y$ is the response value,
$\mbox{\boldmath$\beta$\unboldmath}=(\beta_{1},\ldots,\beta_{p})^{T}$ are the
vector of regression parameters, and the error term $\epsilon$ is normally
distributed with mean zero and variance $\sigma^{2}$. Throughout, assume data
are centered so that the model in (1) has no intercept. Suppose this model has
a sparse structure for which only $p_{0}$ predictor variables are active with
non-zero regression coefficients, where $p_{0}<p$. Let
$\mathcal{A}(\mbox{\boldmath$\beta$\unboldmath})=\\{j:\beta_{j}\neq
0,j=1,\ldots,p\\}$ be the set of the indices of the active variables. Then the
cardinality of the set $\mathcal{A}(\mbox{\boldmath$\beta$\unboldmath})$ is
$p_{0}$.

For a given $n\times p$ regression matrix
$\textbf{X}=(\mbox{\boldmath$x$\unboldmath}_{1},\ldots,\mbox{\boldmath$x$\unboldmath}_{n})^{T}$,
and a given response vector
$\mbox{\boldmath$y$\unboldmath}=(y_{1},\ldots,y_{n})^{T}$, the Lasso solution
is

$\displaystyle\hat{\mbox{\boldmath$\beta$\unboldmath}}=\mbox{arg}\min_{\mbox{\boldmath$\beta$\unboldmath}}[(\mbox{\boldmath$y$\unboldmath}-\textbf{X}\mbox{\boldmath$\beta$\unboldmath})^{T}(\mbox{\boldmath$y$\unboldmath}-\textbf{X}\mbox{\boldmath$\beta$\unboldmath})+\lambda\|\mbox{\boldmath$\beta$\unboldmath}\|_{l_{1}}],$
(2)

where
$\|\mbox{\boldmath$\beta$\unboldmath}\|_{l_{1}}=\sum_{i=1}^{p}|\beta_{i}|$ and
$\lambda$ is a tuning parameter. Because the $l_{1}$ norm $\|\cdot\|_{l_{1}}$
is singular at the origin, a desirable property of the Lasso is that some
coefficients of $\hat{\mbox{\boldmath$\beta$\unboldmath}}$ are exactly zero.
Then $\mathcal{A}(\mbox{\boldmath$\beta$\unboldmath})$ can be estimated by
$\mathcal{A}(\hat{\mbox{\boldmath$\beta$\unboldmath}})=\\{j:\hat{\beta}_{j}\neq
0,j=1,\ldots,p\\}$. The number of false selections of the Lasso is

$\displaystyle\gamma=\\#\\{j:j\in\mathcal{A}(\hat{\mbox{\boldmath$\beta$\unboldmath}})\mbox{
but
}j\notin\mathcal{A}(\mbox{\boldmath$\beta$\unboldmath})\\}+\\#\\{j:j\notin\mathcal{A}(\hat{\mbox{\boldmath$\beta$\unboldmath}})\mbox{
but }j\in\mathcal{A}(\mbox{\boldmath$\beta$\unboldmath})\\},$ (3)

where $\\#$ denotes the set cardinality, the first term counts the number of
false positives and the second term counts the number of false negatives.

The scope of this work is in developing experimental design techniques to
construct the regression matrix $\mathbf{X}$ in (2) in order to minimize the
value of $\gamma$ in (3), the number of false selection. Based on the
probability properties of regularized linear models, it often requires large
randomness in the regression matrix (Jung et al. 2019). Thus, a
straightforward way is to take X to be an independently and identically
distributed (i.i.d.) sample. However, from an experimental design perspective,
the points of an i.i.d. sample is not well stratified in the design space
(Box, Hunter, and Hunter, 2005; Wu and Hamada, 2009). To improve upon this
scheme, we propose to take X to be a nearly orthogonal Latin hypercube design
(NOLHD) that is a Latin hypercube design with nearly orthogonal columns. A
Latin hypercube design is a space-filling design that achieves maximum
uniformity when the points are projected onto any one dimension (McKay,
Beckman, and Conover, 1979). An NOLHD simultaneously possesses two desirable
properties: low-dimensional stratification and nearly orthogonality. Owen
(1992) stated the advantage of using Latin hypercube designs for fitting
additive models. It was discussed in Section 3 of Owen (1992) that the least-
squares estimates of the regression coefficients of an additive regression
with a Latin hypercube design can have significant smaller variability than
their counterparts under an i.i.d. sample. Since the model in (1) is additive,
$\hat{\mbox{\boldmath$\beta$\unboldmath}}$ in (2) associated with a Latin
hypercube design is expected to be superior to that with an i.i.d. sample.
Both random Latin hypercube designs and NOLHDs are popular in computer
experiments (Lin and Tang, 2015). It is advantageous to use NOLHDs instead of
random Latin hypercube designs for the Lasso problem because the former have
guaranteed small columnwise correlations. When the regression matrix X is
taken to be an NOLHD, its small columnwise correlations allow the active
variables less correlated with the inactive variables, thus improving the
selection accuracy of the Lasso.

There is some consistency between the concept of NOLHDs and the sparsity
concept in variable selection. The sparsity assumption we have made earlier
for the model in (1) states that only $p_{0}$ variables in the model are
active and does not specify which $p_{0}$ variables are active. If the
regression matrix X for this model is an NOLHD of $n$ runs for $p$ input
variables, when the points of this design are projected onto any $p_{0}$
dimensions, the resulting design still retains the NOLHD structure for the
$p_{0}$ factors. Note that an NOLHD has more than two levels and spreads the
points evenly in the design space, not restricted to the boundaries only.
Since the number of false selections $\gamma$ in (3) has a nonlinear relation
with the regression matrix X, the use of an NOLHD for the Lasso problem is
more appropriate than a two-level design to exploit this complicated relation
between $\gamma$ and X.

The remainder of the article is organized as follows. Section 2 introduces a
new criterion to measure NOLHDs, and presents two systematic methods for
constructing such designs for the Lasso problem. Section 3 provides numerical
examples to bear out the effectiveness of the proposed method. The numerical
examples in Section 3 clearly indicate the superiority of NOLHDs over two-
level designs for the Lasso problem. We provide a brief discussion in Section
4.

##  2 Methodology

In this section we discuss the construction of NOLHDs and how to use them for
the Lasso problem. We prefer NOLHDs over random Latin hypercube designs
because the latter are not guaranteed to have small columnwise correlations.
An an illustration, let $n=64$ and $p=192$ and compute $\gamma$ in (3) for two
different choices of X in (2). Assume the model in (1) has $\sigma=8$ and
$\mbox{\boldmath$\beta$\unboldmath}=(0.05,0.2,\ldots,3.0,0\ldots,0)^{T}$,
where only the first $20$ coefficients are nonzeros, and the predictor
variables take values on the hypercube $[-(64-1)/2,(64-1)/2]^{p}$. The first
method takes the design matrix X to be a random Latin hypercube design
constructed by (7).

Figure 1: Histogram of the sample correlations of a $64\times 192$ random
Latin hypercube design.

Fig. 1 depicts the histogram of the columnwise sample correlations of one
random Latin hypercube design, where $21\%$ of the columnwise correlations of
the matrix are larger than $0.1$ in absolute values. This method gives
$\gamma=36$. The second method takes the design matrix to be a $64\times 192$
NOLHD from Section 2.1, where the columnwise correlations of the matrix are
very small. The second method gives $\gamma=20$. The difference of $\gamma$
values of the two methods indicates that the Lasso solution with an NOLHD can
be far more superior.

Here are some useful notation and definitions for constructing NOLHDs. The
Kronecker product of an $n\times p$ matrix $\textbf{A}=(a_{ij})$ and an
$m\times q$ matrix $\textbf{B}=(b_{ij})$ is

$\textbf{A}\otimes\textbf{B}=\left[\begin{array}[]{cccc}a_{11}\textbf{B}&a_{12}\textbf{B}&\ldots&a_{1p}\textbf{B}\\\
a_{21}\textbf{B}&a_{22}\textbf{B}&\ldots&a_{2p}\textbf{B}\\\
\vdots&\vdots&\ddots&\vdots\\\
a_{n1}\textbf{B}&a_{n2}\textbf{B}&\ldots&a_{np}\textbf{B}\end{array}\right],$

where $a_{ij}\textbf{B}$ is an $m\times q$ matrix whose ($k,l$) entry is
$a_{ij}b_{kl}$. The correlation matrix of an $n\times p$ matrix
$\textbf{X}=(x_{ij})$ is

$\mbox{\boldmath${\rho}$}=\left(\begin{array}[]{rrrr}\rho_{11}&\rho_{12}&\ldots&\rho_{1p}\\\
\rho_{21}&\rho_{22}&\ldots&\rho_{2p}\\\ \vdots&\vdots&\ddots&\vdots\\\
\rho_{p1}&\rho_{p2}&\ldots&\rho_{pp}\\\ \end{array}\right),$ (4)

where

$\rho_{ij}=\frac{\sum_{k=1}^{n}(x_{ki}-\bar{x}_{i})(x_{kj}-\bar{x}_{j})}{\sqrt{\sum(x_{ki}-\bar{x}_{i})^{2}\sum(x_{kj}-\bar{x}_{j})^{2}}},$
(5)

represents the correlation between the $i$th and $j$th columns of X,
$\bar{x}_{i}=n^{-1}\sum_{k=1}^{n}x_{ki}$ and
$\bar{x}_{j}=n^{-1}\sum_{k=1}^{n}x_{kj}$. The matrix X is orthogonal if
${\rho}$ in (4) is an identity matrix.

Let $\textbf{D}=(d_{ij})$ be an $n\times p$ random Latin hypercube in which
each column is a random permutation of $1,\ldots,n$, and all columns are
generated independently. Using D, a random Latin hypercube design
$\textbf{Z}=(z_{ij})$ on $[0,1]^{p}$ is generated through

$\displaystyle z_{ij}=\frac{d_{ij}-u_{ij}}{n},i=1,\ldots,n;j=1,\ldots,p,$ (6)

where the $u_{ij}$’s are independent uniform random variables on [0,1), and
the $d_{ij}$’s and the $u_{ij}$’s are mutually independent. If Z needs to be
defined on $[a,b]^{p}$ for general $a<b$, rescale $z_{ij}$ in (6) as

$\displaystyle z_{ij}\leftarrow(b-a)z_{ij}+a.$ (7)

We use ${\mbox{\small NOLHD}}(n,p)$ to denote an $n\times p$ NOLHD. For a pre-
specified vector $\textbf{t}=(t_{1},\ldots,t_{q})$ with $0\leq
t_{q}\leq\cdots\leq t_{1}\leq 1$, the orthogonality of an NOLHD X can be
assessed by using the proportion correlation vector given by

$\delta_{\textbf{t}}(\textbf{X})=(\delta_{t_{1}}(\textbf{X}),\ldots,\delta_{t_{q}}(\textbf{X})),$
(8)

where $\delta_{t_{k}}(\textbf{X})=\\{p(p-1)\\}^{-1}\sum_{i=1}^{p}\sum_{j\neq
i}I(|\rho_{ij}|\leq t_{k})$, $k=1,\ldots,q$, and $I(\cdot)$ is an indicator
function. For $k=1,\ldots,q$, this criterion computes the proportion of the
$|\rho_{ij}|$’s not exceeding $t_{k}$. For two designs $\textbf{X}_{1}$ and
$\textbf{X}_{2}$, $\textbf{X}_{1}$ is preferred over $\textbf{X}_{2}$ if
$\delta_{t_{k}}(\textbf{X}_{1})>\delta_{t_{k}}(\textbf{X}_{2})$ for
$t_{1},\ldots,t_{q}$. For the Lasso problem, this new criterion has more
discriminating power than the maximum correlation $\rho_{m}$ and root average
squared correlation $\rho_{ave}$ criteria proposed in Bingham, Sitter, and
Tang (2009), where $\rho_{m}(\textbf{X})=\hbox{max}_{i,j}|\rho_{ij}|$ and and
$\rho_{ave}(\textbf{X})=\\{\sum_{i<j}\rho_{ij}^{2}/[p(p-1)/2]\\}^{1/2}$.
Designs with similar values of $\rho_{m}$ and $\rho_{ave}$ may have different
values of $\delta_{\textbf{t}}$. For illustration, compare a randomly
generated $64\times 192$ i.i.d. sample with an ${\mbox{\small NOLHD}}(64,192)$
from Section 2.1. The former has $\rho_{ave}=0.124$ and $\rho_{m}=0.493$ and
the latter has $\rho_{ave}=0.112$ and $\rho_{m}=0.786$. The two designs are
indistinguishable in terms of $\rho_{ave}$. But for
$\textbf{t}=(0.1,0.05,0.01,0.005)$,
$\delta_{\textbf{t}}=(0.562,0.305,0.064,0.033)$ for the i.i.d. sample and
$\delta_{\textbf{t}}=(0.906,0.894,0.883,0.883)$ for the NOLHD, clearly
indicating the superiority of the latter.

Sections 2.1 and 2.2 present two systematic methods for constructing NOLHDs.
The first method was proposed by Lin, Mukerjee, and Tang (2009) and the second
method is a generalization of the method in Lin et al. (2010). To assist
readers in machine learning who may not be familiar with NOLHDs, we describe
these two methods in a self-contained fashion. These two methods are easy to
implement. Other construction methods for (nearly) orthogonal Latin hypercube
designs include Owen (1994), Tang (1998), Ye (1998), Steinberg and Lin (2006),
Pang, Liu, and Lin (2009), and Sun, Liu, and Lin (2009, 2010), among others.
However, they have run-size constraints and thus we do not consider here. In
the two constructions we will present, an NOLHD with $n$ runs is obtained from
a Latin hypercube in which the $n$ levels in each column are
$\\{-(n-1)/2,\ldots,0,\ldots,(n-1)/2\\}$ if $n$ is odd and
$\\{-(n-1)/2,\ldots,-1/2,1/2,\ldots,(n-1)/2\\}$ if $n$ is even.

###  2.1 A Construction Method Using Orthogonal Arrays

Lin, Mukerjee, and Tang (2009) proposed a method for constructing nearly
orthogonal Latin hypercubes using orthogonal arrays. Recall that an orthogonal
array ${\mbox{\small OA}}(n,p,s)$ of strength two is an $n\times p$ matrix
with levels $1,\ldots,s$ such that, for any two columns, all level
combinations appear equally often (Hedayat, Sloane, and Stufken, 1999). Let A
be an ${\mbox{\small OA}}(s^{2},2f,s)$ and let $\textbf{B}=(b_{ij})$ be an
$s\times p$ Latin hypercube. This method works as follows.

Step 1. For $j=1,\ldots,p$, obtain an $s^{2}\times(2f)$ matrix
$\textbf{A}_{j}$ from A by replacing the symbols $1,\ldots,s$ in the latter by
$b_{1j},\ldots,b_{sj}$, respectively, and partition $\textbf{A}_{j}$ to
$\textbf{A}_{j1},\ldots,\textbf{A}_{jf}$, each of two columns.

Step 2. Let

${\textbf{V}=\left[\begin{array}[]{rr}1&-s\\\ s&1\\\ \end{array}\right].}$

For $j=1,\ldots,p$, obtain an $s^{2}\times(2f)$ matrix

$\textbf{M}_{j}=[\textbf{A}_{j1}\textbf{V},\ldots,\textbf{A}_{jf}\textbf{V}].$

Step 3. For $n=s^{2}$ and $q=2pf$, define an $n\times q$ matrix M =
[$\textbf{M}_{1},\ldots,\textbf{M}_{p}$].

Lemma 1 from Lin, Mukerjee, and Tang (2009) captures the structure of M.

Lemma 1. (a) The matrix M constructed above is an $s^{2}\times(2pf)$ Latin
hypercube.
(b) The correlation matrix of M is
$\mbox{\boldmath${\rho}$}(\textbf{M})=\mbox{\boldmath${\rho}$}(\textbf{B})\otimes\textbf{I}_{2f}$,
where $\textbf{I}_{2f}$ is the identity matrix of order $2f$.

Observe that the proportion correlation $\delta_{t_{k}}$ in (8) of M is

$\delta_{t_{k}}(\textbf{M})=[p(2f-1)+(p-1)\delta_{t_{k}}(\textbf{B})]/(2pf-1),\
\hbox{for}\ k=1,\ldots,q.$ (9)

Example 1. Example Let A be an ${\mbox{\small OA}}(49,8,7)$ from Hedayat,
Sloane, and Stufken (1999) and let B be an ${\mbox{\small NOLHD}}(7,12)$ given
by

${\footnotesize\left(\begin{array}[]{rrrrrrrrrrrr}-3&0&-1&0&3&3&0&-2&1&-3&-1&-3\\\\[-2.0pt]
-2&-1&1&-3&-1&-3&1&-3&-2&-1&1&3\\\\[-2.0pt]
-1&3&0&3&0&-2&2&0&-1&3&-3&-1\\\\[-2.0pt]
0&-2&3&2&-2&2&-2&1&-3&1&2&-2\\\\[-2.0pt]
1&1&-3&-1&-3&1&-3&-1&3&2&0&1\\\\[-2.0pt]
2&-3&-2&1&1&-1&3&2&2&0&3&0\\\\[-2.0pt] 3&2&2&-2&2&0&-1&3&0&-2&-2&2\\\
\end{array}\right),}$

where $\rho_{ave}(\textbf{B})=0.3038$, $\rho_{m}(\textbf{B})=0.9643$, and
$\delta_{\textbf{t}}(\textbf{B})=(\delta_{0.1},\delta_{0.05},\delta_{0.01},\delta_{0.005})$=
$(0.500,0.364,0.136,0.136)$. Here, the matrix M from Lemma 1 is an
${\mbox{\small NOLHD}}(49,96)$ with $\rho_{ave}(\textbf{M})=0.1034$ and
$\rho_{m}(\textbf{M})=0.9643$. From (9),
$\delta_{\textbf{t}}(\textbf{M})=(\delta_{0.1},\delta_{0.05},\delta_{0.01},\\\
\delta_{0.005})$=$(0.942,0.926,0.9,0.9)$. In general, if B is an
${\mbox{\small NOLHD}}(7,p)$, Lemma 1 gives an ${\mbox{\small NOLHD}}(49,8p)$.

###  2.2 A Construction Method Using the Kronecker Product

We now propose a generalization of the method in Lin et al. (2010) for
constructing NOLHDs. This generalization provides designs with better low-
dimensional projection properties than those obtained in Lin et al. (2010).

For $j=1,\ldots,m_{2}$, let $\textbf{C}_{j}=(c_{ik}^{j})$ be an $n_{1}\times
m_{1}$ Latin hypercube and let $\textbf{A}_{j}=(a_{ik}^{j})$ be an
$n_{1}\times m_{1}$ matrix with entries $\pm 1$. Let
$\textbf{B}=(b_{ij})_{n_{2}\times m_{2}}$ be an $n_{2}\times m_{2}$ Latin
hypercube, let $\textbf{D}=(d_{ij})_{n_{2}\times m_{2}}$ be a matrix with
entries $\pm 1$, and let $r$ be a real number. Our proposed method constructs

M
$\displaystyle=\left[\begin{array}[]{rrrr}b_{11}\textbf{A}_{1}+rd_{11}\textbf{C}_{1}&b_{12}\textbf{A}_{2}+rd_{12}\textbf{C}_{2}&\ldots&b_{1m_{2}}\textbf{A}_{m_{2}}+rd_{1m_{2}}\textbf{C}_{m_{2}}\\\
b_{21}\textbf{A}_{1}+rd_{21}\textbf{C}_{1}&b_{22}\textbf{A}_{2}+rd_{22}\textbf{C}_{2}&\ldots&b_{2m_{2}}\textbf{A}_{m_{2}}+rd_{2m_{2}}\textbf{C}_{m_{2}}\\\
\vdots&\vdots&\ddots&\vdots\\\
b_{n_{2}1}\textbf{A}_{1}+rd_{n_{2}1}\textbf{C}_{1}&b_{n_{2}2}\textbf{A}_{2}+rd_{n_{2}2}\textbf{C}_{2}&\ldots&b_{n_{2}m_{2}}\textbf{A}_{m_{2}}+rd_{n_{2}m_{2}}\textbf{C}_{m_{2}}\\\
\end{array}\right].$ (14)

In contrast, the method in Lin et al. (2010) constructs

$\textbf{L}=\textbf{A}\otimes\textbf{B}+r\textbf{C}\otimes\textbf{D},$ (15)

where $\textbf{A}=(a_{ij})_{n_{1}\times m_{1}}$ is a matrix with entries $\pm
1$, $\textbf{C}=(c_{ij})_{n_{1}\times m_{1}}$ is an $n_{1}\times m_{1}$ Latin
hypercube, and B, D and $r$ are as in (14). Lin et al. (2010) provided the
conditions for L to be a nearly orthogonal Latin hypercube. When projected
onto some pairs of predictor variables, points in the design in (15) lie on
straight lines, which may not be desirable for the Lasso problem. Such
projection patterns are due to the use of the same A and the same C for each
entry of B and D in (15). The generalization in (14) uses different
$\textbf{A}_{j}$’s and $\textbf{C}_{j}$’s to eliminate this undesirable
projection pattern. Proposition 1 establishes conditions for M in (14) to be a
Latin hypercube.

######  Proposition 1.

Let $r=n_{2}$. Then the design M in (14) is a Latin hypercube if
one of the following two conditions holds:
(a) For $j=1,\ldots,m_{2}$, the $\textbf{A}_{j}$ and $\textbf{C}_{j}$ satisfy
that for $i=1,\ldots,m_{1}$, $c_{pi}^{j}=-c_{p^{\prime}i}^{j}$ and
$a_{pi}^{j}=a_{p^{\prime}i}^{j}$ hold simultaneously.
(b) For $k=1,\ldots,m_{2}$, the entries of B and D satisfy the condition that
$b_{qk}=-b_{q^{\prime}k}$ and $d_{qk}=d_{q^{\prime}k}$ hold simultaneously.

Proposition 1 can be verified by using an argument similar to the proof of
Lemma 1 in Lin et al. (2010) and thus is omitted. Proposition 2 studies the
orthogonality of M in terms of $\textbf{A}_{j}$’s, B, $\textbf{C}_{j}$’s and
D.

######  Proposition 2.

Suppose $\textbf{A}_{j}$’s, B, $\textbf{C}_{j}$’s, D and $r$ in (14) satisfy
condition (a) or (b) in Proposition 1 and M in (14) is a Latin hypercube. In
addition, assume that $\textbf{A}_{j}$s, B, and D are orthogonal and that
$\textbf{B}^{T}\textbf{D}=0$ or $\textbf{A}_{j}^{T}\textbf{C}_{j}=0$ holds for
all $j$s. Then we have that
(a)
$\rho_{m}(\textbf{M})=\hbox{Max}\\{w_{1}\rho_{m}(\textbf{C}_{j}),j=1,\ldots,m_{2}\\}$,
where $w_{1}=n_{2}^{2}(n_{1}^{2}-1)/(n_{1}^{2}n_{2}^{2}-1)$.
(b)
$\rho_{ave}(\textbf{M})=\sqrt{w_{2}\sum_{j=1}^{m_{2}}\rho_{ave}^{2}(\textbf{C}_{j})/m_{2}}$,
where $w_{2}=(m_{1}-1)w_{1}^{2}/(m_{1}m_{2}-1)]$.
(c)
$\delta_{t_{k}}(\textbf{M})\geq\sum_{j=1}^{m_{2}}\delta_{t_{k}}(\textbf{C}_{j})/m_{2}$
for $k=1,\ldots,q$.
(d) The matrix M is orthogonal if and only if
$\textbf{C}_{1},\ldots,\textbf{C}_{m_{2}}$ are all orthogonal.

###### Proof.

Let $\textbf{M}_{jk}$ and $\textbf{M}_{j^{\prime}k^{\prime}}$ be the
$[(j-1)m_{2}+k]$th and $[(j^{\prime}-1)m_{2}+k^{\prime}]$th columns of M in
(14), respectively. Take $n=n_{1}n_{2}$. Let
$\rho(\textbf{M}_{jk},\textbf{M}_{j^{\prime}k^{\prime}})$ be the correlation
between $\textbf{M}_{jk}$ and $\textbf{M}_{j^{\prime}k^{\prime}}$ defined in
(5). Express ${\footnotesize
12^{-1}n(n^{2}-1)\rho(\textbf{M}_{jk},\textbf{M}_{j^{\prime}k^{\prime}})}$ as

$\sum_{i_{1}=1}^{n_{2}}\sum_{i_{2}=1}^{n_{1}}(b_{i_{1}j}a_{i_{2}k}^{j}+n_{2}d_{i_{1}j}c_{i_{2}k}^{j})(b_{i_{1}j^{\prime}}a_{i_{2}k^{\prime}}^{j^{\prime}}+n_{2}d_{i_{1}j^{\prime}}c_{i_{2}k^{\prime}}^{j^{\prime}}),$

which equals

$\displaystyle\sum_{i_{1}=1}^{n_{2}}b_{i_{1}j}b_{i_{1}j^{\prime}}\sum_{i_{2}=1}^{n_{1}}a_{i_{2}k}^{j}a_{i_{2}k^{\prime}}^{j^{\prime}}+n_{2}\sum_{i_{1}=1}^{n_{2}}d_{i_{1}j}b_{i_{1}j^{\prime}}\sum_{i_{2}=1}^{n_{1}}c_{i_{2}k}^{j}a_{i_{2}k^{\prime}}^{j^{\prime}}$
$\displaystyle\quad+n_{2}\sum_{i_{1}=1}^{n_{2}}b_{i_{1}j}d_{i_{1}j^{\prime}}\sum_{i_{2}=1}^{n_{1}}a_{i_{2}k}^{j}c_{i_{2}k^{\prime}}^{j^{\prime}}+n_{2}^{2}\sum_{i_{1}=1}^{n_{2}}d_{i_{1}j}d_{i_{1}j^{\prime}}\sum_{i_{2}=1}^{n_{1}}c_{i_{2}k}^{j}c_{i_{2}k^{\prime}}^{j^{\prime}}$
$\displaystyle=$
$\displaystyle\sum_{i_{1}=1}^{n_{2}}b_{i_{1}j}b_{i_{1}j^{\prime}}\sum_{i_{2}=1}^{n_{1}}a_{i_{2}k}^{j}a_{i_{2}k^{\prime}}^{j^{\prime}}+n_{2}^{2}\sum_{i_{1}=1}^{n_{2}}d_{i_{1}j}d_{i_{1}j^{\prime}}\sum_{i_{2}=1}^{n_{1}}c_{i_{2}k}^{j}c_{i_{2}k^{\prime}}^{j^{\prime}}.$

Thus, $\rho(\textbf{M}_{jk},\textbf{M}_{j^{\prime}k^{\prime}})$ is zero for
$j\neq j^{\prime}$ and is
$n_{2}^{2}(n_{1}^{2}-1)\rho_{kk^{\prime}}(\textbf{C}_{j})/(n^{2}-1)$ for
$j=j^{\prime}$ and $k\neq k^{\prime}$. By the definitions of $\rho_{m}$ and
$\rho_{ave}$, the results in (a) and (b) hold. Note that for $k=1,\ldots,q$,

$\displaystyle\delta_{t_{k}}(\textbf{M})$ $\displaystyle=$
$\displaystyle\\{m_{2}(m_{2}-1)m_{1}^{2}+m_{1}(m_{1}-1)\sum_{j=1}^{m_{2}}\delta_{t_{k}}(\textbf{C}_{j})\\}/\\{m_{1}m_{2}(m_{1}m_{2}-1)\\}$
$\displaystyle=$
$\displaystyle\sum_{j=1}^{m_{2}}\delta_{t_{k}}(\textbf{C}_{j})/m_{2}+[(m_{2}-1)m_{1}\\{1-\sum_{j=1}^{m_{2}}\delta_{t_{k}}(\textbf{C}_{j})/m_{2}\\}]/(m_{1}m_{2}-1).$

The result in (c) now follows because
$\sum_{j=1}^{m_{2}}\delta_{t_{k}}(\textbf{C}_{j})/m_{2}\leq 1$. By (a), (b)
and (c), (d) is evident. This completes the proof. ∎

Proposition 2 expresses the near orthogonality of M in (14) in terms of that
of $\textbf{C}_{j}$’s and establishes conditions for $\textbf{A}_{j}$, B and D
in order for M to be an orthogonal Latin hypercube. The required matrices in
Proposition 2 can be chosen as follows. First, orthogonal matrices
$\textbf{A}_{j}$’s and D are readily available from Hadamard matrices when
$n_{1}$ and $n_{2}$ are multiples of four. Second, orthogonal Latin hypercubes
B are available from Pang, Liu, and Lin (2009), Lin, Mukerjee, and Tang
(2009), Lin et al. (2010), among others. If $\textbf{A}_{j}$, B and D are
orthogonal, and either $\textbf{B}^{T}\textbf{D}=0$ or
$\textbf{A}_{j}^{T}\textbf{C}_{j}=0$, then M is orthogonal when
$\textbf{C}_{j}$’s are orthogonal Latin hypercubes. If $\textbf{C}_{j}$’s are
NOLHDs like those from Lin, Mukerjee, and Tang (2009) and Lin et al. (2010),
then M is nearly orthogonal. If $\textbf{C}_{1}$ is an NOLHD,
$\textbf{C}_{2},\ldots,\textbf{C}_{m_{2}}$ can be obtained by permuting the
rows of $\textbf{C}_{1}$.

Example 2. Example Let

$\textbf{B}=\frac{1}{2}\left(\begin{array}[]{rrrr}1&-3&7&5\\\\[-2.0pt]
3&1&5&-7\\\\[-2.0pt] 5&-7&-3&-1\\\\[-2.0pt] 7&5&-1&3\\\\[-2.0pt]
-1&3&-7&-5\\\\[-2.0pt] -3&-1&-5&7\\\\[-2.0pt] -5&7&3&1\\\\[-2.0pt]
-7&-5&1&-3\\\
\end{array}\right),\textbf{D}=\left(\begin{array}[]{rrrr}1&1&1&1\\\\[-2.0pt]
1&1&-1&-1\\\\[-2.0pt] 1&-1&1&-1\\\\[-2.0pt] 1&-1&-1&1\\\\[-2.0pt]
-1&1&1&1\\\\[-2.0pt] -1&1&-1&-1\\\\[-2.0pt] -1&-1&1&-1\\\\[-2.0pt]
-1&-1&-1&1\\\ \end{array}\right),$
$\textbf{A}_{1}=\left(\begin{array}[]{rrrrrr}1&1&1&1&1&1\\\\[-2.0pt]
1&-1&1&1&1&-1\\\\[-2.0pt] -1&1&-1&1&1&1\\\\[-2.0pt] -1&-1&1&-1&1&1\\\\[-2.0pt]
1&-1&-1&1&-1&1\\\\[-2.0pt] -1&1&-1&-1&1&-1\\\\[-2.0pt]
-1&-1&1&-1&-1&1\\\\[-2.0pt] -1&-1&-1&1&-1&-1\\\\[-2.0pt]
1&-1&-1&-1&1&-1\\\\[-2.0pt] 1&1&-1&-1&-1&1\\\\[-2.0pt]
1&1&1&-1&-1&-1\\\\[-2.0pt] -1&1&1&1&-1&-1\\\ \end{array}\right),\hbox{ and
}\textbf{C}_{1}=\frac{1}{2}\left(\begin{array}[]{rrrrrr}-11&-9&9&11&5&1\\\\[-2.0pt]
-9&5&-1&-5&-9&11\\\\[-2.0pt] -7&11&-3&3&1&-7\\\\[-2.0pt]
-5&-1&-9&-9&-1&-9\\\\[-2.0pt] -3&-7&5&-11&7&-1\\\\[-2.0pt]
-1&9&-7&5&9&5\\\\[-2.0pt] 1&-3&7&-7&-7&3\\\\[-2.0pt]
3&-11&-11&9&-11&-3\\\\[-2.0pt] 5&7&11&7&-5&-5\\\\[-2.0pt]
7&-5&-5&1&11&7\\\\[-2.0pt] 9&1&3&-3&3&-11\\\\[-2.0pt] 11&3&1&-1&-3&9\\\
\end{array}\right).$

For $j=2,3,4$, obtain $\textbf{A}_{j}$ and $\textbf{C}_{j}$ by permuting the
rows of $\textbf{A}_{1}$ and $\textbf{C}_{1}$, respectively. Using the above
matrices, M in (14) is a $96\times 24$ orthogonal Latin hypercube.

Example 3. Example Let $\textbf{C}_{1}$ be an ${\mbox{\small NOLHD}}(25,24)$
constructed by Lemma 1 using an ${\mbox{\small OA}}(25,6,5)$ from Hedayat,
Sloane, and Stufken (1999) and an ${\mbox{\small NOLHD}}(5,4)$. Permute the
rows of $\textbf{C}_{1}$ to get an NOLHD $\textbf{C}_{2}$. Generate two
$25\times 24$ nearly orthogonal matrices, $\textbf{A}_{1}$ and
$\textbf{A}_{2}$, by using the Gendex DOE software associated with Nguyen
(1996). Using

$\textbf{B}=\left(\begin{array}[]{rr}\frac{1}{2}&-\frac{1}{2}\\\
-\frac{1}{2}&\frac{1}{2}\end{array}\right)\hbox{ and
}\textbf{D}=\left(\begin{array}[]{rr}1&1\\\ 1&1\end{array}\right),$

M in (14) is an ${\mbox{\small NOLHD}}(50,48)$.

##  3 Numerical Illustration

In this section we provide numerical examples to compare the number of false
selections $\gamma$ in (3) with four different types of design matrices.
Method I uses an NOLHD from Section 2. Method II uses a two-level design at
levels $\pm(n-1)/2$. If $p>n-1$, a two-level design is often called a
supersaturated design (Lin, 1993; Wu, 1993). Method III uses a random Latin
hypercube design (RLHD) constructed in (7). Method IV uses an i.i.d. sample.
Denote by $\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$
the $\gamma$ values of these methods, respectively. Since the focus here is to
compare the effect of the regression matrix X on the accuracy of the Lasso
solution, the response vector $y$ from the model in (1) is generated with the
same
$\mbox{\boldmath$\epsilon$\unboldmath}=(\epsilon_{1},\ldots,\epsilon_{n})^{T}$
for the four methods. The tuning parameter $\lambda$ in (2) is selected by the
five-fold cross-validation. The package lars (Efron, Hastie, Johnstone, and
Tibshirani, 2003) in R (R, 2010) is used to compute the Lasso solution
$\hat{\mbox{\boldmath$\beta$\unboldmath}}$ in (2). Examples below have
different $p/n$ ratios.

Example 4. Example For the model in (1), let $p=48$, $\sigma=8$, and
$\mbox{\boldmath$\beta$\unboldmath}=(0.8,1.0,\ldots,3,0,\ldots,0)^{T}$ with
the last $36$ coefficients being zero. Take $n=50$ with $n\approx p$. Method I
takes the ${\mbox{\small NOLHD}}(50,48)$ in Example 2. Method II uses a
$50\times 48$ nearly orthogonal two-level design from the Gendex software
based on the algorithm in Nguyen (1996). Table 1 compares three quartiles of
the $\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$ values
over 50 replications. Fig. 2 depicts the boxplots of $\gamma$ values of these
methods. Table 1 and Fig. 2 clearly indicate that $\gamma_{NOLHD}$ is smaller
than $\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$.

Table 1: Three quartiles of the $\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$ values over 50 replications for Example 3 | NOLHD | FD | RLHD | IID
---|---|---|---|---
median | 13.00 | 18.00 | 18.00 | 20.00
1st quartile | 12.00 | 14.00 | 15.00 | 16.00
3rd quartile | 15.00 | 21.00 | 23.00 | 23.00

Figure 2: Boxplots of the $\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and
$\gamma_{IID}$ values over the 50 replications for Example 3.

Example 5. Example For the model in (1), let $p=96$, $\sigma=8$ and
$\mbox{\boldmath$\beta$\unboldmath}=(0.2,0.4,\ldots,3,0,\ldots,0)^{T}$ with
the last $81$ coefficients being zero. Take $n=49$ with $p>n$. Method I uses
the ${\mbox{\small NOLHD}}(49,96)$ in Example 2.1. Method II uses an
$E(s^{2})$-optimal supersaturated design from the Gendex software associated
with Nguyen (1996). Table 2 compares three quartiles of the $\gamma_{NOLHD}$,
$\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$ values over 50 replications.
Fig. 3 depicts the boxplots of $\gamma$ values of these methods. Table 2 and
Fig. 3 show that $\gamma_{NOLHD}$, once more, significantly outperforms
$\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$.

Table 2: Three quartiles of the $\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$ values over 50 replications for Example 2. | NOLHD | FD | RLHD | IID
---|---|---|---|---
median | 17.50 | 27.00 | 25.00 | 27.00
1st quartile | 15.00 | 24.00 | 22.25 | 23.25
3rd quartile | 22.75 | 30.00 | 28.00 | 29.00

Figure 3: Boxplots of the $\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and
$\gamma_{IID}$ values over the 50 replications for Example 2.

Example 6. Example For the model in (1), let $p=192$, $\sigma=8$ and
$\mbox{\boldmath$\beta$\unboldmath}=(0.05,0.2,\ldots,3,0,\ldots,0)^{T}$ with
the last $172$ coefficients being zero. Take $n=64$ with $p>n$. Method I uses
an ${\mbox{\small NOLHD}}(64,192)$ from Lemma 1 in Section 2.1. Method II uses
an $E(s^{2})$-optimal supersaturated design from the Gendex software
associated with Nguyen (1996). Table 3 compares three quartiles of the
$\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$ values
over 50 replications. Fig. 4 depicts the boxplots of $\gamma$ values for these
methods, where $\gamma_{NOLHD}$ is much smaller than $\gamma_{FD}$,
$\gamma_{RLHD}$ and $\gamma_{IID}$. This example clearly demonstrates that the
use of an NOLHD leads to significant improvement of the Lasso solution.

Table 3: Three quartiles of the $\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and $\gamma_{IID}$ values over 50 replications in Example 3.  | NOLHD | FD | RLHD | IID
---|---|---|---|---
median | 27.00 | 42.50 | 43.00 | 41.00
1st quartile | 23.00 | 40.00 | 34.25 | 34.00
3rd quartile | 33.00 | 46.00 | 45.00 | 45.00

Figure 4: Boxplots of the $\gamma_{NOLHD}$, $\gamma_{FD}$, $\gamma_{RLHD}$ and
$\gamma_{IID}$ values over the 50 replications for Example 3.

These examples suggest that the Lasso solution with an NOLHD is more accurate
than those of the competing designs. Comparison of Fig. 2– Fig. 4 indicates
that the advantage of using NOLHDs in the Lasso problem grows as the ratio
$p/n$ increases.

##  4 Discussion

We have proposed a method using NOLHDs from computer experiments to
significantly enhance the variable selection accuracy of the Lasso procedure.
The effectiveness of this method has been successfully illustrated by several
examples. Design construction for the regularized linear models is a new
research direction in design of experiments, which can be applied in many
areas, such comprehensive sensing (Song et al., 2016; Jung et al. 2019), and
actuator placement (Du et al., 2019). As an alternative to the proposed
method, one may develop a model-based optimal design approach by extending the
ideas of Meyer, Steinberg, and Box (1996) and Bingham and Chipman (2007).
Because the Lasso solution in (2) does not admit an analytic form, a potential
difficulty in developing such an approach is to introduce a sensible and
computationally efficient criterion for the Lasso problem. It will be of
interest in a subsequent project to study the proposed design strategy for
variants of the Lasso. A R package for the proposed method is under
development and will be released in the future.

## References

  * [1] Bingham, D., and Chipman, H. A. (2007), Incorporating Prior Information in Optimal Design for Model Selection, Technometrics, 49, 155–163.
  * [2] Bingham, D., Sitter, R. R., and Tang, B. (2009), Orthogonal and Nearly Orthogonal Designs for Computer Experiments, Biometrika, 96, 51–65.
  * [3] Box, G. E. P., Hunter, W. G., and Hunter, J. S. (2005), Statistics for Experimenters: Design, Innovation, and Discovery, 2nd Edition, New York: John Wiley & Sons.
  * [4] Du, J., Yue, X., Hunt, J. H., and Shi, J. (2019). Optimal Placement of Actuators Via Sparse Learning for Composite Fuselage Shape Control. Journal of Manufacturing Science and Engineering, 141(10) 101004\.
  * [5] Efron, B., Hastie, T., Johnstone, I., and Tibshirani R. (2003), Least Angle Regression, Annals of Statistics, 32, 407–499.
  * [6] Fan J, Li G, and Li R. (2005) An overview on variable selection for survival analysis. InContemporary Multivariate Analysis And Design Of Experiments: In Celebration of Professor Kai-Tai Fang’s 65th Birthday, 315–336.
  * [7] Hedayat, A. S., Sloane, N. J. A., and Stufken, J. (1999), Orthogonal Arrays: Theory and Applications, New York: Springer-Verlag.
  * [8] Heinze, G., Wallisch, C. and Dunkler, D. (2018), Variable Selection - A Review and Recommendations for the Practicing Statistician. Biometrical Journal, 60, 431–449.
  * [9] Hesterberg, T., Choi, N.H., Meier, L. and Fraley, C. (2008), Least Angle and $l_{1}$ Penalized Regression: A Review. Statistics Surveys, 2, 61–93.
  * [10] Huang, J., Breheny, P. and Ma, S. (2012), A Selective Review of Group Selection in High-Dimensional Models. Statistical Science, 27.
  * [11] Jung, P., Kueng, R., and Mixon, D. G. (2019). Derandomizing Compressed Sensing with Combinatorial Design. Frontiers in Applied Mathematics and Statistics, 5, 26.
  * [12] Lin, D. K. J. (1993), A New Class of Supersaturated Designs, Technometrics, 35, 28–31.
  * [13]
  * [14] Lin, C. D., Bingham, D., Sitter, R. R., and Tang, B. (2010), A New and Flexible Method for Constructing Designs for Computer Experiments, Annals of Statistics, 38, 1460–1477.
  * [15] Lin, C. D., Mukerjee, R., and Tang, B. (2009), Construction of Orthogonal and Nearly Orthogonal Latin Hypercubes, Biometrika, 96, 243–247.
  * [16] Lin, C.D. and Tang, B. (2015). Latin Hypercubes and Space-filling Designs, Handbook of Design and Analysis of Experiments, Bingham, D., Dean, A., Morris, M., and Stufken, J. ed. 593–626, CRC Press.
  * [17] McKay, M. D., Beckman, R. J., and Conover, W. J. (1979), A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code, Technometrics, 21, 239–245.
  * [18] Meyer, R. D., Steinberg, D. M., and Box, G. E. P. (1996), Follow-up Designs to Resolve Confounding in Multifactor Experiments, Technometrics, 38, 303–313.
  * [19] Nguyen, N. (1996), A Note on Constructing Near-Orthogonal Arrays with Economic Run Size, Technometrics, 38, 279–283.
  * [20] Owen, A. B. (1992), A Central Limit Theorem for Latin Hypercube Sampling. Journal of the Royal Statistical Society, Series B, 54, 541–551.
  * [21] Owen, A. B. (1994), Controlling Correlations in Latin Hypercube Samples, Journal of the American Statistical Association, 89, 1517–1522.
  * [22] Pang, F., Liu, M. Q., and Lin, D. K. J. (2009), A Construction Method for Orthogonal Latin Hypercube Designs with Prime Power Levels, Statistica Sinica, 19, 1721–1728.
  * [23] R (2010), The R Project for Statistical Computing.
  * [24] Song, P., Mota, J. F., Deligiannis, N., and Rodrigues, M. R. D. (2016). Measurement Matrix Design for Compressive Sensing with Side Information at the Encoder. In 2016 IEEE Statistical Signal Processing Workshop (SSP), DOI: 10.1109/SSP.2016.7551810.
  * [25] Steinberg, D. M., and Lin, D. K. J. (2006), A Construction Method for Orthogonal Latin Hypercube Designs, Biometrika, 93, 279–288.
  * [26] Sun, F., Liu, M. Q., and Lin, D. K. J. (2009), Construction of Orthogonal Latin Hypercube Designs, Biometrika, 96, 971–974.
  * [27] Sun, F., Liu, M. Q., and Lin, D. K. J. (2010), Construction of Orthogonal Latin Hypercube Designs with Flexible Run Sizes, Journal of Statistical Planning and Inference, 140, 3236–3242.
  * [28] Tang, B. (1998), Selecting Latin Hypercubes Using Correlation Criteria, Statistica Sinica, 8, 965–977.
  * [29] Tibshirani, R. (1996), Regression Shrinkage and Selection via the Lasso, Journal of the Royal Statistical Society, Series B, 58, 267–288.
  * [30] Tibshirani, R. J., and Taylor, J. (2011). The Solution Path of the Generalized Lasso. The Annals of Statistics, 39(3), 1335–1371.
  * [31] Wu, C. F. J. (1993), Construction of Supersaturated Designs through Partially Aliased Interactions, Biometrika, 80, 661–669.
  * [32] Wu, C. F. J., and Hamada, M. (2009), Experiments: Planning, Analysis, and Parameter Design Optimization, 2nd Edition, New York: John Wiley & Sons.
  * [33] Ye, K. Q. (1998), Orthogonal Column Latin Hypercubes and Their Application in Computer Experiments, Journal of the American Statistical Association, 93, 1430–1439.
  * [34] Zhao, P., and Yu, B. (2006), On Model Selection Consistency of the Lasso, Journal of Machine Learning Research, 7, 2541–2567.
  * [35] Zhao, P., Rocha, G., and Yu, B. (2009). Grouped and Hierarchical Model Selection through Composite Absolute Penalties.  Annals of Statistics, 37(6A), 3468–3497.
  * [36] Zou, H. (2006). The Adaptive Lasso and its Oracle Properties. Journal of the American Statistical Association, 101, 1418–1429.
  * [37] Zou, H. and Hastie, T. (2005). Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society: Series B, 67, 301–320.