# Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large
Language Models

Bowen Ping1∗ Shuo Wang2 Hanqing Wang3 Xu Han2 Yuzhuang Xu2 Yukun Yan2
Yun Chen3 Baobao Chang1 Zhiyuan Liu2† Maosong Sun2
1Peking University 2Dept. of Comp. Sci. & Tech., Tsinghua University, Beijing,
China
3Shanghai University of Finance and Economics  Equal contribution.
Corresponding authors.

###### Abstract

Fine-tuning is a crucial process for adapting large language models (LLMs) to
diverse applications. In certain scenarios, such as multi-tenant serving,
deploying multiple LLMs becomes necessary to meet complex demands. Recent
studies suggest decomposing a fine-tuned LLM into a base model and
corresponding delta weights, which are then compressed using low-rank or low-
bit approaches to reduce costs. In this work, we observe that existing low-
rank and low-bit compression methods can significantly harm the model
performance for task-specific fine-tuned LLMs (e.g., WizardMath for math
problems). Motivated by the long-tail distribution of singular values in the
delta weights, we propose a delta quantization approach using mixed-precision.
This method employs higher-bit representation for singular vectors
corresponding to larger singular values. We evaluate our approach on various
fine-tuned LLMs, including math LLMs, code LLMs, chat LLMs, and even VLMs.
Experimental results demonstrate that our approach performs comparably to full
fine-tuned LLMs, surpassing both low-rank and low-bit baselines by a
considerable margin. Additionally, we show that our method is compatible with
various backbone LLMs, such as Llama-2, Llama-3, and Mistral, highlighting its
generalizability.

##  1 Introduction

Figure 1:  Left: illustration of BitDelta (Liu et al., 2024b), which employs
1-bit quantization for all the delta weights. Middle: illustration of low-rank
compression (Ryu et al., 2023b), retaining the top-$k$ singular values and the
corresponding singular vectors. Right: illustration of the proposed Delta-CoMe
method, which represents the singular vectors of larger singular values using
high-bit vectors while compressing the singular vectors of smaller singular
values into low-bit representations. This method is inspired by the long-tail
distribution of singular values in delta weights.

Large language models (LLMs) (Touvron et al., 2023; Jiang et al., 2023) are
increasingly becoming the standard for a wide range of downstream tasks (Luo
et al., 2023a; Yu et al., 2023; Wei et al., 2023; Luo et al., 2023b; Liu et
al., 2024a; Wang et al., 2023), significantly surpassing conventional small
models. To meet the demands of various application domains and scenarios, many
researchers direct their attention to developing advanced alignment or
adaptation algorithms together with diverse training data to learn aligned
LLMs based on generally pre-trained models. For instance, Luo et al. (2023a)
propose a reinforcement learning from evol-instruct feedback (RLEIF) method to
construct LLMs with strong mathematical reasoning abilities. Similarly, Yu et
al. (2023) employ a bootstrapping method to diversify mathematical questions
and then fine-tune open-source LLMs to build mathematical models. For code
generation, Luo et al. (2023b) adapt the evol-instruct method to the coding
domain, resulting in the WizardCoder model, which demonstrates superior coding
abilities compared to generally trained LLMs. Additionally, Wei et al. (2023)
enhance the capabilities of open-source code LLMs by using automatically
generated high-quality instruction data based on existing code snippets. Wang
et al. (2023) utilize various resources of mixed quality and design a new
conditioned reinforcement learning fine-tuning method to train the OpenChat
model. Beyond the text modality, some studies propose fine-tuning pre-trained
LLMs to understand other modalities. For instance, Liu et al. (2024a)
construct a multi-modal instruction tuning dataset and develop the Llava
model, which can understand both text and images.

Building on the aforementioned alignment approaches, LLMs are endowed with
specialized capabilities that align with distinct user demands and application
requirements (Liu et al., 2024b). In certain scenarios, deploying multiple
LLMs with different abilities is necessary. For example, in multi-tenant
serving, different LLMs may be needed to satisfy various users. Additionally,
some complex tasks consist of multiple sub-tasks, each requiring different
model capabilities. To address these tasks, we should organize and deploy a
group of LLMs simultaneously. A straightforward question arises: why not use a
single general LLM that encompasses all the necessary capabilities? For
example, we could develop one model that can both understand images and
generate code programs. To our knowledge, LLMs with various capabilities
(e.g., GPT-4111https://chatgpt.com) typically have an enormous number of
parameters, making them impractical for resource-limited situations (e.g.,
edge-side scenarios).

In pursuit of this objective, a field of research advocates for the
minimization of expenses associated with multi-model serving. Delta-
compression emerges as a crucial and viable approach in this context, offering
the potential to decrease both storage requirements and GPU memory utilization
in scenarios involving multiple models. The primary objective of delta-
compression is to minimize the size of the delta weights between aligned and
pre-trained LLMs (e.g., Llama-2-Chat and Llama-2). Ryu et al. (2023b) identify
the low-rank nature of delta weights and enhance storage efficiency through
low-rank approximation. Alternatively, Liu et al. (2024b) propose a 1-bit
quantization approach, termed BitDelta, to further reduce the size of delta
weights. They validate the effectiveness of BitDelta across various chat
models, including Llama-2-Chat (Touvron et al., 2023),
Vicuna222https://lmsys.org/blog/2023-03-30-vicuna, and WizardLM (Xu et al.,
2023). In this work, we reassess the performance of both low-rank and low-bit
delta-compression methods across a diverse range of aligned LLMs, encompassing
mathematical, coding, chat, and multi-modal LLMs. Our experimental results
(e.g., Table 3) reveal that current low-rank and low-bit compression
techniques may significantly degrade the performance of aligned LLMs. These
results motivate us to explore more advanced delta-compression methods capable
of achieving performance nearly equivalent to the aligned LLMs before
compression.

Inspired by the long-tail distribution of singular values, as illustrated in
Figure 1, we propose allocating higher-bit representations for singular
vectors associated with larger singular values, given their greater impact on
the approximation of delta weights prior to compression. Conversely, for
singular vectors associated with smaller singular values, we employ low-bit
formats to reduce the delta size. For singular values that are extremely
small, we omit the corresponding singular vectors altogether. The resulting
method, which we term Delta-CoMe, can be viewed as a hybrid of low-rank and
low-bit compression techniques. Delta-CoMe outperforms both the low-rank
compression method and BitDelta. Moreover, our method achieves performance
comparable to that of the full aligned LLMs. For instance, Delta-CoMe attains
an average score of 53.2 across eight representative tasks, closely matching
the average score of 53.5 achieved by the aligned LLMs. In comparison, the
scores of the low-rank and low-bit baselines are 47.8 and 49.3, respectively.

Finally, we compare the performance of the involved delta-compression methods
to LoRA (Hu et al., 2022), a widely-used delta-tuning approach. The primary
distinction between delta-compression and delta-tuning is that delta-
compression first optimizes the full model and then converts the modified
weights into a lightweight module, reducing inference costs in multi-model
settings. In contrast, delta-tuning primarily aims to lower training costs.
Our experimental results demonstrate that the proposed Delta-CoMe method
significantly outperforms LoRA, with scores of 41.9 versus 29.8, respectively.
These results suggest that delta-compression can deliver superior performance
in multi-model settings compared to delta-tuning.

Our contribution can summarized as follows:

  * •

We propose a mixed-precision delta-compression method that employs varying
bit-widths for different singular vectors based on their singular values;

  * •

We validate the effectiveness of the proposed method across different types of
aligned LLMs of varying sizes, including mathematical, coding, chat, and
multi-modal LLMs;

  * •

We conduct in-depth analyses to understand the superior performance of our
method over low-rank and low-bit baselines. Our method can also outperform
delta-tuning approaches such as LoRA, demonstrating that the proposed delta-
compression method is more practical for multi-model serving scenarios.

##  2 Related Work

###  2.1 Delta-Compression

Recently, delta-compression has garnered increasing interest in the LLM
community due to its ability to substantially diminish the storage and
inference expenses associated with serving multiple models. GPT-Zip extends
the GPTQ approach (Frantar et al., 2023) to compress the delta weights between
aligned models and the backbone model, successfully using 2-bit delta weights
to approximate the model. Additionally, they sparsify the quantized delta
weights to further reduce storage costs. However, the sparsification technique
can hardly reduce GPU memory usage during inference. Similarly, Yu et al.
(2024) find that dropping the majority of the delta weights has a limited
effect on the performance of aligned LLMs. Ryu et al. (2023a) identify the
low-rank property of delta weights and propose reducing the storage
requirements of aligned LLMs through low-rank approximation. Yao & Klimovic
(2023) adopt the concept of delta-compression to develop a multi-tenant
serving system, DeltaZip. Most recently, Liu et al. (2024b) introduced
BitDelta, which successfully quantizes the delta weights into 1-bit. However,
they only examined the performance of this compression method using chat LLMs,
leaving a wide range of other types of aligned LLMs unexplored. In this work,
we propose leveraging the benefits of both low-rank and low-bit compression
methods by using varying bit-widths to represent different components of the
delta weights. We evaluate representative low-rank and low-bit delta-
compression methods across various types of aligned LLMs to provide a
comprehensive comparison of these methods.

###  2.2 Model Compression with Mix-Precision

Using mixed-precision to compress the model weights is an effective technique
that has been investigated in many previous studies. SpQR (Dettmers et al.,
2023) isolates a small number of outlier weights and retains them with high-
precision, while keeping the other weights at low-precision, resulting in a
significant performance improvement. Based on activations, Agile-Quant (Shen
et al., 2024) utilizes token pruning to achieve mixed-precision quantization
of both weights and activations. Bablani et al. (2023) propose employing
varying bit-widths for different layers of the model, while Yao et al. (2021)
propose quantizing activations and model weights with different precisions. In
this work, we propose using mixed-precision compression for different singular
vectors of the delta model, marking the first method to introduce mixed-
precision compression for delta weights.

##  3 Approach

Figure 2:  Illustration of Delta-CoMe, where we utilize varying bit-widths for
singular vectors with different singular values. Singular vectors
corresponding to larger singular values are assigned higher bit-widths. For
extremely small singular values, we omit the singular vectors (i.e., 0-bit).

###  3.1 Preliminaries

For a backbone LLM $\bm{\theta}_{b}$, we can customize it into an aligned
model $\bm{\theta}_{a}$ for a specific purpose using advanced alignment
algorithms (Xu et al., 2023; Luo et al., 2023a; Yu et al., 2023; Luo et al.,
2023b; Wei et al., 2023; Liu et al., 2024a). In some practical scenarios, as
mentioned in Section 1, we may need to deploy multiple LLMs at the same time.
Formally, we should store and deploy a series of aligned LLMs
$\left\\{\bm{\theta}_{a}^{(1)},\cdots,\bm{\theta}_{a}^{(N)}\right\\}$, where
$N$ is the number of aligned models. The total size of the group of aligned
models is $N\times M$, where $M$ is the size of one model. We use
$\bm{\Delta}$ to represent the delta weights between the aligned model and the
backbone model, which is given by

$\bm{\Delta}^{(n)}=\bm{\theta}_{a}^{(n)}-\bm{\theta}_{b},$ (1)

where $\bm{\theta}^{(n)}$ is the $n$-th aligned LLM. Note that the sizes of
$\bm{\Delta}^{(n)}$, $\bm{\theta}_{a}^{(n)}$, and $\bm{\theta}_{b}$ are the
same.

Delta-compression aims to compress the delta weights $\bm{\Delta}^{(n)}$ into
$\hat{\bm{\Delta}}^{(n)}$, where the latter has significantly fewer
parameters. After delta-compression, we can only maintain one backbone model
and $N$ compressed delta models:
$\left\\{\bm{\theta}_{b},\hat{\bm{\Delta}}^{(1)},\cdots,\hat{\bm{\Delta}}^{(N)}\right\\}$.
The total size is decreased from $N\times M$ to $(1+\alpha N)\times M$, where
$\alpha$ is the compression ratio. During inference, we can restore each
aligned LLM in the following way:

$\hat{\bm{\theta}}_{a}^{(n)}=\bm{\theta}_{b}+\hat{\bm{\Delta}}^{(n)}.$ (2)

For a good delta-compression method, we expect it can achieve a smaller
$\alpha$, while making $\hat{\bm{\theta}}_{a}^{(n)}$ attain comparable
performance with $\bm{\theta}_{a}^{(n)}$. BitDelta (Liu et al., 2024b), to our
knowledge, is the most recent study that successfully quantizes delta weights
into 1-bit, which means that $\alpha=1/16$ when the original aligned model is
represented by FP16 or BF16. In this work, we propose to improve the
performance of delta-compression methods by inducing mixed-precision
quantization, which will be detailed in the following sub-sections.

###  3.2 Delta Decomposition

Previous works have investigated mixed-precision model compression methods at
the token (Shen et al., 2024) or layer level (Bablani et al., 2023). For
delta-compression, we propose employing mixed-precision for different singular
vectors. We first use the SVD algorithm to decompose each delta matrix:

$\Delta\mathbf{W}=\mathbf{U}\bm{\Sigma}\mathbf{V}^{\top},$ (3)

where $\Delta\mathbf{W}\in\mathbb{R}^{h_{\mathrm{out}}\times
h_{\mathrm{in}}}$, $\mathbf{U}\in\mathbb{R}^{h_{\mathrm{out}}\times
h_{\mathrm{out}}}$, $\bm{\Sigma}\in\mathbb{R}^{h_{\mathrm{out}}\times
h_{\mathrm{in}}}$, $\mathbf{V}\in\mathbb{R}^{h_{\mathrm{in}}\times
h_{\mathrm{in}}}$. Intuitively, the singular vectors associated with larger
singular values have a greater impact on the approximation of the delta matrix
$\Delta\mathbf{W}$, we thus spend more bits for these vectors to reduce the
quantization error.

###  3.3 Mixed-Precision Quantization

Some representative quantization methods, such as GPTQ (Frantar et al., 2023),
aims to minimize the following objective:

$\hat{\mathbf{W}}=\mathrm{Quant}_{k}(\mathbf{W},\mathbf{X})=\mathop{\mathrm{argmin}}_{\hat{\mathbf{W}}}||\mathbf{W}\mathbf{X}-\hat{\mathbf{W}}\mathbf{X}||^{2},$
(4)

where $\mathbf{X}\in\mathbb{R}^{h_{\mathrm{in}}}$ is the input to the
parameter $\mathbf{W}$ and $\hat{\mathbf{W}}$ is the corresponding quantized
parameter. We use $\mathrm{Quant}_{k}$ to denote the $k$-bit quantization
algorithm. In this work, we employ the widely-used GPTQ method with
$\textrm{group\\_size}=128$ for cases where $k>1$, and BitDelta for 1-bit
quantization. For a certain group of singular vectors, let
$r_{\mathrm{begin}}$ and $r_{\mathrm{end}}$ represent the start and end
indices, respectively. The quantization of the singular vectors can be given
by

$\begin{split}\hat{\mathbf{V}}[:,r_{\mathrm{begin}}:r_{\mathrm{end}}]^{\top}&=\mathrm{Quant}_{k}(\mathbf{V}[:,r_{\mathrm{begin}}:r_{\mathrm{end}}]^{\top},\mathbf{X}),\\\
\hat{\mathbf{U}}[:,r_{\mathrm{begin}}:r_{\mathrm{end}}]&=\\\
\mathrm{Quant}_{k}&(\mathbf{U}[:,r_{\mathrm{begin}}:r_{\mathrm{end}}],\bm{\Sigma}[r_{\mathrm{begin}}:r_{\mathrm{end}},r_{\mathrm{begin}}:r_{\mathrm{end}}]\hat{\mathbf{V}}[:,r_{\mathrm{begin}}:r_{\mathrm{end}}]^{\top}\mathbf{X}).\\\
\end{split}$ (5)

As illustrated in Figure 2, we use varying quantization bits for different
groups of singular vectors. By employing different mixed-precision strategies,
we can control the trade-off between achieving a small delta size and
maintaining high performance. We will provide more details about the
exploration of the mixing strategy in Section 5.1.

##  4 Experimental Setup

To thoroughly investigate the proposed delta-compression method Delta-CoMe and
the involved baselines, we examine the performance of different methods across
several tasks, which are typical applications of recent aligned LLMs.

###  4.1 Tasks

#### Mathematical Problem Solving

Solving mathematical problems is a challenging task for modern LLMs. For this
task, we employ GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021)
as the evaluation datasets, which are among the most popular mathematical
benchmarks for LLMs. The reported score is accuracy, which is estimated by
comparing the ground-truth number with the result calculated by the model.

#### Code Generation

The ability to process code is crucial for numerous practical applications,
including data analysis and LLM-based agents. For this task, we use HumanEval
(Chen et al., 2021) and MBPP (Austin et al., 2021) as the evaluation datasets,
which are widely used in recent studies. The reported score is the pass rate,
indicating that the model-generated code can successfully run the test cases
in one pass (i.e., pass@1).

#### Chat

The chat ability enables LLMs to interact with users, providing helpful and
safe suggestions or responses based on the user’s requests. A good chat model
is expected to be well aligned with human preferences. For evaluating chat
LLMs, we select TruthfulQA (Lin et al., 2022) and SafetyBench (Zhang et al.,
2023) as the evaluation datasets, which measure helpfulness and safety,
respectively. The reported score is the accuracy, indicating that the choice
of the model is correct.

#### Multi-Modal Chat

Vision-language models (VLMs) are attracting increasing attention due to their
ability to process both text and images. Most recent VLMs are based on pre-
trained visual encoders and language models, with the language models fine-
tuned to understand the visual signal. For this task, we use GQA (Hudson &
Manning, 2019) and TextVQA (Singh et al., 2019). The reported score is the
accuracy, indicating that the choice of the model is correct.

###  4.2 Models

Table 1: Selected backbone and aligned models for the examined four tasks.  Task | 7B Models | 13B Models
---|---|---
Backbone | Aligned | Backbone | Aligend
Math | Llama-2 | WizardMath-v1.0 | Llama-2 | WizardMath-v1.0
Code | CodeLlama-Py | MagicoderS-CL | CodeLlama-Py | WizardCoder-Py-v1.0
Chat | Llama-2 | Llama-2-chat | Llama-2 | Llama-2-chat
Multi-Modal | Vicuna-v1.5 | Llava-v1.5 | Vicuna-v1.5 | Llava-v1.5

For the four tasks, we provide the backbone and aligned models in Table 1. All
the model weights are open-sourced by the authors. We use both 7B and 13B
models to make a thorough comparison between different delta-compression
models. During inference, we use greedy search.

###  4.3 Baselines

We employ two representative baselines, including SVD-based low-rank
compression and BitDelta (Liu et al., 2024b). For the low-rank baseline, we
re-implement the method, while for BitDelta, we use the code open-sourced by
the authors.333https://github.com/FasterDecoding/BitDelta. All methods are
evaluated on NVIDIA A100 GPUs.

##  5 Experimental Results

###  5.1 Exploration of Mixed-Precision Strategies

Table 2: Comparison of different mixed-precision strategies.  # Precision | Setting | GSM8K
---|---|---
Single | 1 | 45.6
2 | 50.6
3 | 51.8
4 | 51.6
8 | 47.8
16 | 43.3
Double | 16 + 3 | 52.5
8 + 3 | 53.1
4 + 3 | 52.2
3 + 2 | 52.3
Triple | 16 + 8 + 3 | 53.2
8 + 4 + 3 | 52.2
8 + 3 + 2 | 53.6

To determine which bit-width to use and how many singular vectors to quantize,
we conduct a preliminary experiment using different mixed-precision
strategies. We examine three types of strategies: single-precision, double-
precision, and triple-precision settings. The size of the compressed delta
remains consistent across all settings. For single-precision compression, we
set $r_{\mathrm{begin}}$ to 0, and $r_{\mathrm{end}}$ is set to guarantee that
the delta size is the same as BitDelta (Liu et al., 2024b). In other words,
the compression ratio $\alpha$ for all settings is 1/16. Formally, for a delta
matrix $\Delta\mathbf{W}\in\mathbb{R}^{h_{\mathrm{out}}\times
h_{\mathrm{in}}}$, $r_{\mathrm{begin}}$ and $r_{\mathrm{end}}$ are set to
satisfy the following equation:

$k\times(r_{\mathrm{end}}-r_{\mathrm{begin}})(h_{\mathrm{out}}+h_{\mathrm{in}})=16\times\alpha
h_{\mathrm{out}}h_{\mathrm{in}},$ (6)

where $\alpha$ is set to 1/16 in our experiments, which is the same as
BitDelta. In double-precision settings, $r_{\mathrm{begin}}$ and
$r_{\mathrm{end}}$ are set to 0 and 2, respectively, for the first precision.
For the second precision, $r_{\mathrm{begin}}$ is set to 2, and
$r_{\mathrm{end}}$ is adjusted so that the total delta size is 1/16 of the
uncompressed delta. In triple-precision settings, $r_{\mathrm{begin}}$ and
$r_{\mathrm{end}}$ are set to 0 and 2, respectively, for the first precision.
$r_{\mathrm{begin}}$ and $r_{\mathrm{end}}$ are set to 2 and 34, respectively,
for the second precision. For the third precision, $r_{\mathrm{begin}}$ is set
to 34, and $r_{\mathrm{end}}$ is adjusted so that the total delta size is 1/16
of the uncompressed delta. Since the diagonal matrix $\bm{\Sigma}$ occupies
little storage, the averaged bit-width for triple-precision compression is
approximately

$\frac{h_{\mathrm{out}}+h_{\mathrm{in}}}{h_{\mathrm{out}}h_{\mathrm{in}}}\sum_{i=1}^{3}k^{(i)}(r_{\mathrm{end}}^{(i)}-r_{\mathrm{begin}}^{(i)}).$
(7)

We conduct experiments on the math task, and the results are shown in Table 2.
We find that the 3-bit setting performs best among the single-precision
settings. Therefore, we keep the 3-bit setting and add other bit-widths to
form double-precision settings. Among the double-precision settings, "8+3"
achieves the highest score, which is then combined with an additional bit-
width to form triple-precision settings. We find that the best double-
precision setting can outperform the best single-precision setting, and the
best triple-precision setting achieves the highest score across all the
examined settings. We use “8+3+2” as the default setting in the following
experiments.

###  5.2 Main Results

Tables 3 and 4 show the performance of different delta-compression methods on
7B and 13B models, respectively. Across all tasks, Delta-CoMe outperforms both
baselines. While BitDelta (Liu et al., 2024b) can achieve near lossless
performance on chat models, it significantly degrades the performance of math
and code LLMs, a phenomenon not investigated by Liu et al. (2024b).
Surprisingly, our method achieves good performance in the delta-compression of
VLMs. To our knowledge, we are the first to investigate delta-compression for
VLMs.

Table 3: The performance of different delta-compression methods on 7B aligned models.  Method | $\alpha$ | WizardMath | MagicoderS-CL | Llama-2-chat | Llava-v1.5 | Ave.
---|---|---|---|---|---|---
GSM8K | MATH | HumanEval | MBPP | TruthfulQA | SafetyBench | GQA | TextVQA
Backbone | 1 | 11.0 | 2.9 | 38.4 | 47.6 | 41.7 | 38.9 | n/a | n/a | n/a
Aligned | 1 | 55.2 | 10.9 | 70.7 | 69.2 | 59.5 | 44.6 | 62.0 | 58.2 | 53.5
Low-Rank | 1/16 | 43.2 | 8.0 | 56.7 | 65.7 | 55.4 | 42.5 | 57.7 | 53.3 | 47.8
BitDelta | 1/16 | 45.6 | 8.6 | 57.3 | 65.9 | 59.3 | 41.1 | 59.7 | 56.9 | 49.3
Delta-CoMe | 1/16 | 53.6 | 10.3 | 67.1 | 67.9 | 59.8 | 47.0 | 61.7 | 58.5 | 53.2
Table 4: The performance of different delta-compression methods on 13B aligned models.  Method | $\alpha$ | WizardMath | MagicoderS-CL | Llama-2-chat | Llava-v1.5 | Ave.
---|---|---|---|---|---|---
GSM8K | MATH | HumanEval | MBPP | TruthfulQA | SafetyBench | GQA | TextVQA
Backbone | 1 | 17.8 | 3.9 | 43.3 | 49.0 | 55.0 | 37.3 | n/a | n/a | n/a
Aligned | 1 | 63.9 | 14.0 | 60.4 | 66.9 | 62.7 | 43.9 | 63.2 | 61.3 | 54.5
Low-Rank | 1/16 | 54.2 | 9.4 | 53.0 | 66.9 | 62.3 | 43.7 | 60.2 | 58.3 | 51.0
BitDelta | 1/16 | 54.8 | 10.6 | 51.8 | 64.2 | 62.6 | 41.6 | 60.9 | 60.3 | 50.9
Delta-CoMe | 1/16 | 58.9 | 12.8 | 57.9 | 67.2 | 62.9 | 44.1 | 63.1 | 61.2 | 53.5

###  5.3 Results on More Backbone Models

Table 5: Results on other representative backbones. The backbone of Openchat-3.5-0106 (Wang et al., 2023) is Mistral-7B-v0.1 (Jiang et al., 2023). Both Mistral-7B-v0.1 and Llama-3-8B are widely-used open-source LLMs.  Method | $\alpha$ | Openchat-3.5-0106 | Llama-3-8B-instruct | Ave.
---|---|---|---|---
GSM8K | HumanEval | TruthfulQA | SafetyBench | GSM8K | HumanEval | TruthfulQA | SafetyBench
Backbone | 1 | 52.2 | 28.7 | 61.0 | 42.1 | 44.8 | 33.5 | 43.6 | 43.9 | 43.7
Aligned | 1 | 77.1 | 73.2 | 78.4 | 61.0 | 78.5 | 61.6 | 68.2 | 51.6 | 68.7
Low-Rank | 1/16 | 50.5 | 52.4 | 76.9 | 49.0 | 68.3 | 46.3 | 67.5 | 51.3 | 57.8
BitDelta | 1/16 | 70.3 | 54.9 | 78.4 | 50.0 | 67.6 | 56.1 | 68.6 | 50.2 | 62.0
Delta-CoMe | 1/16 | 74.8 | 59.8 | 78.9 | 62.6 | 77.1 | 60.4 | 69.1 | 51.8 | 66.8

To investigate the generalization abilities of the delta-compression methods,
we conduct experiments on aligned models based on other representative
backbone LLMs. For additional backbones, we utilize Mistral-7B-v0.1 (Jiang et
al., 2023) and Llama-3-8B444https://huggingface.co/meta-llama/Meta-
Llama-3-8B.. The corresponding aligned models are Openchat-3.5-0106 (Wang et
al., 2023) and Llama-3-8B-instruct, respectively. As shown in Table 5, our
proposed Delta-CoMe method maintains superior performance over the two
baselines, demonstrating its generalization ability.

###  5.4 Delta-Compression vs. Delta-Tuning

A closely related area to delta-compression is delta-tuning. While delta-
tuning primarily aims to reduce the training cost of LLMs, delta-compression
focuses on reducing the storage and inference cost for multi-model serving. It
remains unclear whether delta-compression outperforms delta-tuning when using
the same delta size. To investigate this, we trained LoRA (Hu et al., 2022)
modules for all model parameters to compare delta-compression with delta-
tuning. We set the LoRA rank to 128 and the scale factor to 16, using a cosine
warmup schedule with a warmup ratio of 0.04 and a peak learning rate of 1e-4.
For each task, we trained the LoRA for 3 epochs. For mathematical LoRA, the
training dataset is from Yu et al. (2023), which consists of 395K training
examples. For code LoRA, the training set is from Wei et al. (2023), which
contains 186K training examples. For a fair comparison, we fine-tune all model
parameters using the same dataset as used for LoRA training. We then apply
different delta-compression methods to both the fine-tuned mathematical and
code LLMs.

Table 6: Comparison between LoRA and delta-compression methods.  Method | Math | Code | Ave.
---|---|---|---
GSM8K | MATH | HumanEval | HuamnEval
Backbone | 11.0 | 2.9 | 10.5 | 17.7 | 10.5
Aligned | 65.4 | 18.6 | 43.2 | 44.9 | 43.0
LoRA | 58.3 | 11.4 | 17.6 | 31.8 | 29.8
Low-Rank | 54.8 | 5.5 | 26.2 | 42.6 | 32.3
BitDelta | 47.8 | 10.7 | 26.2 | 41.9 | 31.7
Delta-CoMe | 65.1 | 18.0 | 39.6 | 44.9 | 41.9

Table 6 shows the results of both delta-tuning and delta-compression methods.
The results reveal that LoRA achieves superior performance compared to the
low-rank compression approach and BitDelta in the mathematical task. However,
when it comes to the coding task, LoRA exhibits lower performance than both
low-rank compression and BitDelta. By contrast, our proposed delta-compression
method (i.e., Delta-CoMe) consistently outperforms LoRA across all four
benchmarks. Specifically, the performance of our method is close to that of
the uncompressed aligned models (41.9 vs. 43.0), while the average score of
LoRA is only 29.8. These results imply that learning an aligned model and then
compressing it can achieve better results than delta-tuning.

##  6 Analysis

###  6.1 Analysis of Quantization Error

To better understand the performance of various delta-compression methods, we
estimate the quantization error as defined in Eq. (4). It is important to note
that the error we calculate differs from that of GPTQ. Specifically, we use
the mean square error between the activations of the uncompressed aligned
model and those of the combination of the backbone model and the compressed
delta model. The error is estimated on the GSM8K test set using
WizardMath-7B-v1.0 as the aligned model and Llama-2-7B as the backbone model.
Since different layers have varying impacts on the final output (Wu et al.,
2023), we distinguish low-, medium-, and high-layers when estimating the
average quantization error. Specifically, the first 11 layers are designated
as low-layers, the 12th to 22nd layers as medium-layers, and the last 10
layers as high-layers. Moreover, as outliers play a critical role in model
compression (Dettmers et al., 2023; Lin et al., 2023), we also calculate the
average error on outlier parameters. For each delta matrix $\Delta\mathbf{W}$,
we select the top 1% of columns with the largest absolute values as outliers.
Table 7 presents the results. We find that the average error of our methods
(i.e., "Single" and "Triple") is substantially lower than both the low-rank
baseline and BitDelta. Furthermore, the error of "Triple" is consistently less
than that of "Single," reaffirming the necessity of mixed-precision
compression for delta weights.

Table 7: Approximation errors ($\times 10^{-2}$) at the activation level for different model parameters. “Low”, “Medium”, “High” represent low-, medium-, and high-layers, respectively. “All” means the error averaged across all the parameters, while “Out.” denotes the average error estimated only on outliers.  Param | Attn.Q_Proj | Attn.K_Proj
---|---|---
Layer | Low | Medium | High | Low | Medium | High
Type | All | Out. | All | Out. | All | Out. | All | Out. | All | Out. | All | Out.
Low-Rank | 0.75 | 2.24 | 4.24 | 14.31 | 4.47 | 10.28 | 0.87 | 9.90 | 4.79 | 34.04 | 4.82 | 31.41
BitDelta | 0.97 | 2.48 | 4.66 | 14.48 | 4.84 | 10.01 | 1.09 | 10.34 | 5.16 | 33.03 | 5.14 | 28.06
Single | 0.20 | 0.74 | 1.37 | 5.11 | 1.24 | 3.36 | 0.23 | 3.19 | 1.52 | 11.30 | 1.36 | 8.48
Triple | 0.13 | 0.28 | 0.54 | 1.07 | 0.71 | 0.88 | 0.15 | 0.56 | 0.58 | 1.99 | 0.73 | 2.10
Param | Attn.V_Proj | Attn.O_Proj
Layer | Low | Medium | High | Low | Medium | High
Type | All | Out. | All | Out. | All | Out. | All | Out. | All | Out. | All | Out.
Low-Rank | 0.41 | 3.61 | 1.84 | 8.27 | 2.93 | 4.64 | 0.01 | 0.13 | 0.10 | 0.39 | 0.38 | 5.94
BitDelta | 0.45 | 3.60 | 1.95 | 8.02 | 3.18 | 4.85 | 0.01 | 0.13 | 0.11 | 0.44 | 0.37 | 5.45
Single | 0.14 | 1.42 | 0.65 | 3.58 | 0.79 | 1.45 | 0.00 | 0.04 | 0.03 | 0.10 | 0.10 | 1.60
Triple | 0.04 | 0.12 | 0.21 | 0.35 | 0.52 | 0.61 | 0.00 | 0.01 | 0.02 | 0.05 | 0.06 | 0.92
Param | FFN.Up_Proj | FFN.Gate_Proj
Layer | Low | Medium | High | Low | Medium | High
Type | All | Out. | All | Out. | All | Out. | All | Out. | All | Out. | All | Out.
Low-Rank | 0.13 | 0.86 | 0.93 | 3.43 | 2.20 | 11.45 | 0.10 | 0.26 | 0.79 | 1.12 | 1.87 | 9.74
BitDelta | 0.18 | 0.97 | 1.06 | 3.84 | 2.38 | 12.22 | 0.13 | 0.31 | 0.90 | 1.26 | 2.02 | 11.74
Single | 0.03 | 0.17 | 0.27 | 1.08 | 0.56 | 3.10 | 0.02 | 0.06 | 0.23 | 0.35 | 0.47 | 2.14
Triple | 0.03 | 0.11 | 0.15 | 0.49 | 0.39 | 2.01 | 0.02 | 0.03 | 0.14 | 0.15 | 0.35 | 1.64

###  6.2 Case Study

We also present a detailed case study in Figure 3. Three delta-compression
methods are examined: BitDelta, single-precision compression, and triple-
precision compression. The reference answer is "104 hours". We observe that
BitDelta makes mistakes initially, while single-precision compression
generates an incorrect intermediate result at the second reasoning step. In
contrast, our mixed-precision delta-compression method calculates the correct
final answer.

Figure 3: Case study for different delta-compression methods, where only the
triple-precision compression method proposed in this work can give the correct
answer.

##  7 Conclusion

In this paper, we propose Delta-CoMe, a delta-compression method with mixed-
precision inspired by the long-tail distribution of singular values in the
delta weights. Delta-CoMe achieves near-lossless performance compared to
uncompressed aligned models across various typical tasks, including math,
code, chat, and multi-modal tasks. We validate the effectiveness of Delta-CoMe
on several widely-used aligned LLMs, whose backbone pre-trained models include
Llama-2, Llama-3, and Mistral. Experimental results demonstrate that Delta-
CoMe outperforms several representative baselines by a considerable margin. We
believe the newly introduced Delta-CoMe method has significant value for many
practical applications, such as multi-tenant serving.

## References

  * Austin et al. (2021) Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ , 2021.
  * Bablani et al. (2023) Bablani, D., Mckinstry, J. L., Esser, S. K., Appuswamy, R., and Modha, D. S.  Efficient and effective methods for mixed precision neural network quantization for faster, energy-efficient inference.  _arXiv preprint arXiv:2301.13330_ , 2023.
  * Chen et al. (2021) Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W.  Evaluating large language models trained on code, 2021.
  * Cobbe et al. (2021) Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Dettmers et al. (2023) Dettmers, T., Svirschevski, R. A., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D.  Spqr: A sparse-quantized representation for near-lossless llm weight compression.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Frantar et al. (2023) Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D.  OPTQ: Accurate quantization for generative pre-trained transformers.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=tcbBPnfwxS.
  * Hendrycks et al. (2021) Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J.  Measuring mathematical problem solving with the math dataset.  _NeurIPS_ , 2021.
  * Hu et al. (2022) Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Hudson & Manning (2019) Hudson, D. A. and Manning, C. D.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 6700–6709, 2019.
  * Jiang et al. (2023) Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Lin et al. (2023) Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_ , 2023.
  * Lin et al. (2022) Lin, S., Hilton, J., and Evans, O.  Truthfulqa: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 3214–3252, 2022.
  * Liu et al. (2024a) Liu, H., Li, C., Wu, Q., and Lee, Y. J.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024a.
  * Liu et al. (2024b) Liu, J., Xiao, G., Li, K., Lee, J. D., Han, S., Dao, T., and Cai, T.  Bitdelta: Your fine-tune may only be worth one bit.  _arXiv preprint arXiv:2402.10193_ , 2024b.
  * Luo et al. (2023a) Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _arXiv preprint arXiv:2308.09583_ , 2023a.
  * Luo et al. (2023b) Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D.  Wizardcoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_ , 2023b.
  * Ryu et al. (2023a) Ryu, S., Seo, S., and Yoo, J.  Efficient storage of fine-tuned models via low-rank approximation of weight residuals.  _arXiv preprint arXiv:2305.18425_ , 2023a.
  * Ryu et al. (2023b) Ryu, S., Seo, S., and Yoo, J.  Efficient storage of fine-tuned models via low-rank approximation of weight residuals.  _arXiv preprint arXiv:2305.18425_ , 2023b.
  * Shen et al. (2024) Shen, X., Dong, P., Lu, L., Kong, Z., Li, Z., Lin, M., Wu, C., and Wang, Y.  Agile-quant: Activation-guided quantization for faster inference of llms on the edge.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pp. 18944–18951, 2024.
  * Singh et al. (2019) Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M.  Towards vqa models that can read.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8317–8326, 2019.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Wang et al. (2023) Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., and Liu, Y.  Openchat: Advancing open-source language models with mixed-quality data.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Wei et al. (2023) Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L.  Magicoder: Source code is all you need.  _arXiv preprint arXiv:2312.02120_ , 2023.
  * Wu et al. (2023) Wu, X., Huang, S., and Wei, F.  Mole: Mixture of lora experts.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Xu et al. (2023) Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D.  Wizardlm: Empowering large language models to follow complex instructions, 2023.
  * Yao & Klimovic (2023) Yao, X. and Klimovic, A.  Deltazip: Multi-tenant language model serving via delta compression.  _arXiv preprint arXiv:2312.05215_ , 2023.
  * Yao et al. (2021) Yao, Z., Dong, Z., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang, Q., Wang, Y., Mahoney, M., et al.  Hawq-v3: Dyadic neural network quantization.  In _International Conference on Machine Learning_ , pp. 11875–11886. PMLR, 2021.
  * Yu et al. (2023) Yu, L., Jiang, W., Shi, H., Jincheng, Y., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W.  Metamath: Bootstrap your own mathematical questions for large language models.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Yu et al. (2024) Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y.  Language models are super mario: Absorbing abilities from homologous models as a free lunch.  In _International Conference on Machine Learning_. PMLR, 2024.
  * Zhang et al. (2023) Zhang, Z., Lei, L., Wu, L., Sun, R., Huang, Y., Long, C., Liu, X., Lei, X., Tang, J., and Huang, M.  Safetybench: Evaluating the safety of large language models with multiple choice questions.  _arXiv preprint arXiv:2309.07045_ , 2023.


# On Locally Conformally Kähler metrics on Oeljeklaus-Toma Manifolds

Ştefan Deaconu  Faculty of Mathematics and Informatics
University of Bucharest, Academiei st. 14, Bucharest, Romania
<EMAIL_ADDRESS>and  Victor Vuletescu  Faculty of Mathematics and
Informatics
University of Bucharest, Academiei st. 14, Bucharest, Romania
<EMAIL_ADDRESS>
###### Abstract.

We show that Oeljeklaus-Toma manifolds $X(K,U)$ where $K$ is a number field of
signature $(s,t)$ such that $s\geq 1,$ $t\geq 2$ and $s\geq 2t$ admit no lck
metric. Combined with the earlier results by [OeTo05] and [Dub14] this
completely solves the problem of existence of LCK metrics on Oeljeklaus-Toma
manifolds.

###### Key words and phrases:

Locally conformally Kähler metric, Oeljeklaus-Toma manifold.

###### 2010 Mathematics Subject Classification:

53C55, 57T15

Both authors are partially supported by Romanian Ministry of Education and
Research, Program PN-III, Project number PN-III-P4-ID-PCE-2020-0025, Contract
30/04.02.2021

##  1\. Introduction

Oeljeklaus-Toma manifolds were introduced by K.Oeljeklaus and M. Toma in
[OeTo05], as a generalization to higher dimensions of the Inoue surfaces
$S_{M}$ ([Ino74]). Very briefly, their construction goes as follows. Fix a
number field $K$ having $s\geq 1$ real embeddings and $2t\geq 2$ complex ones,
and label its embeddings such that $\sigma_{1},\dots,\sigma_{s}$ are the real
ones, while $\sigma_{s+t+i}=\overline{\sigma}_{s+i}$ for any $i=1,\dots,t.$
Let $\mathcal{O}_{K}$ be the ring of integers of $K,$ $\mathcal{O}_{K}^{*}$
the group of units of $\mathcal{O}_{K}$ and $\mathcal{O}_{K}^{*,+}$ the
subgroup of $\mathcal{O}_{K}^{*}$ of totally positive units, that is elements
$u\in\mathcal{O}_{K}^{*}$ such that $\sigma_{i}(u)>0$ for all $i=1,\dots,s.$
Letting ${\mathbb{H}}:=\\{z\in\mathbb{C}|Im(z)>0\\}$, we see there are natural
actions of $\mathcal{O}_{K}$ and respectivley of $\mathcal{O}_{K}^{*,+}$ on
${\mathbb{H}}^{s}\times\mathbb{C}^{t}\subset\mathbb{C}^{s+t}$ by

$a\cdot(x_{i})_{i=1,\dots s+t}:=\left(x_{i}+\sigma_{i}(a)\right)_{i=1,\dots
s+t},\;\forall a\in\mathcal{O}_{K}$

and respectively

$u\cdot(x_{i})_{i=1,\dots s+t}:=\left(\sigma_{i}(u)x_{i}\right)_{i=1,\dots
s+t},\;\forall u\in\mathcal{O}_{K}^{*,+}.$

The combined resulting action of $\mathcal{O}_{K}^{*,+}\ltimes\mathcal{O}_{K}$
is however not discrete in general. Still, in [OeTo05] it is shown that one
can always find subgroups $U\subset\mathcal{O}_{K}^{*,+}$ such that the action
of $U\ltimes\mathcal{O}_{K}$ is discrete and cocompact: the resulting compact
complex manifold is usually denoted $X(K,U)$ and is called an Olejeklaus-Toma
manifold (OT, for short). The Inoue surfaces $S_{M}$ are corresponding to the
particular case when $K$ has degree $3$ and $U\subset\mathcal{O}_{K}^{*,+}$ is
any subgroup of finite index.

The OT manifolds have a number of very interesting properties. For
instance,they are non-Kälerian, do not respect Hodge symmetry (nor satisfy the
$\partial\overline{\partial}$-lemma), and for appropriate (but rather generic)
choices of the group of units $U$, they have no closed proper complex analytic
subspaces.

As these manifolds do not admit Kähler metrics, it is natural to ask whether
other natural metrics to exist on them: for a detailed account on this
problem, see e.g. [AnDuOtSt22]. One of the most interesting ones are the
locally conformally Kähler metrics (LCK, for short): these are those whose
associated $(1,1)-$ forms $\omega$ has the property

(1) $\displaystyle d\omega=\theta\wedge\omega$

for some closed $1-$form $\theta$ (for more details see [DrOr98]). The
existence of such metrics on OT manifolds $X(K,U)$ can be read on the Galois
properties of the group of units $U$. More precisely, it was shown (see
[Dub14], appendix by L. Battisti) that:

######  Proposition 1.

An Oeljeklaus-Toma manifold $X(K,U)$ admits an LCK metric if if and only if
for any unit $u\in U$ one has

(2) $\displaystyle|\sigma_{s+1}(u)|=\dots=|\sigma_{s+t}(u)|$

Already since these manifolds were introduced in [OeTo05], it was shown that
such metrics exists on $X(K,U)$ when $t=1$ and do not exist when $s=1$ and
$t\geq 2.$ For the remaining possibilities for $(s,t)$, the second-named
author showed in ([Vu14]) that for a certain number of cases, LCK metrics do
not exist. The result was widely extended by A. Dubickas in [Dub14]; still,
some cases remained unclear. The goal of this note is to show the non-
existence of LCK metrics in the remaining cases.

Acknowledgments. The authors thank Liviu Ornea for useful discussions and
valuable suggestions and Alexandra Otiman for a careful reading of a previous
version of the present note.

##  2\. The results

######  Lemma 1.

Let $\Lambda\subset\mathbb{R}^{n}$ be a discrete lattice. Then $\Lambda$ can
not be written as a union

(3) $\displaystyle\Lambda=\bigcup_{i=1}^{m}\Lambda_{i}$

of sublattices of proper rank,
$rank_{\mathbb{Z}}(\Lambda_{i})<rank_{\mathbb{Z}}(\Lambda),\forall
i=1,\dots,m.$

Proof. For a lattice $\Lambda\subset\mathbb{R}^{n}$ freely generated by some
vectors $e_{1},\dots,e_{N}$ we let

$\Lambda\mathbb{Q}:=\\{\sum_{j=1}^{N}q_{i}e_{i},q_{i}\in\mathbb{Q},\forall
i\\};$

then $\Lambda\mathbb{Q}$ is a $\mathbb{Q}-$vector space and if $\Lambda$ is
discrete then
$dim_{\mathbb{Q}}(\Lambda\mathbb{Q})=rank_{\mathbb{Z}}(\Lambda).$

Next, we infer that (3) implies

(4) $\displaystyle\Lambda\mathbb{Q}=\bigcup_{i=1}^{m}\Lambda_{i}\mathbb{Q}$

The only inclusion to see here is ${}^{\prime\prime}\subset^{\prime\prime}.$
Take a vector $v\in\Lambda\mathbb{Q};$ then
$v=\displaystyle\sum_{j=1}^{N}q_{i}e_{i}$ hence
$v=\frac{1}{M}\displaystyle\sum_{j=1}^{N}a_{i}e_{i}$ for some
$M\in\mathbb{N}^{*}$ and $a_{i}\in\mathbb{Z}.$ But then the vector
$w:=\displaystyle\sum_{j=1}^{N}a_{i}e_{i}$ is in $\Lambda$, hence by our
assumption $w$ belongs to some $\Lambda_{i}$; it follows that
$v\in\Lambda_{i}\mathbb{Q}$.

But decomposition (4) leads to a contradiction, since a vector space over an
infinite field cannot be written as a finite union of subspaces of smaller
dimension. Q.E.D.

Notations. Let $K$ be a number field with $s$ real embeddings and $2t$ complex
ones; we suppose we labeled the embeddings $\sigma_{i}(i=1\dots s+2t)$ of $K$
such that $\sigma_{1},\dots,\sigma_{s}$ are the real embeddings and such that
$\overline{\sigma}_{s+k}=\sigma_{s+k+t}$ for all $k=1,\dots,t.$ We denote by
$\Lambda_{K}$ the image of the units of $K$ under the logarithmic embedding

$l(u):=\left(\log|\sigma_{1}(u)|,\dots,\log|\sigma_{s+t}(u)|\right)\subset\mathbb{R}^{s+t}.$

Dirichlet units theorem tells us that $\Lambda_{K}$ is a discrete (and
complete) lattice in the hyperplane ${\mathcal{H}}_{Dir}$ given by

$({\mathcal{H}}_{Dir}):\;x_{1}+\dots+x_{s}+2x_{s+1}+\dots+2x_{s+t}=0$

If $L\subset K$ is a number subfield, we will similarly denote by
$\Lambda_{L}$ the image of the units of $L$ under the previous embedding.

######  Proposition 2.

Let $K$ be a number field of signature $(s,t)$. If $s\geq 1,$ $t\geq 2$ and
$s\geq 2t$ then $\mathcal{O}_{K}^{*}$ has no subgroup $U$ of rank $s$ such
that

$\sigma_{s+1}(u)\sigma_{s+t+1}(u)=\dots=\sigma_{s+t}(u)\sigma_{s+2t}(u)$

holds good for any $u\in U.$

Proof. Assume such an $U$ would exist. First notice that the logarithmic image
$l(U)$ of $U$ lives on the intersection of the hyperplane
${\mathcal{H}}_{Dir}$ above with the $t-1$ hyperplanes
${\mathcal{H}}_{i},i=1,\dots,t-1$ given by

$x_{s+i}=x_{s+i+1}.$

Notice that the linear variety
${\mathcal{H}}_{Dir}\cap\left(\displaystyle\bigcap_{i=1}^{t-1}{\mathcal{H}}_{i}\right)$
is of dimension $s.$

We will prove that there are finitely many sublattices
$\Lambda^{\prime}\subset\Lambda_{K}$ with $rank(\Lambda^{\prime})<s$ such that
any element $l(u)\in l(U)$ lives in (at least) one such $\Lambda^{\prime}$,
getting henceforth a contradiction with the Lemma 1. So take an arbitrary
element $u\in U.$

If $deg(u)<[K:\mathbb{Q}]$ then there exists some proper subfield $L\subsetneq
K$ such that $u\in L;$ in particular $l(u)\in\Lambda_{L}.$ Call
$(s^{\prime},t^{\prime})$ the signature of $L$; then
$rank(\Lambda_{L})=s^{\prime}+t^{\prime}-1.$ Letting $d:=[K:L]$ we have
$s+2t=d(s^{\prime}+2t^{\prime}).$ Hence
$s^{\prime}+2t^{\prime}=\frac{s+2t}{d}$ so

$s^{\prime}+t^{\prime}-1=\frac{s+2t}{d}-t^{\prime}-1.$

Now

$\frac{s+2t}{d}-t^{\prime}-1<s\Leftrightarrow
s+2t<ds+d(t^{\prime}+1)\Leftrightarrow 2t-d(t^{\prime}+1)<(d-1)s.$

But $2t-d(t^{\prime}+1)<2t$ and $(d-1)s\geq s$ as $d\geq 2$ (since
$L\subsetneq K$). Hence $rank(\Lambda_{L})<s.$

We are left with the case when $u$ has maximal degree. As $t\geq 2$ the
relation

(5)
$\displaystyle\sigma_{s+1}(u)\sigma_{s+t+1}(u)=\sigma_{s+2}(u)\sigma_{s+t+2}(u)$

holds good. As the absolute Galois group
$G_{\overline{\mathbb{Q}}/\mathbb{Q}}$ acts transitively on the Galois
conjugates of $\sigma_{s+1}(u)$, we see there exists some $\varphi\in G_{K}$
such that $\varphi(\sigma_{s+1}(u))=\sigma_{1}(u).$ Applying $\varphi$ to
relation (5) we get

(6) $\displaystyle\sigma_{1}(u)\sigma_{j}(u)=\sigma_{k}(u)\sigma_{l}(u)$

for some $j,k,l\in\\{2,\dots,s+2t\\}.$ Taking absolute values we see that the
logarithmic image of $u$ lives in the hyperplane
${\mathcal{H}}_{j^{\prime}k^{\prime}l^{\prime}}$ of $\mathbb{R}^{n}$ given by

$({\mathcal{H}}_{j^{\prime}k^{\prime}l^{\prime}}):\;x_{1}+x_{j^{\prime}}=x_{k^{\prime}}+x_{l^{\prime}}$

where $j^{\prime},k^{\prime},l^{\prime}$ equals respectively $j,k,l$ if they
are $\leq s+t$ or $j-t,k-t,l-t$ otherwise. Since $s\geq 1$ we see that

$dim({\mathcal{H}}_{j^{\prime}k^{\prime}l^{\prime}}\cap{\mathcal{H}}_{Dir}\cap\left(\bigcap_{i=1}^{t-1}{\mathcal{H}}_{i}\right))=s-1<s$

so

$\Lambda_{j^{\prime}k^{\prime}l^{\prime}}:=\Lambda_{K}\cap\left({\mathcal{H}}_{j^{\prime}k^{\prime}l^{\prime}}\cap{\mathcal{H}}_{Dir}\cap\left(\bigcap_{i=1}^{t-1}{\mathcal{H}}_{i}\right)\right)$

is a lattice of rank $<s$ since it is discrete.

We conclude that any $l(u),u\in U$ lives either in a lattice of the form
$\Lambda_{L}$ with $L\subsetneq K$ a proper subfield or in a lattice of the
form $\Lambda_{i^{\prime}j^{\prime}k^{\prime}}$ as above; as all these
lattices are of rank $<s$ and they are finitely many, we got our
contradiction.

######  Corollary 1.

If an Oeljeklaus-Toma manifold $X(K,U)$ admits an LCK metric, then $K$ has
exactly $2t=2$ complex embeddings.

The proof follows at once from the above Propositions 1 and 2.

## References

  * [AnDuOtSt22] Angella, D., Dubickas, A., Otiman, A., Stelzig, J., On metric an cohomological properties of Oeljeklaus-Toma manifolds, ArXiv: 2201.06377
  * [DrOr98] Dragomir,S., Ornea, L., Locally conformally Kähler manifolds, Progress in Math. 55, Birkhäuser, 1998.
  * [Dub14] Dubickas, A., Nonreciprocal units in a number field with an application to Oeljeklaus–Toma manifolds, New York J. Math, 20(2014), p. 257–274.
  * [Ino74] Inoue, M., On surfaces of class V II0, Invent. Math., 24(1974), 269–310
  * [MilneANT] James S. Milne. Algebraic Number Theory. Available at http://www.jmilne.org/math/CourseNotes/ant.html.
  * [OeTo05] Oeljeklaus, K., Toma, M., Non-Kähler compact complex manifolds associated to number fields, Ann. Inst. Fourier, Grenoble, 55(2005), no. 1, 161–171.
  * [Vu14] Vuletescu V., LCK metrics on Oeljeklaus-Toma manifolds versus Kronecker’s theorem, Bull. Math. Soc. Sci. Math. Roumanie, 57 (2014), no. 2, p. 225-231