fix: pre-release audit round 12 — 33 bugs fixed across all tiers

Tier 1 (ship-blockers):
- README/verifier layer count corrected to 7-layer/10-point
- SQL keyword regex now strips quoted strings before checking
- validate_citations counts unique sources, extracts from body only
- Provider-specific error messages for all 3 LLM backends
- Missing config error handling in ingest.py entry point

Tier 2 (quality):
- SQL injection: alphanumeric validation on interpolated values
- LIKE wildcard escaping with ESCAPE clause
- Fallback builds fresh keyword query instead of re-running failed SQL
- NaN/NaT/inf filtering across all tabular readers
- Merged docx table cells deduplicated via element identity
- .env double-escaping fix in setup wizard
- Verification LLM exceptions skip correction instead of triggering it
- References check handles SQL and web sources
- Dead pre-loop validate_citations removed
- KB meta extension detection uses Path.suffix
- QU prompt omits SQL fields when no schema available
- Redundant collection.count() eliminated
- Row padding for zip truncation in Excel/CSV readers

Tier 3 (improvements):
- Gemini SDK migrated from google-generativeai to google-genai
- PyPDF2 migrated to pypdf
- ChromaDB collection caching (one client per session)
- CLI startup uses cached KB brief instead of LLM call
- Temperature/max_tokens validation and clamping
- Docker: removed deprecated version field, fixed bind mounts
- SQL INSERT batched with executemany (10-100x faster)
- Various small fixes (rdata None display, SELECT validation,
max_rows default, search error handling, log message guard,
fuzzy query caching)

Added .dockerignore for clean Docker builds.
All 174 tests passing. Design score 9/10.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.dockerignore

@@ -0,0 +1,18 @@
+.env
+.git
+.gitignore
+.DS_Store
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+chroma_db/
+sql_db/
+knowledge_base/*
+!knowledge_base/.gitkeep
+docs/
+tests/
+CLAUDE.md
+*.md

README.md

@@ -1,6 +1,6 @@
 # RAG Research Chatbot Template
 
-A **citation-verified research assistant chatbot** that answers questions using your own documents -- PDFs, Word files, spreadsheets, datasets, and more. Every answer cites its sources with numbered references, and a 6-layer verification system ensures the chatbot never makes things up.
+A **citation-verified research assistant chatbot** that answers questions using your own documents -- PDFs, Word files, spreadsheets, datasets, and more. Every answer cites its sources with numbered references, and a 7-layer verification system ensures the chatbot never makes things up.
 
 **Built for researchers.** No coding experience required. You provide your documents, choose an AI provider, and the chatbot does the rest.
 
@@ -20,7 +20,7 @@ A **citation-verified research assistant chatbot** that answers questions using
   - [Step 1: Query Understanding](#step-1-query-understanding)
   - [Step 2: Retrieval](#step-2-retrieval)
   - [Step 3: Response Generation](#step-3-response-generation)
-  - [Step 4: Verification](#step-4-verification-6-layer-anti-hallucination-stack)
+  - [Step 4: Verification](#step-4-verification-7-layer-anti-hallucination-stack)
   - [Step 5: Display](#step-5-display)
 - [Configuration Reference](#configuration-reference)
 - [Docker (Optional)](#docker-optional)
@@ -397,9 +397,9 @@ The system prompt tells the LLM to:
 
 A **soft token cap** limits response length proportionally to context size -- less context means shorter answers, reducing the surface area for hallucination.
 
-### Step 4: Verification (6-Layer Anti-Hallucination Stack)
+### Step 4: Verification (7-Layer Anti-Hallucination Stack)
 
-This is the core differentiator. Before showing you the answer, it passes through six verification layers:
+This is the core differentiator. Before showing you the answer, it passes through seven verification layers:
 
 ```
 Layer 0: No-source gate
@@ -414,7 +414,7 @@ Layer 2: Response length cap
    v
 Layer 3: LLM self-verification loop (up to 3 iterations)
    │  A second LLM call audits the response against the sources
-   │  using a 9-point checklist. If errors are found, the response
+   │  using a 10-point checklist. If errors are found, the response
    │  is corrected and re-verified. If it still fails after 3
    │  attempts, the chatbot refuses to answer. (1-3 extra LLM calls)
    v
@@ -423,6 +423,11 @@ Layer 4: Term-overlap check
    │  appear in the source text. Flags claims with < 40% overlap
    │  as potentially ungrounded. (Free, no LLM call)
    v
+Layer 4.5: Citation audit
+   │  Checks that citation numbers [N] don't exceed actual source
+   │  count and that the References section mentions filenames from
+   │  retrieved sources. (Free, no LLM call)
+   v
 Layer 5: Warning-phrase scanner
       Scans for phrases like "based on my knowledge" or "it is well
       known" that suggest the LLM is using training data instead of
@@ -573,7 +578,7 @@ The container mounts your `knowledge_base/`, `chroma_db/`, `sql_db/`, `config.ya
 
 ### Advantages
 
-- **No hallucination by design.** The 6-layer verification stack catches unsupported claims before they reach you. If the chatbot can't verify an answer, it refuses rather than guessing.
+- **No hallucination by design.** The 7-layer verification stack catches unsupported claims before they reach you. If the chatbot can't verify an answer, it refuses rather than guessing.
 - **Full citation trail.** Every factual claim is tied to a specific source with page numbers or URLs. You can trace any claim back to the original document.
 - **Works with your own documents.** Unlike general-purpose chatbots, this one answers from *your* knowledge base. Your PDFs, datasets, and codebooks are the primary authority.
 - **Knows what it knows.** The chatbot builds a meta-overview during ingestion. Ask "What datasets do you have?" and it can answer, instead of refusing because no chunk matches.
@@ -625,9 +630,9 @@ This means it cannot find relevant passages in your documents. Try:
 - Adding more relevant documents to `knowledge_base/` and re-running `python ingest.py`
 - Enabling web search (`/websearch on`) to supplement with academic papers
 
-### PyPDF2 deprecation warning
+### PDF reader
 
-You may see a warning about PyPDF2 being deprecated. This is cosmetic and does not affect functionality.
+This project uses `pypdf` (successor to PyPDF2) for PDF extraction.
 
 ### Ingestion is very slow
 

app_cli.py

@@ -9,7 +9,7 @@ from rich.text import Text
 
 from src.config_loader import load_config, get_api_key
 from src.ingest import ingest_documents
-from src.kb_meta import summarize_kb_for_welcome
+from src.kb_meta import load_kb_meta_brief
 from src.query_engine import understand_query
 from src.retriever import retrieve
 from src.verifier import verify_and_respond
@@ -217,7 +217,7 @@ def main() -> None:
         welcome.append(". Use /websearch on to enable.\n\n")
 
     # KB overview summary (LLM-generated welcome summary)
-    kb_summary = summarize_kb_for_welcome(cfg)
+    kb_summary = load_kb_meta_brief(cfg)
     if kb_summary:
         welcome.append("Knowledge Base:\n", style="bold")
         welcome.append(kb_summary + "\n\n", style="dim")

app_web.py

@@ -119,7 +119,8 @@ def render_sidebar():
                 try:
                     count = ingest_documents(cfg)
                     st.success(f"Ingested {count} chunks.")
-                    # Clear the cached chunk count so it refreshes
+                    # Clear cached data so it refreshes after re-ingest
+                    st.session_state.pop("kb_welcome_summary", None)
                     st.rerun()
                 except Exception as e:
                     st.error(f"Ingestion failed: {e}")
@@ -252,9 +253,10 @@ def render_chat():
                 if result.get("refused"):
                     status.update(label="No sufficient sources found.", state="error")
                 elif result.get("verification_passed") is True:
+                    sql_label = f" + {n_sql} SQL rows" if n_sql else ""
                     status.update(
                         label=f"Verified ({result.get('iterations', 0)} iteration(s)). "
-                              f"{n_local} local + {n_web} web sources.",
+                              f"{n_local} local{sql_label} + {n_web} web sources.",
                         state="complete",
                     )
                 elif result.get("verification_passed") is False:
@@ -263,8 +265,9 @@ def render_chat():
                         state="error",
                     )
                 else:
+                    sql_label2 = f" + {n_sql} SQL rows" if n_sql else ""
                     status.update(
-                        label=f"Done. {n_local} local + {n_web} web sources.",
+                        label=f"Done. {n_local} local{sql_label2} + {n_web} web sources.",
                         state="complete",
                     )
 

docker-compose.yml

@@ -1,4 +1,4 @@
-version: "3.8"
+# Prerequisites: config.yaml must exist. Run 'python setup.py' first.
 services:
   chatbot:
     build: .
@@ -8,8 +8,9 @@ services:
       - ./knowledge_base:/app/knowledge_base
       - ./chroma_db:/app/chroma_db
       - ./sql_db:/app/sql_db
-      - ./config.yaml:/app/config.yaml
-      - ./.env:/app/.env
+      - ./config.yaml:/app/config.yaml:ro  # Must exist before running
+      # API keys: set via environment variables below (preferred)
+      # or mount .env file: - ./.env:/app/.env:ro
     environment:
       - OPENAI_API_KEY=${OPENAI_API_KEY:-}
       - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}

ingest.py

@@ -1,11 +1,17 @@
 """Entry point for document ingestion."""
 
+import sys
+
 from src.ingest import ingest_documents
 from src.config_loader import load_config
 
 
 if __name__ == "__main__":
-    cfg = load_config()
+    try:
+        cfg = load_config()
+    except FileNotFoundError:
+        print("Config file not found. Run 'python setup.py' first.")
+        sys.exit(1)
     count = ingest_documents(cfg)
     if count == 0:
         print("\nNo documents were ingested. Add files to knowledge_base/ and try again.")

requirements.txt

@@ -1,6 +1,6 @@
 # Core
 chromadb>=0.4.0
-PyPDF2>=3.0.0
+pypdf>=4.0.0
 python-docx>=1.0.0
 openpyxl>=3.0.0
 xlrd>=2.0.0
@@ -19,7 +19,7 @@ tokenizers>=0.13.0
 # LLM providers
 openai>=1.0.0
 anthropic>=0.20.0
-google-generativeai>=0.5.0
+google-genai>=1.0.0
 
 # Web UI
 streamlit>=1.30.0

setup.py

@@ -91,7 +91,13 @@ def generate_env(provider: str, api_key: str, existing_env_path: str = None) ->
                 line = line.strip()
                 if line and not line.startswith("#") and "=" in line:
                     k, v = line.split("=", 1)
-                    v = v.strip().strip('"').strip("'")
+                    v = v.strip()
+                    # Unwrap one matched pair of quotes
+                    if (v.startswith('"') and v.endswith('"')) or \
+                       (v.startswith("'") and v.endswith("'")):
+                        v = v[1:-1]
+                    # Unescape previously escaped characters
+                    v = v.replace('\\"', '"').replace('\\\\', '\\')
                     existing[k.strip()] = v
 
     # Update with the new key
@@ -99,7 +105,8 @@ def generate_env(provider: str, api_key: str, existing_env_path: str = None) ->
 
     lines = ["# Auto-generated by setup wizard"]
     for k, v in sorted(existing.items()):
-        lines.append(f'{k}="{v}"')
+        v_escaped = v.replace('\\', '\\\\').replace('"', '\\"')
+        lines.append(f'{k}="{v_escaped}"')
     lines.append("")
     return "\n".join(lines)
 

src/config_loader.py

@@ -32,9 +32,14 @@ def load_config(config_path: str = None) -> dict:
 
     # Normalize None-valued sections to empty dicts so chained .get() never
     # fails with AttributeError (e.g. `llm:` with no sub-keys → None).
-    for key in list(cfg.keys()):
-        if cfg[key] is None:
-            cfg[key] = {}
+    # Recurse into nested dicts so `paths:\n  vector_db:` also gets normalized.
+    def _normalize_nulls(d):
+        for key in list(d.keys()):
+            if d[key] is None:
+                d[key] = {}
+            elif isinstance(d[key], dict):
+                _normalize_nulls(d[key])
+    _normalize_nulls(cfg)
 
     for provider, env_var in _ENV_KEY_MAP.items():
         env_val = os.environ.get(env_var)

src/ingest.py

@@ -196,11 +196,14 @@ def ingest_documents(cfg: dict = None, documents_dir: str = None) -> int:
             if not os.path.isabs(db_path):
                 project_root = Path(__file__).resolve().parent.parent
                 db_path = os.path.join(str(project_root), db_path)
+            cleared_any = False
             for meta_name in ("kb_meta.txt", "kb_meta_brief.txt"):
                 meta_file = os.path.join(db_path, meta_name)
                 if os.path.exists(meta_file):
                     os.remove(meta_file)
-            print("Cleared stale KB overview.")
+                    cleared_any = True
+            if cleared_any:
+                print("Cleared stale KB overview.")
         except Exception as e:
             print(f"Warning: Could not clear KB meta: {e}")
         return 0
@@ -288,18 +291,15 @@ def ingest_documents(cfg: dict = None, documents_dir: str = None) -> int:
                 print(f"SQL ingestion error (non-fatal): {e}")
 
     # ── KB meta overview (LLM-generated) ────────────────────────────
-    # Clear stale meta files before regenerating
     _db_path = cfg.get("paths", {}).get("vector_db", "chroma_db")
     if not os.path.isabs(_db_path):
         _db_path = os.path.join(str(Path(__file__).resolve().parent.parent), _db_path)
-    for _meta_name in ("kb_meta.txt", "kb_meta_brief.txt"):
-        _meta_file = os.path.join(_db_path, _meta_name)
-        if os.path.exists(_meta_file):
-            os.remove(_meta_file)
 
     print("\nGenerating knowledge base overview...")
     try:
         from src.kb_meta import build_and_store_overview
+        # build_and_store_overview overwrites meta files via write_text();
+        # no pre-deletion needed — old files stay intact if generation fails.
         overview = build_and_store_overview(collection, cfg)
         if overview:
             print("KB overview generated and indexed.")

src/kb_meta.py

@@ -281,7 +281,7 @@ def collect_file_records(collection, *, _cache: tuple = None) -> list[dict]:
         if source == META_SOURCE:
             continue
         if source not in source_info:
-            ext = Path(source).suffix if "." in source else ""
+            ext = Path(source).suffix
             source_info[source] = {
                 "source": source,
                 "dataset": meta.get("dataset", "general"),

src/llm/__init__.py

@@ -23,6 +23,9 @@ def generate(system_prompt: str, user_message: str, cfg: dict,
     temperature = llm_cfg.get("temperature", 0.0)
     if max_tokens is None:
         max_tokens = llm_cfg.get("max_tokens", 2048)
+    # Validate parameters
+    temperature = max(0.0, min(float(temperature), 2.0))
+    max_tokens = max(1, min(int(max_tokens), 128000))
     return PROVIDERS[provider].generate(
         system_prompt=system_prompt, user_message=user_message,
         api_key=api_key, model=model, temperature=temperature, max_tokens=max_tokens,

src/llm/anthropic.py

@@ -18,7 +18,16 @@ def generate(system_prompt: str, user_message: str, api_key: str,
         return response.content[0].text or ""
     except Exception as e:
         error_type = type(e).__name__
-        raise RuntimeError(f"Anthropic API error ({error_type}). Check your API key and network connection.") from e
+        error_msg = str(e)
+        # Provide specific guidance based on error type
+        if "auth" in error_type.lower() or "authentication" in error_type.lower():
+            raise RuntimeError(f"Anthropic authentication failed. Check your API key.") from e
+        elif "rate" in error_type.lower() or "429" in error_msg:
+            raise RuntimeError(f"Anthropic rate limit exceeded. Wait a moment and try again.") from e
+        elif "model" in error_msg.lower() or "not found" in error_msg.lower():
+            raise RuntimeError(f"Anthropic model error: {error_msg}") from e
+        else:
+            raise RuntimeError(f"Anthropic API error ({error_type}): {error_msg}") from e
 
 def list_models(api_key: str) -> list[str]:
     return list(SUPPORTED_MODELS)

src/llm/gemini.py

@@ -1,29 +1,49 @@
-"""Google Gemini LLM provider."""
-SUPPORTED_MODELS = ["gemini-2.5-pro", "gemini-2.5-flash", "gemini-2.0-flash"]
+"""Google Gemini LLM provider (google-genai SDK)."""
+
+SUPPORTED_MODELS = [
+    "gemini-2.5-pro",
+    "gemini-2.5-flash",
+    "gemini-2.0-flash",
+]
+
 
 def generate(system_prompt: str, user_message: str, api_key: str,
              model: str = None, temperature: float = 0.0,
              max_tokens: int = 2048) -> str:
-    model = model or "gemini-2.5-flash"
-    import google.generativeai as genai
-    genai.configure(api_key=api_key)
-    gen_model = genai.GenerativeModel(
-        model_name=model, system_instruction=system_prompt,
-        generation_config=genai.GenerationConfig(temperature=temperature, max_output_tokens=max_tokens),
-    )
-    response = None
+    from google import genai
+
+    model = model or SUPPORTED_MODELS[0]
+    client = genai.Client(api_key=api_key)
+
     try:
-        response = gen_model.generate_content(user_message)
+        response = client.models.generate_content(
+            model=model,
+            contents=user_message,
+            config=genai.types.GenerateContentConfig(
+                system_instruction=system_prompt,
+                temperature=temperature,
+                max_output_tokens=max_tokens,
+            ),
+        )
+        # Check for blocked responses
+        if not response.text:
+            return ""
         return response.text
-    except ValueError:
-        # Gemini raises ValueError when the response is blocked by safety filters.
-        if response is not None:
-            block_reason = getattr(response, "prompt_feedback", None)
-            if block_reason:
-                return f"[Gemini blocked: {block_reason}]"
-        return "[Gemini blocked: response filtered by safety settings]"
-    except Exception:
-        return "[Gemini error: request failed]"
+    except ValueError as e:
+        # Safety filter block
+        return f"[Gemini blocked: {e}]"
+    except Exception as e:
+        error_type = type(e).__name__
+        error_msg = str(e)
+        if "auth" in error_type.lower() or "authentication" in error_type.lower() or "api_key" in error_msg.lower():
+            raise RuntimeError(f"Gemini authentication failed. Check your API key.") from e
+        elif "rate" in error_type.lower() or "429" in error_msg:
+            raise RuntimeError(f"Gemini rate limit exceeded. Wait a moment and try again.") from e
+        elif "model" in error_msg.lower() or "not found" in error_msg.lower():
+            raise RuntimeError(f"Gemini model error: {error_msg}") from e
+        else:
+            raise RuntimeError(f"Gemini API error ({error_type}): {error_msg}") from e
+
 
 def list_models(api_key: str) -> list[str]:
     return list(SUPPORTED_MODELS)

src/llm/openai.py

@@ -38,7 +38,16 @@ def generate(system_prompt: str, user_message: str, api_key: str,
         return response.choices[0].message.content or ""
     except Exception as e:
         error_type = type(e).__name__
-        raise RuntimeError(f"OpenAI API error ({error_type}). Check your API key and network connection.") from e
+        error_msg = str(e)
+        # Provide specific guidance based on error type
+        if "auth" in error_type.lower() or "authentication" in error_type.lower():
+            raise RuntimeError(f"OpenAI authentication failed. Check your API key.") from e
+        elif "rate" in error_type.lower() or "429" in error_msg:
+            raise RuntimeError(f"OpenAI rate limit exceeded. Wait a moment and try again.") from e
+        elif "model" in error_msg.lower() or "not found" in error_msg.lower():
+            raise RuntimeError(f"OpenAI model error: {error_msg}") from e
+        else:
+            raise RuntimeError(f"OpenAI API error ({error_type}): {error_msg}") from e
 
 def list_models(api_key: str) -> list[str]:
     return list(SUPPORTED_MODELS)

src/prompts.py

@@ -254,7 +254,7 @@ def build_query_understanding_prompt(
             schema_text=sql_schema_summary
         )
     else:
-        sql_routing_block = ""
+        sql_routing_block = '(No SQL tables available — always use route "vector" and omit sql_query.)\n'
 
     if kb_overview:
         kb_overview_block = (

src/query_engine.py

@@ -93,7 +93,7 @@ def understand_query(
             "You are a query reformulation assistant. Return only JSON.",
             prompt,
             cfg,
-            max_tokens=256,
+            max_tokens=512,
         )
         result = _parse_qu_result(raw, user_query)
     except Exception:

src/readers/csv_tab.py

@@ -19,7 +19,8 @@ def read_csv_tab(file_path: str, delimiter: str = None) -> list[dict]:
     row_texts = []
     for row in rows[1:]:
         parts = []
-        for header, val in zip(headers, row):
+        padded_row = list(row) + [""] * max(0, len(headers) - len(row))
+        for header, val in zip(headers, padded_row[:len(headers)]):
             val = val.strip().strip('"')
             if val:
                 parts.append(f"{header}: {val}")

src/readers/docx.py

@@ -5,6 +5,20 @@ def read_docx(file_path: str) -> list[dict]:
     from docx import Document
     doc = Document(file_path)
     paragraphs = [p.text.strip() for p in doc.paragraphs if p.text.strip()]
+
+    # Extract embedded tables
+    for table in doc.tables:
+        for row in table.rows:
+            seen_elements = set()
+            cells = []
+            for cell in row.cells:
+                if id(cell._element) not in seen_elements:
+                    seen_elements.add(id(cell._element))
+                    if cell.text.strip():
+                        cells.append(cell.text.strip())
+            if cells:
+                paragraphs.append(" | ".join(cells))
+
     if not paragraphs:
         return []
     return [{"page": 1, "text": "\n\n".join(paragraphs)}]

src/readers/excel.py

@@ -44,8 +44,9 @@ def _rows_to_pages(rows: list[tuple], sheet_name: str) -> list[dict]:
     row_texts = []
     for row in rows[1:]:
         parts = []
-        for header, val in zip(headers, row):
-            if val is not None and str(val).strip():
+        padded_row = list(row) + [None] * max(0, len(headers) - len(row))
+        for header, val in zip(headers, padded_row[:len(headers)]):
+            if val is not None and str(val).strip() and str(val).lower() not in ("nan", "nat", "<na>", "inf", "-inf"):
                 parts.append(f"{header}: {val}")
         if parts:
             row_texts.append("; ".join(parts))

src/readers/pdf.py

@@ -2,7 +2,7 @@
 
 
 def read_pdf(file_path: str) -> list[dict]:
-    from PyPDF2 import PdfReader
+    from pypdf import PdfReader
     reader = PdfReader(file_path)
     pages = []
     for i, page in enumerate(reader.pages):

src/readers/rdata.py

@@ -6,8 +6,9 @@ def read_rdata(file_path: str) -> list[dict]:
     result = pyreadr.read_r(file_path)
     pages = []
     for name, df in result.items():
+        name_display = name if name is not None else "data"
         headers = list(df.columns)
-        header_line = f"Object: {name} | Columns: {', '.join(headers)}\n"
+        header_line = f"Object: {name_display} | Columns: {', '.join(headers)}\n"
         block = []
         block_chars = len(header_line)
         block_start = 1
@@ -15,14 +16,14 @@ def read_rdata(file_path: str) -> list[dict]:
             parts = []
             for col in headers:
                 val = row[col]
-                if val is not None and str(val).strip() and str(val) != "nan":
+                if val is not None and str(val).strip() and str(val).lower() not in ("nan", "nat", "<na>", "inf", "-inf"):
                     parts.append(f"{col}: {val}")
             if not parts:
                 continue
             row_text = "; ".join(parts)
             if block and block_chars + len(row_text) + 1 > MAX_CHUNK_CHARS:
                 text = header_line + "\n".join(block)
-                pages.append({"page": f"{name}_rows_{block_start}-{block_start + len(block) - 1}", "text": text})
+                pages.append({"page": f"{name_display}_rows_{block_start}-{block_start + len(block) - 1}", "text": text})
                 block = []
                 block_chars = len(header_line)
                 block_start = idx + 1
@@ -30,5 +31,5 @@ def read_rdata(file_path: str) -> list[dict]:
             block_chars += len(row_text) + 1
         if block:
             text = header_line + "\n".join(block)
-            pages.append({"page": f"{name}_rows_{block_start}-{block_start + len(block) - 1}", "text": text})
+            pages.append({"page": f"{name_display}_rows_{block_start}-{block_start + len(block) - 1}", "text": text})
     return pages

src/readers/stata.py

@@ -29,7 +29,7 @@ def _dataframe_to_pages(df, meta) -> list[dict]:
         parts = []
         for col in headers:
             val = row[col]
-            if val is not None and str(val).strip() and str(val) != "nan":
+            if val is not None and str(val).strip() and str(val).lower() not in ("nan", "nat", "<na>", "inf", "-inf"):
                 parts.append(f"{col}: {val}")
         if not parts:
             continue

src/retriever.py

@@ -4,6 +4,20 @@ import re
 
 from src.search import search, format_web_results_as_context
 
+_collection_cache = {}
+
+
+def _get_cached_collection(cfg: dict):
+    """Get or create cached ChromaDB collection."""
+    from src.ingest import get_chroma_collection
+    db_path = cfg.get("paths", {}).get("vector_db", "chroma_db")
+    embed_provider = cfg.get("embeddings", {}).get("provider", "local")
+    cache_key = f"{db_path}:{embed_provider}"
+    if cache_key not in _collection_cache:
+        _collection_cache[cache_key] = get_chroma_collection(cfg)
+    return _collection_cache[cache_key]
+
+
 NO_SOURCES_REFUSAL = (
     "I don't have any information on this topic in my knowledge base. "
     "No relevant local documents or web sources were found. "
@@ -23,18 +37,17 @@ def retrieve_from_vectordb(query: str, cfg: dict) -> list[dict]:
     is included as a fallback so that meta-questions about the knowledge
     base can still be answered.
     """
-    from src.ingest import get_chroma_collection
-
     retrieval_cfg = cfg.get("retrieval", {})
     top_k = retrieval_cfg.get("top_k", 50)
     max_distance = retrieval_cfg.get("max_distance", 0.55)
-    collection = get_chroma_collection(cfg)
+    collection = _get_cached_collection(cfg)
 
-    if collection.count() == 0:
+    total = collection.count()
+    if total == 0:
         return []
 
     # Query a large candidate pool, then filter by relevance
-    candidate_count = min(top_k, collection.count())
+    candidate_count = min(top_k, total)
     results = collection.query(
         query_texts=[query],
         n_results=candidate_count,
@@ -193,6 +206,8 @@ def _build_fallback_sql_query(query: str, cfg: dict) -> str | None:
         for col in text_cols:
             for word in words:
                 safe_word = word.replace("'", "''")
+                if not re.match(r'^\w+$', safe_word):
+                    continue  # Skip non-alphanumeric words
                 conditions.append(f'"{col}" LIKE \'%{safe_word}%\'')
 
         if conditions:
@@ -265,6 +280,9 @@ def _try_alternate_columns(sql_query: str, cfg: dict) -> tuple[list[dict], str]
 
     max_rows = cfg.get("sql", {}).get("max_rows", 200)
     safe_value = search_value.replace("'", "''")
+    # Strip dangerous characters for defense-in-depth
+    if not re.match(r"^[\w\s.,'-]+$", safe_value):
+        safe_value = re.sub(r"[;'\\\"]", "", safe_value)
     for col in text_cols:
         alt_query = f'SELECT * FROM "{table_name}" WHERE "{col}" LIKE \'%{safe_value}%\' LIMIT {max_rows}'
         rows = execute_sql_query(alt_query, cfg)
@@ -316,7 +334,7 @@ def _run_sql_retrieval(sql_query: str, cfg: dict) -> tuple[list[dict], str, str]
     # Step 3: if still no results, try word-level fuzzy (LIKE '%Korea%')
     if not sql_rows:
         word_fuzzy = make_fuzzy_query(sql_query, word_level=True)
-        if word_fuzzy and word_fuzzy != (make_fuzzy_query(sql_query) or ""):
+        if word_fuzzy and word_fuzzy != (fuzzy or ""):
             sql_rows = execute_sql_query(word_fuzzy, cfg)
             if sql_rows:
                 effective_query = word_fuzzy
@@ -377,7 +395,7 @@ def retrieve(
         and not sql_rows
         and (
             (route in ("vector", "both") and not db_results)
-            or route == "sql"
+            or (route == "sql" and not db_results)
         )
     )
     if sql_fallback_needed:
@@ -387,11 +405,11 @@ def retrieve(
             if fallback_query:
                 sql_rows, sql_context, sql_match_type = _run_sql_retrieval(fallback_query, cfg)
         else:
-            # route is "vector" or "both" — existing fallback logic
-            if not sql_query:
-                sql_query = _build_fallback_sql_query(query, cfg)
-            if sql_query:
-                sql_rows, sql_context, sql_match_type = _run_sql_retrieval(sql_query, cfg)
+            # route is "vector" or "both" — build fresh keyword query
+            # (if sql_query exists, it was already tried above and returned nothing)
+            fallback_sql = _build_fallback_sql_query(query, cfg)
+            if fallback_sql:
+                sql_rows, sql_context, sql_match_type = _run_sql_retrieval(fallback_sql, cfg)
 
     # ── Web search ───────────────────────────────────────────────────
     web_enabled = cfg.get("web_search", {}).get("enabled", False)

src/search/__init__.py

@@ -6,11 +6,14 @@ BACKENDS = {
     "none": lambda query, limit=5: [],
 }
 
-def search(query: str, backend: str = "semantic_scholar", limit: int = 5) -> list[dict]:
+def search(query: str, backend: str = "none", limit: int = 5) -> list[dict]:
     fn = BACKENDS.get(backend)
     if fn is None:
-        raise ValueError(f"Unknown search backend: {backend}. Available: {list(BACKENDS.keys())}")
-    return fn(query, limit=limit)
+        return []
+    try:
+        return fn(query, limit=limit)
+    except Exception:
+        return []
 
 def format_web_results_as_context(results: list[dict]) -> str:
     if not results:

src/search/semantic_scholar.py

@@ -44,7 +44,7 @@ def search_papers(query: str, limit: int = 5) -> list[dict]:
         results.append({
             "title": paper.get("title", "Untitled"), "authors": authors,
             "year": paper.get("year"), "abstract": abstract, "url": url,
-            "citation_count": paper.get("citationCount", 0), "source_type": "web_search",
+            "citation_count": paper.get("citationCount") or 0, "source_type": "web_search",
         })
-    results.sort(key=lambda x: x.get("citation_count", 0), reverse=True)
+    results.sort(key=lambda x: x.get("citation_count") or 0, reverse=True)
     return results

src/sql_ingest.py

@@ -565,6 +565,7 @@ def _ingest_tables(
             placeholders = ", ".join(["?"] * len(safe_headers))
             insert_sql = f'INSERT INTO "{table_name}" VALUES ({placeholders})'
 
+            batch = []
             for row in rows:
                 values = []
                 for col_idx, col_type in enumerate(col_types):
@@ -583,7 +584,12 @@ def _ingest_tables(
                             values.append(None)
                     else:
                         values.append(str(raw).strip())
-                conn.execute(insert_sql, values)
+                batch.append(values)
+                if len(batch) >= 1000:
+                    conn.executemany(insert_sql, batch)
+                    batch = []
+            if batch:
+                conn.executemany(insert_sql, batch)
 
             conn.commit()
 

src/sql_retriever.py

@@ -14,6 +14,15 @@ _DANGEROUS_KEYWORDS = re.compile(
 )
 
 
+def _strip_quoted(sql: str) -> str:
+    """Remove quoted strings and identifiers for safe keyword checking."""
+    # Remove single-quoted strings (handles '' escaping)
+    result = re.sub(r"'(?:[^']|'')*'", "", sql)
+    # Remove double-quoted identifiers
+    result = re.sub(r'"[^"]*"', "", result)
+    return result
+
+
 def _strip_sql_comments(sql: str) -> str:
     """Remove SQL comments (single-line -- and multi-line /* */) from a query."""
     prev = None
@@ -36,12 +45,16 @@ def _validate_sql(sql: str) -> bool:
         return False
     if ";" in stripped:
         return False
-    if not stripped.upper().startswith("SELECT"):
+    if not stripped.upper().startswith("SELECT "):
+        return False
+    unquoted = _strip_quoted(stripped)
+    if _DANGEROUS_KEYWORDS.search(unquoted):
         return False
-    if _DANGEROUS_KEYWORDS.search(stripped):
+    # Block access to SQLite system tables
+    if re.search(r'\bsqlite_(master|schema|temp_master|temp_schema)\b', stripped, re.IGNORECASE):
         return False
     # Block subqueries: reject if more than one SELECT keyword
-    if len(re.findall(r'\bSELECT\b', stripped, re.IGNORECASE)) > 1:
+    if len(re.findall(r'\bSELECT\b', unquoted, re.IGNORECASE)) > 1:
         return False
     return True
 
@@ -102,7 +115,7 @@ def execute_sql_query(sql_query: str, cfg: dict) -> list[dict]:
 
 def format_sql_results_as_context(
     rows: list[dict], sql_query: str, source_file: str,
-    table_info: dict = None, max_rows: int = 0,
+    table_info: dict = None, max_rows: int = None,
 ) -> str:
     """Format SQL result rows as context for the verification pipeline.
 
@@ -121,7 +134,7 @@ def format_sql_results_as_context(
         return ""
 
     row_label = f"Rows returned: {len(rows)}"
-    if max_rows and len(rows) >= max_rows:
+    if max_rows is not None and len(rows) >= max_rows:
         row_label += f" (truncated — more rows may exist, limit was {max_rows})"
 
     parts = [
@@ -186,7 +199,13 @@ def make_fuzzy_query(sql: str, word_level: bool = False) -> str | None:
     )
 
     if not word_level:
-        new_sql, count = pattern.subn(r"\1LIKE '%\2%'", sql)
+        def _phrase_replace(m):
+            col_part = m.group(1)
+            value = m.group(2).replace("'", "''")
+            # Escape LIKE wildcards in the value
+            value = value.replace("%", "\\%").replace("_", "\\_")
+            return f"{col_part}LIKE '%{value}%' ESCAPE '\\'"
+        new_sql, count = pattern.subn(_phrase_replace, sql)
         if count == 0:
             return None
         return new_sql
@@ -194,18 +213,22 @@ def make_fuzzy_query(sql: str, word_level: bool = False) -> str | None:
     # Word-level: extract significant words from each value
     def _word_replace(m):
         col_part = m.group(1)
-        value = m.group(2)
+        value = m.group(2).replace("'", "''")
+        # Escape LIKE wildcards in the value
+        value = value.replace("%", "\\%").replace("_", "\\_")
         words = [
-            w for w in re.split(r'\W+', value)
+            w.replace("'", "''").replace("%", "\\%").replace("_", "\\_")
+            for w in re.split(r'\W+', m.group(2))
             if len(w) >= 3 and w.lower() not in _FUZZY_STOPWORDS
         ]
+        esc = " ESCAPE '\\'"
         if not words:
             # Fallback: use the original value as phrase LIKE
-            return f"{col_part}LIKE '%{value}%'"
+            return f"{col_part}LIKE '%{value}%'{esc}"
         if len(words) == 1:
-            return f"{col_part}LIKE '%{words[0]}%'"
+            return f"{col_part}LIKE '%{words[0]}%'{esc}"
         # Multiple words: join with AND on the same column
-        conditions = [f"{col_part}LIKE '%{w}%'" for w in words]
+        conditions = [f"{col_part}LIKE '%{w}%'{esc}" for w in words]
         return "(" + " AND ".join(conditions) + ")"
 
     new_sql, count = pattern.subn(_word_replace, sql)

src/verifier.py

@@ -1,5 +1,5 @@
 """
-6-Layer Anti-Hallucination Verification Pipeline.
+7-Layer Anti-Hallucination Verification Pipeline.
 
 Layers:
     0 — No-source refusal (pre-LLM gate)
@@ -7,6 +7,7 @@ Layers:
     2 — Soft max-token cap (proportional to context length)
     3 — LLM-as-verifier (structured JSON audit)
     4 — Semantic similarity cross-check (term-overlap heuristic)
+    4.5 — Deterministic citation audit (validate_citations)
     5 — Warning-phrase scanner (advisory flags)
 
 The main entry point is ``verify_and_respond``.
@@ -159,17 +160,25 @@ def validate_citations(response: str, retrieval_result: dict) -> list[str]:
     """
     warnings = []
 
-    # Count actual sources available
-    db_count = len(retrieval_result.get("db_results", []))
+    # Count unique source documents (not raw chunks)
+    db_results = retrieval_result.get("db_results", [])
+    unique_db_sources = set()
+    for chunk in db_results:
+        source = chunk.get("metadata", {}).get("source", "")
+        if source:
+            unique_db_sources.add(source)
+    db_count = len(unique_db_sources) if unique_db_sources else (1 if db_results else 0)
     web_count = len(retrieval_result.get("web_results", []))
-    sql_count = len(retrieval_result.get("sql_results", []))
+    sql_count = 1 if retrieval_result.get("sql_results") else 0
     total_sources = db_count + web_count + sql_count
 
     if total_sources == 0:
         return warnings
 
-    # Extract citation numbers from response body
-    citation_nums = set(int(m) for m in re.findall(r"\[(\d+)\]", response))
+    # Extract citation numbers from response BODY only (not References section)
+    refs_split = re.split(r'(?im)^#+\s*(references|sources)\b|^\*\*(references|sources)\*\*', response)
+    body_text = refs_split[0] if refs_split else response
+    citation_nums = set(int(m) for m in re.findall(r"\[(\d+)\]", body_text))
     if not citation_nums:
         return warnings
 
@@ -181,10 +190,12 @@ def validate_citations(response: str, retrieval_result: dict) -> list[str]:
         )
 
     # Check that source file names from retrieval appear in the References section
-    refs_match = re.search(r"(?i)\b(references|sources)\b.*", response, re.DOTALL)
+    refs_match = re.search(
+        r"(?im)(^#+\s*(references|sources)\b|^\*\*(references|sources)\*\*).*",
+        response, re.DOTALL,
+    )
     if refs_match:
         refs_text = refs_match.group(0).lower()
-        db_results = retrieval_result.get("db_results", [])
         matched_sources = 0
         for chunk in db_results:
             source = chunk.get("metadata", {}).get("source", "")
@@ -193,7 +204,24 @@ def validate_citations(response: str, retrieval_result: dict) -> list[str]:
                 filename = source.rsplit("/", 1)[-1].lower()
                 if filename in refs_text:
                     matched_sources += 1
-        if db_results and matched_sources == 0:
+        # Also check SQL source files
+        sql_results = retrieval_result.get("sql_results", [])
+        for sql_chunk in sql_results:
+            sql_source = sql_chunk.get("metadata", {}).get("source", "")
+            if sql_source:
+                sql_filename = sql_source.rsplit("/", 1)[-1].lower()
+                if sql_filename in refs_text:
+                    matched_sources += 1
+        # Also check web source URLs
+        web_results = retrieval_result.get("web_results", [])
+        for web_chunk in web_results:
+            web_url = web_chunk.get("metadata", {}).get("url", "")
+            if web_url and web_url.lower() in refs_text:
+                matched_sources += 1
+        # Only warn when db_results are present and are the primary source
+        # and no filenames matched from any source type
+        has_any_source = db_results or sql_results or web_results
+        if has_any_source and matched_sources == 0 and db_results:
             warnings.append(
                 "References section does not mention any filenames from "
                 "the retrieved local sources. Citations may be fabricated."
@@ -267,7 +295,7 @@ def verify_and_respond(
     query: str, retrieval_result: dict, cfg: dict,
     original_query: str = "",
 ) -> dict:
-    """Generate a response and run it through the 6-layer verification stack.
+    """Generate a response and run it through the 7-layer verification stack.
 
     Args:
         query: The display query (reformulated by QU layer).
@@ -350,9 +378,6 @@ def verify_and_respond(
             "iterations": 0,
         }
 
-    # ── Layer 4.5: Deterministic citation audit ───────────────────────
-    citation_warnings = validate_citations(response, retrieval_result)
-
     # ── Short-circuit if verification is disabled ─────────────────────────
     verification_cfg = cfg.get("verification", {})
     if not verification_cfg.get("enabled", True):
@@ -407,7 +432,8 @@ def verify_and_respond(
                 max_tokens=1024,
             )
         except Exception:
-            raw_verification = ""
+            # Verification LLM failed — skip correction, continue to next iteration
+            continue
         vr = parse_verification_result(raw_verification)
 
         if vr.get("pass", False):
@@ -483,8 +509,9 @@ def verify_and_respond(
         "against the provided sources. Some claims may lack adequate "
         "grounding. Please cross-check important facts."
     )
-    if citation_warnings:
-        warning += "\n**Citation issues:** " + "; ".join(citation_warnings)
+    final_citation_warnings = validate_citations(response, retrieval_result)
+    if final_citation_warnings:
+        warning += "\n**Citation issues:** " + "; ".join(final_citation_warnings)
     return {
         "response": response + warning,
         "refused": False,