Local Semantic Search over TMFNK on a small Mac

🔧 Local Semantic Search over TMFNK


What it is	Local “search by vibe” over this site’s Hugo `content/` tree
Stack	Python + uv, `llama-server`, bge-m3 embeddings, optional LFM2.5-230M, NumPy cosine
Status	Design locked; not built yet
Code	Sibling repo planned (`tmfnk-semantic-search`); ops copy ee-case-studies

This site already has Lunr for keyword search in the browser. I want the other half: type “books like a library shelf walk” and get pages whose meaning matches, not only the words. The corpus is already on disk: about 761 Markdown pages under /content, skipping _index.md. I paid the tuition for local models on a small Mac in ee-case-studies (LightRAG over Equal Experts write-ups). This is the build plan that steals those ops habits and drops the graph.

Snippets below are the shape I’d ship. They are not a live repo yet.

What I learned (before writing search code)

LightRAG is the wrong tool for site search. On ee-case-studies it is good for entities and multi-hop themes across the case studies data. It also means a third llama-server (extract) and a slow ingest. For “which of my 750+ posts feel like this query?” you need embeddings and cosine. Graph extract over every article would burn an evening for something Lunr search already kind of covers.
Small RAM forces phased servers. Loading query LLM + embed + extract together OOMs this machine. The fix that stuck in EE: start_servers.sh modes. Here, ingest and search run embed only (bge-m3 on :8081). answer is the only dual-load: embed plus LFM2.5-230M (~146 MB) on :8080. Leave the 1.2B Thinking model for EE.
127.0.0.1, not localhost. EE servers bind IPv4 only. localhost can hit ::1 first and look “down.”
One vector per page is enough for v1. Embed title, description, tags, and the first ~2,000 characters of body. Strip YAML, JSON-LD <script> blocks, and “Back to …” nav lines. Use frontmatter canonical as the result URL.
Ship search before chat. Ranked hits are the product. A 230M blurb over top-k is optional. If answers are weak, stop embed and try a bigger model alone.

How I would build it

content/*.md  →  ingest.py  →  POST /v1/embeddings (bge-m3 :8081)
                            →  data/vectors.npz + data/meta.jsonl

query  →  search.py  →  embed query  →  NumPy cosine  →  top-k

optional: answer.py  →  top-k snippets  →  LFM2.5-230M :8080

Repo layout mirrors EE, thinner middle: scripts/start_servers.sh, ingest.py, search.py, optional answer.py, .env for CONTENT_ROOT and GGUF paths. Deps: httpx, numpy, pyyaml. No LightRAG.

1. Phased `llama-server` (steal from EE, drop extract)

EE’s script already starts embed with --embedding --pooling mean and matches -b / -ub to --ctx-size so long inputs do not blow the batch. For TMFNK search I’d keep that embed block and swap modes:

# scripts/start_servers.sh  (trimmed from ee-case-studies)
start_embed() {
  start_one "bge-m3" "${EMBED_PORT}" "${EMBED_ALIAS}" "${EMBED_MODEL}" \
    "${LOG_DIR}/bge-m3.log" "${LOG_DIR}/bge-m3.pid" \
    --embedding --pooling mean --ctx-size 2048 -b 2048 -ub 2048
}

# ingest | search  →  embed only
# answer           →  embed + LFM2.5-230M on :8080 (not the 1.2B)
# never            →  start all three EE servers on this Mac

Same flags EE uses everywhere: -ngl 999 (Metal) and -np 1 (small KV).

2. Turn Hugo pages into embed text

Walk CONTENT_ROOT (default: the site’s content/). Skip _index.md and draft: true. One record per page:

def page_embed_text(fm: dict, body: str) -> str:
    tags = fm.get("tags") or []
    tag_line = ", ".join(tags) if isinstance(tags, list) else str(tags)
    body = strip_nav_and_jsonld(body)[:2000]
    return (
        f"{fm.get('title', '')}\n"
        f"{fm.get('description', '')}\n"
        f"{tag_line}\n"
        f"{body}"
    )

canonical from frontmatter becomes the hit URL. If it is missing, derive https://tmfnk.com/... from the path the way Hugo slugs it, then spot-check ten pages.

3. Embed one page at a time

EE learned the hard way: batching many long strings into one /v1/embeddings call makes llama-server sum tokens and die. Same rule here. One text per request, talk to 127.0.0.1:

import httpx
import numpy as np

EMBED_URL = "http://127.0.0.1:8081/v1/embeddings"


def embed(text: str) -> np.ndarray:
    r = httpx.post(
        EMBED_URL,
        json={"model": "bge-m3", "input": text},
        timeout=120.0,
    )
    r.raise_for_status()
    return np.asarray(r.json()["data"][0]["embedding"], dtype=np.float32)

Ingest loop: embed each page, append a line to data/meta.jsonl, stack vectors, np.savez_compressed("data/vectors.npz", X=X). Log skips. A full rebuild of ~750+ pages is fine for v1; incremental mtime hashing can wait.

Day one:

scripts/start_servers.sh start ingest   # embed :8081 only
uv run python scripts/ingest.py
scripts/start_servers.sh stop

4. Search is cosine, not a framework

At query time start embed again, embed the query, score against the matrix:

def search(query: str, X: np.ndarray, meta: list[dict], k: int = 8) -> list[dict]:
    q = embed(query)
    q = q / (np.linalg.norm(q) + 1e-9)
    Xn = X / (np.linalg.norm(X, axis=1, keepdims=True) + 1e-9)
    scores = Xn @ q
    idx = np.argpartition(-scores, k)[:k]
    idx = idx[np.argsort(-scores[idx])]
    return [{**meta[i], "score": float(scores[i])} for i in idx]

761 × 1024 is nothing. No FAISS, no Vectorize, no Milvus. Print title, score, canonical URL, short snippet.

scripts/start_servers.sh start search
uv run python scripts/search.py "local AI on Apple Silicon"

5. Optional answers with the 230M model

Only after search feels honest. Retrieve top-5, build a short prompt from titles and snippets, call http://127.0.0.1:8080/v1/chat/completions with LFM2.5-230M. Cap --n-predict around 128–256 so it stays snappy. Always print the raw hits under the blurb so you can see when the tiny model invents.

If that dual-load spikes RAM: embed the query, stop the embed server, start 230M, generate. Sequential beats clever.

6. What I would not do in v1

Port LightRAG or start the extract model
Chunk every article into five vectors on day one
Point clients at localhost
Deploy to Cloudflare before local queries feel useful
Default the answer LLM to the 1.2B Thinking checkpoint next to bge-m3

Worth copying if you already run phased llama-server on Apple Silicon and want semantic search over a Markdown tree without a knowledge graph.

Local Semantic Search over TMFNK on a small Mac