Edge RAG compresses the language model to fit a 6 GB envelope. Does that compression change what the system leaks about its private database? We measured datastore membership inference across a four-rung quantization ladder, dense and hybrid retrieval, two datastores, and six attack families. The generator channel turned out to be flat at chance — and the membership signal turned out to live somewhere else entirely: the retrieval layer.
RAG answers a question by retrieving documents from a datastore, then asking a language model to write an answer from them. Prior membership-inference (MIA) work probes the generator. But the generator only ever sees what retrieval hands it. If membership is decided upstream, then quantizing — or otherwise hardening — the generator is aimed at the wrong layer.
We tested both layers on the same datastores. The generator-side ladder is null: every rung within noise of chance, and formally equivalent to a full-precision baseline. The retrieval-side control is the opposite: a document's rank in the fused index is a near-perfect membership oracle, because members are indexed and non-members are not. That gap — flat generator, saturated retriever — is the paper's thesis, and it is a positive result, not a null.
The generator null still matters: it retires a folklore belief that compression hides datastore contents, and it is what makes the channel claim clean — there is no generator leakage to confound the retrieval measurement.
synthetic:false, one
generator lineage (granite-sft).All values below are re-derived this session from the artifacts named in each tooltip. The paper's own verifier checks every decimal in the manuscript against the same evidence tree.
Pooled per-quant attack AUC is 0.586 / 0.506 / 0.558 / 0.532 for IQ1_M / IQ2_XS / Q2_K / Q4_K_M — flat and non-monotone. For the loss/zlib families, all 16/16 configurations are statistically equivalent to the F16 baseline within a pre-registered margin δ=0.1 (largest observed difference 0.018). No configuration survives Holm correction (0/92). Verdict: no quantization level is a privacy control, and none makes leakage worse.
paper/ieee_main.tex §RQ1 · results/t1/stats.json · results/t1/tost_scores.jsonlA grid of nulls is only credible if the instrument is sensitive. On the unquantized base model, the mask-based attack reaches mean AUC 0.712 while the statistical families sit at chance (loss 0.506, zlib 0.509, S2MIA 0.487). The same harness that reports nulls on quantized models reports strong signal at full precision — where the literature says it lives.
results/t1/f16_control.jsonlA retriever-only control scores membership with no generator involved. The rank of the candidate document in the fused top-10 separates members from non-members at AUC 1.000 on both datastores; the dense embedding cosine reaches 0.876, and BM25 0.627–0.940. This is definitional once seen clearly — members are indexed, non-members are not — and it is the thesis: generator-side defenses are aimed at the wrong layer.
results/t1/retriever_only_verdict.md · scripts/retriever_only.pyWe hypothesized BM25+RRF fusion would amplify leakage, since BM25 ranks are invariant under quantization. Over 40 matched dense/hybrid pairs: mean ΔAUC +0.010, range −0.086 to +0.208, 35/40 within ±0.05 (17 positive, 20 negative, 3 exactly zero). An RRF-k sensitivity lane (k=30/90) is flat, so the null is not an artifact of the one fusion constant.
paper/ieee_main.tex §RQ2 · results/t1/rrfk_sens.jsonlThe entailment-stealth family (MEntA) is the only one that tracks fusion. Hybrid per-quant means climb 0.618 → 0.726 → 0.807 → 0.833 while dense stays flat (0.625 → 0.537 → 0.520 → 0.529). It is a 2-seed scoped arm today — the reason a 5-seed replication is the next mandatory experiment, not a claim yet.
results/t1/raw_sweep.jsonl · results/t1/menta_det.jsonlA pre-retrieval firewall (embedding centroid z-score fused with lexical attack features) abstains before embedding and search. On the real pipeline it costs 0.1 ms p50 warm, abstains on 0/60 legitimate member queries, and leaves loss/MBA AUC unchanged (Δ +0.009 / −0.013). Its detection signal covers frame-shaped attack queries (calibration TPR 0.733 at FPR ≤1%), not plain-query probing. Reported as a null, not hidden.
results/t1/guard_summary.json · paper/ieee_main.tex §RQ3Against an FP16 KV baseline at n=100 per arm: loss ΔAUC 0.019 and zlib 0.0067 — 0.19× and 0.07× the ±2·SE resolution band. The edge stack can halve KV memory with no measurable membership cost.
results/t1/kv-q8/bench.jsonMean AUC over the two dataset-level means at k=60. Toggle retrieval mode; hover a cell for its sidecar row. Values above 0.6 (warm) mark the mask-based and differential-calibration columns; the calibration column is under repair — see the correction ledger.
An honest explainer has to show the corrections, not just the results. Four data-integrity defects were found and fixed in a post-draft audit; a fifth is in flight.
| Defect | What was wrong | Correction |
|---|---|---|
| C1 | Table 1 was stale: 14 of 40 cells did not reproduce from the deduped sweep, and the verifier could not see it. | Table regenerated from a canonical aggregator; fresh recompute now yields 0 mismatches. Dedup policy documented (stochastic re-runs averaged). |
| C2 | The claim verifier matched a number against any evidence row, so a wrong-cell value still passed. | Cell-binding verifier added (--strict-cells): 56 table
cells checked by name; a wrong-cell selftest must fail. |
| C3 | The differential-calibration attack built its calibration set from the evaluation non-members themselves — in-sample, so its reported AUC was inflated (1.000 on one configuration). | Disjoint calibration enforced; early re-run collapses the artifact (e.g. 0.671 → 0.473). Full lane completing now. |
| O3 | An early draft credited grounding for pushing loss attacks to chance — but the ungrounded F16 base is already at chance (loss 0.506, zlib 0.509). | Claim scoped/demoted; an F16-SFT control is the clean experiment that would settle it, scheduled. |
| O4 | Cross-family generalization rested on a single-seed, single-quant, domain-mismatched probe. | Downgraded to a probe; a proper second family is deferred to revision (the pinned runtime cannot serve the candidate architecture). |
This ledger is the reason to trust the rest of the page: the audit was adversarial, the corrections are measured, and the verifier was extended so the same class of error cannot pass again.
Every decimal in the manuscript is machine-checked against the evidence tree. The strict mode binds table cells to named sidecar rows so a value that exists but belongs to the wrong cell fails.
scripts/cove_verify.py --strict-cells
307/307 claims · 56 cells · PASS
Four audit tracks (evidence, statistics, code/artifact, threat/novelty) plus an adversarial pass produced 33 findings — 7 critical — including the training-data contamination that forced a full rebuild of the SFT set to exclude all datastore documents.
docs/audits/Q1_WEAKNESS_REVIEW.md
All 38 entries cross-checked against live APIs: 33 clean, 5 corrected (a wrong venue, a wrong DOI, a wrong volume), 0 unverifiable.
Faithfulness is 0.809 against a 0.85 gate — not met. Two defects are named: a query-echo mode and zero abstention. The gate is never printed as met.
If Gate 2 or 3 fails, the paper falls back to an honest negative-results measurement rather than a manufactured positive.
Human-gated items remain: real author block and affiliations, and activating the IEEE AI-use disclosure verbatim. These are submission logistics, not research.