Update README.md

README.md

@@ -6,7 +6,6 @@ tags:
   - geometric-deep-learning
   - cross-modal
   - multi-modal
-  - retrieval
   - pentachoron
   - procrustes
   - bert
@@ -14,41 +13,116 @@ tags:
   - whisper
   - esm2
   - codebert
-  - contrastive-learning
+  - archived
+  - research-precursor
 language:
   - en
 library_name: pytorch
 pipeline_tag: feature-extraction
 ---
 
-# GEOLIP-Bertenstein - AKA the GEOLIP-Conduit prototype
+# GEOLIP-Bertenstein — multimodal Procrustes alignment prototype
+
+**An early test of a single question: can whitened Procrustes alignment build a usable
+shared space across independently-trained encoders of different modalities and different
+dimensionalities?**
+
+Four frozen experts — DINOv2-large, Whisper-large-v3, ESM-2-650M, CodeBERT-base — each
+pre-aligned into BERT-large's text space by a whitened Procrustes map, then fused by a
+single bidirectional transformer layer. 41.5M trainable parameters over ~11 minutes.
+
+> ### Status: archived precursor — superseded, retained for lineage
+>
+> This is one of the **preliminary Procrustes-whitening exercises** that opened the
+> alignment chain in this account. It predates `geolip-captionbert-8192` (Mar 18),
+> `geolip-constellation-core` (Mar 21), the `geolip-vit-*-x3` line and
+> `geolip-vit-captionbank-coco` (Jul 28), and `clip-vitb-mini-distilled` (Jul 30). It
+> follows directly from `procrustes-analysis` (Mar 6), which profiled Procrustes
+> alignability across 17 pretrained models in *weight* space; Bertenstein was the first
+> attempt to apply the same idea in *activation* space across modalities.
+>
+> **A 2026-07-31 audit found the original evaluation protocol invalid, and the headline
+> retrieval numbers on this card have been withdrawn and replaced below.** The
+> architecture is best understood as an alignment *determinant* — an instrument for
+> deciding whether the approach carried — rather than as a deployable model. The answer
+> it returned was largely negative, and it was not pursued because it did not yield the
+> **invariant elements needed for distillation**, which is the direction this line went
+> instead.
+>
+> The negative result was local, not general. Later alignment work found Procrustes to
+> be **substantially more capable than this prototype indicated**, and a corrected form
+> of exactly this operation is now used as a standing instrument (see *Lineage*).
 
-**A multi-expert geometric fusion transformer that bridges 4 independently-trained encoders into a shared embedding space using BERT-large as a universal text hub.**
-
-One layer. One epoch. Perfect cross-modal retrieval. Universal pentachoron geometry.
-
-The entire design is meant to create an alignment bridge with multiple different modalities, multiple different dimensional sizes, and multiple different independent structures into a format
-that can be consumed downstream by another system for useful association.
-
-This system uses a form of procrustes whitening analysis that aligns multiple systems of information through a shared structural boundary,
-with well defined and understood complexity association differentiations.
-
-This is not meant to be a MOE, this is meant to be a collective understanding and cooperation of alignments through direct and indirect association.
-
-## Results
+---
 
-| Expert Pair | R@1 | Cosine | Pentachoron CV |
+## What it was for
+
+The design goal was a shared multimodal space obtained *without* training the encoders:
+take independently-trained models with incompatible dimensionalities (768 / 1024 / 1280),
+Procrustes-align each into a common text-anchored frame, and let a small fusion layer do
+the rest. If that worked, it would be a cheap route to cross-modal association — and, by
+extension, to distillation targets that don't require running the experts at inference.
+
+The intended pay-off was *indirect* association: train only text↔X pairs and get X↔Y
+alignment transitively through the shared text hub.
+
+## Corrected results
+
+The original card reported R@1 ≈ 1.0000 across all expert pairs. Those figures came from
+an evaluation that encoded both modalities in a **single forward pass** and compared the
+two readouts taken from that same pass — so the two vectors had already attended to one
+another. Re-measured with independent encoding (the standard retrieval protocol):
+
+| measurement | value | note |
+|---|---|---|
+| text↔image, independent encoding | **R@1 0.0007** | N=4962 COCO val, chance 0.0002 |
+| text↔image, original joint-pass protocol | R@1 1.0000 | withdrawn — see below |
+| Procrustes map alone, out-of-sample | **R@1 0.0088** | N=4962, 43.5× chance; cos 0.112 vs 0.011 random |
+| text↔{audio, code, protein} | withdrawn | not yet re-measured under the corrected protocol |
+
+Three findings from the audit, kept here so the record is complete:
+
+1. **Joint-pass evaluation.** `<TEXT>` and `<IMAGE>` are read from one unmasked
+   sequence, and the training objective compares the same two readouts. That objective
+   is satisfiable without any cross-modal alignment. Diagnostics: R@1 stayed at 1.0000
+   from gallery size 64 to 4096 (real retrieval decays with gallery size), deliberately
+   mismatched pairs scored identically to matched ones, and the two readouts agreed at
+   cos 0.94 on pure-noise inputs.
+2. **The `image_coco_test` cache has no usable captions.** It was built from
+   COCO-Caption `split="test"`, whose annotations are withheld; upstream encodes that as
+   `answer: ['None']`, and the ingestion embedded the literal string. Every row is the
+   same 3-token sequence. The "40K test" row was additionally computed on n=5000, not
+   40K. **`image_coco`, `audio_librispeech`, `code_csn` and `protein_p2t` are unaffected
+   and carry real text.**
+3. **The alignment figures were in-sample.** The Procrustes map is fit on the first
+   5,000 rows and was reported on those same rows. The rotation maximizes the very
+   statistic being reported, and it has ~524k free parameters fit from 5,000 pairs; at
+   that ratio a no-relationship null reproduces cos ≈ 0.39. The previously reported
+   0.377–0.440 figures should be read as in-sample, in the whitened frame.
+
+### Procrustes pre-alignment — corrected
+
+| Expert | dimension | reported (in-sample, whitened frame) | out-of-sample |
 |---|---|---|---|
-| text ↔ audio (1.5K val) | **1.0000** | 0.972 | **0.203** |
-| text ↔ code (5K val) | **0.9996** | 0.988 | **0.195** |
-| text ↔ image (5K val) | **1.0000** | 0.986 | **0.196** |
-| text ↔ protein (2.3K val) | **0.9987** | 0.979 | **0.200** |
-| text ↔ image (40K test) | **1.0000** | 0.980 | **0.208** |
-
-All CV values converge to the **0.20 ± 0.01 universal band** — the same geometric constant measured across 17 neural architectures before this model existed.
-
-The current results only used a partial protein and partial audio dataset for preliminary, which left the sample size smaller and less conclusively potential
-as a full dataset run can yield due to the increased data capacity for alignment.
+| audio | 1280 → 1024 (PCA) | 0.4404 | not re-measured |
+| code | 768 → 1024 (pad) | 0.4036 | not re-measured |
+| image | 1024 → 1024 | 0.4107 | **cos 0.112, R@1 0.0088** |
+| protein | 1280 → 1024 (PCA) | 0.3771 | not re-measured |
+
+The out-of-sample image figure is small but **not** null — it sits ~43× above chance and
+~17× above a matched no-relationship null. Whitened Procrustes does recover real
+cross-modal structure here; it was the reported *magnitude* that was an artifact of the
+fit protocol.
+
+### Pentachoron CV
+
+The original claim of a universal 0.20 ± 0.01 band across modalities does not hold as
+stated: CV is dimension-dependent. Measured on untrained normalized Gaussians —
+d=16 → 0.211, d=64 → 0.090, d=256 → 0.040, d=1024 → 0.021. This model reads CV ≈ 0.196
+at a nominal d=1024, with an **effective rank of ~20**. That is consistent with the
+0.20 band being the natural value on S¹⁵ and the embedding having collapsed to roughly
+16–20 effective dimensions, rather than with convergence to a universal constant.
+**Report effective rank alongside CV**; CV alone cannot distinguish the two.
 
 ## Architecture
 
@@ -71,111 +145,111 @@ as a full dataset run can yield due to the increased data capacity for alignment
     └──────┘
 ```
 
-**Frozen encoders** (not trained, not included in this repo):
-- **BERT-large** (336M) — universal text hub
-- **DINOv2-large** (302M) — natural images
-- **Whisper-large-v3** encoder (1.5B) — speech audio
-- **ESM-2-650M** (652M) — protein sequences
-- **CodeBERT-base** (125M) — source code
-
-**Trainable fusion** (this repo): **41.5M params**
-- 1 shared transformer layer (8.9M)
-- 5 expert modules (text + 4 modalities, ~7M each)
-- Includes Procrustes pre-alignment buffers per expert
-
-## How It Works
-
-1. **Procrustes Pre-Alignment**: Before training, compute optimal orthogonal rotation to align each expert's coordinate system with BERT's text space. Whitened Procrustes with centering.
-
-2. **Expert Modules**: Each modality gets a projection layer, learned cross-attention pooling (257 image patches → 16 tokens, 1500 audio frames → 16 tokens, etc.), a special `<|MODALITY|>` token, and an output head.
-
-3. **Fusion Sequence**: `[<|TEXT|>] [text_tokens] [<|IMAGE|>] [img_tokens] ...` — bidirectional self-attention across all modalities. Text tokens attend to image patches. Audio tokens attend to text tokens.
-
-4. **Geometric Loss**: InfoNCE contrastive + pentachoron CV variance (Cayley-Menger) + Procrustes SVD alignment. The geometric components drive the embedding manifold toward the universal 0.20 CV band.
-
-5. **Text Hub**: All modalities pair with text during training. Cross-modal alignment (e.g., audio↔image) emerges transitively through the shared text space.
-
-## Procrustes Pre-Alignment
-
-| Expert | cos before | cos after | Dimension |
-|---|---|---|---|
-| audio | 0.0004 | **0.4404** | 1280 → 1024 (PCA down) |
-| code | -0.0016 | **0.4036** | 768 → 1024 (zero-pad up) |
-| image | 0.0038 | **0.4107** | 1024 → 1024 (direct) |
-| protein | 0.0005 | **0.3771** | 1280 → 1024 (PCA down) |
+**Frozen encoders** (not trained, not included here): BERT-large (336M) text hub,
+DINOv2-large (302M), Whisper-large-v3 encoder (1.5B), ESM-2-650M, CodeBERT-base (125M).
+
+**Trainable fusion** (this repo): 41.5M — one shared transformer layer (8.9M) plus five
+expert modules (~7M each), including the Procrustes pre-alignment buffers.
+
+**How it runs.** Each expert is centered, dimension-matched (PCA down or zero-pad up),
+whitened, rotated into text space, and un-whitened. Each modality gets a projection, a
+learned cross-attention pooling (257 image patches → 16 tokens; 1500 audio frames → 16),
+a `<|MODALITY|>` token and an output head. The fusion sequence is
+`[<|TEXT|>] [text] [<|IMAGE|>] [img] …` with bidirectional attention, and each modality's
+embedding is read from its special-token position. Loss: InfoNCE + a pentachoron volume
+term + a Procrustes alignment term.
+
+**Known defect in that description:** because attention is unmasked across modalities and
+both embeddings are read from the same sequence, the `<|TEXT|>` readout is not a function
+of the text alone. Measured: it retains ~0.8% of itself when the attached image is
+swapped, and ~99.5% when the *text* is swapped. An untrained model shows the same
+asymmetry more mildly (34% / 59%), so this is a property of the unmasked joint sequence
+that training amplified. It also means the intended *indirect* association had no
+mechanism to travel on — alignments cannot compose through an anchor that isn't a
+function of its own modality.
 
 ## Training
 
-- **Data**: COCO-Caption (40K), LibriSpeech clean-100 (10K), CodeSearchNet Python (50K), Protein2Text-QA (15K)
-- **Schedule**: 3 epochs, round-robin across experts, cosine LR 3e-4 → 1e-6
-- **Hardware**: NVIDIA RTX PRO 6000 Blackwell (102 GB VRAM)
-- **Time**: ~11 minutes total (3 × ~220s)
-
-## Key Finding: Universal Pentachoron Geometry
-
-The coefficient of variation (CV) of pentachoron (4-simplex) volumes, measured via Cayley-Menger determinants, converges to **0.20 ± 0.01** across:
-
-- 17 pretrained models (T5, BERT, CLIP, DINOv2, SD UNets, VAEs)
-- 5 architecture families (transformer, UNet, convolutional autoencoder)
-- 4 modalities in this fusion model (audio, code, image, protein)
-
-This constant emerges in ANY embedding space that reaches convergence through gradient-based contrastive learning, regardless of modality, architecture, or training objective.
-
-## File Structure
+- **Data**: COCO-Caption (40K), LibriSpeech clean-100 (10K), CodeSearchNet Python (50K),
+  Protein2Text-QA (15K)
+- **Schedule**: 3 epochs, round-robin across experts, cosine LR 3e-4 → 1e-6, batch 256
+- **Hardware**: NVIDIA RTX PRO 6000 Blackwell (102 GB) · **Time**: ~11 min (3 × ~220s)
+
+Precomputed embedding caches (Arrow): [`AbstractPhil/bertenstein-v1`](https://huggingface.co/datasets/AbstractPhil/bertenstein-v1)
+— configs `image_coco`, `image_coco_test`, `audio_librispeech`, `code_csn`, `protein_p2t`.
+
+## Lineage — what carried forward
+
+- **Whitened Procrustes cross-space alignment.** The technique held; the *fit discipline*
+  didn't. Its corrected descendant — an fp64 Procrustes frame fit with a held-out split —
+  is now a standing instrument on the distillation line, where it recovers alignment that
+  absolute gauges otherwise read as a floor. Procrustes turned out to be considerably
+  more capable than this prototype suggested.
+- **Alignment to a reference member, with per-expert projectors.** Bertenstein's
+  soup-style construction is the direct ancestor of `geolip-captionbert-8192`'s cached
+  consensus, which aligns heterogeneous frozen encoders to a reference member rather than
+  to a running mean — the property that lets it fuse experts of differing dimension.
+- **The move toward distillation.** What this prototype did *not* produce was an
+  invariant object stable enough to distill against. That gap is what redirected the line
+  toward consensus targets and student distillation
+  (`geolip-vit-captionbank-coco`, `clip-vitb-mini-distilled`), where the target is cached
+  and model-free at training time.
+- **Evaluation discipline.** Three checks now run by default upstream of any retrieval
+  number: verify the query set actually varies; never score an alignment map on the rows
+  it was fit to; report effective rank beside CV.
+
+## Possible return
+
+The idea is not closed. What it would need:
+
+1. **Independent encoding** — separate passes, or an attention mask forbidding
+   cross-modal attention during the contrastive computation, so the readouts are
+   functions of their own modality.
+2. **A held-out fit for the alignment maps**, and a smaller/regularized map: ~524k free
+   parameters from 5,000 samples is far past where in-sample cosine means anything.
+3. **Data that can actually test transitivity** — items present in two or more *non-text*
+   modalities (video gives audio + frames of the same event; a protein with both sequence
+   and structure encoders). The four corpora here are disjoint, so the central claim was
+   never evaluable on its own data.
+4. **A caption-validity check at ingestion** — the `image_coco_test` failure was a
+   validity flag derived from the image alone, letting empty and placeholder captions
+   through.
+
+## Files
 
 ```
 geolip-bertenstein/
 ├── checkpoints/
-│   ├── epoch_001/
-│   │   ├── model.safetensors
-│   │   ├── loss.safetensors
-│   │   ├── training_state.pt
-│   │   └── config.json
-│   ├── epoch_002/
-│   ├── epoch_003/
-│   └── final/
-│       ├── model.safetensors
-│       ├── loss.safetensors
-│       ├── training_state.pt
-│       ├── config.json
-│       ├── aligner_audio.safetensors
-│       ├── aligner_code.safetensors
-│       ├── aligner_image.safetensors
-│       └── aligner_protein.safetensors
+│   ├── epoch_001/ · epoch_002/ · epoch_003/
+│   └── final/          model.safetensors · loss.safetensors · training_state.pt
+│                       config.json · aligner_{audio,code,image,protein}.safetensors
 ├── tensorboard/
-├── bertenstein_results.json
-├── stage2_bertenstein.py
+├── bertenstein_results.json      # original metrics — see Corrected results above
+├── cell1_prepare_data.py         # Stage 1: encode + cache
+├── cell2_prototype_model_trainer_v1.py   # Stage 2: align, train, evaluate
 └── README.md
 ```
 
-## Usage
-
-```python
-from safetensors.torch import load_file
-import torch
-
-# Load model weights
-state = load_file("checkpoints/final/model.safetensors")
-
-# Reconstruct model (see stage2_bertenstein.py for full class definitions)
-from stage2_bertenstein import BertensteinFusion, ExpertModule, ProcrustesAligner, FusionConfig
-
-# ... build model, load state_dict, run inference
-```
-
-Precomputed embedding caches (Arrow format) for all modalities: [`AbstractPhil/geolip-bertenstein-cache`](https://huggingface.co/datasets/AbstractPhil/geolip-bertenstein-cache)
+`checkpoints/final/model.safetensors` contains 102 tensors; the Procrustes aligner
+buffers ship inside it as `experts.<name>.aligner.*` as well as separately.
+Class definitions (`BertensteinFusion`, `ExpertModule`, `ProcrustesAligner`,
+`FusionConfig`) live in **`cell2_prototype_model_trainer_v1.py`** — earlier revisions of
+this card referenced a `stage2_bertenstein.py`, which was never part of this repo.
 
-## Geometric Terrain Analysis
+## Related
 
-The foundational profiling of 17 models and Procrustes alignment analysis: [`AbstractPhil/procrustes-analysis`](https://huggingface.co/AbstractPhil/procrustes-analysis)
+- [`AbstractPhil/procrustes-analysis`](https://huggingface.co/AbstractPhil/procrustes-analysis) — the weight-space profiling this grew out of
+- [`AbstractPhil/geolip-captionbert-8192`](https://huggingface.co/AbstractPhil/geolip-captionbert-8192) — reference-member consensus, the successor construction
+- [`AbstractPhil/clip-vitb-mini-distilled`](https://huggingface.co/AbstractPhil/clip-vitb-mini-distilled) — where the distillation direction landed
 
 ## Citation
 
 ```bibtex
 @misc{abstractphil2026bertenstein,
-  title={GEOLIP-Bertenstein: Cross-Modal Alignment Through Shared Pentachoron Geometry},
+  title={GEOLIP-Bertenstein: A Multimodal Procrustes Alignment Prototype},
   author={AbstractPhil},
   year={2026},
+  note={Archived research precursor; evaluation protocol corrected 2026-07-31},
   url={https://huggingface.co/AbstractPhil/geolip-bertenstein}
 }
 ```