MSE Graph Language Model:
Deterministic, Explainable,
Zero-Weight Inference

1. Introduction

Most language models are built around the same idea: train a neural network on enormous amounts of text, let it adjust billions of floating-point weights until it learns to predict the next word reasonably well, and then sample from a probability distribution at inference time. The model is powerful, but it is also a black box — you cannot point to the weight that caused a particular word to be chosen, and two runs with the same input can produce different output.

The MSE Graph Language Model (MSE-GLM) takes a different approach entirely. Language is represented as a directed graph: tokens are nodes, observed transitions are edges, and inference is graph traversal under a small set of explicit, inspectable rules. There are no learned weights, no gradients, no probability sampling — and because of that, every generation decision can be traced back to the exact rule and candidate set that produced it.

2. Design Philosophy

The core bet is that language, in many constrained domains, does not need to be modeled probabilistically. If the valid output space is a finite set of token transitions — all valid SQL clauses, all valid JSON keys for a schema, all valid assembly mnemonics — then a graph that memorizes exactly those transitions can generate correctly constrained output with zero chance of emitting something it never observed, and zero need for a GPU.

Where the graph is genuinely ambiguous — two equally plausible next tokens given the same context — the architecture resolves that ambiguity using principled, inspectable rules rather than a probability sample. That is the core engineering problem this system solves, and the three-matrix design described below is how it does it.

3. Architecture Overview

Training is a single O(N) pass over the corpus — no backpropagation, no epochs, no GPU. The trained model persists to a self-contained folder of JSON files (vocabulary, edges, bridges, relationships, metadata) that can be loaded and queried on any machine with Python.

4. Tokenizer

The tokenizer is a from-scratch Byte Pair Encoding (BPE) implementation — the same approach used by GPT-2, but written from the ground up with no external dependencies. It converts raw text into integer token IDs through iterative character-pair merging.

Four reserved special tokens anchor the system:

Sentence boundaries (on . ! ? \n) are preserved during training so the graph learns where sequences legally end. Streaming training from a file is supported, so corpus size is not bounded by available RAM.

5. The Three Matrices

The trained model is three compact, array-backed, CSR-indexed structures. All storage uses Python's array.array('i') — 4 bytes per integer, roughly 7× smaller than equivalent Python lists.

Edge Matrix (E)

A deduplicated list of every adjacent token pair observed in the corpus, sorted by source token and indexed for O(1) successor lookup. This is the bigram graph — it answers "what tokens have I seen follow token X?"

Bridge Matrix (B)

Extends E to three-token context: every observed (source, bridge, target) triple is stored once, giving the model trigram-equivalent context with no attention mechanism. The key structural innovation is a fourth column, cluster_id, described in the next section.

Relationship Matrix (R)

The most novel structure. It stores only two columns:

R contains no copy of the triple's own content. It is a pure many-to-many edge list: a triple shared across multiple training sentences carries one R row per sentence. This is what enables lineage-aware tie-breaking at inference time (see §7) and O(1) batch audit of "every triple belonging to sentence N."

# Example R matrix — corpus: "the cat sat on the mat." + "the dog sat on the carpet."
triple_id  |  relationship_id
-----------+-----------------
    1      |       1           ← triple (the→sat, bridge=cat) appeared in sentence 1
    2      |       1
    3      |       1           ← triple (sat→the, bridge=on) — shared
    4      |       1
    5      |       2
    6      |       2
    3      |       2           ← same triple_id 3, now also in sentence 2
    7      |       2

6. Distributional Clustering

The cluster_id column in B is a weights-free, symbolic analogue of the distributional hypothesis underlying word2vec and other embedding models: tokens that are interchangeable in the same structural slot are functionally related, without any claim of shared meaning.

Clustering is assigned by a dual-axis rule:

• Bridge axis — triples sharing the same (source, target) pair get a shared cluster_id. This groups tokens interchangeable as the middle token (the "bridge").
• Target axis — triples sharing the same (source, bridge) pair, not already grouped on the bridge axis, get a shared cluster_id. This groups tokens interchangeable as the destination.
• cluster_id = 0 — a triple that matches neither axis with any other triple is left unclustered.

A derived T_index maps each token to the non-zero cluster IDs it participates in. Token relatedness is then defined as |T_index[a] ∩ T_index[b]| — a set-intersection analogue of cosine similarity that needs no embedding space.

infer_shared_role()

This enables a new type of query: give the model an unordered set of tokens and ask what they have in common structurally:

model.infer_shared_role(["cat", "dog"])
# → [('sat', 'bridge_axis', {'cluster_id': 1, 'overlap': 2}), ...]
# cat and dog both filled the bridge slot in (the → ___ → sat)

7. Inference Pipeline

At every generation step the engine knows the current token, the previous token (if available), and a running set active_rels — the relationship IDs the path has been consistent with so far. It resolves the next token through four ordered stages:

The lineage-narrowing rule (critical detail)

When a Stage 1 step is resolved, active_rels is updated by intersecting it with the chosen triple's lineage — not replacing it. This is subtle but essential: a triple shared across several training sentences (e.g. a common clause like sat on the) must not erase the more specific lineage a prior step already established.

# BUG (old): replace active_rels entirely
active_rels = new_triple_rels  # ← wipes specificity at shared triples

# FIX: narrow by intersection
narrowed = active_rels & new_triple_rels
active_rels = narrowed if narrowed else new_triple_rels  # reset only on divergence

Without this fix, the dog would generate the dog sat on the mat instead of the correct the dog sat on the carpet, because the shared sat → on → the triple would widen active_rels back out to all lineages just before the branch point. This regression is covered by an automated test in the suite.

8. Explainability

Every generation step is fully traceable. The explain_step() method and /explain chat command return the stage, rule, candidate set, and active_rels for any given step:

you> /explain the | dog
model> next='sat'  stage=1  rule=storage_order_fallback  active_rels={1}

you> /explain sat | on
model> next='the'  stage=1  rule=single_match  active_rels={1}

you> /explain on | the
model> next='carpet'  stage=1  rule=lineage_match  active_rels={1}

The analyse.py CLI provides full step-by-step traces from the command line:

python3 analyse.py --model runs/demo trace "the dog" --max-tokens 12

The output names the chosen token, stage, rule, and active lineages at every step — making the model fully auditable without any post-hoc interpretation.

9. MSE-GLM vs. Transformers

The two architectures optimize for different things. Transformers win on fluency, generalization, and long-range reasoning. MSE-GLM wins on guarantees, auditability, and resource efficiency. They are not competitors — they are tools for different problems.

10. Use Cases

• Grammar-constrained generation — SQL, JSON, config files, shell commands: any domain where the valid output space is closed and can be observed from a corpus.
• LLM guardrails — attach MSE-GLM as a structural validator on top of a transformer's output to catch illegal transitions before they reach users.
• Embedded / edge AI — no GPU, no framework, pure Python + stdlib. Deploy on a Raspberry Pi or a microcontroller-class device.
• Compliance-sensitive tooling — legal, finance, healthcare contexts where every output decision must be auditable by a human reviewer.
• Autocomplete engines — deterministic, fast, constrained to observed patterns.
• Distributional similarity without embeddings — infer_shared_role() identifies which tokens are interchangeable in a given structural slot, with no embedding model required.

11. Track Record

12. Download & Run

The full implementation — tokenizer, graphs, inference engine, training CLI, chat REPL, analysis CLI, and test suite — is on GitHub. No pip installs required beyond the Python standard library.

git clone https://github.com/fodokidza/mse_glm.git
cd mse-glm

# train a model
python3 train.py --text "your corpus here." --out runs/model --vocab-size 500

# chat with it
python3 chat.py --model runs/model

# analyse it
python3 analyse.py --model runs/model report

# run the tests
python3 test.py

© 2026 Clifford Chivhanga. Source code licensed under AGPL-3.0. Commercial licensing available — contact the author.