Multiple issues between README claims and codebase
milla-jovovich/mempalace
I've been doing reviews of agentic memory systems and figured I'd flag this since no other system in my survey has had this pattern before where the README claims do not match what's in the code to such a degree.
| README claim | What the code actually does | Severity |
|---|---|---|
| **"Contradiction detection"** — automatically flags inconsistencies against the knowledge graph | `knowledge_graph.py` has **no contradiction detection**. The only dedup is blocking identical open triples (same subject/predicate/object where `valid_to IS NULL`). Conflicting facts (e.g., two different `married_to` values) accumulate silently. | **Feature does not exist** |
| **"30x compression, zero information loss"** — AAAK described as "lossless shorthand" | AAAK is lossy abbreviation: regex entity codes + keyword frequency + 55-char sentence truncation. `decode()` is string splitting — no original text reconstruction. Token counting uses `len(text)//3` heuristic. **LongMemEval drops from 96.6% to 84.2% in AAAK mode** — a 12.4pp quality loss. | **Claim is false** |
| **96.6% LongMemEval R@5** (headline, positioned as MemPalace's score) | Real score, but measured in "raw mode" — uncompressed verbatim text stored in ChromaDB, standard nearest-neighbor retrieval. **The palace structure (wings/rooms/halls) is not involved.** This measures ChromaDB's default embedding model performance, not MemPalace. | **Misleading attribution** |
| **"+34% retrieval boost from palace structure"** | Narrowing se...
View on GitHub ↗
SaaS Metrics