ENCSRequest code access

System

Knowledge base

A single-tenant, permission-scoped knowledge base for one company: it ingests the company's documents and answers questions grounded strictly in the sources each asker is allowed to see. It is advisory, not autonomous — it answers, it never acts.

The problem

A naive “chat over our documents” gets the one thing that matters wrong: access. Some documents are for the whole company, some are personal. A model that sees everything leaks; a model told to “be careful” refuses the people who are entitled, unpredictably. So the entitlement has to be enforced before the model sees anything, and the model must not re-decide it.

How it works

  1. Access in SQL. One visibility rule — shared with the whole company, or personal and visible to its owner and to the users and roles its ACL names. No bypass, not even for the company owner. Every query that reads chunks applies it: vector search, word search, word statistics and the typo vocabulary. → Decision 0001
  2. Hybrid retrieval. Vector similarity (a local multilingual embedding model; documents never leave the server) fused with a word search by weighted Reciprocal Rank Fusion. Words are matched by lemma + stem keys computed in Python and stored in the database; an accent-free prefix is only a half-weight fallback. → Decision 0006
  3. Evidence or nothing. A chunk counts only with a close vector, or a strong match of rare question words covering most of the idf-weighted question. A typo is corrected against the asker's own vocabulary first. Without evidence the answer is “not in the knowledge base” and the model is not called. → Why vector search alone failed us
  4. Cited sources only. The model cites document titles in brackets; only the cited documents are shown as sources, one per document, with an excerpt. → Decision 0003

Also in the code

  • Members are people or AI agents — both pass the same permission filter.
  • Secrets stay on the server. A token for a linked service is stored encrypted (Fernet, key derived with HKDF from a dedicated key) and used only by a server-side proxy with an allow-list of paths and a body-size cap; the browser never sees it.
  • Single-tenant by design: one company, one knowledge base, one dedicated Postgres + pgvector.

How it is tested

  • Unit tests without a database: word keys (“must find” / “must not find”), hybrid fusion, sources, access, the proxy.
  • Integration tests against a real Postgres + pgvector: the access rule through the real queries for members with different rights, search, migrations.
  • A forced-fail run undoes each fix in a copy and requires its tests to fail; the CI gate runs it first, and a missing database is a failure, not a skipped test. → Decision 0005

Stack

Python 3.11 · FastAPI · SQLAlchemy 2 (async) · Postgres + pgvector · fastembed / ONNX embeddings (multilingual-e5-large) · simplemma + Snowball · Alembic · a small vanilla HTML/JS UI.