Classifier Policy (#6898)

## Overview

Adds **AI document classification** and a **classification-aware Files
sidebar**: uploaded documents are automatically tagged with
document-type labels (Invoice, Contract, Lab report, …), and the sidebar
groups files under editable parent categories so a large library stays
navigable.

> [!IMPORTANT]
> **This feature only runs in the SaaS build.** Classification depends
on the AI engine and team-scoped label storage, so it's gated to SaaS
end-to-end:
> - The sidebar grouping is a `saas/`-layer override of the
`fileSidebarGrouping` seam; every other build (OSS core, self-hosted
proprietary, desktop) gets the null stub and renders the **unchanged
flat, recency-sorted list** — no categories, no "Other", no picker.
> - The classify/labels backend endpoints are gated on
`policies.enabled` (on in SaaS) and live in `app/proprietary`, so
they're absent from pure OSS and dormant in self-hosted unless
explicitly enabled.
> - The Python classifier is reached only via that gated path.
>
> Shared-layer changes that do compile everywhere are inert without the
engine (dormant schema/field additions) or intentional (`GetInfoOnPDF`
surfacing custom metadata).

## What it does

- **Classifier (engine):** reads the first/last two pages of a PDF and
assigns document-type labels from an allowed vocabulary. Labels are
deliberately document-*type* descriptors — no deep-content/PII
detection, since only a page window is read.
- **Team label vocabulary:** ~270 built-in defaults across ~15 families,
seeded per team. Editable by team leaders/admins in the Classification
policy settings (import/export/reset). Team-scoped and shared;
**per-user personal labels are intentionally out of scope** — the
vocabulary is team-level only.
- **Sidebar categories:** files group under parent categories
(Financial, Legal, Medical, …), busiest-first, collapsible, with a
"Recent" group on top and an "Other" group for anything uncategorised.
The category structure (names, icons, membership, custom categories) is
**device-local and user-editable** via a "Customize" picker — the only
per-user personalization; it never changes the team's label vocabulary.
- Classification results are written to PDF metadata
(`StirlingPDFClassification`), read back to keep files in their groups
without re-parsing.

## Architecture

Spans all three layers, mirroring the existing policy/source subsystem
conventions:
- **`frontend/editor`** — sidebar grouping seam + SaaS override,
category manager, labels editor, icon palette, file grouping, tests,
`en-US` i18n.
- **`app/proprietary` + `app/common` + `app/core`** —
`ClassifyLabelController`, team-scoped `ClassificationLabelStore` (Jpa +
in-process impls, same shape as `PolicyStore`/`SourceStore`), metadata
read/write.
- **`engine`** — the document-classifier agent, contracts, routes,
tests.

## Screenshots

**Files sidebar — grouped by category (SaaS)**

### Loading view

<img width="2056" height="1046" alt="Screenshot 2026-07-07 at 5 12
56 PM"
src="https://github.com/user-attachments/assets/1d712da5-50ae-4349-b0cd-e62665c3ec0c"
/>

### Organized in the sidebar

<img width="2056" height="1045" alt="Screenshot 2026-07-07 at 5 14
05 PM"
src="https://github.com/user-attachments/assets/3ea4fe21-da51-4cea-bc3a-18ce040d3d05"
/>

**Customize categories picker**
### Personal settings to change how labels are grouped in an individual
users editor

<img width="2056" height="1044" alt="Screenshot 2026-07-07 at 5 52
42 PM"
src="https://github.com/user-attachments/assets/40be03ce-0f63-4d1e-b58b-cec045d01cb2"
/>

**Classification labels editor (team settings)**

<img width="2056" height="1042" alt="Screenshot 2026-07-07 at 5 53
00 PM"
src="https://github.com/user-attachments/assets/337b0739-15c9-4749-9c6b-22e3b20825b8"
/>

## Testing

- Frontend `task frontend:check` — green (editor + portal tests,
typecheck across all flavors, lint, label-drift guard).
- Backend `task backend:check` (proprietary) and `:saas:test` — green.
- Engine `task engine:check` — green.
This commit is contained in:
EthanHealy01
2026-07-09 11:47:37 +00:00
committed by GitHub
parent f29500c138
commit c64369e56c
121 changed files with 10018 additions and 506 deletions
@@ -0,0 +1,86 @@
// Default classification labels. The SOURCE OF TRUTH is the co-located static
// JSON (`classificationLabels.json`), imported here and shaped into typed
// objects — edit THAT file, not this one. This is the ONLY copy of the label
// data: it seeds a team's editable set and drives the sidebar's grouping,
// icons, and display names. Neither the backend nor the engine keeps a copy.
//
// NOTE: this is only the built-in default vocabulary. A team's own
// (admin-editable) labels live in the backend store and are what the backend
// sends to the engine per classify request — the engine holds no vocabulary of
// its own. Edits to a team's set reach Python on the next run, independent of
// this file.
//
// `labels` is the flat set: each has a stable `id` (slug — the value on the wire,
// in storage and keyed on) and a human `name` (display, translatable via
// `classification.labels.<id>`); `icon` is presentational only, the engine never
// sees it. `families` are presentational sidebar roll-ups (referencing labels by
// id) the classifier never sees.
import labelsData from "@app/data/classificationLabels.json";
export interface ClassificationLabel {
/** Stable identity (slug) — the value on the wire, stored on the doc, and
* keyed on. Independent of the (translatable) display name. */
id: string;
/** Human display name; the en-US default for `classification.labels.<id>`. */
name: string;
/** Material Symbols icon key (see `labelIcons.ts`). */
icon?: string;
}
export interface LabelFamily {
/** Stable identity for sidebar prefs — never rename once shipped. */
id: string;
/** Group header text shown in the sidebar and the group picker. */
name: string;
/** Material Symbols icon key (see `labelIcons.ts`). */
icon: string;
/** The built-in labels this family rolls up in the sidebar. */
labels: ClassificationLabel[];
}
/** Shape of `classificationLabels.json` — the flat label set plus the
* presentational family grouping (which references labels by id). */
interface LabelsFile {
labels: ClassificationLabel[];
families: { id: string; name: string; icon: string; labelIds: string[] }[];
}
const data = labelsData as LabelsFile;
/** Flat default label set — file order, as the classifier/team-seed sees it. */
export const DEFAULT_CLASSIFICATION_LABELS: ClassificationLabel[] = data.labels;
const LABEL_BY_ID = new Map(data.labels.map((label) => [label.id, label]));
/**
* Built-in families with their labels resolved from the flat set by id. Throws
* at module load if a family references an unknown id, so a bad hand-edit of the
* JSON fails fast rather than silently dropping a label from its group.
*/
export const LABEL_FAMILIES: LabelFamily[] = data.families.map((family) => ({
id: family.id,
name: family.name,
icon: family.icon,
labels: family.labelIds.map((id) => {
const label = LABEL_BY_ID.get(id);
if (!label) {
throw new Error(
`Classification family "${family.id}" references unknown label id "${id}"`,
);
}
return label;
}),
}));
/**
* Stable slug id from a label's canonical (English) name — used to derive an id
* for a NEW custom label the user types. Built-in ids are authored in the JSON;
* this must stay in sync with the slug rule used to generate them.
*/
export function labelId(name: string): string {
return name
.toLowerCase()
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "");
}