A Zigbee2MQTT networkmap on a 200+ device mesh takes minutes to build.
Two separate failures fell out of that:
- POST /zigbee/import held the HTTP request open for the whole MQTT
round-trip, so any reverse proxy in front of the API cut it first
(Cloudflare returns a 524 at 120 s) and the browser never saw the map.
It now registers a job, fetches in the background and answers 202; the
client polls GET /zigbee/import/{job_id} until the payload is ready.
Job results are transient and live in memory with a 15 min TTL — the
same single-worker assumption the scheduler already makes. A failed
fetch replays the status the synchronous route used to raise, so a bad
broker is still a 502 and a slow mesh still a 504.
- The networkmap wait was hard-coded at 300 s with no way to raise it.
It now reads ZIGBEE_NETWORKMAP_TIMEOUT, and the shared MQTT round-trip
used by the Z-Wave import reads MQTT_RESPONSE_TIMEOUT. Both default to
300 s, fall back to that if misconfigured to a non-positive value, and
name themselves in the timeout message.
Also corrects the route and doc claims that the wait was 60 s.
The /import tests changed with the contract they cover, not to pass.
Fixes#380
ha-relevant: yes
Completes the split: `nodes` now holds only how a device is drawn on one
canvas, and every device fact reaches the API from the inventory row.
- The device columns are removed from `nodes` (SQLite table rebuild, the
same shape as the existing device_inventory and canvas_state rebuilds).
The drop is skipped, and logged, while any non-furniture node is still
unlinked: those columns are the last copy of that node's facts. The
backfill therefore reads them with raw SQL — by the time it runs, the
model no longer declares them.
- The status checker iterates devices, not nodes: one check per device
however many canvases draw it, writing `status_live` / `last_seen` /
`response_time_ms` on the row. `/ws/status` messages carry `device_id`
and the node ids they light up. Hidden devices are not probed.
- Readers repointed: scanner (last_scan lands on the row), proxmox (its
node tier collapses into the inventory tier, keeping only the cluster
handles), zigbee/zwave (one property refresh serves every canvas), rack
inventory, liveview, stats, node dedupe.
- `POST /scan/pending` merges into the row that already describes the host
instead of minting a second one — one device is one row, whichever way it
was documented.
- Standalone keeps parity: the canvas blob gains `devices`, split on save
and hydrated on load. A blob written before the split still reads.
The rack inventory had a related bug: a mount that names a node explicitly
printed the mount's device rather than the pinned node's. It now reads the
node's own row.
Tests that built a node with device columns are ported to the link; where a
behaviour genuinely moved (properties refresh once on the row, last_scan is
the device's) the assertion moved with it rather than being dropped.
ha-relevant: yes
The rack's link picker read `/nodes`, so it only ever offered devices
someone had already approved onto a logical canvas — two rows on a homelab
holding 74 inventory entries. A device on no canvas is still the record of
a real box, and is exactly what a rack is built out of.
`DevicePickerModal` replaces `NodePickerModal` and lists the Device
Inventory itself. Picking an entry calls the new `relinkDevice`, which
repoints the mount's `deviceId`, adopts that entry's node, status and —
unless the user renamed the plate — its label. One entry, one mount: a row
another plate stands for is not offered, and the store refuses it anyway.
The placeholder a rack-created plate left behind is dropped through the new
`DELETE /api/v1/scan/pending/{id}`, which refuses a device a rack still
mounts (409): foreign keys are off at runtime, so the mount would be left
naming a row that no longer exists.
`LinkedDevicePanel` becomes "Linked device" and now prints what discovery
found even when nothing on a canvas answers for the device; only the
canvas-side rows go missing, under a "Not on a logical canvas." note.
Also renames `pending_devices` to `device_inventory` (and
`pending_device_links` to `device_inventory_links`), with the Python and
TypeScript names that followed it. "Pending devices" was the scanner's word
for a queue of finds awaiting approval; the rows outlive approval, are
edited by hand and are what a rack mounts. Routes, payload keys and MCP
tool names are a published contract and are unchanged — `/scan/pending/*`
and the `pending_devices` key in `/stats` stay as they are.
The rename migration runs before `create_all`, or an empty new table would
be created beside the populated old one and every scanned device would read
as gone; it repairs that state too, for anyone whose app already started
mid-upgrade. Foreign keys are switched on for the rename so SQLite rewrites
the `REFERENCES` clause in `rack_devices`.
ha-relevant: maybe
Mirror the Proxmox auto-sync pattern for the Zigbee2MQTT and Z-Wave JS
UI mesh imports. Connection config + MQTT credentials live in .env only,
are never persisted to scan_config.json, and are never returned by any
API or shown in the UI (single source of truth).
- config: ZIGBEE_* / ZWAVE_* env settings; only sync_enabled+interval
are persisted, connection/credentials stay env-only
- routes: GET/POST /config, POST /sync-now; auto-sync reuses the exact
manual _background_*_import + _persist_pending_import path (fresh
import when empty, update-in-place when nodes exist, ScanRun trace)
- scheduler: zigbee_sync / zwave_sync jobs with live enable + reschedule
- frontend: reusable MeshAutoSync section in Settings (Zigbee, Z-Wave)
- .env.example: documented both blocks
- tests: scheduler jobs, router config/sync-now/auth, credential-never-
persisted, SettingsModal sections
Manual Zigbee/Z-Wave import behaviour is unchanged.
ha-relevant: no
Replace 9 duplicated per-file 'headers' fixtures and the awkward
'auth_headers' coroutine factory with one canonical 'headers' fixture
in conftest.py. Migrate liveview/media tests off the await-based
factory. No behavior change; 627 passed.
ha-relevant: no
A device already on a canvas (Node exists) but with no pending_devices row
never showed in the discovery inventory — the inventory lists pending_devices,
not nodes. This stranded legacy auto-placed coordinators: on a canvas yet
invisible in the inventory list.
Both mesh imports now, in the already-approved-node branch, ensure an
inventory row exists: create one as status="approved" when missing, or
refresh its metadata (preserving status) when present. New/unplaced devices
still land as status="pending"; hidden rows stay hidden.
Tests updated: approved-node path now backfills an approved inventory row
(zigbee + zwave), plus a regression that a hidden inventory row is not
revived.
ha-relevant: yes
The coordinator was special-cased in both mesh imports to auto-create a
canvas Node, so it never appeared in the pending inventory and users could
not approve/hide/type it like every other device.
Remove the auto-placement: the coordinator now flows through the shared
pending path (upsert into pending_devices with suggested_type
zigbee_coordinator / zwave_coordinator). An already-approved coordinator
Node still gets its properties refreshed on re-import via the shared
approved-node path. The response's coordinator/coordinator_already_existed
fields are retained (now always unset) for backward-compatible shape; the
frontend already ignored them.
Tests updated: coordinator lands in pending (counts include it), no Node
auto-created, pending metadata carried, approved coordinator refresh + no
pending re-list.
ha-relevant: yes
Zigbee and Z-Wave imports looked up canvas nodes by ieee_address with
scalar_one_or_none(), assuming one node per IEEE globally. A device placed
on two designs (one Node per canvas — a supported feature) made re-import
crash with MultipleResultsFound.
- Both mesh imports now refresh properties on every matching node instead
of a single row (loop over .scalars().all()).
- approve_device guards against a true duplicate: same IEEE already on the
SAME design reuses that node instead of inserting a second one.
- New node_dedupe service: loss-free repair keyed on (ieee, design_id).
Collapses only genuine same-canvas duplicates (merges properties/services/
missing fields, re-points edges + parent_id, drops self-loops/parallel
edges). Cross-design placements preserved. Runs at start of both imports
and bulk-approve. No-op on healthy DBs.
- IP correlation path already handled multiple nodes; left unchanged.
Tests: dedupe unit tests (collapse, cross-design preservation, edge/parent
re-point, idempotent), zigbee + zwave multi-canvas regression, approve
no-dupe guard.
ha-relevant: yes
When a Zigbee device was approved (PendingDevice.status=approved + Node
created) and the user later deleted the canvas Node, the PendingDevice row
was left orphaned as approved. On re-import _persist_pending_import found
the existing row, refreshed its fields but left status=approved, so it
never reappeared in the Pending list (which filters status==pending) —
the import reported devices found but Pending stayed empty.
Reset such orphaned approved rows back to pending on re-import. Hidden
devices stay hidden (explicit user intent). Adds regression tests for
both revive-approved and keep-hidden paths.
ha-relevant: yes
- New zigbee props (approve + first-time re-import) ship with visible=false
so the canvas card stays clean. User opts in from the right panel.
- On re-import of an already-approved node, merge instead of overwrite:
keys already present keep their visible flag (and any user-edited value
is replaced with the freshly imported one), brand-new keys are appended
hidden. Non-zigbee custom properties are preserved untouched.
- Approval (single + bulk) of zigbee pending devices now writes IEEE, Vendor,
Model, LQI into Node.properties so they show in the right panel.
- Zigbee re-import refreshes properties on existing canvas Nodes and skips
creating a pending row when the device was already approved — keeps
approved devices out of pending/hidden modals.
- Coordinator Node also receives the same properties on first creation and
on re-import.
- Remove the Parent Container selector from the add/edit node modal.
Mirrors IP scan flow: POST /zigbee/import-pending now creates a
ScanRun(kind=zigbee, status=running) and returns immediately.
Networkmap fetch + pending upsert run in the background, status
transitions to done/error when finished.
Frontend: import modal closes on submit, scan history shows the
run with a ZIG/IP kind chip and toasts on completion. Pending
modal auto-refreshes when run finishes.
scan_runs.kind column added (default 'ip', idempotent migration).
Existing zigbee tests refactored to exercise _persist_pending_import
directly (background tasks don't see the test session); route test
verifies the run is created with kind=zigbee.
Coordinator auto-approves to a canvas Node; routers/end devices land in
pending_devices keyed by IEEE. Discovered parent->child edges are stored
in pending_device_links so that approving a pending device later
auto-creates the Edge once both endpoints exist as canvas Nodes.
- new POST /api/v1/zigbee/import-pending (default mode in modal)
- new pending_device_links table; ieee_address on nodes + pending_devices
- pending_devices.ip migrated to nullable (table rebuild on existing DBs)
- approve / bulk-approve return auto-created edges; sidebar pushes them
into the canvas store with bottom -> top-t handles
- ZigbeeImportModal: radio toggle pending vs canvas; reset on close
- PendingDeviceModal: zigbee badge, IEEE/LQI/vendor/model rows, services
hidden for zigbee
- Sidebar pending row: ZIG source badge, LQI badge, friendly_name fallback
- SearchBar: null-safe IP, also searches friendly_name and ieee_address
- Tooltip trigger uses asChild to avoid nested-button hydration error
- Schema: mqtt_tls + mqtt_tls_insecure flags on import + test-connection
requests; model_validator enforces insecure requires tls
- Service: _build_tls_context() using ssl.create_default_context();
logger.warning when verification disabled; tls_context plumbed into
aiomqtt.Client for both fetch_networkmap and test_mqtt_connection
- Route: passes tls flags through to service
- Frontend: TLS checkbox auto-toggles port 1883<->8883 unless user
edited; insecure checkbox disabled until TLS on, red-tinted; password
field marked autocomplete=new-password
- Tests: 7 new backend tests (TLS context build, client kwargs assertion,
router happy path, insecure-without-tls 422)
Pouzor