The node-side log writers now hand their rows to Core\Cluster\LogSink by
type, instead of each building its own INSERT: client request logs
(lines_logs), stream logs (streams_logs), stream errors (streams_errors),
panel errors (panel_logs, INSERT IGNORE) and restream detections
(detect_restream_logs). The legacy backend writes the same multi-row
INSERTs, chunked at 1000 rows, with bound values instead of hand-escaped
SQL (streams_logs used addslashes). A useSink() hook is the seam the
cluster API's log.* events plug into (spec Phase 0/5).
Core\Cluster\Redactor strips credentials before anything is journaled:
password/token/username values, the /user/pass/ segments of Xtream URLs,
and user:pass@ in URLs. The API backend will use it; the SQL backend
writes what it wrote before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Es8UhyPS8MNjuYTagqfGWi
Review follow-up to the log-import fix (71108b20):
- Record last activity with one UPDATE per batch instead of an
INSERT ... ON DUPLICATE KEY UPDATE, so activity for a deleted line no
longer recreates it as an empty `lines` row. Each line is set once,
from its latest row, in id order so nodes lock rows alike, and
`updated` is kept so cache_engine does not rebuild the line.
- Flush a batch at 4 MiB of spool as well as at 1000 rows, so oversized
client rows cannot push an INSERT past max_allowed_packet.
- Skip a claimed spool that cannot be opened instead of failing on fgets.
- Document that a claim left by a crashed run is re-imported from its
start (at-least-once).
- Tests: writer appending mid-import, missing keys, the server_id check,
last_activity_array contents, duplicate lines in a batch, byte-split
batches, the lines_logs failed-INSERT path; reset the injected DB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
ActivityCronJob and LinesLogsCronJob drain a base64(JSON)-per-line spool
(LOGS_TMP_PATH/activity, client_request.log) into MySQL once a minute on
MAIN and on every LB. Both parsers broke out of the read loop after the
first row, and ActivityCronJob also stopped at the first invalid row
instead of skipping it. The caller then deleted the whole spool, so each
node kept at most one lines_activity row and one lines_logs row per
minute. The rest were lost, and lines.last_activity was updated for one
closed connection only. In LinesLogsCronJob a line that did not decode
to an array raised a TypeError in array_map() before the unlink, so a
bad line wedged the spool.
Read every line, skip empty, undecodable and (for activity) incomplete
rows, and insert in batches of 1000 so a busy minute stays well under
max_allowed_packet. Each activity batch upserts `lines` with its own
consecutive ids from last_insert_id(). Values come from a fixed key list
with a '' default, so a record with a missing key cannot break the SQL
for its whole batch. The SQL statements and column lists are unchanged.
The spool is now claimed by renaming it to <spool>.import before it is
read. The writers append by path, so rows written during the import go
to a fresh spool and are not deleted with the claimed file. A claimed
file left by a crashed run is imported first. Rows are still dropped
when their INSERT fails; loss-free delivery is planned for Phase 5.
StreamsLogsCronJob and ErrorsCronJob already read every line and are
unchanged. ErrorsCronJob keeps error_log.log when every line is
filtered or its INSERT fails, and StreamsLogsCronJob unlinks even when
its INSERT fails. Both are left for a separate change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TQHeypQpfAXTdCysmi7b6a
Claude