mirror of
https://github.com/RayLabsHQ/gitea-mirror.git
synced 2026-10-05 06:51:53 +02:00
feat: incremental issue and pull request sync (#449)
After a complete pass, the issues and pull request passes list only the items GitHub reports as updated since the previous listing (minus a 10 minute margin), so per-issue comment listing and per-PR detail, commits and files calls run only for what changed. Pull requests use the issues endpoint for this since pulls.list has no since filter. The watermark lives in the repository metadata state as syncCursors, one per kind, and records when the listing started. It only moves when the pass finishes with no failed item. A full pass still runs when the last one is older than 7 days, when the destination has no mirrored issues or pull requests, and after reset metadata, which clears the state.
This commit is contained in:
@@ -19,6 +19,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- Raised the dependency floors for the advisories published on 2026-09-08: Astro 7.2.8 (remote code execution through AVIF image optimization, and an authorization bypass when stripping the configured base), @xmldom/xmldom 0.8.15 (eight parser and serializer issues), sharp 0.35.4 (libheif), svgo 4.1.0 (removeScripts sanitization) and js-yaml 4.3.2 (merge-key CPU use). Applied to both the application and the documentation site.
|
||||
|
||||
### Added
|
||||
- Incremental issue and pull request sync (#449)
|
||||
- After a complete pass, the issues and pull request passes ask GitHub only for items updated since the last one (with a 10 minute margin), so comments, pull request details, commits and files are fetched only for what changed instead of for every item on every sync
|
||||
- The watermark is stored per repository in the metadata state and only moves when a pass finishes with no failed item
|
||||
- A full pass still runs when the last one is older than 7 days, when the destination has no mirrored issues or pull requests, and after Reset metadata
|
||||
- Releases mirror from Gitea and Forgejo sources, Codeberg included (#440)
|
||||
- The release mirror lists releases through the repository's own source instead of always through the GitHub API, and keeps the same release limit, asset limit, per-destination lock, tag check and retention pruning
|
||||
- Release assets are downloaded with the credentials of the host they live on, so a Gitea source uses its own token and a public one needs none
|
||||
|
||||
@@ -9,8 +9,8 @@ mock.module("@/lib/helpers", () => ({
|
||||
}));
|
||||
|
||||
const mockMirrorGitHubReleasesToGitea = mock(() => Promise.resolve());
|
||||
const mockMirrorGitRepoIssuesToGitea = mock(() => Promise.resolve());
|
||||
const mockMirrorGitRepoPullRequestsToGitea = mock(() => Promise.resolve());
|
||||
const mockMirrorGitRepoIssuesToGitea = mock(() => Promise.resolve(undefined));
|
||||
const mockMirrorGitRepoPullRequestsToGitea = mock(() => Promise.resolve(undefined));
|
||||
const mockMirrorGitRepoLabelsToGitea = mock(() => Promise.resolve());
|
||||
const mockMirrorGitRepoMilestonesToGitea = mock(() => Promise.resolve());
|
||||
const mockSyncRepositoryMetadataToGitea = mock(() => Promise.resolve());
|
||||
|
||||
@@ -1090,13 +1090,15 @@ export async function syncGiteaRepoEnhanced({
|
||||
);
|
||||
} else {
|
||||
try {
|
||||
await dependencies.mirrorGitRepoIssuesToGitea({
|
||||
const issuesCursor = await dependencies.mirrorGitRepoIssuesToGitea({
|
||||
config,
|
||||
octokit,
|
||||
repository,
|
||||
giteaOwner: repoOwner,
|
||||
giteaRepoName: repoName,
|
||||
syncCursor: metadataState.syncCursors.issues,
|
||||
});
|
||||
if (issuesCursor) metadataState.syncCursors.issues = issuesCursor;
|
||||
metadataState.components.issues = true;
|
||||
metadataState.components.labels = true;
|
||||
metadataUpdated = true;
|
||||
@@ -1123,13 +1125,15 @@ export async function syncGiteaRepoEnhanced({
|
||||
);
|
||||
} else {
|
||||
try {
|
||||
await dependencies.mirrorGitRepoPullRequestsToGitea({
|
||||
const pullRequestsCursor = await dependencies.mirrorGitRepoPullRequestsToGitea({
|
||||
config,
|
||||
octokit,
|
||||
repository,
|
||||
giteaOwner: repoOwner,
|
||||
giteaRepoName: repoName,
|
||||
syncCursor: metadataState.syncCursors.pullRequests,
|
||||
});
|
||||
if (pullRequestsCursor) metadataState.syncCursors.pullRequests = pullRequestsCursor;
|
||||
metadataState.components.pullRequests = true;
|
||||
metadataUpdated = true;
|
||||
console.log(
|
||||
|
||||
@@ -0,0 +1,476 @@
|
||||
/**
|
||||
* Behavioural tests for incremental issue and pull request sync (#449).
|
||||
*
|
||||
* With a watermark from a completed pass, the GitHub listing is asked only
|
||||
* for what changed (`since`), and the per-issue comment listing and per-PR
|
||||
* detail, commits and files calls run only for the items that came back.
|
||||
* Without one, the pass is full and unchanged from before.
|
||||
*
|
||||
* The destination is a fake global fetch and GitHub is a fake Octokit. Other
|
||||
* test files replace @/lib/http-client and friends with mock.module, which is
|
||||
* process wide in bun and depends on file order, so the suites run in an
|
||||
* isolated child process (same harness as gitea-source-releases.test.ts).
|
||||
*/
|
||||
|
||||
import { afterEach, describe, expect, it, test } from "bun:test";
|
||||
import type { Config } from "@/types/config";
|
||||
import type { Repository } from "@/lib/db/schema";
|
||||
import type { MetadataSyncCursor } from "@/lib/metadata-state";
|
||||
|
||||
const CHILD_FLAG = "GM_INCREMENTAL_METADATA_ISOLATED";
|
||||
const isChild = !!process.env[CHILD_FLAG];
|
||||
|
||||
if (!isChild) {
|
||||
test("incremental metadata sync - isolated child suite", () => {
|
||||
const res = Bun.spawnSync({
|
||||
cmd: [process.execPath, "test", import.meta.path],
|
||||
env: { ...process.env, [CHILD_FLAG]: "1" },
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
});
|
||||
if (res.exitCode !== 0) {
|
||||
console.error(res.stdout.toString());
|
||||
console.error(res.stderr.toString());
|
||||
}
|
||||
expect(res.exitCode).toBe(0);
|
||||
}, 60_000);
|
||||
}
|
||||
|
||||
const { mirrorGitRepoIssuesToGitea, mirrorGitRepoPullRequestsToGitea } = isChild
|
||||
? await import("@/lib/gitea")
|
||||
: ({} as typeof import("@/lib/gitea"));
|
||||
const { INCREMENTAL_SYNC_SAFETY_MARGIN_MS } = isChild
|
||||
? await import("@/lib/metadata-state")
|
||||
: ({} as typeof import("@/lib/metadata-state"));
|
||||
|
||||
const GITEA_URL = "https://gitea.example.com";
|
||||
const DEST_OWNER = "mirror-owner";
|
||||
const DEST_REPO = "demo";
|
||||
const DEST_API = `${GITEA_URL}/api/v1/repos/${DEST_OWNER}/${DEST_REPO}`;
|
||||
|
||||
const realFetch = globalThis.fetch;
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = realFetch;
|
||||
});
|
||||
|
||||
const config = {
|
||||
userId: "user-1",
|
||||
githubConfig: { owner: "acme", token: "github-token" },
|
||||
giteaConfig: {
|
||||
url: GITEA_URL,
|
||||
token: "gitea-token",
|
||||
defaultOwner: DEST_OWNER,
|
||||
issueConcurrency: 1,
|
||||
pullRequestConcurrency: 1,
|
||||
},
|
||||
} as unknown as Partial<Config>;
|
||||
|
||||
const repository = {
|
||||
id: "repo-1",
|
||||
name: DEST_REPO,
|
||||
fullName: "acme/demo",
|
||||
owner: "acme",
|
||||
isStarred: false,
|
||||
mirrorOverrides: null,
|
||||
organization: null,
|
||||
} as unknown as Repository;
|
||||
|
||||
interface Recorded {
|
||||
method: string;
|
||||
url: string;
|
||||
body?: string;
|
||||
}
|
||||
|
||||
/** Destination Gitea with a fixed set of issues already present. */
|
||||
function fakeGitea(existingIssues: Array<{ number: number; title: string }>) {
|
||||
const requests: Recorded[] = [];
|
||||
let nextNumber = 1000;
|
||||
|
||||
globalThis.fetch = (async (input: any, init?: any) => {
|
||||
const url = typeof input === "string" ? input : String(input);
|
||||
const method = String(init?.method ?? "GET").toUpperCase();
|
||||
const body = typeof init?.body === "string" ? init.body : undefined;
|
||||
requests.push({ method, url, body });
|
||||
|
||||
const json = (payload: unknown, status = 200) =>
|
||||
new Response(JSON.stringify(payload), {
|
||||
status,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
|
||||
if (method === "GET" && url === DEST_API) return json({ id: 1, name: DEST_REPO });
|
||||
if (method === "GET" && url === `${DEST_API}/labels`) return json([]);
|
||||
if (method === "POST" && url === `${DEST_API}/labels`) return json({ id: 7 }, 201);
|
||||
if (method === "GET" && url.startsWith(`${DEST_API}/issues?`)) {
|
||||
// The defensive recheck searches by marker; nothing to recover.
|
||||
if (url.includes("&q=")) return json([]);
|
||||
const page = Number(new URL(url).searchParams.get("page") ?? "1");
|
||||
return json(
|
||||
page === 1
|
||||
? existingIssues.map((issue) => ({ ...issue, body: "", state: "open" }))
|
||||
: []
|
||||
);
|
||||
}
|
||||
if (method === "POST" && url === `${DEST_API}/issues`) {
|
||||
nextNumber += 1;
|
||||
return json({ number: nextNumber, state: "open" }, 201);
|
||||
}
|
||||
if (method === "PATCH" && /\/issues\/\d+$/.test(url)) return json({});
|
||||
if (method === "PUT" && /\/issues\/\d+\/labels$/.test(url)) return json([]);
|
||||
if (method === "GET" && /\/issues\/\d+\/comments\?/.test(url)) return json([]);
|
||||
if (method === "POST" && /\/issues\/\d+\/comments$/.test(url)) return json({ id: 1 }, 201);
|
||||
|
||||
throw new Error(`unexpected request: ${method} ${url}`);
|
||||
}) as unknown as typeof fetch;
|
||||
|
||||
return {
|
||||
requests,
|
||||
createdTitles: () =>
|
||||
requests
|
||||
.filter((r) => r.method === "POST" && r.url === `${DEST_API}/issues`)
|
||||
.map((r) => JSON.parse(r.body!).title as string),
|
||||
createdBodies: () =>
|
||||
requests
|
||||
.filter((r) => r.method === "POST" && r.url === `${DEST_API}/issues`)
|
||||
.map((r) => JSON.parse(r.body!).body as string),
|
||||
};
|
||||
}
|
||||
|
||||
function ghIssue(number: number, extra: Record<string, unknown> = {}) {
|
||||
return {
|
||||
number,
|
||||
title: `Issue ${number}`,
|
||||
body: `body ${number}`,
|
||||
state: "open",
|
||||
html_url: `https://github.com/acme/demo/issues/${number}`,
|
||||
user: { login: "alice", html_url: "https://github.com/alice" },
|
||||
labels: [],
|
||||
assignees: [],
|
||||
created_at: `2026-01-0${number % 9 || 1}T00:00:00Z`,
|
||||
updated_at: "2026-09-23T00:00:00Z",
|
||||
...extra,
|
||||
};
|
||||
}
|
||||
|
||||
function ghPull(number: number, extra: Record<string, unknown> = {}) {
|
||||
return {
|
||||
number,
|
||||
title: `PR ${number}`,
|
||||
body: `pr body ${number}`,
|
||||
state: "open",
|
||||
html_url: `https://github.com/acme/demo/pull/${number}`,
|
||||
user: { login: "bob", html_url: "https://github.com/bob" },
|
||||
created_at: "2026-01-01T00:00:00Z",
|
||||
merged_at: null,
|
||||
base: { ref: "main" },
|
||||
head: { ref: `feature-${number}` },
|
||||
...extra,
|
||||
};
|
||||
}
|
||||
|
||||
interface GitHubData {
|
||||
/** What issues.listForRepo returns (issues and PR entries). */
|
||||
issueListing: any[];
|
||||
/** What pulls.list returns. */
|
||||
pullListing?: any[];
|
||||
/** When set, pulls.get throws for these numbers. */
|
||||
failDetailFor?: number[];
|
||||
}
|
||||
|
||||
/** Fake Octokit that records every call made through it. */
|
||||
function fakeOctokit(data: GitHubData) {
|
||||
const calls: Array<{ endpoint: string; params: any }> = [];
|
||||
|
||||
const issues = {
|
||||
listForRepo: async (params: any) => {
|
||||
calls.push({ endpoint: "issues.listForRepo", params });
|
||||
return { data: data.issueListing };
|
||||
},
|
||||
listComments: async (params: any) => {
|
||||
calls.push({ endpoint: "issues.listComments", params });
|
||||
return { data: [] };
|
||||
},
|
||||
};
|
||||
const pulls = {
|
||||
list: async (params: any) => {
|
||||
calls.push({ endpoint: "pulls.list", params });
|
||||
return { data: data.pullListing ?? [] };
|
||||
},
|
||||
get: async (params: any) => {
|
||||
calls.push({ endpoint: "pulls.get", params });
|
||||
if (data.failDetailFor?.includes(params.pull_number)) {
|
||||
throw new Error("detail unavailable");
|
||||
}
|
||||
return {
|
||||
data: {
|
||||
...ghPull(params.pull_number),
|
||||
additions: 1,
|
||||
deletions: 0,
|
||||
changed_files: 1,
|
||||
merged_by: null,
|
||||
},
|
||||
};
|
||||
},
|
||||
listCommits: async (params: any) => {
|
||||
calls.push({ endpoint: "pulls.listCommits", params });
|
||||
return { data: [] };
|
||||
},
|
||||
listFiles: async (params: any) => {
|
||||
calls.push({ endpoint: "pulls.listFiles", params });
|
||||
return { data: [] };
|
||||
},
|
||||
};
|
||||
|
||||
const octokit = {
|
||||
rest: { issues, pulls },
|
||||
paginate: async (fn: any, params: any, map?: (res: any) => any) => {
|
||||
const res = await fn(params);
|
||||
return map ? map(res) : res.data;
|
||||
},
|
||||
};
|
||||
|
||||
return {
|
||||
octokit: octokit as any,
|
||||
calls,
|
||||
callsTo: (endpoint: string) => calls.filter((c) => c.endpoint === endpoint),
|
||||
};
|
||||
}
|
||||
|
||||
function recentCursor(): MetadataSyncCursor {
|
||||
return {
|
||||
lastPassStartedAt: new Date(Date.now() - 60 * 60 * 1000).toISOString(),
|
||||
lastFullPassStartedAt: new Date(Date.now() - 2 * 24 * 60 * 60 * 1000).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
function expectedSince(cursor: MetadataSyncCursor) {
|
||||
return new Date(
|
||||
Date.parse(cursor.lastPassStartedAt) - INCREMENTAL_SYNC_SAFETY_MARGIN_MS
|
||||
).toISOString();
|
||||
}
|
||||
|
||||
describe.skipIf(!isChild)("mirrorGitRepoIssuesToGitea incremental sync", () => {
|
||||
it("runs a full pass without a watermark and records one", async () => {
|
||||
fakeGitea([]);
|
||||
const github = fakeOctokit({
|
||||
issueListing: [ghIssue(1), ghIssue(2), ghPull(3, { pull_request: {} })],
|
||||
});
|
||||
|
||||
const before = Date.now();
|
||||
const cursor = await mirrorGitRepoIssuesToGitea({
|
||||
config,
|
||||
octokit: github.octokit,
|
||||
repository,
|
||||
giteaOwner: DEST_OWNER,
|
||||
giteaRepoName: DEST_REPO,
|
||||
});
|
||||
const after = Date.now();
|
||||
|
||||
const [listing] = github.callsTo("issues.listForRepo");
|
||||
expect(listing.params.since).toBeUndefined();
|
||||
expect(listing.params.sort).toBe("created");
|
||||
expect(listing.params.direction).toBe("asc");
|
||||
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([1, 2]);
|
||||
|
||||
expect(cursor).toBeDefined();
|
||||
const started = Date.parse(cursor!.lastPassStartedAt);
|
||||
expect(started).toBeGreaterThanOrEqual(before);
|
||||
expect(started).toBeLessThanOrEqual(after);
|
||||
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
|
||||
});
|
||||
|
||||
it("lists with since and only reconciles the returned issues", async () => {
|
||||
fakeGitea([
|
||||
{ number: 11, title: "[GH-ISSUE #1] Issue 1" },
|
||||
{ number: 12, title: "[GH-ISSUE #2] Issue 2" },
|
||||
{ number: 13, title: "[GH-ISSUE #3] Issue 3" },
|
||||
]);
|
||||
const github = fakeOctokit({ issueListing: [ghIssue(2)] });
|
||||
const previous = recentCursor();
|
||||
|
||||
const cursor = await mirrorGitRepoIssuesToGitea({
|
||||
config,
|
||||
octokit: github.octokit,
|
||||
repository,
|
||||
giteaOwner: DEST_OWNER,
|
||||
giteaRepoName: DEST_REPO,
|
||||
syncCursor: previous,
|
||||
});
|
||||
|
||||
const listings = github.callsTo("issues.listForRepo");
|
||||
expect(listings).toHaveLength(1);
|
||||
expect(listings[0].params.since).toBe(expectedSince(previous));
|
||||
expect(listings[0].params.sort).toBe("updated");
|
||||
expect(github.callsTo("issues.listComments").map((c) => c.params.issue_number)).toEqual([2]);
|
||||
|
||||
// The watermark moves to this listing; the last full pass is kept.
|
||||
expect(cursor).toBeDefined();
|
||||
expect(Date.parse(cursor!.lastPassStartedAt)).toBeGreaterThan(
|
||||
Date.parse(previous.lastPassStartedAt)
|
||||
);
|
||||
expect(cursor!.lastFullPassStartedAt).toBe(previous.lastFullPassStartedAt);
|
||||
});
|
||||
|
||||
it("creates new issues from an incremental listing in creation order", async () => {
|
||||
const gitea = fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1" }]);
|
||||
// GitHub returns them by updated_at: #5 changed before #4.
|
||||
const github = fakeOctokit({
|
||||
issueListing: [
|
||||
ghIssue(5, { created_at: "2026-09-22T10:00:00Z", updated_at: "2026-09-23T01:00:00Z" }),
|
||||
ghIssue(4, { created_at: "2026-09-22T09:00:00Z", updated_at: "2026-09-23T02:00:00Z" }),
|
||||
],
|
||||
});
|
||||
|
||||
await mirrorGitRepoIssuesToGitea({
|
||||
config,
|
||||
octokit: github.octokit,
|
||||
repository,
|
||||
giteaOwner: DEST_OWNER,
|
||||
giteaRepoName: DEST_REPO,
|
||||
syncCursor: recentCursor(),
|
||||
});
|
||||
|
||||
expect(gitea.createdTitles()).toEqual(["[GH-ISSUE #4] Issue 4", "[GH-ISSUE #5] Issue 5"]);
|
||||
});
|
||||
|
||||
it("falls back to a full pass when the destination has no mirrored issues", async () => {
|
||||
fakeGitea([]);
|
||||
const github = fakeOctokit({ issueListing: [ghIssue(1)] });
|
||||
const previous = recentCursor();
|
||||
|
||||
const cursor = await mirrorGitRepoIssuesToGitea({
|
||||
config,
|
||||
octokit: github.octokit,
|
||||
repository,
|
||||
giteaOwner: DEST_OWNER,
|
||||
giteaRepoName: DEST_REPO,
|
||||
syncCursor: previous,
|
||||
});
|
||||
|
||||
const listings = github.callsTo("issues.listForRepo");
|
||||
expect(listings).toHaveLength(2);
|
||||
expect(listings[0].params.since).toBe(expectedSince(previous));
|
||||
expect(listings[1].params.since).toBeUndefined();
|
||||
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
|
||||
});
|
||||
|
||||
it("runs a full pass when the last full pass is older than 7 days", async () => {
|
||||
fakeGitea([{ number: 11, title: "[GH-ISSUE #1] Issue 1" }]);
|
||||
const github = fakeOctokit({ issueListing: [ghIssue(1), ghIssue(2)] });
|
||||
|
||||
await mirrorGitRepoIssuesToGitea({
|
||||
config,
|
||||
octokit: github.octokit,
|
||||
repository,
|
||||
giteaOwner: DEST_OWNER,
|
||||
giteaRepoName: DEST_REPO,
|
||||
syncCursor: {
|
||||
lastPassStartedAt: new Date(Date.now() - 60 * 60 * 1000).toISOString(),
|
||||
lastFullPassStartedAt: new Date(Date.now() - 8 * 24 * 60 * 60 * 1000).toISOString(),
|
||||
},
|
||||
});
|
||||
|
||||
const listings = github.callsTo("issues.listForRepo");
|
||||
expect(listings).toHaveLength(1);
|
||||
expect(listings[0].params.since).toBeUndefined();
|
||||
expect(github.callsTo("issues.listComments")).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!isChild)("mirrorGitRepoPullRequestsToGitea incremental sync", () => {
|
||||
it("runs a full pass with pulls.list without a watermark", async () => {
|
||||
fakeGitea([]);
|
||||
const github = fakeOctokit({
|
||||
issueListing: [],
|
||||
pullListing: [ghPull(1), ghPull(2)],
|
||||
});
|
||||
|
||||
const cursor = await mirrorGitRepoPullRequestsToGitea({
|
||||
config,
|
||||
octokit: github.octokit,
|
||||
repository,
|
||||
giteaOwner: DEST_OWNER,
|
||||
giteaRepoName: DEST_REPO,
|
||||
});
|
||||
|
||||
expect(github.callsTo("pulls.list")).toHaveLength(1);
|
||||
expect(github.callsTo("issues.listForRepo")).toHaveLength(0);
|
||||
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([1, 2]);
|
||||
expect(github.callsTo("pulls.listCommits")).toHaveLength(2);
|
||||
expect(github.callsTo("pulls.listFiles")).toHaveLength(2);
|
||||
expect(cursor).toBeDefined();
|
||||
expect(cursor!.lastFullPassStartedAt).toBe(cursor!.lastPassStartedAt);
|
||||
});
|
||||
|
||||
it("lists changed pull requests through the issues endpoint with since", async () => {
|
||||
const gitea = fakeGitea([
|
||||
{ number: 21, title: "[PR #1] PR 1" },
|
||||
{ number: 22, title: "[PR #2] PR 2" },
|
||||
{ number: 23, title: "[PR #3] PR 3" },
|
||||
]);
|
||||
const github = fakeOctokit({
|
||||
// Issue #9 changed too; it is not a pull request and is skipped here.
|
||||
issueListing: [
|
||||
ghIssue(9),
|
||||
{ ...ghIssue(3), title: "PR 3", pull_request: { merged_at: null } },
|
||||
{ ...ghIssue(4), title: "PR 4", pull_request: { merged_at: null } },
|
||||
],
|
||||
});
|
||||
const previous = recentCursor();
|
||||
|
||||
const cursor = await mirrorGitRepoPullRequestsToGitea({
|
||||
config,
|
||||
octokit: github.octokit,
|
||||
repository,
|
||||
giteaOwner: DEST_OWNER,
|
||||
giteaRepoName: DEST_REPO,
|
||||
syncCursor: previous,
|
||||
});
|
||||
|
||||
expect(github.callsTo("pulls.list")).toHaveLength(0);
|
||||
const [listing] = github.callsTo("issues.listForRepo");
|
||||
expect(listing.params.since).toBe(expectedSince(previous));
|
||||
expect(listing.params.sort).toBe("updated");
|
||||
|
||||
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number).sort()).toEqual([3, 4]);
|
||||
expect(github.callsTo("pulls.listCommits")).toHaveLength(2);
|
||||
expect(github.callsTo("pulls.listFiles")).toHaveLength(2);
|
||||
|
||||
// The new PR is built from the detail response, which has the refs.
|
||||
const [created] = gitea.createdBodies();
|
||||
expect(created).toContain("feature-4");
|
||||
expect(gitea.createdTitles()).toEqual(["[PR #4] PR 4"]);
|
||||
|
||||
expect(cursor!.lastFullPassStartedAt).toBe(previous.lastFullPassStartedAt);
|
||||
});
|
||||
|
||||
it("keeps the previous watermark when a pull request fails", async () => {
|
||||
fakeGitea([{ number: 21, title: "[PR #1] PR 1" }]);
|
||||
// Detail fails and the basic fallback create also fails.
|
||||
globalThis.fetch = ((original) =>
|
||||
(async (input: any, init?: any) => {
|
||||
const url = typeof input === "string" ? input : String(input);
|
||||
if (String(init?.method ?? "GET").toUpperCase() === "POST" && url === `${DEST_API}/issues`) {
|
||||
return new Response("boom", { status: 500 });
|
||||
}
|
||||
return original(input, init);
|
||||
}) as unknown as typeof fetch)(globalThis.fetch);
|
||||
const github = fakeOctokit({
|
||||
issueListing: [{ ...ghIssue(5), pull_request: { merged_at: null } }],
|
||||
failDetailFor: [5],
|
||||
});
|
||||
|
||||
const cursor = await mirrorGitRepoPullRequestsToGitea({
|
||||
config,
|
||||
octokit: github.octokit,
|
||||
repository,
|
||||
giteaOwner: DEST_OWNER,
|
||||
giteaRepoName: DEST_REPO,
|
||||
syncCursor: recentCursor(),
|
||||
});
|
||||
|
||||
expect(cursor).toBeUndefined();
|
||||
expect(github.callsTo("pulls.get").map((c) => c.params.pull_number)).toEqual([5]);
|
||||
});
|
||||
});
|
||||
+249
-41
@@ -30,8 +30,11 @@ import {
|
||||
listSources,
|
||||
} from "./sources";
|
||||
import {
|
||||
advanceMetadataSyncCursor,
|
||||
parseRepositoryMetadataState,
|
||||
planMetadataPass,
|
||||
serializeRepositoryMetadataState,
|
||||
type MetadataSyncCursor,
|
||||
} from "./metadata-state";
|
||||
import {
|
||||
normalizeReleaseAssetLimit,
|
||||
@@ -1178,13 +1181,15 @@ export const mirrorGithubRepoToGitea = async ({
|
||||
|
||||
if (shouldMirrorIssuesThisRun) {
|
||||
try {
|
||||
await mirrorGitRepoIssuesToGitea({
|
||||
const issuesCursor = await mirrorGitRepoIssuesToGitea({
|
||||
config,
|
||||
octokit,
|
||||
repository,
|
||||
giteaOwner: repoOwner,
|
||||
giteaRepoName: targetRepoName,
|
||||
syncCursor: metadataState.syncCursors.issues,
|
||||
});
|
||||
if (issuesCursor) metadataState.syncCursors.issues = issuesCursor;
|
||||
metadataState.components.issues = true;
|
||||
metadataState.components.labels = true;
|
||||
metadataUpdated = true;
|
||||
@@ -1209,13 +1214,15 @@ export const mirrorGithubRepoToGitea = async ({
|
||||
|
||||
if (shouldMirrorPullRequests) {
|
||||
try {
|
||||
await mirrorGitRepoPullRequestsToGitea({
|
||||
const pullRequestsCursor = await mirrorGitRepoPullRequestsToGitea({
|
||||
config,
|
||||
octokit,
|
||||
repository,
|
||||
giteaOwner: repoOwner,
|
||||
giteaRepoName: targetRepoName,
|
||||
syncCursor: metadataState.syncCursors.pullRequests,
|
||||
});
|
||||
if (pullRequestsCursor) metadataState.syncCursors.pullRequests = pullRequestsCursor;
|
||||
metadataState.components.pullRequests = true;
|
||||
metadataUpdated = true;
|
||||
console.log(
|
||||
@@ -1950,13 +1957,15 @@ export async function mirrorGitHubRepoToGiteaOrg({
|
||||
|
||||
if (shouldMirrorIssuesThisRun) {
|
||||
try {
|
||||
await mirrorGitRepoIssuesToGitea({
|
||||
const issuesCursor = await mirrorGitRepoIssuesToGitea({
|
||||
config,
|
||||
octokit,
|
||||
repository,
|
||||
giteaOwner: orgName,
|
||||
giteaRepoName: targetRepoName,
|
||||
syncCursor: metadataState.syncCursors.issues,
|
||||
});
|
||||
if (issuesCursor) metadataState.syncCursors.issues = issuesCursor;
|
||||
metadataState.components.issues = true;
|
||||
metadataState.components.labels = true;
|
||||
metadataUpdated = true;
|
||||
@@ -1981,13 +1990,15 @@ export async function mirrorGitHubRepoToGiteaOrg({
|
||||
|
||||
if (shouldMirrorPullRequests) {
|
||||
try {
|
||||
await mirrorGitRepoPullRequestsToGitea({
|
||||
const pullRequestsCursor = await mirrorGitRepoPullRequestsToGitea({
|
||||
config,
|
||||
octokit,
|
||||
repository,
|
||||
giteaOwner: orgName,
|
||||
giteaRepoName: targetRepoName,
|
||||
syncCursor: metadataState.syncCursors.pullRequests,
|
||||
});
|
||||
if (pullRequestsCursor) metadataState.syncCursors.pullRequests = pullRequestsCursor;
|
||||
metadataState.components.pullRequests = true;
|
||||
metadataUpdated = true;
|
||||
console.log(
|
||||
@@ -2604,13 +2615,19 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
repository,
|
||||
giteaOwner,
|
||||
giteaRepoName,
|
||||
syncCursor,
|
||||
}: {
|
||||
config: Partial<Config>;
|
||||
octokit: Octokit;
|
||||
repository: Repository;
|
||||
giteaOwner: string;
|
||||
giteaRepoName?: string;
|
||||
}) => {
|
||||
/**
|
||||
* Watermark from the previous completed issues pass (#449). When set,
|
||||
* only issues updated since then are listed and reconciled.
|
||||
*/
|
||||
syncCursor?: MetadataSyncCursor;
|
||||
}): Promise<MetadataSyncCursor | undefined> => {
|
||||
//things covered here are- issue, title, body, labels, comments and assignees
|
||||
// The source client arrives as `octokit` (anonymous for a public-only
|
||||
// source), so only the destination has to be configured here.
|
||||
@@ -2649,30 +2666,74 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
|
||||
const [owner, repo] = repository.fullName.split("/");
|
||||
|
||||
// Fetch GitHub issues
|
||||
const issues = await octokit.paginate(
|
||||
octokit.rest.issues.listForRepo,
|
||||
{
|
||||
owner,
|
||||
repo,
|
||||
state: "all",
|
||||
per_page: 100,
|
||||
sort: "created",
|
||||
direction: "asc",
|
||||
},
|
||||
(res) => res.data
|
||||
);
|
||||
// Incremental sync (#449): once a pass has completed, ask GitHub only
|
||||
// for issues updated since then. A new comment, a label, state,
|
||||
// milestone, title or body change all bump updated_at, so this is the
|
||||
// set that needs reconciling, and the per-issue comment listing below
|
||||
// only runs for it. The watermark is the time this listing started.
|
||||
let plan = planMetadataPass(syncCursor);
|
||||
const listingStartedAt = new Date();
|
||||
|
||||
// Filter out pull requests
|
||||
const filteredIssues = issues.filter((issue) => !(issue as any).pull_request);
|
||||
const listGitHubIssues = async (since?: string) => {
|
||||
const listed = await octokit.paginate(
|
||||
octokit.rest.issues.listForRepo,
|
||||
since
|
||||
? {
|
||||
owner,
|
||||
repo,
|
||||
state: "all",
|
||||
per_page: 100,
|
||||
sort: "updated",
|
||||
direction: "asc",
|
||||
since,
|
||||
}
|
||||
: {
|
||||
owner,
|
||||
repo,
|
||||
state: "all",
|
||||
per_page: 100,
|
||||
sort: "created",
|
||||
direction: "asc",
|
||||
},
|
||||
(res) => res.data
|
||||
);
|
||||
// Filter out pull requests
|
||||
const onlyIssues = listed.filter((issue) => !(issue as any).pull_request);
|
||||
if (since) {
|
||||
// Process in creation order, like the full pass, so issues opened
|
||||
// since the last pass are created in Gitea in their GitHub order.
|
||||
onlyIssues.sort(
|
||||
(a, b) =>
|
||||
new Date(a.created_at).getTime() - new Date(b.created_at).getTime() ||
|
||||
a.number - b.number
|
||||
);
|
||||
}
|
||||
return onlyIssues;
|
||||
};
|
||||
|
||||
let filteredIssues: Awaited<ReturnType<typeof listGitHubIssues>>;
|
||||
if (plan.mode === "incremental") {
|
||||
console.log(
|
||||
`[Issues] Incremental pass for ${repository.fullName}: listing issues updated since ${plan.since}`
|
||||
);
|
||||
filteredIssues = await listGitHubIssues(plan.since);
|
||||
console.log(
|
||||
`[Issues] Incremental listing returned ${filteredIssues.length} changed issue(s) for ${repository.fullName}`
|
||||
);
|
||||
} else {
|
||||
console.log(
|
||||
`[Issues] Full pass for ${repository.fullName} (${plan.reason})`
|
||||
);
|
||||
filteredIssues = await listGitHubIssues();
|
||||
}
|
||||
|
||||
console.log(
|
||||
`Mirroring ${filteredIssues.length} issues from ${repository.fullName}`
|
||||
);
|
||||
|
||||
if (filteredIssues.length === 0) {
|
||||
if (filteredIssues.length === 0 && plan.mode === "full") {
|
||||
console.log(`No issues to mirror for ${repository.fullName}`);
|
||||
return;
|
||||
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
|
||||
}
|
||||
|
||||
const ghIssueMarkerRegex = /\[GH-ISSUE #(\d+)\]/i;
|
||||
@@ -2736,6 +2797,25 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
titleFallbackMap.set(title, existing);
|
||||
}
|
||||
|
||||
// A watermark with no mirrored issue on the Gitea side means the
|
||||
// destination was recreated or emptied since the last pass. Only a
|
||||
// full listing can repopulate it.
|
||||
if (plan.mode === "incremental" && giteaIssueByGitHubNumber.size === 0) {
|
||||
console.log(
|
||||
`[Issues] No mirrored issues found in Gitea for ${repoName}; ignoring the watermark and running a full pass`
|
||||
);
|
||||
plan = { mode: "full", reason: "destination has no mirrored issues" };
|
||||
filteredIssues = await listGitHubIssues();
|
||||
console.log(
|
||||
`Mirroring ${filteredIssues.length} issues from ${repository.fullName}`
|
||||
);
|
||||
}
|
||||
|
||||
if (filteredIssues.length === 0) {
|
||||
console.log(`No issues to mirror for ${repository.fullName}`);
|
||||
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
|
||||
}
|
||||
|
||||
// Get existing labels from Gitea
|
||||
const giteaLabelsRes = await httpGet(
|
||||
`${config.giteaConfig.url}/api/v1/repos/${giteaOwner}/${repoName}/labels`,
|
||||
@@ -2765,7 +2845,7 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
}
|
||||
|
||||
// Process issues in parallel with concurrency control
|
||||
await processWithRetry(
|
||||
const mirroredIssues = await processWithRetry(
|
||||
filteredIssues,
|
||||
async (issue) => {
|
||||
const githubLabelNames =
|
||||
@@ -3084,6 +3164,18 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
console.log(
|
||||
`Completed mirroring ${filteredIssues.length} issues for ${repository.fullName}`
|
||||
);
|
||||
|
||||
// processWithRetry logs and drops items that still fail after their
|
||||
// retries. Keep the old watermark so the next pass lists them again.
|
||||
const failedIssueCount = filteredIssues.length - mirroredIssues.length;
|
||||
if (failedIssueCount > 0) {
|
||||
console.warn(
|
||||
`[Issues] ${failedIssueCount} issue(s) failed for ${repository.fullName}; keeping the previous watermark so they are retried next run`
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -3964,13 +4056,19 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
repository,
|
||||
giteaOwner,
|
||||
giteaRepoName,
|
||||
syncCursor,
|
||||
}: {
|
||||
config: Partial<Config>;
|
||||
octokit: Octokit;
|
||||
repository: Repository;
|
||||
giteaOwner: string;
|
||||
giteaRepoName?: string;
|
||||
}) {
|
||||
/**
|
||||
* Watermark from the previous completed pull request pass (#449).
|
||||
* When set, only pull requests updated since then are processed.
|
||||
*/
|
||||
syncCursor?: MetadataSyncCursor;
|
||||
}): Promise<MetadataSyncCursor | undefined> {
|
||||
// The source client arrives as `octokit`; see mirrorGitRepoIssuesToGitea.
|
||||
if (
|
||||
!config.giteaConfig?.token ||
|
||||
@@ -4007,27 +4105,97 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
|
||||
const [owner, repo] = repository.fullName.split("/");
|
||||
|
||||
// Fetch GitHub pull requests
|
||||
const pullRequests = await octokit.paginate(
|
||||
octokit.rest.pulls.list,
|
||||
{
|
||||
owner,
|
||||
repo,
|
||||
state: "all",
|
||||
per_page: 100,
|
||||
sort: "created",
|
||||
direction: "asc",
|
||||
},
|
||||
(res) => res.data
|
||||
);
|
||||
// Incremental sync (#449). pulls.list has no `since`, so an
|
||||
// incremental pass lists issues updated since the watermark and keeps
|
||||
// the pull request entries. The detail, commits and files calls below
|
||||
// then only run for those. The watermark is the time this listing
|
||||
// started.
|
||||
let plan = planMetadataPass(syncCursor);
|
||||
const listingStartedAt = new Date();
|
||||
type ListedPullRequest = Awaited<
|
||||
ReturnType<typeof octokit.rest.pulls.list>
|
||||
>["data"][number];
|
||||
|
||||
const listAllPullRequests = () =>
|
||||
octokit.paginate(
|
||||
octokit.rest.pulls.list,
|
||||
{
|
||||
owner,
|
||||
repo,
|
||||
state: "all",
|
||||
per_page: 100,
|
||||
sort: "created",
|
||||
direction: "asc",
|
||||
},
|
||||
(res) => res.data
|
||||
);
|
||||
|
||||
const listChangedPullRequests = async (
|
||||
since: string
|
||||
): Promise<ListedPullRequest[]> => {
|
||||
const changed = await octokit.paginate(
|
||||
octokit.rest.issues.listForRepo,
|
||||
{
|
||||
owner,
|
||||
repo,
|
||||
state: "all",
|
||||
per_page: 100,
|
||||
sort: "updated",
|
||||
direction: "asc",
|
||||
since,
|
||||
},
|
||||
(res) => res.data
|
||||
);
|
||||
return changed
|
||||
.filter((entry) => Boolean((entry as any).pull_request))
|
||||
.sort(
|
||||
(a, b) =>
|
||||
new Date(a.created_at).getTime() - new Date(b.created_at).getTime() ||
|
||||
a.number - b.number
|
||||
)
|
||||
.map(
|
||||
(entry) =>
|
||||
// The issues endpoint has no head/base refs. The per-PR detail
|
||||
// call below replaces this with the full pull request; these
|
||||
// fields are what the basic fallback needs if that call fails.
|
||||
({
|
||||
number: entry.number,
|
||||
title: entry.title,
|
||||
body: entry.body ?? null,
|
||||
state: entry.state,
|
||||
html_url: entry.html_url,
|
||||
user: entry.user,
|
||||
created_at: entry.created_at,
|
||||
merged_at: (entry as any).pull_request?.merged_at ?? null,
|
||||
}) as unknown as ListedPullRequest
|
||||
);
|
||||
};
|
||||
|
||||
let pullRequests: ListedPullRequest[];
|
||||
let listedFromIssues = false;
|
||||
if (plan.mode === "incremental") {
|
||||
console.log(
|
||||
`[Pull Requests] Incremental pass for ${repository.fullName}: listing pull requests updated since ${plan.since}`
|
||||
);
|
||||
pullRequests = await listChangedPullRequests(plan.since);
|
||||
listedFromIssues = true;
|
||||
console.log(
|
||||
`[Pull Requests] Incremental listing returned ${pullRequests.length} changed pull request(s) for ${repository.fullName}`
|
||||
);
|
||||
} else {
|
||||
console.log(
|
||||
`[Pull Requests] Full pass for ${repository.fullName} (${plan.reason})`
|
||||
);
|
||||
pullRequests = await listAllPullRequests();
|
||||
}
|
||||
|
||||
console.log(
|
||||
`Mirroring ${pullRequests.length} pull requests from ${repository.fullName}`
|
||||
);
|
||||
|
||||
if (pullRequests.length === 0) {
|
||||
if (pullRequests.length === 0 && plan.mode === "full") {
|
||||
console.log(`No pull requests to mirror for ${repository.fullName}`);
|
||||
return;
|
||||
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
|
||||
}
|
||||
|
||||
// Note: Gitea doesn't have a direct API to create pull requests from external sources
|
||||
@@ -4104,6 +4272,25 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
prIssuesPage += 1;
|
||||
}
|
||||
|
||||
// Same guard as the issues pass: a watermark with no mirrored pull
|
||||
// request in Gitea means the destination was recreated or emptied.
|
||||
if (plan.mode === "incremental" && existingPrIssuesByNumber.size === 0) {
|
||||
console.log(
|
||||
`[Pull Requests] No mirrored pull requests found in Gitea for ${repoName}; ignoring the watermark and running a full pass`
|
||||
);
|
||||
plan = { mode: "full", reason: "destination has no mirrored pull requests" };
|
||||
pullRequests = await listAllPullRequests();
|
||||
listedFromIssues = false;
|
||||
console.log(
|
||||
`Mirroring ${pullRequests.length} pull requests from ${repository.fullName}`
|
||||
);
|
||||
}
|
||||
|
||||
if (pullRequests.length === 0) {
|
||||
console.log(`No pull requests to mirror for ${repository.fullName}`);
|
||||
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
|
||||
}
|
||||
|
||||
const { processWithRetry } = await import("@/lib/utils/concurrency");
|
||||
|
||||
const rawPullConcurrency = config.giteaConfig?.pullRequestConcurrency ?? 5;
|
||||
@@ -4121,9 +4308,10 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
let successCount = 0;
|
||||
let failedCount = 0;
|
||||
|
||||
await processWithRetry(
|
||||
const processedPullRequests = await processWithRetry(
|
||||
pullRequests,
|
||||
async (pr) => {
|
||||
async (listedPr) => {
|
||||
let pr = listedPr;
|
||||
try {
|
||||
// Fetch additional PR data for rich metadata
|
||||
const [prDetail, commits, files] = await Promise.all([
|
||||
@@ -4131,6 +4319,11 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
octokit.rest.pulls.listCommits({ owner, repo, pull_number: pr.number, per_page: 10 }),
|
||||
octokit.rest.pulls.listFiles({ owner, repo, pull_number: pr.number, per_page: 100 })
|
||||
]);
|
||||
// An incremental listing came from the issues endpoint; the
|
||||
// detail response has every field the pulls.list entry has.
|
||||
if (listedFromIssues) {
|
||||
pr = prDetail.data as unknown as ListedPullRequest;
|
||||
}
|
||||
|
||||
// Build rich PR body with metadata
|
||||
let richBody = `## 📋 Pull Request Information\n\n`;
|
||||
@@ -4408,6 +4601,21 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
);
|
||||
|
||||
console.log(`✅ Mirrored ${successCount}/${pullRequests.length} pull requests to Gitea as enriched issues (${failedCount} failed)`);
|
||||
|
||||
// Keep the old watermark when any pull request failed so the next
|
||||
// pass lists it again.
|
||||
const unfinishedPullRequests = Math.max(
|
||||
failedCount,
|
||||
pullRequests.length - processedPullRequests.length
|
||||
);
|
||||
if (unfinishedPullRequests > 0) {
|
||||
console.warn(
|
||||
`[Pull Requests] ${unfinishedPullRequests} pull request(s) failed for ${repository.fullName}; keeping the previous watermark so they are retried next run`
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return advanceMetadataSyncCursor(syncCursor, plan.mode, listingStartedAt);
|
||||
}
|
||||
|
||||
export async function mirrorGitRepoLabelsToGitea({
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
/**
|
||||
* Incremental metadata sync watermarks (#449): parsing, serialisation and
|
||||
* the full vs incremental decision.
|
||||
*/
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import {
|
||||
FULL_METADATA_PASS_INTERVAL_MS,
|
||||
INCREMENTAL_SYNC_SAFETY_MARGIN_MS,
|
||||
advanceMetadataSyncCursor,
|
||||
createDefaultMetadataState,
|
||||
parseRepositoryMetadataState,
|
||||
planMetadataPass,
|
||||
serializeRepositoryMetadataState,
|
||||
} from "./metadata-state";
|
||||
|
||||
const HOUR = 60 * 60 * 1000;
|
||||
const DAY = 24 * HOUR;
|
||||
|
||||
describe("sync cursor parsing", () => {
|
||||
test("a row written before #449 loads with no cursors", () => {
|
||||
const legacy = JSON.stringify({
|
||||
components: { releases: true, issues: true, pullRequests: true, labels: true, milestones: true },
|
||||
lastSyncedAt: "2026-09-01T00:00:00.000Z",
|
||||
});
|
||||
const state = parseRepositoryMetadataState(legacy);
|
||||
expect(state.syncCursors).toEqual({});
|
||||
expect(state.components.issues).toBe(true);
|
||||
expect(state.lastSyncedAt).toBe("2026-09-01T00:00:00.000Z");
|
||||
});
|
||||
|
||||
test("the default state has no cursors", () => {
|
||||
expect(createDefaultMetadataState().syncCursors).toEqual({});
|
||||
expect(parseRepositoryMetadataState(null).syncCursors).toEqual({});
|
||||
expect(parseRepositoryMetadataState("not json").syncCursors).toEqual({});
|
||||
});
|
||||
|
||||
test("cursors survive a round trip", () => {
|
||||
const state = createDefaultMetadataState();
|
||||
state.syncCursors.issues = {
|
||||
lastPassStartedAt: "2026-09-20T10:00:00.000Z",
|
||||
lastFullPassStartedAt: "2026-09-18T10:00:00.000Z",
|
||||
};
|
||||
state.syncCursors.pullRequests = {
|
||||
lastPassStartedAt: "2026-09-20T11:00:00.000Z",
|
||||
lastFullPassStartedAt: "2026-09-20T11:00:00.000Z",
|
||||
};
|
||||
const reparsed = parseRepositoryMetadataState(serializeRepositoryMetadataState(state));
|
||||
expect(reparsed.syncCursors).toEqual(state.syncCursors);
|
||||
});
|
||||
|
||||
test("malformed cursors are dropped instead of failing the row", () => {
|
||||
const raw = {
|
||||
components: {},
|
||||
syncCursors: {
|
||||
issues: { lastPassStartedAt: "yesterday", lastFullPassStartedAt: "2026-09-18T10:00:00.000Z" },
|
||||
pullRequests: { lastPassStartedAt: "2026-09-20T11:00:00.000Z" },
|
||||
},
|
||||
};
|
||||
expect(parseRepositoryMetadataState(raw).syncCursors).toEqual({});
|
||||
expect(parseRepositoryMetadataState({ syncCursors: "nope" }).syncCursors).toEqual({});
|
||||
expect(parseRepositoryMetadataState({ syncCursors: { issues: 42 } }).syncCursors).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe("planMetadataPass", () => {
|
||||
const now = new Date("2026-09-23T12:00:00.000Z");
|
||||
|
||||
test("no cursor means a full pass", () => {
|
||||
const plan = planMetadataPass(undefined, now);
|
||||
expect(plan.mode).toBe("full");
|
||||
});
|
||||
|
||||
test("a recent cursor gives an incremental pass with the safety margin", () => {
|
||||
const lastPass = new Date(now.getTime() - 2 * HOUR);
|
||||
const plan = planMetadataPass(
|
||||
{
|
||||
lastPassStartedAt: lastPass.toISOString(),
|
||||
lastFullPassStartedAt: new Date(now.getTime() - 3 * DAY).toISOString(),
|
||||
},
|
||||
now
|
||||
);
|
||||
expect(plan).toEqual({
|
||||
mode: "incremental",
|
||||
since: new Date(lastPass.getTime() - INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toISOString(),
|
||||
});
|
||||
expect(INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toBeGreaterThanOrEqual(5 * 60 * 1000);
|
||||
});
|
||||
|
||||
test("a full pass older than the interval forces a full pass", () => {
|
||||
const plan = planMetadataPass(
|
||||
{
|
||||
lastPassStartedAt: new Date(now.getTime() - HOUR).toISOString(),
|
||||
lastFullPassStartedAt: new Date(now.getTime() - FULL_METADATA_PASS_INTERVAL_MS).toISOString(),
|
||||
},
|
||||
now
|
||||
);
|
||||
expect(plan.mode).toBe("full");
|
||||
expect(FULL_METADATA_PASS_INTERVAL_MS).toBe(7 * DAY);
|
||||
});
|
||||
|
||||
test("a full pass just inside the interval stays incremental", () => {
|
||||
const plan = planMetadataPass(
|
||||
{
|
||||
lastPassStartedAt: new Date(now.getTime() - HOUR).toISOString(),
|
||||
lastFullPassStartedAt: new Date(now.getTime() - FULL_METADATA_PASS_INTERVAL_MS + HOUR).toISOString(),
|
||||
},
|
||||
now
|
||||
);
|
||||
expect(plan.mode).toBe("incremental");
|
||||
});
|
||||
|
||||
test("a watermark in the future is not trusted", () => {
|
||||
const plan = planMetadataPass(
|
||||
{
|
||||
lastPassStartedAt: new Date(now.getTime() + DAY).toISOString(),
|
||||
lastFullPassStartedAt: now.toISOString(),
|
||||
},
|
||||
now
|
||||
);
|
||||
expect(plan.mode).toBe("full");
|
||||
});
|
||||
});
|
||||
|
||||
describe("advanceMetadataSyncCursor", () => {
|
||||
const started = new Date("2026-09-23T12:00:00.000Z");
|
||||
|
||||
test("a full pass moves both timestamps", () => {
|
||||
expect(advanceMetadataSyncCursor(undefined, "full", started)).toEqual({
|
||||
lastPassStartedAt: started.toISOString(),
|
||||
lastFullPassStartedAt: started.toISOString(),
|
||||
});
|
||||
});
|
||||
|
||||
test("an incremental pass keeps the last full pass time", () => {
|
||||
const previous = {
|
||||
lastPassStartedAt: "2026-09-23T06:00:00.000Z",
|
||||
lastFullPassStartedAt: "2026-09-20T06:00:00.000Z",
|
||||
};
|
||||
expect(advanceMetadataSyncCursor(previous, "incremental", started)).toEqual({
|
||||
lastPassStartedAt: started.toISOString(),
|
||||
lastFullPassStartedAt: "2026-09-20T06:00:00.000Z",
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -18,12 +18,52 @@ export interface AcknowledgedDeletion {
|
||||
giteaSha: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Watermark for one metadata kind that can be fetched incrementally
|
||||
* from GitHub (#449). Both values are the time the GitHub listing
|
||||
* started, not when the pass finished, so anything that changed while
|
||||
* a long pass was running is picked up by the next one.
|
||||
*/
|
||||
export interface MetadataSyncCursor {
|
||||
/** Start of the last pass (full or incremental) that completed cleanly. */
|
||||
lastPassStartedAt: string;
|
||||
/** Start of the last full pass that completed cleanly. */
|
||||
lastFullPassStartedAt: string;
|
||||
}
|
||||
|
||||
export interface MetadataSyncCursors {
|
||||
issues?: MetadataSyncCursor;
|
||||
pullRequests?: MetadataSyncCursor;
|
||||
}
|
||||
|
||||
export interface RepositoryMetadataState {
|
||||
components: MetadataComponentsState;
|
||||
lastSyncedAt?: string;
|
||||
acknowledgedDeletions: AcknowledgedDeletion[];
|
||||
syncCursors: MetadataSyncCursors;
|
||||
}
|
||||
|
||||
/**
|
||||
* How far before the stored watermark an incremental listing starts.
|
||||
* Covers clock skew between this host and GitHub and items whose
|
||||
* updated_at was written just before the previous listing began.
|
||||
* Reprocessing a few items twice is harmless: every write is matched
|
||||
* by its [GH-ISSUE #N] / [PR #N] marker and comments by their id.
|
||||
*/
|
||||
export const INCREMENTAL_SYNC_SAFETY_MARGIN_MS = 10 * 60 * 1000;
|
||||
|
||||
/**
|
||||
* A full pass still runs when the last one is older than this. It
|
||||
* catches what `since` cannot see, such as a label renamed on GitHub
|
||||
* (which does not touch the issues that carry it) or an issue removed
|
||||
* on the Gitea side by hand.
|
||||
*/
|
||||
export const FULL_METADATA_PASS_INTERVAL_MS = 7 * 24 * 60 * 60 * 1000;
|
||||
|
||||
export type MetadataPassPlan =
|
||||
| { mode: "full"; reason: string }
|
||||
| { mode: "incremental"; since: string };
|
||||
|
||||
const defaultComponents: MetadataComponentsState = {
|
||||
releases: false,
|
||||
issues: false,
|
||||
@@ -36,6 +76,70 @@ export function createDefaultMetadataState(): RepositoryMetadataState {
|
||||
return {
|
||||
components: { ...defaultComponents },
|
||||
acknowledgedDeletions: [],
|
||||
syncCursors: {},
|
||||
};
|
||||
}
|
||||
|
||||
function isValidTimestamp(value: unknown): value is string {
|
||||
return typeof value === "string" && Number.isFinite(Date.parse(value));
|
||||
}
|
||||
|
||||
function parseSyncCursor(raw: unknown): MetadataSyncCursor | undefined {
|
||||
if (!raw || typeof raw !== "object") return undefined;
|
||||
const { lastPassStartedAt, lastFullPassStartedAt } = raw as {
|
||||
lastPassStartedAt?: unknown;
|
||||
lastFullPassStartedAt?: unknown;
|
||||
};
|
||||
if (!isValidTimestamp(lastPassStartedAt) || !isValidTimestamp(lastFullPassStartedAt)) {
|
||||
return undefined;
|
||||
}
|
||||
return { lastPassStartedAt, lastFullPassStartedAt };
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether the next issues or pull request pass can ask GitHub
|
||||
* only for what changed. Without a usable watermark, or when the last
|
||||
* full pass is too old, the pass is full.
|
||||
*/
|
||||
export function planMetadataPass(
|
||||
cursor: MetadataSyncCursor | undefined,
|
||||
now: Date = new Date()
|
||||
): MetadataPassPlan {
|
||||
if (!cursor) {
|
||||
return { mode: "full", reason: "no previous completed pass" };
|
||||
}
|
||||
const nowMs = now.getTime();
|
||||
const lastPassMs = Date.parse(cursor.lastPassStartedAt);
|
||||
const lastFullMs = Date.parse(cursor.lastFullPassStartedAt);
|
||||
if (!Number.isFinite(lastPassMs) || !Number.isFinite(lastFullMs)) {
|
||||
return { mode: "full", reason: "stored watermark is not a valid date" };
|
||||
}
|
||||
if (lastPassMs > nowMs + INCREMENTAL_SYNC_SAFETY_MARGIN_MS) {
|
||||
return { mode: "full", reason: "stored watermark is in the future" };
|
||||
}
|
||||
if (nowMs - lastFullMs >= FULL_METADATA_PASS_INTERVAL_MS) {
|
||||
return { mode: "full", reason: "last full pass is older than 7 days" };
|
||||
}
|
||||
return {
|
||||
mode: "incremental",
|
||||
since: new Date(lastPassMs - INCREMENTAL_SYNC_SAFETY_MARGIN_MS).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The watermark to store after a pass that completed without failures.
|
||||
* `startedAt` is the time the GitHub listing started.
|
||||
*/
|
||||
export function advanceMetadataSyncCursor(
|
||||
previous: MetadataSyncCursor | undefined,
|
||||
mode: MetadataPassPlan["mode"],
|
||||
startedAt: Date
|
||||
): MetadataSyncCursor {
|
||||
const started = startedAt.toISOString();
|
||||
return {
|
||||
lastPassStartedAt: started,
|
||||
lastFullPassStartedAt:
|
||||
mode === "full" || !previous ? started : previous.lastFullPassStartedAt,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -93,6 +197,13 @@ export function parseRepositoryMetadataState(
|
||||
);
|
||||
}
|
||||
|
||||
if (parsed.syncCursors && typeof parsed.syncCursors === "object") {
|
||||
const issues = parseSyncCursor(parsed.syncCursors.issues);
|
||||
const pullRequests = parseSyncCursor(parsed.syncCursors.pullRequests);
|
||||
if (issues) base.syncCursors.issues = issues;
|
||||
if (pullRequests) base.syncCursors.pullRequests = pullRequests;
|
||||
}
|
||||
|
||||
return base;
|
||||
}
|
||||
|
||||
|
||||
@@ -76,6 +76,9 @@ export const POST: APIRoute = async ({ request, locals }) => {
|
||||
);
|
||||
}
|
||||
|
||||
// Clearing the whole state also drops the incremental sync
|
||||
// watermarks (#449), so the next issues and pull request passes are
|
||||
// full ones.
|
||||
await db
|
||||
.update(repositories)
|
||||
.set({
|
||||
|
||||
@@ -152,6 +152,12 @@ Throughout, job status updates are published as server-sent events so the dashbo
|
||||
|
||||
Gitea's API cannot create a pull request through the migration path, so mirrored pull requests are created as issues instead. Each one is tagged with a `pull-request` label, its title is prefixed with the original number and status, and its body includes the commit history, changed files, and merge outcome. This is a deliberate workaround for an API limitation, not a temporary gap.
|
||||
|
||||
### Incremental issue and pull request sync
|
||||
|
||||
The first issues pass and the first pull request pass list everything on GitHub. After a pass completes, the repository's metadata state records when its listing started, and later passes ask GitHub only for items updated since then, minus a 10 minute safety margin. New comments, label, state, milestone, title and body changes all move an item's `updated_at`, so only those items are reconciled and only their comments, pull request details, commits and files are fetched. Pull requests are listed through the issues endpoint for this, because the pull request list has no `since` filter.
|
||||
|
||||
A full pass still runs when the last one is more than 7 days old, when the destination has no mirrored issues or pull requests (for example after it was recreated), and after Reset metadata. If any item fails, the watermark is not moved, so the next pass lists it again. Items deleted on GitHub are left on the destination, as before.
|
||||
|
||||
### Ordering and concurrency
|
||||
|
||||
Gitea assigns issue numbers on creation, so processing issues in parallel can produce numbering that does not match GitHub's. The concurrency settings default to 3 for issues and 5 for pull requests, trading some ordering fidelity for speed. Set both to 1 if you need the numbering to line up.
|
||||
|
||||
@@ -33,6 +33,8 @@ Gitea Mirror uses this on every GET. Each response's ETag is cached; every subse
|
||||
|
||||
The practical effect is that sync frequency mostly stops mattering. A `30m` interval on a stable account costs barely more than a `24h` one, because 47 of the 48 daily sweeps are nearly all cache hits. The requests you spend scale with how much changed, not how often you look.
|
||||
|
||||
Issues and pull requests go one step further. After the first complete pass, a scheduled sync asks GitHub only for the issues and pull requests updated since the previous one, and fetches comments, commits and changed files for those alone. A tracker with thousands of issues and a handful of changes costs a handful of requests. A full pass still runs once a week to catch anything the incremental listing cannot see.
|
||||
|
||||
## Configuration that stays inside the budget
|
||||
|
||||
A few settings still matter for large or busy accounts:
|
||||
|
||||
Reference in New Issue
Block a user