Sources API

Base path /api/sources — JWT required. Sources are the UKS Layer A records inside a project.

Endpoints

Method	Path	Purpose
GET	/api/sources?projectId=&page=&limit=&search=&type=	List + filter sources
POST	/api/sources/batch	Add many at once (e.g. from a research run)
POST	/api/sources/scrape-url	Scrape a URL (Jina Reader → direct fetch fallback)
POST	/api/sources/verify-links	Check source-link liveness
PUT	/api/sources/:id	Update a source
DELETE	/api/sources/:id	Delete a source
POST	/api/sources/:id/kb	Toggle the knowledge-base bookmark
POST	/api/sources/dedup/:projectId	Run entity resolution / dedup

Listing

GET /api/sources supports pagination (page, limit), full-text search, and a type filter, scoped to projectId.

Scrape a URL

POST /api/sources/scrape-url — body { "url": "https://…" } (must be a valid URL).

Every URL is checked against the SSRF guard first (private/loopback/cloud-metadata targets are rejected before any fetch happens), and every redirect hop is re-validated too — a URL can't dodge the guard via a 3xx to an internal host.

Fetch strategy: Jina AI Reader first (https://r.jina.ai/<url>, 20s timeout, returns clean Markdown), falling back to a direct GET (12s timeout, basic HTML tag-stripping, capped at 200 KB) if Jina errors, times out, or returns a non-OK status.

Response shape (200 in all cases — failures are signalled in the body, not the HTTP status):

{ "ok": true, "title": "…", "markdown": "…", "word_count": 812,
  "fetched_at": "2026-07-02T…Z", "method": "jina" }

Field	Notes
ok	false if the page couldn't be scraped
blocked	true if the page actively blocked access (login wall, bot detection, rate limit, or the SSRF guard)
error	Human-readable reason, present whenever ok:false
method	"jina" or "direct", whichever succeeded
word_count	Word count of the extracted markdown/text

Bot-detection pages (Cloudflare challenge, "verify you are human", etc.) and 401/403/422 responses are detected and reported as blocked: true rather than silently returning garbage content.

Verify links

POST /api/sources/verify-links — body { "urls": ["https://…", …] } (max 50 URLs). Checks each URL for liveness and for parked/for-sale/placeholder content — a 200 response isn't enough proof a source is still a real page.

Each URL is checked independently (Promise.allSettled, so one failure doesn't abort the batch), through the same SSRF guard and redirect re-validation as scrape-url, with a 10-second timeout and up to 12 KB of body read to scan for parking markers.

Response:

{ "results": [
  { "url": "https://example.com/page", "status": 200, "ok": true,  "verdict": "ok" },
  { "url": "https://dead.example.com",  "status": 404, "ok": false, "verdict": "broken" },
  { "url": "https://parked.example.com","status": 200, "ok": false, "verdict": "parked" },
  { "url": "https://internal.local",    "status": 0,   "ok": false, "verdict": "blocked", "error": "…" }
] }

verdict is one of ok · broken (HTTP ≥ 400, timeout, or network error) · parked (200 but matches a domain-parking/placeholder pattern — "for sale", "coming soon", Sedo/HugeDomains/Afternic markers, etc.) · blocked (rejected by the SSRF guard before any request was made).

Field shape

Each source follows the UKS Source schema — evidence_grade, clinical_status, credibility_score, and per-record license are preserved. domain_data is merged, never overwritten, so deep-analysis results accumulate.

→ Related: Projects · Research · Source Fields