myskin

Crawlers

Each crawler is one site. Open it for the live run, or its configuration page for settings.

—

idle

Recent pages

Queue tail

Configuration

General

Site ID Name Public base URL Enabled

Crawler

Seed URL Sitemap URLs URL match regex Content rules Local sitemap path Local sitemap requeue Max depth Max pages Request delay User agent Respect robots.txt Refresh known URLs Resume on startup Sitemap only Follow file links HTML to markdown Passthrough enabled Extract PDF text Passthrough extensions Progress mode

Scheduler

Scheduler enabled Cron Timezone Interval hours Interval minutes Run on startup

RAGFlow

RAGFlow enabled API URL Dataset ID Sync after crawl Parse on upload Delete missing docs

New crawler

Create the site, then adjust the rest on its configuration page.