Self-hosting

knxprod-index is a Go program with an embedded SQLite database, built as one static binary. The source is on git.tinkerbus.org.

Build

CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X main.version=0.1.0" -o knxprod-index ./cmd/knxprod-index

Configure

Copy sources.yaml and edit it:

• user_agent has to say who runs the crawler and how to reach you (it must contain +http or an @). The service refuses to start its jobs while it still holds the example placeholder. Set it to a contact that site operators can use.
• min_delay is the shortest time between requests to one host (default 2 s, never below 1 s).
• data_dir, cache_dir say where stored files and cached pages go. Relative paths are relative to the working directory.
• max_file_mb (default 300) and max_total_gb (default 20) bound the stored copies.
• schedule sets how often each job runs (defaults: registry weekly, crawl daily, fetch every six hours, check weekly, retry after one hour).
• sources: a source is crawled by crawl only when enabled is true. mirror: true additionally lets fetch keep and serve the files, and needs enabled. Mirroring is redistribution of the manufacturer’s files. Whether a site’s terms of use allow crawling and the licence of its files allows copies is not recorded in the configuration: check it and keep track of it yourself before you set either switch. The shipped file does not decide that for you.
• Adding a plain page of links needs only an entry of kind: links in sources.yaml. Under links: it takes every link to a file with one of the listed suffixes; there href_match (a regular expression) narrows them, row, name, article and series say where in the page the details of a file are, file_param reads the file name from a query parameter of the address (some download links carry it as p_File_Name), and raw_urls: true also takes addresses that appear only in the page text, such as embedded JSON. file_hosts, next to links:, names the hosts the files may come from besides the host of the page.

Run

./knxprod-index run -config sources.yaml -db index.db -addr 127.0.0.1:8080

run serves the API and runs the four jobs. Useful flags: -rate N (requests per minute per client, default 120), -trust-proxy (take the client address from X-Forwarded-For; use it only behind a proxy that sets it), -no-jobs (serve only).

A first start, step by step: set user_agent; knxprod-index sources -config sources.yaml shows what is enabled; registry; crawl -source ID -dry-run per source; crawl; fetch -limit 5 as a test; then run.

Docker

docker build --build-arg VERSION=0.1.0 -t knxprod-index:0.1.0 .
docker run -d --name knxprod-index -p 127.0.0.1:8080:8080 \
  -v knxprod-data:/var/lib/knxprod-index \
  -v "$PWD/my-sources.yaml:/etc/knxprod-index/sources.yaml:ro" \
  knxprod-index:0.1.0

• The image runs knxprod-index run (the API and the four jobs) as user knx (uid 10001) and exposes port 8080. The sources.yaml inside the image is the one of the repository, with a placeholder contact: the container stops with a message until you mount your own, as above. To serve what is in a volume without crawling, run the image with the command serve -db /var/lib/knxprod-index/index.db -data /var/lib/knxprod-index/data.
• /var/lib/knxprod-index is the volume: the database, the page cache and the stored files. Back up index.db and data/.
• Put a reverse proxy with TLS in front, which sets X-Forwarded-For, and do not publish port 8080 directly, or clients can fake their address and avoid the rate limit.
• The health check calls /healthz.
• Takedown: docker exec knxprod-index knxprod-index purge -config /etc/knxprod-index/sources.yaml -db /var/lib/knxprod-index/index.db -url-prefix URL, then add the prefix to exclude in the configuration.

With docker compose up -d --build and your sources.yaml next to the docker-compose.yml of the repository you get the same without the long command line. The compose file keeps the data in ./data.

The container image has not been published, and the repository notes that the Dockerfile had not been built when the documentation was written. Build it yourself and check it.

Use it from Tinkerbus

In the application, open Catalog > Online, enter the address of your instance and switch it on. See Catalog.