Crawler and contact

If you operate a website that offers KNX product databases (.knxprod files) and you found requests from knxprod-index in your logs, this page is for you.

Who we are

The crawler belongs to knxprod-index, part of the open-source Tinkerbus project, run at index.tinkerbus.org by Lukas Matt. It lists the product databases that manufacturers publish for ETS so that users of KNX tools can find them in one place. It is not a general crawler and does not read anything but the pages and files listed for each manufacturer.

Contact: [email protected]

How to recognise it

Every request carries the header

User-Agent: knxprod-index/<version> (+https://tinkerbus.org/docs/index/bot/)

where <version> is the version of the service. The address points to this page.

How it behaves

• robots.txt is obeyed, per host, also for redirects. Address the crawler in your robots.txt with the token knxprod-index. If your robots.txt cannot be read, the crawler treats it as forbidding crawling.
• Crawl-delay is obeyed when it is longer than our own pause, up to 60 seconds.
• Slow. At least two seconds between two requests to the same host, and never less than one second.
• Conditional requests. It sends If-None-Match and If-Modified-Since, so an unchanged file costs a 304, and it caches the pages it reads.
• Few requests. The page list of a source is read once a day, new files are fetched every six hours in small rounds (at most 100 files per round, only for sources that are mirrored), dead links are checked once a week. The list of manufacturers is refreshed once a week.
• Only what you list. It follows file links only on the host of the page that lists them, or on a host that is configured for that source, and never leaves it otherwise.
• Size limits. Files above the configured size are not downloaded.
• No logins. It never signs in anywhere and never uses accounts, keys or sessions of any other service.

The list of the sources that are read, and the result of the last read of each, is public at https://index.tinkerbus.org/v1/sources.

What it does with the files

• It lists them: name, version, the ETS version they are for, link to your page and to the file.
• For sources that are marked as mirrored, it also keeps a copy and serves it, unchanged, with its checksum. A source is mirrored only after the licence of the files was checked and found to allow copies. For all other sources the index only links to your page; its file endpoint answers 404 and the link endpoint redirects to you.
• Users of the Tinkerbus desktop application can install a file from the index. The application checks the checksum and size.

A source is crawled at all only after the terms of use of the site were checked. If we read them wrongly, tell us.

Asking us to stop or to remove something

Write to [email protected] and say what you want:

• Stop crawling my site. We disable the source. A Disallow for knxprod-index in your robots.txt is honoured automatically at the next run.
• Remove a file or a whole manufacturer. We delete the entries and the stored copies (the service has a command for exactly this, and a list of address prefixes that are never indexed again).
• Do not mirror, only link. We switch the source to links only.
• Slow down. Tell us which delay you want; Crawl-delay in robots.txt is the fastest way.

Requests to remove entries or stop crawling are honoured.

Questions

If something looks wrong, for example requests faster than described here, send us the time and your log lines.