How it works

registry ──> crawl ──> fetch ──> serve (API)
                         check (independent, weekly)

1. registry loads the list of KNX manufacturers (id, name, country, website) from public data: the KNX master data and the public manufacturers directory. No contact persons, no login to any account. Weekly.
2. crawl reads the download pages of the enabled sources and records which files exist (name, version, ETS version, file URL, page URL). Daily. A source whose layout changed (fewer than a set share of the files found) is rejected and the index stays as it was.
3. fetch downloads the files of sources that are mirrored, reads their metadata, and stores them under their SHA-256. Every six hours, up to 100 files per round. Later rounds ask conditionally, so unchanged files cost a 304.
4. check tests the links of entries that were not seen in the last crawl and marks dead ones. Weekly.
5. serve answers search requests and serves the stored files.

The service process runs the API and the four jobs on their schedule. When a job last ran is kept in the database, so a restart does not start everything again, and a failed job is retried after an hour. GET /v1/status shows the last run of each job.

What happens to a downloaded file

The file is downloaded within a size limit, checked to be a zip, and read: the manufacturer folders, the hardware description and the headers of the application programs (also inside a zip that holds a knxprod, as some manufacturers deliver them). Only then is it moved to its place under its checksum. A file that is not a readable knxprod is not kept. The same content under two addresses is stored once. When a manufacturer replaces a file, the new copy gets its own checksum and the old one is removed when nothing refers to it.

Readers limit the number of entries, the XML size and the nesting depth, because the files come from the internet.

Where the data comes from

Files must be served from the host of the page that lists them, or from a host named in the source’s configuration. Anything else is dropped. The list of sources and the result of their last crawl is public at /v1/sources.

New manufacturers are added with a small parser for their page, tested against a saved excerpt of the page, or, for a plain page of links, with only a configuration entry. A source is added disabled and enabled only after the terms of use of the site were checked.

Sources

The shipped configuration knows these sources. Whether a source is switched on, and whether its files are mirrored, depends on the instance: https://index.tinkerbus.org/v1/sources shows what the hosted one reads.

The two Schneider dummies are placeholder applications for ETS. They are useful for devices that publish no knxprod of their own: Shelly devices, for example, are configured in their own web interface, and Shelly recommends such a dummy (or the Eelectron “Advanced Dummy”, which is not on the manufacturer’s site and so not in the index) to show them in an ETS project.

Not done yet

Automatic discovery of download pages from the registry websites, sites that need scripting or have one page per product, and a published container image. Some manufacturers cannot be indexed at present, for example because their robots.txt forbids downloading the files.