FOIwiki:ScraperSync

ScraperSync is a tool for maintaining an FOIwiki page as a mirror of a dataset from ScraperWiki. It attempts to preserve markup and changes in the page while propagating changes from the dataset.

Dataset requirements

The dataset must have a table called "swdata" containing a column called "name". The contents of this column form the texts of entries in a bulletted list in the wiki page.

List requirements

The start of the maintained list is marked by a special comment like this:

<!-- ScraperSync start { "scraper": "thing" } -->

After the word "start" is a JSON object which configures ScraperSync. It can contain the following entries:

• scraper: names the scraper from which the data should be pulled.
• sort: if set to name, sorts list entries by name.

ScraperSync understands several different list formats, and tries to remove markup and notes to work out what public authority is named by the item.

The maintained list is ended by another special comment:

<!-- ScraperSync end -->

Output

ScraperSync's main output is a new version of the page source. It attempts to:

• cross out bodies that no longer appear in the scraper
• add bodies to the list that appear in the scraper but not in the page
• mark as "gone" any bodies with the defunct tag in WhatDoTheyKnow
• link bodies to WhatDoTheyKnow if they aren't already linked

Creating a new page

ScraperSync has a mode for creating an entirely new page from a dataset. This can be most easily activated by running it with no options and choosing "Create new" and entering the URL name of the Scraper to use as the source.