August saw us publish our update schedule https://airtable.com/appNYgYNqE6TOsm3g/shriBRrUHef67L7sb Therefore, October had to necessarily involve putting the meat to the bones of the schedule.
Existing Laws
Three sets of fields were identified as surfacing important information about changes to the underlying laws:
- Live? status. Is the law still in force or have parts or indeed the full law been repealed or revoked?
- Metadata. A key field is the md_modified date. A clear signal that something has changed!
- Amendments.
The workflows are simple enough in theory but proved time consuming to deliver in practice. In essence the current content of each field of interest is compared to the latest content of the field fetched from legislation.gov.uk. Differences (delta) are captured and stored in 3 new change log fields:
- Live? Change Log
- md Change Log
- Amended By Change Log
When no changes are identified no entry is made to the change log. The workflow is designed to add to the change log over time. So each entry is date stamped.
Here's an example entry for the md Change Log field:
5/10/2023
md_schedule_paras………..156 -> 145
md_total_paras………………258 -> 247
md_modified…………………..2021-08-04 -> 2023-02-13
md_description………………..New value
The work saw us refactor lots of the code underpinning the metadata, amendment and live? field sets and the quality of the data improved. Certainly some of the change log content reflected the improved quality of the data rather than material underlying change. Over time the change logs will develop into a really helpful resource to look back and see the journey a piece of law has been on.
New Laws
Ahead of time we didn't really have a solution in mind to resolving the challenge of identifying new laws. Only when we started to research did we discover that https://legislation.gov.uk had our backs covered. They have an API (that might be undocumented) to capture the laws published on any particular date. Here's the standard landing page for this service: https://www.legislation.gov.uk/new
Obvious changes to this url provide lists of the laws published by date and type code eg uksi, ukpga, ssi. Therefore, we've built a workflow that uses this service.
The workflow filters the results based on simple title and SI Code text searches and checks that we haven't already got the law in our Bases. The final part is a manual QA check over the laws that the auto filter has discarded just in case relevant laws have been inadvertently missed. On the back of this work we've published our results. The PUBLIC views show all the laws but limit the number of available fields. Create an account here at spongl.com for more data!
Obvious changes to this url provide lists of the laws published by date and type code eg uksi, ukpga, ssi. Therefore, we've built a workflow that uses this service.
The workflow filters the results based on simple title and SI Code text searches and checks that we haven't already got the law in our Bases. The final part is a manual QA check over the laws that the auto filter has discarded just in case relevant laws have been inadvertently missed. On the back of this work we've published our results. The PUBLIC views show all the laws but limit the number of available fields. Create an account here at spongl.com for more data!
Environment
Health and Safety
We think these are really helpful resources. The published Grouping is in reverse month order and sorting is reverse date. Which basically means you get the most recent ones first! Let us know if the STARTER views which are available to anyone with an account here should have data download switched on. We'd be interested to learn of your uses for .csv or Airtable download.
… November
The clocks have just gone back and the days are noticeably shorter. What better time to listen to EDM and write code! I stumbled on Amsterdam Dance Event for the first time in October and have pencilled in to visit in 2024. So, what's planned for November? Well, the updates of the Family groups in the update schedule will be progressed. Certainly this is lots easier this month because we now have the workflows! We have two big improvements planned for November.
Amending
Amending. The Amending field lists the laws a law amends. To date we've populated this with a 'backlinks' script running in Airtable. We're going to write the code to forward populate that field. This will help us when creating new laws in the Bases. And will improve the overall data quality in the_ amender - amendee _relationship.
Full Text Article Models
As if this isn't enough, we're going to start the work to bring into the legal registers information from our full text article tables. In particular the duty, dutyholder and POPIMAR tags. These are models that categorise the separate clauses of a law. Therefore, what you'll see appearing in the legal register tables are fields that list each clause / provision / article (use your own term!) by the members of each model. If you're familiar with the 'Geo Extent' field (SUPPORTER & SPONSOR plans) you'll have a good idea of what they'll look like. No decision yet on which plans these new fields will be available on. Certainly we see the duty model making its way into the PUBLIC legal register views.
Neo4j
The stretch target for this month is to export our EARM model and the relationships it embodies into a https://neo4j.com/ database. We think this will provide a new and compelling way to navigate the network of H&S or E laws. Hopefully we'll be able to connect this with the resources in our Airtable Bases. The expectation is that the neo4j database will be made publicly available.
Jason