2.8 Wikibase to Jupyter Notebooks – publishing from digital collection

Summary

The project Wikibase to Jupyter Notebooks (WB2JN) set out with the goal to improve the Python code library being used to enable publishing directly from digital cultural collections that use Linked Open Data. For the project the open licensed collection Corpus der barocken Deckenmalerei in Deutschland (CbDD) was used and which is available online.

The WB2JN project was built on top of two pre-existing proof of concept prototypes that had established the workflow and key elements of the tech stack, but which were not scalable or generalisable:

A suite of four Python modules were made for the WB2JN project which were deposited on the Python package management repository PyPI.

The top level tool performing the end publishing has been called CPS Impress.

The Python modules and subsequent testing have resolved the issues in the two previous prototypes, Demo #1 and Demo #2, and now enable the production of a production-ready command line tool, CPS Impress.

  • PS Impress: It performs a user defined SPARQL query with a result set filter, and then generates output utilising Jinja2 templating.
  • CPS Selenium GET: A test to establish that Selenium, the popular web scraping tool, is properly installed and functioning.
  • CPS Ceiling Paintings: A Selenium custom scape configuration able ingest the whole of CbDD objects based on its website data. The results are stored in SQLITE as an intermediary storage using an Entity Attribute Value (EAV) data model, data is then imported into Wikibase. SQLITE is an important addition as it makes project portable as no server is required for operation and local testing can be carried out.
  • CPS Wikibase: Wikibase/Wikidata import and query tool utilising WikibaseIntegrator Python module.

Below is a demonstration painting catalogue using CPS Impress and a CPS Impress for Wikibase to Jupyter Notebooks – schematic:

Fig. 1: Demo project
Fig. 2: CPS Impress for Wikibase to Jupyter Notebooks – schematic

Credits and acknowledgements

Project lead: Simon Worthington
Lead developer: Darron M. Broad (RUNSTOP)
Student assistant: Anna Rahr

Thank you to the team Wikibase4Research for Wikibase and data modelling guidance.

Thank you to students of the Bachelor of Arts Information Management, Hannover University of Applied Sciences and Arts-seminar: BIM-126-02 Data Science (2025), March–June 2025. Contributions as beta testers.

Innovation

Technical

  • Bulk data harvest (scrape) and Wikibase import using industry standard software Selenium.
  • Intermediate storage of linked open data using EAV and SQLITE.
  • Wikibase/Data upload and query tool using existing Python module WikibaseIntegrator.
  • Templating for publication output: Using the industry stand Jinja2 templates for SPARQL query data structuring and delivery to Quarto and Jupyter Notebooks.
  • Multi-format publication from Wikibase to Jupyter Notebooks and Quart semi-automation.

Publishing model

A collection clones complete collection data and exhibition data in a Wikibase instance, while still using its existing databases. The cloned Wikibase knowledge base can then be used to automatically create exhibition catalogues for the public visitors in any format. If an exhibit item is temporarily removed for loan etc., the catalogue can be updated.

Here is an example of one such exhibition of paintings at Schloss Solitude. This example is the demonstration catalogue made using open licensed content.

Methodology

A workflow to import semi-structured collections into Wikibase and templates for automated publication.

A prototyping software workflow that can move from command line technical to GUI interface for general use.

Deliverables

Demo: Ein Gemäldekatalog, Stuttgart, Schloss Solitude

Software

Information linking to all the parts can be found here: https://github.com/NFDI4Culture/WB2JN

Documentation

Follow up work

Further development work packages are as follows (work not done within the project):

  • Text sections containing figures or other discrete objects: For the harvesting (scraping) of such objects a workflow needs implementing. The ideas have been explored but could not be implemented in the current project. The challenges are how to read, identify, and structure the parts found.
  • Creating a prototype GUI for the system. The underlying architecture has been designed to support GUI components based on Python applications, one such example would be Wagtail CMS.
  • Creating dashboard interfaces. It became apparent that users will need dashboards, this was not in scope for the project. Options can be Wikibase SPARQL queries or Python Dash and Plotly from data generated.

Original work packages and results

Work package #1: Schema testing in the proposed system to ensure generic data model and creation of benchmark publication.

Results : The solution chosen was to use Jinja2 templating in a Python module that was named CPS Impress. It performs a user defined SPARQL query with a result set filter and then generates output utilising Jinja2 templating.

The challenge was to move data processing out of the Notebooks as this was not the place for the core logic to be performed. The assumption at the start was that schema would be the solution to normalisation and abstraction. This assumption was not a workable solution as user data was heterogeneous preventing the system being abstracted. Instead using a Jinja2 templating was chosen as the solution, used alongside a methodology of having a hierarchical tree data structure in the data model. The tree data structure allows segments and objects to be included or omitted from a publication. In the demo examples each castle or palace represents the root of the data tree, subsequently by writing a SPARQL query to designate a specific castle, then any containers, objects, or attributes. This is how catalogues are designed by the query.

Work package #2: Alpha testing with sample collection of Corpus der barocken Deckenmalerei in Deutschland (CbDD). This would use real world content to check for errors.

Results: The complete CbDD collection was ingested into the WB2JN system, and three sample palaces and castles used as examples for demonstrations.

Work package #3: Quality assurance validation tests. This is needed to demonstrate to other developers that schemas have been carefully adhered to.

Results: The complete software packages have been released as open-source on PyPI and can be inspected by all. https://github.com/NFDI4Culture/WB2JN

Work package #4: Developer and user (publication manager or digital humanist) documentation.

Results: Full documentation has been created as step-by-step guides.

Simon Worthington