About The Project

Project Staff

Principal Investigator

Professor Barry Godfrey, University of Liverpool

Co-Investigators

• Professor Robert Shoemaker, University of Sheffield
• Professor Tim Hitchcock, University of Sussex
• Professor Deborah Oxley, University of Oxford
• Professor Hamish Maxwell Stewart, University of Tasmania

Research Associates

• Dr Richard Ward, University of Sheffield (now University of Exeter)
• Dr Lucy Williams, University of Liverpool
• Dr Zoe Alker, University of Liverpool
• Dr Kim Price, University of Liverpool

Public Engagement Officer

Dr Larissa Allwork, University of Sheffield

Project Manager

Dr Sharon Howard, University of Sheffield

Digital Humanities Institute Sheffield

• Jamie McLaughlin, technical development
• Michael Pidd, technical management

PhD students

• Eleanor Bland, University of Sheffield, 'Policing and the Identification of Offenders in Metropolitan London, 1780-1850'
• Cerian Griffiths, University of Liverpool, 'The Prosecution of Fraud in the Metropolis, 1760-1820'
• Lucy Huggins, University of Sheffield, 'The Social and Spatial Worlds of Old Bailey Convicts, 1750-1800'
• Aoife O Connor, University of Sheffield, 'An Ancestor in Crime: Digitisation and the Discovery of Family Deviance'
• Emma Watkins, University of Liverpool, 'The lives and criminal careers of nineteenth-century juvenile offenders'

Funding Bodies

The project was made possible by a large grant from the Arts and Humanities Research Council Digital Transformations Theme.

Technical Methods

Digitisation

While most of the data included in the Digital Panopticon website already existed in some form, the project created a number of new datasets, including both full-text and summary data. Where possible these datasets are made available as open data under a Creative Commons licence.

A number of new datasets were transcribed by project staff: Dead Prisoners Index 1800-1869, Registers of Prison Licences for London convicts and WA Convict Probation Records 1850-1868.

However, other records were too large for this to be feasible. In these cases the approach depended on the nature of the original source. For printed sources, optical character recognition (OCR) was used, while handwritten sources were manually typed using the process known as double rekeying, in which two different typists type the text, the two versions are compared by computer and discrepancies resolved manually.

Two printed sources were digitised and transcribed using OCR: Middlesex House of Detention Calendars 1836-1889 and Metropolitan Police Register of Habitual Criminals 1881-1925. The Middlesex House of Detention Calendars are held by the London Metropolitan Archives, who digitised high-quality images for us. The Metropolitan Police Register of Habitual Criminals is held by The National Archives which supplied us with images. In both cases, the original records were generally in good condition and the text was clear. However, their complex tabular layouts make OCR more difficult and, in particular, they often lacked clear horizontal boundaries between rows of information, which could lead to errors. The overall accuracy of the transcription is at least 99%.

Selected documents from UK Licences for the Parole of Convicts 1853-1925 were rekeyed (in addition to the basic Prison Licence Registers metadata transcribed by the project). In addition, a number of Western Australia record series have been rekeyed and will be added to the website in the near future. The accuracy of rekeying is affected by the sometimes difficult handwriting, particularly in the case of the Western Australia records, and the overall accuracy is approximately 98-99%.

Data Preparation

For effective record linkage data needs to be highly structured and standardised. The project received data in a variety of forms and formats, which required varying degrees of processing to be usable. Each dataset received was audited and documented in detail, paying particular attention to the presence (or lack) of key information for linkage (in particular: name, age or date of birth, gender, occupation, address, birth place, offence, date and place of trial, sentence), and whether it was already structured so as to facilitate linkage or it would require further processing and cleaning.

Record Linkage

The records were linked using a mixture of automated scripts and manual work. The automatic linking was implemented using node.js. Any record sets we felt might contain the same individuals were first filtered to remove impossible combinations of date, criminal sentence, place of birth, age or similar. Then names with compatible attributes were compared using a variety of string comparison algorithms such as Levenshtein distance, DICE and Jaro-Winkler. We found that soundex techniques were not effective for us, probably because we were not usually dealing with material which had been transcribed from speech, but rather records which had been copied from other written sources. Most missed links were due to small clerical errors such as missing letters or slightly different spellings.

Manual linking was undertaken through a bespoke web interface. This was implemented using MySQL and Java servlets, and allowed our researchers to display records which conformed to certain criteria (for example, not being linked to a specific other dataset) and then search again for records which might provide a match.

Website Development

The public Digital Panopticon website is published by the Digital Humanities Institute Sheffield. The website is implemented using Java servlets. The historical background and project guides were developed using Mediawiki. The search is implemented using elasticsearch. The data visualisations use d3.js. The HTML and CSS use bootstrap.

Access to subscription sources

Some of the resources searched by Digital Panopticon are only accessible via subscription. The Digital Panopticon has negotiated free access to aspects of the data held by commercial providers (findmypast and Ancestry), but full access requires a subscription. The Digital Panopticon, a non-profit publicly funded project, does not profit from this arrangement, but it does allow us to provide users with access to aspects of data which would otherwise only be accessible via commercial websites. We are grateful to both findmypast and Ancestry for their participation in the project.

While this website, therefore, allows users to search these resources and examine summary results free of charge, with links to the transcriptions and images at the relevant websites, we cannot provide full access to these resources for non-subscribers. To obtain access, it will be necessary to sign up for an account at their websites.

Partnerships with Archives and Data Providers

This project would not have been possible without substantial amounts of data created by several public and commercial organisations which have worked with us during the course of the project. We are very grateful to these providers for sharing their data with us.

Findmypast provided several crucial datasets from their major collection of Crime and Punishment records from The National Archives and their Census Returns for England and Wales 1841-1911. They also provided access to their images of male UK Licences for the Parole of Convicts 1853-1925 to facilitate transcription.

Ancestry provided a number of UK and Australian convict datasets.

The National Archives UK supplied high-quality images of female UK Licences for the Parole of Convicts 1853-1925 and Metropolitan Police Register of Habitual Criminals 1881-1925, in addition to data obtained via their Discovery catalogue.

The London Metropolitan Archives digitised high-quality images of the Middlesex House of Detention Calendars 1836-1889.

The Founders and Survivors projects at the University of Tasmania and the University of Melbourne provided the key Tasmanian data for VDL Founders and Survivors Convicts 1802-1853 and VDL Founders and Survivors Convict Biographies 1812-1853.

State Library of Queensland created the British Transportation Registers database and made it available as open data.

Tasmanian Archives created the Names Index of Tasmanian convicts and host the portal for linking to VDL Founders and Survivors Convicts 1802-1853 records.

New South Wales State Archives created the series of New South Wales Convict Indexes 1788-1873.

State Records Office of Western Australia suppplied images of several series of convict records of transcription, which will appear on this website in due course.

Free UK Genealogy provided transcriptions data from the Civil Registration index of births, marriages and deaths for England and Wales, 1837-1925, which will appear on this website in due course.

Professor Simon Devereaux of the University of Victoria provided a copy of his Capital Convictions at the Old Bailey 1760-1837 database.

Acknowledgements

We are grateful to the following for their help in bringing this project to completion:

• Professor Andrew Prescott (University of Glasgow), Leadership Fellow for the AHRC Digital Transformations theme.
• The members of our Advisory Board: Professor Peter King (University of Leicester), Professor Mark Finnane (Griffith University), Professor Kris Inwood (University of Guelph), Dr Rebecca Kippen (University of Melbourne), Professor Cormac O'Grada (University College Dublin), Dr Anita Dockley (Howard League for Penal Reform).
• Dr Nick Barratt (formerly of The National Archives), for his help in the early stages of the project.
• Student interns: Samuel Hadfield (University of Sheffield), Alex Traves (University of Sheffield) and Fiona Milne (University of York).
• Paul Nixon, Findmypast UK.
• Trudy Cowley, Founders and Survivors (Tasmania).
• Janet McCalman, Sandra Silcot and Len Smith, Founders and Survivors (Melbourne).
• Darren Wright and Pat Reynolds, Free UK Genealogy.
• Laurence Ward and Charlotte Scott, London Metropolitan Archives.
• Ben Jackson (University of Sussex), who developed the 3D models of the Old Bailey courtroom for the Voices of Authority research theme.
• Mickey and Mallory Ltd, for the website design.
• Anthony Xavier, eAthena Solutions Pvt. Ltd, optical character recognition and rekeying services.
• Tim Sherratt (University of Canberra).
• All the volunteers who generously gave feedback on website development at various stages of the project.

Contact

For information on how to contact the project please see Contact Us.

Author Credits

This page was written by Sharon Howard and Jamie McLaughlin, with additional contributions by other members of the Digital Panopticon project team.