From a repository's record to a link

Every night each source is asked for what has changed, its records are read into theses, the people in them are told apart, their roles become links, and the whole source is replaced in the record at once. Repositories describe theses in many dialects, so each format has one reader, and each source's own habits are met by small rules declared beside it. A change in how a source is read reaches the record only once it has been compared with the reading before and the difference declared.

The five steps every source goes through each night, left to right: its records gathered as the repository served them, read into theses, the people in them told apart, their roles turned into links, and the whole source published at once. Between reading and the rest stands the check: where new code reads a source differently, the change goes on only if it was declared.

Five steps, a source at a time

The record is not written by hand. It is made by a pipeline of five steps that the site's server runs for each of its sources, and runs again every night: gather the source's records, read them into theses, tell the people in them apart, turn the roles they hold into links, and publish the source into the record. A source here is one machine-readable feed: a university's repository, a national register, or an open aggregator that gathers many repositories.

Each step keeps a note of what it was given and what code it ran, and a step whose inputs and code are the same as last time does nothing, so a night in which a repository added nothing changes nothing in the record. The steps work on one source at a time, and a source is always replaced whole, so one repository's trouble stays its own.

Gathering the records

Most repositories serve their records through OAI-PMH, an open protocol for exactly this: a program asks for the records of a set, in a named format, a page at a time, and may ask only for those changed since a given day. The site asks each source for what has changed since its last complete read, so the first read of a large repository can take hours or days and a night's read of it far less. A few sources serve their theses another way, through a list a register exports, a site's own static pages, a register's own search, the protocol libraries use for their catalogues, or a file to download whole, and each of these ways has a reader of its own.

Every record is kept exactly as the repository served it, never edited. The one exception is one export format a common repository platform offers, which carries the email addresses of authors and committee members: the site keeps no contact details, so those fields are dropped before the record is stored. The records go to Amazon S3, the file storage that is the data's home, as soon as a source has been gathered, so the raw record behind any link can always be read again.

Each host is asked under the site's own name and at the pace it asks for, and a host that refuses is recorded as refusing and left alone; the page on the sources says how. A page of records the host fails to serve after three tries is stepped over where the next page can still be found, the gap kept with the read and mailed to the site's maintainer, and two such pages in a row end the read until the next night.

Reading a record into a thesis

Repositories describe a thesis in many dialects. The same facts arrive as Dublin Core, as ETD-MS (the format made for theses and dissertations), as MODS or MARC 21 (the libraries' own), as DSpace's internal format, as Germany's XMetaDissPlus, as the TEI that HAL serves, as DataCite's or OpenAIRE's schema, or as Brazil's national format, among others. The site has one reader for each format. A reader knows the format's fields, the words its vocabulary uses for roles and degrees, and the shapes its values take, and it reads every repository serving that format the same way.

A reader turns each record into one shape: a title, a year, a school, a level (a doctorate, a licentiate, a master's or a bachelor's, or unknown), the page and the file where the thesis can be read, its licence, and the people it names, each with their role and any identifier the record gives them, such as an ORCID iD. A record that is not a thesis, an article or a poster, is counted by its type and set aside here. A role the reader does not know is kept as another role and counted, never dropped. A value that names nobody, such as "Not listed" or "n/a" in a supervisor's field, is left off the record and counted as no name.

A made-up record, written as a repository serving the ETD-MS format of theses might serve it, beside the thesis the record holds of it: each field the reader takes is joined to what it became. The names turn round to be read given name first; the advisor and the committee member become links to the author, each marked as cited by the repository's record; the type and the degree's level together make it a doctoral thesis; and the identifier becomes the link to the thesis's own page.

A format is not the whole story. Each repository has its own habits, and the site meets them with small rules, each written for one source or one collection and declared with the sources, collections and years it reads. Three repositories on one platform head the people listed on a thesis's page as committee members and name the chair apart, in a second format the platform serves; a rule reads the chair from that format, and the chair stands as the supervisor. Many of Liberec's records name no supervisor, but the thesis's own title page does, on a line headed "Vedoucí práce"; a rule reads that line and lends the name to the record. The Czech national repository keeps a new copy of a thesis each time it gathers a school's repository; a rule keeps the latest copy and counts the rest. At the University of West Bohemia a supervisor's or a referee's name is written family name first with the titles after a comma; a rule turns it round.

A rule acts at one of five moments: before any record is read (copies of one thesis), on the people a record names, on whether it is a thesis and at which level, on its other fields, and last, to lend a supervisor found elsewhere to a record that names none. The code that holds the rules refuses to start if a rule is declared wrongly, or if two rules could meet on one record with nothing to say which runs first, so no source is read by an accident of order. Many rules were checked on a sample of their source, and not every rule has a check yet; the page on the checks lists every check as it was measured.

The check that holds back an undeclared change

The code that reads the sources changes often: a reader learns a field it missed, a rule is written for a repository's habit. Each change is meant to improve what some source gives, and any change can also alter what another source gives without anyone meaning it to. So every reading of a source keeps a fingerprint of the code that made it: the format's reader and what it draws on, the source's own rules, and the code that runs them.

When new code reaches the server, a source whose code has changed is not simply read again by the night. Its records are still gathered, but its last reading goes on serving, untouched, until the check has compared the two. The check reads the source again with the new code, over exactly the records its last reading was made from, so any difference can come only from the code, and compares the two readings record by record and field by field, and their counts. Then one of three things happens.

The readings are the same: the new one is published, as any night's would be.
They differ, and the change was declared for that source, either by the rule that changed, which names the sources its change is meant for, or in a list of changes kept beside the code, which names the source, the change and why: the new reading is published, with the number of records that differ written beside it. A declaration lets one change through once, and is spent.
They differ, and nothing declared it: the new reading is held, the old one keeps serving, and the source is counted among the night's failures, which mails the site's maintainer.

A change to the code every source shares, such as the code that runs the rules, is a change to every source at once. After such a change every source is read again and compared before it is published again.

Who is who, and the links

Within one source, a name written the same way after the same small clean-up (a title such as "Prof." taken off, a given name first turned round to follow the family name, spaces and capitals set aside) is one person, known by that name within that source. Across sources nothing is joined by name. Two records are one person only where an identifier they share, such as an ORCID iD, a rule that was measured and checked, or a link verified by hand says so; the page on who is who explains why, and what that leaves apart.

Then the roles become links. Each author of a thesis is linked to each supervisor and co-supervisor its record names, and that link is the lineage, marked as cited by the repository's record. A committee chair stands as the supervisor only on a record that names no supervisor at all. Every other role a record gives (a committee member, a referee, an examiner, an opponent, a custos, a member of the jury) is a link of that role, drawn beside the lineage and never walked as part of it. One rule suggests a supervisor where none is named: at six German repositories, where by custom the first referee is often the supervisor, the first referee is linked as an inferred supervisor, drawn in a style of its own and never counted as cited, and a cited link on the same pair always wins.

A thesis from which no cited supervision comes is not forgotten. It is listed with the reason in a word, such as that it names no author, or only a committee, or that its supervisor is the author again; the page on where links come from gives every reason as a thesis's page says it.

Publishing a source whole

A source is published into the record in one transaction of the PostgreSQL database that holds it: its old theses, people and links are deleted, its new ones written, and the list of its theses that name no supervisor written with them, and all of it is committed at once. A reader whose page is drawn while a source is being published sees the source as it was or as it now is, never half of each and never nothing. A publish whose inputs and code are the ones the record already holds is skipped, so the same reading is never published twice.

One night

Every night the server works through every source it publishes. Most of a night is waiting: on each host's pause between requests, on a slow answer, on a large repository's first read. So the sources are gathered in 8 lanes side by side. A lane is a machine, not a name: many universities' repositories on one hosting platform answer from the same machines, so they share a lane and are asked one after another, and no machine is asked by two of the site's reads at once.

Only the waiting goes side by side. What is heavy for the server, copying a source's records to and from Amazon S3 and reading, relating and publishing it, takes turns, one source at a time, and each step runs as a program of its own, so a step that runs out of memory fails its own source and no other. A source that fails is named, and its lane goes on with the next.

An illustration of one night, not a measured one: the sources gathered in 8 lanes side by side, each lane a machine's sources one after another at the pace its host asks; each source read, related and published in turn, one at a time, as its gathering ends; and once every lane has ended, the counts renewed, the landing's lists and the coverage counted, and the copies the network keeps cleared.

When every lane has ended, the database's statistics are renewed once, the landing's lists and the shelves are drawn again from the record as it now stands, the counts of the coverage page are made, and the copies of pages and answers that Amazon CloudFront keeps near its readers around the world are cleared, so a reader's next page comes from the record as it is that morning. A night that skipped a page of records or failed a source mails the maintainer. A source whose first read takes days runs apart from the nights, and the nights leave it alone until it ends.