Academic Ancestry

About

How it runs

A source is first read on a short-lived Amazon EC2 machine, and every raw record is kept in Amazon S3. One server holds the record in PostgreSQL, answers in Python and draws the pages with React on Node, behind Amazon CloudFront. Every night it reads each source again, many hosts side by side and the heavy work one source at a time.

every nightthe sourcesrepositories,registers,WikidataAmazon EC2short-livedAmazon S3raw recordsthe serverPostgreSQLPythonReact on NodeCaddyAmazonCloudFronta readeran assistantthrough MCP
The system: the sources, read once on short-lived Amazon EC2 machines and every night by the server itself, every raw record kept in Amazon S3; the server, holding the record in PostgreSQL and answering through Python, React on Node and Caddy; Amazon CloudFront in front of it; and a reader, or an assistant through MCP, on the right.

Reading the sources

A source read for the first time can take hours or days, at the pause its host asks for between requests. That first reading runs on a computer rented by the hour from Amazon EC2, Amazon's service of computers for hire: it starts, reads the source, stores what it read, and switches itself off. Every record is kept as the source served it, contact details apart, in Amazon S3, Amazon's storage for files, which is the data's home. The server and every rented machine take their records from there, so nothing is lost when a machine ends, and a source can be read again from what was kept without asking its host twice.

After that first reading the server reads each source itself, every night, asking only for what changed since. The hosts that asked to know where the requests would come from were told one machine, and they see that one machine.

Heavier work that comes seldom goes to larger rented machines in the same way. When new code changes how sources are read, every source can be read again at once on several of them, rented at the spot price, Amazon's spare capacity sold cheaply on the understanding that it may be taken back. The whole record can be rebuilt from the kept records on one such machine and put in place on the server in one step.

The site keeps its own copy of Wikidata's advisor graph, every person Wikidata names as someone's doctoral advisor and their students, in its database. A lineage's people and the links between them are read from that copy; only the facts beside them, such as a thesis's title or a school, are asked of Wikidata, and kept. The copy is moving to Wikidata's weekly dump of every item, read whole on a rented machine of its own, after Wikidata's live search service turned the nightly reading away.

The server

Everything a reader asks is answered by one computer: an Amazon EC2 machine on Graviton, Amazon's own processors, running Linux. On it are a database, two programs that answer, a front door and two helpers.

The database is PostgreSQL. It holds the record: the theses and works, the people, the links between them with where each came from, and the copy of Wikidata's advisor graph. Beside it, in a database of its own, is the site's own: its count of its own use, which keeps no reader's address, the accounts, and what it has asked of outside services, kept so it need not ask again. They are apart so that the record can be replaced whole by a new build without touching anyone's account.

The record's answers, every address under /api, come from a small program in Python. Two copies of it run side by side, so one can restart while the other answers, and every answer passes the same removal of anyone who asked not to be shown. The pages are drawn by React and React Router, run on the server by Node, so a page arrives whole, readable by a program that runs no script, and then comes alive in the browser.

Caddy, a web server, stands in front of both. It holds the site's certificates for HTTPS, renewing them itself, and sends each request on: a page to Node, a question of the record to Python, an assistant's call to the MCP server. The MCP server is built on FastMCP and gives an assistant its tools, each a question to the same Python answers in the reader's name (your assistant and the record). When a link to the site is shared, the picture shown beside it is the page itself, photographed by a browser the server runs without a screen.

a request, through Amazon CloudFrontCaddyHTTPS, and where each request goesthe pagesReact on Nodethe record's answersPython, side by sidean assistant's toolsFastMCPlink cardsa browser, no screenPostgreSQLthe recordPostgreSQLthe site's own
Inside the server: Caddy takes each request that comes through CloudFront and sends it on, a page to Node, which draws it with React, a question of the record to the Python answers, and an assistant's call to the FastMCP server. The pages and the assistant's tools ask the Python answers in turn, which read the record and the site's own database, both PostgreSQL. The link cards photograph the pages.

The owner reaches the server through AWS Systems Manager, with no door open to the world but the web's own. The accounts are copied to Amazon S3 every night.

In front of the server

Amazon CloudFront is a network of servers around the world. A reader's request goes to the one nearest them, and a page or an answer it has kept is given at once, without the request crossing to the server. It keeps each page and answer for a day, and clears every copy when the record changes at the end of a night, so a reader sees each night's record once the night has ended. What depends on the moment, such as a search, a sign-in or a report sent, is never kept. Amazon Route 53, Amazon's service for names, points academicancestry.org at CloudFront, and AWS Certificate Manager holds the certificate CloudFront shows.

Amazon CloudWatch watches the numbers that matter, the site's health, the night's work and what the day has cost, and its alarms send mail through Amazon SNS to the site's contact address when one of them goes wrong.

One night

Every night the server reads each source again. Most of that time is spent waiting: a host asks for a pause between requests, and some answer slowly. So the sources go in lanes, one lane for each machine they live on, told by the address the host's name leads to, since one machine can serve many names. A lane's sources go one after another, each at the pause its own host asked for, and eight lanes run side by side, so that, as far as the addresses tell, a machine is asked by one source at a time.

Only the waiting is side by side. What is heavy for the server, reading a source's records, telling its people apart, turning its roles into links and publishing it, is done one source at a time. A source is published in one transaction: its old rows go and its new ones come in a single step, so a reader in the middle of it sees the old source or the new and never half of one. When every lane has ended, the database's statistics, which it uses to plan how to answer, are renewed once; renewing them after each source took most of a night's publishing. Alarms go out for whatever skipped or failed, the night's counts are drawn, and CloudFront's copies are cleared.

asking each host, side by sidelane 1lane 2lane 3lane 4lane 5lane 6lane 7lane 8then, oncestatistics renewedthe night's countsCloudFront clearedwaiting on a hostread and published, one at a time
One night as a timeline: the sources in lanes by their host, the lanes side by side while each waits on its host at the pace it asked for, and each source's reading, relating and publishing taken one at a time across every lane. When every lane has ended, the database's statistics are renewed once, the night's counts are drawn and CloudFront's copies are cleared.

New code is a different night. When a change to the code changes how a source is read, that night reads the source but does not publish it. The source is read again under the new code and compared, record by record, with what the old code read, and published only where they agree or where the difference was declared beforehand; until then the old reading goes on being shown. How a record becomes a link is on from a repository record to a link.

The services asked as a page opens

Some of what a page shows is asked of other services as it opens, and each is limited and kept.

A search waits a few seconds at most for a source outside the record, and each reader's network address has a number of answers a minute; the page for developers gives the limit (for developers).

Why so small

One server holds everything a reader asks, and the work that is heavy and rare is done on machines rented for hours and then gone. The project's rule is to build the smallest thing that answers the question and to grow it when a measurement asks, not before. The server has been made larger twice, each time when a measurement showed the night's work outgrowing it, and it will be again, or joined by a second, only when a number says so.