Asking 24,500 Repositories a Question

In 1945, Vannevar Bush published an essay called As We May Think in The Atlantic. In it, he described a hypothetical device called the Memex — a desk-sized machine that could store an entire library and, crucially, allow its user to create trails of association between documents. The problem Bush identified was not that knowledge didn't exist, but that we had no good way of finding the specific piece we needed when we needed it. "The summation of human experience is being expanded at a prodigious rate," he wrote, "and the means we use for threading through the consequent maze to the momentarily important item is the same as was used in the days of square-rigged ships."

Portrait of Vannevar Bush — Vannevar Bush during his time in the Office for Emergency Management (part of the US Gov during World War 2) https://commons.wikimedia.org/w/index.php?curid=1633052

I was reminded of Bush's Memex last week, after writing about the X-UK-Gov Public Repository Leaderboard and how we now have over 24,500 public repositories across UK government organisations, complete with Software Bills of Materials. The response was gratifying — a number of people got in touch to say they found the data useful, or that they hadn't realised the scale of what was out there. But one question kept coming up, in various forms: "This is interesting, but how do we actually find the thing we need in 24,500 repositories?"

Article content — Screenshot of the X-UK-Gov Public Repository Leaderboard

It's a fair question. A sorted table is a fine thing (I should know, I've been running one since 2018), but it's not particularly useful if you're a developer in a council trying to work out whether anyone else in government has already solved the problem you're about to spend three months building. We have, in effect, built an enormous library and forgotten to hire a librarian.

I think that's the real gap we've been living with. The code is in the open. The data is available. But discoverability has been, frankly, terrible.

Searching With Intent

Which is why we now have govreposcrape — a semantic search layer over the entire UK government open source estate. It uses Google's Vertex AI Search to index all 24,500+ repositories (and growing), and exposes the results through an API and, more interestingly, through a Model Context Protocol (MCP) server.

For those not yet familiar with MCP, it's an open standard (developed by Anthropic, but designed to be universal) that allows AI assistants to connect to external data sources and tools. In practical terms, it means you can add a single configuration block to Claude Desktop, GitHub Copilot, or any MCP-compatible tool, and your AI assistant suddenly has the ability to search across every public UK government repository as part of its normal workflow.

The setup takes about two minutes. You add a few lines of JSON to your configuration, restart the application, and from that point on we can ask natural language questions about what exists across government code. No API keys to manage, no authentication to configure — the service is freely available.

And yet.

The interesting thing is not the technology. It's what happens when we actually use it.

"Who Else Has Done This?"

I think the most powerful question a developer in government can ask is not "how do I build this?" but "has someone already built this?" So I tried it, asking the kind of questions that come up regularly from teams across the public sector. The results were genuinely illuminating.

The first scenario comes up constantly: "I'm thinking of building a case management system that will OCR text and help the user process casework documents faster. Who else in government has done that? What components can we reuse?"

The response surfaced projects that none of us would have found through keyword searching. Bath & North East Somerset's ProcessMaker implementation — a complete workflow and case management system. Dorset Council's Flood Online Reporting Tool, which is essentially a case intake and processing system. East Sussex County Council's form controls and database file handling libraries. York's Lucene Entity Search Tools for text processing.

None of these are a drop-in solution (they rarely are), but together they represent a considerable body of work that teams could learn from, adapt, or build upon. The alternative — starting from scratch in ignorance of what already exists — is the kind of waste that we should find genuinely frustrating when public money is involved.

"How Should We Store an Address?"

The second question was more specific, and perhaps more revealing: "How should we store an address for portability with other UK government organisations?"

This is one of those problems that every government service encounters, and which (I must admit) I assumed would have been solved definitively years ago. The search surfaced East Sussex County Council's address and personal details library, which implements the BS7666 British Standard for addresses and uses UPRNs (Unique Property Reference Numbers) as the canonical identifier. Dorset Council's GDS Blazor Components showed GOV.UK Design System-compliant address entry patterns.

The emerging consensus from the code is clear: use the BS7666 standard, always include the UPRN, split addresses into structured fields rather than storing them as free text, validate postcodes properly, and design for integration with Ordnance Survey AddressBase. It seems to me that this is exactly the kind of practical, hard-won knowledge that should be easy to find — and until now, it really hasn't been.

From Catalogue to Conversation

It seems to me likely that this is what "reuse" actually looks like in practice. Not a central repository of blessed components (we've tried that approach before, and it tends to go stale), but a searchable, AI-augmented view of what teams are actually building and shipping. The catalogue provides the data. The SBOM collection provides the dependency graph. And the semantic search layer makes it conversational — we can ask questions in the language of the problem we're trying to solve, rather than needing to know which repository to look in.

However, I believe there's something more significant happening here than just better search. When an AI assistant can draw on the entire UK government open source estate as context, it changes the nature of the conversation between developers and their tools. Instead of "write me a function that validates a postcode," the question becomes "show me how other government services validate postcodes, and what patterns have they settled on." The answer comes with provenance, with links to real implementations running in production today, maintained by teams who face the same constraints and compliance requirements we do.

(I should note that the search is only as good as what's in the open. If your department hasn't published its code, it can't be found, it can't be reused, and the rest of government can't benefit from it. Yet another reason to code in the open.)

Getting Started

If you'd like to try this yourself, the govreposcrape repository has full setup instructions. For Claude Desktop, the configuration is minimal:

{
  "mcpServers": {
    "govscraperepo": {
      "url": "https://govreposcrape-api-1060386346356.us-central1.run.app/mcp",
      "description": "UK Government code discovery - semantic search over 24k government repositories"
    }
  }
}

The API is also available directly at the production endpoint with an OpenAPI specification for building your own integrations. The service is free, requires no authentication, and the source code is open (naturally).

I should be upfront: this is running on Google Cloud Run backed by Vertex AI Search, and I have genuinely no idea what the operating costs will settle at. This is, after all, a side quest of a side quest — built in spare time alongside spare time. I intend to keep it running for as long as I can, but if it starts costing more than a few quid a month I may need to scale it back or take it down. The source code will always be there if someone with a more generous cloud budget wants to pick it up.

Reuse in Practice: The National Digital Exchange

This is, in many ways, the same problem I'm trying to solve in my day job. I work on the National Digital Exchange (NDX), and one of the things we've built is NDX:Try — a platform that gives local government organisations free cloud sandboxes to experiment with, no procurement required. The idea is simple: lower the barrier to trying things out. Over 50 organisations are currently using it to explore everything from council chatbots to planning application AI to FOI redaction tools, and every use case is shared openly for others to learn from.

I think the thread connecting all of this — the leaderboard, the SBOMs, the semantic search, NDX:Try — is the same one that's run through my career in public sector technology: we should make it easier for government teams to find, try, and reuse what already exists, rather than building everything from scratch behind closed doors. NDX:Try lowers the barrier to experimentation. The leaderboard and govreposcrape lower the barrier to discovery. They're different tools solving different parts of the same problem.

If you're in local government (or any department with an interesting use case you're willing to share), I'd genuinely encourage you to have a look at NDX:Try. There's a two-minute quiz to find relevant scenarios, or you can browse the full catalogue. It's free, the environments are completely isolated from production, and everything is cleaned up automatically afterwards.

What Comes Next

I think we're at the beginning of something genuinely useful. With over 24,500 repositories catalogued, SBOMs mapping out the dependency landscape, semantic search making the whole thing queryable in natural language, and platforms like NDX:Try making it possible to experiment without a procurement exercise, the infrastructure for meaningful cross-government reuse is starting to exist in a way that it simply hasn't before.

Perhaps the most important shift is a cultural one. We've spent years making the case for coding in the open — and that case has been won, at least in principle. The next challenge is making sure that openness translates into actual reuse, actual collaboration, actual reduction in duplicated effort. That requires making our code not just available but findable, not just findable but understandable in context, and not just understandable but tryable without a six-month procurement cycle.

We're not there yet. But we're closer than we've ever been, and I'm cautiously optimistic that the combination of open data, SBOMs, AI-powered search, and free experimentation platforms might be the thing that finally bridges the gap between "we publish our code" and "we build on each other's work."

Bush imagined his Memex as a desk for one person. What we're building is something rather more collaborative — a shared memory for everyone building public services in the open. Perhaps we should keep going.