Turning your Spotify playlist into a record collection, and why that could be really disruptive

Quick summary

I’ve long wanted to make it easy to buy the music of artists I like, not least because they get better £pay on physical, but when (not if) online services de-list things and hard drives break, they don’t vanish forever.

So, here’s a first experiment: pulling out one of my annual playlists into a ‘catalogue/shop window’ that links to various retailers, CD and Vinyl, and Bandcamp if that exists for the artist.

This is only 2025 for now, while I work out some of the bugs. https://dgen.net/w/playlists/2025.html

Update: now live at https://secondpress.club/p/5lx398

Some reflections (over the past three decades)

This is an example of shifting power dynamics in consumption, engagement and how we can help deliver open markets that work for everyone. I’ve also noticed that the music sector is often an early adopter/trailblazer for broader societal shifts.

Music streaming has solved discovery: a near-infinite library at your fingertips, finding new artists, forgotten classics and new connections has never been easier.

However, there’s an issue buried in that convenience: you don’t really ‘have’ any of it. If a label dispute pulls an album, or you cancel your subscription, your carefully curated listening history evaporates. This can also apply to your ‘downloads’ (even with good hard drives and cloud services, we still lose our digital stuff).

I’ve long thought about what it actually means to have a music collection. Back in the day (1998!) I put Virgin Megastores online: all 60,000 products (which is how may CDs were in a typical store), and designed a service with a colleague, Rick Glanville, where we could sell a subscription to digital downloads and give away a tiny MP3 player as part of the package. We even got Cambridge Electronics to make a little postage-stamp sized player. [yes, these pre-dated the iPod, iPod Shuffle and Spotify by many, many years].

While on that journey it struck me that we were in the process of reducing ‘music’ to a search box, killing most of the actual experience outside of listening, destroying record stores, and as humans we were likely to eventually push back on something so techno-reductionist.

Many of the things we enjoy about music listening are physical: this has played out in the long-term success of stores like Rough Trade, and the resurgence of vinyl (even cassettes are back!). At the time, as a bit of rebellion, I also released my own music as a 19kg solid granite MP3 player and radio transmitter.

In one of my many roles as CEO of Consolidated Independent, we helped get over 20% of the world’s music online: millions of tracks from thousands of labels distributed to hundreds of retailers and services. Some of the labels were terrified of the web destroying their business due to piracy, but the whole system had to embrace it, including sorting out commercial realities. Sadly some of those realities massively skewed market value to ‘the big guys’, including ludicrous statements from some (let’s say ‘commercial’) CEOs that if artists wanted more money they should increase their output (as if art and soup cans are the same thing ).

Don’t get me wrong, I love the fact we have vast access to new music, and listen to a very diverse range of music. And, it creates different challenges for us as consumers and music fans. Now we can have more agency in balancing out the way the music market works – the cathedrals are not going anywhere, but the bazaar has some new spaces.

What we can now do

One of the things I missed when listening was album art, and a few years back I made a 12″ sized screen to pull down and display whatever I’m playing on Spotify (AI helped me code this, and I would never have got around to it without that help).

Building things just for me is fun, and…

Interestingly, with just a few AI prompts and some lightweight code, I created a tool that takes my Spotify playlist, cross-references every album against Discogs, and produced a personalised catalogue.

Based on the individual tracks I’d added to my annual playlist (which I have going back 8 years now), it looks up the album that it was on. It then works out which are available on CD and Vinyl (or digital–only), where to find them, and who sells them. And, not ‘just Amazon’, but others: Rough Trade, HMV and directly linking back to the artist’s Bandcamp. This is just the start of and idea, and took less than half a day to create it as a prototype.

This makes me wonder about a quiet promise of AI that doesn’t get talked about so much: enabling each of us to build the world we’d like. This will disrupt the big platforms (Spotify, Amazon, Apple) who have invested billions in making their aggregated experience frictionless and hard to leave. They’re using AI to make them more personalised and ‘sticky’. We can do the same for ourselves.

The same AI tools they’re racing to build are equally available to you and me. We can now spin up personalised services—in an afternoon—that would have taken a dedicated development team a decade ago.

So, this is a prototype of a custom, personalised catalogue and record-buying assistant that can be built around your taste, routing money to your preferred retailers that you can share with your friends. It is a small but real act of reclaiming your relationship with artists, and their music, from the big platforms.

We can help bridge the gaps and connect the smaller services together (back to the original vision of the web).

And, looking forward, we can start blending, sharing, cross-connecting using federated services like Mastodon and then connecting people, places, gigs, merch, in a way that could actually help everyone.

Isn’t that exciting!?

I wonder if AI might actually help us actually break away from native mobile apps for a lot of things & ‘get back to the web’?

Many modern hi-fi amplifiers have ‘an App’. They are, uniformly (from what I can see), terrible.

In this post I show how I ‘fixed this’ in under an hour using ai to do something I would never, ever have got around to doing. Given how much I listen to music, I know it’s going to save me many hours of irritation. It even left me with enough time to write this post.

Let’s start with the challenge. In the UK we spend about £600M a year on hi-fi stuff: that seems like quite a lot. And the higher-end kit costs more than your PC. However, while they ‘try and be digital’, it’s like walking back into the 1990s as a user.

Here are some screenshots from the ‘flagship’ Marantz app starting up.

I should point out that not only is the user interface terrible (including trying to get you to sign up to their HEOS garbage) but it takes needless ages to start up and navigate. Those spinning disks spin for a long time, and then present with a page with a tiny round button for the volume and mostly meaningless other stuff that’s too small to consistently accurately tap. It’s really easy to accidentally turn the volume up a lot (which on a >3kW system can be alarming if you’re not expecting it).

Let’s say you want to turn the volume down (the #1 thing you do on an amp ‘instantly’), it can take up to 10 seconds to get there. This means you revert back to the remote control, or – heaven forbid – have to actually stand up and turn actual the volume control – there are plenty of other apps that tell us when we need to move . Further, to get into any granular controls such as changing sound processing, or changing the volume on individual surround channels is painfully slow (tens of seconds to get to, seconds for each change), cumbersome and, actually just unusable. I often want to do this for specific tracks when in a proper listening session.

Clearly in 2026 this really isn’t good enough. Fortunately this is now something (almost) anyone can fix without waiting for manufacturers to wake up and do something.

Googling around the docs there are some API standards and I’d done some previous command-line work to pull status and push things like volume to see if it all actually worked (it did).

But now, with the wonders of AI (in this case, Claude.ai) it is almost magically trivial to make your own interface.

In this post I show how easy (and things that can be frustrating).

Let’s start with a bit of magic: I asked Claude (thinking that this wouldn’t be enough)

> develop a web interface for the Marantz SR7015 using its API

It wasn’t just “enough”, it even styled the page with custom fonts and an image of a surround setup. To get it actually working (locally, on my Mac) I just had to give it the error I got

> it says "Cannot reach 192.168.1.46"

At which point the AI created a local proxy server, told me how to install it and some supporting packages (with copy/paste instructions). If I rewind to when I used to be able to actually code, I think this would have all easily taken me a day. Instead it took this aging hacker about 30 mins of faffing around. This was it – a fully working, way better visual interface and way better user experience (it loads instantly with no noticeable latency on the controls), and individual controls, all on one screen. And, against a black background as usually listening when it’s dark (or watching a movie).

I then spent a few hours playing around with layouts to see what was possible, then asked the AI to make a version I would run on my Pi-Fi (the Raspberry Pi I have that displays album art when Spotify is playing). Which, of course, it did at a similar pace.

This means I can access a web page on my home’s wifi network on my phone, laptop or desktop – as can anyone I give the web address to who is with me. I can use it as a remote for the amplifier, not just for volume, but including controlling all the parameters I actually want to access instantly, with nice big buttons and only the things I need. It still wasn’t able to do some things (like more nuanced reading/writing of custom fields).

I then added new functionality – to pair and group left and right speakers into ‘front’, ‘surround’ so I can control the top-front-side-back distribution more easily (unless you’re a sound engineer, you’ll be surprise how much more this varies than you’d think based on the mix and type of the music).

And, finally, for now, a joystick control to move the balance around between all 11 speakers. Which was done with a single prompt:

> create a joystick control to balance the audio between all the speakers

Next up I’m thinking of keying together listening room profiles per track, keyed into what’s playing on Spotify so the entire balance of the system can auto-configure based on what’s playing. This would then include custom listening profiles and sound stages I’d set up for specific favourite tracks. It’s almost as if one could take all this too far*…?

* Looking out at the hi-fi geek landscape, there is no end of completely stupid nonsense things people do (from £1,000 audiophile USB cables to ‘elevated speaker cable risers‘), none of which make a single bean of difference to anything but your wallet. Once you’ve got a basic system in place by far the biggest differences you’ll be able to actually hear are (a) treating the room with sound insulation, and/or (b) treating the distribution, balance and EQ of the signal itself in your space, which is what I’m doing here.

Those who set the rules and govern data flows, control the value

Data infrastructure is not a technical concern nor a subset of digital policy.

It is a core determinant of national power. It shapes how a country governs itself, secures its citizens and its businesses, how it shapes and grows its economy, and how it positions itself in an increasingly hostile geopolitical and commercial landscape. Data infrastructure also increasingly moderates the perceived legitimacy of relationships between government, institutions, business, and the general population.

As data mediates finance, energy, health, transport, housing and public services, the ability to govern how data flows, who controls it, through what processes and on what terms, is now inseparable from national interest.

Without sovereign control of the rules, permissions and layers of trust that determine data access, use and re-use, the UK risks outsourcing core economic and civic functions to foreign platforms, to standards outside of its control and incentives outside its jurisdiction.

This is not just a question of efficiency or cost: it is a question of agency. Who sets the rules of the digital economy? Who captures the value? Whose values and what incentives are encoded into the systems that increasingly shape everyday life and trade?

Addressing these questions can help to shine light on where important centers of power and decision-making capacity lie across the economy, and stimulate debate about what incentives, risks and benefits may exist. These should be considered on a short, medium and long-term commercial and geopolitical basis. They also highlight that a modern data economy requires a secure, trusted and domestically accountable, legally enforceable layer for organisations who use and control data. This is not just technical plumbing but foundational to democratic legitimacy and social trust.

At the same time, the opportunity is substantial. Countries that invest in trusted, interoperable, market-enabling data infrastructures are laying the foundations for a new wave of productivity and growth. The UK’s experience with Open Banking showed what is possible: clear rules, shared standards and strong governance unlocked a multibillion-pound fintech ecosystem used by millions, and positioned the UK as a global exemplar. As it extends this approach with primary legislation across energy, property, health, transport and other sectors it can unlock new markets, attract investment, and reduce friction across the economy.

Countries that can verify business identities while managing consumer permissions and protecting citizen rights in the digital sphere are better equipped to counter fraud, organised crime and interference from hostile actors. Those that can’t expose themselves to systemic risk.

In an adversarial geopolitical environment, data assets such as energy telemetry, financial behaviours and mobility patterns are strategic resources. We must proceed by assuming all digital systems are under continuous both commercial and ‘bad actor’ attack. Resilience depends on open markets, transparency, interoperability, assurance and governance, not just ‘technology infrastructure’ at scale.

This opportunity is also threatened by corporate capture. A small number of (predominantly US-based or Chinese) technology firms increasingly control critical digital infrastructure: cloud, identity, payments, AI models, developer ecosystems and now the data layer itself (data is a different category to technology).

Through proprietary or ‘high friction’ standards, high switching costs and vertically integrated platforms, these firms are de-facto rule-makers. They shape access, pricing and innovation trajectories without public accountability. This concentrates economic power, weakens regulatory leverage and displaces public value with private extraction. Such challenges have been well documented for some time (see selected references).

This is not inevitable and the UK has a credible alternative path. It can use its legal institutions, regulatory sophistication, academic depth and civil-society capability to lead a different model, based on open standards, open source and open governance. Open source in this context is not an ideology but a strategic instrument that underpins the principles of a free market economy. Properly governed, it reduces dependency, improves transparency, lowers barriers to entry, cost and friction, and allows domestic firms to build on shared foundations rather than renting infrastructure from abroad.

Crucially, open governance preserves cultural and democratic sovereignty. It ensures that the rules of the data economy reflect UK legal norms, social values and public-interest objectives.

Through modular, interoperable trust frameworks, the UK can export not just technology but governance itself, positioning the country as a global leader in trusted data markets. This is how economic value can be created at scale: enabling many firms to innovate on shared infrastructure, rather than entrenching a small number of dominant platforms.

The strategic choice is clear: data infrastructure must be treated not merely as national infrastructure, but as a core component of the economy itself. The UK should act accordingly: owning the rules, investing in open components, institutionalising trust frameworks, and using this architecture to drive growth, resilience and global influence. Nations that do this will shape the rules of the digital age. Those that do not will inherit rules written elsewhere.

Selected references

Khan, L.M. (2017) ‘Amazon’s antitrust paradox’, Yale Law Journal, 126(3), pp. 710–805. https://yalelawjournal.org/note/amazons-antitrust-paradox

Kenney, M. and Zysman, J. (2016) ‘The rise of the platform economy’, Issues in Science and Technology, 32(3), pp. 61–69. https://issues.org/the-rise-of-the-platform-economy

OECD (2019) Competition policy in the digital age. https://oecd.org/competition/competition-policy-in-the-digital-age.htm

OECD (2022) The evolving concept of market power in the digital economy. https://oecd.org/daf/competition/the-evolving-concept-of-market-power-in-the-digital-economy.htm

Srnicek, N. (2017) Platform capitalism. https://politybooks.com/bookdetail?book_slug=platform-capitalism–9781509504862

Zuboff, S. (2019) The age of surveillance capitalism: The fight for a human future at the new frontier of power. New York: PublicAffairs. https://publicaffairsbooks.com/titles/shoshana-zuboff/the-age-of-surveillance-capitalism/9781610395694/

Fetter J., Rao, K., Eaves, D. (2025) State of Digital Public Infrastructure Report https://www.ucl.ac.uk/bartlett/publications/2025/nov/2025-state-digital-public-infrastructure-report

I like to have local copies of everything, so when I upgraded to a new Mac I selected “Mirror files” below and set it running. It immediately moved everything on my GDrive cloud storage to the bin (ie. treated my local empty machine as the ‘new thing’ and overwrote everything in the cloud). This is, obviously, not the behaviour anyone wants.

The ‘solution’ – after you’ve gone online into the Bin, very carefully selected everything and clicked ‘Restore‘ (not Delete forever(!)) – is to launch the application in “Stream files” mode. Then, when it starts to stream go into the finder, right-click and select “Download Now“.

The app itself has a terrible user experience for such a tiny number of options: it’s not always clear which account your are in (I have many Gdrive accounts), it’s not consistent in language and the menus often loop back to the same thing. As for trying to set it to save to an external drive, do let me know if you have any tips as this seems impossible now on the Mac M4 (apparently symlinking is the workaround … it’s 2025!).

We’ve never had more data. Some think it’s the new oil, the new gold, the new soil, something of a revolution. Is it? Or are we just still humans stumbling around in the dark?

Earth observation satellites scan our planet, sensors (there are more sensors on Earth than there are people) track the movements of people and things, energy flows, air quality, water usage, and behaviours. Our global financial systems process billions of transactions a day. Most organisations, public and private, are drowning in dashboards, APIs, spreadsheets, data lakes, clouds, compute, applications, and now ‘ai’.

“Every age thinks it’s the modern age…but this one really is”
Dreams Rewired, Manu Luksch

And yet, when we need to make decisions that matter — how to finance a green retrofit, where to send emergency flood support, or how to measure progress toward net zero, the data we need is often hard to access or nowhere to be found. Or it exists, but it’s in the wrong format, in the wrong place, scattered across organisations and systems, under the wrong licence, or wrapped around the wrong market incentives, or just poorly governed and hard to trust.

Why is this important to our future?

We’re in an era where our financial economy is demanding ‘quality’ data from the real economy. Our financial systems have developed over centuries to measure, track, verify, use and report on the rate of change of ‘one dimension’: money. As we continue the shift to make our environment measurable, we must get the ‘exchange rate’ right between financial investment and environmental outcomes (climate, nature, air quality, water, biodiversity, and so on).

Right now the ‘exchange rates’ are all over the place and, importantly, as we step forward we need to measure the rate of change of many parameters, not just one: we need a step-change in how we think about ‘data’ and data sharing.

It’s hard to understate quite how far away the real economy is from generating financial-grade data

We’re still mostly in the 1970s. Environmental disclosures still rely on highly manual processes, despite widespread digitisation and hundreds of application vendors in the market, it’s still mostly clipboards and spreadsheets or, for 99% of the world, nothing at all. This is not a problem of data scarcity, it’s a problem of incentives, structure, and alignment.

We’ve built systems that are optimised for accumulation, not coordination. We have engineered data pipelines that extract value, but not the wiring to deliver reciprocity (shared benefits).

In the language of infrastructure: we’ve laid the cables, but we’ve neglected the grid

We are told to “make data-driven decisions” while simultaneously being denied the means to do so at scale. Meanwhile, we waste more time (and money, opportunity and emissions) trying to reconcile, reformat, clean, or even find the data than the actual decisions that we are trying to make.

The result is a kind of systemic gaslighting: we pretend that the data is ‘there’ when what we really mean is that it exists … but existing is not the same as usable, and very far away from trusted. Just because something is stored on a server doesn’t mean it’s infrastructure. Just because it’s digital doesn’t mean it’s useful. Just because it’s available doesn’t mean it’s accessible.

McKinsey (The data-driven enterprise of 2025) suggests that data professionals still spend up to 80% of their time preparing data rather than using it. This is the same number I have from over 25 years ago.

So where do we go from here?

Data is everywhere but until it flows to where it’s needed (when it’s needed, and with the right context) we’re not building a digital future we’re just building more digital noise

We need to stop pretending that technology (including ai) alone will solve this. We must stop framing data as a ‘commodity’ to be owned and sweated, and start treating it as infrastructure to be governed.

We must design for connection not monopoly control, for shared purpose not private hoarding. We must invest in the ‘boring’ bits: governance, legal standards, licensing, permission/consent, metadata, interoperability.

This is the stuff that doesn’t make the headlines, but makes everything else work.

The National Data Library can deliver its vision,
but it must pivot to build trust in the age of today’s web

This post builds on observations and conversations across government and industry (and internationally) since my previous post on “How can we find the Goldilocks Zone of our National Data Infrastructure?”.

When we’re building into a future where technology has such clear benefits it is often hard to be heard above the noise of the possible. And yet I believe we must also ‘grasp the nettle’ regarding equally possible negative or unintended consequences. While we must not “stifle innovation and growth”, we also must ensure we don’t scuttle our ship in the process.

The first misconception I want to address is that data is not a technology.

Data is used to represent information, analysis, insights, opinions, facts, responsibilities and decisions. It carries many different kinds of value, as well as material rights and material risks. Data is used to find things out and to make decisions. To look into its past, data is the plural of datum, from Latin, meaning something given.

I labour this point to make clear that it is not software (e.g. AI is software that uses and produces data, as are all algorithms codified in software) and it is also not hardware. A lot of debate about data (and AI in particular) seem to conflate these facts.

Treating data as if it were software is comparing apples and carts.

Context

Having been a tech entrepreneur for 30 years, I’ve sat in meetings with Silicon Valley investors where the “move fast and break things, then ask for forgiveness” mantra has been very present. Innovators love to build; building is fun and very rewarding — both intellectually and, sometimes, financially.

It’s common to worry about the consequences later, even when ‘later’ can be measured in decades. Those raising alarm bells are often badged as ‘activists’. This is a label I’ve never felt helpful: we can draw parallels with the past when ‘health risks of smoking’, through to catastrophic climate change, risked people being boxed as ‘alarmist’ instead of ‘anxious-with-cause’.

I’ve always advocated for fair-weather innovation, while ensuring that we build in measures for foul weather. In the age of the web, commercial (and political) ambition can often be seduced by the fair-weather arguments, while policy is rarely able to move at the speed required to keep pace and safeguard us from foul-weather conditions.

A lot has changed since my previous posts on data infrastructure. We must now, right now, be proactive and preemptive to the fact that our threat model has changed (due to shifting geopolitical and technocratic-utopian positions). Foul weather is closer than we may have wished.

Trust isn’t just a buzzword; it underpins our society and, in a digital age, this means addressing our data infrastructure, ensuring our safety, de-risking innovation and enabling growth in turbulent times. Trust helps us build all of these, for the long term.

Going from vision to implementation is hard

There is a natural tendency towards overreach in most systems — this is a very human attribute. When trying to innovate, our bureaucratic systems can often feel frustrating (good entrepreneurs — both inside and outside of government — know how to navigate around, or sometimes ‘through’, such systems).

Today, in 2025, with data and the web and AI, the pace and scale of impact outpaces and out-scales any other time in history. We can create a critical mass, or a critical mess.

To that end, I believe we must take steps to better understand and balance what is in front of us. When it comes to enabling data sharing, whether in finance, health, transport, energy, property, water, or across all research, we need to first take stock of something that is very hard to build and yet can be lost in the blink of an eye: trust.

The National Data Library (NDL) represents significant intent from the UK Government to do data sharing differently, and we (citizens, consumers, businesses, sectors, markets, and the state) can all benefit from this. However, implementation really matters.

Icebreaker One has already made a strong case (selected for publication by the Wellcome Trust) for the National Data Library to:

1. Focus on defined users and use cases rather than datasets, just like any effective data infrastructure.
2. Begin life as a simple, decentralised version, that would curate and improve the discovery of, and timely access to, public sector data for research.
3. Experiment with more complex architectures for harmonising access to data drawn from across multiple government departments.

Cutting across all of this, the NDL must be built, from its core, in ways that engender trust. Whether consumer distrust, competitive interest or geopolitical threats, we must have in place the people, processes and protections to cement trust between businesses (b2b), with consumers (b2c), and with our citizens.

The NDL can still maintain its grand vision to improve outcomes for all of us, connecting data, rapidly unlocking insights, and leading the way.

More than its technical architecture, the nature of its funding and staffing will shape what it can achieve. I’d argue we can do more together, and move faster, with common principles and parallel actions than a large centralised effort. Not only should the data never be centralised (other than for the limited use cases that require that), but neither should the absolute control.

We can go far together if we each do one thing well.

We can bind our work, together, with common principles and practices. There will be no ‘one hub’ — if you are a hub in a collection of hubs, you’re a node. Even a National Library is one node in a network of knowledge. There isn’t only one search engine in the world, nor will there be one AI.

We must also approach, with a clear perspective, that one-time broad-based consent is just one limited use case. It may apply to certain, relatively static, data but a different approach is required for highly variable data: data is rarely static. The conditions on which that ‘one-time’ consent, as we have also seen, will also change based on lived experience (e.g. changes in corporate and political governance).

In a world of rapid change, we must pivot rapidly but
never forget the foundations of trust beneath our feet

Building on the Smart Data model, there are known ways to apply decentralised, user-controlled, real-time, granular consent. There are dynamic solutions in place today which enable peer-to-peer data sharing via secure APIs. Such systems can include immediate revocation (by the user, or by a neutral governing entity/representative), and they are designed and built to enable cross-sector interoperability.

\Equally, such systems will not (and should not) be the approach for all use cases. What works will depend on the purpose (which will span personal, business, national, research as well as small, medium and large aggregate/collective datasets). What matters is how they are governed.

This is where implementation matters. If we over-centralise and over-reach in our ambition we run the risk of the litany of failed ‘IT projects’ which have cost the taxpayer billions.

I see many projects, even some with the word ‘governance’ in their titles, take a technology-led approach rather than a governance-led approach: data is not a technology—it enables insights about its subjects and those subjects have rights.

Recommendations

Without trust in the systems upon which we run our lives we will, at best, stifle innovation, productivity and growth. At worst, we risk undermining far more.

The good news is that we can run, at pace, fast but not foolhardy, to build the data infrastructure we need for today and scale for all our tomorrows, to deliver trust, ensure our safety and unlock the value of data for everyone.

Phase 1: Immediate reorientation on purpose and language (0–6 Months)

Step 1: Transition the “Library Card” model to a Trust-based, scalable model

• Pivot from the concept of ‘library’ to a ‘web of interoperability and trust’ (WIT)

For example, announce an evolution from the “National Data Library” to something like a “Web of Interoperability and Trust” (WIT) or “National Data Web” or “Trusted Data Web”, which would align with work already in place in the financial sector and give more scope to connecting existing initiatives, including Smart Data, without being specific about technology preferences.

• Define six exemplar use cases for pilot delivery in 2025

Pick six lighthouse use cases across sectors, assign to different entities for delivery, and shift governance to the model that best enables them (e.g. from “pre-approved datasets” to a federated trust framework where applicable).

Adopt an API-first architecture to allow controlled, on-demand data sharing. Use them to produce quantitative evidence as to the pros and cons of the approach.

Step 2: Introduce an adaptive Smart Data consent framework (6–12 months)

• Develop and mandate common principles

Common principles should include application for ‘Data Consent/Permission’ for consumers, SMEs, researchers, businesses and government departments to view, grant, and revoke access to datasets. These should be evidenced through use cases with specific cost benefit and threat analysis.

• Require and mandate data best practices

Require that all third-party users (e.g. AI developers, financial institutions) justify the minimum data required per use case. Mandate that every material access request is logged, auditable, and policy-compliant (e.g. GDPR, Data Bill, and related policies). Where applicable, develop formal Schemes that codify multilateral contracts and enforce them. Note that Schemes can also be industry-led and voluntary, with appropriate governance: political signalling can be an efficient way to accelerate implementation.

Phase 2: Build systemic trust (6–18 Months)

Step 1: Shift to an API-enabled Data Access Model (including trust frameworks) based on user needs

In this instance ‘API-driven’ is an architectural choice aligned with the web, not a specific technology, that enables both interoperability and controls to be implemented. Other use cases won’t need ‘APIs’ (e.g. a CSV file of Open Data), but where security and access control are required, any technical solution must be able to operate at web scale. Key is the development of the governance mechanisms that help implement trust frameworks.

• Support development of sandboxes

Support the rapid, cross-sector creation of sandbox environments with political and financial support where required.

• Support open communication, transparency, and build collaboration and reciprocity with a broad stakeholder group. Address business, public and government skills and knowledge gaps.

Embed open communications, transparency and reciprocity into all programmes. Demonstrate to businesses and consumers that technologies such as AI can deliver value without undermining trust. This can be achieved by facilitating co-design across the value chain (e.g. govt, industry and end users, whether b2b or b2c)

• Create systemic links with digital identity developments

Incorporate systems-level authentication (e.g. OAuth-based which is already ubiquitous) — data access should be controlled via digital identity-linked permissions for companies and research organisations. This is aligned with the Data Bill and FCA approaches. Digital identity is a huge topic which I do not want to dive into here, but I do assume that it will (also) be federated (there should and will be no ‘one place’ for it).

Step 2: Maximise privacy enhancement for people and businesses (e.g. SMEs)

• Create mandates and incentives for privacy enhancement

Broaden and define Schemes to enable ‘Access Rulesets’ and enforce data-minimisation principles for citizens, consumers and businesses (including b2b SME data sharing), so datasets can be queried without bulk transfers (e.g. an app can check fact without revealing the underlying data, e.g. “are they over 18 — yes” not “what’s their birthdate?”)

Phase 3: National Smart Data Infrastructure (18–36 Months)

(yes, this may feel ambitious: why shouldn’t it be this rapid?)

Step 1: Legislate Data Interoperability

• Mandate interoperable systems where consent/permission or access control is required

Draft legislation ensuring every public-sector dataset must be ‘API-accessible’ under standardised, interoperable formats. As noted, not all public-sector data needs an API. In fact Open Data should be made available in any way that makes it discoverable and usable, and even CSV files can be catalouged by other systems and made ‘API accessible’. However, where some form of access control is needed, secure APIs are the standard that works for the web (e.g. Open Banking has 12 million monthly active users across the UK).

Also note that interoperability at the API level does not mean that all internal systems must do the same thing in the same way: data can be repurposed and presented without rebuilding entire estates (this will create an incentive for doing so over time, but it doesn’t need to happen first). Interoperability and decentralisation can work together, with appropriate governance, to enhance competition.

• Mandate open access and empower regulators to enforce compliance

Mandate open access to government datasets where possible, while protecting sensitive data via consent-based permissions, and democratically set rules. Pick regulators or code bodies to oversee the development of the specific rules and compliance required, including modes of redress. Note that open access and Open Data are different things.

Step 2: Scale the Model Beyond Government Data

• Support the development of connected demonstrators and pilots

Launch pilots with commercial ecosystems to enable interoperability (e.g. real estate, sustainability finance, public health). This includes cross-border and global data-sharing ecosystems (e.g. finance, goods).

• Enable interoperability beyond the UK, within the desired framework

Develop cross-border data-sharing agreements (‘Schemes’). Ensure interoperability with international frameworks (e.g. EU Digital Identity Wallet, OECD AI Data Principles and related emerging programmes). Ensure every data use and dataset are AI-ready and privacy-compliant via automated monitoring and governance.

Summary

Strategic Benefits

• Accelerates mission-driven government: Enables implementation of the government’s five missions via trusted, scalable data access.
• Supports economic growth: Unlocks interoperability for data-driven innovation across UK sectors.
• Protects society through accountable systems: Ensures that data-enabled systems are governed with clear and enforceable safeguards. Enables a coherent approach to data sovereignty across our economy and society, to minimise the risks of misuse and systemic harms in a rapidly evolving landscape.
• Future-proofs data governance: Aligns with existing data sharing frameworks, and with evolving standards for AI training, licensing, and consent. Licensing will be a cornerstone in all future data work whether AI-linked or not.
• Embeds assurability at scale: Operationalises assurability across domains.

Five WIT principles (for discussion)

1. Interoperability by design: Common standards, APIs, and metadata frameworks to ensure frictionless cross-sector data use.
2. Trust infrastructure: Consent-based access, verified credentials, and relevant-time provenance tracking.
3. Federated governance: Sectoral autonomy with shared rules, coordinated through a Web of Interoperability and Trust.
4. User empowerment: Individuals and organisations have control their data-sharing preferences. Strong governance-led approached apply to aggregate data, with the option to revoke.
5. Composable architecture: Modular, user needs based and cross-sector enabling use cases, using sharable registries, vocabularies, and open interfaces.

NB: This post contains (as-always) personal opinions and thinking-in-progress (strong opinions, weakly held) as I navigate through the maze(s) of data governance. In amongst many experiences, I sat on the MiData energy sector board; co-chaired the creation of the Open Banking Standard; was founding CEO of the Open Data Institute; co-Chaired of the Smart Data Council. I run a non-profit (IB1.org) working on data governance at sector, national and international scale.

With thanks to Jack, Peter, Chris, Frank, Jeni, Simon, Julia, Kathryn, and many others who have helped shape my thinking on this.

(please note my disclosures at the end of this post)

This post contains (as-always) personal opinions and thinking-in-progress (strong opinions, weakly held) as I navigate through the maze(s) of data governance. Persistent in my questions are how we apply the Goldilocks principle of governance vs innovation: to remain flexible and avoid brittleness.

Given the conversations I’m in at the moment I believe it very important to have as much debate as openly as possible, as this impacts everyone.

Here’s a question:

How should the UK implement its National Data Infrastructure so that it provides consistent control, sharing, and security for data, much like other public infrastructures such as roads and utilities?

“Everything should be as simple as it can be, but not simpler”
[often attributed to Einstein]

Having initiated ‘data as infrastructure’ at a political level in the UK in 2013, I’ve been watching and learning on this for a long time. We’ve made remarkable progress in some areas, less so in others.

Let’s explore, starting with some lessons learned.

One immediate reflection is that as over 60 countries moved to copy the Open Banking Standard, those who took only ‘parts’ of the system tended to experience less ideal outcomes (to the point that they often add them back in later to help course-correct — most notable are those who thought this was ‘just a tech problem’ and only took the open API principles).

We can look to past UK data initiatives like MiData and Open Banking and potentially infer that voluntary-only initiatives and centralisation do not work, and that the decentralised nature of Open Banking was more successful. However, these are shallow reflections.

We can point to three factors beyond centralised or voluntary which are dependent on the use case:

1. Market incentives: there must be an economic argument that policy can then amplify or mandate. If there is no financial incentive, there will be no movement (regardless of central vs decentralised, mandate or not — mandates are also often ignored). Midata was mostly tech-led, Open Banking was use-case led.
2. Removal of friction: Removing transactional friction may seem like something everyone wants, but not if your current business model relies on it. There must be “something in it” for everyone, or at least a path to cost reduction or a new business model. Removing friction can help everyone go together: this is never a ‘technology problem’ (e.g. absence of a data ontology).
3. Mandates: if the friction is ‘too high’, regulatory intervention is necessary to mandate participation and move the market. However, if the market can demonstrate self-initiation, then ‘endorsement’ from government can be sufficient to drive impact. Value creation must be visible in both cases.

I believe that we need to embrace these factors in our systems design.

We need to:

1. work out where the lines are drawn between central, pre-competitive and competitive areas;
2. ask what the role of regulation is, and at what granularity
3. query how things can be implemented in a way that humans (and machines) can and want to adopt
4. ensure those governing, operating and participating can be helped in their assessments of compatibility and interoperability
5. work out how to prioritise and create clear (and stable) roadmaps that enable investment to be made

I believe that we can build on the success of sector-specific approaches:

The Open Banking model’s success lies precisely in its sector-specific, incremental approach. It focused on the unique needs of banking, where security, standardisation, and customer trust were paramount.

Copying the Open Banking model into other sectors (e.g., energy, transport, health) isn’t about duplication but about applying tested frameworks tailored to each industry.

Without considering the market incentives for participation we risk techno-utopian thinking

Instead of trying to create a complex, overarching architecture that risks being too broad, a sectoral approach allows for flexibility and adaptation to industry-specific challenges. This carries a risk of fragmentation, and equally a potential to mandate cross-sector interoperability without defining exactly what this means in detail for everyone. The risk of push back on over-reach is material and catastrophic (in terms of adoption, if it fails). Further, market-incentives will be market-specific.

Avoiding Over-Standardisation

National Data Architecture risks over-standardisation and could stifle innovation instead of enhancing it. Different sectors have wildly varying levels of data maturity, hugely variable data governance needs and challenges, and a one-size-fits-all approach is highly likely to impose unnecessary constraints on sectors that are not as ready for them.

For example, imposing the level of data governance on the industrial sector that is required in the financial sector is (today) not realistic. We are, however, on the frontline of that journey with ESG reporting today, and it highlights the complexity of that real-financial sector bridging. Incremental improvements, like extending Open Banking principles to energy or transport can allow industries to evolve organically while keeping governance controls and security as a priority.

Pragmatism of the Incremental Approach

The incremental, domain-by-domain approach may seem less ambitious, but it offers a practical pathway to innovation that can be adopted today. Trying to design a top-down, comprehensive national architecture from the outset risks paralysis by analysis and over-planning. Learning from Open Banking and rolling out smart data initiatives sector-by-sector allows for continuous improvement and responsiveness to real-world feedback, which may be more effective than a sweeping, coordinated architecture.

Interoperability Requires Industry Engagement

Interoperability relies on industry engagement. This is best fostered through practical, sector-specific initiatives like Open Banking. Imposing an overarching system from above risks of alienating key industry stakeholders, who may resist changes that threaten their business models (this happens in every sector already, today). Open Banking’s success came from the balance of regulatory oversight and industry cooperation, a formula that can be replicated in other sectors.

Market Failures Can Be Addressed Incrementally

There are (always) risks of market failures such as data hoarding (castle & moat is still the prevalent investment model for data businesses). These issues should be tackled incrementally through sector-specific mandates rather than a top-down architecture, but can be accelerated with top-down principles.

The competition fostered by Open Banking and similar initiatives already shows promise in tackling data monopolies. An incremental, practical approach allows policymakers to address market failures as they emerge, rather than trying to solve all potential problems with a one-time top-down design.

Trust is Earned, Not Imposed

Trust in data systems cannot be imposed from the top. While Open Banking was catalysed by regulation, it earned user trust through transparency, security, and gradual adoption, not through a top-down imposition. Similarly, rolling out frameworks sector by sector allows consumers, businesses and citizens to see the benefits and gradually gain confidence in the approach. Attempting to ‘mandate trust’ top-down could backfire, especially in sectors like health, where data privacy concerns are particularly sensitive.

At IB1, our Trust Frameworks are one tool (of many) that can help foster multilateral collaboration, building on Open Banking principles.

Decentralisation Encourages Innovation

The decentralised, minimised and pre-competitive nature of Open Banking has led to its adoption and innovation. A large, centralised approach would have risked locking in certain standards or technological pathways that could hinder future innovations. Allowing sectors to develop their own interoperable standards fosters a more competitive, flexible landscape, where innovations in one sector can inform developments in another, without being dictated by a single framework.

Data governance should aim to balance impact on rights and security while minimising its own footprint and reach

While the idea of a unified, overarching data framework may sound appealing in theory, in practice it would risk being too rigid, slow to adapt, and detached from the unique needs of individual sectors. A more pragmatic approach — building on the lessons of Open Banking and extending these principles to other industries — may offer the best balance of innovation, security, and user control.

Please feel free to comment, or message me (via LinkedIn or directly)

Disclosure: I sat on the MiData energy sector board (mostly tearing my hair out on these points); co-chaired the creation of the Open Banking Standard (through which I learned so many lessons it could fill a book or two); was founding CEO of the Open Data Institute; am co-Chair of the Smart Data Council, and run a non-profit (IB1.org) working on data governance at sector and national scale.

Thanks to Chris, Frank, Paul, Hadley and others for their feedback and inputs.

Useful links

Lessons learned from Gaia-X Data Spaces https://www.sitra.fi/en/articles/eight-lessons-from-building-data-spaces/

ODI on Data Institutions
https://theodi.org/insights/projects/rd-data-institutions/

Icebreaker One on how it implements data sharing
https://ib1.org/what-we-do

Open Banking (implementation entity)
https://openbanking.org.uk

Additional narrative on cities
https://agentgav.medium.com/the-porous-city-92ae986cd43c

“Knowledge Miles: The 695th Lord Mayor’s Lecture Series” is an online webinar series that explores the connections of the Square Mile and its roles as “The World’s Coffeehouse” addressing diverse global challenges.  Ref: https://www.greshamsociety.org/webinar/11680/

Background:

As a Longplayer trustee, my role is to help us think through what our long view might be. Here I will recount some of our thinking and ideas from over the years. 

Longplayer helps us ask many questions about our world and our role in its future. It helps frame questions that are much bigger than us — but they are not ‘infinite’. The time-bound nature of the project leads to many different questions: what might be happening in the future? what might our role be? what might our impact be? how might we communicate across 40 generations (if a generation is 25 years)? what will be happening on its fifth thousand-year loop?

“Don’t let the perfect be the enemy of the good”

Oft-cited to help motivate us to get on with making something.

But what happens when we’re not even at ‘good’ yet?

Sorting out today’s challenges needs consideration of what gets ‘embedded in the system’ so that, instead of the 100x impact we need, we stumble into creating 100x friction in a few years.

There is a bar for good enough.

Watching the unfolding developments in data sharing, with environmental reporting as an exemplar of the size of the challenge, we are still very far away from good.

For example, when I look at the landscape of carbon reporting, I see necessary progress from global standards bodies and regulators to mandate that companies must report. This will impact every business (not just the big ones) because those standards mandate that lenders (e.g. banks) must report on the businesses they are lending to (i.e. pretty much every business).

Note the emphasis on mandate and must. This isn’t going to be optional.

The challenges

The natural response to Big Problems is to say things like:

1. We need to start somewhere, so let’s create some approximations that look like things we already do
2. Let’s start by gathering all the data in one place
3. We know this is a global issue, but we need to create something centralised to start with so we can control it

(1) We need to start somewhere so let’s create some approximations that look like things we already do

Our approximations of investment risk for climate are not yet deeply rooted in reality — we don’t know what the real impact of most of our low-carbon investments will be. If I was to ask you to make a financial investment where I could say your bank balance was ‘about $100 plus or minus $50’ you wouldn’t give me your money. Right now, ‘carbon reporting’ has error bars. Large ones.

People are reporting (because they must) and investing (because they also, must) but there is very scant data that shows that $X invested will link to exactly Y tonnes of climate impact. I don’t (just) mean offsetting, I mean exactly how much will I reduce my footprint by if I invest in changing a process. This is ‘okay’ (for some definitions of okay) for the business-as-usual economy, where market corrections emerge over time.

I’m confident we’re investing in a lower-carbon future, but I’m not confident that we’re investing in a net zero future. With climate, physics doesn’t care about our economy. It matters today, not tomorrow, that we have the right numbers.

(2) Let’s start by gathering all the data in one place

This is still the go-to.

It ‘feels’ safer and easier to control. Everyone can kind of get their heads around ‘building a database’. I’ve, covered, this, many, times.

Except, everyone makes their own. They’re mostly tech-first, not market-first. They don’t scale. They are, despite the marketing hype, not interoperable. It’s [still] a mess. It’s going to get a lot worse.

The challenge is that everyone is happy to collect data as long as they don’t need to share theirs.

And, there are $100M’s (hundreds of millions) being invested in private sector companies to create the next generation of eco-unicorns (whether software solutions, insurance companies or rating agencies). The castle & moat business models that these rely on are already at play, and that’s baked into their design.

(3) We know this is a global issue but we need to create something centralised to start with so we can control it

There are over 300 million companies in the world. Most large companies report over 70% of their carbon footprint is in their supply chains (e.g. SMEs). For lenders, the majority of their carbon footprint is in their customer chain (e.g. SMEs).

How are we going to collect, check, calculate, verify, audit and share this data between organisations, across borders, in a comparable manner — and doesn’t undermine business confidentiality or national security?

Part of the solution is to think differently about the market architecture for a data-enabled future

So, what can we do differently? What things can change, and what things won’t?

I doubt we can change the castle & moat business model, so let’s not try.

What we can do now is change how we design data markets from the outset to connect, not [just] collect.

Doing so would be good news for businesses. If they do this, they’ll save money and get more and better data faster, and at scale.

The good news for everyone else is it can help create a race-to-the-top on insights instead of just relentless monetisation of the raw data, while also putting in place the right protections and security that everyone needs (including the businesses themselves), so everyone can engage.

The key is, in business language, to shift where the line is on ‘pre-competitive’ access to data. To achieve this, we must ensure that the right foundations (robust data governance, technical and legal interoperability, common processes) are in place. The outcome will be that we can commoditise access to data, while enabling the value of data to be paid for or exchanged.

The bar for good should be increased competition on analytics and measurable impact, not on the ‘biggest database’ (big data is so 1990s).

In 2024 good should assume (secure, permission-based, valued) access and unleash the power of our systems (human, financial, technical — including AI) to deliver the changes we need, quickly. Really quickly.

The foundations of this have very little to do with ‘web tech’ (it’s already built), and a lot to do with culture, process, law and policy. If you’d like to learn about how we’re doing in one area, to help automate carbon reporting for every UK SME, see https://ib1.org/perseus

What else?

I may write equivalent pieces on data sharing in other areas we are working on, such as finance, energy, water, transport, buildings, and agriculture.

But the tl;dr is the patterns are all identical.

The challenges as large and (really) nothing to do with web technology, despite all the hype. The likelihood of making outcomes good, or mediocre, are the same in every sector I’ve engaged with.

Of course, with climate, mediocre means ‘absolutely catastrophically terrible’.
(see the solid black line)