Turning your Spotify playlist into a record collection, and why that could be really disruptive

Quick summary

I’ve long wanted to make it easy to buy the music of artists I like, not least because they get better £pay on physical, but when (not if) online services de-list things and hard drives break, they don’t vanish forever.

So, here’s a first experiment: pulling out one of my annual playlists into a ‘catalogue/shop window’ that links to various retailers, CD and Vinyl, and Bandcamp if that exists for the artist.

This is only 2025 for now, while I work out some of the bugs. https://dgen.net/w/playlists/2025.html

Update: now live at https://secondpress.club/p/5lx398

Some reflections (over the past three decades)

This is an example of shifting power dynamics in consumption, engagement and how we can help deliver open markets that work for everyone. I’ve also noticed that the music sector is often an early adopter/trailblazer for broader societal shifts.

Music streaming has solved discovery: a near-infinite library at your fingertips, finding new artists, forgotten classics and new connections has never been easier.

However, there’s an issue buried in that convenience: you don’t really ‘have’ any of it. If a label dispute pulls an album, or you cancel your subscription, your carefully curated listening history evaporates. This can also apply to your ‘downloads’ (even with good hard drives and cloud services, we still lose our digital stuff).

I’ve long thought about what it actually means to have a music collection. Back in the day (1998!) I put Virgin Megastores online: all 60,000 products (which is how may CDs were in a typical store), and designed a service with a colleague, Rick Glanville, where we could sell a subscription to digital downloads and give away a tiny MP3 player as part of the package. We even got Cambridge Electronics to make a little postage-stamp sized player. [yes, these pre-dated the iPod, iPod Shuffle and Spotify by many, many years].

While on that journey it struck me that we were in the process of reducing ‘music’ to a search box, killing most of the actual experience outside of listening, destroying record stores, and as humans we were likely to eventually push back on something so techno-reductionist.

Many of the things we enjoy about music listening are physical: this has played out in the long-term success of stores like Rough Trade, and the resurgence of vinyl (even cassettes are back!). At the time, as a bit of rebellion, I also released my own music as a 19kg solid granite MP3 player and radio transmitter.

In one of my many roles as CEO of Consolidated Independent, we helped get over 20% of the world’s music online: millions of tracks from thousands of labels distributed to hundreds of retailers and services. Some of the labels were terrified of the web destroying their business due to piracy, but the whole system had to embrace it, including sorting out commercial realities. Sadly some of those realities massively skewed market value to ‘the big guys’, including ludicrous statements from some (let’s say ‘commercial’) CEOs that if artists wanted more money they should increase their output (as if art and soup cans are the same thing ).

Don’t get me wrong, I love the fact we have vast access to new music, and listen to a very diverse range of music. And, it creates different challenges for us as consumers and music fans. Now we can have more agency in balancing out the way the music market works – the cathedrals are not going anywhere, but the bazaar has some new spaces.

What we can now do

One of the things I missed when listening was album art, and a few years back I made a 12″ sized screen to pull down and display whatever I’m playing on Spotify (AI helped me code this, and I would never have got around to it without that help).

Building things just for me is fun, and…

Interestingly, with just a few AI prompts and some lightweight code, I created a tool that takes my Spotify playlist, cross-references every album against Discogs, and produced a personalised catalogue.

Based on the individual tracks I’d added to my annual playlist (which I have going back 8 years now), it looks up the album that it was on. It then works out which are available on CD and Vinyl (or digital–only), where to find them, and who sells them. And, not ‘just Amazon’, but others: Rough Trade, HMV and directly linking back to the artist’s Bandcamp. This is just the start of and idea, and took less than half a day to create it as a prototype.

This makes me wonder about a quiet promise of AI that doesn’t get talked about so much: enabling each of us to build the world we’d like. This will disrupt the big platforms (Spotify, Amazon, Apple) who have invested billions in making their aggregated experience frictionless and hard to leave. They’re using AI to make them more personalised and ‘sticky’. We can do the same for ourselves.

The same AI tools they’re racing to build are equally available to you and me. We can now spin up personalised services—in an afternoon—that would have taken a dedicated development team a decade ago.

So, this is a prototype of a custom, personalised catalogue and record-buying assistant that can be built around your taste, routing money to your preferred retailers that you can share with your friends. It is a small but real act of reclaiming your relationship with artists, and their music, from the big platforms.

We can help bridge the gaps and connect the smaller services together (back to the original vision of the web).

And, looking forward, we can start blending, sharing, cross-connecting using federated services like Mastodon and then connecting people, places, gigs, merch, in a way that could actually help everyone.

Isn’t that exciting!?

We’ve never had more data. Some think it’s the new oil, the new gold, the new soil, something of a revolution. Is it? Or are we just still humans stumbling around in the dark?

Earth observation satellites scan our planet, sensors (there are more sensors on Earth than there are people) track the movements of people and things, energy flows, air quality, water usage, and behaviours. Our global financial systems process billions of transactions a day. Most organisations, public and private, are drowning in dashboards, APIs, spreadsheets, data lakes, clouds, compute, applications, and now ‘ai’.

“Every age thinks it’s the modern age…but this one really is”
Dreams Rewired, Manu Luksch

And yet, when we need to make decisions that matter — how to finance a green retrofit, where to send emergency flood support, or how to measure progress toward net zero, the data we need is often hard to access or nowhere to be found. Or it exists, but it’s in the wrong format, in the wrong place, scattered across organisations and systems, under the wrong licence, or wrapped around the wrong market incentives, or just poorly governed and hard to trust.

Why is this important to our future?

We’re in an era where our financial economy is demanding ‘quality’ data from the real economy. Our financial systems have developed over centuries to measure, track, verify, use and report on the rate of change of ‘one dimension’: money. As we continue the shift to make our environment measurable, we must get the ‘exchange rate’ right between financial investment and environmental outcomes (climate, nature, air quality, water, biodiversity, and so on).

Right now the ‘exchange rates’ are all over the place and, importantly, as we step forward we need to measure the rate of change of many parameters, not just one: we need a step-change in how we think about ‘data’ and data sharing.

It’s hard to understate quite how far away the real economy is from generating financial-grade data

We’re still mostly in the 1970s. Environmental disclosures still rely on highly manual processes, despite widespread digitisation and hundreds of application vendors in the market, it’s still mostly clipboards and spreadsheets or, for 99% of the world, nothing at all. This is not a problem of data scarcity, it’s a problem of incentives, structure, and alignment.

We’ve built systems that are optimised for accumulation, not coordination. We have engineered data pipelines that extract value, but not the wiring to deliver reciprocity (shared benefits).

In the language of infrastructure: we’ve laid the cables, but we’ve neglected the grid

We are told to “make data-driven decisions” while simultaneously being denied the means to do so at scale. Meanwhile, we waste more time (and money, opportunity and emissions) trying to reconcile, reformat, clean, or even find the data than the actual decisions that we are trying to make.

The result is a kind of systemic gaslighting: we pretend that the data is ‘there’ when what we really mean is that it exists … but existing is not the same as usable, and very far away from trusted. Just because something is stored on a server doesn’t mean it’s infrastructure. Just because it’s digital doesn’t mean it’s useful. Just because it’s available doesn’t mean it’s accessible.

McKinsey (The data-driven enterprise of 2025) suggests that data professionals still spend up to 80% of their time preparing data rather than using it. This is the same number I have from over 25 years ago.

So where do we go from here?

Data is everywhere but until it flows to where it’s needed (when it’s needed, and with the right context) we’re not building a digital future we’re just building more digital noise

We need to stop pretending that technology (including ai) alone will solve this. We must stop framing data as a ‘commodity’ to be owned and sweated, and start treating it as infrastructure to be governed.

We must design for connection not monopoly control, for shared purpose not private hoarding. We must invest in the ‘boring’ bits: governance, legal standards, licensing, permission/consent, metadata, interoperability.

This is the stuff that doesn’t make the headlines, but makes everything else work.

The National Data Library can deliver its vision,
but it must pivot to build trust in the age of today’s web

This post builds on observations and conversations across government and industry (and internationally) since my previous post on “How can we find the Goldilocks Zone of our National Data Infrastructure?”.

When we’re building into a future where technology has such clear benefits it is often hard to be heard above the noise of the possible. And yet I believe we must also ‘grasp the nettle’ regarding equally possible negative or unintended consequences. While we must not “stifle innovation and growth”, we also must ensure we don’t scuttle our ship in the process.

The first misconception I want to address is that data is not a technology.

Data is used to represent information, analysis, insights, opinions, facts, responsibilities and decisions. It carries many different kinds of value, as well as material rights and material risks. Data is used to find things out and to make decisions. To look into its past, data is the plural of datum, from Latin, meaning something given.

I labour this point to make clear that it is not software (e.g. AI is software that uses and produces data, as are all algorithms codified in software) and it is also not hardware. A lot of debate about data (and AI in particular) seem to conflate these facts.

Treating data as if it were software is comparing apples and carts.

Context

Having been a tech entrepreneur for 30 years, I’ve sat in meetings with Silicon Valley investors where the “move fast and break things, then ask for forgiveness” mantra has been very present. Innovators love to build; building is fun and very rewarding — both intellectually and, sometimes, financially.

It’s common to worry about the consequences later, even when ‘later’ can be measured in decades. Those raising alarm bells are often badged as ‘activists’. This is a label I’ve never felt helpful: we can draw parallels with the past when ‘health risks of smoking’, through to catastrophic climate change, risked people being boxed as ‘alarmist’ instead of ‘anxious-with-cause’.

I’ve always advocated for fair-weather innovation, while ensuring that we build in measures for foul weather. In the age of the web, commercial (and political) ambition can often be seduced by the fair-weather arguments, while policy is rarely able to move at the speed required to keep pace and safeguard us from foul-weather conditions.

A lot has changed since my previous posts on data infrastructure. We must now, right now, be proactive and preemptive to the fact that our threat model has changed (due to shifting geopolitical and technocratic-utopian positions). Foul weather is closer than we may have wished.

Trust isn’t just a buzzword; it underpins our society and, in a digital age, this means addressing our data infrastructure, ensuring our safety, de-risking innovation and enabling growth in turbulent times. Trust helps us build all of these, for the long term.

Going from vision to implementation is hard

There is a natural tendency towards overreach in most systems — this is a very human attribute. When trying to innovate, our bureaucratic systems can often feel frustrating (good entrepreneurs — both inside and outside of government — know how to navigate around, or sometimes ‘through’, such systems).

Today, in 2025, with data and the web and AI, the pace and scale of impact outpaces and out-scales any other time in history. We can create a critical mass, or a critical mess.

To that end, I believe we must take steps to better understand and balance what is in front of us. When it comes to enabling data sharing, whether in finance, health, transport, energy, property, water, or across all research, we need to first take stock of something that is very hard to build and yet can be lost in the blink of an eye: trust.

The National Data Library (NDL) represents significant intent from the UK Government to do data sharing differently, and we (citizens, consumers, businesses, sectors, markets, and the state) can all benefit from this. However, implementation really matters.

Icebreaker One has already made a strong case (selected for publication by the Wellcome Trust) for the National Data Library to:

1. Focus on defined users and use cases rather than datasets, just like any effective data infrastructure.
2. Begin life as a simple, decentralised version, that would curate and improve the discovery of, and timely access to, public sector data for research.
3. Experiment with more complex architectures for harmonising access to data drawn from across multiple government departments.

Cutting across all of this, the NDL must be built, from its core, in ways that engender trust. Whether consumer distrust, competitive interest or geopolitical threats, we must have in place the people, processes and protections to cement trust between businesses (b2b), with consumers (b2c), and with our citizens.

The NDL can still maintain its grand vision to improve outcomes for all of us, connecting data, rapidly unlocking insights, and leading the way.

More than its technical architecture, the nature of its funding and staffing will shape what it can achieve. I’d argue we can do more together, and move faster, with common principles and parallel actions than a large centralised effort. Not only should the data never be centralised (other than for the limited use cases that require that), but neither should the absolute control.

We can go far together if we each do one thing well.

We can bind our work, together, with common principles and practices. There will be no ‘one hub’ — if you are a hub in a collection of hubs, you’re a node. Even a National Library is one node in a network of knowledge. There isn’t only one search engine in the world, nor will there be one AI.

We must also approach, with a clear perspective, that one-time broad-based consent is just one limited use case. It may apply to certain, relatively static, data but a different approach is required for highly variable data: data is rarely static. The conditions on which that ‘one-time’ consent, as we have also seen, will also change based on lived experience (e.g. changes in corporate and political governance).

In a world of rapid change, we must pivot rapidly but
never forget the foundations of trust beneath our feet

Building on the Smart Data model, there are known ways to apply decentralised, user-controlled, real-time, granular consent. There are dynamic solutions in place today which enable peer-to-peer data sharing via secure APIs. Such systems can include immediate revocation (by the user, or by a neutral governing entity/representative), and they are designed and built to enable cross-sector interoperability.

\Equally, such systems will not (and should not) be the approach for all use cases. What works will depend on the purpose (which will span personal, business, national, research as well as small, medium and large aggregate/collective datasets). What matters is how they are governed.

This is where implementation matters. If we over-centralise and over-reach in our ambition we run the risk of the litany of failed ‘IT projects’ which have cost the taxpayer billions.

I see many projects, even some with the word ‘governance’ in their titles, take a technology-led approach rather than a governance-led approach: data is not a technology—it enables insights about its subjects and those subjects have rights.

Recommendations

Without trust in the systems upon which we run our lives we will, at best, stifle innovation, productivity and growth. At worst, we risk undermining far more.

The good news is that we can run, at pace, fast but not foolhardy, to build the data infrastructure we need for today and scale for all our tomorrows, to deliver trust, ensure our safety and unlock the value of data for everyone.

Phase 1: Immediate reorientation on purpose and language (0–6 Months)

Step 1: Transition the “Library Card” model to a Trust-based, scalable model

• Pivot from the concept of ‘library’ to a ‘web of interoperability and trust’ (WIT)

For example, announce an evolution from the “National Data Library” to something like a “Web of Interoperability and Trust” (WIT) or “National Data Web” or “Trusted Data Web”, which would align with work already in place in the financial sector and give more scope to connecting existing initiatives, including Smart Data, without being specific about technology preferences.

• Define six exemplar use cases for pilot delivery in 2025

Pick six lighthouse use cases across sectors, assign to different entities for delivery, and shift governance to the model that best enables them (e.g. from “pre-approved datasets” to a federated trust framework where applicable).

Adopt an API-first architecture to allow controlled, on-demand data sharing. Use them to produce quantitative evidence as to the pros and cons of the approach.

Step 2: Introduce an adaptive Smart Data consent framework (6–12 months)

• Develop and mandate common principles

Common principles should include application for ‘Data Consent/Permission’ for consumers, SMEs, researchers, businesses and government departments to view, grant, and revoke access to datasets. These should be evidenced through use cases with specific cost benefit and threat analysis.

• Require and mandate data best practices

Require that all third-party users (e.g. AI developers, financial institutions) justify the minimum data required per use case. Mandate that every material access request is logged, auditable, and policy-compliant (e.g. GDPR, Data Bill, and related policies). Where applicable, develop formal Schemes that codify multilateral contracts and enforce them. Note that Schemes can also be industry-led and voluntary, with appropriate governance: political signalling can be an efficient way to accelerate implementation.

Phase 2: Build systemic trust (6–18 Months)

Step 1: Shift to an API-enabled Data Access Model (including trust frameworks) based on user needs

In this instance ‘API-driven’ is an architectural choice aligned with the web, not a specific technology, that enables both interoperability and controls to be implemented. Other use cases won’t need ‘APIs’ (e.g. a CSV file of Open Data), but where security and access control are required, any technical solution must be able to operate at web scale. Key is the development of the governance mechanisms that help implement trust frameworks.

• Support development of sandboxes

Support the rapid, cross-sector creation of sandbox environments with political and financial support where required.

• Support open communication, transparency, and build collaboration and reciprocity with a broad stakeholder group. Address business, public and government skills and knowledge gaps.

Embed open communications, transparency and reciprocity into all programmes. Demonstrate to businesses and consumers that technologies such as AI can deliver value without undermining trust. This can be achieved by facilitating co-design across the value chain (e.g. govt, industry and end users, whether b2b or b2c)

• Create systemic links with digital identity developments

Incorporate systems-level authentication (e.g. OAuth-based which is already ubiquitous) — data access should be controlled via digital identity-linked permissions for companies and research organisations. This is aligned with the Data Bill and FCA approaches. Digital identity is a huge topic which I do not want to dive into here, but I do assume that it will (also) be federated (there should and will be no ‘one place’ for it).

Step 2: Maximise privacy enhancement for people and businesses (e.g. SMEs)

• Create mandates and incentives for privacy enhancement

Broaden and define Schemes to enable ‘Access Rulesets’ and enforce data-minimisation principles for citizens, consumers and businesses (including b2b SME data sharing), so datasets can be queried without bulk transfers (e.g. an app can check fact without revealing the underlying data, e.g. “are they over 18 — yes” not “what’s their birthdate?”)

Phase 3: National Smart Data Infrastructure (18–36 Months)

(yes, this may feel ambitious: why shouldn’t it be this rapid?)

Step 1: Legislate Data Interoperability

• Mandate interoperable systems where consent/permission or access control is required

Draft legislation ensuring every public-sector dataset must be ‘API-accessible’ under standardised, interoperable formats. As noted, not all public-sector data needs an API. In fact Open Data should be made available in any way that makes it discoverable and usable, and even CSV files can be catalouged by other systems and made ‘API accessible’. However, where some form of access control is needed, secure APIs are the standard that works for the web (e.g. Open Banking has 12 million monthly active users across the UK).

Also note that interoperability at the API level does not mean that all internal systems must do the same thing in the same way: data can be repurposed and presented without rebuilding entire estates (this will create an incentive for doing so over time, but it doesn’t need to happen first). Interoperability and decentralisation can work together, with appropriate governance, to enhance competition.

• Mandate open access and empower regulators to enforce compliance

Mandate open access to government datasets where possible, while protecting sensitive data via consent-based permissions, and democratically set rules. Pick regulators or code bodies to oversee the development of the specific rules and compliance required, including modes of redress. Note that open access and Open Data are different things.

Step 2: Scale the Model Beyond Government Data

• Support the development of connected demonstrators and pilots

Launch pilots with commercial ecosystems to enable interoperability (e.g. real estate, sustainability finance, public health). This includes cross-border and global data-sharing ecosystems (e.g. finance, goods).

• Enable interoperability beyond the UK, within the desired framework

Develop cross-border data-sharing agreements (‘Schemes’). Ensure interoperability with international frameworks (e.g. EU Digital Identity Wallet, OECD AI Data Principles and related emerging programmes). Ensure every data use and dataset are AI-ready and privacy-compliant via automated monitoring and governance.

Summary

Strategic Benefits

• Accelerates mission-driven government: Enables implementation of the government’s five missions via trusted, scalable data access.
• Supports economic growth: Unlocks interoperability for data-driven innovation across UK sectors.
• Protects society through accountable systems: Ensures that data-enabled systems are governed with clear and enforceable safeguards. Enables a coherent approach to data sovereignty across our economy and society, to minimise the risks of misuse and systemic harms in a rapidly evolving landscape.
• Future-proofs data governance: Aligns with existing data sharing frameworks, and with evolving standards for AI training, licensing, and consent. Licensing will be a cornerstone in all future data work whether AI-linked or not.
• Embeds assurability at scale: Operationalises assurability across domains.

Five WIT principles (for discussion)

1. Interoperability by design: Common standards, APIs, and metadata frameworks to ensure frictionless cross-sector data use.
2. Trust infrastructure: Consent-based access, verified credentials, and relevant-time provenance tracking.
3. Federated governance: Sectoral autonomy with shared rules, coordinated through a Web of Interoperability and Trust.
4. User empowerment: Individuals and organisations have control their data-sharing preferences. Strong governance-led approached apply to aggregate data, with the option to revoke.
5. Composable architecture: Modular, user needs based and cross-sector enabling use cases, using sharable registries, vocabularies, and open interfaces.

NB: This post contains (as-always) personal opinions and thinking-in-progress (strong opinions, weakly held) as I navigate through the maze(s) of data governance. In amongst many experiences, I sat on the MiData energy sector board; co-chaired the creation of the Open Banking Standard; was founding CEO of the Open Data Institute; co-Chaired of the Smart Data Council. I run a non-profit (IB1.org) working on data governance at sector, national and international scale.

With thanks to Jack, Peter, Chris, Frank, Jeni, Simon, Julia, Kathryn, and many others who have helped shape my thinking on this.

(please note my disclosures at the end of this post)

This post contains (as-always) personal opinions and thinking-in-progress (strong opinions, weakly held) as I navigate through the maze(s) of data governance. Persistent in my questions are how we apply the Goldilocks principle of governance vs innovation: to remain flexible and avoid brittleness.

Given the conversations I’m in at the moment I believe it very important to have as much debate as openly as possible, as this impacts everyone.

Here’s a question:

How should the UK implement its National Data Infrastructure so that it provides consistent control, sharing, and security for data, much like other public infrastructures such as roads and utilities?

“Everything should be as simple as it can be, but not simpler”
[often attributed to Einstein]

Having initiated ‘data as infrastructure’ at a political level in the UK in 2013, I’ve been watching and learning on this for a long time. We’ve made remarkable progress in some areas, less so in others.

Let’s explore, starting with some lessons learned.

One immediate reflection is that as over 60 countries moved to copy the Open Banking Standard, those who took only ‘parts’ of the system tended to experience less ideal outcomes (to the point that they often add them back in later to help course-correct — most notable are those who thought this was ‘just a tech problem’ and only took the open API principles).

We can look to past UK data initiatives like MiData and Open Banking and potentially infer that voluntary-only initiatives and centralisation do not work, and that the decentralised nature of Open Banking was more successful. However, these are shallow reflections.

We can point to three factors beyond centralised or voluntary which are dependent on the use case:

1. Market incentives: there must be an economic argument that policy can then amplify or mandate. If there is no financial incentive, there will be no movement (regardless of central vs decentralised, mandate or not — mandates are also often ignored). Midata was mostly tech-led, Open Banking was use-case led.
2. Removal of friction: Removing transactional friction may seem like something everyone wants, but not if your current business model relies on it. There must be “something in it” for everyone, or at least a path to cost reduction or a new business model. Removing friction can help everyone go together: this is never a ‘technology problem’ (e.g. absence of a data ontology).
3. Mandates: if the friction is ‘too high’, regulatory intervention is necessary to mandate participation and move the market. However, if the market can demonstrate self-initiation, then ‘endorsement’ from government can be sufficient to drive impact. Value creation must be visible in both cases.

I believe that we need to embrace these factors in our systems design.

We need to:

1. work out where the lines are drawn between central, pre-competitive and competitive areas;
2. ask what the role of regulation is, and at what granularity
3. query how things can be implemented in a way that humans (and machines) can and want to adopt
4. ensure those governing, operating and participating can be helped in their assessments of compatibility and interoperability
5. work out how to prioritise and create clear (and stable) roadmaps that enable investment to be made

I believe that we can build on the success of sector-specific approaches:

The Open Banking model’s success lies precisely in its sector-specific, incremental approach. It focused on the unique needs of banking, where security, standardisation, and customer trust were paramount.

Copying the Open Banking model into other sectors (e.g., energy, transport, health) isn’t about duplication but about applying tested frameworks tailored to each industry.

Without considering the market incentives for participation we risk techno-utopian thinking

Instead of trying to create a complex, overarching architecture that risks being too broad, a sectoral approach allows for flexibility and adaptation to industry-specific challenges. This carries a risk of fragmentation, and equally a potential to mandate cross-sector interoperability without defining exactly what this means in detail for everyone. The risk of push back on over-reach is material and catastrophic (in terms of adoption, if it fails). Further, market-incentives will be market-specific.

Avoiding Over-Standardisation

National Data Architecture risks over-standardisation and could stifle innovation instead of enhancing it. Different sectors have wildly varying levels of data maturity, hugely variable data governance needs and challenges, and a one-size-fits-all approach is highly likely to impose unnecessary constraints on sectors that are not as ready for them.

For example, imposing the level of data governance on the industrial sector that is required in the financial sector is (today) not realistic. We are, however, on the frontline of that journey with ESG reporting today, and it highlights the complexity of that real-financial sector bridging. Incremental improvements, like extending Open Banking principles to energy or transport can allow industries to evolve organically while keeping governance controls and security as a priority.

Pragmatism of the Incremental Approach

The incremental, domain-by-domain approach may seem less ambitious, but it offers a practical pathway to innovation that can be adopted today. Trying to design a top-down, comprehensive national architecture from the outset risks paralysis by analysis and over-planning. Learning from Open Banking and rolling out smart data initiatives sector-by-sector allows for continuous improvement and responsiveness to real-world feedback, which may be more effective than a sweeping, coordinated architecture.

Interoperability Requires Industry Engagement

Interoperability relies on industry engagement. This is best fostered through practical, sector-specific initiatives like Open Banking. Imposing an overarching system from above risks of alienating key industry stakeholders, who may resist changes that threaten their business models (this happens in every sector already, today). Open Banking’s success came from the balance of regulatory oversight and industry cooperation, a formula that can be replicated in other sectors.

Market Failures Can Be Addressed Incrementally

There are (always) risks of market failures such as data hoarding (castle & moat is still the prevalent investment model for data businesses). These issues should be tackled incrementally through sector-specific mandates rather than a top-down architecture, but can be accelerated with top-down principles.

The competition fostered by Open Banking and similar initiatives already shows promise in tackling data monopolies. An incremental, practical approach allows policymakers to address market failures as they emerge, rather than trying to solve all potential problems with a one-time top-down design.

Trust is Earned, Not Imposed

Trust in data systems cannot be imposed from the top. While Open Banking was catalysed by regulation, it earned user trust through transparency, security, and gradual adoption, not through a top-down imposition. Similarly, rolling out frameworks sector by sector allows consumers, businesses and citizens to see the benefits and gradually gain confidence in the approach. Attempting to ‘mandate trust’ top-down could backfire, especially in sectors like health, where data privacy concerns are particularly sensitive.

At IB1, our Trust Frameworks are one tool (of many) that can help foster multilateral collaboration, building on Open Banking principles.

Decentralisation Encourages Innovation

The decentralised, minimised and pre-competitive nature of Open Banking has led to its adoption and innovation. A large, centralised approach would have risked locking in certain standards or technological pathways that could hinder future innovations. Allowing sectors to develop their own interoperable standards fosters a more competitive, flexible landscape, where innovations in one sector can inform developments in another, without being dictated by a single framework.

Data governance should aim to balance impact on rights and security while minimising its own footprint and reach

While the idea of a unified, overarching data framework may sound appealing in theory, in practice it would risk being too rigid, slow to adapt, and detached from the unique needs of individual sectors. A more pragmatic approach — building on the lessons of Open Banking and extending these principles to other industries — may offer the best balance of innovation, security, and user control.

Please feel free to comment, or message me (via LinkedIn or directly)

Disclosure: I sat on the MiData energy sector board (mostly tearing my hair out on these points); co-chaired the creation of the Open Banking Standard (through which I learned so many lessons it could fill a book or two); was founding CEO of the Open Data Institute; am co-Chair of the Smart Data Council, and run a non-profit (IB1.org) working on data governance at sector and national scale.

Thanks to Chris, Frank, Paul, Hadley and others for their feedback and inputs.

Useful links

Lessons learned from Gaia-X Data Spaces https://www.sitra.fi/en/articles/eight-lessons-from-building-data-spaces/

ODI on Data Institutions
https://theodi.org/insights/projects/rd-data-institutions/

Icebreaker One on how it implements data sharing
https://ib1.org/what-we-do

Open Banking (implementation entity)
https://openbanking.org.uk

Additional narrative on cities
https://agentgav.medium.com/the-porous-city-92ae986cd43c

This insightful report, from the Royal Societies DELVE initiative, “Data Readiness: Lessons from an Emergency” delves into the crucial role of data in emergency situations and highlights valuable lessons learned. Drawing from real-life experiences, it emphasises the importance of being prepared to leverage data effectively when faced with unexpected challenges.

The initiative has gathered experience with emergency situations and stresses the significance of having a data-driven approach. It highlights the need for organisations to recognise the value of data and incorporate it into their decision-making processes. By doing so, organisations can gain critical insights that help them respond swiftly and effectively.

The report outlines key factors that contribute to successful data readiness, specifically

• the significance of investing in data infrastructure
• developing a holistic approach including tools, systems, and processes, to ensure the availability, accessibility, and quality of data
• that data readiness is not solely a technical issue but also requires organisational commitment and cultural transformation.

It stresses the importance of collaboration and knowledge sharing between different stakeholders involved in emergency response efforts, highlights the need for establishing partnerships and creating platforms that facilitate data sharing and interoperability: that through open collaboration, organisations can harness the power of collective intelligence and make informed decisions based on comprehensive and up-to-date information.

It touches upon the ethical considerations surrounding data usage in emergency situations, emphasising the need for transparency, privacy protection, and responsible data governance. By adhering to ethical guidelines, organisations can build trust with the public and ensure that data is used for the benefit of affected communities.

It offers valuable insights into achieving data readiness in emergency scenarios and underscores the significance of investing in data infrastructure, fostering collaboration, and upholding ethical standards.

Summary of key elements:

1. Importance of prepared data systems: Having well-prepared data systems is crucial for effective emergency response. It enables timely access to relevant data, which is vital for decision-making and coordination.
2. Data availability: Ensuring data availability is essential during emergencies. It involves having data collection mechanisms in place and establishing protocols for data sharing among relevant stakeholders.
3. Data accessibility: Easy and secure access to data is critical for timely decision-making. This requires establishing appropriate data access controls, and considering data privacy and security concerns.
4. Data quality: Data quality is essential for reliable analysis and decision-making. Implementing data quality assurance measures and validation processes can help maintain accurate and reliable data during emergency situations.
5. Interoperability: Ensuring interoperability among different data systems and platforms is crucial for effective coordination among various entities involved in emergency response. This facilitates seamless data exchange and collaboration.
6. Standardised approaches: Implementing standardised approaches for data collection, sharing, and analysis promotes consistency and efficiency in emergency response efforts. This includes adopting common data formats, protocols, and procedures.
7. Coordination and collaboration: Effective coordination and collaboration among stakeholders involved in emergency response are essential. This involves sharing data, aligning efforts, and establishing clear communication channels to maximise the impact of data-driven decision-making.

This very much echoes our experiences in developing data infrastructure and the data triage process.

We are in the midst of two exponential changes: our environmental crises (e.g. climate change) and the web of data. I created Icebreaker One to try and catalyse ‘green swan’ moments: to help break through the complete stalemate of private-public-sector interactions that we are stuck in.

“Too good to be true”

This comment has been made to me many times throughout my career. Leading mission-driven organisations, it seems, sometimes is beyond the realms of believability with people who don’t, can’t or won’t believe that there might be a set of incentives beyond financial and/or power.

And, I’ve never felt the need to justify why or convince anyone who has such doubts. Aside from being time-consuming, it is also defensive and can even undermine the point. Ultimately it’s better to show by doing.

However, it’s not unwarranted, given the rapacious nature of business and the desire of certain organisations and/or individuals to ‘win’ (sometimes at almost any cost).

It is, therefore, also to clearly set out the vision, mission and values of why an organisation is doing something and to make clear its culture, principles and purpose.

This post is for three audiences:

• those who are trying to build open systems and ‘fight’ against the underlying capitalist system we’re all in;
• for those who are suspicious of the motives of non-commercial actors; and,
• for those who believe that ‘the market will fix it’.

I’m going to apply three lenses to these opinions.

In the first section, I touch on ‘commercial thinking’ and why it must be embraced (e.g. the entire web).

In the second section, I reflect on lessons from the ‘free’ communities, some of the benefits (e.g. the entire web) and some of the challenges.

In the third section, I try and narrate some of the experiences I’ve had in trying to bring things together, in particular how government can enable or kill deep innovation, and the impossible challenges of the policymaker.

I then close with why I created Icebreaker One, based on some of the lessons learned over the last 25+ years, and what this means for the next 25+ years.

1. Commercial-first

I’ve spent many years [cough, decades] helping create commercial companies, some bootstrapped, some venture-backed, across many sectors from media (e.g. Virgin Media) to finance (e.g. Asset TV) to environment (e.g. AMEE) to venture funds (e.g. Two Magnolias).

Starting with the obvious, we are in a financial economy-based culture. That means that our metrics of growth, the interactions between money, the private sector and public goods have a complex dance. Companies, and their directors, are legally bound to increase value to shareholders. We might not always like it, but it is a fact — and one that enables us all to satisfy our basic needs.

And, because of those facts, they are — by definition — not mandated to look after the public good. The incentives of the system drive behaviours (even if they come with positive intent) towards bad outcomes. And at the same time, the economic engine at the heart of this drives incredible progress and innovation.

The purpose of government is to ensure that the citizens who elected them to run our societies, and the economies upon which they are based, create the right balances. They are designed (however well or poorly they are implemented) to seek the best outcome for all of us.

And, the third sector (NGOs, charities, non-profits) are there to fill the gaps: to hold both government and industry accountable when they, inevitably, don’t get it right. I say inevitably here because all of this is run by humans, and humans are always messy.

When things go well, we put people on the moon, create global vaccines and educate billions. When they don’t we end up with all manner of bad, from monopolies to mass surveillance to wars.

2. Free-first

I’ve spent many years [cough, decades] working with ‘open’ communities: ranging from technical (free and open source software) and data (open data) to structural interventions (open banking, open energy), to co-creating non-profits, and sitting as a trustee on charities.

One, very human observation, and one of the reasons I love working with young (in spirit, not just age!) people is that we either don’t know something isn’t meant to be possible, or ignore that it might not be, and just get on with trying to build the future we want to be in.

In early free software communities (e.g. early web) there was a highly utopian (and highly energising) vision that we could make something that was for everyone, that would be free, that would be radically transparent, and vastly inclusive. I often view the web as a social response to globalisation: as communities and family units were being dismantled by mobility, the web helped us stay connected. And, as a foundation for maintaining social connections with friends and family, there was/is — of course — an imperative that such communication should be open, free and inclusive.

Many communities felt strongly that even talking about money would undermine efforts, and actively shunned, or struggled with, ideas about how to fund such developments, assuming that there were enough people to maintain things without needing money to keep things pure. And, in some cases, this works. However, in many, I watched a repeat pattern of such communities reaching a ‘certain life stage’ and wanting to settle down and have a family. Such priority shifts can destroy free approaches if there are no new people who can pick up the pieces.

This is important because it underpins a deep-rooted culture that can shun commercialisation or even the idea of commercialisation.

One of my lessons learned from being (briefly) on the board of the Open Knowledge Foundation was how negatively that can play out. Not long after joining the OKF board, I was appointed as the founding CEO of the Open Data Institute. With that role, which included a commitment of £10M of government funding over five years, we suddenly had a commercial remit for the Open Data community. Part of the rationale for the ODI was to stimulate demand, incubate startups and engage directly with the private sector to unlock the value of open data. And this was wholly correct: you don’t create markets by priming only one side. The creation of open data wasn’t enough to produce users that would make open data work at scale. Supply and demand make markets.

However, the OKF board at the time felt that having me on their board would be a conflict of interest: that the ODI would be immediately competing with them on the small pots of money that existed at the time. While I could appreciate that, for me the point was to raise the tide to float many boats. That having a tight cohesion and strategic approach together, with as many of the other small organisations out there as we could join with, would create something investable by government and industry alike. That with more funding, we could scale resources to develop and deliver the mission.

After many conversations, however, we couldn’t reach alignment on that, and I stepped down.

This was a pity. Two years later I was on stage with the President of the World Bank announcing the Data Revolution. Afterwards, a number of organisations sat down to work out how to unlock a $300M fund for open data globally ($70M had already been secured). If we had had such a strong coalition at that point, I believe we would have had sufficient political support to make that happen, and unlock more investment into the open data landscape in one go than in its lifetime combined. I don’t say this lightly: I was tasked by the Prime Minister at the time to try and land it, the President of the ODI invented the web, it’s hard to imagine a more ‘open goal’. And yet, we’d not done the groundwork to build a broad-enough church.

Change. Takes. Time. Tenacity. Connections. Luck.

To go far, we need to go together.

3. The balance

I co-created a place many years [cough, over a decade ago] in which I live, where we mix the models: co-operatively owned infrastructure where there is a blend of private (homes) and community (common infrastructure) ownership. Individuals can do what they want in their homes, buy and sell them, but the common infrastructure has common governance and is actively managed and enforced. Even in a small community (of ~50 people) this can be challenging: I often find it is a microcosm of the broader challenges we all face at larger scales.

One of the core challenges in governance is fear. For policymakers, charities, or even small communities, the idea of changing something is worrying. “What if” gets in the way of progress. This can translate into infinite consultations, groupthink, or even just a lot of meetings and circular conversations that can eventually destroy any enthusiasm to do anything.

Startups are, in the Valley model, told to move fast and break things. To worry about what the rules are later (if they have enough money to be sued). I’m quoting from a board meeting in one of my own companies.

For a policymaker, at national levels, this is incredibly hard work. How to make a decision that, if taken, could negatively impact millions, or create accidental monopolies? And if not taken…well, the same consequences are entirely possible.

The balance is then between the reckless, the foolhardy and the cautious, decision paralysis and continuous debate about the what-ifs and the buts and the maybes.

And we never know until we try. We learn by making. And trying includes making mistakes.

One difference between policymaker and business is that a business has to act now. There is no ‘later’ because payroll needs to be met, competition needs to be kept at bay and investors are hungry. Snooze and businesses lose.

That doesn’t mean policymakers don’t act, but it’s harder. Investors often want to see a perfectly investable (aka derisked) business before investing (product complete, market established, team in place, profit showing) which makes you wonder why it even needs any investment, then a policymaker has a much harder time. An investor might ‘just’ lose their money. A policymaker might create deeper trouble and, while businesses can ‘ignore’ other people’s opinions, policymakers are the recipients of everyone’s opinion, all the time. By design.

So the challenge is how to strike the right balance between ‘now’ vs ’later’.

We are lucky in the UK to have the most remarkable public innovation funding. Billions are spent on research, pilots, and demonstrators, to help move ideas forward from concept to reality. Notwithstanding its challenges (there are vast funding gaps in the ‘scale up’ zone) these can bring together stakeholders in very meaningful ways.

Our experience in developing Open Energy (under the non-profit, Icebreaker One) has seen just that. We convened literally hundreds of organisations, small and large, national companies, government and regulators. Its Steering Group has convened over three years to discuss how to best take it forward. Commercial organisations have lent in as paying members. And ‘government’ has lent in, and away again, and back, depending on who was in power (‘challenging times’) and what the risk appetite was at the time.

So, to keep this very anonymous and not ‘pointy’, here are some quotes I’ve had from government(s) over the last 15+ years:

“We absolutely see the value in this and will add it to the budget next year”

(in some cases they did, in others, they did not)

“Government can’t fund private sector companies”

(this is simply untrue, the ODI, all the Catapults and many other initiatives have been. Other examples are left as an exercise for the reader)

“We forgot [to put the funding in the budget]”

(note that in both the public and private sectors, teams often change every nine months. I feel sometimes that I’ve briefed a new set of civil servants every nine months for over ten years)

“The private sector should fund it”

(both from people in government and the private sector)

“Government should fund it”

(both from people in government and the private sector)

“He’ll move on in 9–14 months, this is just a stepping stone”

(said to one of the board of the ODI when I joined, somewhat missing the point that being mission-driven is, for some people, more important than money, status and/or power)

“That sounds too good to be true”

(that I’d set up a non-profit company for the benefit of both the public good and could unlock billions for the private sector without wanting to become a billionaire in the process. Someone also followed up with, “why don’t you pivot to a VC-backed company”, which misses the point — by about a lightyear)

So what?

All this is a long preamble to the fact that I think we are at a particular moment, in an Overton Window, that could change the future of our economy and our liveable environment.

We’re out of time. We need to act, now.

We have as many of the pieces as I think can reasonably exist to give confidence to what needs to happen next.

There is a ‘raise the tide’ moment that, done well, could unlock tens of billions in value, for everyone. If we can go far, together.

So, to recap: I created Icebreaker One to try and catalyse ‘green swan’ moments. To help break through the complete stalemate of private-public-sector interactions that we are stuck in: there are two exponential changes we are right in the midst of: the web of data and our environmental crises (e.g. climate change).

As such it is a disruptor. But it acts as a convener, a bridge, a community, and — crucially — an implementor. Its role is incredibly thin to ensure it minimises the potential that it might compete with the things it’s trying to accelerate.

Its vision is to raise the pre-competitive access to data (at the scale of the web) to help everyone address the crises that are upon us.

It is based on the fact that we need to do everything, everywhere, all at once. It assumes that the best shot we have to enable that is using the tools we already have.

It is anchored on the most successful mechanism humans have ever created to share information and connect: the web. This time it’s the age of the web of data.

And, this time, it needs to be for everyone, while including the protections we missed the first time around.

We aim to do one thing well and to go far, together.

We are a constellation of innovators, influencers, doers, makers, explorers and navigators. Join us at http://ib1.org/join!

There was confusion around ‘where is the data?’, ‘who owns what?’ and ‘what can be done with it?’.

My synopsis is that:

The ‘data gap’ is usually leadership
Data governance needs to be taken seriously at the C-suite, management and internal processes. Not doing so both creates risks and misses opportunities. This must be addressed ‘today’.

1. Legal rules need to better balance risks and benefits
   There are likely areas of legal exposure on contracts that must be (re)thought through in a way that both protects the organisation and helps to unlock innovation through data sharing. Often the default is too restrictive and/or misses entire categories of risk.
2. Company policies must be enforceable, monitored and applied
   Organisations should use procurement and contracting processes and ensure that they are fit for purpose, and have standard operating procedures for data governance. This should include clear definitions of who is responsible (e.g. ‘Data Controller’, Chief Data Officer), for what areas and purposes, and how data processing will be managed. All third-party services must contain clear definitions of IP, security, support, and maintenance (even if some services are offered for ‘free’).

One issue I heard a lot of confusion about was the ‘location’ of data.
Its location can be interpreted in two ways:

1. Its physical location

Data can be stored on computers and in systems that are under the organisation’s direct ownership or control through third parties that it uses. These might include cloud computing providers like Google, Amazon, or third-party analysis systems. It is completely fine to have data in many physical places (and ‘the cloud’ is exactly that).

The impact of ‘many physical systems’ does create an operational burden and the organisation’s data teams should aim to minimise this burden by storing data in as few systems as possible. But they must maintain direct contractual control to ensure that the data is well managed and can be held to account (e.g. such as direct contracts with third parties).

2. Its legal location

Regardless of physical location, the legal basis for data sharing must be clearly defined, contracted and enforced.

Organisations should aim to act as (and it will be in many jurisdictions) the primary data controller for all data that it collects. Making this clear is especially important when dealing with countries where rules around data are ‘early stage’. The concept of data ‘ownership’ is highly complex (especially with personal data) and a focus on ‘rights’ (rather than just ‘ethics’) can often help people focus on what’s important.

The organisation can, and should, own the IP (intellectual property) on certain data it collects ( e.g. raw such as non-personal data that its teams have collected), derivative data (aggregate statistics), reports, analyses, insights, visualisations, etc. Owning the IP allows the organisation to explicitly license it to others if it wishes to (whether under Open Data or Shared Data licenses).

The organisation will not ‘own’ personal data about individuals (e.g. EU citizens’ rights are covered by GDPR). Instead, it has a role as a data controller or as a data processor. Depending on its contracts it may have the right to do things with the data, including analysing it and sharing outputs with others.

If the organisation contracts a third party to do, for example, some data cleaning or analysis, it must (a) have the rights to do so; and, (b) do so under a contractual agreement that allows data to be processed by that third party. The organisation should not assign any ownership rights to any ‘primary’ data and nor should it assign rights to any derivate outputs (e.g. analysis) without assessing the risks or benefits of doing so. If it does, such rights must be codified in a contract.

It may be the case that sharing data with a third party can help create additional benefits (e.g. they can improve their analytics systems), in the same way using data from others can create additional benefits for the organisation.

However, there are also new risks, such as using an organisation’s data to train machine-learning / artificial intelligence systems in ways that cannot be predicted. These risks include commercial, competitive, legal, liability, IP, ethical and moral hazards.

The impact of ‘many legal contracts’ creates compound risks and the organisation must be crystal clear about its approach to data governance, including protections, licensing, processing and security. Creating common legal frameworks with multiple parties can take time, but they can also create cohesion, reducing risk and unlocking permission for innovation.

One way to think about this is to imagine ‘technical systems’ as consultants. It doesn’t matter where they are, we don’t give a consultant the right to do anything other than what we need them to do. It doesn’t stop them from learning and building on their own experience. As we move forward with new ‘ai’ systems, this takes on materially different dimensions and creates new types of risk and new types of opportunities.

The purpose of this piece was to help non-technical people understand some of the basis of ‘where’ is data, who can use it, and for what purposes. I hope you found it useful and please leave comments if you have further questions or feedback.

One personal note: I find it far more useful to think about data rights (can this data be used for that purpose in this way) rather than data ‘ethics’ (which can mean very different things to different people).

As we continue on our journey to a data-enabled world, we must ensure that data governance processes are in place to help everyone understand what they can and can’t do, and what to do when things go wrong.

1. create an algorithm for trading based on 1% value swings

Here’s a basic algorithm for trading based on 1% value swings:

1. Define your target security: Select a security that you wish to trade based on 1% value swings.
2. Set your trading parameters: Define the parameters for your trades, including the amount of capital you’re willing to invest, your desired profit margin, and your stop-loss limit.
3. Monitor the security’s value: Continuously monitor the value of your target security. You can use various tools such as trading software, financial news sources, or chart analysis to track its value.
4. Identify 1% value swings: When the security’s value changes by 1% in either direction, take note of it.
5. Buy or sell: If the value of the security has decreased by 1%, and you believe it will rebound, consider buying the security. Conversely, if the value of the security has increased by 1%, and you believe it may drop, consider selling the security.
6. Set stop-loss limit: To manage risk, set a stop-loss limit for each trade. This is the maximum amount of money you’re willing to lose if the trade doesn’t go in your favor.
7. Take profit: Set a profit target for each trade, which is the desired amount of profit you want to earn. When the security’s value reaches this target, close the trade and take your profit.
8. Repeat the process: Continue monitoring the security’s value, identifying 1% swings, and making trades as necessary.

Note: This is a basic algorithm, and there are many factors to consider when trading, such as market conditions, liquidity, volatility, and more. It’s important to do your research and fully understand the risks involved before implementing any trading strategy.

2. write this is pseudocode

Sure, here’s the pseudocode for the trading algorithm based on 1% value swings:

Note that this is a basic pseudocode and it does not include error handling or other advanced features. Additionally, some variables and functions may need to be defined and implemented separately.

3. write this in python

Sure, here’s an example implementation of the trading algorithm based on 1% value swings in Python:

target_security = "AAPL"  # define the target security

# define the trading parameters
capital = 1000
profit_margin = 1.01  # 1% profit margin
stop_loss = 0.99  # 1% stop-loss limit

while True:
    current_value = get_current_value(target_security)  # get the current value of the target security
    if current_value >= profit_margin:
        sell(target_security)  # sell the security if it has increased by 1%
    elif current_value <= stop_loss:
        buy(target_security)  # buy the security if it has decreased by 1%
    else:
        continue  # otherwise, continue monitoring the value

    # set stop-loss limit and profit target
    stop_loss_value = current_value * stop_loss
    profit_target_value = current_value * profit_margin

    while True:
        updated_value = get_current_value(target_security)  # get the updated value of the target security
        if updated_value <= stop_loss_value:
            sell(target_security)  # sell the security if it hits the stop-loss limit
            break
        elif updated_value >= profit_target_value:
            sell(target_security)  # sell the security if it hits the profit target
            break
        else:
            continue  # otherwise, continue monitoring the value

Note that this is just an example implementation and may need to be adapted to your specific trading platform or strategy. Also, the get_current_value, buy, and sell functions are not defined here and need to be implemented separately based on the API or service you’re using.

And now for some fun

write this for a quantum computer

While quantum computers have the potential to solve certain financial problems more efficiently than classical computers, it’s currently unclear how exactly they could be used for trading based on 1% value swings. As quantum computing is still in its early stages of development, there are few, if any, quantum trading algorithms that have been developed and tested at scale.

However, one possible approach to leveraging quantum computing for trading is by using quantum machine learning algorithms to analyze financial data and identify patterns that can be used to make trading decisions. For example, one could use quantum annealing to optimize portfolio selection and risk management, or quantum algorithms to perform efficient optimization of trading strategies.

Here’s an example of how one could use the Qiskit framework in Python to implement a simple quantum machine learning algorithm for financial prediction:

import numpy as np
from qiskit import Aer
from qiskit.circuit.library import ZZFeatureMap
from qiskit.aqua.algorithms import VQC
from qiskit.aqua.components.optimizers import COBYLA

# Define the target security and trading parameters
target_security = "AAPL"
capital = 1000
profit_margin = 1.01
stop_loss = 0.99

# Define the quantum feature map and backend
feature_map = ZZFeatureMap(feature_dimension=2)
backend = Aer.get_backend('qasm_simulator')

# Define the VQC classifier and optimizer
classifier = VQC(feature_map, None, None)
optimizer = COBYLA()

# Train the classifier on historical financial data
data = np.array([[0.5, 0.1], [0.1, 0.9], [0.2, 0.8], [0.3, 0.7], [0.4, 0.6]])
labels = np.array([0, 1, 1, 0, 0])
classifier.train(data, labels)

# Use the classifier to make trading decisions
while True:
    current_data = np.array([[get_current_price(target_security), get_current_volume(target_security)]])
    predicted_label = classifier.predict(current_data)[0]
    if predicted_label == 0:
        buy(target_security)
    elif predicted_label == 1:
        sell(target_security)
    else:
        continue

    # Set the stop-loss limit and profit target
    stop_loss_value = get_current_price(target_security) * stop_loss
    profit_target_value = get_current_price(target_security) * profit_margin

    while True:
        updated_price = get_current_price(target_security)
        if updated_price <= stop_loss_value:
            sell(target_security)
            break
        elif updated_price >= profit_target_value:
            sell(target_security)
            break
        else:
            continue

Again, note that this is just a simple example of how one could use quantum machine learning for trading, and many details would need to be filled in depending on the specific application and data available.

So, there we have it. I can now learn how to write code for a quantum computer that might have some bearing on financial insights, with zero knowledge.

I’m often asked to unpack ‘data’. There’s a lot of confusion and, often, people think it’s a function of ‘IT’ and just for the geeks.

A ‘data strategy’ can often be quite shallow in its thinking and tends not to focus on data sharing—which is a cultural shift in the way collaboration needs to work.

Collaboration in a data-enabled business needs to be about how humans and machines can combine their intelligence, systems, and accelerate and optimise solutions to problems. At the core of this is the ability to easily access, share and use systems internally and externally.

Even if there’s an understanding that data should be used at a strategic level, it’s rare that there is a good understanding of data strategy at C-level, and often it’s deferred to CIO/CTO to make decisions that, in my view, should be more collaboratively defined by the whole team, aligned with business targets, and explicit contributions to ROI.

For example, I would like to think that the age of “build a data lake” or “let’s build a portal” as a starting point are over. However, my observation is that we are very far away from that.

We need to begin by thinking of data as part of our infrastructure and, with this, thinking about how it can be shared between systems regardless of where those systems are.

The Open Data Institute (ODI), frames data infrastructure as the “systems, processes, and tools that enable the creation, management, and use of data.” This includes both physical infrastructure (e.g. servers and storage systems) and virtual infrastructure (e.g. software and networks).

The definition goes further to include: the people, policies, and practices that support the creation, management, processing and usage of data; roles and responsibilities for managing data; the standards and protocols for how data should be collected, stored, and shared.

A strong data infrastructure is the foundation for data management, analytics, visualisation, and value creation. And, in a networked world, the systems for management, analysis, visualisation may exist anywhere.

A strong strategy must, therefore, address interoperability at its core.

A data infrastructure strategy is a plan for building, managing, and maintaining systems, processes, and tools that enable data creation, management and use in a way that can be shared across systems.

It is important to understand the role of data sharing in supporting outcomes and impacts and, based on this, to create a framework for sharing data.

A well-designed strategy can save time and money by rapidly connecting data and systems together: to help people make better, more informed decisions and use data to drive growth and innovation. It must take into account the types of data as well as the business processes (logistical, operational and legal), and understand the applications that rely on that data.

Critically, it must consider the organisation’s data governance policies and procedures, legal and regulatory requirements.

In summary, four core elements of a robust data infrastructure strategy must include:

1. Data governance: policies, procedures, and standards (that can be enforced) to ensure the usability, quality, security, and integrity of data, its usage and sharing (internally and externally to any organisation).
2. Data access and security: how data will be accessed and shared, both across organisations and externally, including protecting data from unauthorized access or misuse. This will combine technical and legal measures.
3. Data analytics and visualization: understanding the tools used to analyse, process, visualise and share outputs, and the processes and skills needed to use them effectively.
4. Data storage and management: where and how data will be stored, the tools and processes needed to manage data throughout its lifecycle, regardless of where the data ‘physcially’ resides. This will include time-based, jurisdictional and related policies.

Note that only point four (and part of point two) of this are functions of ‘IT’. Points one and three span the entire organisation, as managers and as users.

To focus attentiion, a data infrastructure strategy should directly support goals and outcomes, minimise risks, unlock effective usage and sharing to drive growth and innovation.

Finally, it would be prudent to begin to include ‘algorithms’ (as opposed to code) as part of strategic development as while in many cases it may be the data that is ‘moving to the processing’, it is also the case that the algorithms may have to ‘go where the data is’ for efficiency and/or compliance reasons. This may be the subject of a future post.

https://dgen.net/d0

Having spent quite a number of decades working on data at scale, we’ve worked to codify some of the baseline principles that could underpin a trusted web of data.

The purpose is to unlock open markets and increase the precompetitive spaces to both

1. commoditize current parts of the data ecosystem; and,
2. increase the efficiency, value and potential of using data to solve problems.

To me, this is a step we need to take to create data as infrastructure and unlock the next generation of data-enabled innovation.  Open document for discussion here

On one level, we could draw an analogy with energy: there were many ways of connecting, different voltages, sockets, etc. Then we created the national grid and built our economy on it. Getting it right was tough, but it shifted the value chain to delivering services at web-scale.

Another is open banking: banks had ‘ways of being interoperable’ but standardising this process is driving a different kind of innovation at application, business and consumer levels. It’s now in 80 countries (5 years since launch), and is a $23B market that is projected to grow at 23% CAGR to $150B by 2032 [citation].

Why wouldn’t we enable this for all parts of our economy?

Learn more here: