research https://dgen.net/0 Designing our data infrastructure Sat, 02 Aug 2025 21:30:52 +0000 en-GB hourly 1 https://wordpress.org/?v=6.9.4 https://dgen.net/0/wp-content/uploads/2017/02/cropped-dgen-D-logo-2017-32x32.png research https://dgen.net/0 32 32 232530259 Diome: definition https://dgen.net/0/2023/09/12/diome-definition/ Tue, 12 Sep 2023 12:56:00 +0000 https://dgen.net/0/?p=6687 Diome [noun] A data biome that represents a temporal cluster of data from which specific insights can be learned, analysed and/or derived. Discussion This is […]]]>

Diome
[noun]
A data biome that represents a temporal cluster of data from which specific insights can be learned, analysed and/or derived.

Discussion

This is public on LinkedIn (open gdoc here if you’d like to contribute)

  • when training AI, if you have a load of similar diomes you’ll end up with monocultural outcomes.
  • in the same way, ‘monoculture’ biomes means you cannot have the degree of biodiversity to sustain a planet, if you don’t have enough diomes you wont be able to represent the richness of our world (natural and human)
  • ensure that the diome has as complete a representation of the system as possible, with appropriate granularity, resolution, quality, volume and systems ontology?

Builds on the definition of Biome

“A biome is a biogeographical unit consisting of a biological community that has formed in response to the physical environment in which they are found and a shared regional climate. Biomes may span more than one continent. Biome is a broader term than habitat and can comprise a variety of habitats.”

]]> 6687 ai code example: animated planetary orbit https://dgen.net/0/2023/07/21/ai-code-example-animated-planetary-orbit/ Fri, 21 Jul 2023 20:31:49 +0000 https://dgen.net/0/?p=6502 Today’s example of how easy AI can make it to write code. The interactive animation below is the final output from various experiments below.[My words […]]]>

Today’s example of how easy AI can make it to write code. The interactive animation below is the final output from various experiments below.
[My words have this colour background. My instructions to the AI are in bold.
The rest of the content on this page is the AI. All of it, including words, code and highlighting.]

To begin with I just asked it the following:
write html, css and javascript to demonstrate a planet orbiting the sun
It replied…

Creating a realistic 3D planet orbiting the sun requires more advanced graphics and libraries, but I can show you a basic demonstration using HTML, CSS, and JavaScript that simulates a simple planet orbiting the sun

example.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Orbiting Planet</title>
  <link rel="stylesheet" href="styles.css">
</head>
<body>
  <div class="sun"></div>
  <div class="planet"></div>
  <script src="script.js"></script>
</body>
</html>

CSS (styles.css):

body {
  margin: 0;
  padding: 0;
  overflow: hidden;
}

.sun {
  width: 100px;
  height: 100px;
  background-color: orange;
  border-radius: 50%;
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
}

.planet {
  width: 30px;
  height: 30px;
  background-color: blue;
  border-radius: 50%;
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
}

JavaScript (script.js):

let angle = 0;
const orbitRadius = 150;
const orbitSpeed = 0.005;

function animatePlanet() {
  const planet = document.querySelector('.planet');
  const sun = document.querySelector('.sun');
  const sunRect = sun.getBoundingClientRect();
  const centerX = sunRect.left + sunRect.width / 2;
  const centerY = sunRect.top + sunRect.height / 2;

  const x = centerX + orbitRadius * Math.cos(angle);
  const y = centerY + orbitRadius * Math.sin(angle);

  planet.style.left = x + 'px';
  planet.style.top = y + 'px';

  angle += orbitSpeed;
  requestAnimationFrame(animatePlanet);
}

animatePlanet();

This code creates a basic representation of a planet (blue circle) orbiting the sun (orange circle). It uses CSS to position and style the sun and the planet, and JavaScript to calculate the position of the planet as it orbits the sun.

Please note that this is a simplified demonstration and does not include realistic physics or 3D graphics. For more advanced and realistic animations, you would need to use libraries like Three.js or other WebGL frameworks.”

This is pretty astonishing [output]

but I wasn’t happy with it being a circular orbit, so I asked it to:
redo with an elliptical orbit

Creating an elliptical orbit requires more complex mathematical calculations and is beyond the scope of basic HTML, CSS, and JavaScript. However, I can show you an example using the p5.js library, which is a JavaScript library for creative coding and visualization. The p5.js library simplifies handling graphics and animation, making it easier to demonstrate an elliptical orbit.

  1. First, include the p5.js library in your HTML file:
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Elliptical Orbit</title>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.4.0/p5.js"></script>
  <style>
    body {
      margin: 0;
      padding: 0;
    }
  </style>
</head>
<body>
  <script src="sketch.js"></script>
</body>
</html>
  1. Next, create a JavaScript file (e.g., “sketch.js”) that will handle the animation and display of the elliptical orbit:
let sun;
let planet;
let orbitRadiusX = 150;
let orbitRadiusY = 100;
let angle = 0;
let orbitSpeed = 0.01;

function setup() {
  createCanvas(windowWidth, windowHeight);
  sun = createVector(width / 2, height / 2);
  planet = createVector(sun.x + orbitRadiusX, sun.y);
}

function draw() {
  background(240);

  // Draw the sun
  fill('orange');
  ellipse(sun.x, sun.y, 40, 40);

  // Calculate the position of the planet in the elliptical orbit
  let x = sun.x + orbitRadiusX * cos(angle);
  let y = sun.y + orbitRadiusY * sin(angle);

  // Draw the planet
  fill('blue');
  ellipse(x, y, 20, 20);

  // Update the angle for the next frame
  angle += orbitSpeed;
}

The above code sets up a canvas, defines the sun and planet as vectors, and uses trigonometry to calculate the position of the planet as it moves along an elliptical orbit around the sun. The orbitRadiusX and orbitRadiusY variables determine the shape and size of the elliptical orbit, while the orbitSpeed variable controls the speed of the orbit.

Save these HTML and JavaScript files in the same folder and open the HTML file in a web browser. You should see an animation of a blue planet orbiting an orange sun in an elliptical path. Feel free to adjust the values of orbitRadiusX, orbitRadiusY, and orbitSpeed to experiment with different elliptical orbits.

This is a great example of how game-changing AI is becoming. To see the two bits of code (copy & pasted) in action, click Circular or Elliptical. Remarkable.

And then my friend Andy noticed it wasn’t doing a Keplerian orbit, so
redo with a keplerian orbit
led to this

Creating a Keplerian orbit involves implementing more complex mathematical formulas to simulate the motion of celestial bodies following Kepler’s laws of planetary motion. To do this, we’ll use the p5.js library to handle graphics and animation.

  1. First, include the p5.js library in your HTML file:
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Keplerian Orbit</title>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.4.0/p5.js"></script>
  <style>
    body {
      margin: 0;
      padding: 0;
    }
  </style>
</head>
<body>
  <script src="sketch.js"></script>
</body>
</html>
  1. Next, create a JavaScript file (e.g., “sketch.js”) that will handle the animation and display of the Keplerian orbit:
let sun;
let planet;
let a; // Semi-major axis
let b; // Semi-minor axis
let angle = 0;
let orbitSpeed = 0.01;

function setup() {
  createCanvas(windowWidth, windowHeight);
  sun = createVector(width / 2, height / 2);
  a = 150; // Set the semi-major axis
  b = 100; // Set the semi-minor axis
  planet = createVector(sun.x + a, sun.y);
}

function draw() {
  background(240);

  // Draw the sun
  fill('orange');
  ellipse(sun.x, sun.y, 40, 40);

  // Calculate the position of the planet in the Keplerian orbit
  let x = sun.x + a * cos(angle);
  let y = sun.y + b * sin(angle);

  // Draw the planet
  fill('blue');
  ellipse(x, y, 20, 20);

  // Update the angle for the next frame
  angle += orbitSpeed;
}

The above code sets up a canvas, defines the sun and planet as vectors, and uses trigonometry to calculate the position of the planet following Kepler’s laws of planetary motion. The a and b variables determine the size and shape of the Keplerian orbit, while the orbitSpeed variable controls the speed of the orbit.

Save these HTML and JavaScript files in the same folder and open the HTML file in a web browser. You should see an animation of a blue planet following a Keplerian orbit around an orange sun. Feel free to adjust the values of a, b, and orbitSpeed to experiment with different Keplerian orbits.

Interesting but still not quite there

So let’s
place the sun at a focus point and add gravity

To add gravity to the Keplerian orbit simulation, we can use Newton’s law of universal gravitation to calculate the gravitational force between the sun and the planet. The force will affect the acceleration of the planet, causing it to move in a curved path around the sun.

Here’s the updated code that includes gravity:

let sun;
let planet;
let a; // Semi-major axis
let b; // Semi-minor axis
let c; // Distance from the center to focus (c = sqrt(a^2 - b^2))
let angle = 0;
let orbitSpeed = 0.01;
let G = 0.5; // Gravitational constant

function setup() {
  createCanvas(windowWidth, windowHeight);
  sun = createVector(width / 2, height / 2);
  a = 150; // Set the semi-major axis
  b = 100; // Set the semi-minor axis
  c = sqrt(a * a - b * b);
  planet = createVector(sun.x + c + a, sun.y);
}

function draw() {
  background(240);

  // Draw the sun at the focus point
  fill('orange');
  ellipse(sun.x, sun.y, 40, 40);

  // Calculate the position of the planet in the Keplerian orbit
  let x = sun.x + c + a * cos(angle);
  let y = sun.y + b * sin(angle);

  // Draw the planet
  fill('blue');
  ellipse(x, y, 20, 20);

  // Calculate the distance between the planet and the sun
  let distance = dist(x, y, sun.x, sun.y);

  // Calculate the gravitational force between the planet and the sun
  let force = G * (1 / (distance * distance));

  // Calculate the angle of the force vector
  let forceAngle = atan2(sun.y - y, sun.x - x);

  // Calculate the acceleration of the planet due to gravity
  let acceleration = force / 1; // We can simplify the mass of the planet as 1

  // Update the planet's velocity based on the acceleration
  orbitSpeed += acceleration;

  // Update the angle for the next frame
  angle += orbitSpeed;
}

In this updated code, we introduce the gravitational constant G (you can adjust its value to control the strength of gravity) and calculate the gravitational force between the sun and the planet using Newton’s law of universal gravitation. The planet’s acceleration is then updated based on the gravitational force, and its velocity is adjusted accordingly. The result is a more realistic simulation of a Keplerian orbit with gravity affecting the motion of the planet around the sun.

Save these HTML and JavaScript files in the same folder and open the HTML file in a web browser. You should now see an animation of a blue planet following a Keplerian orbit around an orange sun, with gravity affecting its movement. Feel free to experiment with different values of G and other parameters to observe different orbital behaviors.

Very interesting. It get’s it much more right, but there’s an acceleration that shouldn’t be there. The planet spins faster and faster…

so, rather than try and troubleshoot it, let’s just ask it
why does the speed of the planet accelerate?

Apologies for the confusion in the code. The speed of the planet should not accelerate as a result of gravitational force. In reality, the speed of a planet in a stable orbit around the sun remains constant.

To accurately simulate a stable orbit, we need to use the principles of circular motion and centripetal force. The centripetal force acts perpendicular to the velocity vector, causing the planet to follow a circular path around the sun. In this scenario, the speed of the planet remains constant while its direction continuously changes.

Let’s correct the code to achieve a stable circular orbit:

let sun;
let planet;
let a; // Semi-major axis
let b; // Semi-minor axis
let c; // Distance from the center to focus (c = sqrt(a^2 - b^2))
let angle = 0;
let orbitSpeed = 0.01;
let G = 0.5; // Gravitational constant

function setup() {
  createCanvas(windowWidth, windowHeight);
  sun = createVector(width / 2, height / 2);
  a = 150; // Set the semi-major axis
  b = 100; // Set the semi-minor axis
  c = sqrt(a * a - b * b);
  planet = createVector(sun.x + c + a, sun.y);
}

function draw() {
  background(240);

  // Draw the sun at the focus point
  fill('orange');
  ellipse(sun.x, sun.y, 40, 40);

  // Calculate the position of the planet in the Keplerian orbit
  let x = sun.x + c + a * cos(angle);
  let y = sun.y + b * sin(angle);

  // Draw the planet
  fill('blue');
  ellipse(x, y, 20, 20);

  // Update the angle for the next frame
  angle += orbitSpeed;
}

And, there we have it. We didn’t even need to understand why the planet was accelerating, or if there was a bug in the code, just ask a question and it corrected itself.

I wanted to change the colours and layout a bit, so I asked it to do that too
left-align the scene, make the sun yellow, the background black and the planet blue
and then I thought it’d be fun if we made it interactive, so I asked
add a slider so I can change the value of G and ensure that the animation updates automatically if the value is changed. put the slider at the top in the middle
which it didn’t get right the first time (the slider moved but the planet speed didn’t change) so,
that isn’t updating the value of G in the animation
which fixed it, and then a final bit of fiddling
print the value of G on the screen as well. align the slider so it’s just above the animation
move it 200px left. Keep the value of G next to the slider
and so on, which resulted in the interactive example at the top of this page

And, lastly, for fans of Processing, simply asking it to
rewrite this in Processing
gave this working code

int sunX, sunY;
float a; // Semi-major axis
float b; // Semi-minor axis
float c; // Distance from the center to focus (c = sqrt(a^2 - b^2))
float angle = 0;
float orbitSpeed = 0.01;
float G = 0.5; // Gravitational constant

void setup() {
  size(800, 600);
  sunX = width / 4;
  sunY = height / 2;
  a = 150; // Set the semi-major axis
  b = 100; // Set the semi-minor axis
  c = sqrt(a * a - b * b);
}

void draw() {
  background(0); // Set background color to black

  // Draw the sun at the focus point
  fill(255, 255, 0); // Change sun color to yellow
  ellipse(sunX, sunY, 40, 40);

  // Calculate the position of the planet in the Keplerian orbit
  float x = sunX + c + a * cos(angle);
  float y = sunY + b * sin(angle);

  // Draw the planet
  fill(0, 0, 255); // Change planet color to blue
  ellipse(x, y, 20, 20);

  // Update the angle for the next frame based on the slider value
  angle += G * orbitSpeed; // Use G to modify orbit speed

  // Display the value of G next to the slider
  fill(255);
  textSize(20);
  text("Gravitational Constant (G): " + G, 20, 20);
}

// Event handler for the slider
void mouseReleased() {
  if (mouseX >= 20 && mouseX <= 320 && mouseY >= 40 && mouseY <= 60) {
    // Map the x-coordinate of the mouse to a value between 0 and 2 for G
    G = map(mouseX, 20, 320, 0, 2);
  }
}
]]> 6502 Testing out this AI thing https://dgen.net/0/2023/03/24/testing-out-this-ai-thing/ Fri, 24 Mar 2023 17:58:08 +0000 https://dgen.net/0/?p=6281

  1. create an algorithm for trading based on 1% value swings

Here’s a basic algorithm for trading based on 1% value swings:

  1. Define your target security: Select a security that you wish to trade based on 1% value swings.
  2. Set your trading parameters: Define the parameters for your trades, including the amount of capital you’re willing to invest, your desired profit margin, and your stop-loss limit.
  3. Monitor the security’s value: Continuously monitor the value of your target security. You can use various tools such as trading software, financial news sources, or chart analysis to track its value.
  4. Identify 1% value swings: When the security’s value changes by 1% in either direction, take note of it.
  5. Buy or sell: If the value of the security has decreased by 1%, and you believe it will rebound, consider buying the security. Conversely, if the value of the security has increased by 1%, and you believe it may drop, consider selling the security.
  6. Set stop-loss limit: To manage risk, set a stop-loss limit for each trade. This is the maximum amount of money you’re willing to lose if the trade doesn’t go in your favor.
  7. Take profit: Set a profit target for each trade, which is the desired amount of profit you want to earn. When the security’s value reaches this target, close the trade and take your profit.
  8. Repeat the process: Continue monitoring the security’s value, identifying 1% swings, and making trades as necessary.

Note: This is a basic algorithm, and there are many factors to consider when trading, such as market conditions, liquidity, volatility, and more. It’s important to do your research and fully understand the risks involved before implementing any trading strategy.

2. write this is pseudocode

Sure, here’s the pseudocode for the trading algorithm based on 1% value swings:

Note that this is a basic pseudocode and it does not include error handling or other advanced features. Additionally, some variables and functions may need to be defined and implemented separately.

3. write this in python

Sure, here’s an example implementation of the trading algorithm based on 1% value swings in Python:

target_security = "AAPL"  # define the target security

# define the trading parameters
capital = 1000
profit_margin = 1.01  # 1% profit margin
stop_loss = 0.99  # 1% stop-loss limit

while True:
    current_value = get_current_value(target_security)  # get the current value of the target security
    if current_value >= profit_margin:
        sell(target_security)  # sell the security if it has increased by 1%
    elif current_value <= stop_loss:
        buy(target_security)  # buy the security if it has decreased by 1%
    else:
        continue  # otherwise, continue monitoring the value

    # set stop-loss limit and profit target
    stop_loss_value = current_value * stop_loss
    profit_target_value = current_value * profit_margin

    while True:
        updated_value = get_current_value(target_security)  # get the updated value of the target security
        if updated_value <= stop_loss_value:
            sell(target_security)  # sell the security if it hits the stop-loss limit
            break
        elif updated_value >= profit_target_value:
            sell(target_security)  # sell the security if it hits the profit target
            break
        else:
            continue  # otherwise, continue monitoring the value

Note that this is just an example implementation and may need to be adapted to your specific trading platform or strategy. Also, the get_current_value, buy, and sell functions are not defined here and need to be implemented separately based on the API or service you’re using.

And now for some fun

write this for a quantum computer

While quantum computers have the potential to solve certain financial problems more efficiently than classical computers, it’s currently unclear how exactly they could be used for trading based on 1% value swings. As quantum computing is still in its early stages of development, there are few, if any, quantum trading algorithms that have been developed and tested at scale.

However, one possible approach to leveraging quantum computing for trading is by using quantum machine learning algorithms to analyze financial data and identify patterns that can be used to make trading decisions. For example, one could use quantum annealing to optimize portfolio selection and risk management, or quantum algorithms to perform efficient optimization of trading strategies.

Here’s an example of how one could use the Qiskit framework in Python to implement a simple quantum machine learning algorithm for financial prediction:

import numpy as np
from qiskit import Aer
from qiskit.circuit.library import ZZFeatureMap
from qiskit.aqua.algorithms import VQC
from qiskit.aqua.components.optimizers import COBYLA

# Define the target security and trading parameters
target_security = "AAPL"
capital = 1000
profit_margin = 1.01
stop_loss = 0.99

# Define the quantum feature map and backend
feature_map = ZZFeatureMap(feature_dimension=2)
backend = Aer.get_backend('qasm_simulator')

# Define the VQC classifier and optimizer
classifier = VQC(feature_map, None, None)
optimizer = COBYLA()

# Train the classifier on historical financial data
data = np.array([[0.5, 0.1], [0.1, 0.9], [0.2, 0.8], [0.3, 0.7], [0.4, 0.6]])
labels = np.array([0, 1, 1, 0, 0])
classifier.train(data, labels)

# Use the classifier to make trading decisions
while True:
    current_data = np.array([[get_current_price(target_security), get_current_volume(target_security)]])
    predicted_label = classifier.predict(current_data)[0]
    if predicted_label == 0:
        buy(target_security)
    elif predicted_label == 1:
        sell(target_security)
    else:
        continue

    # Set the stop-loss limit and profit target
    stop_loss_value = get_current_price(target_security) * stop_loss
    profit_target_value = get_current_price(target_security) * profit_margin

    while True:
        updated_price = get_current_price(target_security)
        if updated_price <= stop_loss_value:
            sell(target_security)
            break
        elif updated_price >= profit_target_value:
            sell(target_security)
            break
        else:
            continue

Again, note that this is just a simple example of how one could use quantum machine learning for trading, and many details would need to be filled in depending on the specific application and data available.

So, there we have it. I can now learn how to write code for a quantum computer that might have some bearing on financial insights, with zero knowledge.

]]> 6281 Designing the UK’s shared data infrastructure https://dgen.net/0/2020/08/15/designing-the-uks-shared-data-infrastructure/ Sat, 15 Aug 2020 09:13:06 +0000 https://www.dgen.net/0/?p=4649 Smart Data: Research on Consent, Liability & Authentication The government has estimated that the impact of personal data mobility enabled productivity and efficiency is worth […]]]>

Smart Data: Research on Consent, Liability & Authentication

The government has estimated that the impact of personal data mobility enabled productivity and efficiency is worth approximately £27.8bn. Smart Data could be a key part of the Government’s arsenal in the fight against Covid-19 and climate change.  

Up until now the UK has taken a piecemeal approach to unlocking data. This creates data bottlenecks that hinder us from allocating resources better within the home, at work, and across society. Open Banking has pioneered data sharing in the UK and provides a foundation of knowledge and learning for Open Finance and Smart Data. However, Open Banking is underpinned by a complex regulatory structure, faces governance challenges and has not yet resolved its longer-term funding.

Following the publication of BEIS reports about Consent and Liability for Smart Data Initiatives in the UK, Dgen (Faith Reynolds, Sharon Cunliffe, Miles Cheetham, Gavin Starks) held an overview webinar on Wednesday 16th September 2020.

Covering two new publications:

 

 

 

 

 

 

 

 

 

 

 

 

UK Government links:  https://gov.uk/government/publications/smart-data-research-on-consent-liability-and-authentication

2020-09 BEIS response: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/915973/smart-data-consultation-response.pdf

 

]]> 4649 An evolution in process https://dgen.net/0/2019/10/03/an-evolution-in-process/ Thu, 03 Oct 2019 08:39:38 +0000 https://www.dgen.net/0/?p=3729 The above chart is an evolution of the work of Simon Wardley, which I’ve been following for many years. It feels like we’re in another […]]]>

The above chart is an evolution of the work of Simon Wardley, which I’ve been following for many years. It feels like we’re in another phase-change based on the impact of the web and technology: of radically centralised+decentralised tensions, of systems that self-organise/self-actuate, where teams are spread as constellations, and where ‘open’ (for many varying definitions of the word) is a central forcing function of change.

“Innovators must deter competition to get some of the value they created”

This view has been at the heart of our industrial economic drive—I think most of my career has been anchored in trying to invert it.

In creating open businesses I’ve continuously sought to increase the number of connections, not the size of the moat: building federated systems not centralised fiefdoms, actively encouraging imitation and rapid evolution. In the long-term, I think the impact of being open is far more profound as a culture shift.

Moats draw their power to prevent imitation from one of four basic sources:

The state
We’ve created regulation that mandates ‘open’ systems, such as Open Banking. This is now in-force in the UK and in development in 20 countries. The impact, I think, will be both innovation and the radical erosion of ‘excess profit’ to make open markets operate the way they were intended.

Special know-how
The web removes this as a barrier pretty quickly. The cost of setting up, for example, a bank is now 1/10th to 1/100th what it used to be. Moore’s Law applies to business models not just technology.

Scale
Ditto scale. Cloud compute + web + mobile decimates the path to the customer faster and at greater scale than anything else we’ve invented.

System rigidity
Systems rigidity is a bug, not a feature. Startups route around rigidity faster than incumbents can hold a meeting. Note that rigidity is not the same thing as resilience or robustness.

And, with a startup lens: “Startups that aim to create value can’t have a moat when they begin, uncertainty is what protects them from competition until a proper moat can be built. Uncertainty becomes their moat.”

So, what next, moat-economy enthusiasts? How do these ideas evolve in a web-first society & economy?

My friend Peter commented:

“Innovation must be open and collaborative, value is created by societies not individuals, through innovation, because it is not an absolute, it is a differential, it is how we rescue wealth from the inevitable decay imposed by obsolescence and environmental “externalities”. Individuals who hoard end up sitting on a pile of worthless rubbish … and our society is geared up to pay them for the privilege – i.e. we pay for the decay of the wealth of others. That’s what being rich means in this unsustainable zero sum game.”

]]> 3729 The Data Spectrum: defining Shared & Closed https://dgen.net/0/2019/05/06/the-data-spectrum-defining-shared-closed/ Mon, 06 May 2019 01:23:02 +0000 https://www.dgen.net/0/?p=3438 One of the things I’m most proud of creating at the ODI is the Data Spectrum, to us help understand the language of data. When […]]]>

One of the things I’m most proud of creating at the ODI is the Data Spectrum, to us help understand the language of data.

When ODI launched in 2012, there was a lot of uninformed rhetoric about data. Ministers would talk on-stage about the revolution in “big, open, personal data”, mixing categories and leaving audiences at best confused, at worst deeply concerned about their personal data.   We tried immediately to combat this with a graphic that separated those ideas and provoke questions about their intersection.

But this wasn’t good enough.

It took many hundreds of presentations, trying repeatedly to reset the conversation before we arrived at something that did (perhaps my background in astrophysics led me to use the word ‘spectrum’).

The Data Spectrum reframes the discussion about one simple question — “can I use the data?”

In a practical sense, the answer to this question is about legal permission: do I have a license to use the data?

This is then irrespective of whether the data is ‘big, medium or small’ or ‘personal, commercial or government’.

With Open Data, this is [relatively] easy. Open Data is that anyone can use for any purpose for free: there are a range of open licenses that can be applied to ensure this (e.g. Creative Commons, Open Government Licence)

While at the ODI, for Shared Data and Closed Data we gave examples, but we did not focus on trying to ‘solve’ for those categories as this was outside its remit.

Having reflected on this now for some years, I think there is potential to define Shared in a more concrete way—that is enabling to the web of data at-scale.

Shared data is, largely, held in the private sector.  The users are also, mostly, in the private sector.

The collective action challenge is that data licensing is typically done on a point-to-point basis, is often complex and between just two parties (that can also represent a cascade of third-party rights).

There are a number of ‘data as a service’ companies who provide blanket licensing, but these tend to be very high-level and become custom deals at the point of either scale or complexity.

We’ve attempted (and failed) to implement rights management schemes in the past (e.g. DRM in the music industry) and have been making some progress on machine-readable descriptions of rights since — the W3C adopted the Open Digital Rights Language in 2018, but such schemes have been slow in adoption. We’ve also seen huge progress on the technical development of the semantic web and schema.org but we’ve not, yet, translated this into ways non-technical people can engage with, especially around their corporate data.

Data increases in value the more it is connected

To increase these connections we must catalyse data sharing. We also need to protect people, confidentiality, intellectual property and security.

We need good use cases that change the culture of closed that permeates our current thinking that can illustrate the benefits. Sir Jeremey Heywood (former head of the UK civil service) believed that this was fundamental to the future.

I hope that Icebreaker One is one such catalyst — we won’t fix our climate emergency without more sharing of public and private data.

Creative Commons defined a step-change in thinking. It enabled us all to say, for example “it’s okay to use this image for free”,  in advance. As of May 2018, there were an estimated 1.4 billion works licensed using a CC licence.

Another interesting example is the UK Open Banking Standard, which pre-emptively defines and mandates ways to share personal data, it is now regulated and every UK high street bank has engage.

We need to make it easier for organisations to pre-emptively licence non-open data: for both public and private good. Can we, for example, say “for this use-case, this organisation can access and use information for a fee” and have a different licence for a different set of users?

So, why don’t we define Shared Data on the Data Spectrum as data that has a pre-emptive licence? (Closed Data is then data that requires a per-use, custom licence).

To do this, we must have clear, Open Data descriptions of the Shared Data that exists and how it might be used (how it is licensed).

Publishing open data that describes the shared data will enable search engines (and therefore you) to find it. If the licensing is clear, then the friction between discovery and usage is reduced.

Doing so will increase the size of the observable dataverse and help to unlock innovation while protecting the interests of individuals, organisations and countries to use it for both public and private good.

To summarise:

Open data: data that anyone can use, for any purpose, for free

Shared data: data that has pre-emptive licences for specific use-cases, and open data descriptions of both the data and the licence conditions

Closed data: data that requires a per-use, custom licence, negotiated on a case-by-case basis

]]> 3438 The utterance of a cosmological model? https://dgen.net/0/2011/10/09/the-utterance-of-a-cosmological-model/ Sun, 09 Oct 2011 17:24:15 +0000 http://www.dgen.net/blog/?p=481 “On 12 April 1961 Yuri Gagarin became the first human in outer space and the first to orbit the Earth. 2011 sees the fiftieth anniversary […]]]>

“On 12 April 1961 Yuri Gagarin became the first human in outer space and the first to orbit the Earth. 2011 sees the fiftieth anniversary of that event…”

As part of my ongoing work on Binary Dust, I am speaking at Heavenly Discourses on Sunday 16th October 16:45 – 18:00. PANEL: Music

I’ll be presenting new work (including sounds and pictures derived from ALMA) that my great collaborators, Andrew Newsam and Julie Freeman, have helped me with (thank you!).

Here’s the abstract of my paper. I am delighted to have been accepted – esp. as I’m one of the few/the only non-institutional presenters at the conference.

The utterance of a cosmological model?

A conjoining of languages, Acoustic Cosmology is an attempt to describe our audible worlds – a 21st century progression of the music of the spheres – a narrative of acoustic sculpture within n-dimensional space. With no intentional stance on sound as a cultural construct or phenomenology, we openly explore links between cosmology and music, using the language of mathematics and sonic art.

Building on the works Trevor Wishart and Jean-Pierre Luminet, and developed by professional astronomers and musicians, we question and connect the fabric of these non-verbal languages.

Using cosmology and sonic art as its basis, this paper will provide a journey of discovery – a basis for discussion in the junction between music and astronomy, opening up new methods of comprehending scale, connection, depth and complexity. Sound examples and visuals will be included in the presentation.

]]> 481 Project Sigma https://dgen.net/0/2003/03/29/project-sigma/ Sat, 29 Mar 2003 14:42:31 +0000 https://www.dgen.net/0/?p=4745 © 2003 Precis Sigma aims to demonstrate the human scale of infinity. Its aim is to make human infinity a tangible concept and to allow its […]]]>

© 2003

Precis

Sigma aims to demonstrate the human scale of infinity.

Its aim is to make human infinity a tangible concept and to allow its exploration, ultimately showing its diversity as an opposition to the general trends to reductionism and globalisation.

Sigma is the sum of all perceivable knowledge in a single, known dataverse.

It reflects the progress of humanity to analyse, store and measure everything around them.

It demonstrates that technology has already reached a point where all that can be measured, can be measured.

Whereas the search for the Grand Unification Theory seeks to unite all of science into a single reductionist theory, Sigma pre-renders all possible outcomes of the known universe into a single point and thus demonstrates the finite and infinite nature of the universe.

  • Number of protons in the universe: 10^17
  • number of synapses in the brain: 5^11
  • the combined number of synapses on earth: 5^11 * 6×10^9 = 3×10^21
  • the combined number of synaptic events per annum for Earth (avg. 10 per second) = 10^30
    • and for the entire history of “thought” = 10^30 * 5,000 years = 5*10^35
  • number of web pages = 3×10^8
  • number of grains of sand on earth: 10^12
  • number of CDs produced = 10^7
  • number of books published = 3*10^7
  • number of permutations of an encrypted 128 bit message= 2^128
  • storage required for all music produced = 500TB = 5*10^14 bytes
  • storage per $1000 following Moore’s law (18 months): 0.25TB 2003
  • so, for $2m you can store all the CDs, today.
  • In 10 years: $35,000
  • In 20 years: $500
  • number of words in Sigma::everyWord: 10^100
  • number of images in Sigma::everyImage: 10^150

Things not covered by Sigma.

  • Other infinities. For example, Pi cannot be stored. It may be explored within Sigma to any precision but not stored in its entirety.

Discussion

The nature of the universe is believed to be infinite. That as finer detail is resolved, greater detail is discovered. However, the human species is self-defined by its limitations as a biological organism. It has only stereo vision and hearing. It has monaural verbal communication, a sense of touch, taste and smell. 

Any quantifiable data can be digitised. This includes everything from the concept of love as described in a poem, to the image of an individuals face in 3-dimensions, to the sound of a train, to the structure of DNA.

Using simple arithmetical models, any data pattern can be generated.

It is not, however, a “random” number generator. All output is predicated on some form of deterministic input.

Sigma has two distinct implementations.

  1. The first pre-generates all possible data and stores it in a single place. The aim of which is to demonstrate that human knowledge is quantifiable. The necessity to actually implement this is debatable (and costly), but the principle is solid.
  2. The second is a collection of tools to enable the exploration and navigation of this “ultimate” data space. This may be done without the pre-generation of any data. The data is only generated when a request is made that it be observed. (note there are many parallels between this approach and classical quantum mechanics – e.g. Schrodinger’s Cat). The system exists in all possible states until an observation is made that “collapses” the decision tree at that instant to create an observable result.

Following late 20th and early 21st century trends, it is intended that the entire dataverse and, importantly, its methods of access be patented. Thus, any subset or realisation of any data realisable with the system will be the patent of Sigma, and any future discovery via other means would have to site Sigma as prior art. Since Sigma encapsulates all data, this means anything that can be digitised is patented by Sigma.

Implementation

The implementation of the project is split into phases, each covering the exploration of a specific subset of the dataverse.

Dataverse subset definitions

  1. The written word (everyWord)
  2. The audible sound (everySound)
  3. The static image (everyPicture)
  4. The moving image (everyMovie)
  5. The stereoscopic image (everySpace)

Implementation of the everyWord

everyWord is the generation of every possible combination of 100 words (each of 10 characters or less using a 40 characters alphabet). This is described numerically as

  • 40^1000 combinations

If this is reduced to dictionary words, say 60,000 words, then every 100-word combination can be represented by

  • 60,000^100 combinations

Sigma Definitions

The dataverse is the set of all information that can be stored digitally.

Within the Sigma project, this potentially infinite set is constrained within the constraints of the data being investigated and the human perception of the output data.

The definition of the text/word dataverse is the number of characters in the western alphabet, namely 26 letters, and 13 punctuation marks (space, dash, comma, full stop, semi-colon, question mark, exclamation mark, brackets, quotes and colon).

The definition of the image dataverse is based on a maximum of 1600×1200 pixels with 24-bit colour depth.

The definition of the video dataverse is based on a maximum of a series of images played at up to 30 frames per second.

The definition of the stereo image dataverse is based on a maximum of 1600×1200 pixels with 24-bit colour depth, one for each eye.

The definition of the audio dataverse is based on a maximum of 44.1KHz sample rate with 16-bit resolution in stereo.

See also

https://en.wikipedia.org/wiki/The_Library_of_Babel

]]> 4745 Internet Music Survey https://dgen.net/0/1993/01/10/internet-music-survey/ Sun, 10 Jan 1993 12:00:08 +0000 http://www.dgen.net/blog/?p=40 Working as a researcher at the University of Glasgow, I surveyed all the free digital signal processing (DSP) and music software available on the Internet. […]]]>

Working as a researcher at the University of Glasgow, I surveyed all the free digital signal processing (DSP) and music software available on the Internet. In those days you could – there were about 120 references to “music software” if you did an Archie search…

Glasgow was ‘quite unusual’ that we had a lab of over 30 NeXT computers networked across the music, comp sci and engineering departments, and the recording studio – and I was one of only 3 people who knew how it actually worked. We were doing ‘realtime audio over the internet’ in 1993.

Such early days…funny to think that years later it’d be so relevant to my later work.

Copy for archive below:

Music ftp sites

The information in this document was derived from: a file maintained by Piet van Oostrum who is setting up a midi archive at ftp.cs.ruu.nl [131.211.80.17] and would welcome submissions; from the results of searches carried out on behalf of the Music TLTP Project by Gavin Starks of the the Dept. of Music of the University of Glasgow; and from information provided by colleagues here at the University of Leeds: Dr. David Cooper Director of the Electronic Studio in the Department of Music and Kia Ng of the School of Computer Studies. (See also Music FTP Sites without information about contents). Neither file is actively maintained by me though I welcome any corrections and additions that I am sent by readers.

]]> 40