Open Data – The Promises, the Disillusion and the Panacea | Part 2. The Disillusion

A new era of open data is encouraging diverse public expectations about the role of open government. Changes in our technological landscape allow for greater connectivity and transparency between governments and their citizens. With increased public oversight and engagement comes a need for deeper understanding of the implications of open information and the context of their applications.

The consumption of open data and how it addresses specific economic and social issues is still being researched and, while there have been case studies examining the use of open data for specific companies, a broader view of the global impact is yet to be fully explored. 

Commentators of open data viability have identified gaps in the open data market between data providers and users, and aim to close this gap by shifting the focus from data release to data quality. However, traditional market theories that examine the separate roles of data producers and consumers may, due to the nature of digital goods and market interactions, be unable to adequately describe the dynamics of an evolving digital ecosystem. 

An open digital market offers a new generation of public goods that are not just non-rivalrous and nonexcludable, but also modifiable or linkable’, allowing for the creation and proliferation of new goods that may be consumed with little to no diminishment. New digital technologies and analytics such as machine learning, data visualisations and sophisticated user interfaces have the capability to better facilitate digital market transactions through the provision of better information and services, and would serve to dynamically change the responsiveness between data producers and consumers.

Whether or not open data has delivered the economic impacts expected across various industries, there is a renewed focus on how efficiently open data translates to real economic gains. Just as crude oil needs to be refined into petrol before it can be used by automobiles for transportation, open data may require processing before it can be utilised by new innovative apps for the public to gain substantial benefits. 

But it’s much more complex with a mutable commodity like data. What determines how data should be processed depends primarily on how it is to be used or consumed. Hence, the need for ‘open data infomediaries’, as recognised by many in the open data community, and the importance of data literacy are critical in sustaining a vibrant open digital market ecosystem, yet remain underdeveloped. 

The role of these data infomediaries could span between cleaning and managing data to repurposing it or creating new apps for data interpretation, so that the general public could extract insights specific to their decision making needs. Consumer interaction with open data products would also benefit from innovation such as a new generation of smart apps, which may be able to harness Artificial Intelligence (AI) for cognitive processing, so that users can seamlessly function more efficiently in a complex big data environment.

The real challenge lies in determining to what extent the current open data supply is able to provide consumers with the information needed for significant benefits to economic, social and community outcomes in the first instance. This requires a detailed assessment of the data itself.

Over the last few years there have been several measures of open data such as the Global Open Data Index which aims to provide a snapshot of government open data, and the Open Data Barometer which aims to uncover the prevalence and impact of open data initiatives around the world. While these measures have been useful in tracking the progress of open data releases, implementation and impacts, they are extensively survey-based and fall short of interrogating the data directly. 

Apart from the obvious advantages of cost effectiveness, timeliness, consistency and potential improvements in accuracy, data-driven measures also have the capacity to employ AI to provide new insights into not just the current state of open data in real-time, but also to diagnose and identify gaps in data supply which may impede optimal public innovation.

What determines the potential impact of addressing these gaps ultimately depends on the inherent value of the data and how it may be used. While datasets may be identified as high-value if they play a significant role in directly contributing towards social outcomes, the true value of a dataset also depends on its ability to enable other datasets or applications to contribute towards greater efficiencies and improvements in our society. 

How industries use open data to supplement their own private data for better insights and what portion of their economic and social benefits may be attributed to open data is difficult to quantify. Nevertheless, it is useful to estimate this value in determining to what extent government and industry should support the provision of open data in the future.


Dr. Audrey Lobo-Pulo is a Senior Adviser in the Australian Public Service, and is an advocate for open government and open source software in government modelling. Views expressed in this article are those of the author and not necessarily those of the Australian Government, nor of the Global Digital Foundation, which does not hold corporate views.