Case Study: Australian Government Data Reforms 2015-2019: the evolving role of the Australian Bureau of Statistics
Topic
Collaboration across the Australian Government to ensure that the national data system can meet the emerging and future data and analytics needs facing the Australian Public Service.
Situational description
The Australian Bureau of Statistics (ABS) is Australia’s national statistical agency, providing trusted official statistics on a wide range of economic, social, population and environmental matters of importance to Australia. As an independent statutory authority, the ABS has a legislated function to be the central statistical authority for the Australian Government and to provide statistical services for the state and territory governments. In this capacity, the ABS has an important leadership role, to maximize the use of public data for statistical purposes, and to provide technical advice, develop standards and provide assistance to Australian and state/territory governments in relation to statistics.
The ABS is also the primary (but not only) Australian Government Accredited Integrating Authority (AIA) within the whole of the Australian Government reform initiative, the Data Integration Partnership for Australia (DIPA). In this role, the ABS plays a significant role in helping Government to maximize the use of public data and statistics to enable better informed government policy and service delivery. As an AIA, the ABS undertakes high-risk data linkage projects on behalf of data custodians across Governments and provides safe access to unidentified, integrated microdata to government and non-government researchers.
Over the last three years, the ABS has undertaken an increasing range and volume of work to support the evolution of the Australian National Data System.
Starting point
In partnership with Australian Government data custodian agencies, the ABS began developing a business longitudinal linked dataset in 2005 and a person-based integrated dataset in 2010. The person-based integrated asset and infrastructure was initially championed by the ABS and the Department of Health, before growing into a six-agency partnership project. Recognizing increasing interest in data by government, the ABS worked with its partners to test the feasibility of linking various cross-portfolio datasets longitudinally. However, access to these integrated microdata was limited to partner agencies for exploratory research purposes, and progress in developing integrated data methods and assets was slow without dedicated resourcing. Since 2017, the production and expansion of these assets has been further enabled by government funding for the DIPA initiative.
During this time, the Australian Government, in recognition of the growing need for enhanced management of public data, stated in its Public Data Policy Statement that data “is a strategic national resource that holds considerable value for growing the economy, improving service delivery and transforming policy outcomes for the nation”. To leverage its data holdings, the Australian Government committed to:
- harnessing the value of data;
- publishing, linking and sharing data to stimulate innovation and create opportunities; and
- optimizing the use and reuse of public data.
In May 2017, the Productivity Commission released the report of its Inquiry into Data Availability and Use, providing a further catalyst for change in the Australian data landscape. This inquiry looked at options to increase the availability and use of public and private sector data and found significant barriers to data sharing including:
- a culture of risk aversion
- a dense web of legislative requirements
- a lack of a whole of government approach.
An ABS senior official was outposted to work on the Australian Government’s response to the inquiry, which resulted in a Government commitment of $65 million over four years to reform Australia’s data system. Reforms aim to streamline and modernize how the Government shares data, while ensuring privacy and security, to support better decisions, policy, programs, services and research. More specifically, the reforms include establishing the Office of the National Data Commissioner (ONDC) and introducing Commonwealth Data Sharing and Release Legislation. The Department of the Prime Minister and Cabinet is leading these reforms, and the ABS is contributing substantially to their design through its technical expertise and advice.
Within the ABS, work to support reform of the national data system is centered within the Statistical Data Integration Division, under the Deputy Australian Statistician, Census and Data Services. A small team of six staff provide direct support and technical advice to the Office of the National Data Commissioner, whilst approximately another 140 staff work across the end-to-end data integration process. Around half of these staff are involved in producing and providing safe access to integrated data. Remaining staff undertake governance, privacy and other risk management activities; engage with researchers and data custodians; build and enhance key people and business integrated data assets; maintain safe and secure data environments, systems and processes; develop new infrastructure; and undertake transparency initiatives to maintain public trust and social license.
New developments
The ABS plays an important role in these data system reforms as Australia’s national statistics office, with a wealth of experience in collecting, analyzing and publishing data using secure and privacy preserving methods, across a suite of economic, demographic, social and environmental areas. This role has been recognized by Government, in providing the ABS with funding:
- under the DIPA to develop enduring integrated data assets, which bring together data from across the Australian Public Service, and build linkage capabilities, technical infrastructure and tools to enable greater use of public data across government to inform research and policy. As the lead AIA, the ABS received the largest funding to a single agency under this program; and
- to provide technical support to the National Data Commissioner.
As part of the national Data System reforms, new formal governance arrangements were established across the Australian Government (Figure 1). The ABS is an active participant in all levels of data governance fora. The Australian Statistician is a member of the Secretaries Data Group and the National Data Advisory Council; the Deputy Australian Statistician, Census and Data Services is a member of the Deputy Secretaries Data Group and DIPA Board; and Program Managers and Executive level staff are involved in the Data Champions Group. Through its representation on these formal fora, ABS is a partner in driving strategy, work program and innovation in the Australian data landscape. ABS staff also participate regularly in technical working groups to progress collaboration and knowledge sharing as system changes are implemented. The ABS has established integrated data asset user groups to provide advice to the ABS on products, data use and analysis methods, and to enable members to share analytical results, work collaboratively and share knowledge.
In parallel to national data system reforms, similar legislative changes and cultural reforms are occurring within Australia’s constituent states and territories. The ABS has been involved in and supported a number of these initiatives in a variety of ways. The ABS provides a Director level strategic outposted officer in each jurisdiction to engage with, and support, state and territory data and statistical needs, including maximizing the value of public data. Through these staff, awareness of the availability and uses of the MADIP and BLADE key assets, as well as opportunities for data sharing between Australian and State and Territory Governments have grown. 14 linkage and analytical projects for states and territories are currently underway in 2019-20. By sharing its data integration capability, secure integration and analysis environments, and its integrated data products with jurisdictions, the ABS has been able to extend the statistical services it provides to jurisdictions as the Australian NSO.
The ABS is also working closely with the Australian Institute of Health and Welfare (AIHW) to develop interoperability methods to enable data integrated by either the ABS and AIHW to be brought together safely and efficiently. This is of particular value for state and territory data in the health and welfare sectors, which has already been shared with and integrated by AIHW, enabling it to be combined with a broader range of Australian Government data through MADIP.
Whilst provision of trusted statistics remains central to ABS’ role, in recognition of the changing environment and opportunities presented by sector-wide data reforms, ABS’ strategic direction has evolved to reflect the increasingly important role of partnerships in informing Australia’s important decisions, and the development of new statistics to support emerging priorities, both through traditional direct collection methods and re-use of public and private sector data.
The ABS also recognizes it has an important role to play in building capability across the Australian Public Service to enable embedding of data analytics into the design of policy, program and services to best meet Australians’ needs.
Actions taken
The Australian Government’s response to the Productivity Commission Report on Data Availability and Use included an announcement of three reforms:
- A new Consumer Data Right giving citizens greater transparency and control over their own data;
- A National Data Commissioner to implement and oversee a simpler, more efficient data sharing and release framework; and
- New legislative and governance arrangements to enable better data use while ensuring appropriate safeguards are in place to protect sensitive information.
The ABS is providing technical advice and expertise on two of these three key reforms – advising the National Data Commissioner and contributing to new legislative and governance arrangements for better data use.
The National Data Commissioner is responsible for overseeing and regulating the data sharing system and will drive change and support best practice for data across the Australian Public Service. To support this role, the ABS has collaborated with the Office of the National Data Commissioner to develop a Best Practice Guide to Applying Data Sharing Principles, which was published in March 2019. Although the principles are a requirement for using the Data Sharing and Release legislation, they are intended to be applicable for all data sharing activities. A consultation process is currently underway to inform the introduction of Commonwealth Data Sharing and Release Legislation.
The Government has established a new governance mechanism, the National Data Advisory Council, to advise the National Data Commissioner on ethical data use, community engagement, technical best practice, and industry/international developments. The Council comprises members from the Australian government, business and industry, civil society groups and academia. The Australian Statistician is one of the Government representatives along with the Australian Information and Privacy Commissioner and the Australian Chief Scientist. The Council is working to support the National Data Commissioner to find the optimal balance between streamlining the sharing and release of data and ensuring the protection of privacy.
In addition to these reforms, the ABS is also contributing to whole of government data integration capabilities. The ABS is a partner in the Data Integration Partnerships for Australia (DIPA) program, an investment to maximize the use and value of the Government’s data assets, which is leveraging collaboration between over 20 Commonwealth government agencies. Through DIPA, the ABS is improving technical data infrastructure, enhancing whole of government data assets and leveraging these to support cross-portfolio analytical projects on a range of economic, social and environmental issues.
The DIPA program has provided opportunities for ABS to expand its role as an NSO. At the current time (October 2019), ABS is playing a range of critical roles within the public data system: as steward of both the business (BLADE) and person (MADIP) based integrated data assets; co-developer of new place based assets, such as the Location Index (LOC-I[1]) API to streamline data integration of geospatial data; providing data brokerage services to assist researchers and analysts ensure that their use of integrated data is feasible and appropriate as they scope their research proposals; opening up and streamlining safe access to unidentified integrated data to a wider range of government and non-government users; providing expert integrated data analysis, advice and support for cross-portfolio research projects; and developing and delivering training in integrated data linking and analysis to build this capability across the Australian Public Service.
The ABS is also collaborating with the Data Champions Network, the Department of Health and the Department of the Prime Minister and Cabinet, on behalf of the Secretaries Data Group, to investigate a range of strategies to further uplift data literacy and analytics capability. It is anticipated there will be three areas of focus:
- Implementing a data professions stream across the Australian Public Service, with consistently defined roles, skills and competencies. This would create a professional network of staff with data and analytics skills, help the public service attract and retain qualified and capable data practitioners and create clear career pathways.
- Implementing a more coordinated approach to data and analytics capability across the Australian Public Service, commencing with an audit of existing capability building activities and relevant external initiatives.
- Further increasing collaboration beyond agency boundaries, for example, by using more flexible working models.
Results
To assess progress in undertaking data reforms on a national scale, the Government commissioned a Review of Australian Government Data Activities (Department of the Prime Minister and Cabinet, July 2018) to provide a snapshot of data activities and data resources across government agencies. The Review identified four key areas where Government reforms to the public sector data system are yielding improved outcomes:
- Access to public sector data is improving
- Agencies are using data more efficiently to provide agile and effective government services
- Public sector data skills and capabilities are improving
- Government data protections are building community trust and confidence in how public sector data is collected and used.
Undertaking reforms across a national data system takes time. While good progress has been made, further work is required to develop, implement and evaluate reforms. The ABS is working with its government partners to continue to drive change and build momentum to develop data infrastructure and capabilities for the future to expand better data sharing and data use.
In recognition of the fast-paced changing data landscape in the ‘information age’, the Government is currently developing a refreshed Public Data Strategy Roadmap across five pillars: legislation and regulation, governance and culture, capability, technical and trust. The ABS is partnering with the Department of the Prime Minister and Cabinet to develop the strategy and roadmap, through its role as a technical adviser to the National Data Commissioner and in its capacity as Australia’s national statistics office.
With the current phase of DIPA in its final year of funding, the ABS is also working closely with its partner agencies across government to leverage the data sharing and capability building initiatives that have been built under the DIPA and create further opportunities to embed data system reform. Opportunities are also being pursued to extend data sharing, integration and access infrastructure and delivery of training to states and territories over the next few years.
Lessons learned
In seeking to reform the Australian data system, many opportunities and challenges have arisen, and will continue to arise, as the legislative environment, culture, public trust and technical capability across government change. As for many reforms, there is a general tendency to set up dedicated new structures and new organisations to move new ideas forward, rather than seek to drive reforms through existing arrangements, and this is playing out in the Australian National Data System context.
The ABS cannot navigate the complexity of the Australian public data environment successfully on its own as risks and obligations are often shared, and sometimes competing, between actors in the data system. A key learning to date has been the importance of ongoing collaboration and effective engagement with ABS’ partners across Government. This is most apparent in managing the need to continually balance maximizing the public value and utility of data, with maintaining privacy and trust in the ways the Australian Government shares and uses data.
At the start of the reforms in 2015-17, there was widespread appreciation of the value of data across Government, but little understanding of the complexities and effort involved in managing its safe handling, production and use. Whilst there is a continued focus on technical solutions for safe and efficient data sharing, integration, access and analysis, appreciation of the data governance capabilities and judgments that are an essential enabler of data sharing is growing. However, it is still far from being universally mature.
The realization of the value of the volume of data continues to be hindered by the slow (but growing) recognition across the Australian Public Service of data capability as a specialized and professional capability stream. In contrast, progress in adoption of digital technologies across government, especially those that facilitate service delivery to the public, has progressed relatively quickly. This has been supported by concerted efforts to improve the public’s experience of interacting with government and enabled by the long standing (but rapidly evolving) establishment of a professionalized ICT stream across government.
The trust pact between an NSO, data custodians and the public is both fragile and essential, not only to achieving the goal of maximising the value of public data, but to its fundamental operations as an information organisation. Navigating the complex legal, privacy and social licence obligations successfully requires continued effort, stakeholder collaboration, public communication and engagement, and problem solving.
Prior to the commencement of DIPA and other national data reforms around 2017, the ABS was a trusted provider of national statistics, but was not well geared towards rapid data innovation and provision of data services to government. The DIPA initiative, in particular, has provided an opportunity for ABS to work collaboratively with both technical and policy agencies to create a stronger and better functioning data system, to re-cast its role and to broaden the value it provides, as an NSO, to Government. There is more work needed, but ABS is playing an increasingly active role in building capability across the Australian Government in data governance, data linking and analysis of integrated data.
[1] In partnership with Department of Energy and Environment, Department of Agriculture and Water Resources, Department of Industry, Innovation and Science, Geoscience Australia and CSIRO.