Grant to IDInsight to support NITI Aayog in developing  the National Data Analytics Platform (NDAP) for India

It is a long standing mantra in public policy that data and evidence driven policy making lead to better outcomes. But years of experience tells us this is easier said than done. Take the case of a principal secretary in a state in India, trying to tackle the vexed issue of malnutrition. If she is data driven, it can take months of effort and many complex steps – from leveraging personal networks to access the right files from other departments like food distribution, child welfare, and education; to getting a data entry operator to enter the information (which may be in PDF format, even if it is ‘digitized’); to identifying researchers who can analyse the data and derive insights, and finally getting a consultant to help visualize and present the analysis in a compelling way to all stakeholders involved in decision making. Covid-19 has further surfaced the challenges (and benefits) of accessing and analysing public data from disparate sources to make literally ‘life-saving’ policy decisions

These challenges in accessibility and usability of data are faced not just by government officials, but many others who use public datasets in their work. Companies creating products and services for India’s ‘next half billion’ internet users, need demographic and socioeconomic population data to inform their market penetration strategy for different regions. A journalist tracking environment & climate change needs to access data from pollution control boards, the meteorological department, health, agriculture and public transport departments, then conduct analysis across these different datasets to tell a better, more holistic story to inform public opinion.

For all these different types of users of government data, large volumes of rich datasets are in fact, technically ‘available’ – but rarely accessible in a format that is useful for analysis. Even highly sophisticated policy researchers spend significant time and resources to request access to datasets or download them, only to find that they are in unwieldy formats. Every researcher has to then repeat the same task of  converting these datasets to machine readable formats, cleaning, organizing and codifying data to prepare it for analysis.

In this context it is heartening to see that NITI Aayog, the ‘think-tank’ of the Government of India, has developed a vision to transform India’s open data architecture. The stated ambition of NITI Aayog’s National Data and Analytics Platform (NDAP) is to “serve as a single point for accessing data across all Ministry(ies) of Government of India combined with intuitive visualization and self-service analytics”[1]. If it succeeds, NDAP will become a valuable resource for data driven policy and decision making, research and reporting. It will also empower citizens by bringing about greater transparency and accountability. In the medium-long run, by exposing government data to public scrutiny, and by driving adoption of ‘data standards’ and SOPs, NDAP can lead to significant improvement in the quality of data collected and maintained by the government. Beyond these direct impacts, there is an opportunity to shape the design of the platform as a “digital commons”, organizing public data in a manner that can be used by entrepreneurs to create products & services on top for various end-users.

At Omidyar Network India, we believe ‘National Open Digital Ecosystems (NODEs)’ have the potential to transform societal outcomes for India, but only if the critical ‘non-tech’ layers of such ecosystems are in place – these include accountable institutions, transparent rules of engagement that ensure protection of data privacy and fair value distribution, and a vibrant community of users as well as collaborators. We believe NDAP can be a powerful national data asset, if designed well, based on a deep understanding of end-user needs, a robust underlying data integration model to enable comparison across datasets, and an ‘open’ tech architecture based on the principles of modularity, interoperability and extensibility, to allow users to build various applications on top.

With this vision, Omidyar Network India is supporting NITI Aayog during the design phase of NDAP, by supporting a team for the Program Management Unit (PMU), led by IDinsight, and Development Data Lab. Before the tech platform for NDAP is built, it would be important to get inputs from the end-users of NDAP, as well as technology and domain specific advisors. The PMU will play an important role in supporting NITI Aayog. It will conduct user-research to understand the pain points and needs of data users from research, civil society and the government so that the platform is fit for purpose. It will use these inputs to help design the front-end features and the back-end data organization model. It will also serve as the ‘glue’ for the project, coordinating across the ecosystem, as the success of NDAP will depend on active collaboration with the data provider and data user community.

A few initiatives like data.gov.in and the DISHA dashboard, have already laid the foundations for India to take the next steps in its journey towards open data. Data.gov.in was a pioneering effort to open up access to public datasets. DISHA emanated from the need to give our Members of Parliament a constituency level view of progress made by various schemes, so they could become a more effective custodians of the interests of their constituents.

While NDAP will learn from (and source data from) these older platforms, it will also enable India’s open data ecosystem to leapfrog in the following ways:

1. User-centric design and advanced front-end features: Government data sets are typically uploaded into the public domain usually without additional features that can unlock their value (e.g., customizable visualization, machine-learning enabled ‘intelligent’ queries to identify trends, etc). NDAP aims to create analytical and reporting tools using the latest technology, to facilitate usage of public data.
2. Ease of integration across multiple datasets: Current public data repositories are managed in siloes by multiple line departments/Ministries, and therefore cannot be combined to provide a ‘single view’, at say a geographic unit of analysis (e.g., co-relate mortality rates with access to sanitation and education in a block). NDAP can resolve this by creating an underlying data organization layer for the major datasets that are accessed through it, with ‘common identifiers’ to help users get an integrated view across datasets. In time, NDAP can introduce national standards for maintaining all such public datasets so that data siloes are broken down at the source.
3. Enabling wider range of use-cases through third-party apps: Currently public data is maintained in static web services, with heavy post-processing required to unlock benefits (e.g., Urban Clap cleaning up NSDC’s data to find skilled workers). NDAP aims to provide real-time access to public data in easily usable formats, which can lower barriers for private entrepreneurs to leverage government data for creating third party services (e.g., the National Weather Service of the United States government provides real-time meteorological updates to enable third-party services such as Accuweather which are widely used by individuals and enterprises). At the same time, it will be important for NDAP to create appropriate ‘standards’ to ensure that personal data of residents is protected, and safeguards are in place to prevent misuse by private entities.

At Omidyar Network India, we are committed to supporting ‘GoodTech’ – har zindagi behtar, i.e. technology which is both beneficial and responsible. The National Data Analytics Platform is a great example of ‘GoodTech’ – it has the potential to transform the national open data ecosystem in a way that can accelerate societal outcomes for India across sectors, while at the same time setting standards for the governance, accessibility, and ‘rights of use’ for government data. It is our hope that this effort will be an empowering tool for the health secretary trying to tackle malnutrition, the climate change researcher, and the civil society activist holding the government accountable, as well as a valuable resource for innovators creating products and services for every Indian.

[1] NDAP RFP circulated by NITI Ayog to procure a tech vendor for the platform

‍