How to measure innovation ecosystems
How do you measure capability and outcomes of an innovation ecosystem? The answer is involved but worth finding a resolution.
The value and need to measure
The focus here is on ‘how’ you measure, with an emphasis on measuring visible aspects including jobs and investment. These outcomes can be considered ‘lag measures’ and are often the result of other factors such as talent, culture, activity, and infrastructure. I consider ‘what’ to measure in a previous post here.
Corporations, universities, community groups, investors, and governments are making significant investments into innovation. This is for good reason, as these investments benefit from the network effect of innovation ecosystems. The Australian bureau of Statistics estimates Australian businesses spent between $26 billion and $30 billion on innovation between 2014 and 2015 and earned $60 billion from innovative goods and services during this same period. Regions where there are industry-facing research organisations have a greater number of entrepreneurs and patent and trademark applications. Small business that collaborate introduce twice as many innovations as their non-collaborative counterparts.
Building an innovation ecosystems is an economic development exercise that benefits society overall. This places the investment in the remit of national, state, and local levels of government. The Australian federal government maintains their $1.1 billion National Innovation and Science Agenda. States follow suite with programs including Victoria’s LaunchVic program, Western Australia’s Innovation Strategy, Innovation South Australia, Canberra’s CBR Innovation Network, the Tasmania Innovation and Investment fund, Northern Territory Business Innovation Support Initiatives, and the Advance Queensland initiative.
Measuring the outcomes of these investment in a timely manner is critical. The public, media, and oppositional parties are quick to look for a return on the innovation investment. The impacts of innovation investments can take years to realise and data can be difficult to obtain. We measure to better understand the effectiveness of investment and to better inform more rapid and higher quality decisions on innovation policy and programs.
Current approaches
Government data and Consultancy reports
The speed of measuring innovation impact is not in keeping with the pace at which decisions need to be made. The majority of the data in the federal government’s Australian Innovation System report is over two years old and focused on the national level. The 2016 Regional Australia Institute’s interactive insight innovation provides a great methodology to compare regions across Australia, but many of the data points come from the 2011 Census.
Consultancies and not for profit associations create reports to fill in the blanks and answer questions not readily available in traditional data sets. Yet these reports can also suffer from time lag and do not always provide information for decision making in specific regions. Examples such as the StartupAus Crossroads report, Microsoft’s JoinedUp report, or the Regional Queensland Startup Ecosystem Report are quality and necessary bodies of work and the regular Crossroads report has become a valuable reference, but even with annual updates can become outdated in the fast-paced world of innovation.
Analytic platforms
As would be expected, the market responds to this need with a range of real-time analysis tools. Platforms that provide a means to analyse data by region include Crunchbase, AngeList, Datafox, CBInsights, PrivCo, Mattermark, Funderbeam, Pitchbook, and Gust. A good comparison of a few of these can be found here.
I signed up for a few of the platforms to determine if they might be an option to measure aspects of innovation ecosystems related to investment and business activity. As a test, I downloaded data from Crunchbase for Australian companies as of September 2017, as many platforms reference Crunchbase as a source.
The data for Australia included 8,272 records. Of these, 1,203 had Queensland listed as a city or state. Of the companies identified as being in Queensland, around 243 had a founded date greater than 2011, assuming they started in the past five years. For efficiency, I excluded the 359 records that did not have a founded date.
I then reviewed each of the 243 records to 1) determine if they were still in operation, 2) assess their business model to see if they were a startup, and 3) associate them with an ABS industry sector. This involved website searches and comparing datasets in other platforms.
The review left 120 companies still in operation that could be considered to be a startup. The test for a “startup” is largely accepted as the potential to reach a significant market and leveraging technology to scale.
The inaccuracies and gaps in data, particularly with early stage businesses, made it impossible to rely on without significant data cleansing. If some data is missing or inaccurate, then all data needs to be questioned. Analytics platforms have teams of people scouring the web performing research similar to the exercise I undertook, although I expect they do it more efficiently. The platforms solve a need to identify fast growth potential companies and companies that are receiving investment or significant PR, but are not suitable as yet to provide a reliable measure of startup activity in an ecosystem.
Surveys (Startup Muster, Victoria) and curated lists (Vest)
Other platforms worth considering include the Startup Muster survey and the curated list Vest.
Startup Muster is an annual survey identifying Australian startups. The survey continues to improve year on year and is moving to a platform to provide participants with connections. The benefit of the survey is the broad reach and the connections feature is aimed at incentivising greater participation. The 2016 survey validated 685 verified startup founders, 239 potential founders and 474 startup supporters. As its name suggests, Startup Muster is a representative sample dependent on startups and businesses nominating themselves for inclusion. It remains however the best current option for close to real-time information available.
Victoria’s startup survey is also worth a mention. The survey was released in March and the high quality report was delivered in August. The survey is good example of a state-based approach to measurement, and also raises the question as to how many surveys can be done without experiencing survey fatigue.
A second platform to consider is Vest, a repository of Australian startups designed and managed by design and digital agency JosephMark. The platform captures similar information as the other analytics platforms, but the content appears more accurate given the local curation.
The examples above demonstrate the two broad approaches to measurement: participant data entry through a survey or platform, and third-party curation. Survey options are challenged by providing sufficient incentives for user entry on an ongoing basis and providing access to data. Platforms curated by third parties can be labour intensive and depend on the curator to have sufficient access to the target market and depth of information in that market.
The end solution needs to source information from a platform that sufficiently solves a business need for those who own the data. The solution also needs to provide privacy and data ownership considerations to allow access to the data. This is my current focus, to provide a platform that addresses a business need where there is a high percentage of ecosystem activity.
My why and where to from here
My phone buzzed as I prepared to hit publish on this post. I looked down and saw a news notification that at least 50 people have been killed in a shooting in Las Vegas.
I am angry, frustrated, and deeply sad. This academic analysis of innovation ecosystem measurement seems so distant to the pain happening on the other side of the world. There is no comparison, but it is a pain echoed from natural disasters, war, and poverty globally.
And this leads to my why. I believe that innovation, entrepreneurship, and ecosystem development may be the only pathway to address the wicked problems we face. It is clear that repeatedly experiencing the same pain and responding with traditional approaches will not create a solution. I feel helpless and ineffective with my focus on regional startup ecosystems. Surely addressing gun control requires a drastic policy change and political shift. Yet my hope is that fostering the right people in the right environment at the right time will create the right culture, get the right people in a room, and develop new solutions we have not even considered.
This is the reason for my focus on measurement, to provide better information for improved decision making, so we can more rapidly develop entrepreneurs to address some of the systemic challenges facing society. Reliable feedback loops will inform us as to how we can more effectively help a greater number of people make an impact.
I look with interest at overseas models such as Israel’s Startup Nation Central, the work coming out of the Kauffman Foundation and KCSourceLink in Kansas City, and Nesta in the UK. The challenge is global, addressed at national and local levels.
My current approach includes development of a platform to measure at the source of entrepreneurial activity, feeding into and validating other platforms where possible.
We will collectively continue towards achieving more timely, accurate, and accessible information about the innovation ecosystems that drive entrepreneurial activity in our regions.
Thank you for those who have connected me with people doing this in Australia and globally. Any and all collaborations are welcome.