IDEAS:Rather than coming to people in a flash of inspiration, argues author Steven Johnson, the best ideas are generated over time, and in the right conditions

IT’S AN UNFORTUNATE truth that we often take ideas for granted. We assume they just happen, little gifts from the brain, and that some people are more prone than others to having good ones. Above all, we are under the misapprehension that the best ideas come to geniuses in flashes of fully formed inspiration – Eureka moments. These assumptions illustrate how hard it is to conceptualise where ideas come from and how difficult it is to encourage better ones.

Tackling such theoretical terrain is a difficult mission, which is why Steven Berlin Johnson received unanimous praise when he published Where Good Ideas Come From: A Natural History of Innovationlast autumn.

Johnson is an acclaimed popular science author whose work has often concerned groundbreaking innovators. His best-selling Everything Bad Is Good for You: How Today's Popular Culture Is Actually Making Us Smarteris required reading for anybody interested in digital culture.

“This was the hardest of all seven of the books that I’ve written, by a wide margin,” he says, on the phone from his home in Brooklyn. “I knew I could make a lot of connections, but I wanted it to not just feel like a stream of consciousness.”

Johnson seems an exceptionally sunny individual, expansive and generous, given to frequent laughter. There is a sense of irrepressible enthusiasm about him, a giddy optimism unusual in a 42-year-old, though maybe that’s because we talked just as the “hyperlocal” news site he co-founded, Outside.in, was being bought by AOL. Yes, on top of being a celebrated thinker, Johnson is also a web entrepreneur – he not only writes about good ideas, he acts on his own, too.

Johnson sees the web as analogous to the diverse biosystem of the coral reef: a teeming platform that encourages innovation by lowering the barriers to entry for anyone with an idea. He has been further convinced of this since “seeing how healthy the web has been through , recognising that there has been something resilient about the web culture, that it just kept innovating right through the crisis”.

“Facebook and Twitter were being built as the rest of the world was falling apart. It was almost like nothing had happened to them,” he says.

That resilience is the hallmark of vibrant ecosystems. The web and the ecosystems display similar patterns to the process of innovative thinking described in Where Good Ideas Come From.His most ambitious work yet, it aims to present a grand thesis of creative thinking.

The venerable technology writer Kevin Kelly summarised it as “a book about how ideas are networks that are made up of a network of ideas”. Indeed, the book itself is an intricate network of anecdote and theory that ultimately coheres into a compelling portrait of that most evanescent of mental processes – having an idea.

“It wasn’t ready until I had the patterns, with each chapter being a pattern . It took me three years to get to that idea. Originally it was five, then I padded it out to seven. It took a long time to figure out a way to make it all work,” he says.

The structure he mentions is a clever grouping of “patterns” central to strong idea creation. Each chapter is devoted to exploring a pattern: the premise of the “adjacent

possible”, which describes how each phase of innovation opens doors for more innovation, allowing good ideas to lead to, and build upon, other good ideas, and so on; “liquid networks” that encourage information flow and collaboration; the long development of “slow hunches” rather than flashes of inspiration, as epitomised by the long gestation of Darwin’s theory of evolution; the vital role played by sheer serendipity and learning from error; the potential of exaptation, or adapting existing innovations for new purposes; and the foundational role played by platforms, such as the tangled banks of coral reefs, cities or the internet.

Johnson illustrates each concept with vivid, interlinked anecdotes, creating an enlightening web of innovation history.

He seems justifiably proud of and gratified at the positive reaction that met Good Ideas– it immediately had the air of a zeitgeist-defining work.

"When I was putting together the idea and writing it, I thought that innovation was an evergreen subject matter that people were always interested in . . . But I didn't think of it as being particularly timely," he says.

"I think, in a weird way, that it's partially [due to] the post-economic crisis and being in a recovery period, or making ourselves feel like we're in a recovery period. The book is about innovation environments and has a special place in its heart for why the web itself has been such a driver of change and new ideas. I think that resonated."

According to Johnson, the web points the way to the next phase of the "adjacent possible", a phrase borrowed from the scientist Stuart Kauffman to describe the exponentially iterative process of innovation. Such a model suggests an ever-increasing rate of technological development.

Some thinkers, however, doubt the sustainability of such improvement. US economist Tyler Cowen, for instance, suggests in his most recent book, The Great Stagnation, that the pace of life-changing innovation has slowed dramatically in recent decades, with many of the changes seen in the 20th century representing the "low-hanging fruit" of progress, and that we have reached a "technological plateau". Johnson isn't so sceptical.

"The two doors that have just been unlocked are location [based] and social. We're just starting to think about what it means to organise everything around true social networks," he says. "To have that as part of the structure of all the technology that we have is going to be unbelievably disruptive. All major consumer decisions, I believe, will be predominantly filtered through some kind of social graph in the next five years. You'll make all your choices through that network. And that is very different; it's not the way it works now. We still have a mass-culture system, not a system of clusters of 200 people. Walking around with location-aware devices, with a world of data that is geographically tagged, that is just starting to happen.

"Frankly, I think that alone will give us another 10 years of messing around and experimenting and inventing new things," he says, breaking into a hearty laugh. "By that point the computers will be sentient, and then we'll have plenty of stuff to deal with."

Apart from the notion of the Eureka moment, another myth Johnson is eager to puncture is that innovation stems predominantly from competitive markets. For instance, the story of how Willis Carrier developed air-conditioning in the early 20th century is often held up as "the archetypal myth of modern innovation" – an individual inventor motivated by the promise of great wealth hits upon a brilliant and lucrative idea. Our collective belief in the Carrier model of innovation underpins our faith in market capitalism as an unparalleled innovation engine.

In the compelling final chapter of Good Ideas, however, Johnson persuasively argues that Carrier is an anomaly: the exception rather than the rule. Innovative thought is more likely to spring from collaborative networks than from individuals or small groups working in isolation, and market incentives have a surprisingly small role to play in encouraging innovation, which often thrives in non-market environments such as universities and open-source development communities – a space Johnson calls the "fourth quadrant". Crucially, the liberal flow of ideas in nonproprietary networks is more conducive to pioneering breakthroughs than in private firms competing in a marketplace and determined to protect their intellectual property.

This conclusion sparked a predictable capitalism-versus-socialism debate during the recent Good Ideasbook tour. Johnson sounds bemused by the issue now.

"That was totally fascinating," he says. "The politics are very interesting. They don't fit clear left-right distinctions because you're all for decentralisation and open networks, but you're not just for markets; and you're not for big government, big state-centralised solutions to things, but you think that some of the best things ever have come out of non-market environments. Trying to figure out what that means as a political philosophy, not just as a prescription for innovation in society, gets really interesting."

The problem, Johnson feels, is the lack of a pre-existing political vocabulary to define the "fourth quadrant". What is most apparent from Good Ideasis the organic nature of innovation networks, the various elements of chance and circumstance required for them to develop, which would suggest that creating something such as a "smart economy" is rather more complex a task than merely investing in computer science departments and broadband. Notoriously, the top-heavy bureaucracies that typify our public sector institutions, from the Government down, are, in Johnson's words, "innovation sinkholes". In Good Ideas, he suggests that "these institutions have an opportunity to fundamentally alter the way they cultivate and promote good ideas. The more the government thinks of itself as an open platform instead of a centralised bureaucracy, the better it will be for all of us."

How can this work in practice?

"You know, it was very funny," he says. "I've gone down to some folks at the White House since this book came out, and the intelligence community down there, to talk about the importance of openness and trying to figure out ways to allow ideas to connect in new, surprising and serendipitous forms. It turned out to be the week that the last big WikiLeaks dump happened, so they were all like, 'we're all full on openness right now, we're trying to figure out how to close things down a little bit'," Johnson says, laughing.

"But I think this is one thing the Obama administration has been very good about and very committed to, trying to figure out ways to encourage people outside the government to do interesting things with all the data that governments have.

"There are all these services, all these things governments do; all this information about the resources government has, and about society and where tax dollars are going. Historically they've done a terrible job sharing that information, and making it accessible . . . If governments can take some of these systems and get a lot of people making new tools to help folks realise the value that governments provide, I think that's a great opportunity."

Whether Johnson can help governments foster innovative environments remains to be seen, but after reading Good Ideas,it's impossible to take the creation of good ideas for granted ever again. Far from being an abstraction, it is, clearly, an awe-inspiring process of human creativity.