Can we predict the future with big data?

As a lifelong fan of science fiction, I was thrilled to learn that Apple TV+ is bringing Isaac Asimov’s classic Foundation series to the small screen. I have to admit that my excitement was just accompanied by a soup of bewilderment, unsure as I was that anyone would be able to do justice to the story’s vast multi-generational expanse. But it was good that an effort was also being made.

The reason why the Foundation trilogy is so timeless is its connection to psycho-history—the fictional science around which the entire series’ plot revolves. Using a combination of mathematics, history and sociology, psychology makes it possible for the protagonist, Hari Seldon, to predict the flow of future historical events with near-perfect accuracy. Science is based on the premise that, while no one can expect to predict what a single human is going to do, if you are modeling a large population, it is possible to describe the order in which Future events will be of unmistakable accuracy. The Galactic Empire in which the Foundation series is set has a population exceeding a quintillion, allowing Seldon to accurately predict its downfall and develop a plan to shape the course of these future events in order to achieve its worst. effects can be reduced.

The appeal of psycho-history lies in its credibility. Even though it was conceptualized 80 years ago, long before the wonders of big data and modern data-driven innovation, Asimov intuitively drew attention to the fact that human behavior can be erratic in isolation, when collected at a population scale. If done, it can be inferred.

Science fiction has always been presentational. Jules Verne wrote about space travel and submarines before it even reached the technology needed to make it a reality. Arthur C. Clarke predicted satellite communications before the first satellite was placed in geosynchronous orbit. Even Douglas Adams’ vision of a universal translator (the babelfish—a live fish that lives in your ear) long predated Google Translate.

Seeing as how science fiction got all of these things right, I’d like to think that it’s only a matter of time until psychology becomes a reality.

A few weeks ago, The Economist Weekly had a feature on a new revolution that it called the third wave of economics. Unlike the first wave, which was largely driven by individual thinkers who wrote books and papers about a singular big idea, or the second wave which was a bit more experimental with new claims supported by empirical studies, the third wave of economics. Almost fully powered. Huge availability of real time data.

Tech companies have long been able to draw on the data they generate to predict customer behavior. E-commerce firms use this information to promote products and ensure that their private-label brands line up with items that are likely to have strong demand. Streaming media companies use this data to green-light movies and TV series based on what is most likely in favor of a global audience.

Third-wave economists use similar techniques, drawing on large amounts of real-time granular data, to explain real-world problems. By studying fine-grained mobility data obtained from social media companies and telecom service providers during the pandemic, they were able to understand the impact of lockdown restrictions on disease transmission. By studying live data on the ships’ day-to-day activities, they were able to pinpoint where the bottlenecks lie in supply-chain logistics. In the US, economists analyzed live data from restaurant booking sites to gather evidence in support of a stimulus package for the industry.

The availability of real-time granular data is only going to increase. In addition to the fact that more and more people are getting online every day, the rapid increase in the amount and variety of wearables and Internet of Things devices has resulted in a rapid increase in the availability of new information. This real-time data is available on the cloud in easily accessible, inter-operable formats that are suitable for cross-platform analysis. And, as the amount of data available increases, the accuracy of early trends identified is only going to improve.

While it may not be anything like Seldon’s psychology yet, we can already see how, over time, it could develop into something along those lines. If economists can model the data generated by commercial transactions to predict the behavior of markets, it may not take long to use this data to predict social outcomes. With real-time data at their disposal, it should be possible to create tight feedback loops that continually refine and reorder these models based on how they perform in the real world, with continuous iterations helping them improve accuracy. Help is available.

When we have all the data needed to predict outcomes as well as the models needed to do so accurately, we will also have all the tools needed to shape those outcomes in desirable ways. Although, at present, the predictions of these models are relatively short-lived, it seems entirely within the realm of possibility that with more data and better models, it will soon be possible to look further into the future, not just predict the immediate predicted response. Doing. But also the sequence of events that will occur. Once that happens, it will only be a matter of time before there is little to separate third wave economics from psychology.

Rahul Mathan is a participant in Trilegal and also has a podcast called Ex Machina. His twitter handle @matthan . Is

subscribe to mint newspaper

, Enter a valid email

, Thank you for subscribing to our newsletter!

Never miss a story! Stay connected and informed with Mint.
download
Our App Now!!

,