While the pace has picked up in recent years, AI has been revolutionizing finance behind the scenes for decades now. Due to the complexity of the data, systems, and strategies involved in financial markets, there are numerous situations where the actionable, monetizable patterns elude human perception and understanding but succumb to AI algorithmics.
Typical AI technologies applied to financial markets have also shown some significant limitations — mostly notably the inability to deal with ‘regime changes’, i.e. situations where fundamental characteristics of a market change radically during a short period of time. A typical pattern is that a certain quantitative or AI technique works really well for a certain interval on a certain market, yielding dramatic positive returns with manageable risk, and then suddenly the market regime shifts and the performance rapidly becomes quite the opposite.
Given the nature of the beast, this sort of phenomenon will probably never be entirely conquered. However, the clear potential exists to do a much better job of robust AI market prediction and financial management by leveraging ensemble-based AI tools and progress toward Artificial General Intelligence.
AI in Finance: Transcending the Curve-Fitting Trap
The use of AI in finance is a natural extension of the use of advanced statistical methods, which arguably dates back at least to Bachelier’s 1900 thesis ‘The theory of speculation’. Machine learning methods, and broader sorts of AI techniques, exceed standard statistical methods in their ability to deal with complex interdependencies between multiple variables, which are ever-present in finance, due to the complexities of the economic systems underlying financial instruments and also the diversity of players in real-world markets. Today AI methods are everpresent on the back end of financial applications including process automation, security, underwriting and credit scoring, robo-advisory, algorithmic trading, and more.
Fundamentally, though, every AI-for-finance application still suffers from the same basic problem: what the algorithms are doing is a little too close to curve-fitting on the historical market and market-context data used to train the algorithms. This problem is not necessarily unique to AI methods; simpler quantitative techniques, standard ‘technical trading’ approaches, and approaches based on expert fundamental analysis or behavioral finance can also overfit to historical situations and regimes. However, the sophistication of AI systems means that sometimes the ways in which AI financial models are overfit to their historical training data can be less obvious to tease out.
These issues have not stopped AI from delivering tremendous value. Funds like Renaissance Technologies and 2 Sigma have famously provided investors extraordinary AI-driven returns, but these just scratch the surface. Looking just at AI-driven fund management — a small part of the story — it is well known that AI-based hedge funds as an overall sector have delivered significant returns above that of funds traded using more conventional methods.
Given the important role that healthy financial markets play in advancing technology and society by helping provide broad access to capital, we can say that financial AI has been a major contributor to recent growth in the world economy. It’s also clear, though, that AI has been substantially deployed for the benefit of the largest investing entities rather than smaller players — which is not intrinsic to the nature of AI as a technology, but rather a manifestation of the scarcity of AI talent and the general centralization of the AI sphere.
I believe that returns from financial AI can be made even greater and more consistent to the extent regime changes can be more adeptly managed by moving further beyond curve-fitting type methodologies and using AI models that better abstract fundamental market dynamics transcending particular historical situations. I also believe that the overall social and economic impact of financial AI can be made far more positive if it is deployed appropriately — leveraging decentralized software frameworks and focusing on a broader variety of financial instruments including those that drive entrepreneurial ventures forward.
Crypto-finance’s Special Challenges
Crypto-finance presents some particular challenges beyond those present in traditional equity and futures/options markets. These challenges confront all investors and traders, not only those using AI approaches, but they can present themselves to AI system developers in especially subtle ways.
One challenge is the relative lack of historical data. The crypto markets simply haven’t been around as long as traditional markets, which means the data available for humans or AIs to study for inspiration or historical testing is much less.
Another major challenge gets back to regime changes, which in the crypto domain are relentless and extreme. Ongoing improvements in technology, shifts in the regulatory landscape, and expansions in the nature of the crypto investment community (e.g. one year China heavily restricts crypto, the next year a huge number of traditional Wall Street firms come into crypto, etc.) mean that the markets at any given point in time generally have radically different characteristics from what they had a year before.
Difficulties are also presented by the heavy manipulation of crypto markets by a small number of relatively well-capitalized parties. This is similar to situations seen in equities or commodities in various emerging markets, and it means that a lot of what one’s predictive algorithms are doing is predicting the future attitudes and activities of a handful of powerful individuals or funds. In fact, the situation goes beyond mere price manipulation and often includes illegal and unethical practices like wash trading and making fake orders on exchanges.
Finally the relatively small size of many crypto markets — and this especially applies to altcoins more than to BTC or ETH — means that once one’s trading activity gets above a certain relatively modest size, one’s activities are directly influencing market behavior. This can be both a positive and a negative — in an AI context it opens the door to using reinforcement learning techniques as well as more standard non-interactive pattern analysis techniques. But it complexities things and renders a significant portion of the traditional AI-based financial approaches not precisely applicable.
None of these complicating factors are show-stoppers for the application of AI to crypto finance. But they are all things that AI algorithms need to deal with in the crypto context, and they raise the bar for the sophistication of the AI techniques one can profitably deploy.
Meeting the Challenges with AGI and Ensemble Methods
The key to overcoming the challenges presented by crypto markets, and the difficulties with regime changes that have plagued AI’s application to finance generally, is the development of AI techniques that can learn more “broad-minded” models of the historical data at their disposal, rather than basing their future predictions too closely on the highly specific patterns they’ve seen in the past.
This can be achieved in multiple ways. One avenue is the development of diverse ensembles of AI models, each model embodying different insights, perhaps resultant from different algorithms, different datasets, or having different humans involved in the model creation. Numerai and Cindicator are two well-known blockchain projects that have leveraged ensemble methods for financial prediction, at times to excellent effect.
Another avenue is the utilization of AI techniques, such as neural-symbolic AI, that focus on forming more abstract models of the data they ingest. This is where financial AI intersects the higher end of the AI research world, leveraging concepts like transfer learning, lifelong learning, and Artificial General Intelligence.
Both ensemble methods and advanced AGI-oriented algorithmic techniques can be applied in a pure data-analytics mode for larger and more liquid markets, and in a reinforcement-learning mode for smaller markets in which analyzing the impacts of one’s own trading actions plays a key role. Further, the two approaches can be used together by including AI tools based on transfer learning, lifelong learning, and proto-AGI techniques within one’s broader model ensembles.
…. and this of course brings us precisely to SingularityDAO and its use of the SingularityNET platform.
SingularityNET is designed specifically for robust integration of diverse AI tools in the service of common goals, and for exploring the boundaries between narrow AI and AGI as they manifest themselves in various vertical areas as AI algorithmic sophistication gradually increases.
SingularityDAO — a newly launching project incubated by SingularityNET — is designed to amplify the value of SingularityNET-based AI in the specific context of predicting the value of sets of altcoins, with the goal of creating healthier and more liquid markets for these altcoins and thus fostering growth in the decentralized technology sector.
In the course of pursuing its mission in the altcoin markets, SingularityDAO will also be driving R&D and practical experimentation with AI algorithms in some very challenging contexts, thus driving the state of the art forward.
We hope you will join us in our mission to democratize AI and to use its transformative powers to enact real, positive and lasting change. The best way forward is to come together and work practically towards creating a better future.
If you are looking to monetize your AI services or create new ones, we invite you to learn more about the nature of our platform and what it has to offer by visiting the SingularityNET developer portal.
We are proud of our developers and researchers that are actively publishing their research for the benefit of the community; you can read the research here. For any additional information, please refer to our new roadmap. To stay informed about all of our developments, please subscribe to our newsletter. You can connect with our passionate and talented community by visiting our community forum.
Why Cryptofinance Needs Decentralized AGI was originally published in SingularityNET on Medium, where people are continuing the conversation by highlighting and responding to this story.