There is a lot of hype around machine learning and AI in the technology industry today. Banks and FIs are making bold steps towards implementing an AI roadmap. If you are startup and have AI in your name you are virtually guaranteed to get funded. However, what most people are missing is that AI is not the answer to everything.
DataSeers and LexisNexis recently did a hackathon at KSU and the problem given was to homogenize merchant names. Kids quickly took the machine learning approach using Metaphone and Levenstein algorithms and advanced machine learning techniques. Needless to say, they did not succeed. However, they learnt an important lesson. ML is not the solution to every problem. Certainly, you cannot dive into a problem ML first.
A few years ago, I attended a keynote at a conference and the speaker talked about the acronym LWAR, which immediately stuck. Listen, Watch, Analyze and then Read. In that specific order. This approach is not only important in life but very much so in any ML based solution. The first step is to understand the problem at hand. This includes data analysis and profiling. A simple question, how good (or bad) is my data? The next step is to do the actual cleansing of the data. In the case of our hackathon problem this would have solved the problem close to 90%. No amount of ML or AI would have needed to get here!
Before picking any ML algorithmAt its most general level, an algorithm is a finite, deterministic, and effective problem-solving method suitable for implementation as a computer program. In modern equity and debt markets, many problems are solved and decisions made in this computational, algorithmic manner. For more information please see the SEC website More, clean and labelled data is required. Once this is available, the next important state is to identify if and what features from your dataset apply to the specific problem you are solving. In any case, there is not a one solution that fits all. There are several different algorithms and libraries that can get you started. But what will make you succeed is the domain expertise, a careful and methodical approach towards solving the problem and then picking the right tool set.
If you were given a broken plane and a bunch of tools, would you start repairing it? No! you won’t. You probably won’t know where to start. You would spend time understanding the problem, understanding the tools you have. Then figuring out which tool is suitable to fix the problem and then try to fix it.
Having said all that, machine learning is really good at recognizing patterns. The old approach of “If it walks like a duck and it quacks like a duck it must be a duck” is gone. Now it’s time to ask the additional questions, does it look like a duck? Does swim like a duck? Does it smell like a duck? Fraudsters exploit schemes that are rule based, by presenting a machine with patterns that it expects and lets you in. This is a very typical approach to gain access in the cyber world today. ML takes an anomaly approach, it looks for anything out of the ordinary. This could mean you would need access to additional data, what if we weight the so-called duck. Looked at the shape of its beak. Looked and analyzed its webbed feet for additional datapoints. Your algorithmAt its most general level, an algorithm is a finite, deterministic, and effective problem-solving method suitable for implementation as a computer program. In modern equity and debt markets, many problems are solved and decisions made in this computational, algorithmic manner. For more information please see the SEC website More suddenly became very strong and now it would work so much better.
DataSeers has ML/AI at its core, what sets us apart is that we know the data inside and out. We have clean, labelled and rich data that feeds out models making them very accurate. We also collect data from disparate data sources and use it in our decisioning engine. If you are embarking on a ML/AI journey, make sure to remember the most important letter a wise man once told me
LWAR – Listen, Watch, Analyze and then React
