Cell Knight

Artificial Intelligence techniques power CellKnight

CellKnight was launched in the Silicon Valley and its early adopters were our friends and family members who happened to be, for the lack of a better word, "techies". When we first explained CellKnight to one of our early users, the first question was: Isn't it simply web-scraping? Your software is logging onto the cellphone provider's website and fetching minutes usage data from the web page that is then being monitored.

Well, yes and no! The description of what the CellKnight software does is correct at a very high level. However, the devil is in the details. Web-scraping, as it is commonly known, or POWS (Plain Old Web Scraping) as we like to call it is like a high maintenance car. Once you buy the car, it will keep breaking every now and then and the repairs are prohibitively expensive. And once you own that car, it is hard to get rid of it because no body else will buy it.

Anybody who has ever written a web-scraping software will attest to the countless hours it requires to fix it when the website from where it is fetching data changes its structure and/or content. If your software is looking for a particular element with a particular ID on a web page, you obviously know what happens when the web page changes and that element no longer exists? Say, the button became an image and the table became a sequence of DIVs and that element with a particular ID that you were looking for? Lets not even go there :)

To cut a long story short, webpages change very often. In fact, they change far more often than we would like to imagine, especially when it comes to the website of a big corporation such as a large Telecom company (for example, Cellphone providers). These companies have a huge IT department who have got to keep their job. So what does the IT department responsible for creating and maintaining the website do? They keep changing the webpages - usually a small change at at time (and drastic changes every few months) that is enough to break most plain old web scraping software.

So how does CellKnight do it? As opposed to the POWS softwares, CellKnight uses a proprietary natural language understanding technique that tries to make sense of the content of a web page similar to how you and I understand the meaning of what is on a particular web page. It uses this technique not only to identify items of interest on a web page but also to direct the controlled search for pages where the items of interest might be found. The result is an intelligent web scraping technique that is independent of the structure, layout, and content of webpages. The website and all its pages may change in design and content but as long as the minute usage information is available on some page, the software will find it.

Of course, as one can well imagine, the applicability of this intelligent web page understanding technology is not limited to CellKnight. It can be used in many different contexts such as to understand and find product and travel deals. *hint, hint* :)

The story behind CellKnight

One of the founders had just gotten married and his wife was then in a different country for a month or so. And he used his cellphone to talk to her constantly. He had exceeded his minutes once before but that was almost 4 years ago. So the very thought that he needed to watch his cellphone minutes usage did not occur to him until much later. And that was a little too late. The wakeup call was the cellphone bill that month - more than $250 over his usual cellphone bill!

How dearly he wished that someone would have reminded him that he was about to go over his minutes. He called the customer service of his cellphone provider many times hoping to talk to a different person every time and to different supervisors but that did not help. All that was said was, "Sorry, we cannot help you now. If you had called us even one day before the end of your billing cycle, we could have moved you to a higher minutes plan for that month and you would not have gone over your minutes."

A few months later, the second founder had a similar experience with his cellphone bill.

What was needed was something that could have watched our minutes usage and told us when we were about to go over our monthly quota. Had we known that we were about to exceed our minutes, then we could have done either one of the following: (a) Stop using cellphone for the remaining days in that billing cycle (basically, use it only for Emergencies), or (b) Call the cellphone provider's customer service and request them to switch our plans to a higher plan that allows more minutes.

Oh, how much we wished that such a thing existed!

So we wrote a quick web-scraping software that would check our minutes usage and alert us when we were about to go over our minutes. It was just for our own personal use and ran on one of our home desktops. Whenever the cellphone provider's web pages changed, we would have to fix our software. We fixed it 4-5 times and then eventually gave up on it. Then about a year or so later, we happened to be working on an Artificial Intelligence tool that would understand Natural Language and make sense out of what is written on a web page. We were writing a test case for the initial version when we stumbled upon our broken web-scraping software.

It was decided that we will re-write that software using the technology that we were developing. This software would be a first test case for what we were developing. Moreover, if this alerting software was so useful to us, surely there must be many others who would benefit from it. So it was decided to make this alerting software available to everybody else as a service. And that's how CellKnight came into existence.