Big Data Behavioral Analytics - A Spotlight Q&A with Quantivo’s Amrit Williams

Originally published 14 December 2011

BeyeNETWORK Spotlights focus on news, events and products in the business intelligence ecosystem that are poised to have a significant impact on the industry as a whole; on the enterprises that rely on business intelligence, analytics, performance management, data warehousing and/or data governance products to understand and act on the vital information that can be gleaned from their data; or on the providers of these mission-critical products.

Presented as a Q&A-style article, these interviews conducted by Ron Powell of the BeyeNETWORK present the behind-the-scene view that you won’t read in press releases.

This BeyeNETWORK spotlight features Ron Powell's interview with Amrit Williams, Chief Technology Officer at Quantivo. Ron and Amrit talk the value of analytics and the benefits analytics as a service, a scalable and elastic approach to "big data" analytics.

These are definitely the days of big data. Everybody's talking about it, and it seems every vendor is rushing to bring a big data solution to market. Quantivo is uniquely positioned because it you have always focused on extremely large data sets. Can you give us a brief history of Quantivo?

Amrit Williams: Originally, the technology was co-developed with a very large credit card company, and the problem they were trying to solve was that they needed to analyze every credit card transaction. They were trying to identify fraud, violations of policy, business rules and misuse. Essentially, they wanted to understand the behaviors of consumers when they were doing credit card transactions. They needed a technology that could handle massive data volumes and very high velocity transactions, and they needed access to the raw event-level data. Summarizing or sampling the data wasn't going to be appropriate. They needed to know, for example, if a point-of-sale device at a gas station in the middle of the country had a business violation or there was a fraud attempt there.

At the time, there wasn’t great technology to do this. Some of the folks at this credit card company sought out analytic and database experts in the market. They brought these experts in, and they co-developed this technology to address that problem.

By 2008, the original developers decided they want to commercialize this technology. They saw that behavioral analytics had great applicability to a wider market, so they founded Quantivo. They set about optimizing the solution to take advantage of cloud computing  – the elasticity, scalability and other attributes of the cloud – to deliver a very powerful analytic experience to the market.

Now here we are in 2011 with a mature technology that has been used in very large environments to address big data problems for quite some time. Now we are focused on applying and delivering that technology to a wider market.

Enterprises are concerned that big data is going to add another layer of infrastructure – hardware and software – that demands increased support and maintenance. How is Quantivo's approach different?

Amrit Williams: Data analytics is an interesting process in most organizations because it's not something that's done continuously – it is a cyclical process. There are peaks and valleys – times of very high analytic activity and times of low analytic activity. When dealing with massive volumes of data, there is a natural tendency in most organizations to compress that data for storage and transfer. The problem is they need to uncompress it to analyze it. When you have a very cyclical process, trying to deal with the infrastructure to support data volumes that compress and uncompress becomes very challenging.

Quantivo provides a full-stack analytics solution delivered as a service from data storage, to processing, to the ETL, to the query environment and the user interface. The entire infrastructure is managed and secured by Quantivo. This eliminates all the CapEx costs and provides a very fast time to value and productivity to the market and the customers.

Additionally, to support the cyclical nature of analytics, we’ve spent a lot of time optimizing the way we can dynamically scale our infrastructure. For example, during times of peak load or usage times, or if you have a lot of folks from your company interacting with the system, we automatically spin up a bunch of computing resources to ensure that the analytic experience stays the same regardless of the usage of the system. The power of being able to deliver analytics as a service is that ability to ensure – regardless of usage, data volumes and velocity – that we're delivering the same type of service levels. Then the organization can benefit and enjoy those services without worrying about the maintenance, management, and administration of the infrastructure. They can focus on the real value that the business needs – the insights gleaned from their data.

From service-level perspective, are you finding you're able to maintain a service level regardless of any of those factors?

Amrit Williams: That's the beauty of some of the things that we've been working on over the years in the cloud. We've tried to ensure that regardless of what happens to the infrastructure, we can still accommodate and maintain service levels. For example, we have the ability to very quickly run full tolerance and load balancers in different geographies. We also have the ability to very quickly spin-up separate clusters in different geographies. If our North American infrastructure data center becomes impacted, we can very quickly bring up an additional server in Asia, Europe, or different clusters of servers. The way that we're transferring data and ensuring synchronization ensures that the process has very limited downsides.

Even though we're an analytics technology company, a lot of our focus has been ensuring that we can deliver a very strong analytic experience regardless of usage, data volumes or velocity. It has been very beneficial to use the elasticity of the cloud to ensure that we can dynamically scale when any of those things happen. Cloud capacity isn’t infinite, but there is a lot of capacity that can be used and leveraged very, very quickly.

When you talk about the cloud, do your potential customers express any concerns about putting their data in the cloud? If so, how do you address those concerns?

Amrit Williams: I think it's the loss of visibility, control, and the concerns around governance and compliance that drive the security issue. But a lot of people voice security as the number one concern for cloud computing. Prior to joining Quantivo as the CTO, I ran Emerging Security Technologies for IBM, and some of the areas that were part of my responsibility were cloud computing, mobility, security analytics and things of that nature. We wanted to understand how we could deliver better services to our customers and what their concerns were. Without question, security was the number one concern. It all came down to issues of visibility and control. I think a lot of companies recognize that some of the cloud computing providers and folks that deliver services have some pretty strong incentives to ensure that they have the expertise, sophistication and controls to manage and secure the environment.

That being said, governance and control are very important, but we do a lot to try to address the concerns. For one thing, we're fully multitenant. When I say fully multitenant, I mean not only are we logically separating data, and customers, and tenants, but we're also physically separating them on the file system. The segmentation of data is better. There is no data that can be in any way interspersed with other data, which is different than how some other technologies work. Additionally, we provide the ability for the organization to manage, understand and administer their data environments so they can see, audit and get the visibility they need into that infrastructure.

Sometimes customers have concerns about a particular set of data, and we work with them to find ways to anonymize or tokenize the data so it's not compromised.

You're right, security is a big concern – not just for us, but for cloud computing in general, but I think enterprises are starting to understand that the benefits make the risk acceptable. Companies like SuccessFactors,, and others have done quite well delivering services across a live set of domains.

One of the intriguing features with the Quantivo platform is your behavioral analytics. Could you explain behavioral analytics and the types of data that can be analyzed, and how it differs from other analytic methodologies?

Amrit Williams: Human behavior is intrinsically chaotic and dynamic, and it's very hard to guess what people will do. Behavioral analytics can provide incredibly powerful contributions to the way businesses interact with their customers and maximize their revenues.

Our behavioral analytic capabilities are based on what we call a pattern-based data processing engine. What we do when we ingest, enrich and transform the data is look for things that have a lot of high affinity or cardinality to each other. We can actually look at how things influence each other.

Traditional technologies tend to look at what happened. They can do counts very effectively, and they can tell that a certain thing happens more than another. But they're not very good at telling you what might occur, or what will occur, or what the probability is of something occurring if you change your behavior or how you talk to your customers, and that's really what we're trying to do. We find patterns of behavior, what influences behavior and how to impact the behavior you want – both negative and positive behavior. The interesting thing about behavioral analytics is that there is a set of things that seem to be very common sense. Then there is a set of things that are not, and those are the things that have very, very high value.

For example, we had a customer that was an online media company, and one of the things that they were looking at was the video game content segment. Their highest revenue-generating segment was Xbox 360 users who looked at sports and action games. Common sense would tell you that if you wanted to maximize that segment – if you wanted to cross-sell or up-sell them something – you'd probably want to look at things that they would have an affinity for, such as first person shooters or wrestling games. But it turns out that the highest affinity they had was for music games. That segment of the population was four times more likely to visit music games content than anybody else in the population. And when they did, they spent almost double the amount of time on the page and they looked at more pages. That's an incredibly high value segment for this company, and it has nothing to do with common sense behavior.

The real beauty of behavioral analytics is that it gives you knowledge to make decisions. In the case of this company, the highest opportunity they had to optimize and maximize the value of this segment was actually something that no one in the company thought it would be. That's the real value of behavioral analytics, and you can extend this to almost any domain.

For example, we worked with the credit card company, and they were looking at fraud alerts. They wanted to understand behavior and its influencers. They wanted to know if we saw someone doing X and they did Y, what would be the probability that they do Z, in this case fraud. They wanted to know how to decrease the probability that they would actually do Z, or stop it before it happened. Understanding behavior is very, very complex, but it's incredibly beneficial and powerful for companies to optimize how they interact with their customers.

It does sound complex, and it certainly seems like something a typical business user would not do on his or her own. Does a company have to hire a statistician if they want to use Quantivo?

Amrit Williams: Well, this is the other part of about Quantivo that's incredibly intriguing. The technology was really designed for business users. In order to take advantage of the pattern base and affinity matching that we do within the data, we built out a lot of the stack ourselves. We built our own ETL and created an environment and interface. We spent a lot of time to be sure the interface was easy for folks to understand and didn't have a high barrier for entry. We developed a set of pre-canned things for certain domains to get folks started and made it easy for people to ask and answer questions.

The interesting thing about a lot of the behavioral analytics that we see with our customers is that there are things that are very simple to express in English. For example, what else do people do when they do X? Can I look at a segment of the population that does X and learn to influence them to do more of Y?

When you say that in English, it sounds very simple; but executing that with data and analytics technologies can be very complicated. That’s why we built out the back end and  developed an interface so that we could better align the simplicity of English with the complexity of technology, and provide an abstraction layer so that a regular business user can get access to this type of powerful analytics. That is the mission of the company. We deliver powerful analytics to a wide set of the market that has not had the ability to access to this type of analytic power. We want to make a difference in how these companies run their business.

What kind of insights can behavioral analytics provide for your customers and what types and sizes of companies can benefit from a Quantivo solution?

Amrit Williams: I'll give you a couple of examples of how we've been used to optimize marketing programs, increase revenues, lower costs, or reduce fraud and abuse.

One interesting example is about how we provide influence versus the importance of something. In this case, it was a pharmaceutical company. You can't sell pharmaceuticals online so it's hard for them to measure a conversion and understand how their online marketing works against the marketing that they do with the doctors, etc. The only thing they could really determine was if the content that they were putting on their website was important. They could do that based on page rank or visits, but they couldn't figure out the content that was actually leading people to take an action. That's what Quantivo was able to do. We could show them that the content on a specific page was influencing people to take an action.

The company took that page and moved it to the home page because it was buried about seven clicks deep. The result was  that they had a five times conversion rate increase, which is incredibly powerful. That’s an example of how we provide the ability for a company to optimize their marketing programs.

Another example is a retailer. They manage a wide set of national chain stores, and one of their cigarette distributors came in and said, “We are planning to increase your costs. We think our margins aren't high enough, and we recognize that our cigarettes are what are driving the sale of other items in your store.” This was going to result in a very significant increase in cost for this company so they used our technology to determine two things.

They wanted to segment every customer who purchased cigarettes, find out what they purchased with the cigarettes, and compare that to the general population. They were able to determine with the data that the cigarettes actually didn't influence this segment of the population to purchase anything more than the general population. They took the data back to the cigarette distributor company and prevented the increased costs. That is an example of the very high-value, transient activity we're very good at providing. Because we provide our analytics as a service, there's a very low barrier of entry for this company to use us. They went on, and now they use to do merchandising, and they make sure that when they have campaigns that we're monitoring and optimizing them.

Here's another great example. We were working with another retailer, and they actually incented their cashiers to take customer contact information. They thought this would be a great way for them to amass a better database of their customers. They sell items that are not high-end so their market basket size on average is probably about $50. They loaded and used our tool to understand customer behavior on their e-commerce online site versus their in-store, brick-and-mortar site. Within five minutes of inputting the tool, we automatically identified associations between customers and purchases and market basket size, and they quickly realized that what was being input for customer data was, in fact, not correct. The cashiers were just inputting the same data over and over again so that they could get their bonuses, but it was all false data. That behavior was determined very quickly and changed how they were running their business.

Ultimately, we help our customers optimize, target, and drive greater value to their customer base, the services that they deliver, and their interactions with the market. We help them maximize and excel at these things. That’s value in a solution. We believe we can provide insights with behavioral analytics that are optimizing, driving assumptions to knowledge, driving data to knowledge, and driving the business to take that knowledge and make very good decisions on how they're going to use their investments and their money.

It’s evident that from a behavioral perspective, you can solve some very complex problems, but it seems that you have to have a certain amount of data from a click perspective or a historical perspective to determine some of these insights. How long does it take for a company to get started with the Quantivo solution?  

Amrit Williams: You need the raw event-level data. This is why summarizing data to try to do behavioral analytics would never work. As soon as you get rid of some set of the data, you get rid of the accuracy. For data and analytics, accuracy of data is really important.

It’s not difficult or challenging to get set up with Quantivo. We find that a lot of our customers have data online or data that is being aggregated by somebody. For example, there are companies that aggregate point-of-sale device data for multiple retailers, and that data sits in the cloud. We have access and have worked with those aggregation points for multiple companies. We already have integrations built with lead or e-mail optimization companies like Responsys, Marketo and others so we can bring in the data very, very quickly.

Amazon S3, as an example, has dealt with very large data loads, and we do very similar things. If the original data load is very large, we have WAN acceleration techniques with some applications we have to help them move the data around. The original movement of the data is the big hurdle. Once that data is moved, the incremental movement of data is quite simple.

The time to get started for most companies can be a couple days to weeks. It depends on how long it takes them to give us the data. Training takes about four hours. Once we have access to the data or the data is being moved over, it’s generally about a day for us to work with the company to prep the environment and get them ready. The majority of customers that we have are up and running and productive within a couple weeks.

Well, you answered my last question, which was moving the data. I like the approach that once you have the original data loaded, you can take care of it on an incremental basis so you don't have to reload over and over again.

Amrit Williams: We do see very big data loads. Recently, we had one that was about 45 terabytes. But again, this is a space and time issue.  Moving terabytes of data takes time, but the cost and time associated with moving data and working with a service like Quantivo is still less expensive and less challenging than trying to develop all the infrastructure internally and moving it that way.

Thank you so much for sharing Quantivo's value proposition with BeyeNetwork readers.


SOURCE: Big Data Behavioral Analytics - A Spotlight Q&A with Quantivo’s Amrit Williams

  • Ron PowellRon Powell
    Ron is an independent analyst, consultant and editorial expert with extensive knowledge and experience in business intelligence, big data, analytics and data warehousing. Currently president of Powell Interactive Media, which specializes in consulting and podcast services, he is also Executive Producer of The World Transformed Fast Forward series. In 2004, Ron founded the BeyeNETWORK, which was acquired by Tech Target in 2010.  Prior to the founding of the BeyeNETWORK, Ron was cofounder, publisher and editorial director of DM Review (now Information Management). He maintains an expert channel and blog on the BeyeNETWORK and may be contacted by email at 

    More articles and Ron's blog can be found in his BeyeNETWORK expert channel.

Recent articles by Ron Powell



Want to post a comment? Login or become a member today!

Be the first to comment!