• Application of Big Data in Finance

    Data is increasing at a tremendous rate in today’s world and the immersion of Big data has gained significant momentum in driving innovative business models to create new customer services. In recent years Big Data has also made an impact with regard to contextualising transaction data in real-time across multiple finance institutions.

    The three keys of Big Data – Volume, Velocity and Variety (3 V’s of Big Data) is what has helped overcome the challenges of the traditional approach of data handling, analysis and data extraction. Veracity is an additional dimension of big data to consider: Establishing and maintaining trust in data presents a challenge as its sources and varieties grow.

    A major challenge for the finance sector is being able to process increasingly large volumes of data in a timely manner. Big data now helps them to gain insights into their operations, customers, and market opportunities. It is known that Big data is changing the world of finance sector and affecting how they operate. Finance is the area where big data is making its big mark, as emerging technologies such as Hadoop, Storm and NoSQL allow investors to analyse large volumes of data.

    Though financial institutes are finding it a bit challenging to enable data-oriented business capabilities with Big Data, they are finding ways to integrate their processes and get the advantages of Big Data. Here are some of the many ways, Big Data is applied in the financial sector: -

    Successfully harnessing big data can help banks achieve three critical objectives for banking transformation: Create a customer-focused enterprise; Optimize enterprise risk management; and Increase flexibility and streamline operations. Big Data helps to overcome many banking problems by optimizing the data, MIS (management Information System)/Regulatory Reporting, fraud detection and investigation and best for counterparty credit risk management.

    Asset Management
    Opportunities for Big Data include enabling post trade analytics to help asset managers to evaluate key metrics like transaction costs, order execution performance and portfolio returns measurement in real time. When we talk about asset management in finance, we are covering the business emotional factor to make the economy better in institutes like trade sentiment analysis, investment product quality management and risk management.

    80% of our data is unstructured and not stored in the manageable and friendly confines of a database. Despite the volume of data and content available today, decision makers are often starved for true insight. Financial institutes helps insurance companies harness big data to drive business results. Insurance leaders are focusing on major imperatives to drive competitive advantage and differentiation in current market conditions, like customer-focused enterprise creation, optimizing risk management and multi-channel interaction and by increasing flexibility and streamlining operations.

    Capital Market
    Insights from market indicators, economic indicators, and sentiment analysis for stocks and events may be used to enrich the information set used by Investment and portfolio managers for investment and asset balancing decisions. Big Data capabilities also enable the enterprise to develop comprehensive check-points for KYC initiatives, fraud detection, investigation and prevention. We expect that the overall relevance of Big Data will continue to increase manifold and will play a significant role in defining the overall performance of firms in capital markets over the years.

    To compete in current economy system, it is increasingly clear that financial firms must leverage their information assets to gain a comprehensive understanding of markets, customers, channels, products, regulations, competitors, suppliers, employees and more. Financial institutions will realize value by effectively managing and analysing the rapidly increasing volume, velocity and variety (3 V’s) of new and existing data, and definitely putting the right skills and tools in place to better understand their financial operations, the marketplace and the customers as a whole.

  • Big Data in Marketing

    The promise of Big Data has been an exciting idea for marketing analysts worldwide. The potential of targeting customers based on not just transaction history and standard demographic profiles but also on highly individualized patterns like social media behaviour and browsing patterns is extraordinary, and marketing analysts have been quick to embrace the concept of Big Data

    How does Big Data impact marketing efforts? As discussed, traditionally marketing relied on aggregated transactional data supplemented by limited customer demographic data and for a smaller percentage of customers, loyalty data to understand, segment, profile, and target customers. Clearly differentiated strategies are better than a one size fits all approach. The same idea can now be expanded to create differentiated, personalized marketing outreach at a micro level with mini segments or even individuals. Customers are increasingly expecting customization in their life experiences, and that extends also to brands that they buy into.

    As with any new technology however, implementation has not necessarily kept pace with potential. As with any Big Data system, part of the problem is how to keep pace with the volume of data being generated, with such high variety. While data is being compiled, not all data is of the same quality or usefulness. Keeping pace with such massive amounts of data, in structured, semi structured, and unstructured formats, is a big barrier to many. The other problem is to effectively measure the impact of Big Data investment, in terms of incremental ROI.

    What is clear however is that Big Data application to marketing is here to stay. There are many success stories already – like using location data realtime to offer discounts at nearest stores, to being able to target customers to different landing pages on an e-commerce site based on their browsing history.  While there is a lot of hype around Big Data, and it is not necessarily the solution to every problem, companies and their marketing functions can certainly not ignore the implications of the transformational power of Big Data in the medium to long run.

  • Application of Big data in various business functions

    We often hear experts on Big Data talk about how Big Data powers business transformation. Indeed an effective Big Data strategy that integrates and services various business functions with the objective of raising revenues, reducing risks, and bringing down costs can truly bring successful reform and sustainable profitability to a business. The key here is integration and developing a strategy that looks at each business function both separately and as a whole.

    Many businesses use Big Data to improve their sales and marketing function. Big Data insights helps them acquire more customers, while better retaining current ones, boosts store traffic, conversion rates, and improves advertising effectiveness. They often do all this without looking at the business as a whole and though such a strategy can definitely improve the bottom line, in the long run the effects may start to slow down.

    What’s critical for long term financial success therefore is to try and develop a Big Data strategy that encompasses all the critical business functions. This will help bring efficiency and improved performance to all areas of operations by optimizing network performance, predicting utilization/capacity, maintaining a more efficient supply chain and consolidating suppliers, while also hiring and retaining the most effective workforce.

    As per a study by TCS (http://sites.tcs.com/big-data-study/big-data-benefits-challenges/) activities, which companies believe have the greatest potential for Big Data benefits, go far beyond marketing and sales. In fact, of the 25 highest-rated activities, there are an equal number in logistics and sales (six). In addition, marketing and customer service had four each.

    Over a series of articles, we will write in the next few weeks, we will talk about how Big Data initiatives can be implemented across various functions in an organization, the main ones being:

    1. Marketing
    2. Finance
    3. IT
    4. Supply Chain
    5. Pricing
    6. Risk
    7. HR
    8. Manufacturing
    9. Strategy

    However before we begin any Big Data initiative there are some critical steps businesses need to follow. Let’s quickly outline them:

    1. Define responsibilities, build a team: Designate who is responsible for collecting data (IT and analytics team), who identifies areas within their respective functions where big data could drive value and ofcourse the Big Data Team.

    2. Identify what data is worth analyzing: Valuable business insight can come from many sources. It is important to explore these sources and see which of them are viable and can answer the business problem and add value to the Big Data strategy.

    3. Match big data with business functions: Some big data programmes can be implemented in a variety of settings, but many will need to be matched to specific functions.

    4. Assess the IT architecture: Ensure that your businesses information architecture can accommodate massive, high-speed, variable data flows.


  • What it takes to Succeed with Big Data

    It is official – big data jobs have been declared the sexiest jobs of the 21st century.  For a long time, nobody could make sense of big data.  There are still so many myths that shroud the data analytics field.  Hasn’t big data been there all along?  Isn’t it just another name for the universal set, or just a catch phrase?  Is it just a blanket term with no decipherable specifics?  In effect, the whole world is talking about it.  And how there’s a dearth of talent to support the huge impact it can have in the reshaping of corporate decision making.


    Well, if you’re looking for a change in career, to gather more knowledge on data analytics, or if you’re just interested in understanding more about big data, you’re on the right page.  Here are some of the things you need to know to succeed in the big data world!


    Big data isn’t just for big companies

    It’s important to first understand that big data can apply anywhere.  The most famous biggies in big data – Amazon, Google, Twitter, Facebook, LinkedIn – these companies are the pioneers and can definitely boast of huge impacts that data mining has had for them.  But there are also companies that have “little data” who can put it to use just as effectively – and have! It doesn’t all have to be larger than life.  Sometimes, the very big differences can be made of small inferences from little, but relevant data.


    It’s not just in one industry

    Big data isn’t just for the IT enthusiasts.  It applies in such a variety of fields that its scope is close to unimaginable.  Be it marketing, sales, HR, IT, FMCG, banking – Big data jobs are sought after everywhere.  There’s a universal need now for people who can ask the right business questions and interpret the right answers from available data.


    Engineers, Visualizers, Scientists – Roles galore

    Being a data scientist might probably be the sexiest job.  What if you’re trying to make a career shift?  You could fit in anywhere!  What matters more than your designation is your commitment to making more data driven decisions; willingness to learn on the job and the ability to think out of the box.  Degrees in mathematics, statistics, MBAs, work experience – all of these are just added advantages that can help you make it big!


    Using the right data

    There’s so much data available to everyone.  How do you know what matters?  It all starts with understanding business needs.  Take time to understand what you / the company intends to do with the data.  If it’s just very specific things you’re looking for, so much the better! Identifying clear goals can make the potential of implementing a big data program more successful!


    The people are still important

    The big data boom has definitely not replaced people.  And analytics tools and engines have not replaced human intelligence.  It isn’t enough to rely on advanced algorithms to give us answers to our business needs.  What matters is being able to make sense out of what software or machines offer as possible solutions.  Recommending a solution or actually making a data driven decision is driven by analytics tools, which make data more easy to visualize and decrypt.  But that doesn’t reduce the value of people in an organization.


    Never stop learning: Keep reading and updating yourself on the developments in data science, big data and analytics.  If you truly want to make it big (anywhere, for that matter), you’ve got to realize that there’s no end to learning.  Big data is such a huge opportunity, and its application has so many facets.  Joining and following various social network discussion threads, and reading up content available in so many forms of media – they will take you closer to succeeding in the big data world.

  • Big Data vs Analytics

    There is a common confusion about analytics and Big Data. What exactly is the difference between Big Data and Analytics? And if there is a difference, what is then Big Data Analytics?

    Big Data simply is a catch-all term used to refer to multiple things – Big Data Collection, Big Data Storage, Big Data Processing, Big Data Reporting, Big Data Analysis, and so on. The enormous increase in the amount of data that is being generated, which itself is because pretty much everything we do is now being captured as data, has led to a need for much more robust and sophisticated infrastructure and systems requires to process the data. There are multiple tools and technologies developed to efficiently store, process, and retrieve these vast volumes of data, and all of these are part of the Big Data landscape

    Within Big Data, there is also business intelligence, reporting, and analytics, performed using Big Data technologies. So Analytics is one part of what is enabled with Big Data technology. However, analytics is performed on datasets of all sizes, big or not. Of course, dealing with very large datasets with a variety of different types of data poses its own specific challenges, so there is some specialized expertise required to perform analytics on Big Data.

    At every stage of the data mining process, starting from collection through analysis, there a variety of new technologies developed to manage Big Data volumes, and therefore there are multiple specialized job roles within the Big Data sphere of work. A lot of job roles are also heaving focused on IT skills, especially as related to data storage, processing, and querying.

    If you are looking to build a career in this industry and are wondering – Big Data or Analytics: the answer is really both. It is important to have a solid base of analytics knowledge and statistical and predictive modelling skills to work in analytics. It is also increasingly becoming important to have knowledge of Big Data technologies like Hadoop for distributed computing, and MapReduce for efficient querying, as well as more specialized analytics software packages built for handling Big Datasets.

    The key is in having a combination of skills that will give you the edge as this whole Big Data and Analytics industry evolves. We don’t know what the future has in store, so its best to keep your skill set updated with what is in demand in the industry.

  • Why is Big Data important to an organization?

    Big Data is now the next big thing, expected to change everything about how businesses understand their customers, generate strategy, and go to market. While there is certainly a great amount of hype, Big Data and the associated technologies and tools do bring a lot of value to most organizations, and in many cases transformative impacts.

    First though, it is important to understand that Big Data does not mean the same thing to every organization, and therefore the benefits of Big Data processing may be very different from organization to organization. While Big Data refers to very vast datasets that are being generated at high speeds, there is no one consensus on how big is “Big”. For any organization, the point at which existing infrastructure is not able to handle the volume of data being generated and needs to be stored is really when it needs to start looking at technology that can handle much higher volumes of data. Big Data is also not just about volume, it’s also about variety – data that is being collected is much more than transactional data, it could include a wide variety of data points like customer reviews in text formats, facebook likes, and images and videos. Traditional database systems are not built to handle data that is in unstructured format, and so newer Big Data technologies are required to handle non-traditional data.

    So how can Big Data tools help organizations?  First, the speed at which analysis is performed and insights are generated can be increased dramatically, with real time analysis as opposed to retrospective analysis. This is enabled by sophisticated new Big Data technology that allows much faster querying and processing of even extremely large datasets

    Second, Big Data analysis outcomes are much more powerful because they are generated using a much wider set of information that includes more than the traditional data contained in transactional databases. For example, in healthcare, doctors are increasingly able to generate evidence based treatment programmes that not only take into account previous medical history of a patient but also such data as daily fitness activities and diet components over time.

    Third, Big Data algorithms and analytics have increased predictive accuracy because of a fundamental shift in approach that has been enabled by the availability of vast volumes of data. The new approach to predictive analysis is based on Bayesian statistics, which essentially allows analytical algorithms to constantly improve accuracy based on newer data.

    There are many examples of Big Data success stories and Big Data enabled high impact strategies across organizations across many sectors and industries. While large companies have been quick to embrace Big Data and have invested heavily in Big Data programmes and manpower, in anticipation of big returns, even smaller organizations should plan and generate a Big Data strategy to survive and thrive in an increasingly data driven environment.

  • The Human Face of Big Data

    Though the mystery of missing Malaysian airlines MH370 plane is still to be resolved even after a span of two months; the multi-national search co-ordination committee and other supporting partners are processing massive amounts of satellite, flight path, and ocean data commonly referred as “Big Data” to find clues that would lead them to possible debris location. Big Data is one of the most popular buzzwords in technology industry today with a promise of transforming our daily lives.

    Rapid evolution of internet and social networks spanning 100’s of millions of users has resulted in explosion of massive information that can be analysed for trends and correlations. Many experts believe that not long from now, we will all wear devices capable of capturing and storing every possible human interaction in real-time so that they can be retrieved and accessed whenever needed. To some extent we are already witnessing the rise of such devices in the likes of google glass project and wearable fitness trackers that are growing in terms of popularity and adoption rates. Even organizations worldwide have realized the value of the immense volume of data available, and are trying their best to manage, analyze, and unleash the power of data to build strategies and develop a competitive edge. Definitely the future of Big Data looks very promising with potential applications for enterprises and individuals alike.

    Apart from providing enterprise benefits, Big Data is also addressing many challenges of our planet in smarter ways. It has led to the beginning of new thinking which starts looking at entire human ecosystem as a nervous system with intricate connections spread across and abound with information. This machine enabled connectivity of billions of people not only enables us to contribute and consume information but also making each one of us play a more central role in the entire information lifecycle. These exabytes of information we generate coupled with the processing capabilities of emerging IT technologies such as Hadoop can lead to insights that can have a bigger impact on civilization beyond ever possibly achieved.

    In the field of utility consumption, a computer scientist and entrepreneur named Shwetak Patel has developed a way for households to track and monitor their utility consumption and further provides a better way to save on their bills. This innovative idea runs smart algorithms on data generated by wireless sensors plugged in every home to provide saving tips for households to act on a daily basis. On similar lines, Opower is another publicly held company that partners with utility providers around the world and provides energy consumption monitoring services to their customers with the help of smart meter technology. According to an official statement, an average customer using the Opower platform has cut energy usage by more than 2.5 percent.

    Another great example in the field of medicine would be the use of Big Data in Canada to detect infections in ICU babies by harnessing millions of heartbeat measurements each day, and detect any potential threats at least 24 hours before. This early detection would allow doctors to get a head-start on providing relevant treatment and save many innocent lives.

    Even in terms of early detection of earthquakes, Japan invested about half-a-billion dollars in installing hardwired sensor system on the ground to track the wave that comes before a violent earthquake. As a direct result of this, they managed to stop every bullet train and every factory 43 seconds before an earthquake hit back in 2011. However these examples are only a few out of many advances through Big Data across diverse fields such as social networks, smart cities, DNA sequencing, medicine, geophysical and ocean depth tracking and it is not hard to imagine a future where Big Data will become part of everything.

  • What is Big Data?

    Why is everyone talking so much about Big Data? What is it about this term that is getting the industry all in a frenzy? Is Big Data and analytics one and the same? Well I too had all these questions some time back. I went out and read a whole lot of books and blogs on the subject and today can say that I have some understanding of Big Data which I will share with you.

    One of the best definitions of Big Data I havefound isquite naturally on Wikepedia. They say that Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. As compared to traditional structured data which is typically stored in a relational database, big data is characterised by it’s volume, velocity, and variety. Lets understand more about these three Vs.

    By the way, it was Gartner analyst Doug Laney who first  introduced the concept of the 3Vs in his 2001 MetaGroup research publication, 3D data management: Controlling data volume, variety and velocity. It’s worthy to mention that of late few additional Vs are doing the rounds, variability – the increase in the range of values typical of a large data set — value, which addresses the need for valuation of enterprise data and volatility.


    Big data at the outset first means huge, gargantuan volumes of data. This data is generated by people, machines and networks. It is very common to have Terabytes and Petabytes of the storage system for enterprises. As the database grows in volume, the applications and architecture built to support the data needs to keep pace. This huge size of data is what represents Big Data.


    We are in the digital age and ‘recent data’ has a whole new meaning. Everything is real time and updates are reduced to fractions of seconds. This high velocity of data is another characteristic of Big Data. It refers to the pace at which data flows in from a variety of sources. The flow of data is continuous and in great amounts.


    When data comes in from a variety of sources both traditional and untraditional, in both structured and unstructured form, we call it variety. These varied sources create challenges in terms of storage, mining and analyzing and is another representation of what can be called Big Data.

    So why is the industry so excited about Big Data. Well in simple terms big data can create a significant competitive advantage for companies in every kind of industry. Big Data can also help to create new growth opportunities. There is even a whole new industry spurned out involving all those who manage this Big Data and aggregate and analyse it. And best of all, all of us as consumers also stand to gain from Big Data. It can and will even more potently in future impact our daily lives and make it better.

    It now makes it easier to understand the difference between Big Data and analytics. Big Data is the raw data that we have access to. It is huge, comes from various sources, it can be unstructured and untraditional and its velocity can be real time. Now the tools and technologies that is used to analyze this Big Data is what is called as analytics. Simple isn’t it? Yes that’s Big Data for you in a nutshell.

  • What Can Big Data Do For A Small Business?

    The goal of any business, big or small is to improve sales and profitability. The goal of any big data effort is to improve business. However complex big data may sound, if used correctly, small businesses can gain many insights and use it to make smart and intelligent decisions.

    There are several key advantages for small businesses that use big data. They can use it to identify key customers and improve their service to them. They can understand customer patterns, know when they’re likely to come in and reward them for multiple visits. This is also known as Loyalty Analytics. Whether one is in health care or the service industry, there is a wide scope of applying big data analytics. All it takes is creatively asking the right kind of questions that the data can answer. More or less data and analytics tools have been long used. While the analytical approach remains the same, what adds in with Big data is the use of adequateand more sophisticated technology.

    As data grows, so do the IT requirements. The challenge is to meet the gap of the business need and the IT infrastructure. To use Big Data smartly, the first requirement is to design the scope and objective of the analytics projects at hand. Based on the objective, the relevant data can be obtained. For example – Humungous amount of data can be generated at the customer transaction level. But suppose the business objective is to focus on customers in a certain geography, one has to think about how to filter the data to address the business problem.

    Big Data technology uses Hadoop and Mapreduce which is great but very expensive and not needed for small or mid- size businesses. Always keep an eye on the business objective before any investment is made purely based on price or hype. In most cases, we think about leveraging information management technologies like data integration and data quality to prepare data for analytics. Although this is certainly an important step, the biggest differentiator will be how business analytics can be applied to determine what to do with your organizational data, determine which data is relevant, and how or whether data should be stored.  While there is a lot of data, obtaining resources with the right kind of skills is a growing challenge. Some level of interaction and training within teams may also help in building skills relevant to generate business insights from big data.

    These days an increasing number of small businesses are collecting and crunching volumes of data to lift their sales.  There are many insights one can generate using Google Analytics for analysing web traffic, Facebook Insights, SumAll, etc. These tools are all easily available. The CRM tool collects all kinds of data which enables businesses to enhance user experience. Salesforce is a commonly used CRM platform.

    As big data grows and analytic tools become more affordable, small business must leverage from the big data movement. It’s time to look beyond what is a common belief that big data can only benefit large businesses. Small businesses that utilize big data will have a stronger understanding of their target markets and will be able to better cater to customers’ demand. Big data is a valuable asset for businesses of all size.