Three reasons why Big Data projects fail

technology-3200401_1280

I have not been regular with my personal blog because I have been blogging elsewhere.

Here are the links to my latest blog posts about why Big Data projects fail and how to attract more women into tech.

Having worked extensively in the Big data & IoT space I have closely observed failures over and over again and the reasons for failure being repetitive :

  • Wrong use cases
  • Wrongly staffed projects
  • Obsolete technology

Read the blog post for more details:

Three reasons why Big Data projects so often fail

Being a woman in tech or woman in data I am often the only woman in meetings, trainings and discussions which feels weird. With not many women in tech it gets easier to discriminate the few that do exist. Incidents of mansplaining, gaslighting are rampant and it’s the victim that gets labelled as drama queen while the abusers fo scot free. Organisations that are serious about increasing the number of women in tech need to address glass ceiling, gender wage gaps & bro-culture and cultivate an inclusive work atmosphere. Read my post on how to get more women into tech.

How to Get More Women in Tech

How to become big data – data analyst

Anyone who works in the tech industry is aware of the rising demand of Analytics/ Machine learning professionals. More and more organisations have been jumping on to the data driven decision making bandwagon, thereby accumulating loads of data pertaining to their business. In order to make sense of all the data gathered, organisations will require Big Data Analysts to decipher the data.

  Data Analysts have traditionally worked with pre formatted data, that was served by the IT departments, to perform analysis. But with the need for real time or near-real time Analytics to serve end customers better and faster, analysis needs to be performed faster, thereby making the dependency on IT departments a bottleneck. Analysts are required to understand data streams that ingest millions of records into databases or file systems, Lambda architecture and batch processing of data to understand the influx of data.

Also analysing larger amounts of data requires skills that range from understanding the business complexities, the market and the competitors to a wide range of technical skills in data extraction, data cleaning and transformation, data modelling and statistical methods.

Analytics being a relatively new field, is struggling to resource the market demands with highly skilled Big Data Analysts. Being a Big Data Analyst requires a thorough understanding of data architecture and the data flow from source systems into the big data platform. One can always stick to a specific industry domain and specialize within that, for example Healthcare Analytics, Marketing Analytics, Financial Analytics, Operations Analytics, People Analytics, Gaming Analytics etc. But mastering the end-to-end data chain management can lead to plenty of opportunities, irrespective of industry domain.

The entire Data and Analytics suite includes the following gamut of stages:

  • Data integrations – connecting disparate data sources
  • Data security and governance – ensuring data integrity and access rights
  • Master data management – ensuring consistency and uniformity of data
  • Data Extraction, Transformation and Loading – making raw data business user friendly
  • Hadoop and HDFS – big data storage mechanisms
  • SQL/ Hive / Pig – data query languages
  • R/ Python –  for data analysis and mining programming languages
  • Data science algorithms like Naive Bayes, K-means, AdaBoost etc. – Machine learning algorithms for clustering, classification
  • Data Architecture – solutionizing all the above in an optimized way to deliver business insights

The new age data analysts or a versatile Big Data Analyst is one who understands the complexity of data integrations using APIs or connectors or ETL (Extraction, Transformation and Loading), designs data flow from disparate systems keeping in mind data security and quality issues, can code in SQL or Hive and R or Python and is well acquainted with the machine learning algorithms and has a knack at understanding business complexities.

Since Big Data and Analytics is constantly evolving, it is imperative for anyone aiming at a career within the same, to be well versed with the latest tech stack and architectural breakthroughs. Some ways of doing so:

  • Following knowledgeable industry leaders or big data thought leaders on Twitter
  • Joining Big Data related groups on LinkedIn
  • Following Big Data influencers on LinkedIn
  • Attending events, conferences and seminars on Big Data
  • Connecting with peers within the Big Data industry
  • Last but not the least (probably the most important) enrolling in MOOC (Massive Open Online Course) and/ or Big Data books

Since Analytics is a vast field, encompassing several operations, one could choose to specialise in parts of the Analytics chain like data engineers – specializing in highly scalable data management systems or data scientists specializing in machine learning algorithms or data architects – specializing in the overall data integrations, data flow and storage mechanisms. But in order to excel and future proof a career in the world of Big Data, one needs to master more than one area. A data analyst who is acquainted with all the steps involved in data analysis from data extraction to insights is an asset to any organization and will be much sought after!

Data integration is not a choice!

samsung-793043_640Every organization irrespective of industry has several business processes, each business process being supported by several IT products. Each of these IT products have an insurmountable amount of information that can generate insights which are paramount for any organization. Businesses that have been around for a while have obsolete processes and legacy systems that support the same. A typical organization independent of industry has transaction processing systems, CRM systems, ERP, billing and business analytics solutions. Each solution in itself is a silo if not integrated with the rest of the solutions. Granted that each of these solutions harbour valuable information but the the information residing in each system does not generate a holistic view of the business.

Integrating the silos is a Herculean task, or so it may seem, if the solutions are outdated and do not support APIs, plug-ins and adapters. Most CRM, ERP, Marketing automation products, lately are equiped with some form of connector, enabling data blending. If an organization has systems that do not support the above, then it is wise to migrate or upgrade the solutions to versions compatible with data extraction. Migrating legacy systems is a rocky road but the trade off being elimination of data silos. Often the implementation cycle of new software solutions are so long that the idea becomes outdated even before the roll out. Ofcourse there exist solutions with shorter time-to-market, for example data analytics platform that are run on Spark have a faster implementation cycle and are scalable, providing the flexibility that growing businesses need.

It was not long ago that marketing and data analytics borders got blurred due to new business needs. This has resulted in complex technological challenges. Not all businesses have the budget and resources to invest in migrating and upgrading most of the legacy systems. But in order to appease todays demanding customers, data integration is the key. No customer would like to remember or rummage through their homes to find old reciepts or mails when they call the customer care for a service or to complain. They would very much expect that on identifying themselves, the customer care representative not only solves their grievances but also comes up with suggestions to improve their customer lifecycle, which can be only attained by integrating data from disparate systems to gain a 360 degree view of the customer journey. Data integration, thus is a matter of being in business or out.

To start with, businesses should identify each data silo that exists and the function that each of them fulfill. (There maybe exist examples of one business process that is fulfilled by several software solutions. If an organization lacks data governance, then the number of redundant solutions and products can be plenty.) Listing and mapping business processes to softare solutions clarifies the current architecture. The next process is

  • To identify the to-be roadmap
  • Map solutions that support data blending, to each of the business process whiteboard-849810_640

The solutions that are adapted for new age businesses require to embody the following characteristics:

  • Easy to implement
  • Short implementation time
  • Compatability with a wide range of disparate systems
  • Easy to implement data security and access rights
  • Scalable
  • Forward compatible

Businesses need technology that support business gain and growth and the ever changing rules of the game (read disrutption).

The Start-up Lifecycle

The start-up industry may appear very glamorous from the surface, but it entails endless meticulous planning and back breaking hard work, it’s anything but a cake-walk. Most success stories that come into limelight have already gone through testing times and been bitten by failure at some stage or the other and have survived the winds of change.FullSizeRender

The typical stages in a start-up are more or less as below:

  • Ideate – An idea that has been incubating for a while gets more concrete and is at a stage where it can be implemented. It has to be beyond the pen and paper stage, on the path to a more concrete objective.
  • Feasibility study – validate your idea, think through all the possibilities, market demands, fall back plan and get an initial feedback from friends and family about the viability of the idea.
  • Conceptualize the idea into a business case, planning the inception, the initial capital required, the source of fund, the launch of beta version of the product and the marketing of the same to acquire a customer base.
  • While you’re at it, you’ll need to create an online presence in this age of digitalization, you will have to be visible. Brand building and audience buying begins even before the actual launch of the product. Send out teasers in the social media, engage your potential customers and get them interested in your product.
  • To allure the venture capital funding, your product has to be foolproof in this age of competition. The VC firms have to see a potential market for the product to be convinced to invest in it. Be prepared to be grilled.
  • Midway, if gets bumpy and you realize start-up life is just not your cuppa, you should return to your plan B, in case you give up. But if you have a heart made of steel, there’s no stopping you….
  • Then slog even harder. Learn from mistakes. Curate and analyze:
  1. consumer behavior both offline and online
  2. abandoned baskets
  3. competitive analysis
  4. social media analysis
  5. shopping history

Success is hard work and in order to acquire new customers and retain the existing, there’s no short cut but to get the above steps right, if it does not work at once – reiterate. When success does come by, in the form of reaching targeted number of customers or reaching the targeted revenue, all the hard work does pay off. The incredible feeling of having achieved something, which is your own, based on your accomplishments, is almost hedonistic. Handmade success!