Four Reasons Why Data Engineering is a Zero-Sum Game for Most Organizations

September 17th, 2018   Data engineering is hard and getting it exactly 100% right results in a single outcome – machine learning engineers and quants can now do their job effectively.  The analysis of data and the subsequent execution of those insights is the competitive differentiator and core competency of business – its heart and soul.  Data engineering is the commoditized heavy-lifting every organization needs to perform to get the analysis correct.  This is why we see data engineering as a zero-sum game.  Getting data engineering right means organizations are just breaking even – it simply allows other employees to do their job properly.  Getting it wrong means everything and everyone else dependent on data engineering cannot operate effectively. Outsourcing the commoditized heavy-lift data engineering is the least risky and most cost-efficient path to achieve the economic and market-leading competitive advantages organizations need to compete.     Prioritize Algorithm Development Over Data Engineering Modern organizations should prioritize and invest in the algorithm development, quantitative research, and machine learning aspects of data science.  These activities can make or break firms who use data for a competitive advantage.  Applying machine learning in a meaningful way using data formatted specifically for those algorithms is not a trivial task.  To be successful, organizations should recognize the undifferentiated and differentiated activities associated with extracting insight from data and decouple the activities required to get data into a specific format (or schema) to support those algorithms from the development and tuning of those algorithms.   Race Car Drivers and Data Mechanics An interesting social phenomenon we’ve observed over the past several years is that we have yet to meet a data engineer that wasn’t secretly plotting a career change to become a machine learning engineering and/or quant and with a more data science-centric job title to-boot.  If machine learning engineers and quants...