This post is part of us “Netflix’s data engineer”Series, where our own data engineers talk about their journey Data Engineering: Netflix.
Pallavi Fadnis One of them is a senior software engineer Product Data Science and Engineering Team. In this post, Pallavi talks about her journey on Netflix and the challenges that keep her work interesting.
Pallavi received his master’s degree from Carnegie Mellon. Prior to joining Netflix, he worked in the advertising and e-commerce industries as a backend software engineer. In his spare time, he watches Netflix and enjoys traveling.
Her favorite shows: Stranger Things, Gilmore Girls and Breaking Bad.
Netflix’s unique work culture and petabyte-scale data issues have attracted me to Netflix.
In the early years of my career, I worked primarily as a backend software engineer, designing and creating backend systems that enable big data analysis. I have created many batches and real-time data pipelines using open source technology for AOL Advertising and eBay. I have also created an online service and microservices that drive Walmart’s e-commerce.
That year’s experience in solving technical problems for different businesses has taught me that data has the potential to maximize the potential of any product.
Before I joined Netflix, I was always fascinated by my experience as a Netflix member which made a great impression on me from the Netflix engineering teams. When I read Netflix Culture Memo For the first time, I was fascinated by how candid, direct and transparent the work culture was. These cultural points resonated the most with me: freedom and responsibility, high bar for performance, and no hiring of bright shakes.
Over the years, I have followed the Big Data open-source community and Netflix technology blogs closely and learned a lot about Netflix’s innovative engineering solutions and active contributions to the open-source ecosystem. In 2017, I attended the Women in Big Data Conference on Netflix and met some amazing women in data engineering, including our VP. At this conference, I also got to know my current team: Integrated Logging (CL).
CL provides end-to-end solutions for logging, processing, and analyzing user interactions in the Netflix app from all devices. This is important for fast-paced product innovation on Netflix because CL provides basic data for personalization, A / B testing, and performance analysis. Moreover, its petabyte scale also brings unique engineering challenges. Job opportunities, business impact and CL’s engineering challenges were very exciting to me. Also, the CL team roles require a mix of data engineering, software engineering and distributed system skills, which line up really well with my interests and skills.
What I am most proud of for the project is building an integrated logging V2 platform that processes and enriches data on a huge scale (5 million + events per second) in real-time. I have redesigned our legacy data pipelines and built a new platform on top of Apache Flink and Iceberg. The scale has brought some interesting lessons and involves working closely with the Apache Flink and Kafka communities to address key issues. Thanks to the transfer to V2, we’ve been able to significantly improve data availability and usability for our customers. The skills gained in processing and storage levels have brought us huge savings in computing and storage costs. You can learn more about it from my lecture at the Flink Forward Conference. Over the past few years, we’ve been constantly improving the V2 platform to support more logging usage outside of the Netflix streaming app and to provide more analytics capabilities. For example, I’m currently working on a project to create a general analysis solution for 100s Netflix Studio and internal apps.
The role of data engineering in Netflix is similar in many ways to the role of software engineering. Both roles design and develop large-scale solutions using a variety of open-source technologies. In addition to business logic, ensuring production stability requires a good understanding of the structure and infrastructure of their structures, for example, maintaining SLA to reduce impact on upstream and downstream applications. At Netflix, it is fairly common for data engineers and software engineers to collaborate on similar projects.
In addition, data engineers are responsible for designing data logging specifications and optimized data models to ensure that the desired business questions are answered. Therefore, they need to understand the depth of data in terms of product and business use.
In other words, data engineers bridge the gap between data producers (such as client UI teams) and data consumers (such as data analysts and data scientists).
Interested in learning more about the role of data in Netflix? You are in the right place! Keep an eye out for our open role in data science and engineering by visiting our job site here. Our culture is the key to our influence and growth: read about it here. To learn more about our data engineers, check out our chats with Dhevi Rajendran, Samuel Setagne and Kevin Wiley.