Talk details

As businesses increasingly rely on data to drive decision-making, the demand for effective data engineering solutions continues to grow. Central to this field are data pipelines—structured processes that ensure data flows seamlessly from its source to its final destination, enabling comprehensive analysis and informed decision-making.
There is a wide variety of tools and technologies available for building data pipelines. Major cloud providers offer robust SaaS solutions, and there are numerous open-source options that can be self-hosted on IaaS or PaaS platforms. However, in certain edge cases, these solutions may not be suitable due to specific requirements or performance constraints.
If you’ve encountered challenges with existing tools or are seeking a more customized approach to data pipeline development, this talk is for you. I will share my experience in designing lightweight, tailored data pipelines using Python for specific use cases. We will explore best practices for managing priorities, ensuring availability, and optimizing performance while also covering essential aspects of monitoring and error handling.
Join me to learn how to craft efficient data pipelines that meet your unique requirements and drive better business results.