Lead Big Data Engineer

Data Engineer (AWS/Databricks)

About Us

“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery.

WHY JOIN CAPCO?

You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry.

MAKE AN IMPACT

Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.

BEYOURSELFATWORK

Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.

CAREER ADVANCEMENT

With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.

DIVERSITY & INCLUSION

We believe that diversity of people and perspective gives us a competitive advantage.

Job Title: Senior Data Engineer (AWS/Databricks)
Experience: 5+ Years
Location: Pune
Interview Process: One mandatory in-person interview round at the client's office in Pune

Job Summary

We are looking for a highly skilled and experienced Senior Data Engineer (AWS/Databricks) with 5+ years of hands-on experience in designing, developing, and optimizing scalable data engineering solutions.

The ideal candidate should have strong expertise in Python, SQL, PySpark, AWS, Databricks, ETL/ELT pipelines, and cloud-based data platforms. The role requires strong technical capabilities in distributed data processing, data modeling, performance optimization, and building enterprise-grade data solutions.

The candidate should be capable of independently delivering complex data engineering solutions, collaborating with cross-functional teams, and implementing engineering best practices.

Key Responsibilities

• Design, develop, and maintain scalable, reliable, and high-performance data pipelines using Python, PySpark, SQL, and Databricks.
• Build and optimize ETL/ELT pipelines for large-scale data ingestion, transformation, and integration.
• Develop and manage cloud-based data engineering solutions leveraging AWS services and Databricks.
• Write complex SQL queries, perform data modeling, and implement query optimization and performance tuning techniques.
• Design and implement distributed data processing solutions using Apache Spark and PySpark.
• Work with data lakes, cloud-based storage solutions, and enterprise data platforms.
• Implement CI/CD pipelines, automated deployments, and DevOps best practices for data engineering workloads.
• Ensure data quality, governance, security, compliance, and reliability across data pipelines.
• Troubleshoot pipeline failures, identify performance bottlenecks, and implement optimization strategies.
• Collaborate with data architects, data scientists, business analysts, and engineering teams to deliver end-to-end data solutions.
• Participate in Agile/Scrum ceremonies, technical discussions, design reviews, and code reviews.
• Contribute to the adoption of modern cloud-native data engineering practices and emerging technologies.

Required Technical Skills

1. Python

• Strong hands-on programming expertise in Python.
• Experience developing production-grade data engineering applications, automation scripts, and reusable components.
• Ability to write clean, efficient, maintainable, and scalable code.

2. SQL & Data Modeling

• Expert-level SQL skills, including complex joins, window functions, CTEs, and advanced data transformations.
• Strong experience in data modeling, query optimization, and performance tuning.
• Ability to design efficient data structures for large-scale analytical workloads.

3. PySpark & Distributed Data Processing

• Strong hands-on experience with PySpark and Apache Spark.
• Expertise in distributed data processing, Spark transformations, partitioning, caching, and performance optimization.
• Experience processing large datasets in distributed computing environments.

4. AWS & Databricks

• Hands-on experience with AWS cloud-based data engineering solutions.
• Experience working with Databricks for large-scale data processing and transformation.
• Understanding of cloud-based data platforms, data lakes, and modern data architectures.
• Exposure to relevant AWS services such as S3, Glue, Lambda, EMR, and Redshift is advantageous.

5. ETL/ELT & Data Integration

• Strong experience designing, developing, maintaining, and optimizing scalable ETL/ELT pipelines.
• Experience integrating data from multiple structured and semi-structured sources.
• Understanding of data ingestion, transformation, orchestration, and pipeline monitoring.

6. CI/CD & DevOps

• Experience implementing CI/CD pipelines for data engineering applications.
• Understanding of version control, automated testing, deployment automation, and DevOps best practices.
• Knowledge of containerization, orchestration, and cloud-native architectures.

7. Data Governance, Quality & Security

• Strong understanding of data governance, data quality frameworks, and data security principles.
• Experience implementing data validation, monitoring, and compliance controls.
• Ability to build reliable and maintainable enterprise data solutions.

Good to Have Skills

• Experience with Azure or GCP cloud platforms.
• Familiarity with machine learning concepts, MLOps practices, and AI/ML-enabled data platforms.
• Exposure to Generative AI, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), AI agents, and modern AI technologies.
• Knowledge of Docker, Kubernetes, and cloud-native deployment patterns.
• Experience with modern data lakehouse architectures and advanced Databricks capabilities.

Required Experience & Qualifications

• Minimum 5+ years of relevant experience in Data Engineering.
• Strong practical experience in Python, advanced SQL, PySpark, and ETL/ELT development.
• Hands-on exposure to AWS and/or other cloud platforms, with Databricks experience preferred.
• Proven experience working with large-scale data platforms and distributed data processing frameworks.
• Strong analytical, debugging, problem-solving, and communication skills.
• Experience working in Agile/Scrum environments.
• Ability to collaborate effectively with cross-functional technical and business stakeholders.
• Bachelor's or Master's degree in Computer Science, Information

If you are keen to join us, you will be part of an organization that values your contributions, recognizes your potential, and provides ample opportunities for growth. For more information, visit www.capco.com. Follow us on Twitter, Facebook, LinkedIn, and YouTube.