Engenheiro de Dados SR

Accountabilities:: Design, develop, and maintain robust, scalable, and efficient data pipelines to support advanced analytics and AI initiatives. Manage data ingestion, extraction, transformation, cleaning, standardization, and enrichment processes across large and diverse datasets. Design and implement modern data architectures based on Lakehouse concepts and the Medallion architecture (Bronze, Silver, and Gold). Transform raw data into structured, reliable, and reusable data products, with particular emphasis on the Gold layer. Prepare and deliver high-quality datasets for consumption by Artificial Intelligence solutions, Machine Learning models, and Large Language Models (LLMs). Develop and maintain ETL/ELT processes to integrate structured and unstructured data from multiple sources. Implement data validation, quality assurance, lineage, traceability, and monitoring mechanisms. Structure and process legal, judicial, contractual, and documentary datasets to support business and technology requirements. Develop solutions for processing large volumes of documents and extracting valuable information from unstructured data. Collaborate with Data Scientists, AI professionals, Technology teams, and Legal specialists to ensure effective data integration and delivery. Ensure compliance with data security, governance, access control, and Brazil's General Data Protection Law (LGPD). Identify opportunities to improve infrastructure performance, optimize costs, and increase the efficiency and scalability of data operations. Requirements: Solid professional experience in Data Engineering and data solution architecture. Strong proficiency in SQL and Python for data manipulation, transformation, and processing. Hands-on experience developing and maintaining ETL/ELT pipelines. Knowledge of Lakehouse architecture and the Medallion data model (Bronze, Silver, and Gold). Experience with Big Data platforms and distributed processing technologies such as Apache Spark and PySpark. Familiarity with cloud-based data storage and processing platforms. Experience integrating and processing both structured and unstructured data. Knowledge of data modeling and the development of Data Lakes and Data Warehouses. Experience with pipeline orchestration tools such as Apache Airflow or similar technologies. Understanding of data governance, quality management, lineage, and security best practices. Familiarity with preparing and optimizing datasets for Machine Learning models and Large Language Models (LLMs). Nice to have: Previous experience working on Generative AI projects and LLM-based solutions. Knowledge of Retrieval-Augmented Generation (RAG) architectures. Experience with document processing, Optical Character Recognition (OCR), and the extraction and structuring of textual information. Familiarity with vector databases and embedding techniques. Experience working with legal, judicial, contractual, or banking-related datasets. Knowledge of cloud-based data and AI services offered by AWS, Microsoft Azure, or Google Cloud. Experience with DataOps methodologies and CI/CD practices applied to Data Engineering. Benefits: The original job description does not specify salary, healthcare coverage, allowances, or additional employee perks. These details may be clarified during the recruitment process. Full-time employment. Remote working arrangement. Opportunity to work on innovative Data Engineering and Artificial Intelligence projects. Exposure to modern data architectures, Big Data technologies, and Generative AI solutions. Collaboration with multidisciplinary teams specializing in Data Science, AI, Technology, and Legal operations. Professional development opportunities in a technology-focused and innovation-driven environment.