Senior Data Engineer (databricks)
Project overview
Our client is embarking on a significant transformation project aimed at replatforming our financial system. Two initiatives will run in parallel: one for all financial and second for all non-financial data. We are in the early stages of an ambitious project to consolidate data from various sources into a single Delta Lake data warehouse.
Responsibilities
- Lead the integration and ingestion of data from diverse sources into the Delta Lake platform, ensuring data consistency and availability.
- Collaborate with various teams to identify all data sources, including SFTP / flat files, and APIs.
- Design and implement systematic processes for data ingestion and integration.
- Conduct extensive data analysis to support the consolidation of organizational data into a single data warehouse.
- Utilize Databricks for data ingestion and reporting, transitioning data from multiple data warehouses to a unified Delta Lake platform.
- Work closely with the development team to understand project scope, gather full requirements, and plan subsequent phases.
- Participate in daily standups, sprint planning, backlog refinement, and retrospectives within a scaled scrum framework.
- Exhibit flexibility and adaptability in navigating unknowns and evolving project requirements.
Requirements
- Bachelorâs or Masterâs degree in Computer Science, Data Science, or a related field.
- Solid understanding of the software development life cycle.
- Proven experience with Python, Spark and SQL for data engineering and analysis.
- Strong data analysis skills, with the ability to interpret and manipulate large datasets.
- Experience with Databricks for data ingestion and transformation.
- Familiarity with Delta Lake and data warehousing concepts.
- Excellent problem-solving skills and the ability to see the big picture and understand the broader context of assigned tasks.
- Strong teamwork and collaboration skills, with a demonstrated ability to work effectively in a team-oriented environment.
- Knowledge of reporting tools like Power BI.
- Knowledge of Data Science concepts is a plus.
- Strong communication, interpersonal and presentation skills.
Bądź pierwszy, który ubiega się o to miejsce pracy!
-
Dlaczego szukać pracy na HitPraca.pl?
Subskrybuj oferty pracy
Codziennie nowe oferty pracy Możesz wybierać z bardzo szerokiej gamy ofert pracy - naszym celem jest posiadanie jak najszerszej oferty pracy Otrzymuj nowe oferty e-mailem Bądź pierwszym, który odpowie na nowe oferty pracy Wszystkie oferty pracy w jednym miejscu (od pracodawców, agencji pośrednictwa pracy i innych portali) Wszystkie usługi dla kandydatów do pracy są bezpłatne Pomożemy Ci znaleźć nową pracę