Data Engineer
technologies-expected :
- Microsoft Azure
- Google Cloud Platform
- Rockset
- SQL
- MongoDB
- Python
- Microsoft Excel
- Git
technologies-optional :
- Torch/PyTorch
- TensorFlow
- MySQL
- PostgreSQL
- Redis
- Snowflake Data Cloud
- AWS
about-project :
- We aim to develop a comprehensive Business Artificial Intelligence (BAI) platform tailored specifically for production companies. The architecture of this platform will be designed to encompass the following key components:
- Integrations:
- APIs of Various Systems: The platform will integrate with the APIs of various systems that businesses currently use, ensuring seamless data flow and interaction.
- Data Warehouse: A centralized data warehouse will be established to aggregate and store data from different sources, enabling efficient data management and retrieval.
- AI Agents:
- SQL Agents for Analytics: These agents will be responsible for querying the data warehouse, performing complex analytics, and generating insights.
- API Agents: These agents will interact with external APIs, facilitating both read and write operations to ensure real-time data synchronization and interaction.
- Generative UI:
- The AI will have the capability to generate the platform’s user interface dynamically. This feature is particularly useful for companies using systems without existing APIs, allowing them to transfer and streamline their operations into our platform seamlessly. ]
- Extensions:
- Prediction Models and More: The platform will support various extensions such as predictive modeling to forecast trends, anomalies, and other critical metrics, enhancing decision-making processes.
- Ultimately, our vision is to develop AI that operates on employees' machines within the company. This AI will observe their activities, gradually assist in optimizing tasks, and provide feedback to the board on where operational bottlenecks exist. This continuous improvement loop aims to enhance overall productivity and operational efficiency.
- Our initial focus will be on production companies, ensuring the platform addresses the unique challenges and requirements of this sector.
responsibilities :
- Develop, implement, and maintain procedures for data cleansing to ensure data accuracy and consistency.
- Managing data ingestion aspects, batch and near-real-time, with multiple source types
- Design, build, and maintain data warehouse solutions that meet business needs.
- Develop and optimize ETL processes to efficiently load data into data warehouses.
- Integrate data from various sources, including databases, APIs, and flat files.
- Troubleshooting and bug fixing
- Work closely with business analysts, data scientists, and other stakeholders to understand data requirements.
- Collaborate with IT teams to ensure proper data infrastructure and architecture.
- Stay updated with the latest trends and technologies in data engineering and data warehousing.
- Utilize advanced tools and technologies such as SQL, Python, ETL tools, and cloud-based data warehousing solutions (e. g. , AWS Redshift, Google BigQuery).
- Develop automation scripts and tools to improve data processing efficiency.
- Develop and maintain data pipelines to support continuous data flow between systems.
requirements-expected :
- 2+ years of experience in data engineering, data warehousing, or a similar role
- Strong experience with data warehouse technologies like Azure Synapse Analytics, Amazon Redshift, Google BigQuery, Snowflake, or similar.
- Proficiency with ETL
- Proficiency in SQL and Python
- Experience with real-time data processing technologies
- Strong understanding of relational databases (e. g. , MySQL, PostgreSQL, Oracle) and NoSQL databases (e. g. , MongoDB).
- Experience with cloud platforms such as AWS, Google Cloud Platform, or Microsoft Azure, including data storage and processing services.
- Knowledge of data integration tools and practices, including APIs, data streaming, and data replication.
offered :
- Young and Energetic Team: Join a dynamic and enthusiastic team that thrives on creativity and innovation. Our collective energy and passion drive us to challenge the status quo and make a significant impact.
- Fast-Paced Learning: Engage in continuous learning and professional development. Working with cutting-edge technology and solving complex problems will rapidly enhance your skills and knowledge.
- Mentorship and Collaboration: Benefit from close collaboration with experienced professionals and peers, fostering a culture of mentorship and collective growth.
- Making a Difference: Contribute to groundbreaking projects that have the potential to revolutionize business operations. Your work will directly impact our clients, helping them optimize tasks and improve efficiency.
- Ownership and Responsibility: Take on meaningful responsibilities from day one. Your contributions will be valued, and you’ll have the autonomy to drive projects forward and see your ideas come to life.
- Cutting-Edge Technology: Work with the latest technologies in artificial intelligence, machine learning, and data analytics. Be at the forefront of technological advancements and contribute to the development of pioneering solutions.
- Startup Culture: Experience the excitement and agility of a startup environment. Our small, ambitious team allows for quick decision-making and the ability to make a tangible impact.
Bądź pierwszy, który ubiega się o to miejsce pracy!
-
Dlaczego szukać pracy na HitPraca.pl?
Subskrybuj oferty pracy
Codziennie nowe oferty pracy Możesz wybierać z bardzo szerokiej gamy ofert pracy - naszym celem jest posiadanie jak najszerszej oferty pracy Otrzymuj nowe oferty e-mailem Bądź pierwszym, który odpowie na nowe oferty pracy Wszystkie oferty pracy w jednym miejscu (od pracodawców, agencji pośrednictwa pracy i innych portali) Wszystkie usługi dla kandydatów do pracy są bezpłatne Pomożemy Ci znaleźć nową pracę