CloudIngest hiring Data Engineer

Key Responsibilities Phase 1 Deliverables • Following EDB standard, design and establish AWS S3 bucket structure for data lake (Bronze/Silver/Gold zones) with Red CCI security controls • Build new data pipelines to extract data from the selected SaaS-based HSE systems • Enhance/extend data pipelines using the dataset from AWS EDB Asset related data products and digital solutions • MES selection to be finalized by the end of Jan 2027 - Build data ingestion pipeline for selected MES via REST API (Lambda-based extraction). If a different MES is chosen, change course accordingly • Data pipeline to SaaS based HSE tools to integrate data into the data warehouse • Implement CDC pipeline for LabVantage LIMS using AWS DMS from Oracle database (or Azure Data Factory) • Develop Bronze-to-Silver transformations using AWS Glue or Azure Data Factory depending on data domains • Configure AWS Glue Data Catalog with appropriate metadata and Red CCI classification tags Phase 2 Deliverables • Connect to EDB marketplace for enterprise reference data • Build Silver-to-Gold transformations creating batch-centric data products • Implement data quality checks and monitoring dashboards • Develop orchestration workflows using AWS Step Functions • Support Power BI Developer in validating OneLake shortcut connectivity to Gold zone • Document data lineage, schema definitions, and pipeline architecture Ongoing Responsibilities • Monitor pipeline health, troubleshoot failures, and optimize performance • Collaborate with Power BI Developer on data model requirements and data quality issues • Support data governance and security compliance reviews • Respond to ad-hoc data requests from engineers • Coordinate with enterprise EDB team on data sharing agreements and standards Show less [{"type":"people_also_viewed","href":"https://www.linkedin.com/jobs/view/data-analyst-at-sti-4368833518","jobId":"4368833518","label":"Data Analyst"},{"type":"people_also_viewed","href":"https://www.linkedin.com/jobs/view/data-engineer-manager-at-pwc-4369137679","jobId":"43691376...