Data Engineer (Databricks, AI & Data Governance)
Data Engineer (Databricks, AI & Data Governance)
Job type:
Skillhouse Contract
Specialization:
Data and Analytics
Language Level:
English Level - Advanced (TOEIC 860), Japanese Level - None
Location:
Sumida-Ku
Salary:
¥700,000 - ¥900,000 Yearly
Job Reference:
507781
A global and one of the world’s largest Insurance Service provider is seeking an experienced Data Engineer (Databricks, AI & Data Governance) to join its Data Management Team.
The successful candidate will design, develop, and maintain scalable data pipelines and data platforms using Azure Databricks, Python, SQL, and Azure Cloud technologies.
The role will focus on modernizing enterprise data platforms, supporting AI/ML initiatives, and implementing strong Data Governance practices to ensure data quality, security, and reliability.
Responsibilities:
- Design, develop, and maintain scalable data pipelines using Azure Databricks, Python, PySpark, and SQL
- Build and optimize ETL/ELT processes for large-scale data ingestion, transformation, and processing
- Develop and maintain data solutions using Apache Spark, Delta Lake, and Azure data services
- Implement Data Governance practices, including data quality, security, access controls, and data lineage
- Utilize Databricks Unity Catalog for centralized data management and governance
- Support AI/ML and Generative AI initiatives by preparing reliable, high-quality datasets for model development and deployment
- Automate data workflows, testing, monitoring, and deployments using CI/CD tools
- Monitor and troubleshoot data pipelines to improve system performance and reliability
- Collaborate with technical and business teams to translate requirements into scalable data solutions
- Maintain technical documentation and follow enterprise data engineering best practices
Required Skills:
- 5+ years of experience in Data Engineering, software development, or enterprise data platforms
- Strong hands-on experience with Azure Databricks, Apache Spark, and PySpark
- Strong programming skills in Python and SQL for building and optimizing data pipelines
- Experience with ETL/ELT development, data integration, and large-scale data processing
- Knowledge of Delta Lake, Databricks Unity Catalog, and Data Governance, including data quality, security, and lineage
- Experience with Microsoft Azure Cloud, including Azure Data Lake Storage, Azure Data Factory, or similar services
- Familiarity with AI/ML data pipelines, MLFlow, or supporting AI applications
- Experience with Git, CI/CD, data pipeline monitoring, and troubleshooting
- Strong analytical, problem-solving, and communication skills
Why should you apply:
- Work on enterprise Databricks, AI, and Data Governance initiatives at a leading global insurance company
- Gain experience with modern cloud data platforms and AI-ready data infrastructure
- Long-term contract opportunity with potential extensions
- International working environment with English as the primary communication language
Company Details:
A US based world’s leading insurance providers, offering a broad range of life, health, and retirement solutions to individuals, families, and businesses. The company is heavily invested in digital transformation, utilizing advanced technologies like cloud computing, data analytics, AI, and cybersecurity to enhance customer experience and streamline operations. As part of its values, it has a strong focus on creating a diverse environment, and in particular on the appointment of women in high-level position.
Working Hours: 9:00 - 18:00 (Mon-Fri)
Working Style: 3 days’ work in office, and 2 days’ work from home
Holidays: Saturday, Sunday, National Holidays, Year-end and New Year Holidays, Paid Holidays
Services/Benefits: Transportation expenses up to 20,000 yen per month, plus Paid leave, plus social insurance (health insurance, welfare pension, and work-related accident insurance), Periodic health examination, and Employment insurance
Share this job
