Data Platform Section (DPS) is building out the next generation enterprise data platform that will change the way users find, query and analyse data at massive scale and become the hub of analytical innovation. The next generation platform not only looks to automate the processes involved with ingesting, discovering, governing and querying data but also support the company's drive to become completely data-driven across more than 70 services.
They are looking for an end-to-end data engineer with experience in the development of data pipeline platforms and in the modelling and querying of data for Business Intelligence purposes.
The technical architecture comprises a Teradata and Hadoop platform utilizing Python/Linux batch and also Kafka data ingestion mechanisms.
This is a Dev/Ops role where you would be responsible for supporting existing production data pipelines and adhoc BI enhancement requests, while expanding our new Hadoop based data platform.
- Develop, enhance and maintain data pipeline applications and data models on a rotational on-call basis in a 24x7x365 environment.
- Trouble-shoot the causes of adhoc daily production failures and provide effective and documented solutions.
- Continuous improvement initiatives in data ingestion performance, ingestion models, data integrity and data availability.
- Work with the business in analyzing and documenting new functionality requests and managing the implementation of those within an Agile ownership model.
【会社概要 | Company Details】
Our client is a large global Internet service company that has enjoyed sustained growth as they continue to expand their business in various new areas and industries. This is a great opportunity to work in a diverse and international environment in Japan. Our client actively strives to be an equal opportunity employer, and they have many female and foreign nationals in upper management positions. Their brand has also gained global recognition as they sponsor some of the world's most famous sports teams. Our client prides themselves in providing a comfortable working environment for their employees. Engineers are welcome to choose their own setup (Windows/Mac, etc.); whatever makes them comfortable! Free meals are also provided at the company cafeteria. Their chefs work to create exciting new menus and dishes, so employees never get tired of the food!
【就業時間 | Working Hours】
Flextime（Mon - Fri）
【休日休暇 | Holidays】
Saturday, Sunday, and National Holidays, Year-end and New Year Holidays, Paid Holidays, Other Special Holidays
【待遇・福利厚生 | Services / Benefits】
各種社会保険完備（厚生年金保険、健康保険、労災保険、雇用保険）、 屋内原則禁煙（屋外に喫煙所あり）、 通勤交通費支給等
Social insurance, Transportation Fee, No smoking indoors allowed (Designated smoking area), etc.
- Expert SQL capability in querying Big Data/ large data sets (Teradata, Hadoop, etc.) to extract BI- insights.
- Programming languages such as Python/Scala/PLSQL/Java.
- Development and operation of data pipeline leveraging big data technologies such as Spark, Map Reduce, Hive, Kafka, Sqoop, NoSQL Databases as well as traditional DB and file based data integration solutions.
- Database development (eg TeraData, Oracle, MySQL, SQLServer, DB2.)
- Shell-scripting languages such as Bash.
- Formal analysis and documentation of BI solutions.
- Distributed version control system such as Git.
- Initiative and the ability to work independently and in a team. (The team uses Agile methodology.)
- Application development using workflow engines such as Airflow, Oozie, Rundeck.
- BI reporting tools, including administration, modeling, and report/dashboard development.
- BI Modelling of data marts using ER hybrid, Kimball, Data Vault methodologies.
- Experience in ELK Stack.
- Experience in AtScale and Presto.
- Operational experience in developing and supporting high availability applications / systems.
- Capability to self-manage and also manage small projects.