Big data Engineer (4101)

Job Type
Haken
Salary
JPY 3,400 - JPY 3,700 per hour
Japanese Level
None
English Level
Advanced (TOEIC 860)
Start Date
ASAP
Location
Tokyo

Description

Data Platform Section (DPS) is building out the next generation enterprise data platform that will change the way users find, query and analyse data at massive scale and become the hub of analytical innovation. The next generation platform not only looks to automate the processes involved with ingesting, discovering, governing and querying data but also support the company's drive to become completely data-driven across more than 70 services.

They are looking for an end-to-end data engineer with experience in the development of data pipeline platforms and in the modelling and querying of data for Business Intelligence purposes.

The technical architecture comprises a Teradata and Hadoop platform utilizing Python/Linux batch and also Kafka data ingestion mechanisms.

This is a Dev/Ops role where you would be responsible for supporting existing production data pipelines and adhoc BI enhancement requests, while expanding our new Hadoop based data platform.

 

【Responsibilities】

- Develop, enhance and maintain data pipeline applications and data models on a rotational on-call basis in a 24x7x365 environment.

- Trouble-shoot the causes of adhoc daily production failures and provide effective and documented solutions.

- Continuous improvement initiatives in data ingestion performance, ingestion models, data integrity and data availability.

- Work with the business in analyzing and documenting new functionality requests and managing the implementation of those within an Agile ownership model.

 

【会社概要 | Company Details】
グローバルに展開する一部上場インターネットサービス企業です。様々な事業を展開し、成長を続けています。日本にいながら異文化コミュニケーションができる国際的な環境かつ、女性や外国籍の管理職登用にも積極的なダイバーシティな環境でもあります。世界的にも知名度アップを図り、スポーツチームのスポンサーなども積極的に行っています。
エンジニアの方には嬉しい、WindowかMacを自分で選択することができます!社内カフェテリアは明るく、毎日食べても飽きないほど、メニューが豊富です!
スキルハウスから就業しているスタッフも多く、年間成約数もトップレベルを誇ります。会社同士の信頼関係も厚く、面接のプロセスもスムーズです。実際に企業に応募する前に、経験豊富な企業の専任担当より会社の様子など、詳しくお伝えいたします。また、日本語・英語でのレジュメの書き方のアドバイスや面接の準備もいたします。ケースbyケースですが、社内ツアーを設定することもできます。

Our client is a large global Internet service company that has enjoyed sustained growth as they continue to expand their business in various new areas and industries. This is a great opportunity to work in a diverse and international environment in Japan. Our client actively strives to be an equal opportunity employer, and they have many female and foreign nationals in upper management positions. Their brand has also gained global recognition as they sponsor some of the world's most famous sports teams. Our client prides themselves in providing a comfortable working environment for their employees. Engineers are welcome to choose their own setup (Windows/Mac, etc.); whatever makes them comfortable! Free meals are also provided at the company cafeteria. Their chefs work to create exciting new menus and dishes, so employees never get tired of the food!
 
【就業時間 | Working Hours】
Flextime(Mon - Fri)
 
【休日休暇 | Holidays】
完全週休2日制(土日祝休み)、年末年始、年次有給休暇、その他特別休暇など
Saturday, Sunday, and National Holidays, Year-end and New Year Holidays, Paid Holidays, Other Special Holidays
 
【待遇・福利厚生 | Services / Benefits】
各種社会保険完備(厚生年金保険、健康保険、労災保険、雇用保険)、 屋内原則禁煙(屋外に喫煙所あり)、 通勤交通費支給等
Social insurance, Transportation Fee, No smoking indoors allowed (Designated smoking area), etc.

 

 

Required Skills

- Expert SQL capability in querying Big Data/ large data sets (Teradata, Hadoop, etc.) to extract BI- insights.

- Programming languages such as Python/Scala/PLSQL/Java.

- Development and operation of data pipeline leveraging big data technologies such as Spark, Map Reduce, Hive, Kafka, Sqoop, NoSQL Databases as well as traditional DB and file based data integration solutions.

- Database development (eg TeraData, Oracle, MySQL, SQLServer, DB2.)

- Shell-scripting languages such as Bash.

- Formal analysis and documentation of BI solutions.

- Distributed version control system such as Git.

- Initiative and the ability to work independently and in a team. (The team uses Agile methodology.)

Preferred Skills

- Application development using workflow engines such as Airflow, Oozie, Rundeck.

- BI reporting tools, including administration, modeling, and report/dashboard development.

- BI Modelling of data marts using ER hybrid, Kimball, Data Vault methodologies.

- Experience in ELK Stack.

- Experience in AtScale and Presto.

- Operational experience in developing and supporting high availability applications / systems.

- Capability to self-manage and also manage small projects.