Kỹ năng
Mô tả công việc
Build software according to the requirements of the Company and the Customer.
Research and bring research results as required
Perform other duties as assigned by the head of department
Research and bring research results as required
Perform other duties as assigned by the head of department
Yêu cầu công việc
Apache Airflow Skills:
Understanding of Airflow's core concepts like DAGS, Operators, Hooks, and Executors. Ability to write and maintain DAGS.
Experience with the Airflow UI to monitor and troubleshoot workflows.
Knowledge of Airflow plugins and extensions.
Familiarity with deployment and scaling of Airflow in production environments. AWS Glue Skills:
Proficiency with Glue ETL jobs, using both the visual interface and scripting.
Knowledge of Glue Data Catalog as a centralized metadata repository.
Familiarity with Glue Crawlers and how to automate data discovery. Ability to optimize Glue job performance.
Experience with Glue Studio for ETL job design.
Programming and Scripting:
Python is crucial, as it's the primary language for Airflow DAGS and can also be used in Glue scripts.
Familiarity with PySpark, since AWS Glue uses Apache Spark for processing. Cloud & Infrastructure:
Deep understanding of AWS services, especially those related to data storage (e.g., S3) and computation (e.g., EC2, Lambda).
Ability to set up and manage AWS IAM roles and permissions for Glue and related services.
Knowledge of cloud infrastructure best practices for security, cost optimization, and performance.
Databases & Data Warehouses:
Experience with source and target databases that may be integrated with Glue and Airflow. This includes relational databases like RDS, Aurora, and data warehouses like Amazon Redshift.
Data Modeling & ETL Best Practices:
Knowledge of data normalization, denormalization, star schema, and snowflake schema designs.
Best practices for ETL (Extract, Transform, Load) processes to ensure data integrity and performance.
Performance Tuning:
Ability to optimize Airflow and Glue for better performance, including partitioning strategies, parallel processing, and efficient ETL patterns.
Monitoring & Logging:
Familiarity with tools and services for monitoring the health of Airflow and Glue jobs, such as CloudWatch for AWS Glue.
Ability to debug and troubleshoot failed runs or performance bottlenecks.
Version Control & CI/CD:
Experience with version control systems, primarily Git.
Knowledge of CI/CD best practices, and tools like Jenkins or AWS CodePipeline, to automate the deployment of Airflow DAGS and Glue
Understanding of Airflow's core concepts like DAGS, Operators, Hooks, and Executors. Ability to write and maintain DAGS.
Experience with the Airflow UI to monitor and troubleshoot workflows.
Knowledge of Airflow plugins and extensions.
Familiarity with deployment and scaling of Airflow in production environments. AWS Glue Skills:
Proficiency with Glue ETL jobs, using both the visual interface and scripting.
Knowledge of Glue Data Catalog as a centralized metadata repository.
Familiarity with Glue Crawlers and how to automate data discovery. Ability to optimize Glue job performance.
Experience with Glue Studio for ETL job design.
Programming and Scripting:
Python is crucial, as it's the primary language for Airflow DAGS and can also be used in Glue scripts.
Familiarity with PySpark, since AWS Glue uses Apache Spark for processing. Cloud & Infrastructure:
Deep understanding of AWS services, especially those related to data storage (e.g., S3) and computation (e.g., EC2, Lambda).
Ability to set up and manage AWS IAM roles and permissions for Glue and related services.
Knowledge of cloud infrastructure best practices for security, cost optimization, and performance.
Databases & Data Warehouses:
Experience with source and target databases that may be integrated with Glue and Airflow. This includes relational databases like RDS, Aurora, and data warehouses like Amazon Redshift.
Data Modeling & ETL Best Practices:
Knowledge of data normalization, denormalization, star schema, and snowflake schema designs.
Best practices for ETL (Extract, Transform, Load) processes to ensure data integrity and performance.
Performance Tuning:
Ability to optimize Airflow and Glue for better performance, including partitioning strategies, parallel processing, and efficient ETL patterns.
Monitoring & Logging:
Familiarity with tools and services for monitoring the health of Airflow and Glue jobs, such as CloudWatch for AWS Glue.
Ability to debug and troubleshoot failed runs or performance bottlenecks.
Version Control & CI/CD:
Experience with version control systems, primarily Git.
Knowledge of CI/CD best practices, and tools like Jenkins or AWS CodePipeline, to automate the deployment of Airflow DAGS and Glue
Thời gian làm việc
Trong tuần: Từ thứ 2 - thứ 6
Trong ngày: Từ 08:30 giờ - 18:00 giờ
Quyền lợi ứng viên
- No need to try a job, get an official job with 100% salary
- Opportunities onsite and working with big customers, advanced technology, self-development...
- The opportunity to work in groups with many leading experts in the field of domestic and international IT.
- Opportunity to implement ambitious projects in many countries, exposure to the latest technologies and learn from good colleagues.
- Working in a young, vibrant, modern and multicultural environment; Communication activities and events on holidays take place regularly.
- Opportunity for advancement based on ability with corresponding increase in rank and salary increase.
- Have the right to participate in soft skills training courses (logical thinking, creative thinking, communication skills, project management skills, negotiation skills…) and Japanese language classes.
- And many other attractive benefits...
- Opportunities onsite and working with big customers, advanced technology, self-development...
- The opportunity to work in groups with many leading experts in the field of domestic and international IT.
- Opportunity to implement ambitious projects in many countries, exposure to the latest technologies and learn from good colleagues.
- Working in a young, vibrant, modern and multicultural environment; Communication activities and events on holidays take place regularly.
- Opportunity for advancement based on ability with corresponding increase in rank and salary increase.
- Have the right to participate in soft skills training courses (logical thinking, creative thinking, communication skills, project management skills, negotiation skills…) and Japanese language classes.
- And many other attractive benefits...
Địa chỉ làm việc
Remote