Location: – 18850 N 56th St, Phoenix, Arizona 85050
Job Description: –
- Architect, Design, Develop, Test, Deploy and Support Big Data Applications on Hadoop Cluster with Map Reduce (Java), Sqoop, Hbase, Hive, OOZIE.
- Work on large-scale Hadoop YARN cluster (600+ nodes) for distributed data Storage, processing and analysis.
- Gather all required information and requirements for the project.
- Import of data from various data sources, performed transformations using Hive, Map Reduce and loaded data into HDFS.
- Involve in encrypting the sensitive data like card number, acct number etc. on more than 100 million records.
- Schedule Oozie workflow and Spring Batch for configuring the workflow for different jobs like Hive, MapReduce, Shell.
- Develop and document design/implementation of requirements based on business needs.
- Automat all the jobs for pulling data from FTP server to load data into Hive tables using Oozie workflows.
- Store, Access and Process data from different file formats i.e. Text, ORC and Parquet.
- Involve in creating Hive Tables, loading with data and writing Hive queries.
- Write Shell scripts to start the use case and for pre validations.
- Involve in code review and bug fixing for improving the performance
- Develop and optimize Hive UDFs (User-Defined Functions) to implement the functionality of external languages as and when required.
- Manage and review Hadoop log files to identify issues when job fails and finding out the root cause.
- Co-ordinate with Administrator team to analyze Map Reduce Jobs performance for resolving any cluster related issues.
- Manage and schedule Jobs on a Hadoop cluster using Cron Tab, Oozie Scheduler, Event Engine (Inhouse Tool).
- Reduce number of open tickets for couple of use-cases by analyzing, categorizing & prioritizing all open issues recorded. This required motivation of the offshore team & thoughtful delegation of workable tickets to the offshore resources to maximize on efficiency.
- Use the Service Now ITSM tool for creating Incidents, Problems, Knowledge Articles, Change Requests.
- Guide offshore programmers assigned to the Production Support group.
- Work on Spark with Python to migrate from Hive to Spark for faster data transfer
Minimum Education Required: – Bachelor’s degree in computer science, computer information systems, information technology, or a combination of education and experience equating to the U.S. equivalent of a Bachelor’s degree in one of the aforementioned subjects.