Software Developer/ Big Data

Location: – 18850 N 56th St, Phoenix, Arizona 85050

Job Description: –

  • Architect, Design, Develop, Test, Deploy and Support Big Data Applications on Hadoop Cluster with Map Reduce (Java), Sqoop, Hbase, Hive, OOZIE.
  • Work on large-scale Hadoop YARN cluster (600+ nodes) for distributed data Storage, processing and analysis.
  • Gather all required information and requirements for the project.
  • Import of data from various data sources, performed transformations using Hive, Map Reduce and loaded data into HDFS.
  • Involve in encrypting the sensitive data like card number, acct number etc. on more than 100 million records.
  • Schedule Oozie workflow and Spring Batch for configuring the workflow for different jobs like Hive, MapReduce, Shell.
  • Develop and document design/implementation of requirements based on business needs. 
  • Automat all the jobs for pulling data from FTP server to load data into Hive tables using Oozie workflows.
  • Store, Access and Process data from different file formats i.e. Text, ORC and Parquet.
  • Involve in creating Hive Tables, loading with data and writing Hive queries. 
  • Write Shell scripts to start the use case and for pre validations.
  • Involve in code review and bug fixing for improving the performance
  • Develop and optimize Hive UDFs (User-Defined Functions) to implement the functionality of external languages as and when required.
  • Manage and review Hadoop log files to identify issues when job fails and finding out the root cause.
  • Co-ordinate with Administrator team to analyze Map Reduce Jobs performance for resolving any cluster related issues.
  • Manage and schedule Jobs on a Hadoop cluster using Cron Tab, Oozie Scheduler, Event Engine (Inhouse Tool).
  • Reduce number of open tickets for couple of use-cases by analyzing, categorizing & prioritizing all open issues recorded. This required motivation of the offshore team & thoughtful delegation of workable tickets to the offshore resources to maximize on efficiency.
  • Use the Service Now ITSM tool for creating Incidents, Problems, Knowledge Articles, Change Requests.
  • Guide offshore programmers assigned to the Production Support group.
  • Work on Spark with Python to migrate from Hive to Spark for faster data transfer

Minimum Education Required: – Bachelor’s degree in computer science, computer information systems, information technology, or a combination of education and experience equating to the U.S. equivalent of a Bachelor’s degree in one of the aforementioned subjects.