Collabera
Big data engineer
Job Location
Schiller Park, IL, United States
Job Description
Position Details:
Title: Big Data Engineer
Location: Chicago, IL- Hybrid - 3x/week onsite
Compensation: $65-$68.26/hour
Interview Process: One Inperson round
Why open: New project
Scope of role: Experience with Kafka, Pyspark and real-time data processing, Elastic Search, Kibana (nice to have)
Project: Dealing with streaming data
Expected to accomplish:
Processing Data in Real-time
Building system from scratch
Designing abstract layers
Must have:
Strong communication skills
Kafka
Pyspark
Real time data processing
SQL
Spark
Coding/ development in Python
Nice to have:
Elastic Search
Kibana
Designing system from scratch/ designing abstract layers
Required Skills:
Strong SQL Skills – one or more of MySQL, HIVE, Impala, SPARK SQL
Data ingestion experience from message queue, file share, REST API, relational database, etc. and experience with data formats like json, csv, xml
Experience working with SPARK Structured steaming
Experience working with Hadoop/Big Data and Distributed Systems
Working experience with Spark, Sqoop, Kafka, MapReduce, NoSQL Database like HBase, SOLR, CDP or HDP, Cloudera or Hortonworks, Elastic Search, Kibana, etc.
Hands on programming experience in at least one of Scala, Python, PHP, or Shell Scripting, to name a few
Performance tuning experience with spark /MapReduce or SQL jobs
Experience and proficiency with Linux operating system is a must
Experience in end-to-end design and build process of Near-Real Time and Batch Data Pipelines
Experience working in Agile development process and deep understanding of various phases of the Software Development Life Cycle
Experience using Source Code and Version Control systems like SVN, Git, Bit Bucket etc.
Experience working with Jenkins and Jar management
Self-starter who works with minimal supervision and the ability to work in a team of diverse skill sets
Ability to comprehend customer requests and provide the correct solution
Strong analytical mind to help take on complicated problems
Desire to resolve issues and dive into potential issues
Ability to adapt and continue to learn new technologies is important
Location: Schiller Park, IL, US
Posted Date: 9/19/2024
Title: Big Data Engineer
Location: Chicago, IL- Hybrid - 3x/week onsite
Compensation: $65-$68.26/hour
Interview Process: One Inperson round
Why open: New project
Scope of role: Experience with Kafka, Pyspark and real-time data processing, Elastic Search, Kibana (nice to have)
Project: Dealing with streaming data
Expected to accomplish:
Processing Data in Real-time
Building system from scratch
Designing abstract layers
Must have:
Strong communication skills
Kafka
Pyspark
Real time data processing
SQL
Spark
Coding/ development in Python
Nice to have:
Elastic Search
Kibana
Designing system from scratch/ designing abstract layers
Required Skills:
Strong SQL Skills – one or more of MySQL, HIVE, Impala, SPARK SQL
Data ingestion experience from message queue, file share, REST API, relational database, etc. and experience with data formats like json, csv, xml
Experience working with SPARK Structured steaming
Experience working with Hadoop/Big Data and Distributed Systems
Working experience with Spark, Sqoop, Kafka, MapReduce, NoSQL Database like HBase, SOLR, CDP or HDP, Cloudera or Hortonworks, Elastic Search, Kibana, etc.
Hands on programming experience in at least one of Scala, Python, PHP, or Shell Scripting, to name a few
Performance tuning experience with spark /MapReduce or SQL jobs
Experience and proficiency with Linux operating system is a must
Experience in end-to-end design and build process of Near-Real Time and Batch Data Pipelines
Experience working in Agile development process and deep understanding of various phases of the Software Development Life Cycle
Experience using Source Code and Version Control systems like SVN, Git, Bit Bucket etc.
Experience working with Jenkins and Jar management
Self-starter who works with minimal supervision and the ability to work in a team of diverse skill sets
Ability to comprehend customer requests and provide the correct solution
Strong analytical mind to help take on complicated problems
Desire to resolve issues and dive into potential issues
Ability to adapt and continue to learn new technologies is important
Location: Schiller Park, IL, US
Posted Date: 9/19/2024
Contact Information
| Contact | Human Resources Collabera |
|---|