Azure Data Engineer (Databricks Expert) Job at Eviden, Canada

WmhPNW12N2ZpSHV4VlZ5NG1WcFUvWVI1VWc9PQ==
  • Eviden
  • Canada

Job Description

Job Description

Role : Azure Data Engineer (Databricks Expert)

Location : Toronto, ON

Hire Type : Full-Time with Eviden

Job Description:

About the Role:

We are seeking a highly skilled and experienced Senior Azure Data Engineer to join team and tackle critical performance challenges within Azure data platform. You will play a pivotal role in optimizing and re-architecting key components of data infrastructure, directly impacting the performance and scalability of core data marts. This is not a maintenance role; this is a chance to make a significant impact by redesigning and implementing solutions for a high-visibility project.

Responsibilities:

  • Performance Optimization: Lead the performance tuning and optimization of Azure-based data pipelines, specifically addressing the current bottleneck where processing 1000 records takes 40 minutes. This requires a deep understanding of Azure Data Factory (ADF), Databricks (Spark), and data processing best practices.
  • Architecture Review and Redesign: Evaluate the existing Azure data platform architecture, identify bottlenecks and design flaws, and propose and implement solutions for a more efficient and scalable system. This will involve working with Delta tables and optimizing data storage and retrieval.
  • Databricks Expertise: Utilize your advanced PySpark skills, particularly with structured streaming, to re-engineer data transformation and processing logic within Databricks. The ideal candidate will have a proven track record of optimizing Spark jobs for performance.
  • Data Integration: Work with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Database (DB), and file exchange processes (batch mode) to ensure seamless data flow and integration.
  • MDM Integration (Desirable): Contribute to the ongoing development and optimization of Master Data Management (MDM) system, which follows a hub-and-spoke architecture. Experience with large-scale MDM implementations is highly desirable.
  • Collaboration: Work closely with other engineers, business stakeholders, and vendors to understand requirements, communicate progress, and ensure successful project delivery. You will be expected to mentor and guide other team members.

Qualifications:

  • Minimum experience of implementing 2 projects on Azure data platform
  • Minimum experience of around 8-12 years
  • "Ninja" Level Expertise: A proven track record of success in optimizing and scaling Azure data platforms, particularly with Databricks and Spark. We are looking for an expert who can quickly diagnose and resolve complex performance issues.
  • Deep Azure Knowledge: Extensive experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Synapse, Event hub, and other relevant Azure services. Experience with Azure SaaS offerings is a plus.
  • Databricks Mastery: Advanced proficiency in PySpark and structured streaming. Demonstrated ability to write and optimize complex Spark queries for performance.
  • Data Warehousing and MDM: Solid understanding of data warehousing principles and experience with Master Data Management (MDM) implementations, preferably with a hub-and-spoke architecture.
  • Problem-Solving Skills: Exceptional analytical and problem-solving skills, with the ability to identify root causes of performance bottlenecks and design effective solutions.
  • Communication Skills: Excellent communication and collaboration skills, with the ability to work effectively 1 with both technical and non-technical 2 stakeholders.

Desirable:

  • Experience with AI Search and Cosmos DB.
  • Experience with Informatica MDM in an Azure environment.

Job Tags

Full time,

Similar Jobs

Johnson & Johnson

Clinical Data Engineer Summer Intern Job at Johnson & Johnson

 ...Job Sub Function: Non-LDP Intern/Co-Op Job Category: Career...  ...for Clinical Data Engineer Summer Intern to be in Irvine, CA....  ...about statistical and data science applications specific to clinical...  ...writing, and writing of computer programs or software applications... 

Adecco

Forklift Operator - Cherry Picker Job at Adecco

 ...Job Description Job Description Hiring immediately for Forklift Operator Cherry Picker jobs at Elogistics. Pay starting at $19.00 per hour, openings on all shifts. High School Diploma or GED 1+ year related forklift, cherry picker experience Valid driver... 

Taco Bell

Restaurant General Manager Job at Taco Bell

Up to $70,000.00 per year for a proven QSR General Manager!!!!!!!!Apply today, Interview today,top Pay...  ...schedules and great work environment!Restaurant General ManagerFREE FOOD! GREAT PAY! DISCOUNTS!Thinking the fast food industry isn't for you....we may surprise... 

Malinis Property Services

Residential Cleaner Job at Malinis Property Services

 ...Job Description Job Description NOW HIRING: RESIDENTIAL CLEANERS Serving: Milton, Oakville, Burlington & West Mississauga Start Date: Immediately Hiring Job Type: Part-Time (with opportunity for Full-Time) Pay: $18$23/hr (based on experience & compensation... 

Pediatrix Medical Group

Intensivist (Part-time) Job at Pediatrix Medical Group

 ...Requisition ID: 2025-49453 Location: US-UT-Orem Specialty: Physician - Pediatric Intensivist Position Type: Full Time HR Rep / Recruiter: Jillian Larkin Contact: ****@*****.*** Overview Pediatric Intensivist Opportunity with Outdoor...