Senior Data Engineer

If you are looking for the opportunity to build highly scalable services and rich data insights on Azure’s largest distributed platforms, then the Security Platform team is the right place for you to come and take on your next challenge.

Our team owns multiple data pipelines that process tens of terabytes of data per day and stores multiple petabytes of data. Our challenge is to do this efficiently, reliably, and with low latency. The Security Platform LT team need to understand how the Security platform is used, to make data driven decision and build product features based on customer usage and feedback. Also to ensure the data is compliant with data privacy regulations and other privacy regulations. Our job is to create data pipelines to produce meaningful insights about their level of compliance, and to create understandable actions that they can take to improve.

We are looking for engineers who have a track record of success, who deliver results, and who (preferably) have a background in large-scale data processing systems in a distributed service environment. Everyone on the team owns significant areas of the system, directly contributes to the vision, direction, and implementation of new features, and are key to the success of our products. You will collaborate on your designs across our partner engineering teams and work closely with our PMs to design, build, and maintain a growing set of features that will serve the needs of our customers.

Basic Qualifications:

• Overall 9+ Years of experience in Software development
• Bachelor’s / Master’s degree in Computer Science or Engineering or Mathematics or IT technical discipline
• 4+ years of experience in Designing highly scalable data solutions for Analytics / Business Intelligence, Data science and Data Engineering
• Guiding the Team members on resolving the technical challenges and following best practices
• Data scientist with right aptitude & passion towards telemetry and product/customer intelligence.  
• Data Engineering background and familiarity with big data analytics tools (Spark/PySpark, Synapse, ADF, SQL, Kusto, Power BI, etc).
• Proficiency in programming using Python, .NET/Java needed.
• Experience in Datawarehouse concepts
• Experience in building reports / dashboards using Power BI
• Experience in writing complex SQLs.
• Experience in working with Data Lakes
• Experience in cloud eco system like Azure / AWS

 Preferred Qualifications: 

• Experience building and shipping production grade software or services.
• Experience creating and shipping V1 products using modern development practices. 
• Experience using agile methodologies or test-driven development (TDD).
• Great curiosity and willingness to question.
• High enthusiasm, integrity, ingenuity, results-orientation, self-motivation, and resourcefulness in a fast-paced competitive environment. 
• Love the next problem, the next experiment, the next partner.
• Have a deep desire to work collaboratively, solve problems with groups, find win/win solutions and celebrate successes.
• Get excited by the challenge of hard technical problems.
• Solve problems by always leading with deep passion and empathy for customers.
• Experience with Open-Source Big Data Stack such as Hadoop, HIVE, SPARK, HBASE etc is added advantage.
• Understanding of data structures, algorithms, and distributed systems

#ENGGJOBS #MSEC #MSFTSECURITY #SPDGP

Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable laws, regulations and ordinances. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request via the Accommodation request form.

Benefits/perks listed below may vary depending on the nature of your employment with Microsoft and the country where you work.

We are looking for engineers who have a track record of success, who deliver results, and who (preferably) have a background in large scale data processing systems. Everyone on the team owns significant areas of the system, directly contributes to the vision, direction, and implementation of new features, and are key to the success of our products. You will collaborate on your designs across our partner engineering teams and work closely with our PMs to design, build, and maintain a growing set of features that will serve the needs of our customers.

• Requirements and Modelling: Collaborate with stakeholders, document data requirements, evaluate feasibility, negotiate data agreements, and proposes data metrics for project-relevant features. 
• Data Model Design and Specification: Contribute to the data model design, draft design specification documents, improve data schemas, and balance analytical requirements and compute/storage consumption for data. 
• Data Extraction and Validation: With guidance, implement code to extract raw data from upstream sources, perform code reviews, implement orchestration techniques, validate data quality and reliability, and record data source control and versioning. 
• Data Transformation and Merging: Apply standard operations to transform raw data into compatible forms, use software and tools to wrangle and analyze data, and merge data into distributed systems or products. 
• Data Tool Design and Maintenance: Design and maintain data tools/platforms that are used to transform, manage, and access data, write efficient code to test and validate data platforms, and implement sustainable design patterns. 
• Data Monitoring and Troubleshooting: Follow performance monitoring protocols, build visualizations and aggregations to monitor data quality and pipeline/report health, troubleshoot and fix problems or anomalies, and document and share insights. 
• Data Compliance and Governance: Follow data modeling and handling procedures to comply with laws and policies, tag and classify data, govern data accessibility and traceability, and contribute to the data glossary.