Req ID: 335675
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.
We are currently seeking a Databricks Engineers to join our team in Bangalore, Karnātaka (IN-KA), India (IN).
\"Job Duties: 1. Data Extraction & Source Analysis
Extract and analyze data from SQL Server and Teradata sources.
Review and interpret existing ETL logic from DataStage, BTEQ scripts, and custom SQL.
Identify filtering conditions, incremental load logic, and audit/control columns.
Work with SMEs to understand data models, including workflow, reference, and source tables.
2. ETL Pipeline Modernization & Transformation
Translate legacy SQL/DataStage transformations into Databricks-native code (e.g., PySpark/SQL).
Re-engineer pipelines with consideration for:
Surrogate key generation
Date/time formatting
Audit trail logging (create/update date, process ID)
Ensure incremental load logic is retained in the new platform.
3. Job Design, Development & Automation
Build and orchestrate jobs within Databricks using tools like:
Databricks Workflows
Delta Lake for data consistency
Auto Loader or Query Federation for real-time extracts
Incorporate AI/LLM-based tools for code generation where appropriate.
Create reusable components for audit, rollback, and logging.
4. Metadata, Documentation & Lineage
Generate and maintain:
Data flow diagrams
Job documentation
Table definitions and transformation logic
Work with AI-based tools to auto-generate data lineage and mapping sheets.
Ensure documentation supports compliance and audit-readiness.
5. Testing & Validation
Design and execute unit tests and integration tests for data pipelines.
Validate:
Row counts between source and target
Accuracy of transformation logic
Integrity of audit and surrogate columns
Collaborate with QA and SMEs to verify business rules.
6. Collaboration & Communication
Work closely with:
Business SMEs to clarify logic and validate results
Data architects on schema, data model, and standards
DevOps/infra teams to provision environments and access
Participate in agile ceremonies (stand-ups, retrospectives, planning)
7. Performance Tuning & Optimization
Optimize data ingestion and transformation for performance and cost efficiency.
Implement partitioning, caching, or Z-ordering as needed in Databricks.
Classify and handle cold/warm/hot data appropriately.
8. Security, Governance, and Compliance
Ensure compliance with data governance policies (e.g., masking PHI/SPI if necessary).
Implement access control via Unity Catalog or equivalent.
Work within approval processes for metadata extraction and AI tool usage.
Minimum Skills Required: 1. Data Engineering & ETL Fundamentals
Strong understanding of ETL/ELT principles and data pipelines
Experience with data ingestion, data transformation, and data validation
2. Technical Skills
a. Databricks & Spark
Proficiency with Databricks platform (notebooks, workflows)
Strong hands-on with PySpark or Spark SQL
Familiarity with Delta Lake, data versioning, partitioning
b. SQL
Advanced SQL skills for complex joins, window functions, filtering logic
Ability to translate legacy SQL logic into Databricks SQL/PySpark
c. Teradata & SQL Server
Familiarity with Teradata and SQL Server environments
Experience reading and interpreting BTEQ scripts and DataStage jobs
3. Data Modeling & Metadata Understanding
Ability to read and understand data models, schemas, and ERDs
Understanding of surrogate keys, audit fields, and incremental load strategies
4. Version Control & DevOps
Basic proficiency with Git for code versioning
Familiarity with CI/CD pipelines for data projects (preferred)
5. Testing & Debugging
Ability to write and validate unit/integration tests
Debugging data discrepancies between source and target systems
6. Documentation & Lineage
Document data flows, job dependencies, and transformation logic
Use of lineage tools or familiarity with AI-generated documentation is a plus
7. Soft Skills
Strong communication skills to collaborate with SMEs, architects, QA
Ability to T-shirt size job complexity (easy/medium/complex)
Comfortable working in agile/scrum teams
8. Security & Governance Awarenes\"
About NTT DATA
NTT DATA is a $30 billion trusted global innovator of business and technology services. We serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize and transform for long term success. As a Global Top Employer, we have diverse experts in more than 50 countries and a robust partner ecosystem of established and start-up companies. Our services include business and technology consulting, data and artificial intelligence, industry solutions, as well as the development, implementation and management of applications, infrastructure and connectivity. We are one of the leading providers of digital and AI infrastructure in the world. NTT DATA is a part of NTT Group, which invests over $3.6 billion each year in R&D to help organizations and society move confidently and sustainably into the digital future. Visit us at us.nttdata.com
Whenever possible, we hire locally to NTT DATA offices or client sites. This ensures we can provide timely and effective support tailored to each client’s needs. While many positions offer remote or hybrid work options, these arrangements are subject to change based on client requirements. For employees near an NTT DATA office or client site, in-office attendance may be required for meetings or events, depending on business needs. At NTT DATA, we are committed to staying flexible and meeting the evolving needs of both our clients and employees. NTT DATA recruiters will never ask for payment or banking information and will only use @nttdata.com and @talent.nttdataservices.com email addresses. If you are requested to provide payment or disclose banking information, please submit a contact us form, https://us.nttdata.com/en/contact-us.
NTT DATA endeavors to make https://us.nttdata.com accessible to any and all users. If you would like to contact us regarding the accessibility of our website or need assistance completing the application process, please contact us at https://us.nttdata.com/en/contact-us. This contact information is for accommodation requests only and cannot be used to inquire about the status of applications. NTT DATA is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status. For our EEO Policy Statement, please click here. If you'd like more information on your EEO rights under the law, please click here. For Pay Transparency information, please click here.