Hadoop Jobs

Hiring: Lead Data Engineer (Data Platforms & Analytics Infrastructure) Job ID: GKS2005_1 --- Role: Lead Data Engineer Mode: Remote (Monthly visit to Coimbatore office preferred) Budget: ₹50,000 – ₹60,000/month Contract Duration: 8 Months Experience: 7 – 10 Years --- About the Role We are looking for a Lead Data Engineer to build and scale a robust data platform supporting analytics and AI/ML use cases. This role involves end-to-end ownership of the data lifecycle along with leading a team to deliver reliable and high-performance data solutions. --- Key Responsibilities Build scalable batch and real-time data pipelines Design and implement Lakehouse architecture (Delta Lake / Iceberg) Develop ETL/ELT workflows using Airflow, dbt, or Prefect Ensure data quality,...

Informatica Data Quality (IDQ) Specialist Availability- Immediate Mode - WFH Job Summary: Informatica Data Quality (IDQ) Specialist Allocation: Dedicated / Full-Time Objective: Own the end-to-end Data Quality management framework for the project, covering data profiling, rules definition, and remediation pipelines across the 8 in-scope data sources. Key Responsibilities & Scope of Work (SoW): Data Profiling & Analysis: Conduct detailed data profiling and analysis for 8 complex data sources (including systems like SAP, Moamalat, Bawaba, etc.) to identify data anomalies and gaps. Rules Catalog Development: Collaborate with business stakeholders to define, document, and build a reusable catalog of Data Quality (DQ) rules, thresholds, and issue classifications. Remediation & Cle...

We are currently looking for enterprise-grade software codebases and related SDLC/operational artifacts for engineering evaluation, AI benchmarking, architecture analysis, and technical due diligence projects. We are open to source code from GitHub, Bitbucket, GitLab, private repositories . Preferred requirements include: 50K+ LOC with production-like architecture Backend, frontend, database, and infrastructure components Enterprise workflows beyond basic CRUD Tech stacks like Java, .NET, Python, Node.js, React, Docker, Kubernetes, etc. SDLC artifacts such as Jira tickets, PRD/BRD, API docs, architecture diagrams, deployment guides, and test artifacts If your team has suitable repositories or archived enterprise projects, please share: Tech stack Approximate LOC/module count Available SD...

I'm seeking an expert to secure our Oracle Java environment and assist with an AWS migration project using AWS Corretto. Key Responsibilities: - Address security vulnerabilities in Oracle Java, specifically: - Data breaches - Injection attacks - Misconfigurations - Utilize AWS Corretto's monitoring tools and security features - Work with Open Java SDK versions 17 and 21 - Provide recommendations for transitioning from WebLogic 14.1.1 and Java 11 to a suitable non-Oracle Java platform. - Assist in preparing a proposal by the deadline of 22-May 2026, focusing on removing vulnerabilities and ensuring a secure migration strategy. Ideal Skills & Experience: - In-depth knowledge of Oracle Java security - Strong background in AWS, especially AWS Corretto - Experience with mo...

We want a experienced Candidate who has worked on the given jd Objective Lead the migration from Visual SourceSafe (VSS) to GitHub, build a Jenkins–Docker CI/CD pipeline from scratch, and integrate with Jira and Mattermost for automated workflows and communication. Job Description / Scope & Key Deliverables 1. Version Control Transformation Migrate repositories from VSS to GitHub with full history Define branching strategies (GitFlow/GitHub Flow) and access controls Implement security best practices (branch protection, SSH keys, secret scanning) 2. CI/CD Architecture (Jenkins & Docker) Set up and secure on-premise Jenkins Build reusable pipelines (Pipeline-as-Code using Groovy) Standardize Docker containerization and private registry Enable automated deployments via Docker...

We are currently looking for enterprise-grade software codebases and related SDLC/operational artifacts for engineering evaluation, AI benchmarking, architecture analysis, and technical due diligence projects. We are open to source code from GitHub, private repositories, internal servers, or ZIP archives. Preferred requirements include: 50K+ LOC with production-like architecture Backend, frontend, database, and infrastructure components Enterprise workflows beyond basic CRUD Tech stacks like Java, .NET, Python, Node.js, React, Docker, Kubernetes, etc. SDLC artifacts such as Jira tickets, PRD/BRD, API docs, architecture diagrams, deployment guides, and test artifacts If your team has suitable repositories or archived enterprise projects, please share: Tech stack Approximate LOC/module cou...

I need a cloud-native ETL pipeline built end-to-end on AWS, coded in PySpark and designed for production reliability. The pipeline will ingest data from three sources—databases, APIs, and file systems—then standardise and load it into an analytics-ready destination. Source files arrive in a mix of CSV, JSON, and Parquet, so the job must include automatic format detection, schema inference, and efficient column-wise writes. Beyond raw transformation, I want solid engineering practices: parameter-driven jobs, modular Spark code, unit tests, logging, alerting, and retry logic. Leveraging AWS native services such as Glue, EMR, Lambda, and S3 is expected, but I’m open to other AWS components if they shorten development time or lower cost. Candidates must have expertise in da...

I’m setting up an intensive, expert-level course for a team that already designs and operates complex AWS workloads. The objective is to deepen their Data Engineering skills on AWS, so we need a trainer who can move straight past fundamentals and into real-world architecture, scaling strategies, and cost-efficient design. Core focus: Data Engineering with AWS, specifically • Data Storage Solutions • Data Processing & ETL • Big Data Analytics Your role is to craft and deliver a hands-on program that blends concise theory, architectural white-boarding, and live labs. Glue, EMR, Redshift, Lake Formation, Kinesis, Lambda, Step Functions, and related services should feature prominently, but I’m open to your proven toolset and workflows. Deliverables 1...

Automated Terraform infrastructure deployments, achieving a 97% security score in CSPM, ensuring cloud infrastructure met enterprise-grade security and compliance standards Hardened cloud infrastructure to CIS Benchmark v3/v5 and PCI-DSS v4 standards, directly contributing to enterprise compliance certification and reducing security vulnerabilities across all environments. Built Ansible playbooks automating software installation and configuration across multiple servers simultaneously, cutting provisioning time by 20% and eliminating manual configuration errors. Architected and managed production AWS infrastructure across 15+ services, including EMR, WAF, GuardDuty, and Global Accelerator, supporting a highly available and secure multi-environment setup. Managed hybrid deployments spanning...

Hiring: GCP DevOps Engineer (2 Positions) Experience: 5+ Years Location: Remote Contract Duration: 3 Months Salary: ₹40,000 – ₹50,000/month --- Key Responsibilities: Infrastructure & Platform Engineering: - Design and manage cloud infrastructure (GCP preferred / AWS / Azure) using Terraform - Build and manage Kubernetes clusters (GKE/EKS/AKS) with multi-tenancy - Implement auto-scaling (HPA/VPA) and optimize resource usage - Ensure high availability, disaster recovery, and backup strategies CI/CD & Release Engineering: - Design and maintain CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins, ArgoCD) - Implement GitOps workflows for automated deployments - Enable zero-downtime deployments (blue-green, canary) - Automate database migrations, rollbacks, and environment s...

Hiring: ML Engineer – Test & Learn Platform Experience: 3+ Years Location: Remote (1–2 visits to Bangalore required) Salary: ₹40,000 – ₹50,000/month --- Role Overview: We are looking for an ML Engineer to build and scale experimentation and causal inference systems. You will work on statistical engines, APIs, and cloud-based pipelines to enable data-driven decision-making. --- Key Responsibilities: - Develop ML/statistical models (DID, Synthetic Control, A/B Testing) in Python - Build and integrate FastAPI-based services - Design large-scale data pipelines using PySpark, Delta Lake, and Azure Data Lake - Optimize Spark jobs (memory, partitioning, performance tuning) - Work with Databricks for job orchestration and data workflows - Containerize and deploy applica...

Cancer Detection Using Machine Learning ## Project Overview I have developed a Machine Learning based Lung Cancer Detection system that predicts the likelihood of lung cancer using patient medical parameters and data analysis techniques. This project is suitable for: * Final year engineering students * Academic submissions * Research work * Machine Learning learning purposes * Healthcare AI projects ## Features * Data preprocessing and cleaning * Exploratory Data Analysis (EDA) * Machine Learning model training and testing * Accuracy evaluation and prediction system * Visualization graphs and reports * Easy-to-understand code structure * User-friendly implementation ## Technologies Used * Python * Scikit-learn * Pandas * NumPy * Matplotlib * Jupyter Notebook ## Deliverables * Com...

Hope you are doing well. We are currently looking for enterprise-grade software codebases and related SDLC/operational artifacts for engineering evaluation, AI benchmarking, architecture analysis, and technical due diligence projects. We are open to source code from GitHub, private repositories, internal servers, or ZIP archives. Preferred requirements include: 50K+ LOC with production-like architecture Backend, frontend, database, and infrastructure components Enterprise workflows beyond basic CRUD Tech stacks like Java, .NET, Python, Node.js, React, Docker, Kubernetes, etc. SDLC artifacts such as Jira tickets, PRD/BRD, API docs, architecture diagrams, deployment guides, and test artifacts If your team has suitable repositories or archived enterprise projects, please share: Tech stack A...

# Lung Cancer Detection Using Machine Learning ## Project Overview I have developed a Machine Learning based Lung Cancer Detection system that predicts the likelihood of lung cancer using patient medical parameters and data analysis techniques. This project is suitable for: * Final year engineering students * Academic submissions * Research work * Machine Learning learning purposes * Healthcare AI projects ## Features * Data preprocessing and cleaning * Exploratory Data Analysis (EDA) * Machine Learning model training and testing * Accuracy evaluation and prediction system * Visualization graphs and reports * Easy-to-understand code structure * User-friendly implementation ## Technologies Used * Python * Scikit-learn * Pandas * NumPy * Matplotlib * Jupyter Notebook ## Deliverables...

I’m looking for an experienced consultant who can take the lead in architecting and standing up a full-scale data lakehouse, with the immediate goal of seamless data integration across my current and future sources. What I need from you: • A clear, vendor-agnostic architecture blueprint that balances performance, cost, and governance. • Hands-on implementation of the core storage layer (Delta/Parquet or comparable), compute engine (Spark or equivalent), and ingestion pipelines. • Robust security and data-quality controls baked in from day one, including role-based access, lineage, and monitoring. • Documentation and a concise knowledge-transfer session so my internal team can extend and maintain the platform confidently. Acceptance criteria: 1. A reproducible i...

# Lung Cancer Detection Using Machine Learning ## Project Overview I have developed a Machine Learning based Lung Cancer Detection system that predicts the likelihood of lung cancer using patient medical parameters and data analysis techniques. This project is suitable for: * Final year engineering students * Academic submissions * Research work * Machine Learning learning purposes * Healthcare AI projects ## Features * Data preprocessing and cleaning * Exploratory Data Analysis (EDA) * Machine Learning model training and testing * Accuracy evaluation and prediction system * Visualization graphs and reports * Easy-to-understand code structure * User-friendly implementation ## Technologies Used * Python * Scikit-learn * Pandas * NumPy * Matplotlib * Jupyter Notebook ## Deliverables...