Senior DevOps Engineer (AI/ML & On-Premise)

AI & ML & On-Premise

Senior DevOps Engineer

Why Polygraf AI?

At Polygraf, we envision a future where AI augments human capabilities without compromising safety, privacy, or ethical standards. Trust in our commitment to building this future with you.

About the Role

We are looking for a Senior DevOps Engineer to lead the deployment of our high-throughput, KubeRay-based AI architecture into customer environments. Unlike standard cloud-native SaaS roles, this position focuses on on-premise, edge, and air-gapped deployments.

You will bridge the gap between our internal distributed AI systems and the varied, often restrictive infrastructure of our enterprise clients. A critical component of this role is ensuring that our proprietary software and AI models remain secure, encrypted, and tamper-proof when running on hardware we do not control.

Required Skills (The Core)

Kubernetes Internals: Deep understanding of Kubernetes architecture beyond managed cloud services (EKS/GKE). Proven experience administering local/on-prem distributions (e.g., K3s, Rancher RKE2, MicroK8s, or similar).
Python Ecosystem: Knowledge of Python application packaging and dependency management (UV, Poetry, pip, wheels, sdist) specifically for complex ML
Ray & KubeRay: Hands-on experience deploying and tuning Ray clusters or similar distributed compute frameworks (Dask, PySpark or similar) on Kubernetes.
Infrastructure as Code: Experience with Helm (creating charts from scratch) and Terraform or its alternative.
Linux Networking: Strong grasp of low-level Linux networking, container networking (CNI), and debugging connectivity issues in restricted corporate networks.
Containerization: Experience creating minimized, secure container images (distroless, nvidia-cuda or similar docker images) optimized for large ML dependencies (PyTorch/TensorFlow).

Key Responsibilities

Architect Local Clusters: Design and maintain Kubernetes configurations for local and bare-metal deployments (using distributions like K3s, RKE2, or MicroK8s) to support KubeRay clusters.
KubeRay Management: Orchestrate the lifecycle of Ray clusters on Kubernetes, ensuring high availability and fault tolerance for distributed Python applications.
Secure Packaging: Develop robust strategies for packaging Python applications and AI models (Docker, Helm, Terraform/Pulumi or similar) that function in offline/air-gapped environments without external repository access.
IP Protection Implementation: Implement encryption at rest and in transit for proprietary model weights and Integrate obfuscation and licensing enforcement mechanisms into the deployment pipeline.
Client Support: Serve as the technical escalation point for installation issues within complex customer network topologies (firewalls, proxies, custom PKI).

Preferred Skills (The "IP Protection" & Niche)

Code & Model Security: Encrypting model artifacts, Rust programming language, and Experience with Python code obfuscation tools (e.g., PyArmor).
Confidential Computing: Familiarity with Trusted Execution Environments (TEE) Trusted Platform Modules (TPM) or deploying to secure enclaves (Intel SGX, AMD SEV, Nvidia) is a massive plus.
Licensing Automation: Experience integrating on-premise license servers or “phone-home” validation mechanisms.
GPU Passthrough: Experience configuring NVIDIA Container Toolkit and Multi-Instance GPU (MIG) on bare metal.
Security Compliance: Knowledge of FIPS compliance or hardening Kubernetes specifically for highly regulated industries (Finance, Healthcare).

As an equal opportunity employer, we highly value diversity and inclusion. We recognize that a diverse team brings unique perspectives, fosters ongoing innovation, and deepens our connection to the global community we serve. If you’re enthusiastic about spearheading the next era of generative AI and leaving a profound impact on how individuals and brands create, we’re eager to have you join our team.

AI/ML & On-Premise