When asking, "How is this nightly batch job run?", hearing, "Tanaka logs into the production server via SSH and manually executes the script," remains a surprisingly common response when taking over system operations in custom development. It works for now. However, only individuals with keys can run it, no records exist of when, who, or what was executed, and if the connection drops, jobs terminate halfway through. Security audits will almost certainly flag this, and every team departure or person-dependent dependency sparks anxiety. A permanently open SSH gateway to production is substantial debt, even if everyone has grown accustomed to it.
A notable case study solving this exact challenge at scale is Slack's publication, From SSH to REST: Modernizing Slack's EMR Data Pipelines (Slack Engineering) (with an accessible breakdown on InfoQ). Slack migrated over 700 jobs that Airflow initiated via direct SSH into EMR master nodes to Quarry, a REST-based job execution platform, completely eliminating direct SSH into production clusters. Although the scale differs, the transition from person-dependent SSH to an auditable execution platform applies directly to business systems in small and mid-sized enterprises.
Why manual SSH execution is technical debt
Logging into production via SSH to execute jobs is convenient at first. The issue is that as operations mature, three major problems emerge.
First, an expansive attack surface. Permanent pathways and keys granting direct access to production exist continuously, turning key distribution, rotation, and auditing into heavy operational burdens. As more people hold keys, tracking exactly who can access production becomes impossible.
Second, the lack of auditability. Manual script execution leaves no structured record of when, by whom, or with which parameters a job was run. Being unable to trace what happened after an incident is fatal for reliable operations.
Third, poor reliability. If an SSH session drops, jobs can terminate prematurely or continue running invisibly in the background, risking duplicate runs. Slack similarly cited jobs failing silently due to broken connections.
| Dimension | Manual SSH execution | REST-based job platform |
|---|---|---|
| Access | Requires persistent SSH access and keys to production | No direct access to production required |
| Execution logs | Manual process; no structured records retained | Full record of who executed what via API |
| Connection drops | Jobs fail halfway or run duplicate executions | Server tracks and continues jobs reliably |
| Owner | Only key holders can fix issues (person-dependent) | Anyone with proper permissions can execute safely |
Moving away from SSH means unifying entry points and recording logs
The essence of moving away from SSH is closing convenient shortcuts (direct production logins) and routing all execution through a single authenticated, logged entry point. With Slack's Quarry, Airflow simply sends HTTP requests instead of opening SSH connections, leaving job dispatching and status monitoring to the server. Even if the Airflow process restarts, jobs continue running on the platform. The core is to insert a layer handling authentication and logging between the operator and the production environment.
Bringing this concept to SME business systems does not require infrastructure as massive as Slack's. The key is stopping direct manual logins to production and structuring jobs so they are triggered from permissioned APIs or admin consoles with execution logs retained. Routing execution through CI/CD pipelines is also effective; much like shortening deployment lead times, this transforms operations from manual human procedures into repeatable mechanisms (see CI/CD Improvements to Speed Up Deployment by Roughly 50% (GH Media)).
A realistic sequence for moving away from SSH in custom development
Attempting to migrate everything to a unified platform all at once paralyzes field operations and causes projects to stall. In our custom development, we adopt an approach that shuts down SSH access starting with high-risk pathways, without interrupting ongoing business operations.
First, take inventory of all production access paths
The first step is not building platforms, but understanding the current reality: who can SSH into production, which keys are distributed where, and what jobs are being manually triggered. In one handover project for an enterprise system, our inventory uncovered active keys belonging to former employees and scripts being edited directly on production servers. Before closing pathways, you must map them out. For untangling situations where no one possesses a complete picture, Inventory Design for Visualizing Dependencies (GH Media) serves as a helpful reference.
Migrate high-risk jobs to logged execution first
Among the jobs identified in the inventory, start by replacing manual SSH execution with permissioned, logged execution for frequently run tasks and those where failure carries high business impact. Rather than migrating all jobs at once, running systems in parallel and verifying them one by one eases anxiety for operational teams. Once jobs run via REST endpoints or admin panels, execution records accumulate naturally, making it possible to trace who ran what and when.
Eliminate standing SSH access and use time-limited access only for emergencies
Once primary jobs are migrated to the platform, close permanently open SSH entry points. For emergencies where access cannot be eliminated entirely, switch to time-limited access that is requested, opened temporarily, and closed automatically after use. This eliminates permanent keys to production and ensures every entry is logged. For embedding secrets management and authentication into development workflows, please also read Secrets Management and DevSecOps Automation in GitLab 19 (GH Media).
Common pitfalls to avoid
First, revoking keys without verifying operational procedures with teams on the ground. Blocking SSH without understanding the execution pathways of running nightly batch jobs will cause processes to break that very night. Always follow the sequence of inventory → parallel operation → cutover.
Second, undermining security by leaving too much emergency access in place. Leaving permanent SSH access "just in case" inevitably leads back to old habits. Emergency access must be strictly time-limited and logged, with exceptions formalized as operational rules.
Summary — graduating from "only key holders can fix it"
Relying on manual SSH logins to run jobs on production servers is a habit that carries technical debt across attack surface, auditability, and reliability. Slack migrated over 700 jobs to a REST-based platform and eliminated direct production access. The same philosophy applies to business systems in small and mid-sized enterprises. When tackling this through custom development, auditing production access paths, migrating high-risk jobs to logged execution first, and replacing standing SSH with time-limited emergency access—following this sequence is the pragmatic way to move past person-dependent SSH without disrupting operations.
If you are concerned about nightly batch jobs relying on individual engineers' SSH access, or if direct production access was flagged during a security audit, please reach out via our contact form. We can start with an audit of your production access pathways.
Sources
- From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines（Slack Engineering）
- Slack Eliminates SSH in EMR Pipelines, Migrates 700+ Jobs to REST-Based Architecture（InfoQ）
- CI/CD Improvements to Speed Up Deployment by Roughly 50% (GH Media)
- Inventory Design for Visualizing Dependencies (GH Media)
- Secrets Management and DevSecOps Automation in GitLab 19 (GH Media)
