Self-Healing Recovery Actions

Overview

Self-healing recovery actions allow upti.my agents to automatically respond to check failures without waiting for human intervention. When a local check fails, the agent can execute a configured recovery action to restore the service, clean up resources, or run a diagnostic script. This reduces mean time to recovery (MTTR) and keeps your systems running even during off-hours.

Each recovery action is linked to a specific local check. When the check fails, the agent evaluates the retry and cooldown settings before executing the action. All execution results are logged and reported back to the upti.my dashboard for full visibility.

Common Settings

All recovery action types share the following configuration options:

Recovery Action Types

1. Restart Service (systemd)

Restart a systemd-managed service on the host. The agent runs systemctl restart for the specified service name. This is the most common recovery action for Linux servers.

{
  "type": "restart_service",
  "service_name": "nginx",
  "max_retries": 3,
  "retry_interval": 60,
  "cooldown_period": 300
}

2. Restart Docker Container

Restart a Docker container by name. The agent issues a docker restart command for the specified container. Useful for recovering crashed or unresponsive containers.

{
  "type": "restart_docker_container",
  "container_name": "redis-cache",
  "max_retries": 2,
  "retry_interval": 30,
  "cooldown_period": 180
}

3. Restart Kubernetes Pod

Delete a Kubernetes pod to trigger a restart by the controller (Deployment, StatefulSet, etc.). The agent uses kubectl to delete the pod in the specified namespace.

{
  "type": "restart_kubernetes_pod",
  "pod_name": "api-server-7d8f9b6c4-x2k9p",
  "namespace": "production",
  "kubectl_path": "/usr/local/bin/kubectl",
  "max_retries": 2,
  "retry_interval": 60,
  "cooldown_period": 300
}

4. Execute Shell Script

Run a custom shell script on the host. This is the most flexible recovery action, allowing you to execute any command or sequence of commands. The script runs with the agent's permissions and is subject to the configured timeout.

{
  "type": "execute_script",
  "script": "#!/bin/bash\ncd /var/app && ./restart.sh && echo 'Recovery complete'",
  "timeout": 60,
  "max_retries": 2,
  "retry_interval": 30,
  "cooldown_period": 600
}

5. Send HTTP Webhook

Send an HTTP request to an external endpoint as a recovery action. This is useful for triggering external automation pipelines, notifying third-party services, or calling a custom recovery API.

{
  "type": "http_webhook",
  "url": "https://automation.example.com/recover",
  "method": "POST",
  "headers": { "Authorization": "Bearer token123" },
  "body": "{ \"service\": \"api\", \"action\": \"restart\" }",
  "expected_status": 200,
  "max_retries": 3,
  "cooldown_period": 300
}

6. Clean Disk Space

Remove old files from specified directories to free up disk space. The agent deletes files older than the configured number of days. This pairs well with the Disk Usage local check.

{
  "type": "clean_disk_space",
  "paths": ["/var/log", "/tmp"],
  "older_than_days": 7,
  "max_retries": 1,
  "cooldown_period": 3600
}

7. Kill Process

Terminate a specific process by name or PID. You can choose between a graceful shutdown (SIGTERM) or a forced kill (SIGKILL). This is useful for stopping runaway processes that consume excessive resources.

{
  "type": "kill_process",
  "process_name": "stuck-worker",
  "signal": "SIGTERM",
  "max_retries": 2,
  "retry_interval": 15,
  "cooldown_period": 120
}

8. Clear Application Cache

Clear an application's cache by deleting a cache directory or running a cache-clearing command. This is helpful when stale cache data causes application errors or performance degradation.

{
  "type": "clear_cache",
  "command": "redis-cli FLUSHALL",
  "max_retries": 1,
  "cooldown_period": 600
}

9. DNS Flush

Flush the local DNS resolver cache. This can resolve issues caused by stale DNS records, such as after a DNS failover event. The agent uses the system default flush command or a custom command you provide.

{
  "type": "dns_flush",
  "custom_command": "systemd-resolve --flush-caches",
  "max_retries": 1,
  "cooldown_period": 300
}

10. Custom Recovery

A fully custom recovery action that lets you define an arbitrary script with complete control over the recovery logic. Use this for complex recovery workflows that don't fit into the other action types.

{
  "type": "custom_recovery",
  "script": "#!/bin/bash\necho 'Starting custom recovery...'\nsystemctl stop myapp\nrm -rf /tmp/myapp-locks\nsystemctl start myapp\necho 'Recovery complete'",
  "max_retries": 2,
  "retry_interval": 60,
  "cooldown_period": 600
}

Recovery Action Summary