Machines

What runs on a deploy machine, what to back up, how the OS stays patched, and the difference between a deploy and looking after a machine.

A Deploy Is Never Verified

elements deploy -json with no errors is a successful deploy. The report is the fact. There is nothing to confirm afterwards: no request to the URL, no systemctl status, no shell on the box to look at /elements, no log to tail. A failed deploy is reported the same way, with the machine's own diagnostics, and the fix goes in the source tree or config.jsoc, never on the machine. See What Deploy Does (opens in a new tab).

Looking after a machine is a different activity, and elements ssh is for it: a kernel update, a reboot, a full disk, a question about memory. Do it when the user asks or when there is a reason, never as a follow-up to a deploy.

What Is on the Box

Elements installs and runs everything: the project server, the app, the load balancer, the job runner, and on a prototype the Postgres cluster. All of it runs in a systemd slice sized to the machine's RAM minus a reserve for the OS, so the services together can never take the memory the system needs. If they outgrow the slice the kernel stops one, systemd restarts it, the machine stays reachable, and the next deploy reports it as an out-of-memory diagnostic with the machine's RAM. That diagnostic is the signal to move to the large machine. See Sizing (opens in a new tab).

Nothing on the Box Is Worth Backing Up

The app machine holds code that is in your repository and a release that the next deploy rebuilds. Files the app stores go in the database or in object storage, where every machine reads them (elements man recipes/file-upload). So there is no machine backup, no disk snapshot, and no snapshot schedule: the database is the one thing with data in it, and on the production tier the provider backs it up daily. A prototype's database lives on the box and is not backed up, which is what makes it a prototype.

Security Updates and Reboots

Ubuntu installs security updates on its own. A kernel update waits for a reboot. The agent checks for one when the user asks or an advisory lands, over elements ssh:

Terminal
elements ssh                                    # a shell on the first production machine
apt list --upgradable 2>/dev/null | grep -i security
test -f /var/run/reboot-required && cat /var/run/reboot-required
df -h /                                         # disk
free -m                                         # memory

A reboot takes the machine down for a few minutes. Warn the user and get a yes first. With one machine the site is down for those minutes, and the load balancer is down with it. With several machines, reboot one at a time; the site stays up through it only if DNS points at every machine, as in Load Balancer (opens in a new tab). Otherwise it is still one at a time, with the site down briefly each time.

Terminal
elements ssh -remote=production#production0     # then: sudo reboot

After the machine is back, nothing else is needed. The services start with the machine, the load balancer serves the maintenance page until the app answers, and the next elements deploy is a normal one.

A planned outage that is not a reboot is elements maint on, in Load Balancer (opens in a new tab).

Firewall and SSH

The provider's firewall is the firewall: the app machine takes 22, 80 and 443 from anywhere, and the database takes 5432 from the app machine's security group only. SSH is key-only on every cloud image. Nothing further is configured on the box, and nothing needs to be.