# Operations

Services, migrations on production, redirects, packages, the preDeploy
hook, and logging in to a machine by hand.

## Services

Custom backend services deploy alongside the main app. Each service runs as a
systemd unit, listens on its own port, and routes through the load balancer
based on its declared domains.

```jsoc
deploy: {
  services: [
    {
      name: "transcode-worker",
      build: "go build -o services/transcode-worker/bin/transcode-worker ./cmd/transcode",
      bin:   "services/transcode-worker/bin/transcode-worker",
      port: 8082,
      domains: ["transcode.example.com"],
    },
  ],
}
```

`build` runs locally before any SSH. The output binary at `bin` is then synced
to each machine. If `build` fails, the deploy aborts before any machine is
touched.

## Migrations on Production

All pending migrations apply inside a single Postgres transaction. If migration
7 of 10 fails, all 10 roll back and the database is left exactly as it was
before the deploy started. The schema either advances completely or not at all.

Migrations are forward-only. There is no automatic rollback for a migration that
has already succeeded. If a release ships a bad migration, fix it with a new
migration.

Long-running migrations block the deploy, and that is by design. The new release
does not start serving traffic until the app database schema is ready.

A few Postgres DDL statements (`CREATE INDEX CONCURRENTLY`, `VACUUM`,
`REINDEX CONCURRENTLY`) cannot run inside a transaction and will fail under this
model. Apply those out of band: run the SQL directly via
`elements db -connect=production` and add a no-op migration file to keep
environments in sync.

## Redirects

```jsoc
deploy: {
  redirects: {
    "www.example.com": "example.com",
  },
}
```

The load balancer issues a 301 from source to destination before any service
routing. A typical use is redirecting `www` to apex, or apex to `www`.

## Packages

OS packages are installed on every deploy machine during provisioning.

```jsoc
deploy: {
  packages: ["ffmpeg", "imagemagick"],
}
```

Elements diffs the package list against the per-machine state, so
already-installed packages are not reinstalled. Per-environment or per-machine
`packages` entries extend this list.

## preDeploy

Local commands run before any service builds and before any SSH. They run
sequentially and fail fast.

```jsoc
deploy: {
  preDeploy: [
    "npm run lint",
    "go build ./...",
  ],
}
```

Use for cross-cutting prep that doesn't belong to a single service.

## Logging In to a Machine

`elements deploy -json` reports every failure, local and remote, and it is the
only thing a deploy needs. The commands below are for a human operator holding a
question the diagnostics did not answer. They are not a step in a deploy, and
reaching for one after a failed deploy trades a report the user can act on for a
tour of a machine.

```
elements ssh -connect=production               # ssh into the first production machine
elements ssh -connect=production#production2   # specific machine
elements build -connect=production -json       # query build state on the machine
```

Server logs are written to
`/elements/projects/<project>/.elements/logs/project.log` on the machine.

`elements db -connect=production` is not in that category. It is the normal way
to run sql against a deployed database, including the out-of-band DDL above.

## Related

- Build: `elements man build`. The deploy is a build that ships the release to a
  remote machine.
- Migrations: `elements man migrations`. The frozen-after-deploy rule and
  transaction batching.
- Database: `elements man database`. The bundled Postgres cluster and the