# Operations

Services, migrations on production, redirects, packages, the preDeploy
hook, and logging in to a machine by hand.

## Services

Custom backend services deploy alongside the main app. Each service runs as a
systemd unit, listens on its own port, and routes through the load balancer
based on its declared domains.

```jsoc config.jsoc
deploy: {
  services: [
    {
      name: "transcode-worker",
      build: "go build -o services/transcode-worker/bin/transcode-worker ./cmd/transcode",
      bin:   "services/transcode-worker/bin/transcode-worker",
      port: 8082,
      domains: ["transcode.example.com"],
    },
  ],
}
```

`build` runs locally before any SSH. The output binary at `bin` is then synced
to each machine. If `build` fails, the deploy aborts before any machine is
touched.

## Migrations on Production

Pending migrations apply in filename order, each in its own Postgres
transaction together with the record that it applied. If migration 7 of 10
fails, 7 rolls back, 1 through 6 stay applied, and the deploy stops before the
release swap, so the running release keeps serving. Ship the fix and the next
deploy applies 7 through 10. The running release serves traffic while
migrations apply, so write each migration to keep it working against the new
schema.

Migrations are forward-only. There is no automatic rollback for a migration that
has already succeeded. If a release ships a bad migration, fix it with a new
migration.

The new release starts serving traffic once the app database schema is ready,
however long the migrations take.

One deploy migrates a database at a time: Elements holds a Postgres advisory
lock on the database for the whole run.

`CREATE INDEX CONCURRENTLY`, `REINDEX CONCURRENTLY`, and `VACUUM` go in a
migration tagged `@migration tx=false`, which applies in the deploy like any
other, so every environment stays in step. Production tables keep taking
writes while a concurrent index builds. See `elements man migrations`.

## Redirects

```jsoc config.jsoc
deploy: {
  redirects: {
    "www.example.com": "example.com",
  },
}
```

The load balancer issues a 301 from source to destination before any service
routing. A typical use is redirecting `www` to apex, or apex to `www`.

## Packages

OS packages are installed on every deploy machine during provisioning.

```jsoc config.jsoc
deploy: {
  packages: ["ffmpeg", "imagemagick"],
}
```

Elements diffs the package list against the per-machine state, so
already-installed packages are not reinstalled. Per-environment or per-machine
`packages` entries extend this list.

## preDeploy

Local commands run before any service builds and before any SSH. They run
sequentially and fail fast.

```jsoc config.jsoc
deploy: {
  preDeploy: [
    "npm run lint",
    "go build ./...",
  ],
}
```

Use for cross-cutting prep that doesn't belong to a single service.

## Logging In to a Machine

`elements deploy -json` reports every failure, local and remote, and it is the
only thing a deploy needs. The commands below are for a human operator holding a
question the diagnostics did not answer. They are not a step in a deploy, and
reaching for one after a failed deploy trades a report the user can act on for a
tour of a machine.

```bash Terminal
elements ssh -remote=production               # shell in /elements/app on the first production machine
elements ssh -remote=production#production2   # specific machine
elements build -remote=production -json       # query build state on the machine
```

`elements ssh` opens its shell in the app, `/elements/app`. Every Elements
service on the machine logs to the systemd journal. `elements logs -remote`
reads it from your machine (see `elements man logs`), and on the machine
`journalctl -u elementsd` shows the project server's.

`elements db -remote=production` is not in that category. It is the normal way
to run SQL against a deployed database, including the statements above.

## Related

- Build: `elements man build`. The deploy is a build that ships the release to a
  remote machine.
- Migrations: `elements man migrations`. The frozen-after-deploy rule and
  transaction batching.
- Database: `elements man database`. The bundled Postgres cluster and the