# Logs

What the app logs, where it goes, and how to read it back. The record format
below is a public contract: `v: 1` keeps its fields and their meaning, and a
change to either comes with a new version.

## Reading Logs

```bash Terminal
elements logs                          # everything, oldest first
elements logs -f                       # then keep going as records are written
elements logs -service app -since 10m  # the app's last ten minutes
elements logs -since 2026-10-13T15:00:00Z -until 2026-10-13T15:05:00Z
elements logs -json                    # the records as stored, one per line
```

- `-service <name>`: one service's records. `app` is the program
  `elements start` or `elements run` runs; `job` is the job server and its
  workers; `project` is the project server itself (the `elementsd`
  process).
- `-f` (`-follow`): print what is there, then new records as they arrive.
- `-since` / `-until`: a duration back from now (`30s`, `10m`, `2h`, `1d`) or
  a time. A time is RFC 3339 (`2026-10-13T15:04:05Z`) or a local
  `2026-10-13 15:04`.
- `-json`: the stored lines, for `jq` and other tools.

Without `-service` every service is merged in time order. `elements logs`
reads the files in `.elements/logs` and needs no running server, so it works
while the app is down, from any terminal or subagent. It reads the current
file of each service, not the rotated backups.

## Reading a Deploy Machine's Logs

```bash Terminal
elements logs -remote                     # every production machine, merged
elements logs -remote=staging#web2 -f     # one machine, then keep going
elements logs -remote -service app -since 1h -json
```

`-remote[=<env>[#<machine>]]` reads the systemd journal on the deploy
machines over ssh, the connection `elements db -remote` uses. The environment
defaults to `production`. The journal holds `elements-app` (service `app`),
`elementsd` (`project`), `elements-job` (`job`) and one unit per custom
service, named after it. `-service`, `-since`, `-until`, `-f` and `-json` work
as they do locally.

On a deploy machine `elements start` writes each of the app's records to the
journal as its JSON line, stamped with `service`, `pid`, `run` and `stream`,
and every other line as the app printed it. A record keeps its own `ts`; any
other line is a raw line at the time the journal received it. Lines are drawn
as they are locally. With no `#<machine>` every machine of the environment is
read and the lines are merged by `ts`. When there is more than one machine,
each line starts with its machine's name, and `-json` sets `machine` on each
record.

A damaged line never stops a read. A line that is not a record (a write cut
off by a full disk or a crash, or bytes that are not text) is shown as a raw
line, unreadable bytes replaced, and `-json` prints it wrapped as a raw record,
so every line it prints is a record. A record with no usable `ts` is placed
beside the line before it. A line over 1 MB is cut there. With `-f`, a file
that is rotated or deleted and made again is read from its start.

## Where Logs Go

`elements start` reads everything the app writes to stdout and stderr, one
line at a time. Each line is shown in the terminal and appended to
`.elements/logs/<service>.jsonl`, `app.jsonl` for the app. `elements run`
logs the same way, to `app.jsonl`. The project server, `elementsd`, does the
same for the processes it runs: job workers write `job.jsonl`, beside the job
server's own records. Its own records go to `project.jsonl`, which holds
records and nothing else; a panic it recovers from is an `error` record there.
A crash nothing recovers from (a runtime fatal error, or a `SIGQUIT` dump) is
printed by the Go runtime as plain text to stderr, which is appended to
`.elements/logs/crash.log`. That file is kept as written: nothing reads,
moves or clears it, and `elements logs` does not show it. If the server can't
open `project.jsonl`, it logs to stderr, so to `crash.log`. Under systemd on a
deploy machine both go to the journal instead.

Several processes can append to one file at once, as `elements start` and
`elements run` both do to `app.jsonl`; each line lands whole. A file rotates
at 100 MB to `<service>-<time>.jsonl`, and the last three backups are kept, as
written, for up to seven days. `<service>.jsonl.lock` is the lock the writers
take to rotate. The terminal shows a line as

```text Terminal
[15:02:44 info ] http://localhost:4000 → postgres://127.0.0.1:5433/myapp
[15:02:45 info ] get 200 5ms /todos
[15:02:45 info ] call 31ms app/pages/todos/services:addTodo
[15:02:46 error] get 500 2ms /todos/7
                 TypeError: Cannot read properties of undefined (reading 'title')
                     at TodoPage.render (app/pages/todo.ts:14:22)
[15:02:47      ] hello from console.log
```

The time is local. Each line has two columns: the time and level, always 17
characters wide, then the message. The further lines of a message and an
error's stack are indented to the message column. A line the program printed
itself, such as a `console.log`, is shown in the message column under the time
it was read and no level, as it was written and with its own colours.

Nothing wraps. On a terminal each line is one row, cut at the terminal's width
with `…`, and cut again to the new width when the terminal is resized; each
further line of a message or a stack is its own row, cut the same way.
`elements start` and `elements logs` both cut on a terminal. `y` in
`elements start` copies whole lines, and `elements logs` into a pipe or a
file, `elements logs | less` for one, prints them whole.

A request draws as method, status and duration, then the path, so a cut line
loses the end of the path. A call to a service function draws as `call`, the
duration and the function's name. The fields are separated by single spaces,
with no column padding after the time and level. The startup line names the url the app serves and the
database it uses, without credentials. Any other record's `attrs` follow its
message as `key=value`, leaving out a value the message already says: the
project server's `child up` carries `child=job-1`.

Colour marks only what needs attention. The time and level are dim, but for
the level word of a `warn` (yellow) or an `error` (red). A 2xx status is green,
a 3xx yellow, and a 4xx or 5xx red; every message and
stack is the terminal's own colour. A request's or a call's duration is dim. In the startup
line the links are blue and the rest of the line is dim. In VS Code's terminal
the url opens in VS Code's browser, the tab the app runs in, and the database
is a link too, which opens `elements db` (see `editors/vscode`); elsewhere the
url opens in the default browser.

The log is never allowed to slow the app down. If the disk cannot keep up,
records are dropped rather than making the app wait, and a `warn` record from
service `elements` says how many. A write that fails partway (a full disk)
drops the rest of what it held, counted the same way, and the next record
starts on a new line.

## Two Kinds of Line

- **A record from the logger.** The framework's logger writes each record as
  the byte `0x1E`, one JSON object and a newline: an RFC 7464 JSON text
  sequence. Request lines, startup lines and framework warnings are records.
- **Everything else is raw:** `console.log`, an uncaught exception's stack,
  Node's own fatal output. Each raw line is stored as a record with no `level`,
  its `ts` the time it was read and its colour codes stripped.

A program's own `console.log(JSON.stringify(x))` never starts with `0x1E`, so
JSON the app prints is never mistaken for a record. Run without a parent, as
plain `node`, the logger's records print as they are.

## The Record, `v: 1`

Each line of a `.jsonl` file is one JSON object:

```json .elements/logs/app.jsonl
{"v":1,"ts":"2026-10-13T15:02:45.123000Z","level":"info","msg":"get /todos 200 (5ms)","attrs":{"method":"get","path":"/todos","status":200,"ms":5},"service":"app","pid":41872,"run":"mgo3k2xq","stream":"stderr"}
```

| Field | Set by | Meaning |
|---|---|---|
| `v` | writer | Schema version, `1`. |
| `ts` | logger, or the parent for raw lines | RFC 3339, UTC, microseconds. |
| `level` | logger | `debug`, `info`, `warn` or `error`. Absent on a raw line. |
| `msg` | logger, or the parent for raw lines | The message. May hold newlines. |
| `attrs` | logger | Structured fields. Absent when there are none. |
| `err` | logger | `{type, message, stack}`, an error the line carries. |
| `service` | parent | `app`, `job`, `project`, or `elements` for a parent's note about the log itself. |
| `pid` | parent | The process that wrote the line. |
| `run` | parent | One start of the process. A restart is a new `run`; a hot reload is not. |
| `stream` | parent | `stdout` or `stderr`. |
| `truncated` | parent | `true` when the line was over 64 KB and was cut. |
| `machine` | `elements logs -remote` | The deploy machine, when the lines of more than one are merged. |

Fields the framework sets in `attrs`:

- **A request:** `method` (lowercase), `path`, `status`, `ms`, and `host`
  when the route has a `domain`.
- **A service call:** `call`, the function as `<module>:<name>`, and `ms`.
- **Inside a request:** `session`, the last four characters of the session id.
- **Startup:** the `listening` record carries `listen`, the url the app
  serves, and `db`, the database as `postgres://<host>:<port>/<name>`, left
  off when there is no database (a `warn`, `postgres not connected`,
  follows).

A reader should ignore fields it does not know: later records in `v: 1` may
add some.

## Levels

`LOG_LEVEL` in the app's environment sets the lowest level written: `debug`,
`info` (the default), `warn` or `error`. A 4xx response logs at `info`, since
any client can produce one on demand; a 5xx logs at `error` with the error
behind it in `err`. A built asset the app serves from `/assets` logs at
`debug`, so a page load logs the page and not each file it pulls in. Records
below the level are never written, so by default they are in neither the
terminal nor `app.jsonl`.