* automate setup-go-work
It's all to easy to forget to `make setup-go-work`, only to run into mysterious build failures. Let's default to doing this automatically, unless `SKIP_SETUP_GO_WORK` is true (or the legacy `IGNORE_GO_WORK_IF_EXISTS`, which was oddly named, since we can't actually ignore it.)
* Make setup-go-work recipe fail-fast with set -e
* Add container name to Docker logs
This will allow for querying Loki by container's name:
{job="docker",container_name="mattermost-postgres"}
* Configue Loki to prepare for OTLP ingestion
- Add a volume to Loki container to get the config
- Configure Loki with the expected labels so that we can query by job,
app, container.name...
* Add OpenTelemetry collector configuration
There are three pipelines:
1. logs/mattermost scrapes the logs from mattermost.log, parsing the
timestamp and severity, and pushes them to Loki.
2. logs/docker scrapes the Docker logs from *-json.log, parsing the
timestamp, the log itself and the container name, and pushes them to Loki.
3. metrics/docker scrapes the Docker socket to retrieve the containers'
uptime values and pushes them to Prometheus.
* Replace Promtail with OpenTelemetry collector
* Update build tooling for OpenTelemetry collector
1. Make sure that the logs directory is created
2. Swap Promtail with OpenTelemetry collector
* Scrape collector to get Docker stats
Prometheus needs to scrape the OpenTelemetry collector in the exposed
port to get the Docker stats, so that we can query the uptime with
metric container_uptime_seconds, which has a container_name label to
filter by container.
* Update Grafana dashboard for Docker health checks
1. Use Prometheus as the datasource in all queries
2. Simplify the mappings to either 0 (offline, red) or 1 (online,
green).
3. Unify all queries on container_uptime_seconds, filtering by
container_name, and making sure that the latest value we got is at most
15 seconds old, so that it does not show stale data.
4. Add Redis health check, that was missing
5. Update the dashboard title to Docker containers
* Tune Loki and OTel collector configs for local dev
- Switch filelog receivers to start_at: beginning so existing logs are
ingested on collector startup, not just new entries.
- Fix Docker log timestamp layout to use 9s (variable-length nanos)
instead of 0s (fixed-width), matching actual Docker JSON log format.
- Add ingester max_chunk_age to keep chunks open longer in the
single-instance dev setup, so that we can ingest older logs (the
window is max_chunk_age/2).
- Relax Loki limits for local development: allow unordered writes,
disable old-sample rejection, and raise ingestion rate/burst to 64 MB
to avoid throttling during bulk ingest.
- Update library version.
- Added MaxFlush delay to help reduce CPU usage.
- Fall back to LRU cache for the caches which use SCAN.
- Added mattermost-redis and running for all api layer
tests in Postgres.
https://mattermost.atlassian.net/browse/MM-59934
```release-note
NONE
```
* configure Loki for local development
Extend our docker compose setup to include Loki (along with promtail), scraping `logs/*.log` to ingest development logs as well as all logs from running containers.
While we're in here, teach Prometheus to scrape metrics from Docker containers too.
* tweak promtail labels
* document extra services in config.mk
* update home.json
Jesse HallamandGitHub
Alejandro García MontoroandGitHub
Pavel ZemanandGitHub
Agniva De SarkerandGitHub
Doug LauderandGitHub