# Which Problems Are Solved
Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.
# How the Problems Are Solved
All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...).
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.
## New features
- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
- Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)
## Exporters
This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.
- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics
The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.
## Configuration and structure
- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.
# Additional Changes
- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
* fix: compatability for cockroach v22.1
* update cr version in docs and compose
* bind dev cockroach to 127.0.0.1
* Revert "bind dev cockroach to 127.0.0.1"
This reverts commit 1c7c6304ae.
* remove --listen-addr flag
* use v2.0.0-v2-alpha.16
* use v2.0.0-v2-alpha.17
Co-authored-by: Elio Bischof <eliobischof@gmail.com>
* feat: Configurable Unique Machine Identification
This change fixes Segfault on AWS App Runner with v2 #3625
The change introduces two new dependencies:
* github.com/drone/envsubst for supporting AWS ECS, which has its metadata endpoint described by an environment variable
* github.com/jarcoal/jpath so that only relevant data from a metadata response is used to identify the machine.
The change ads new configuration (see `defaults.yaml`):
* `Machine.Identification` enables configuration of how machines are uniquely identified - I'm not sure about the top level category `Machine`, as I don't have anything else to add to it. Happy to hear suggestions for better naming or structure here.
* `Machine.Identifiation.PrivateId` turns on or off the existing private IP based identification. Default is on.
* `Machine.Identification.Hostname` turns on or off using the OS hostname to identify the machine. Great for most cloud environments, where this tends to be set to something that identifies the machine uniquely. Enabled by default.
* `Machine.Identification.Webhook` configures identification based on the response to an HTTP GET request. Request headers can be configured, a JSONPath can be set for processing the response (no JSON parsing is done if this is not set), and the URL is allowed to contain environment variables in the format `"${var}"`.
The new flow for getting a unique machine id is:
1. PrivateIP (if enabled)
2. Hostname (if enabled)
3. Webhook (if enabled, to configured URL)
4. Give up and error out.
It's important that init configures machine identity first. Otherwise we could try to get an ID before configuring it. To prevent this from causing difficult to debug issues, where for example the default configuration was used, I've ensured that
the application will generate an error if the module hasn't been configured and you try to get an ID.
Misc changes:
* Spelling and gramatical corrections to `init.go::New()` long description.
* Spelling corrections to `verify_zitadel.go::newZitadel()`.
* Updated `production.md` and `development.md` based on the new build process. I think the run instructions are also out of date, but I'll leave that for someone else.
* `id.SonyFlakeGenerator` is now a function, which sets `id.sonyFlakeGenerator`, this allows us to defer initialization until configuration has been read.
* Update internal/id/config.go
Co-authored-by: Alexei-Barnes <82444470+Alexei-Barnes@users.noreply.github.com>
* Fix authored by @livio-a for tests
Co-authored-by: Livio Amstutz <livio.a@gmail.com>
* fix: rename iam command side to instance
* fix: rename iam command side to instance
* fix: rename iam command side to instance
* fix: rename iam command side to instance
* fix: rename orgiampolicy to domain policy
* fix: merge conflicts
* fix: protos
* fix: md files
* implement deprecated org iam policy again
Co-authored-by: Livio Amstutz <livio.a@gmail.com>
* begin init checks for projections
* first projection checks
* debug notification providers with query fixes
* more projections and first index
* more projections
* more projections
* finish projections
* fix tests (remove db name)
* create tables in setup
* fix logging / error handling
* add tenant to views
* rename tenant to instance_id
* add instance_id to all projections
* add instance_id to all queries
* correct instance_id on projections
* add instance_id to failed_events
* use separate context for instance
* implement features projection
* implement features projection
* remove unique constraint from setup when migration failed
* add error to failed setup event
* add instance_id to primary keys
* fix IAM projection
* remove old migrations folder
* fix keysFromYAML test
Tim MöhlmannandGitHub
Livio SpringandGitHub
SilvanandGitHub
