# Which Problems Are Solved
In Zitadel's append-only event-sourced architecture, the
`eventstore.events2` table grows indefinitely. PostgreSQL's default
autovacuum uses a percentage-based scale factor, so as the table grows,
the number of changed rows required to trigger a `VACUUM` or `ANALYZE`
drifts towards infinity. Without regular vacuums, the table's Visibility
Map becomes stale, disabling fast Index-Only Scans and forcing expensive
heap reads. Without regular analyzes, query planner statistics become
stale, leading to suboptimal execution plans.
This causes eventstore operations to progressively degrade as `events2`
grows, even without CPU, memory, or I/O saturation. A manual `VACUUM
ANALYZE` immediately restores performance, confirming the root cause.
- If `Eventstore.Autovacuum` is left at its default, `events2` keeps
using PostgreSQL's default, percentage-based autovacuum/autoanalyze
scale factors, which become impractically infrequent on large tables.
- There was previously no supported way to apply static, table-level
autovacuum tuning to `events2` through Zitadel's own configuration/setup
process.
# How the Problems Are Solved
- Added an `Eventstore.Autovacuum` runtime configuration block to
`cmd/defaults.yaml` (disabled by default):
```yaml
Eventstore:
Autovacuum:
Enabled: false # ZITADEL_EVENTSTORE_AUTOVACUUM_ENABLED
VacuumThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_VACUUMTHRESHOLD
AnalyzeThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_ANALYZETHRESHOLD
```
- Added a repeatable `zitadel setup` migration step
(`cmd/setup/eventstore_autovacuum.go`) that:
- When `Enabled: true`, disables the percentage-based
`autovacuum_vacuum_scale_factor`, `autovacuum_analyze_scale_factor`, and
`autovacuum_vacuum_insert_scale_factor` on `eventstore.events2`, and
applies static thresholds (`autovacuum_vacuum_insert_threshold`,
`autovacuum_vacuum_threshold`, `autovacuum_analyze_threshold`) from the
config instead.
- When `Enabled: false`, resets those storage parameters on
`eventstore.events2` back to the cluster defaults.
- Implements `Repeatable.Check()` so the step only re-runs when the
configuration actually changed since the last `zitadel setup` run.
- Added documentation in the new "Performance tuning" page.
# Additional Changes
- Move the projection documentation into performance tuning page
- Expand and update the projection documentation to the latest state in
zitadel. (contained some stale information)
# Additional Context
Several other issues describe read-performance symptoms consistent with
this same root cause (stale `events2` visibility map / planner
statistics at scale, without resource saturation). Since this PR
addresses the shared root cause:
- Closes#12448
- Closes#10754
- Closes#10260
- Closes#8585
- Closes#9239
---
_Generated by [Claude
Code](https://claude.ai/code/session_01HoKvEY7niCVgLCz7CajBwW)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
* Use slices.Contains over custom function
* Correctly remove roles from granted roles
* fix(setup): repair user grants with stale roles (GHSA-v859-c572-qh5p)
Add setup step 73 that reconciles existing user grants whose roles were
left too broad by the buggy cascade removal in removeRoleFromUserGrant.
The corruption lives in the eventstore event payloads, so the step pushes
a corrective user.grant.cascade.changed event per affected grant (roles
intersected with the currently valid set) and re-triggers the user grant
projection. Runs in the second setup slice, after the projection tables
it reads have been created.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(setup): scope GHSA-v859-c572-qh5p repair to grant-based user grants
Direct user grants can never be hit by this bug (only ChangeProjectGrant's
multi-role cascade to grant-based grants can trigger it), so drop the
direct-grant branch from the finder query to avoid stripping unrelated,
legitimate roles that merely mismatch for other reasons (e.g. stale
role_key drift). Also exclude removed instances from the migration scope,
and log the number of grants fixed per instance.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Livio Spring <9405495+livio-a@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
# Which Problems Are Solved
Zitadel exposes the secrets generator configuration through its admin
api. This allows instance admins to manage them on their own and they
can create overwrite the system / runtime defaults (incl. expiration).
This very much needed in multi-instance scenarios such as zitadel.cloud.
Currently the invite code configuration was not manageable through the
API, but only runtime config.
# How the Problems Are Solved
- added the `invite_code` type to the API allowing it to be set and
retrieved.
- added the type to console's management list
- added the type to be stored on instance setup
- change the `GetSecretGenerator` endpoint to fall back to the runtime
config if no config is stored on the instance itself
- ensure the `length` and at least one charset is enabled, return an
error otherwise
- expiry is not enforced, so 0 allows codes with no expiry (current
state)
# Additional Changes
None
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10474
# Which Problems Are Solved
Login v2 uses the `users.v2.ListUsers`-endpoint to get a user by login
name. This query had an inefficient `WHERE`-clause.
# How the Problems Are Solved
Update the view and use a specific clause for this query.
# Additional information
Added in https://github.com/zitadel/zitadel/pull/10475
---------
Co-authored-by: Marco A. <kwbmm1990@gmail.com>
Co-authored-by: Livio Spring <9405495+livio-a@users.noreply.github.com>
# Which Problems Are Solved
During the setup step we saw rare cases which caused setup to fail after
executing steps 40, 64 and 70.
# How the Problems Are Solved
Close currently open database connections so that they fetch the correct
type mapping for the `eventstore.command2` database type.
# Additional Changes
Ensure correct order of setup steps 64 and 70.
# Additional Context
None
---------
Co-authored-by: Livio Spring <9405495+livio-a@users.noreply.github.com>
# Which Problems Are Solved
- The eventstore did not support intentionally overwriting the resource
owner when creating events for aggregates that may be reused across
owners.
- Resource owner handling was implicit and could not be controlled per
command/event type.
- We needed a safe way to distinguish between:
- keeping the existing aggregate owner, and
- explicitly setting a new owner for specific create-like events.
# How the Problems Are Solved
- Introduced a new eventstore command type with an explicit
enforce_owner flag.
- Updated eventstore.commands_to_events and eventstore.push so owner
assignment is now explicit:
- if enforce_owner is true, the command owner is written
- if enforce_owner is false, the existing aggregate owner is retained
when present
- Added EnforceResourceOwnerCommand and wiring so command types can opt
in to enforced owner behavior.
- Wired the new behavior through the v3 eventstore push path, including
compatibility fallback for older command type mapping.
- Added migration/setup changes to register and use the new command type
and SQL functions.
- Added and updated tests for owner overwrite and aggregate ID reuse
scenarios.
# Additional Changes
- Added small migration/setup robustness improvements related to
eventstore setup ordering and helper reuse.
- Added focused test coverage for enforced owner behavior and
sequencing.
- Events that currently allow owner changes (implement
EnforceResourceOwner) are:
- AddedEvent (action)
- GroupAddedEvent
- StartedEvent (idp intent)
- ProjectAddedEvent
- HumanAddedEvent
- HumanRegisteredEvent
- MachineAddedEvent
- CreatedEvent (schema user)
# Additional Context
- Follow-up for eventstore owner-handling correctness in create flows
and aggregate ID reuse cases.
- No additional issue link was attached for this change.
---------
Co-authored-by: abhishek kumar gupta <abhishek818t@gmail.com>
# Which Problems Are Solved
Opaque tokens now use authenticated encryption.
# How the Problems Are Solved
- Upgrade zitadel/oidc to v3.47
- Copy crypto implementation for refresh and session tokens (internal to
zitadel)
- Added config that allows validating old tokens for gradual roll-out
# Additional Changes
- Set NX cache for `integration-test-build` to `false`, working on a
seperate fix.
# Additional Context
- closes#11315
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: muhlemmer <5411563+muhlemmer@users.noreply.github.com>
# Which Problems Are Solved
https://github.com/zitadel/zitadel/pull/11390 renamed "Console" to
"Management Console". While
https://github.com/zitadel/zitadel/pull/11706 already reverted an
unintended rename of the feature key to enable the management console to
use the V2 API for user creation. It was now also discovered that the
rename of the feature itself also broke existing (default)
configurations.
# How the Problems Are Solved
Added a `mapstructure` tag on the instance feature to handle existing
configs.
# Additional Changes
Removed unused `TokenExchange` from the default configuration.
# Additional Context
- relates to #11390
- relates to #11706
- requires backport to v4.x
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Marco A. <marco@zitadel.com>
## Problem description
- Setup currently creates/updates relational tables, but there is no
built-in way to fully reset the relational schema during iterative
development.
- Re-running setup after schema/projection changes can leave stale
relational objects and projection state behind, which makes local/dev
validation harder.
- There is no explicit, configurable switch in setup steps for
destructive schema recreation behavior.
## How the Problems Are Solved
- Adds a new setup step configuration section for relational tables with
a `ShouldRecreateSchema` flag (default `false`).
- Wires the new config flag into the transactional tables setup step.
- Extends the transactional tables execution logic to optionally:
- Drop the `zitadel` schema with `CASCADE`
- Clean related projection state entries for relational tables in
`projections.current_states`
- Recreate tables through the existing setup flow afterward
- Wraps the destructive operations in a transaction and keeps
error/rollback handling plus logging for visibility.
## Additional Changes
- Adds an explicit warning in step configuration that schema recreation
is intended for development and not production use.
- Keeps behavior fully backward-compatible by default
(`ShouldRecreateSchema: false`), so existing setups are unchanged unless
the flag is enabled.
## Additional Context
- Follow-up for relational setup/dev workflow improvements.
- PR: #11833
## How to use it
### Env `export ZITADEL_RELATIONALTABLES_SHOULDRECREATESCHEMA=true`
### Config
Add the following to your custom setup steps config:
```yaml
RelationalTables:
ShouldRecreateSchema: true
```
# Which Problems Are Solved
The total number of configured IDPs seemed of.
We were counting only the IDP table for non-templated IDPs.
# How the Problems Are Solved
Add a count trigger migration for the IDP template table.
Entries will be counted under the existing
`ResourceCountIdentityProvider`
# Additional Changes
- none
# Additional Context
- Reported internally
- Implemented in #9979
- Related https://github.com/zitadel/zitadel/issues/9957
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
If the `cache.objects`-table does not exist setup failes with the
following error:
`ERROR: relation "cache.objects" does not exist at character 44`
# How the Problems Are Solved
Changed statement to prevent immediately throw an error if the table is
missing.
# Additional Context
- reported in
https://discord.com/channels/927474939156643850/1473769282880934089
- backport to v4
# Which Problems Are Solved
When running Zitadel behind a reverse proxy and especially when the API
and the login UI don't run on the same domain, Zitadel needs to be
configured to trust the corresponding domains and use them in public
responses, like email links and more.
This can be done by adding a trusted domain. However it's currently only
possible through the API and not in the instance setup process.
# How the Problems Are Solved
Added a possibility to configure multiple trusted domains in the first
instance setup process.
# Additional Changes
None
# Additional Context
- closes#11153
# Which Problems Are Solved
As part of the consistent naming effort, this PR focuses on
"Organization domain".
# How the Problems Are Solved
- All terms referring referring to Organization Domains where changed to
be Organization Domain
# Additional Changes
None
# Additional Context
- closes [#11283](https://github.com/zitadel/zitadel/issues/11283)
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
# Which Problems Are Solved
Group attributes weren't masked if their corresponding key was
configured. For example setting `data` as a masked key entry in the
runtime config would still print unmasked event data. This was because
ReplaceAttr does not receive group attributes, only the flattened
attributes with group information.
# How the Problems Are Solved
Check the current groups stack in the replacer. When a configured key is
found in the group, mask the current attribute. This means that the
attribute structure is preserved and all sub-keys of a masked group are
still printed.
# Additional Changes
- Use a binary search for key matching. As we are now comparing slice
against slice, a binary search should make the process a little faster
if many keys are configured.
# Additional Context
- Follow-up for PR #11435
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: muhlemmer <5411563+muhlemmer@users.noreply.github.com>
# Which Problems Are Solved
There were still some emails (passkey registration and domain claimed)
sent with links pointing to login v1 even when the login v2 was enabled
for the instance.
Also while looking into the issue, it was discovered that some links
pointing to login V2 were not correctly generated.
# How the Problems Are Solved
- Added default paths for passkey registration and domain claimed
notifications
- Fixed the existing paths to properly handle concatenation (resp. use
`url.ResolveReference`)
- Change their go types (from string) to `*url.URL`
- Added a mapstructure hook for string to url
- Removed unnecessary `InstanceSetupFeatures` and corresponding
conversions
- Refactored the methods on the `login.DefaultPaths` struct and added an
interface to the `Commands` to only need to pass a single config (and
not every method)
- Added an `OriginURL` method to the `DomainCtx` to prevent going from
url to string and back
- Added the use of the templates in case of enabled login v2 for passkey
registration and domain claimed)
# Additional Changes
None
# Additional Context
closes#10643
---------
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Livio Spring <livio@zitadel.com>
Co-authored-by: Max Peintner <peintnerm@gmail.com>
Co-authored-by: Gayathri Vijayan <66356931+grvijayan@users.noreply.github.com>
# Which Problems Are Solved
Inconsistent naming of service account, found in the following
variations:
- Machine User
- machine user
- Service User
- Machine Account
- Technical Account
- User: Type Machine
# How the Problems Are Solved
Attentive search and replace.
Localizations have been translated using Copilot
# Additional Changes
Some unused methods have been removed from the Go code.
# Additional Context
- Closes#11285
## Which problems are solved
https://github.com/zitadel/zitadel/pull/11484 introduced a regression
that causes the setup process to fail on existing Zitadel deployments.
This prevents users from upgrading to recent versions without
encountering setup failures.
## How the problems are solved
Setup step 69 has been corrected to properly handle existing deployment
configurations and prevent setup failures during initialization and
upgrades.
## Additional Context
introduced by
[7a41fe968b](https://github.com/zitadel/zitadel/commit/7a41fe968b9fcb69b378336740b74e5448c1ff81)
### Testing
- [x] Verified setup succeeds on fresh deployments
- [x] Verified setup succeeds on existing deployments
- [x] Verified migration from PostgreSQL 17 to 18 after running setup of
this version
# Which Problems Are Solved
Streams allow differentiating logs produced by different components of
Zitadel.
# How the Problems Are Solved
The `backend/v3/instrumentation/logging` package now exposes convenience
function for setting and getting a logger from the context. As well as
high-level functions to emit log records at various levels. When
constructing a new logger a "stream" needs to be specified:
- **runtime**: General runtime logs, such as startup and shutdown
messages. Default for logs that do not belong to the other categories.
- **request**: Logs for incoming API and HTTP requests.
- **event_handler**: Logs for event handling in projections.
- **queue**: Logs for the job queue processing.
- **event_pusher**: Logs for event pushing to the database. Disabled by
default, contains sensitive information.
Each line from the returned logger contains a `stream` field as well as
a `version` field with the current Zitadel version.
## Runtime config
Streams can be enabled by passing an array of stream names in the
runtime config. Because some log streams may contain sensitive data
(especially events), it is now also possible to mask values by their
key.
# Additional Changes
- Wrap `slogctx` in the `logging` package. (Except API error converter
packages, because of import cycle)
- Add some docs to `logging` package so other devs understand how to add
logging to Zitadel
- Add `logging.OnError` and `logging.WithError` helper functions with
`Panic()` and `Fatal()` methods, to preserve current calls in the `cmd`
packages.
- Add instance context extractor.
- Only output request details in the request info log. Request ID
remains propagated through context.
- Moved middleware functionality into protocol specific packages.
- Removed setting of URI to context in metric middleware. There were ony
setters and no getters. (Unused value)
- Reuse a single statusWriter in the middleware package for middlewares
that need to know the response status.
# Additional Context
- Closes#11333
- Closes#11331
- Partly #11330
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
When starting Zitadel with Postgres version 18, setup fails with the
following error:
`level=error msg="migration failed" caller=".../cmd/setup/setup.go:373"
code=0A000 detail= error="ERROR: partitioned tables cannot be unlogged
(SQLSTATE 0A000)" hint= message="partitioned tables cannot be unlogged"
name=34_add_cache_schema severity=ERROR`
# How the Problems Are Solved
- Modify setup step 34 to ensure compatibility with PostgreSQL 18 by
changing the creation of the partitioned tables to`LOGGED` tables but
keep the partitions `UNLOGGED`.
- Added an additional setup step which alters the table persistence of
the partitioned tables to `LOGGED`.
# Additional Changes
- Bumped Postgres compatibility to version 18 in docs.
- Ensure default partitions for cache tables
## Additional Context
- closes https://github.com/zitadel/zitadel/issues/10712
- backport to v4
- migration from PostgreSQL version 17 to 18 was verified using
`pg_dumpall` and restoring the created backup file
- and new setups using PostgreSQL version 18 directly
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
# Which Problems Are Solved
The current back-channel logout notification still used a single
projection handler for handling terminating sessions and notifying the
necessary clients. This can lead to back pressure and delayed
notifications.
# How the Problems Are Solved
The handler now only creates a job in the (river) worker queue to handle
the notification.
The worker will then search for all necessary clients to be informed and
create a job for each.
These jobs will then be picked up by the (same) worker(s) again, which
will create, sign and send the logout token to the client. If one
notification fails, this one job will be retired, but others can still
be processed in parallel and ore not affected.
Each successful job will still create an event on the session stating
the successful notification of the client.
The existing `OIDC.DefaultBackChannelLogoutLifetime` configuration has
been deprecated in favor of making it part of the new
`OIDC.BackChannelLogout` config.
# Additional Changes
None
# Additional Context
- feature check is still executed and will be removed seperately for
easier review: https://github.com/zitadel/zitadel/issues/11277
- closes https://github.com/zitadel/zitadel/issues/9279
- requires backport to v4.x
# Which Problems Are Solved
Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.
# How the Problems Are Solved
All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...).
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.
## New features
- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
- Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)
## Exporters
This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.
- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics
The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.
## Configuration and structure
- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.
# Additional Changes
- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
# Which Problems Are Solved
The payload in actions V2 is currently sent as JSON to the target
endpoint. It might get exposed to intermediary infrastructure or logging
systems.
For these scenarios there needs to be an application-layer encryption,
where the provider of the endpoint can define an key to be used for the
encryption.
# How the Problems Are Solved
- Added an additional option to the target to specify the payload type:
`JSON` (current and default), `JWT`, `JWE` (api and console)
- added endpoints to upload and manage public keys (to be used for
encryption) for a target
- updated all action v2 executions (interceptors, oidc, saml, ...) to
provide the `GetActiveSigningWebKey` from queries
- implemented jwt and jwe in the exections incl. refactoring of code and
tests
- changes to the authn_keys table:
- added a `fingerprint` column
- dropped not null constraint on expiration
- moved the `GetSignerOnce` into its own package to prevent circular
dependencies
# Additional Changes
None
# Additional Context
closes#11061
---------
Co-authored-by: Marco A. <marco@zitadel.com>
Co-authored-by: conblem <mail@conblem.me>
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
The change was introduced in
https://github.com/zitadel/zitadel/pull/11178.
The fix is to prevent wiping the memberships because the projection did
not init yet.
### Changes
- Introduces a check to determine if the member role synchronization
should be executed based on the existence of a specific database table
(`projections.instance_members4`).
- Ensures that the synchronization process only runs if the required
table is present in the database.
# Which Problems Are Solved
Zitadel v4.7.2 fixed a security issue by switching to the permission v2
framework for user APIs. It appears that systems that are running since
before v2.68 that were affected by a precision bug in the eventstore,
which was fixed in that version. The precision bug results in certain
events being "skipped" while being projected into the fields table, used
by the new permission system. This caused certain membership roles to be
missing, resulting in empty user lists when executed by the affected
member. The permission system basically finds no matching memberships
and therefore returns no users at all.
# How the Problems Are Solved
After research we concluded that the legacy membership projections are
projected correctly. This PR synchronizes the projected state into the
fields table. As the membership roles are not marked unique, all rows
are first deleted and then the correct membership roles are then
inserted. The operation happens in a single transaction, during which
the fields table will remain locked for modifications. This to prevent
possible concurrent modifications to membership states.
# Additional Changes
- none
# Additional Context
- Introduced in
https://github.com/zitadel/zitadel/commit/0e17d0005a98ccbf92139961ef702ef03208ffd3
- Released in
[v4.7.2](https://github.com/zitadel/zitadel/releases/tag/v4.7.2)
- Related: https://github.com/zitadel/zitadel/issues/8863
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
<!--
Please inform yourself about the contribution guidelines on submitting a
PR here:
https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md#submit-a-pull-request-pr.
Take note of how PR/commit titles should be written and replace the
template texts in the sections below. Don't remove any of the sections.
It is important that the commit history clearly shows what is changed
and why.
Important: By submitting a contribution you agree to the terms from our
Licensing Policy as described here:
https://github.com/zitadel/zitadel/blob/main/LICENSING.md#community-contributions.
-->
One-time recovery codes are a common multi-factor authentication (MFA)
backup method, letting users access their account if they lose other MFA
devices. Support for recovery codes can also reduce support burden for
users locked out of their accounts and provide a more secure and
reliable form of verification than security questions.
# Which Problems Are Solved
Zitadel currently lacks support for recovery codes.
# How the Problems Are Solved
This PR partially addresses recovery code support in Zitadel.
Importantly, it adds recovery codes as a new 2FA `Factor` and an
additional `Check` type for the Session API.
```
Example recovery code flow:
1. User generates N new recovery codes using `POST /v2/users/{user_id}/recovery_codes`
2. Zitadel hashes and stores these codes and returns the un-hashed codes in the response
3. User creates new session with an additional check: `recoveryCode`
4. Code is checked against hash and, if valid, cannot be used again
5. User attempts to adds N more codes using the same endpoint
6. If `remaining_codes + N <= RecoveryCodes->MaxCount` config value, then recovery codes are added in addition to original codes
7. User can remove all recovery codes using `DEL /v2/users/:userId/recovery_codes`
```
This PR adds:
- [x] Session recovery_code check support on `POST+PATCH /v2/sessions`
endpoints
- [x] Adds `mfa_recovery_code_checked_at` column (default null) to
`projections.sessions8` table
- [x] Support for `SECOND_FACTOR_TYPE_RECOVERY_CODES` as available 2FA
method on login policy
- [x] Support for importing recovery codes in /import code
Missing, will _not_ implement in this PR:
- [ ] Admin console support for displaying Recovery Code settings for
user(s)
- [ ] Zitadel Typescript login support for recovery codes
TODO:
- [x] Additional unit and integration tests
- [x] Error translations
# Additional Changes
None
# Additional Context
- Closes#6898
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Replaces Turbo by Nx and lays the foundation for the next CI
improvements. It enables using Nx Cloud to speed the up the pipelines
that affect any node package.
It streamlines the dev experience for frontend and backend developers by
providing the following commands:
| Task | Command | Notes |
|------|---------|--------|
| **Production** | `nx run PROJECT:prod` | Production server |
| **Develop** | `nx run PROJECT:dev` | Hot reloading development server
|
| **Test** | `nx run PROJECT:test` | Run all tests |
| **Lint** | `nx run PROJECT:lint` | Check code style |
| **Lint Fix** | `nx run PROJECT:lint-fix` | Auto-fix style issues |
The following values can be used for PROJECT:
- @zitadel/zitadel (root commands)
- @zitadel/api,
- @zitadel/login,
- @zitadel/console,
- @zitadel/docs,
- @zitadel/client
- @zitadel/proto
The project names and folders are streamlined:
| Old Folder | New Folder |
| --- | --- |
| ./e2e | ./tests/functional-ui |
| ./load-test | ./benchmark |
| ./build/zitadel | ./apps/api |
| ./console | ./apps/console (postponed so the PR is reviewable) |
Also, all references to the TypeScript repo are removed so we can
archive it.
# How the Problems Are Solved
- Ran `npx nx@latest init`
- Replaced all turbo.json by project.json and fixed the target configs
- Removed Turbo dependency
- All JavaScript related code affected by a PRs changes is
quality-checked using the `nx affected` command
- We move PR checks that are runnable using Nx into the `check`
workflow. For workflows where we don't use Nx, yet, we restore
previously built dependency artifacts from Nx.
- We only use a single and easy to understand dev container
- The CONTRIBUTING.md is streamlined
- The setup with a generated client pat is orchestrated with Nx
- Everything related to the TypeScript repo is updated or removed. A
**Deploy with Vercel** button is added to the docs and the
CONTRIBUTING.md.
# Additional Changes
- NPM package names have a consistent pattern.
- Docker bake is removed. The login container is built and released like
the core container.
- The integration tests build the login container before running, so
they don't rely on the login container action anymore. This fixes
consistently failing checks on PRs from forks.
- The docs build in GitHub actions is removed, as we already build on
Vercel.
# Additional Context
- Internal discussion:
https://zitadel.slack.com/archives/C087ADF8LRX/p1756277884928169
- Workflow dispatch test:
https://github.com/zitadel/zitadel/actions/runs/17760122959
---------
Co-authored-by: Florian Forster <florian@zitadel.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
# Which Problems Are Solved
Depending on the metadata values (already existing), the newly created
index (#10415) cannot be created or error in the future.
# How the Problems Are Solved
- Create the index using `sha256` and change the query to use sha256 as
well when comparing bytes values such as user_metadata.
- Added a setup step to cleanup potentially created index on
`projections.user_metadata5`
# Additional Changes
None
# Additional Context
- relates to #10415
- requires backport to v4.x
A timing issue (a race condition) was identified in our event processing
system. Under specific circumstances, it was possible for the system to
skip processing certain events, leading to potential data
inconsistencies.
## Which problems are solved
The system tracks its progress through the event log using timestamps.
The issue occurred because we were using the timestamp from the start of
a database transaction. If a query to read new events began after the
transaction started but before the new event was committed, the query
would not see the new event and would fail to process it.
## How the problems are solved
The fix is to change which timestamp is used for tracking. We now use
the precise timestamp of when the event is actually written to the
database. This ensures that the event's timestamp is always correctly
ordered, closing the timing gap and preventing the race condition.
This change enhances the reliability and integrity of our event
processing pipeline. It guarantees that all events are processed in the
correct order and eliminates the risk of skipped events, ensuring data
is always consistent across the system.
## Additional information
original fix: https://github.com/zitadel/zitadel/pull/10560
# Which Problems Are Solved
There was an left-behind index introduced to optimize the old and
removed event execution handler. The index confuses prostgres and it
sometimes picks this index in favor of the projection specific index.
This sometimes leads to bad query performance in the projectio handlers.
# How the Problems Are Solved
Drop the index
# Additional Changes
- none
# Additional Context
- Forgotten in https://github.com/zitadel/zitadel/pull/10564
# Which Problems Are Solved
Using the service ping, we want to have some additional insights to how
zitadel is configured. The current resource count report contains
already some amount of configured policies, such as the login_policy.
But we do not know if for example MFA is enforced.
# How the Problems Are Solved
- Added the following counts to the report:
- service users per organization
- MFA enforcements (though login policy)
- Notification policies with password change option enabled
- SCIM provisioned users (using user metadata)
- Since all of the above are conditional based on at least a column
inside a projection, a new `migration.CountTriggerConditional` has been
added, where a condition (column values) and an option to track updates
on that column should be considered for the count.
- For this to be possible, the following changes had to be made to the
existing sql resources:
- the `resource_name` has been added to unique constraint on the
`projection.resource_counts` table
- triggers have been added / changed to individually track `INSERT`,
`UPDATE`(s) and `DELETE` and be able to handle conditions
- an optional argument has been added to the
`projections.count_resource()` function to allow providing the
information to `UP` or `DOWN` count the resource on an update.
# Additional Changes
None
# Additional Context
- partially solves #10244 (reporting audit log retention limit will be
handled in #10245 directly)
- backport to v4.x
# Which Problems Are Solved
HTTP Request to HTTP providers for Email or SMS are not signed.
# How the Problems Are Solved
Add a Signing Key to the HTTP Provider resources, which is then used to
generate a header to sign the payload.
# Additional Changes
Additional tests for query side of the SMTP provider.
# Additional Context
Closes#10067
---------
Co-authored-by: Marco A. <marco@zitadel.com>
This PR initiates the rework of Zitadel's backend to state-persisted
objects. This change is a step towards a more scalable and maintainable
architecture.
## Changes
* **New `/backend/v3` package**: A new package structure has been
introduced to house the reworked backend logic. This includes:
* `domain`: Contains the core business logic, commands, and repository
interfaces.
* `storage`: Implements the repository interfaces for database
interactions with new transactional tables.
* `telemetry`: Provides logging and tracing capabilities.
* **Transactional Tables**: New database tables have been defined for
`instances`, `instance_domains`, `organizations`, and `org_domains`.
* **Projections**: New projections have been created to populate the new
relational tables from the existing event store, ensuring data
consistency during the migration.
* **Repositories**: New repositories provide an abstraction layer for
accessing and manipulating the data in the new tables.
* **Setup**: A new setup step for `TransactionalTables` has been added
to manage the database migrations for the new tables.
This PR lays the foundation for future work to fully transition to
state-persisted objects for these components, which will improve
performance and simplify data access patterns.
This PR initiates the rework of ZITADEL's backend to state-persisted
objects. This is a foundational step towards a new architecture that
will improve performance and maintainability.
The following objects are migrated from event-sourced aggregates to
state-persisted objects:
* Instances
* incl. Domains
* Orgs
* incl. Domains
The structure of the new backend implementation follows the software
architecture defined in this [wiki
page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel).
This PR includes:
* The initial implementation of the new transactional repositories for
the objects listed above.
* Projections to populate the new relational tables from the existing
event store.
* Adjustments to the build and test process to accommodate the new
backend structure.
This is a work in progress and further changes will be made to complete
the migration.
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Iraq Jaber <iraq+github@zitadel.com>
Co-authored-by: Iraq <66622793+kkrime@users.noreply.github.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
# Which Problems Are Solved
I noticed some outdated / misleading logs when starting zitadel:
- The `init-projections` were no longer in beta for a long time.
- The LRU auth request cache is disabled by default, which results in
the following message, which has caused confusion by customers:
```level=info msg="auth request cache disabled" error="must provide a positive size"```
# How the Problems Are Solved
- Removed the beta info
- Disable cache initialization if possible
# Additional Changes
None
# Additional Context
- noticed internally
- backport to v4.x
# Which Problems Are Solved
When a SAML IDP is created, the signing algorithm defaults to
`RSA-SHA1`.
This PR adds the functionality to configure the signing algorithm while
creating or updating a SAML IDP. When nothing is specified, `RSA-SHA1`
is the default.
Available options:
* RSA_SHA1
* RSA_SHA256
* RSA_SHA512
# How the Problems Are Solved
By introducing a new optional config to specify the Signing Algorithm.
# Additional Changes
N/A
# Additional Context
- Closes#9842
An existing bug in the UpdateSAMLProvider API will be fixed as a
followup in a different
[PR](https://github.com/zitadel/zitadel/pull/10557).
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
…urable
# Which Problems Are Solved
BaseURI defined in environment variables or configuration files was
ignored for Login v2 feature flag.
# How the Problems Are Solved
Define BaseURI as string so that the environment variables and
configuration files can be parsed into it.
# Additional Changes
None
# Additional Context
Closes#10405
# Which Problems Are Solved
Emails are still send only with URLs to login v1.
# How the Problems Are Solved
Add configuration for URLs as URL templates, so that links can point at
Login v2.
# Additional Changes
None
# Additional Context
Closes#10236
---------
Co-authored-by: Marco A. <marco@zitadel.com>
This PR is still WIP and needs changes to at least the tests.
# Which Problems Are Solved
To be able to report analytical / telemetry data from deployed Zitadel
systems back to a central endpoint, we designed a "service ping"
functionality. See also https://github.com/zitadel/zitadel/issues/9706.
This PR adds the first implementation to allow collection base data as
well as report amount of resources such as organizations, users per
organization and more.
# How the Problems Are Solved
- Added a worker to handle the different `ReportType` variations.
- Schedule a periodic job to start a `ServicePingReport`
- Configuration added to allow customization of what data will be
reported
- Setup step to generate and store a `systemID`
# Additional Changes
None
# Additional Context
relates to #9869
# Which Problems Are Solved
We provide a seamless way to initialize Zitadel and the login together.
# How the Problems Are Solved
Additionally to the `IAM_OWNER` role, a set up admin user also gets the
`IAM_LOGIN_CLIENT` role if it is a machine user with a PAT.
# Additional Changes
- Simplifies the load balancing example, as the intermediate
configuration step is not needed anymore.
# Additional Context
- Depends on #10116
- Contributes to https://github.com/zitadel/zitadel-charts/issues/332
- Contributes to https://github.com/zitadel/zitadel/issues/10016
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
Remove the feature flag that allowed triggers in introspection. This
option was a fallback in case introspection would not function properly
without triggers. The API documentation asked for anyone using this flag
to raise an issue. No such issue was received, hence we concluded it is
safe to remove it.
# How the Problems Are Solved
- Remove flags from the system and instance level feature APIs.
- Remove trigger functions that are no longer used
- Adjust tests that used the flag.
# Additional Changes
- none
# Additional Context
- Closes#10026
- Flag was introduced in #7356
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
Stabilize the optimized introspection code and cleanup unused code.
# How the Problems Are Solved
- `oidc_legacy_introspection` feature flag is removed and reserved.
- `OPStorage` which are no longer needed have their bodies removed.
- The method definitions need to remain in place so the interface
remains implemented.
- A panic is thrown in case any such method is still called
# Additional Changes
- A number of `OPStorage` methods related to token creation were already
unused. These are also cleaned up.
# Additional Context
- Closes#10027
- #7822
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
We are preparing to roll-out and stabilize webkeys in the next version
of Zitadel. Before removing legacy signing-key code, we must ensure all
existing instances have their webkeys generated.
# How the Problems Are Solved
Add a setup step which generate 2 webkeys for each existing instance
that didn't have webkeys yet.
# Additional Changes
Return an error from the config type-switch, when the type is unknown.
# Additional Context
- Part 1/2 of https://github.com/zitadel/zitadel/issues/10029
- Should be back-ported to v3
# Which Problems Are Solved
The resource usage to query user(s) on the database was high and
therefore could have performance impact.
# How the Problems Are Solved
Database queries involving the users and loginnames table were improved
and an index was added for user by email query.
# Additional Changes
- spellchecks
- updated apis on load tests
# additional info
needs cherry pick to v3
# Which Problems Are Solved
Add the ability to keep track of the current counts of projection
resources. We want to prevent calling `SELECT COUNT(*)` on tables, as
that forces a full scan and sudden spikes of DB resource uses.
# How the Problems Are Solved
- A resource_counts table is added
- Triggers that increment and decrement the counted values on inserts
and deletes
- Triggers that delete all counts of a table when the source table is
TRUNCATEd. This is not in the business logic, but prevents wrong counts
in case someone want to force a re-projection.
- Triggers that delete all counts if the parent resource is deleted
- Script to pre-populate the resource_counts table when a new source
table is added.
The triggers are reusable for any type of resource, in case we choose to
add more in the future.
Counts are aggregated by a given parent. Currently only `instance` and
`organization` are defined as possible parent. This can later be
extended to other types, such as `project`, should the need arise.
I deliberately chose to use `parent_id` to distinguish from the
de-factor `resource_owner` which is usually an organization ID. For
example:
- For users the parent is an organization and the `parent_id` matches
`resource_owner`.
- For organizations the parent is an instance, but the `resource_owner`
is the `org_id`. In this case the `parent_id` is the `instance_id`.
- Applications would have a similar problem, where the parent is a
project, but the `resource_owner` is the `org_id`
# Additional Context
Closes https://github.com/zitadel/zitadel/issues/9957
# Which Problems Are Solved
Currently if a user signs in using an IdP, once they sign out of
Zitadel, the corresponding IdP session is not terminated. This can be
the desired behavior. In some cases, e.g. when using a shared computer
it results in a potential security risk, since a follower user might be
able to sign in as the previous using the still open IdP session.
# How the Problems Are Solved
- Admins can enabled a federated logout option on SAML IdPs through the
Admin and Management APIs.
- During the termination of a login V1 session using OIDC end_session
endpoint, Zitadel will check if an IdP was used to authenticate that
session.
- In case there was a SAML IdP used with Federated Logout enabled, it
will intercept the logout process, store the information into the shared
cache and redirect to the federated logout endpoint in the V1 login.
- The V1 login federated logout endpoint checks every request on an
existing cache entry. On success it will create a SAML logout request
for the used IdP and either redirect or POST to the configured SLO
endpoint. The cache entry is updated with a `redirected` state.
- A SLO endpoint is added to the `/idp` handlers, which will handle the
SAML logout responses. At the moment it will check again for an existing
federated logout entry (with state `redirected`) in the cache. On
success, the user is redirected to the initially provided
`post_logout_redirect_uri` from the end_session request.
# Additional Changes
None
# Additional Context
- This PR merges the https://github.com/zitadel/zitadel/pull/9841 and
https://github.com/zitadel/zitadel/pull/9854 to main, additionally
updating the docs on Entra ID SAML.
- closes#9228
- backport to 3.x
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
Co-authored-by: Zach Hirschtritt <zachary.hirschtritt@klaviyo.com>
# Which Problems Are Solved
We saw high CPU usage if many events were created on the database. This
was caused by the new actions which query for all event types and
aggregate types.
# How the Problems Are Solved
- the handler of action execution does not filter for aggregate and
event types.
- the index for `instance_id` and `position` is reenabled.
# Additional Changes
none
# Additional Context
none
Mridang AgarwallaandGitHub
