Database Configuration

For configurable Google, Microsoft, Keycloak, and custom OIDC login, see Identity providers and SSO.

This article will detail configuration of the Logship database.

The database is the backbone of any logship deployment. Configuration is specified between the database and backend sections, each providing configurable services with a wide range of capabilities.

You can find an example configuration here or in the logship deployments repository on GitHub.

• Database Configuration
  • Example
  • Production defaults
  • Logging
    • LogLevel
  • agent
  • database
    • endpoints
      • worker
      • master
    • services
      • keyvalue
      • transactionManager
      • transactionLog
        • disk
        • memory
      • gateway
      • partitionManager
  • backend
    • endpoints
      • worker
      • master
    • database
      • worker
      • master
    • services
      • databaseService
        • sqlite
      • schemaManager
      • configService
      • inflowService
      • collectorService
      • searchExecutorService
      • searchQueryService
      • accountsService
        • jwt
        • oauth
          • claimMappings[]
          • providers[]
        • Keycloak SSO example
      • messengerService
      • dashboardService
      • evaluatorManagerService
      • evaluatorRunnerService
      • setupService
        • accounts[]
        • users[]
        • users[].defaultAccounts[]
      • master

Example

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "agent": {
    "udpMetricsEndpoint": "host.docker.internal:49999"
  },
  "database": {
    "enable": true,
    "master": {
      "enable": true
    },
    "endpoints": {
      "worker": ["net.tcp://0.0.0.0:6000"],
      "master": ["net.tcp://127.0.0.1:6000"]
    },
    "services": {
      "keyvalue": {
        "enable": true,
        "dataPath": "/logship/kvs"
      },
      "transactionManager": {
        "enable": true
      },
      "transactionLog": {
        "enable": true,
        "disk": {
          "enable": true,
          "path": "/logship/tlog"
        },
        "memory": {
          "enable": false
        }
      },
      "gateway": {
        "enable": true
      },
      "partitionManager": {
        "enable": true
      }
    }
  },
  "backend": {
    "enable": true,
    "listenPort": 5000,
    "endpoints": {
      "worker": ["net.tcp://0.0.0.0:6002"],
      "master": ["net.tcp://127.0.0.1:6002"]
    },
    "database": {
      "worker": ["net.tcp://127.0.0.1:6000"],
      "master": ["net.tcp://127.0.0.1:6000"]
    },
    "services": {
      "databaseService": {
        "enable": true,
        "sqlite": {
          "enable": true,
          "fileName": "sqlite.db"
        }
      },
      "schemaManager": {
        "enable": true
      },
      "configService": {
        "enable": true
      },
      "inflowService": {
        "enable": true
      },
      "collectorService": {
        "enable": true,
        "tempPath": "/logship/tmp/"
      },
      "searchExecutorService": {
        "enable": true,
        "dataPath": "/logship/searchdata/",
        "minimumFreeSpaceBytes": 5368709120,
        "minimumFreeSpacePercentage": 15.0
      },
      "searchQueryService": {
        "enable": true
      },
      "accountsService": {
        "enable": true,
        "jwt": {
          "enable": true,
          "issuer": "logship-example",
          "audience": "logship-example",
          "signingKey": "this is a secret signing key for locally-signed JWT tokens"
        },
        "oauth": {
          "enable": false
        }
      },
      "messengerService": {
        "enable": true
      },
      "dashboardService": {
        "enable": true
      },
      "evaluatorManagerService": {
        "enable": true
      },
      "evaluatorRunnerService": {
        "enable": true
      },
      "setupService": {
        "enable": true,
        "provision": true,
        "accounts": [
          {
            "accountId": "00000000-0000-0000-0000-000000000000",
            "accountName": "Default Account"
          }
        ],
        "users": [
          {
            "userId": "00000000-0000-0000-0000-00000000cafe",
            "username": "admin",
            "password": "admin",
            "firstname": "Logship",
            "lastname": "Admin",
            "email": "admin@logship.io",
            "defaultGlobalPermissions": ["Logship.Global.Admin"],
            "defaultAccounts": [
              {
                "accountName": "Default Account",
                "userPermissions": ["Logship.Account.Admin"]
              }
            ]
          }
        ]
      }
    },
    "master": {
      "enable": true
    }
  }
}

Config validation can be ignored for services with enable: false.

Production defaults

• Pin explicit image tags (e.g., ghcr.io/logship-io/logship-database:0.8.0) and roll forward deliberately.
• Persist /logship on SSD/NVMe; separate data/log volumes when possible.
• Keep minimumFreeSpaceBytes above 5 GB and minimumFreeSpacePercentage above 15% to avoid throttling.
• Run distinct masters for backend and database on separate nodes when scaling horizontally.
• Secure endpoints with network ACLs and TLS offload (via reverse proxy) where required.

Logging

See .NET Logging Configuration for more information.

LogLevel

agent

database

endpoints

worker

master

services

keyvalue

transactionManager

transactionLog

disk

memory

gateway

partitionManager

backend

endpoints

worker

master

database

worker

master

services

databaseService

Backing metadata store for accounts, dashboards, alerts, and other relational state.

sqlite

When createDirectory is false (the default), the service fails to start if the directory in fileName does not exist, surfacing a misconfigured path instead of silently writing to an unexpected location. Set it to true to have the service create the directory on startup, matching the behavior of collectorService and searchExecutorService.

schemaManager

configService

inflowService

CSV/TSV and negotiated columnar uploads produce immutable typed column batches. Frames cap at 1,024 rows and wide frames split into smaller buffers. A larger individual record occupies its own batch. Parsing overlaps bounded indexing work. Indexed row groups combine at most 8,192 numeric rows or 4,096 rows containing variable fields, with a source-buffer target of at most 8 MiB reduced according to the shared memory limit and upload concurrency. Completed files are checksummed and flushed into the persistent ingestion directory; a background publisher transfers them to executors without reconstructing records over the collection RPC. Small uploads flush their last segment at request completion.

When the shared work budget or ingestion disk capacity is exhausted, uploads stop reading until capacity becomes available. Total input file size is unlimited. Work reservations cover retained input, merged columns, statistics and encoding scratch/dictionaries. Each upload's retained headers, buffered workspace and next reservation must fit maximumBufferedBytes / maximumConcurrentUploads to ensure concurrent uploads can make progress. Batches split and segments flush to respect that share. Scratch spills to the ingestion volume rather than waiting while holding memory. Each admitted request additionally retains at most two bounded decoder staging batches; protocol metadata, stream/compressor state, runtime and catalog caches are outside this budget. A single record whose conservative encoding workspace cannot fit the per-upload share returns HTTP 413 even if its text fits maximumRecordBytes; increase maximumBufferedBytes or reduce maximumConcurrentUploads to increase that share.

Each record must fit the memory budgets of both its inflow service and its collector. Accounting includes payloads and object overhead, so a wide record can require more memory than its UTF-8 representation. These budgets bound pending record buffers, not the total process working set: parsers, RPC messages, schema metadata, indexing and other services require additional memory. Ingestion settings take effect when the service starts.

The indexed-file format also limits an individual encoded field to 65,535 bytes. CSV/TSV decoders report the column and physical record start line when a value exceeds that limit.

Successful CSV/TSV responses acknowledge local disk-flushed acceptance, including a catalog receipt. Pending files survive process/server restart and remain until executor publication is committed. Publication retries reuse the same file identity, and catalog receipts fence replay after publication, compaction, truncation or deletion. This is local durability; loss of the ingestion disk requires a separate replication policy. Query visibility follows publication and can lag acceptance.

Uploads do not provide whole-file transactions or client retry identities. Earlier segments may have been accepted if a later row is malformed or the request is interrupted. Retrying the whole file can duplicate those rows. A concurrent incompatible declaration can reject an unfinished segment before acceptance. JSON and internal-metric uploads retain their existing queue semantics.

Monitor ingestion.memory.retained_bytes and ingestion.memory.wait_ms with stage=inflow, stage=collector or stage=inflow-spool, and metrics.inflow.uploads.active, to distinguish memory/disk backpressure from downstream publication delays. The spool stage measures accounted disk reservations. Corrupt segments are retained and logged for repair. The catalog retains ingestion replay receipts after source reclamation; include this metadata in long-term capacity planning.

Upgrade catalog and executor nodes before inflow nodes. Durable CSV publication uses a distinct internal RPC, so an older executor cannot silently acknowledge an unflushed file. An unavailable compatible executor leaves accepted segments on disk and eventually applies backpressure. Before downgrading inflow, drain its pending segments; older versions cannot replay them.

Collectors request bounded fetches of at most 16,384 rows or approximately 8 MiB of accounted memory per inflow endpoint, and include at most two endpoints in each collection job. Individual RPC messages and writer batches remain bounded by 1,024 rows or approximately 1 MiB. This also bounds data queued by the RPC transport and the size of each temporary file export. Remaining records stay charged to inflow until a later collection pass; collection continues without a polling delay while it makes progress. A record larger than a batch target is fetched alone. Older RPC callers retain their existing drain behavior; upgrade inflow and collector nodes together to apply these bounds throughout the path.

collectorService

searchExecutorService

searchQueryService

accountsService

jwt

oauth

All identity-provider connection details (client id, authority, endpoints, issuers, audiences) live on each entry of providers[] — see providers[] below. The oauth object itself holds only the cross-provider settings:

Logship only enables the OAuth bearer scheme when oauth.enable is true. Each enabled provider is exposed through /auth/oauth, rendered as a separate login option in the frontend, and accepted by the backend bearer auth pipeline.

When a token is accepted, Logship resolves the user from the configured emailClaimType. If the user already exists, Logship signs that user in. If the user does not exist and createDefaultUser is true, Logship creates the user and adds them to defaultUserAccount with a least-privileged Logship.Data.Viewer account permission.

claimMappings[]

Each mapping grants a set of Logship permissions when a token carries a matching claim value. On every login, Logship recomputes the set of granted permissions and syncs the user's global permissions to match: permissions listed in any mapping are considered "managed by SSO" and are added when a rule matches, removed when no rule matches. Permissions not referenced by any mapping are left alone, so permissions assigned through the admin UI are preserved.

"claimMappings": [
  { "claimType": "email",  "values": ["admin@example.com"], "permissions": ["Logship.Global.Admin"] },
  { "claimType": "groups", "values": ["logship-admins"],    "permissions": ["Logship.Global.Admin"] }
]

providers[]

Use providers when you need multiple SSO or need to accept tokens from more than one OIDC identity provider. Claim-to-role mapping remains shared across all providers. Each provider is disabled by default, so set enable to true for every provider you want Logship to expose and accept.

"providers": [
  {
    "enable": true,
    "id": "keycloak-main",
    "displayName": "Keycloak",
    "clientId": "logship-web",
    "scope": "openid profile email",
    "metadataAddress": "https://keycloak.example.com/realms/logship/.well-known/openid-configuration",
    "authority": "https://keycloak.example.com/realms/logship",
    "tokenEndpoint": "https://keycloak.example.com/realms/logship/protocol/openid-connect/token",
    "authorizeEndpoint": "https://keycloak.example.com/realms/logship/protocol/openid-connect/auth",
    "deviceEndpoint": "https://keycloak.example.com/realms/logship/protocol/openid-connect/auth/device",
    "requireHttpsMetadata": true,
    "validIssuers": ["https://keycloak.example.com/realms/logship"],
    "validAudiences": ["logship-web"]
  },
  {
    "enable": true,
    "id": "entra-prod",
    "displayName": "Microsoft Entra ID",
    "clientId": "11111111-2222-3333-4444-555555555555",
    "scope": "openid profile email",
    "metadataAddress": "https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration",
    "authority": "https://login.microsoftonline.com/<tenant-id>/v2.0",
    "tokenEndpoint": "https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token",
    "authorizeEndpoint": "https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/authorize",
    "deviceEndpoint": "https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/devicecode",
    "requireHttpsMetadata": true,
    "validIssuers": ["https://login.microsoftonline.com/<tenant-id>/v2.0"],
    "validAudiences": ["11111111-2222-3333-4444-555555555555"]
  }
]

Keycloak SSO example

{
  "backend": {
    "services": {
      "accountsService": {
        "enable": true,
        "oauth": {
          "enable": true,
          "providers": [
            {
              "enable": true,
              "id": "keycloak-main",
              "displayName": "Keycloak",
              "clientId": "logship-web",
              "scope": "openid profile email",
              "metadataAddress": "https://keycloak.example.com/realms/logship/.well-known/openid-configuration",
              "authority": "https://keycloak.example.com/realms/logship",
              "tokenEndpoint": "https://keycloak.example.com/realms/logship/protocol/openid-connect/token",
              "authorizeEndpoint": "https://keycloak.example.com/realms/logship/protocol/openid-connect/auth",
              "deviceEndpoint": "https://keycloak.example.com/realms/logship/protocol/openid-connect/auth/device",
              "requireHttpsMetadata": true,
              "validIssuers": ["https://keycloak.example.com/realms/logship"],
              "validAudiences": ["logship-web"]
            }
          ],
          "createDefaultUser": true,
          "defaultUserAccount": "11111111-1111-1111-1111-111111111111",
          "userIdClaimType": "sub",
          "firstNameClaimType": "given_name",
          "lastNameClaimType": "family_name",
          "nickNameClaimType": "preferred_username",
          "emailClaimType": "email",
          "claimMappings": [
            {
              "claimType": "email",
              "values": ["admin@example.com"],
              "permissions": ["Logship.Global.Admin"]
            },
            {
              "claimType": "groups",
              "values": ["logship-admins"],
              "permissions": ["Logship.Global.Admin"]
            }
          ]
        }
      }
    }
  }
}

For Keycloak:

1. Create a realm for Logship.
2. Create a client with client id logship-web.
3. Allow the redirect URI used by your Logship frontend, including /login/oauth/redirect.
4. Ensure the issued token includes sub, email, given_name, family_name, and preferred_username claims, or adjust the claim mapping fields above to match your provider.
5. If you want to grant Logship permissions by group membership, add an oidc-group-membership-mapper to the client (token claim name groups, include in access token) so the access token carries a groups claim you can reference in claimMappings.

messengerService

dashboardService

evaluatorManagerService

evaluatorRunnerService

setupService

The setup service can optionally provision accounts and users on startup (useful for local/dev and automation).

accounts[]

users[]

users[].defaultAccounts[]

master