# 📓 January 24, 2026 — Tallinn, Estonia

📓 January 24, 2026 — Tallinn, Estonia

Learning, PKI, Dagster and life signals

Today — another small badge, bigger picture 🧠

Received another certificate from Dagster UniversityDagster & ETL
https://courses.dagster.io/certificates/69aog4jfji

At this point Dagster stopped being “just another orchestrator” and started to look like a control plane pattern — especially interesting when viewed through a PKI / governance / audit lens.

Which naturally leads to the next thing brewing 👇


PKI × Dagster — the plan

The idea is simple but powerful:

  • EJBCA stays a CA
  • SoftHSM stays a crypto backend
  • Dagster becomes the PKI control plane
  • Policies become data, not shell scripts
  • Every action leaves an audit trail

This is not about replacing PKI components —
it’s about orchestrating them safely, transparently and vendor‑independently.

A minimal PoC is coming:

  • Dagster orchestrates certificate lifecycle
  • EJBCA issues
  • SoftHSM stores keys
  • Everything observable, replayable and explainable

Full plan below as an attachment ⬇️


Life, buses and perspective 🚍❤️

Katikene rides the bus to our grandchildren twins again.
That quiet reminder: we really need to visit them more often — not “when things calm down”, but now.
Control planes are important, but time is the only non‑renewable resource.


📎 Attachment — PKIC Working Group Proposal (1 page)

PKIC Working Group Proposal

Title

Vendor-Independent PKI Control Plane using Orchestration Layer

Problem Statement

Public Key Infrastructures are increasingly required to support:

  • crypto-agility (RSA → ECC → PQC)
  • multiple CA and HSM vendors
  • strict audit and compliance requirements

Today, PKI implementations are often tightly coupled to specific vendors and operational scripts, making crypto transitions and governance changes costly and risky.

Proposal

This working group proposes defining a vendor-independent orchestration layer for PKI, separating:

  • cryptographic implementation (CA, HSM)
  • operational workflows
  • governance and policy logic

The orchestration layer does not replace CA software, but coordinates PKI lifecycle steps in a transparent, auditable and testable manner.

Architecture Overview

LayerResponsibility
CryptographyCA, HSM (EJBCA, others)
OrchestrationWorkflow engine (e.g. Dagster)
GovernancePolicies as data
OperationsDeployment, monitoring
AuditExecution history & evidence

Each layer can evolve independently.

Proof of Concept

A minimal PoC demonstrates:

  • PKI lifecycle orchestration using Dagster
  • certificate issuance via EJBCA
  • key management via SoftHSM
  • policy-driven crypto selection
  • full execution trace for audit

The PoC confirms feasibility without vendor lock-in.

Expected Outcomes

  • Reference architecture for vendor-independent PKI orchestration
  • Best practices for crypto-agile transitions
  • Guidance for audit-ready PKI automation
  • Input for future PKIC specifications

Scope (Out of Scope)

  • Defining new cryptographic algorithms
  • Replacing existing CA products
  • Certification or compliance schemes

Conclusion

A vendor-independent PKI control plane enables safer crypto evolution, clearer governance separation and improved auditability, aligning PKI operations with modern infrastructure practices.

🗣 15‑second финал (для VP / PKIC)

“The PoC shows that PKI lifecycle can be orchestrated independently of CA and HSM vendors. Dagster acts as a control plane, giving us crypto‑agility, auditability and governance separation without replacing existing PKI components.”

📎 Attachment — Dagster для PKI‑инфраструктуры и DevOps‑процессов (MS Copilot) # Dagster для PKI‑инфраструктуры и DevOps‑процессов

Архитектура, модель asset‑ов, expectations (SLA) и примеры пайплайнов для:

  • ротации сертификатов;
  • контроля CRL/OCSP с алертами.

1. Архитектура Dagster под PKI‑процессы

1.1. Компоненты

Dagster User Code (репозитории)
Содержат:

  • PKI‑asset‑ы и ops:
    • генерация ключей и CSR;
    • запрос сертификатов у CA;
    • публикация/проверка CRL и OCSP;
    • деплой сертификатов в сервисы (Kubernetes, Nginx, LB и т.п.);
    • проверки сроков действия, цепочки доверия, политик.

Dagster Daemon

  • Schedules: плановая ротация, периодические проверки CRL/OCSP.
  • Sensors: реакция на события (например, скоро истекает сертификат).

Dagster Webserver (UI)

  • наблюдаемость (граф asset‑ов, статусы, логи);
  • аудит (кто/когда/что запустил);
  • ручные триггеры (форс‑ротация, emergency‑обновление).

Инфраструктура и интеграции

  • PostgreSQL — метаданные Dagster;
  • Vault/HSM — ключи и секреты;
  • CA/RA, OCSP, CRL endpoints;
  • Kubernetes API, Ansible, REST/gRPC для деплоя;
  • централизованный логинг/алертинг (Prometheus/Alertmanager, ELK, etc.).

2. Разложение PKI‑ландшафта на data assets

Ниже — примерная модель asset‑ов, которую можно адаптировать под конкретную инфраструктуру.

2.1. CA и цепочка доверия

  • root_ca_cert
    Описание: корневой сертификат CA.
    Источник: вручную загружен/зашит, редко меняется.

  • intermediate_ca_cert[ca_id]
    Описание: промежуточные CA.
    Зависит от: root_ca_cert.
    Используется для: выпуска конечных сертификатов.

  • ca_crl[ca_id]
    Описание: актуальный CRL для конкретного CA.
    Зависит от: intermediate_ca_cert[ca_id].
    Обновляется по расписанию.

  • ocsp_responder_status[ca_id]
    Описание: состояние OCSP‑респондера для CA.
    Используется для SLA‑контроля доступности и корректности ответов.

2.2. Ключи и CSR

  • service_keypair[service_id]
    Описание: ключевая пара сервиса (ссылка на ключ в HSM/Vault).
    Секретность: приватный ключ не хранится в Dagster, только reference.

  • service_csr[service_id, cert_profile]
    Описание: CSR для сервиса под конкретный профиль/политику.
    Зависит от: service_keypair[service_id].

2.3. Сертификаты и деплой

  • service_cert[service_id, cert_profile]
    Описание: выданный сертификат сервиса.
    Зависит от: service_csr[service_id, cert_profile], intermediate_ca_cert.
    Содержит: not_before, not_after, SAN, EKU, policy OID и т.п.

  • deployed_service_cert[service_id, environment]
    Описание: факт деплоя сертификата в конкретную среду (prod, test, etc.).
    Зависит от: service_cert[service_id, cert_profile], инфраструктуры (K8s, LB).
    Содержит: версию, время деплоя, результат health‑check.

2.4. Статусы и мониторинг

  • service_tls_health[service_id, environment]
    Описание: результат проверки TLS‑handshake/health‑check.
    Зависит от: deployed_service_cert[service_id, environment].

  • pki_sla_snapshot
    Описание: агрегированный snapshot состояния PKI (CRL, OCSP, cert lifetime).
    **Используется для отчётности и алертинга.


3. Expectations и SLA для PKI‑asset‑ов

Примеры expectations, которые можно реализовать в Dagster (через AssetChecks / custom logic):

3.1. CRL

  • crl_freshness_ok[ca_id]
    Правило:
    • now - this_update <= CRL_MAX_AGE (например, не старше 24 часов).
  • crl_next_update_reasonable[ca_id]
    Правило:
    • next_update - this_update <= CRL_MAX_INTERVAL (например, не более 7 дней).

3.2. OCSP

  • ocsp_responder_available[ca_id]
    Правило:
    • успешный ответ в течение OCSP_TIMEOUT (например, 2 секунды).
  • ocsp_status_good[service_id]
    Правило:
    • статус сертификата GOOD для заданного service_cert.

3.3. Срок действия сертификатов

  • cert_lifetime_threshold[service_id]
    Правило:
    • not_after - now >= MIN_REMAINING_LIFETIME (например, ≥ 30 дней).
  • cert_max_lifetime_policy[service_id]
    Правило:
    • not_after - not_before <= MAX_ALLOWED_LIFETIME (например, ≤ 398 дней).

3.4. Интеграция с алертингом

При нарушении expectations:

  • лог в Dagster + метрика (Prometheus);
  • алерт в Alertmanager/почту/чат;
  • опционально — автоматический remediate (например, форс‑ротация).

4. Пайплайн ротации сертификатов

Ниже — упрощённый пример job для ротации сертификатов, основанный на сроке действия.

from dagster import asset, job, op, ScheduleDefinition
from datetime import datetime, timedelta

ROTATION_THRESHOLD_DAYS = 30


@asset
def service_cert_metadata() -> dict:
    """
    Метаданные по сертификатам сервисов:
    {service_id: {"not_after": datetime, "profile": "..."}}
    В реальности — чтение из CMDB/PKI-реестра/БД.
    """
    return {
        "service-a": {"not_after": datetime(2026, 5, 1), "profile": "tls_server"},
        "service-b": {"not_after": datetime(2026, 2, 15), "profile": "tls_server"},
    }


@op
def select_services_for_rotation(metadata: dict) -> list[dict]:
    """Выбираем сервисы, у которых скоро истекает сертификат."""
    now = datetime.utcnow()
    to_rotate = []
    for service_id, info in metadata.items():
        if info["not_after"] - now < timedelta(days=ROTATION_THRESHOLD_DAYS):
            to_rotate.append({"service_id": service_id, "profile": info["profile"]})
    return to_rotate


@op
def generate_keypair(service_id: str) -> dict:
    """Генерация ключевой пары (в реальности — HSM/Vault)."""
    return {"service_id": service_id, "private_key_ref": f"vault://keys/{service_id}"}


@op
def generate_csr(keypair_info: dict, profile: str) -> dict:
    """Генерация CSR на основе ключа и профиля."""
    service_id = keypair_info["service_id"]
    return {
        "service_id": service_id,
        "profile": profile,
        "csr_pem": f"CSR_FOR_{service_id}_PROFILE_{profile}",
    }


@op
def request_certificate_from_ca(csr_info: dict) -> dict:
    """Запрос сертификата у CA."""
    service_id = csr_info["service_id"]
    csr_pem = csr_info["csr_pem"]
    cert_pem = f"CERT_FOR_{service_id}_BASED_ON_{csr_pem}"
    # Здесь же можно получить not_before/not_after из ответа CA
    return {
        "service_id": service_id,
        "profile": csr_info["profile"],
        "cert_pem": cert_pem,
        "not_before": datetime.utcnow(),
        "not_after": datetime.utcnow() + timedelta(days=365),
    }


@op
def validate_certificate(cert_info: dict) -> dict:
    """Проверка цепочки, EKU, policy, сроков и т.п."""
    # Здесь — реальная криптопроверка, проверки policy OID, EKU и т.д.
    # При нарушении — raise Exception/Failure
    return cert_info


@op
def deploy_certificate(cert_info: dict, environment: str = "prod"):
    """Деплой сертификата в инфраструктуру (K8s secret, Nginx, LB и т.п.)."""
    service_id = cert_info["service_id"]
    cert_pem = cert_info["cert_pem"]
    # Здесь — вызовы k8s API / Ansible / REST к LB
    print(f"[{environment}] Deploying cert for {service_id}: {cert_pem}")


@job
def rotate_expiring_certs():
    metadata = service_cert_metadata()
    services = select_services_for_rotation(metadata)

    # В реальности лучше использовать DynamicOut/expand,
    # здесь — упрощённый пример.
    for item in services:
        service_id = item["service_id"]
        profile = item["profile"]
        keypair = generate_keypair(service_id)
        csr = generate_csr(keypair, profile)
        cert = request_certificate_from_ca(csr)
        validated = validate_certificate(cert)
        deploy_certificate(validated)


rotate_certs_daily = ScheduleDefinition(
    job=rotate_expiring_certs,
    cron_schedule="0 3 * * *",  # каждый день в 03:00
)
📎 Attachment — PKI Observability Pattern (ChatGPT)

EJBCA on VM + ELK Stack on k3s

Overview

This document describes a production-style observability pattern where:

  • EJBCA (Certificate Authority) runs on a dedicated virtual machine
  • ELK stack (Elasticsearch + Kibana) runs inside a lightweight Kubernetes cluster (k3s)
  • Logs are shipped securely from the VM into the Kubernetes-based observability platform

This pattern mirrors real enterprise and government PKI deployments.


High-Level Architecture

[ VM ]
EJBCA
  |
  | audit.log / server.log
  | Filebeat / Elastic Agent

-------------------------------
        Network
-------------------------------

[ k3s cluster ]
Elasticsearch

Kibana

EJBCA does not depend on Kubernetes and remains fully isolated.


Why This Architecture Is Correct

Security

  • CA runs on a dedicated VM (recommended by PKI auditors)
  • No Kubernetes access on CA host
  • Logs are outbound-only

Operational Separation

  • PKI lifecycle ≠ observability lifecycle
  • Logging stack can be upgraded independently
  • Kubernetes restarts do not affect CA

Audit-Friendly

  • Immutable execution history
  • Centralized log retention
  • Clear evidence trail

This design is widely used in:

  • banks
  • national PKI
  • trust service providers
  • regulated environments

Log Shipping Strategy

Install Filebeat directly on the EJBCA VM.

Why Filebeat:

  • Lightweight
  • Designed for Java logs
  • Supports TLS + client certificates
  • Standard enterprise tool

Typical log sources:

/opt/ejbca/logs/server.log
/opt/ejbca/logs/audit.log

Data Flow

EJBCA event

Audit log written

Filebeat detects event

Log shipped via HTTPS

Elasticsearch indexes event

Kibana visualizes timeline

This creates a verifiable PKI execution trail.


Mutual TLS

  • Filebeat authenticates using a client certificate
  • Elasticsearch validates certificate chain
  • TLS termination via Ingress or LoadBalancer

Result:

PKI logs are protected using PKI itself

This is often called: “PKI eating its own dog food”


What Can Be Observed in Kibana

Operational Events

  • Certificate issuance
  • Enrollment requests
  • Revocations
  • Key generation
  • CRL updates

Security Events

  • Failed authentication
  • Unauthorized admin attempts
  • Policy violations

Audit Timeline

  • Who did what
  • When it happened
  • Which CA
  • Which profile
  • Which key

This is extremely valuable during audits.


Integration with Dagster (Control Plane)

When combined with orchestration:

Dagster

EJBCA REST API

Certificate lifecycle action

EJBCA audit log

ELK stores immutable record

Roles:

  • Dagster → PKI control plane
  • EJBCA → cryptographic engine
  • ELK → audit & evidence plane

Together they form a vendor-independent PKI operating model.


Why This Pattern Matters

  • Enables crypto-agility
  • Improves audit readiness
  • Reduces vendor lock-in
  • Aligns PKI with DevOps practices
  • Scales from lab → production

Summary

This PoC architecture demonstrates:

  • Proper CA isolation
  • Modern observability practices
  • Enterprise-grade auditability
  • Clean separation of concerns

It is suitable for:

  • internal demos
  • PKIC working groups
  • architecture proposals
  • security reviews

Status:
✔ Lab-ready
✔ Enterprise-aligned
✔ PKIC-discussable


🧠 Итог — 3 строки

  1. PKI × Dagster оформился в PKIC Working Group Proposal — control plane, audit trail, vendor independence, всё как у взрослых.
  2. Катикене едет к внукам-близнецам — тихий reminder: time — единственный non-renewable resource.
  3. Dagster — не оркестратор, а философия: policies as data, every action auditable, PKI без vendor lock-in.

TOPICS.md — index of themes across the diary. Here: PKI × Dagster, vendor-independent control plane, EJBCA, SoftHSM, audit trail → 2026-01-23 (PKIC, Bart Symons), 2026-01-29 (trust, observable), 2026-01-30 (AI+PKI+timestamp), 2026-02-01 (PKI consortium PoC), 2026-02-02 (Aletheia audit trail).

Tags / Metadata

#Dagster #PKI #PKIC #ControlPlane #DevSecOps #LearningDiary #LifeSignals