# 📓 January 24, 2026 — Tallinn, Estonia
📓 January 24, 2026 — Tallinn, Estonia
Learning, PKI, Dagster and life signals
Today — another small badge, bigger picture 🧠
Received another certificate from Dagster University — Dagster & ETL ✅
https://courses.dagster.io/certificates/69aog4jfji
At this point Dagster stopped being “just another orchestrator” and started to look like a control plane pattern — especially interesting when viewed through a PKI / governance / audit lens.
Which naturally leads to the next thing brewing 👇
PKI × Dagster — the plan
The idea is simple but powerful:
- EJBCA stays a CA
- SoftHSM stays a crypto backend
- Dagster becomes the PKI control plane
- Policies become data, not shell scripts
- Every action leaves an audit trail
This is not about replacing PKI components —
it’s about orchestrating them safely, transparently and vendor‑independently.
A minimal PoC is coming:
- Dagster orchestrates certificate lifecycle
- EJBCA issues
- SoftHSM stores keys
- Everything observable, replayable and explainable
Full plan below as an attachment ⬇️
Life, buses and perspective 🚍❤️
Katikene rides the bus to our grandchildren twins again.
That quiet reminder: we really need to visit them more often — not “when things calm down”, but now.
Control planes are important, but time is the only non‑renewable resource.
📎 Attachment — PKIC Working Group Proposal (1 page)
PKIC Working Group Proposal
Title
Vendor-Independent PKI Control Plane using Orchestration Layer
Problem Statement
Public Key Infrastructures are increasingly required to support:
- crypto-agility (RSA → ECC → PQC)
- multiple CA and HSM vendors
- strict audit and compliance requirements
Today, PKI implementations are often tightly coupled to specific vendors and operational scripts, making crypto transitions and governance changes costly and risky.
Proposal
This working group proposes defining a vendor-independent orchestration layer for PKI, separating:
- cryptographic implementation (CA, HSM)
- operational workflows
- governance and policy logic
The orchestration layer does not replace CA software, but coordinates PKI lifecycle steps in a transparent, auditable and testable manner.
Architecture Overview
| Layer | Responsibility |
|---|---|
| Cryptography | CA, HSM (EJBCA, others) |
| Orchestration | Workflow engine (e.g. Dagster) |
| Governance | Policies as data |
| Operations | Deployment, monitoring |
| Audit | Execution history & evidence |
Each layer can evolve independently.
Proof of Concept
A minimal PoC demonstrates:
- PKI lifecycle orchestration using Dagster
- certificate issuance via EJBCA
- key management via SoftHSM
- policy-driven crypto selection
- full execution trace for audit
The PoC confirms feasibility without vendor lock-in.
Expected Outcomes
- Reference architecture for vendor-independent PKI orchestration
- Best practices for crypto-agile transitions
- Guidance for audit-ready PKI automation
- Input for future PKIC specifications
Scope (Out of Scope)
- Defining new cryptographic algorithms
- Replacing existing CA products
- Certification or compliance schemes
Conclusion
A vendor-independent PKI control plane enables safer crypto evolution, clearer governance separation and improved auditability, aligning PKI operations with modern infrastructure practices.
🗣 15‑second финал (для VP / PKIC)
“The PoC shows that PKI lifecycle can be orchestrated independently of CA and HSM vendors. Dagster acts as a control plane, giving us crypto‑agility, auditability and governance separation without replacing existing PKI components.”
📎 Attachment — Dagster для PKI‑инфраструктуры и DevOps‑процессов (MS Copilot)
# Dagster для PKI‑инфраструктуры и DevOps‑процессовАрхитектура, модель asset‑ов, expectations (SLA) и примеры пайплайнов для:
- ротации сертификатов;
- контроля CRL/OCSP с алертами.
1. Архитектура Dagster под PKI‑процессы
1.1. Компоненты
Dagster User Code (репозитории)
Содержат:
- PKI‑asset‑ы и ops:
- генерация ключей и CSR;
- запрос сертификатов у CA;
- публикация/проверка CRL и OCSP;
- деплой сертификатов в сервисы (Kubernetes, Nginx, LB и т.п.);
- проверки сроков действия, цепочки доверия, политик.
Dagster Daemon
- Schedules: плановая ротация, периодические проверки CRL/OCSP.
- Sensors: реакция на события (например, скоро истекает сертификат).
Dagster Webserver (UI)
- наблюдаемость (граф asset‑ов, статусы, логи);
- аудит (кто/когда/что запустил);
- ручные триггеры (форс‑ротация, emergency‑обновление).
Инфраструктура и интеграции
- PostgreSQL — метаданные Dagster;
- Vault/HSM — ключи и секреты;
- CA/RA, OCSP, CRL endpoints;
- Kubernetes API, Ansible, REST/gRPC для деплоя;
- централизованный логинг/алертинг (Prometheus/Alertmanager, ELK, etc.).
2. Разложение PKI‑ландшафта на data assets
Ниже — примерная модель asset‑ов, которую можно адаптировать под конкретную инфраструктуру.
2.1. CA и цепочка доверия
-
root_ca_cert
Описание: корневой сертификат CA.
Источник: вручную загружен/зашит, редко меняется. -
intermediate_ca_cert[ca_id]
Описание: промежуточные CA.
Зависит от:root_ca_cert.
Используется для: выпуска конечных сертификатов. -
ca_crl[ca_id]
Описание: актуальный CRL для конкретного CA.
Зависит от:intermediate_ca_cert[ca_id].
Обновляется по расписанию. -
ocsp_responder_status[ca_id]
Описание: состояние OCSP‑респондера для CA.
Используется для SLA‑контроля доступности и корректности ответов.
2.2. Ключи и CSR
-
service_keypair[service_id]
Описание: ключевая пара сервиса (ссылка на ключ в HSM/Vault).
Секретность: приватный ключ не хранится в Dagster, только reference. -
service_csr[service_id, cert_profile]
Описание: CSR для сервиса под конкретный профиль/политику.
Зависит от:service_keypair[service_id].
2.3. Сертификаты и деплой
-
service_cert[service_id, cert_profile]
Описание: выданный сертификат сервиса.
Зависит от:service_csr[service_id, cert_profile],intermediate_ca_cert.
Содержит: not_before, not_after, SAN, EKU, policy OID и т.п. -
deployed_service_cert[service_id, environment]
Описание: факт деплоя сертификата в конкретную среду (prod, test, etc.).
Зависит от:service_cert[service_id, cert_profile], инфраструктуры (K8s, LB).
Содержит: версию, время деплоя, результат health‑check.
2.4. Статусы и мониторинг
-
service_tls_health[service_id, environment]
Описание: результат проверки TLS‑handshake/health‑check.
Зависит от:deployed_service_cert[service_id, environment]. -
pki_sla_snapshot
Описание: агрегированный snapshot состояния PKI (CRL, OCSP, cert lifetime).
**Используется для отчётности и алертинга.
3. Expectations и SLA для PKI‑asset‑ов
Примеры expectations, которые можно реализовать в Dagster (через AssetChecks / custom logic):
3.1. CRL
crl_freshness_ok[ca_id]
Правило:now - this_update <= CRL_MAX_AGE(например, не старше 24 часов).
crl_next_update_reasonable[ca_id]
Правило:next_update - this_update <= CRL_MAX_INTERVAL(например, не более 7 дней).
3.2. OCSP
ocsp_responder_available[ca_id]
Правило:- успешный ответ в течение
OCSP_TIMEOUT(например, 2 секунды).
- успешный ответ в течение
ocsp_status_good[service_id]
Правило:- статус сертификата
GOODдля заданногоservice_cert.
- статус сертификата
3.3. Срок действия сертификатов
cert_lifetime_threshold[service_id]
Правило:not_after - now >= MIN_REMAINING_LIFETIME(например, ≥ 30 дней).
cert_max_lifetime_policy[service_id]
Правило:not_after - not_before <= MAX_ALLOWED_LIFETIME(например, ≤ 398 дней).
3.4. Интеграция с алертингом
При нарушении expectations:
- лог в Dagster + метрика (Prometheus);
- алерт в Alertmanager/почту/чат;
- опционально — автоматический remediate (например, форс‑ротация).
4. Пайплайн ротации сертификатов
Ниже — упрощённый пример job для ротации сертификатов, основанный на сроке действия.
from dagster import asset, job, op, ScheduleDefinition
from datetime import datetime, timedelta
ROTATION_THRESHOLD_DAYS = 30
@asset
def service_cert_metadata() -> dict:
"""
Метаданные по сертификатам сервисов:
{service_id: {"not_after": datetime, "profile": "..."}}
В реальности — чтение из CMDB/PKI-реестра/БД.
"""
return {
"service-a": {"not_after": datetime(2026, 5, 1), "profile": "tls_server"},
"service-b": {"not_after": datetime(2026, 2, 15), "profile": "tls_server"},
}
@op
def select_services_for_rotation(metadata: dict) -> list[dict]:
"""Выбираем сервисы, у которых скоро истекает сертификат."""
now = datetime.utcnow()
to_rotate = []
for service_id, info in metadata.items():
if info["not_after"] - now < timedelta(days=ROTATION_THRESHOLD_DAYS):
to_rotate.append({"service_id": service_id, "profile": info["profile"]})
return to_rotate
@op
def generate_keypair(service_id: str) -> dict:
"""Генерация ключевой пары (в реальности — HSM/Vault)."""
return {"service_id": service_id, "private_key_ref": f"vault://keys/{service_id}"}
@op
def generate_csr(keypair_info: dict, profile: str) -> dict:
"""Генерация CSR на основе ключа и профиля."""
service_id = keypair_info["service_id"]
return {
"service_id": service_id,
"profile": profile,
"csr_pem": f"CSR_FOR_{service_id}_PROFILE_{profile}",
}
@op
def request_certificate_from_ca(csr_info: dict) -> dict:
"""Запрос сертификата у CA."""
service_id = csr_info["service_id"]
csr_pem = csr_info["csr_pem"]
cert_pem = f"CERT_FOR_{service_id}_BASED_ON_{csr_pem}"
# Здесь же можно получить not_before/not_after из ответа CA
return {
"service_id": service_id,
"profile": csr_info["profile"],
"cert_pem": cert_pem,
"not_before": datetime.utcnow(),
"not_after": datetime.utcnow() + timedelta(days=365),
}
@op
def validate_certificate(cert_info: dict) -> dict:
"""Проверка цепочки, EKU, policy, сроков и т.п."""
# Здесь — реальная криптопроверка, проверки policy OID, EKU и т.д.
# При нарушении — raise Exception/Failure
return cert_info
@op
def deploy_certificate(cert_info: dict, environment: str = "prod"):
"""Деплой сертификата в инфраструктуру (K8s secret, Nginx, LB и т.п.)."""
service_id = cert_info["service_id"]
cert_pem = cert_info["cert_pem"]
# Здесь — вызовы k8s API / Ansible / REST к LB
print(f"[{environment}] Deploying cert for {service_id}: {cert_pem}")
@job
def rotate_expiring_certs():
metadata = service_cert_metadata()
services = select_services_for_rotation(metadata)
# В реальности лучше использовать DynamicOut/expand,
# здесь — упрощённый пример.
for item in services:
service_id = item["service_id"]
profile = item["profile"]
keypair = generate_keypair(service_id)
csr = generate_csr(keypair, profile)
cert = request_certificate_from_ca(csr)
validated = validate_certificate(cert)
deploy_certificate(validated)
rotate_certs_daily = ScheduleDefinition(
job=rotate_expiring_certs,
cron_schedule="0 3 * * *", # каждый день в 03:00
)
📎 Attachment — PKI Observability Pattern (ChatGPT)
EJBCA on VM + ELK Stack on k3s
Overview
This document describes a production-style observability pattern where:
- EJBCA (Certificate Authority) runs on a dedicated virtual machine
- ELK stack (Elasticsearch + Kibana) runs inside a lightweight Kubernetes cluster (k3s)
- Logs are shipped securely from the VM into the Kubernetes-based observability platform
This pattern mirrors real enterprise and government PKI deployments.
High-Level Architecture
[ VM ]
EJBCA
|
| audit.log / server.log
| Filebeat / Elastic Agent
↓
-------------------------------
Network
-------------------------------
↓
[ k3s cluster ]
Elasticsearch
↓
Kibana
EJBCA does not depend on Kubernetes and remains fully isolated.
Why This Architecture Is Correct
Security
- CA runs on a dedicated VM (recommended by PKI auditors)
- No Kubernetes access on CA host
- Logs are outbound-only
Operational Separation
- PKI lifecycle ≠ observability lifecycle
- Logging stack can be upgraded independently
- Kubernetes restarts do not affect CA
Audit-Friendly
- Immutable execution history
- Centralized log retention
- Clear evidence trail
This design is widely used in:
- banks
- national PKI
- trust service providers
- regulated environments
Log Shipping Strategy
Recommended approach: Filebeat on VM
Install Filebeat directly on the EJBCA VM.
Why Filebeat:
- Lightweight
- Designed for Java logs
- Supports TLS + client certificates
- Standard enterprise tool
Typical log sources:
/opt/ejbca/logs/server.log
/opt/ejbca/logs/audit.log
Data Flow
EJBCA event
↓
Audit log written
↓
Filebeat detects event
↓
Log shipped via HTTPS
↓
Elasticsearch indexes event
↓
Kibana visualizes timeline
This creates a verifiable PKI execution trail.
Security Enhancements (Recommended)
Mutual TLS
- Filebeat authenticates using a client certificate
- Elasticsearch validates certificate chain
- TLS termination via Ingress or LoadBalancer
Result:
PKI logs are protected using PKI itself
This is often called: “PKI eating its own dog food”
What Can Be Observed in Kibana
Operational Events
- Certificate issuance
- Enrollment requests
- Revocations
- Key generation
- CRL updates
Security Events
- Failed authentication
- Unauthorized admin attempts
- Policy violations
Audit Timeline
- Who did what
- When it happened
- Which CA
- Which profile
- Which key
This is extremely valuable during audits.
Integration with Dagster (Control Plane)
When combined with orchestration:
Dagster
↓
EJBCA REST API
↓
Certificate lifecycle action
↓
EJBCA audit log
↓
ELK stores immutable record
Roles:
- Dagster → PKI control plane
- EJBCA → cryptographic engine
- ELK → audit & evidence plane
Together they form a vendor-independent PKI operating model.
Why This Pattern Matters
- Enables crypto-agility
- Improves audit readiness
- Reduces vendor lock-in
- Aligns PKI with DevOps practices
- Scales from lab → production
Summary
This PoC architecture demonstrates:
- Proper CA isolation
- Modern observability practices
- Enterprise-grade auditability
- Clean separation of concerns
It is suitable for:
- internal demos
- PKIC working groups
- architecture proposals
- security reviews
Status:
✔ Lab-ready
✔ Enterprise-aligned
✔ PKIC-discussable
🧠 Итог — 3 строки
- PKI × Dagster оформился в PKIC Working Group Proposal — control plane, audit trail, vendor independence, всё как у взрослых.
- Катикене едет к внукам-близнецам — тихий reminder: time — единственный non-renewable resource.
- Dagster — не оркестратор, а философия: policies as data, every action auditable, PKI без vendor lock-in.
Related entries (linked topics)
TOPICS.md — index of themes across the diary. Here: PKI × Dagster, vendor-independent control plane, EJBCA, SoftHSM, audit trail → 2026-01-23 (PKIC, Bart Symons), 2026-01-29 (trust, observable), 2026-01-30 (AI+PKI+timestamp), 2026-02-01 (PKI consortium PoC), 2026-02-02 (Aletheia audit trail).
Tags / Metadata
#Dagster #PKI #PKIC #ControlPlane #DevSecOps #LearningDiary #LifeSignals