Kafka integration

ligo-gracedb includes a Kafka producer and consumer that authenticate with SciTokens over the OAUTHBEARER SASL mechanism. These classes are designed for long-running analysis pipelines that need to publish or consume GraceDB events in real time.

Note

The Kafka classes require the confluent-kafka C library and jsonschema (messages are validated before they are produced). Neither is installed with the base package; install the optional dependencies with:

pip install ligo-gracedb[kafka]

With conda, install python-confluent-kafka and jsonschema alongside ligo-gracedb. A producer constructed without them raises ImportError naming the missing package.

Authentication

Both the producer and the consumer discover a SciToken automatically using igwn-auth-utils. The token is located following the WLCG Bearer Token Discovery specification:

  1. $BEARER_TOKEN environment variable (token content)

  2. $BEARER_TOKEN_FILE environment variable (path to file)

  3. $XDG_RUNTIME_DIR/bt_u<uid> or /tmp/bt_u<uid>

If igwn-auth-utils cannot match the required audience or scope, a direct file read of the above paths is attempted as a fallback.

You can obtain a fresh SciToken with htgettoken:

htgettoken -a vault.ligo.org -i igwn

By default, both classes set reload_cred=True, which installs a SciTokenRefreshManager that monitors the token’s exp claim and re-reads the token from disk before it expires. This mirrors the reload_cred behaviour of the REST client and is essential for long-running processes.

Note

In addition to having a valid, non-expired SciToken, a user must have the gracedb.read scope in order to publish to Kafka. On event submission, GraceDB’s existing per-user/per-pipeline permission model will control which users can publish to individual pipelines.

Once changes in GraceDB scopes are implemented, access to pipeline-specific topics will be controlled on a per-user basis.

Write access to the test topic will remain available to all LVK users with the gracedb.read scope.

Upgrading from 2.15

Version 2.16 made the producer fail fast and report causes. Code written against 2.15 keeps working in the common case, but four behaviours changed:

Exceptions. create_event(..., kafka=producer) raises the KafkaError subclasses described under Error handling instead of RuntimeError. Catch KafkaError (or ligo.gracedb.kafka.exceptions.KafkaError) where you caught RuntimeError before. KafkaError does not inherit from RuntimeError.

Construction contacts the broker. By default the constructor looks up the SciToken (validate_token=True) and asks the broker for its topics (check_for_topics=True), so a missing token, an unreachable broker or a wrong instance_name now raises at construction with a message that says what to fix. A producer that must start before its broker or token is available can pass check_for_topics=False and validate_token=False to get the 2.15 behaviour of constructing silently and failing on the first flush.

Default token scope. token_scope defaults to gracedb.read, the scope the GraceDB brokers actually require, instead of the non-existent kafka.produce. Producers that passed token_scope explicitly are unaffected; producers that relied on the default now find a usable token where they previously did not.

Message contents. Outgoing messages no longer include content_id (it is still sent with the graceid reservation and with HTTP uploads), and every message is validated against the Message schema before it is produced. Downstream consumers that read content_id from Kafka messages should take it from the reservation instead.

Topic namespacing

Kafka topics are instance-namespaced: each topic is prefixed with the GraceDB instance name so that multiple GraceDB deployments (development, test, playground, production) can share the same broker without message collisions. The format is <instance_name>.<pipeline>, for example:

  • gracedb-dev.gstlal

  • gracedb-test.cwb

  • gracedb.aframe

The instance name is derived automatically from the GraceDB service URL (using the first component of the hostname). For example, https://gracedb-dev.ligo.org/api/ yields instance name gracedb-dev.

Both the producer and consumer handle namespacing transparently: you pass pipeline names (e.g. "gstlal") and the classes prepend the instance name for you. You can supply the instance identity in two ways:

  • service_url — the GraceDB service URL; the instance name is extracted automatically.

  • instance_name — an explicit instance name string.

Exactly one of these must be provided.

Three utility functions are available for working with namespaced topics directly:

from ligo.gracedb.kafka import (
    derive_instance_name,
    make_topic_name,
    parse_topic_name,
)

derive_instance_name("https://gracedb-dev.ligo.org/api/")
# => "gracedb-dev"

make_topic_name("gracedb-dev", "gstlal")
# => "gracedb-dev.gstlal"

parse_topic_name("gracedb-dev.gstlal")
# => ("gracedb-dev", "gstlal")

GraceDB Kafka brokers

Two brokers, hosted at Caltech, serve the GraceDB instances. Pass the one matching your service_url as bootstrap_servers:

Broker

GraceDB instances served

kafkagracedb1.igwn.org:9092

gracedb.ligo.org (production)

kafkagracedbtest1.igwn.org:9092

gracedb-dev.ligo.org, gracedb-test.ligo.org, gracedb-playground.ligo.org, gracedb-cit1.ligo.org

Each broker only carries topics for the instances it serves, so a producer pointed at the wrong broker for its service_url fails at construction with a KafkaTopicError that lists the instances the broker does have (see Startup checks).

Both brokers present certificates from a public CA, so ca_cert_path=certifi.where() is sufficient; no custom CA bundle is needed.

The brokers run the container from the scitokens-kafka-broker repository. If you have trouble reaching a broker, or believe a topic or permission is missing, please open a ticket on the IGWN Computing Help Desk.

Notes on Downstream Consumers

Topics on the GraceDB event submission Kafka broker correspond to approved and testing pipelines on GraceDB. A test topic is available to submit events to the Test group. Please see the above note regarding topic permissions.

Uploading a GW trigger via Kafka does not directly insert a new event into GraceDB, as was the use case of the create_event REST API call. Rather, publishing an event to Kafka allows the simultaneous processing of GW data products by multiple downstream processes.

Ahead of IR1, a dedicated service called the gracedb-kafka-supervisor is running in AWS alongside GraceDB. This application consumes Kafka messages and injects event triggers and data products into GraceDB. Please see the section below on Skipping server-side processing to control how events are handled by the gracedb-kafka-supervisor.

As the IR1 and beyond low-latency alert infrastructure evolves, follow-up pipelines may opt to delay uploading g-events until after time-critical periods of the alert generation process.

Producer

A GraceDbKafkaProducer wraps a confluent_kafka.Producer with SciToken authentication and sensible defaults for GraceDB usage.

Creating a producer

import certifi
from ligo.gracedb.kafka import GraceDbKafkaProducer

producer = GraceDbKafkaProducer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topic="test",
    service_url="https://gracedb-dev.ligo.org/api/",
    ca_cert_path=certifi.where(),
)

The topic argument is the pipeline name (e.g. "test", "gstlal"). It is automatically namespaced with the instance name derived from service_url, so the actual Kafka topic above would be gracedb-dev.test. See Topic namespacing for details.

During development, when the broker’s TLS certificate does not match its hostname, pass skip_hostname_verification=True.

Startup checks

Constructing a producer is not free: by default it finds your SciToken and contacts the broker, so that a misconfiguration is reported immediately, in a sentence, rather than as a silent timeout on the first upload. Two keyword arguments control this:

Flag

Default

What it does

validate_token

True

Looks up the SciToken once and raises KafkaTokenError if none is found or the one found has expired. No network access is needed. Set to False if the token will only be written to disk after the producer is created. See Failing fast on token problems.

check_for_topics

True

Fetches the broker’s topic list (10 s timeout) and refuses to instantiate if the topics this producer needs are not there. With a default topic, that topic must exist and the token must be authorized for it. Without one, the broker must have at least one topic for the instance. An unreachable broker raises KafkaConnectionError. Set to False for a producer that must start before its broker is up. See Missing or unauthorized topics.

With both checks on, the constructor can raise:

  • KafkaTokenError — no usable SciToken (message says where it looked and how to get one);

  • KafkaConnectionError — the broker could not be reached, or confluent-kafka refused the configuration;

  • KafkaAuthenticationError — the broker rejected the token;

  • KafkaTopicError — the default topic is missing or unauthorized, or the instance has no topics on this broker (message lists the instances that do);

  • ValueError — neither instance_name nor service_url was given;

  • ImportError — confluent-kafka is not installed.

All of the Kafka exceptions share the KafkaError base class, so a pipeline that wants a single handler can catch that:

from ligo.gracedb.kafka import GraceDbKafkaProducer, KafkaError

try:
    producer = GraceDbKafkaProducer(
        bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
        service_url="https://gracedb-test.ligo.org/api/",
        ca_cert_path=certifi.where(),
    )
except KafkaError as exc:
    sys.exit(f"Cannot start Kafka producer: {exc}")

A producer created with check_for_topics=False is still protected later: ping() checks its topic before sending, and a stalled strict flush asks the broker about the queued topics.

Producing messages

The produce() method accepts a dict, str, or bytes value. Dicts are serialised to JSON automatically:

producer.produce({"graceid": "G12345", "group": "CBC"})
producer.flush(strict=True)   # wait for delivery; raise if it fails
producer.close()              # flush remaining messages and release resources

produce() only queues the message; delivery happens in the background. flush() waits for the queue to drain and, like the underlying confluent-kafka call, returns the number of messages still undelivered when the timeout expires. Pass strict=True to have it raise instead, with an explanation of what went wrong (see Diagnosing undelivered messages).

The producer also works as a context manager:

with GraceDbKafkaProducer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topic="test",
    service_url="https://gracedb-dev.ligo.org/api/",
    ca_cert_path=certifi.where(),
) as producer:
    producer.produce({"graceid": "G12345", "group": "CBC"})

Testing connectivity

Call ping() to verify end-to-end connectivity. This checks that the topic exists, produces a small test message, waits for delivery confirmation, and returns the delivery metadata:

result = producer.ping(topic="test")
print(result)
# {'topic': 'gracedb-dev.test', 'partition': 0, 'offset': 42,
#  'message': {'type': 'ping', 'sub': 'albert.einstein@ligo.org', ...}}

The topic argument is namespaced like everywhere else; with no argument, ping() uses the producer’s default topic, or <instance>.test if none was set. On failure it raises KafkaTopicError (topic missing or unauthorized), KafkaConnectionError (broker unreachable), KafkaAuthenticationError (token rejected), or KafkaMessageDeliveryError (anything else), each with a message that says what to check.

Uploading events with Kafka

The preferred way to upload events through Kafka is the two-step reserve + produce workflow provided by create_event(). Passing a producer to its kafka parameter causes create_event to:

  1. Reserve a graceid on the GraceDB server via an HTTP call to the reservations endpoint. This returns a graceid and submitter before any event data is transmitted.

  2. Produce a Kafka message containing the reserved graceid, the base64-encoded event file, pipeline metadata, and optional labels. The GraceDB server-side consumer receives this message and completes the upload.

This approach decouples the latency-sensitive part of event submission (obtaining a graceid) from the bulk data transfer, and provides an optional HTTP fallback if Kafka delivery fails.

Note

GraceDB will not dispatch an igwn-alert when a graceid is reserved. Once a reserved event is uploaded, then GraceDB will dispatch a “new”-type igwn-alert.

Basic example

import certifi
from ligo.gracedb.rest import GraceDb
from ligo.gracedb.kafka import GraceDbKafkaProducer

client = GraceDb('https://gracedb-test.ligo.org/api/', api_version='v2')

producer = GraceDbKafkaProducer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    service_url="https://gracedb-test.ligo.org/api/",
    ca_cert_path=certifi.where(),
)

response = client.create_event(
    "CBC", "gstlal", "coinc.xml",
    search="AllSky",
    kafka=producer,
)
# Message is published to gracedb-test.gstlal
print(response.json()["graceid"])   # e.g. "G123456"

producer.close()

In the above example, by not specifying topic in the GraceDbKafkaProducer, the producer publishes to a topic that is determined by the group and pipeline arguments passed to create_event. Events in the Test group are routed to {instance}.test; all other groups are routed to {instance}.{pipeline} (lowercased).

create_event automatically flushes the producer after publishing, so the message is guaranteed to be delivered (or an error raised) before the call returns. The flush timeout defaults to 10 seconds and can be configured via the flush_timeout parameter on the producer:

producer = GraceDbKafkaProducer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    service_url="https://gracedb-test.ligo.org/api/",
    ca_cert_path=certifi.where(),
    flush_timeout=30,  # wait up to 30s for delivery
)

Using the convenience factory

The create_kafka_producer() method on the client creates a pre-configured producer. The client’s service URL is passed automatically, so topics are namespaced without any extra arguments:

client = GraceDb('https://gracedb-test.ligo.org/api/', api_version='v2')
producer = client.create_kafka_producer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topic="test",
    ca_cert_path=certifi.where(),
)

HTTP fallback

If http_fallback=True and the Kafka path fails (reservation failure or produce failure), create_event automatically falls back to the standard HTTP POST:

response = client.create_event(
    "CBC", "gstlal", "coinc.xml",
    kafka=producer,
    http_fallback=True,
)

When a graceid has already been reserved but the Kafka produce fails, the fallback uses upload_reserved_event() to complete the upload over HTTP. A RuntimeWarning is emitted that names the Kafka error that triggered the fallback.

With the default http_fallback=False, a Kafka delivery failure raises the corresponding KafkaError subclass (see Error handling); the graceid stays reserved and can be completed later with upload_reserved_event. A failed reservation raises HTTPError.

Skipping server-side processing

Two boolean flags control what the server does with the Kafka message:

  • gracedb_upload=False — publish the event to Kafka without reserving a graceid or processing it on the server. Useful when a downstream consumer other than GraceDB handles the message.

  • skip_supervisor=True — the server-side supervisor will not process the message (i.e. will not call upload_reserved_event). Default False.

Consumer

A GraceDbKafkaConsumer wraps a confluent_kafka.Consumer with SciToken authentication.

Note

The consumer does not perform the producer’s startup checks. Subscribing to a topic that does not exist is not an error in Kafka; the consumer simply receives nothing. Double-check instance_name and the pipeline name if a stream stays silent.

Live streaming

By default the consumer generates an ephemeral group ID, reads from the end of the topic (auto.offset.reset=latest), and auto-commits offsets. This means it only sees messages produced after it starts:

import certifi
from ligo.gracedb.kafka import GraceDbKafkaConsumer

with GraceDbKafkaConsumer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topics="test",
    service_url="https://gracedb-dev.ligo.org/api/",
    ca_cert_path=certifi.where(),
) as consumer:
    for msg in consumer.stream():
        print(msg["value"])

Pressing Ctrl-C stops the stream cleanly; the consumer installs SIGINT and SIGTERM handlers that cause the current poll to finish and the generator to exit.

Persisting read position

Supply a stable group_id to resume from the last committed offset across restarts:

import certifi

with GraceDbKafkaConsumer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topics="test",
    service_url="https://gracedb-dev.ligo.org/api/",
    group_id="my-pipeline-consumer",
    ca_cert_path=certifi.where(),
) as consumer:
    for msg in consumer.stream():
        process(msg)

Replaying history

Combine a fresh group_id with auto_offset_reset="earliest" to replay all messages from the beginning of the topic:

import certifi

with GraceDbKafkaConsumer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topics="test",
    service_url="https://gracedb-dev.ligo.org/api/",
    group_id="my-replay-group",
    auto_offset_reset="earliest",
    ca_cert_path=certifi.where(),
) as consumer:
    for msg in consumer.stream():
        print(msg)

Polling manually

If the stream() generator is too opinionated, call poll() directly:

import certifi

consumer = GraceDbKafkaConsumer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    service_url="https://gracedb-dev.ligo.org/api/",
    ca_cert_path=certifi.where(),
)
consumer.subscribe("test")

while True:
    msg = consumer.poll(timeout=1.0)
    if msg is not None:
        handle(msg)

Token reloading

Long-running producers and consumers must cope with SciToken expiry. By default both classes set reload_cred=True, which creates a SciTokenRefreshManager. The manager parses the JWT exp claim and re-discovers a fresh token reload_buffer seconds before expiry (default: 300 s).

If an external process (e.g. htgettoken in a cron job) periodically writes a fresh token to disk, the refresh manager will pick it up transparently. The manager cannot mint tokens itself, so make sure the external job replaces the file before the old token expires; with htgettoken that means setting --minsecs comfortably above the cron interval (e.g. --minsecs 4000 for an hourly job; 3600 is not enough). A job that only renews at expiry produces a short burst of SASL errors every few hours; see SASL errors every time the token is renewed.

To disable proactive reloading:

producer = GraceDbKafkaProducer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    service_url="https://gracedb-dev.ligo.org/api/",
    reload_cred=False,
)

Error handling

All Kafka-specific exceptions inherit from KafkaError:

KafkaError
+-- KafkaAuthenticationError
|   +-- KafkaTokenError
+-- KafkaProducerError
|   +-- KafkaMessageDeliveryError
|       +-- KafkaTopicError
+-- KafkaConsumerError
+-- KafkaConnectionError

Exception

Raised when

What to check

KafkaTokenError

No SciToken could be found, or the only one found has expired. Raised by the constructor (validate_token), by ping(), and inside the OAuth callback on token refresh.

Run htgettoken -a vault.ligo.org -i igwn; the message lists the locations searched.

KafkaAuthenticationError

The broker rejected the token (wrong audience or scope, or it expired while the producer was running). Raised by the next produce(), flush(), or ping() after the failure.

Token scope and audience; whether the token was refreshed on disk in time.

KafkaConnectionError

The broker cannot be reached: DNS failure, connection refused, TLS handshake failure, or the topic list could not be fetched at construction. Raised by the constructor (check_for_topics), a strict flush(), ping(), and create_event.

bootstrap_servers, network access, ca_cert_path, use_ssl; the message carries librdkafka’s own description.

KafkaTopicError

The topic does not exist on the broker (automatic creation is disabled) or the token is not authorized for it, or, at construction without a default topic, the instance has no topics at all. Raised by the constructor, ping(), a strict flush(), and create_event.

instance_name/service_url and the pipeline name; the message lists the instances present on the broker.

KafkaMessageDeliveryError

The broker rejected a message for a reason other than the topic, or messages were still queued when a strict flush() timed out with no connection or topic error to explain it.

flush_timeout; broker health. The message includes the last error librdkafka reported, if any.

KafkaProducerError

confluent-kafka itself raised while producing or flushing (for example a message larger than message.max.bytes).

The wrapped KafkaException in the message.

KafkaConsumerError

A consumer-side error from subscribe() or poll().

The wrapped error in the message.

Authentication failures that occur inside librdkafka (for example, an expired token that was not refreshed in time) are surfaced as KafkaAuthenticationError on the next produce(), flush(), or poll() call.

Failing fast on token problems

The producer looks up the SciToken once when it is constructed (validate_token=True, the default). A missing or expired token therefore raises KafkaTokenError immediately, with a message that lists every location that was searched and the htgettoken command that fixes it:

KafkaTokenError: No valid SciToken found for Kafka broker
'kafkagracedbtest1.igwn.org' (scope 'gracedb.read'). Searched:
$BEARER_TOKEN (unset); $BEARER_TOKEN_FILE (unset);
/tmp/bt_u1000 (not found). Obtain a fresh token with:
htgettoken -a vault.ligo.org -i igwn

KafkaTokenError: SciToken from /tmp/bt_u1000 expired at
2026-09-25 14:59:08 UTC. Obtain a fresh token with:
htgettoken -a vault.ligo.org -i igwn

Without this check, confluent-kafka would swallow the error, wait ten seconds, and report a generic “OAuth token not set” failure. Pass validate_token=False if the token will only be written to disk after the producer is created.

Diagnosing undelivered messages

An unreachable broker does not raise on produce(): librdkafka queues the message and retries in the background, so the first sign of trouble is a flush() that returns with messages still queued. The producer remembers the last error librdkafka reported (DNS failure, connection refused, TLS handshake failure, and so on) and uses it to explain the stall.

create_event() always flushes with strict=True, so a delivery problem raises rather than returning a count:

KafkaConnectionError: Delivery timed out after 10s with 1 message(s)
still queued for Kafka broker kafkagracedbtest1.igwn.org:9092. Last
error: sasl_ssl://kafkagracedbtest1.igwn.org:9092/bootstrap: Failed
to resolve 'kafkagracedbtest1.igwn.org:9092': nodename nor servname
provided, or not known [_RESOLVE]. Check bootstrap_servers and DNS.

When calling the producer directly, opt in the same way:

producer.produce({"graceid": "G12345"})
producer.flush(timeout=10, strict=True)

With strict=True, flush() raises KafkaConnectionError when the client reported a connection-level error, KafkaTopicError when the topic is the problem (see below), and KafkaMessageDeliveryError when the broker rejected a message or messages remain queued for any other reason. With the default strict=False, flush() keeps the plain confluent-kafka behaviour of returning the number of undelivered messages and logs a warning with that count and the last error librdkafka reported. Only a strict flush asks the broker about the queued topics (see below), since that costs a metadata request per topic.

Missing or unauthorized topics

Automatic topic creation is disabled on the GraceDB brokers, so a message for a topic that does not exist (typically a wrong instance_name or pipeline) is held by librdkafka until its delivery timeout without any error being reported. The producer guards against this in three places:

  • At construction (check_for_topics=True, the default), the producer fetches the broker’s topic list and refuses to instantiate if what it needs is not there. With a default topic, that topic must exist and the token must be authorized for it. Without one, the broker must have at least one topic for the instance; the error otherwise names the instances that do have topics, so a wrong instance_name/service_url is caught before any graceid is reserved. An unreachable broker raises KafkaConnectionError at this point. Pass check_for_topics=False for a producer that must start before its broker is up.

  • On ping(), the topic is checked before the test message is sent.

  • When a strict flush stalls with no connection error recorded, the producer asks the broker about every topic that still has messages queued and raises KafkaTopicError if one is the cause. This covers create_event, where the topic is chosen per call.

At construction, without a default topic:

KafkaTopicError: No topics found for GraceDB instance 'gracedb' on
Kafka broker kafkagracedbtest1.igwn.org:9092. Instances with topics
on this broker: gracedb-dev, gracedb-test. Check
instance_name/service_url, or pass check_for_topics=False to skip
this check.

From ping() or a stalled flush, where the topic is known:

KafkaTopicError: Topic 'gracedb.test' does not exist on Kafka broker
kafkagracedbtest1.igwn.org:9092 (automatic topic creation is
disabled). Topics are named '<instance>.<pipeline>'; check
instance_name/service_url and the pipeline name.

Message schema

Kafka messages produced by ligo-gracedb conform to JSON Schemas that ship with the package. The schemas define the required and optional fields, their types, and allowed values. Every message carries a type field that says which schema applies:

  • "upload": an event upload produced by create_event

  • "ping": a connectivity test produced by ping()

Both are validated against their schema before publishing. Consumers and downstream services can use the same schemas to validate incoming messages: validate_message picks the schema from the type field, so it accepts both kinds of message.

Validating messages

from ligo.gracedb.kafka.schema import validate_message

msg = {
    "type": "upload",
    "gracedb_server": "https://gracedb-test.ligo.org/api/",
    "graceid": "G123456",
    "pipeline": "gstlal",
    "gracedb_upload": True,
    "eventFile": ["coinc.xml", "<base64 data>"],
}

validate_message(msg)  # passes silently

# Missing a required field:
del msg["pipeline"]
validate_message(msg)
# raises jsonschema.ValidationError: 'pipeline' is a required property

# A ping is checked against the ping schema, not the upload one:
validate_message({"type": "ping", "timestamp": 1774900000.0,
                  "client": "ligo-gracedb"})  # passes silently

Accessing the schema directly

The raw schema dicts are available for inspection or integration with other tools:

from ligo.gracedb.kafka.schema import (
    UPLOAD_MESSAGE_SCHEMA, PING_MESSAGE_SCHEMA, MESSAGE_SCHEMAS,
)

print(UPLOAD_MESSAGE_SCHEMA["required"])
# ['type', 'gracedb_server', 'pipeline', 'gracedb_upload', 'eventFile']
print(PING_MESSAGE_SCHEMA["required"])
# ['type', 'timestamp', 'client']
print(list(MESSAGE_SCHEMAS))  # keyed by message type
# ['upload', 'ping']

The schema files themselves are at ligo/gracedb/kafka/schemas/upload_message.json and ligo/gracedb/kafka/schemas/ping_message.json inside the installed package.

Required fields

These fields must be present in every upload message:

Field

Type

Description

type

string

Message type, always "upload"

gracedb_server

string

GraceDB service URL the event belongs to

pipeline

string

Analysis pipeline name (e.g., "gstlal", "pycbc")

gracedb_upload

boolean

Whether the server should upload and process the event

eventFile

array

Two-element array: [filename, base64_encoded_contents]

Optional fields

These fields are validated if present but do not cause failures if absent. This ensures backward compatibility with older clients:

Field

Type

Description

graceid

string or null

Reserved GraceDB ID (null when gracedb_upload is false)

submitter

string or null

Submitter identity from the graceid reservation

skip_supervisor

boolean

If true, the supervisor should not process this message

labels

array or null

Labels to attach to the event upon creation

content_id

string

SHA-256-based UUID derived from the event file contents

_producer_timestamp

number

Unix timestamp when the message was produced (for telemetry)

Ping messages

ping() publishes a small message that carries no event. Consumers should recognise it by its type and skip it.

Field

Type

Description

type

string

Message type, always "ping" (required)

timestamp

number

Unix timestamp when the ping was produced (required)

client

string

Producing library, "ligo-gracedb" (required)

sub

string or null

Subject claim of the SciToken used to produce the ping

scope

string or null

Scope claim of that token

Forward compatibility

Both schemas set additionalProperties: true, which means messages with extra fields (e.g., from a newer client version) will pass validation. This allows new fields to be added without breaking existing consumers.

Using the schema in a consumer service

A downstream service (such as the gracedb-kafka-supervisor) can validate incoming messages by importing from ligo-gracedb:

from ligo.gracedb.kafka.schema import validate_message
from jsonschema import ValidationError

for msg in consumer.stream():
    try:
        validate_message(msg["value"])
    except ValidationError as exc:
        log.error(f"Invalid message: {exc.message}")
        continue

    # Pings from producer.ping() are valid messages but carry no
    # event; the topics see one whenever someone tests connectivity.
    if msg["value"]["type"] == "ping":
        log.debug(f"Ping from {msg['value'].get('sub')}")
        continue

    # Process the validated upload message
    process(msg["value"])

API reference

Producer class

class ligo.gracedb.kafka.producer.GraceDbKafkaProducer(bootstrap_servers, topic=None, instance_name=None, service_url=None, use_ssl=True, ca_cert_path=None, skip_hostname_verification=False, token_scope='gracedb.read', token_audience=None, reload_cred=True, reload_buffer=300, flush_timeout=10, guaranteed_delivery=True, validate_token=True, check_for_topics=True, auto_refresh_token=None, **kafka_config)

Kafka producer with SciToken authentication for GraceDB events.

This producer automatically handles SciToken discovery and OAuth authentication for publishing messages to Kafka brokers.

Parameters:
  • bootstrap_servers (str) – Kafka broker address (e.g., “localhost:9092”)

  • topic (str, optional) – Default topic name for messages (pipeline name, e.g., "gstlal"). Will be automatically namespaced with the instance name to produce the actual Kafka topic (e.g., "gracedb-dev.gstlal"). When omitted, a topic must be specified at produce() time. When using the producer with create_event(), the topic is determined automatically from the event’s group and pipeline arguments, so setting a default here is not required.

  • instance_name (str, optional) – GraceDB instance name used to namespace topics (e.g., "gracedb-dev"). Exactly one of instance_name or service_url must be provided.

  • service_url (str, optional) – GraceDB service URL from which the instance name is derived (e.g., "https://gracedb-dev.ligo.org/api/").

  • use_ssl (bool) – Enable SSL/TLS encryption (default: True)

  • ca_cert_path (str, optional) – Path to CA certificate for SSL verification

  • skip_hostname_verification (bool) – Skip SSL hostname verification (default: False). Use for development/testing when cert CN doesn’t match broker hostname.

  • token_scope (str) – SciToken scope to look for (default: “gracedb.read”, the scope GraceDB requires for publishing).

  • token_audience (str, optional) – Required audience. Defaults to broker hostname

  • reload_cred (bool) – Proactively reload SciTokens before expiry (default: True). When enabled, a SciTokenRefreshManager monitors token expiration and re-reads the token from disk before it expires.

  • reload_buffer (int) – Seconds before token expiry to trigger a reload (default: 300). Only used when reload_cred=True.

  • flush_timeout (int) – Seconds to wait for message delivery when create_event() flushes the producer after publishing (default: 10).

  • guaranteed_delivery (bool) – When True (default), configures the producer for reliable delivery: acks=all, retries on transient failures, and delivery verification via callback. Set to False for higher throughput at the risk of silent message loss.

  • validate_token (bool) – Look up the SciToken once, immediately, before connecting (default: True). A missing or expired token then fails fast with a KafkaTokenError that says where it looked and how to get a new token, instead of a generic connection error ten seconds later. Set to False if the token will only become available after construction.

  • check_for_topics (bool) – Fetch the broker’s topic list at construction (default: True) and refuse to instantiate if the topics this producer needs are not available. With a default topic, that topic must exist and the token must be authorized for it. Without one (the usual create_event() setup), the broker must have at least one topic for instance_name; the error otherwise names the instances that do have topics, which catches a wrong instance_name/service_url before any graceid is reserved. An unreachable broker is also an error here. Set to False for a producer that must start before its broker is up; topic problems are then reported when a delivery stalls.

  • **kafka_config – Additional confluent_kafka.Producer configuration options

Raises:
  • ImportError – If confluent-kafka or jsonschema is not installed.

  • ValueError – If neither instance_name nor service_url is given.

  • KafkaTokenError – If validate_token is True and no valid SciToken can be found.

  • KafkaAuthenticationError – If the Kafka client rejects the credentials while the producer is being created.

  • KafkaTopicError – If check_for_topics is True and the default topic does not exist or is not authorized, or no topic exists for instance_name.

  • KafkaConnectionError – If check_for_topics is True and the broker’s topic list cannot be fetched, or if the Kafka client cannot be created for any other reason.

Example

Using with create_event (topic determined automatically):

producer = GraceDbKafkaProducer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    service_url="https://gracedb-dev.ligo.org/api/",
    ca_cert_path="/tmp/cacert.pem",
)
gdb.create_event(
    group="CBC", pipeline="gstlal", filename="coinc.xml",
    filecontents=data, kafka=producer,
)
# Message is published to gracedb-dev.gstlal

Using with a default topic for direct produce() calls:

producer = GraceDbKafkaProducer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topic="test",
    service_url="https://gracedb-dev.ligo.org/api/",
    ca_cert_path="/tmp/cacert.pem",
)
producer.produce(value={"graceid": "G12345"})
# Message is published to gracedb-dev.test
close(timeout=30)

Close the producer and release resources.

This method flushes any pending messages before closing.

Parameters:

timeout (float) – Maximum time to wait for flush in seconds

flush(timeout=-1, strict=False)

Wait for all messages in the producer queue to be delivered.

Parameters:
  • timeout (float) – Maximum time to wait in seconds. -1 = wait indefinitely (default) 0 = non-blocking check >0 = wait up to timeout seconds

  • strict (bool) – When True, raise instead of returning if any message is still queued when timeout expires, or if any delivery report since the last flush carried an error. The exception message includes the last error librdkafka reported (for example “Failed to resolve ‘kafka.example.org:9092’”) and a hint about what to check. Default False preserves the plain confluent-kafka behaviour of returning the count; a stall is then logged as a warning with the count and the last error, without the per-topic broker diagnosis a strict flush performs.

Returns:

Number of messages still in queue after timeout

Return type:

int

Raises:
  • KafkaAuthenticationError – If the SciToken was rejected.

  • KafkaConnectionError – With strict=True, if messages are still queued and the client reported a connection-level error (DNS, connection refused, TLS).

  • KafkaTopicError – With strict=True, if a queued topic does not exist on the broker or the token is not authorized for it.

  • KafkaMessageDeliveryError – With strict=True, if the broker rejected a message or messages are still queued for any other reason.

  • KafkaProducerError – If flush itself fails.

property pending_messages

Number of messages currently pending delivery.

ping(topic=None, timeout=10)

Send a test message to verify broker connectivity and authentication.

Produces a small JSON message whose body includes the sub and scope claims from the SciToken used to authenticate, then waits for delivery confirmation. This exercises the full path: token discovery, OAUTHBEARER handshake, SSL connection, and message delivery.

Parameters:
  • topic (str, optional) – Topic to ping. Uses default_topic, or 'gracedb-test' if no default is set.

  • timeout (float) – Seconds to wait for delivery confirmation (default: 10).

Returns:

Delivery details with keys 'topic', 'partition', 'offset', and 'message' (the test payload that was sent).

Return type:

dict

Raises:
produce(value, topic=None, key=None, headers=None, partition=None, on_delivery=None)

Produce a message to Kafka.

Parameters:
  • value – Message value. Can be: - dict: Automatically serialized to JSON - str: Encoded to UTF-8 bytes - bytes: Sent as-is

  • topic (str, optional) – Pipeline or topic name (e.g., "gstlal"). Automatically namespaced with the instance name (e.g., "gracedb-dev.gstlal"). Falls back to the default topic set at construction if not specified.

  • key (str or bytes, optional) – Message key for partitioning

  • headers (dict, optional) – Message headers as key-value pairs

  • partition (int, optional) – Specific partition to send to

  • on_delivery (callable, optional) – Callback for delivery reports Signature: on_delivery(err, msg)

Raises:
  • ValueError – If topic is not specified and no default was set

  • KafkaProducerError – If message production fails

Example

>>> producer.produce(
...     value={"graceid": "G12345"},
...     topic="gstlal",
...     key="G12345",
... )

Consumer class

class ligo.gracedb.kafka.consumer.GraceDbKafkaConsumer(bootstrap_servers, topics=None, instance_name=None, service_url=None, group_id=None, use_ssl=True, ca_cert_path=None, skip_hostname_verification=False, token_scope='kafka.consume', token_audience=None, reload_cred=True, reload_buffer=300, auto_refresh_token=None, auto_offset_reset='latest', **kafka_config)

Kafka consumer with SciToken authentication for GraceDB events.

This consumer automatically handles SciToken discovery and OAuth authentication for subscribing to Kafka topics and reading messages.

By default, the consumer is configured for live streaming: it generates a unique ephemeral group ID, starts reading from the end of the topic (auto.offset.reset=latest), and automatically commits offsets. This means a fresh consumer only sees messages produced after it connects.

To persist read position across restarts, supply a stable group_id. Kafka will track committed offsets for that group, so subsequent runs resume where the previous one left off.

To replay all historical messages, pass auto_offset_reset="earliest" (useful with a new group_id that has no committed offsets yet).

Parameters:
  • bootstrap_servers (str) – Kafka broker address (e.g., “localhost:9092”)

  • topics (str or list, optional) – Topic(s) to subscribe to immediately. These should be pipeline names (e.g., "gstlal"); they are automatically namespaced with the instance name.

  • instance_name (str, optional) – GraceDB instance name used to namespace topics (e.g., "gracedb-dev"). Exactly one of instance_name or service_url must be provided.

  • service_url (str, optional) – GraceDB service URL from which the instance name is derived (e.g., "https://gracedb-dev.ligo.org/api/").

  • group_id (str, optional) – Consumer group ID. When None (the default), a unique ephemeral ID is generated from the hostname, PID, and current time so that each consumer instance operates independently. Provide a stable string to share offsets across restarts or multiple consumers.

  • use_ssl (bool) – Enable SSL/TLS encryption (default: True)

  • ca_cert_path (str, optional) – Path to CA certificate for SSL verification

  • skip_hostname_verification (bool) – Skip SSL hostname verification (default: False). Use for development/testing when cert CN doesn’t match broker hostname.

  • token_scope (str) – Required SciToken scope (default: “kafka.consume”)

  • token_audience (str, optional) – Required audience. Defaults to broker hostname

  • reload_cred (bool) – Proactively reload SciTokens before expiry (default: True). When enabled, a SciTokenRefreshManager monitors token expiration and re-reads the token from disk before it expires.

  • reload_buffer (int) – Seconds before token expiry to trigger a reload (default: 300). Only used when reload_cred=True.

  • auto_offset_reset (str) – Where to start reading when there is no committed offset for the consumer group. Default: "latest" (only new messages). Set to "earliest" to replay from the beginning of the topic.

  • **kafka_config – Additional confluent_kafka.Consumer configuration options. For example, pass **{"enable.auto.commit": False} to manage offset commits manually.

Example

Stream only new messages (default behaviour):

with GraceDbKafkaConsumer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topics="test",
    ca_cert_path="/tmp/cacert.pem",
) as consumer:
    for msg in consumer.stream():
        print(msg)

Resume from last committed offset across restarts:

with GraceDbKafkaConsumer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topics="test",
    group_id="my-pipeline-consumer",
    ca_cert_path="/tmp/cacert.pem",
) as consumer:
    for msg in consumer.stream():
        process(msg)

Replay all messages from the beginning of a topic:

with GraceDbKafkaConsumer(
    bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
    topics="test",
    group_id="my-replay-group",
    auto_offset_reset="earliest",
    ca_cert_path="/tmp/cacert.pem",
) as consumer:
    for msg in consumer.stream():
        print(msg)
close()

Unsubscribe from topics and close the consumer.

poll(timeout=1.0)

Poll for a single message.

Parameters:

timeout (float) – Maximum time to wait for a message in seconds (default: 1.0)

Returns:

Message dict with keys topic, partition, offset, key, value, timestamp, headers, or None if no message is available.

Return type:

dict or None

Raises:

KafkaConsumerError – If a consumer error occurs

stop()

Signal the stream() loop to stop at the next poll iteration.

stream(timeout=1.0, count=None)

Generator that yields messages from subscribed topics.

Installs signal handlers for SIGINT and SIGTERM so that the first signal causes the loop to exit cleanly (within at most timeout seconds). Original handlers are restored when the generator exits.

Parameters:
  • timeout (float) – Poll timeout in seconds per iteration (default: 1.0)

  • count (int, optional) – Maximum number of messages to yield. None means stream indefinitely.

Yields:

dict – Message dicts (same format as poll())

subscribe(topics)

Subscribe to one or more Kafka topics.

Topic names are automatically namespaced with the instance name. For example, subscribing to "gstlal" with instance name "gracedb-dev" subscribes to "gracedb-dev.gstlal".

Parameters:

topics (str or list) – Pipeline name or list of pipeline names

Raises:

KafkaConsumerError – If subscription fails

Authentication helpers

ligo.gracedb.kafka.auth.get_scitoken_for_kafka(broker_url, scope='gracedb.read', audience=None)

Find and return a valid SciToken for Kafka authentication.

This function discovers a SciToken using igwn-auth-utils and returns the serialized JWT string suitable for Kafka OAUTHBEARER authentication.

Parameters:
  • broker_url (str) – Kafka broker URL (e.g., “localhost:9092”)

  • scope (str) – Required token scope (default: “gracedb.read”)

  • audience (str, optional) – Required audience. If None, extracted from broker_url

Returns:

Serialized JWT token string

Return type:

str

Raises:

KafkaTokenError – If no token can be found, or if the only token available (read directly from disk) has already expired. The message names every location that was searched and how to obtain a fresh token.

ligo.gracedb.kafka.auth.create_oauth_callback(broker_url, scope='gracedb.read', audience=None)

Create an OAuth callback function for confluent_kafka Producer.

The callback is called by the Kafka client to obtain the OAuth token before establishing a connection.

Parameters:
  • broker_url (str) – Kafka broker URL

  • scope (str) – Required token scope

  • audience (str, optional) – Required audience

Returns:

OAuth callback function that returns (token_str, expiry_time)

Return type:

callable

class ligo.gracedb.kafka.auth.SciTokenRefreshManager(broker_url, scope='gracedb.read', audience=None, refresh_buffer=300)

Manages automatic SciToken refresh for long-running Kafka producers.

This class monitors token expiration and triggers callbacks when tokens need to be refreshed.

create_callback()

Create an OAuth callback that uses this manager.

Returns:

OAuth callback function

Return type:

callable

get_token()

Get current token, refreshing if necessary.

Returns:

Current valid token string

Return type:

str

Raises:

KafkaTokenError – If token cannot be obtained

Topic utilities

ligo.gracedb.kafka.utils.derive_instance_name(service_url)

Extract the instance name from a GraceDB service URL.

The instance name is the first component of the hostname (i.e., everything before the first dot, or the full hostname if there is no dot).

Parameters:

service_url (str) – GraceDB service URL (e.g., "https://gracedb-dev.ligo.org/api/")

Returns:

Instance name (e.g., "gracedb-dev")

Return type:

str

Raises:

ValueError – If the hostname cannot be parsed from the URL

Examples

>>> derive_instance_name("https://gracedb-dev.ligo.org/api/")
'gracedb-dev'
>>> derive_instance_name("https://gracedb.ligo.org/api/")
'gracedb'
>>> derive_instance_name("https://localhost:8080/api/")
'localhost'
ligo.gracedb.kafka.utils.make_topic_name(instance_name, pipeline)

Build an instance-namespaced Kafka topic name.

Parameters:
  • instance_name (str) – GraceDB instance name (e.g., "gracedb-dev")

  • pipeline (str) – Pipeline name (e.g., "gstlal")

Returns:

Namespaced topic name (e.g., "gracedb-dev.gstlal")

Return type:

str

ligo.gracedb.kafka.utils.parse_topic_name(namespaced_topic)

Split an instance-namespaced topic into its components.

Parameters:

namespaced_topic (str) – Namespaced topic name (e.g., "gracedb-dev.gstlal")

Returns:

(instance_name, pipeline)

Return type:

tuple

Raises:

ValueError – If the topic does not contain a dot separator

Exception classes

Exceptions for Kafka integration.

exception ligo.gracedb.kafka.exceptions.KafkaAuthenticationError

The broker rejected the SciToken.

Typically the token has the wrong audience or scope, or it expired while the producer or consumer was running and no fresh token was written to disk in time. Raised by the next produce(), flush(), ping() or poll() after librdkafka reports the failure, and by the producer constructor when confluent-kafka reports it while the client is being created.

exception ligo.gracedb.kafka.exceptions.KafkaConnectionError

The broker cannot be reached.

DNS failure, connection refused, TLS handshake failure, or the topic list could not be fetched at construction (check_for_topics=True). Also raised when confluent-kafka refuses the producer configuration. The message carries librdkafka’s own description of the last error and a hint about what to check.

exception ligo.gracedb.kafka.exceptions.KafkaConsumerError

A consumer-side error from subscribe() or poll().

exception ligo.gracedb.kafka.exceptions.KafkaError

Base class for every exception raised by ligo.gracedb.kafka.

Catch this to handle any Kafka problem with one handler.

exception ligo.gracedb.kafka.exceptions.KafkaMessageDeliveryError

A message was not delivered.

Raised by a strict flush(), ping() and create_event when the broker rejected a message, or when messages were still queued at the timeout and neither a connection error nor a topic error explains it.

exception ligo.gracedb.kafka.exceptions.KafkaProducerError

confluent-kafka raised while producing or flushing.

For example a message larger than message.max.bytes. The wrapped KafkaException text is included in the message.

exception ligo.gracedb.kafka.exceptions.KafkaTokenError

No usable SciToken could be found.

Raised before any broker contact: by the producer constructor when validate_token=True (the default), by ping(), and inside the OAuth callback when a token is (re)loaded. The message lists every location searched, or the expiry time of the expired token found, and the htgettoken command that fixes it.

exception ligo.gracedb.kafka.exceptions.KafkaTopicError

The target topic does not exist, or the token is not authorized for it.

Automatic topic creation is disabled on the GraceDB brokers, so this usually means a wrong instance_name/service_url or pipeline name. Raised by the producer constructor (check_for_topics=True; without a default topic it means the instance has no topics at all on the broker), by ping(), and by a strict flush() or create_event once a stalled delivery has been traced to the topic. The message names the instances that do have topics on the broker where that is known.