Kafka integration
ligo-gracedb includes a Kafka producer and consumer that authenticate with SciTokens over the OAUTHBEARER SASL mechanism. These classes are designed for long-running analysis pipelines that need to publish or consume GraceDB events in real time.
Note
The Kafka classes require the confluent-kafka C library and
jsonschema (messages are validated before they are produced).
Neither is installed with the base package; install the optional
dependencies with:
pip install ligo-gracedb[kafka]
With conda, install python-confluent-kafka and jsonschema
alongside ligo-gracedb. A producer constructed without them
raises ImportError naming the missing package.
Authentication
Both the producer and the consumer discover a SciToken automatically using igwn-auth-utils. The token is located following the WLCG Bearer Token Discovery specification:
$BEARER_TOKENenvironment variable (token content)$BEARER_TOKEN_FILEenvironment variable (path to file)$XDG_RUNTIME_DIR/bt_u<uid>or/tmp/bt_u<uid>
If igwn-auth-utils cannot match the required audience or scope, a
direct file read of the above paths is attempted as a fallback.
You can obtain a fresh SciToken with htgettoken:
htgettoken -a vault.ligo.org -i igwn
By default, both classes set reload_cred=True, which installs a
SciTokenRefreshManager that
monitors the token’s exp claim and re-reads the token from disk
before it expires. This mirrors the reload_cred behaviour of the
REST client and is essential for long-running processes.
Note
In addition to having a valid, non-expired SciToken, a user must
have the gracedb.read scope in order to publish to Kafka. On
event submission, GraceDB’s existing per-user/per-pipeline permission
model will control which users can publish to individual pipelines.
Once changes in GraceDB scopes are implemented, access to pipeline-specific topics will be controlled on a per-user basis.
Write access to the test topic will remain available to all LVK users
with the gracedb.read scope.
Upgrading from 2.15
Version 2.16 made the producer fail fast and report causes. Code written against 2.15 keeps working in the common case, but four behaviours changed:
Exceptions. create_event(..., kafka=producer) raises the
KafkaError subclasses described under
Error handling instead of RuntimeError. Catch
KafkaError (or ligo.gracedb.kafka.exceptions.KafkaError) where
you caught RuntimeError before. KafkaError does not inherit
from RuntimeError.
Construction contacts the broker. By default the constructor
looks up the SciToken (validate_token=True) and asks the broker for
its topics (check_for_topics=True), so a missing token, an
unreachable broker or a wrong instance_name now raises at
construction with a message that says what to fix. A producer that
must start before its broker or token is available can pass
check_for_topics=False and validate_token=False to get the 2.15
behaviour of constructing silently and failing on the first flush.
Default token scope. token_scope defaults to gracedb.read,
the scope the GraceDB brokers actually require, instead of the
non-existent kafka.produce. Producers that passed token_scope
explicitly are unaffected; producers that relied on the default now
find a usable token where they previously did not.
Message contents. Outgoing messages no longer include
content_id (it is still sent with the graceid reservation and with
HTTP uploads), and every message is validated against the
Message schema before it is produced. Downstream
consumers that read content_id from Kafka messages should take it
from the reservation instead.
Topic namespacing
Kafka topics are instance-namespaced: each topic is prefixed with
the GraceDB instance name so that multiple GraceDB deployments
(development, test, playground, production) can share the same broker
without message collisions. The format is
<instance_name>.<pipeline>, for example:
gracedb-dev.gstlalgracedb-test.cwbgracedb.aframe
The instance name is derived automatically from the GraceDB service URL
(using the first component of the hostname). For example,
https://gracedb-dev.ligo.org/api/ yields instance name
gracedb-dev.
Both the producer and consumer handle namespacing transparently: you
pass pipeline names (e.g. "gstlal") and the classes prepend the
instance name for you. You can supply the instance identity in two
ways:
service_url — the GraceDB service URL; the instance name is extracted automatically.
instance_name — an explicit instance name string.
Exactly one of these must be provided.
Three utility functions are available for working with namespaced topics directly:
from ligo.gracedb.kafka import (
derive_instance_name,
make_topic_name,
parse_topic_name,
)
derive_instance_name("https://gracedb-dev.ligo.org/api/")
# => "gracedb-dev"
make_topic_name("gracedb-dev", "gstlal")
# => "gracedb-dev.gstlal"
parse_topic_name("gracedb-dev.gstlal")
# => ("gracedb-dev", "gstlal")
GraceDB Kafka brokers
Two brokers, hosted at Caltech, serve the GraceDB instances. Pass the
one matching your service_url as bootstrap_servers:
Broker |
GraceDB instances served |
|---|---|
|
|
|
|
Each broker only carries topics for the instances it serves, so a
producer pointed at the wrong broker for its service_url fails at
construction with a KafkaTopicError
that lists the instances the broker does have (see
Startup checks).
Both brokers present certificates from a public CA, so
ca_cert_path=certifi.where() is sufficient; no custom CA bundle is
needed.
The brokers run the container from the scitokens-kafka-broker repository. If you have trouble reaching a broker, or believe a topic or permission is missing, please open a ticket on the IGWN Computing Help Desk.
Notes on Downstream Consumers
Topics on the GraceDB event submission Kafka broker correspond to approved
and testing pipelines
on GraceDB. A test topic is available to submit events to the Test
group. Please see the above note regarding topic permissions.
Uploading a GW trigger via Kafka does not directly insert a new event
into GraceDB, as was the use case of the create_event REST API call.
Rather, publishing an event to Kafka allows the simultaneous processing
of GW data products by multiple downstream processes.
Ahead of IR1, a dedicated service called the
gracedb-kafka-supervisor
is running in AWS alongside GraceDB. This application consumes Kafka
messages and injects event triggers and data products into GraceDB.
Please see the section below on Skipping server-side processing to
control how events are handled by the gracedb-kafka-supervisor.
As the IR1 and beyond low-latency alert infrastructure evolves, follow-up pipelines may opt to delay uploading g-events until after time-critical periods of the alert generation process.
Producer
A GraceDbKafkaProducer wraps a
confluent_kafka.Producer with SciToken authentication and sensible
defaults for GraceDB usage.
Creating a producer
import certifi
from ligo.gracedb.kafka import GraceDbKafkaProducer
producer = GraceDbKafkaProducer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
topic="test",
service_url="https://gracedb-dev.ligo.org/api/",
ca_cert_path=certifi.where(),
)
The topic argument is the pipeline name (e.g. "test",
"gstlal"). It is automatically namespaced with the instance name
derived from service_url, so the actual Kafka topic above would be
gracedb-dev.test. See Topic namespacing for details.
During development, when the broker’s TLS certificate does not match
its hostname, pass skip_hostname_verification=True.
Startup checks
Constructing a producer is not free: by default it finds your SciToken and contacts the broker, so that a misconfiguration is reported immediately, in a sentence, rather than as a silent timeout on the first upload. Two keyword arguments control this:
Flag |
Default |
What it does |
|---|---|---|
|
|
Looks up the SciToken once and raises
|
|
|
Fetches the broker’s topic list (10 s timeout) and refuses to
instantiate if the topics this producer needs are not there.
With a default |
With both checks on, the constructor can raise:
KafkaTokenError— no usable SciToken (message says where it looked and how to get one);KafkaConnectionError— the broker could not be reached, orconfluent-kafkarefused the configuration;KafkaAuthenticationError— the broker rejected the token;KafkaTopicError— the default topic is missing or unauthorized, or the instance has no topics on this broker (message lists the instances that do);ValueError— neitherinstance_namenorservice_urlwas given;ImportError—confluent-kafkais not installed.
All of the Kafka exceptions share the
KafkaError base class, so a pipeline
that wants a single handler can catch that:
from ligo.gracedb.kafka import GraceDbKafkaProducer, KafkaError
try:
producer = GraceDbKafkaProducer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
service_url="https://gracedb-test.ligo.org/api/",
ca_cert_path=certifi.where(),
)
except KafkaError as exc:
sys.exit(f"Cannot start Kafka producer: {exc}")
A producer created with check_for_topics=False is still protected
later: ping()
checks its topic before sending, and a stalled strict flush asks the
broker about the queued topics.
Producing messages
The produce()
method accepts a dict, str, or bytes value. Dicts are
serialised to JSON automatically:
producer.produce({"graceid": "G12345", "group": "CBC"})
producer.flush(strict=True) # wait for delivery; raise if it fails
producer.close() # flush remaining messages and release resources
produce() only queues the message; delivery happens in the
background. flush() waits for the queue to drain and, like the
underlying confluent-kafka call, returns the number of messages
still undelivered when the timeout expires. Pass strict=True to
have it raise instead, with an explanation of what went wrong (see
Diagnosing undelivered messages).
The producer also works as a context manager:
with GraceDbKafkaProducer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
topic="test",
service_url="https://gracedb-dev.ligo.org/api/",
ca_cert_path=certifi.where(),
) as producer:
producer.produce({"graceid": "G12345", "group": "CBC"})
Testing connectivity
Call ping()
to verify end-to-end connectivity. This checks that the topic exists,
produces a small test message, waits for delivery confirmation, and
returns the delivery metadata:
result = producer.ping(topic="test")
print(result)
# {'topic': 'gracedb-dev.test', 'partition': 0, 'offset': 42,
# 'message': {'type': 'ping', 'sub': 'albert.einstein@ligo.org', ...}}
The topic argument is namespaced like everywhere else; with no
argument, ping() uses the producer’s default topic, or
<instance>.test if none was set. On failure it raises
KafkaTopicError (topic missing or
unauthorized), KafkaConnectionError
(broker unreachable),
KafkaAuthenticationError (token
rejected), or KafkaMessageDeliveryError
(anything else), each with a message that says what to check.
Uploading events with Kafka
The preferred way to upload events through Kafka is the two-step
reserve + produce workflow provided by
create_event(). Passing a
producer to its kafka parameter causes create_event to:
Reserve a graceid on the GraceDB server via an HTTP call to the reservations endpoint. This returns a
graceidandsubmitterbefore any event data is transmitted.Produce a Kafka message containing the reserved
graceid, the base64-encoded event file, pipeline metadata, and optional labels. The GraceDB server-side consumer receives this message and completes the upload.
This approach decouples the latency-sensitive part of event submission
(obtaining a graceid) from the bulk data transfer, and provides an
optional HTTP fallback if Kafka delivery fails.
Note
GraceDB will not dispatch an igwn-alert when a graceid is reserved.
Once a reserved event is uploaded, then GraceDB will dispatch a “new”-type
igwn-alert.
Basic example
import certifi
from ligo.gracedb.rest import GraceDb
from ligo.gracedb.kafka import GraceDbKafkaProducer
client = GraceDb('https://gracedb-test.ligo.org/api/', api_version='v2')
producer = GraceDbKafkaProducer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
service_url="https://gracedb-test.ligo.org/api/",
ca_cert_path=certifi.where(),
)
response = client.create_event(
"CBC", "gstlal", "coinc.xml",
search="AllSky",
kafka=producer,
)
# Message is published to gracedb-test.gstlal
print(response.json()["graceid"]) # e.g. "G123456"
producer.close()
In the above example, by not specifying topic in the
GraceDbKafkaProducer, the producer publishes to a topic that is
determined by the group and pipeline arguments passed to
create_event. Events in the Test group are routed to
{instance}.test; all other groups are routed to
{instance}.{pipeline} (lowercased).
create_event automatically flushes the producer after publishing, so
the message is guaranteed to be delivered (or an error raised) before the
call returns. The flush timeout defaults to 10 seconds and can be
configured via the flush_timeout parameter on the producer:
producer = GraceDbKafkaProducer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
service_url="https://gracedb-test.ligo.org/api/",
ca_cert_path=certifi.where(),
flush_timeout=30, # wait up to 30s for delivery
)
Using the convenience factory
The create_kafka_producer() method
on the client creates a pre-configured producer. The client’s service
URL is passed automatically, so topics are namespaced without any extra
arguments:
client = GraceDb('https://gracedb-test.ligo.org/api/', api_version='v2')
producer = client.create_kafka_producer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
topic="test",
ca_cert_path=certifi.where(),
)
HTTP fallback
If http_fallback=True and the Kafka path fails (reservation failure
or produce failure), create_event automatically falls back to the
standard HTTP POST:
response = client.create_event(
"CBC", "gstlal", "coinc.xml",
kafka=producer,
http_fallback=True,
)
When a graceid has already been reserved but the Kafka produce fails,
the fallback uses
upload_reserved_event() to
complete the upload over HTTP. A RuntimeWarning is emitted that
names the Kafka error that triggered the fallback.
With the default http_fallback=False, a Kafka delivery failure
raises the corresponding KafkaError
subclass (see Error handling); the graceid stays reserved
and can be completed later with upload_reserved_event. A failed
reservation raises HTTPError.
Skipping server-side processing
Two boolean flags control what the server does with the Kafka message:
gracedb_upload=False— publish the event to Kafka without reserving a graceid or processing it on the server. Useful when a downstream consumer other than GraceDB handles the message.skip_supervisor=True— the server-side supervisor will not process the message (i.e. will not callupload_reserved_event). DefaultFalse.
Consumer
A GraceDbKafkaConsumer wraps a
confluent_kafka.Consumer with SciToken authentication.
Note
The consumer does not perform the producer’s
startup checks. Subscribing to a
topic that does not exist is not an error in Kafka; the consumer
simply receives nothing. Double-check instance_name and the
pipeline name if a stream stays silent.
Live streaming
By default the consumer generates an ephemeral group ID, reads from
the end of the topic (auto.offset.reset=latest), and auto-commits
offsets. This means it only sees messages produced after it starts:
import certifi
from ligo.gracedb.kafka import GraceDbKafkaConsumer
with GraceDbKafkaConsumer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
topics="test",
service_url="https://gracedb-dev.ligo.org/api/",
ca_cert_path=certifi.where(),
) as consumer:
for msg in consumer.stream():
print(msg["value"])
Pressing Ctrl-C stops the stream cleanly; the consumer installs
SIGINT and SIGTERM handlers that cause the current poll to
finish and the generator to exit.
Persisting read position
Supply a stable group_id to resume from the last committed offset
across restarts:
import certifi
with GraceDbKafkaConsumer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
topics="test",
service_url="https://gracedb-dev.ligo.org/api/",
group_id="my-pipeline-consumer",
ca_cert_path=certifi.where(),
) as consumer:
for msg in consumer.stream():
process(msg)
Replaying history
Combine a fresh group_id with auto_offset_reset="earliest" to
replay all messages from the beginning of the topic:
import certifi
with GraceDbKafkaConsumer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
topics="test",
service_url="https://gracedb-dev.ligo.org/api/",
group_id="my-replay-group",
auto_offset_reset="earliest",
ca_cert_path=certifi.where(),
) as consumer:
for msg in consumer.stream():
print(msg)
Polling manually
If the stream() generator is too opinionated, call
poll()
directly:
import certifi
consumer = GraceDbKafkaConsumer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
service_url="https://gracedb-dev.ligo.org/api/",
ca_cert_path=certifi.where(),
)
consumer.subscribe("test")
while True:
msg = consumer.poll(timeout=1.0)
if msg is not None:
handle(msg)
Token reloading
Long-running producers and consumers must cope with SciToken expiry.
By default both classes set reload_cred=True, which creates a
SciTokenRefreshManager. The
manager parses the JWT exp claim and re-discovers a fresh token
reload_buffer seconds before expiry (default: 300 s).
If an external process (e.g. htgettoken in a cron job) periodically
writes a fresh token to disk, the refresh manager will pick it up
transparently. The manager cannot mint tokens itself, so make sure the
external job replaces the file before the old token expires; with
htgettoken that means setting --minsecs comfortably above the
cron interval (e.g. --minsecs 4000 for an hourly job; 3600
is not enough). A job that only
renews at expiry produces a short burst of SASL errors every few
hours; see SASL errors every time the token is renewed.
To disable proactive reloading:
producer = GraceDbKafkaProducer(
bootstrap_servers="kafkagracedbtest1.igwn.org:9092",
service_url="https://gracedb-dev.ligo.org/api/",
reload_cred=False,
)
Error handling
All Kafka-specific exceptions inherit from
KafkaError:
KafkaError
+-- KafkaAuthenticationError
| +-- KafkaTokenError
+-- KafkaProducerError
| +-- KafkaMessageDeliveryError
| +-- KafkaTopicError
+-- KafkaConsumerError
+-- KafkaConnectionError
Exception |
Raised when |
What to check |
|---|---|---|
|
No SciToken could be found, or the only one found has expired.
Raised by the constructor ( |
Run |
|
The broker rejected the token (wrong audience or scope, or it
expired while the producer was running). Raised by the next
|
Token scope and audience; whether the token was refreshed on disk in time. |
|
The broker cannot be reached: DNS failure, connection refused,
TLS handshake failure, or the topic list could not be fetched
at construction. Raised by the constructor
( |
|
|
The topic does not exist on the broker (automatic creation is
disabled) or the token is not authorized for it, or, at
construction without a default topic, the instance has no
topics at all. Raised by the constructor, |
|
|
The broker rejected a message for a reason other than the
topic, or messages were still queued when a strict |
|
|
|
The wrapped |
|
A consumer-side error from |
The wrapped error in the message. |
Authentication failures that occur inside librdkafka (for example, an
expired token that was not refreshed in time) are surfaced as
KafkaAuthenticationError on the next
produce(),
flush(), or
poll() call.
Failing fast on token problems
The producer looks up the SciToken once when it is constructed
(validate_token=True, the default). A missing or expired token
therefore raises KafkaTokenError
immediately, with a message that lists every location that was
searched and the htgettoken command that fixes it:
KafkaTokenError: No valid SciToken found for Kafka broker
'kafkagracedbtest1.igwn.org' (scope 'gracedb.read'). Searched:
$BEARER_TOKEN (unset); $BEARER_TOKEN_FILE (unset);
/tmp/bt_u1000 (not found). Obtain a fresh token with:
htgettoken -a vault.ligo.org -i igwn
KafkaTokenError: SciToken from /tmp/bt_u1000 expired at
2026-09-25 14:59:08 UTC. Obtain a fresh token with:
htgettoken -a vault.ligo.org -i igwn
Without this check, confluent-kafka would swallow the error, wait
ten seconds, and report a generic “OAuth token not set” failure. Pass
validate_token=False if the token will only be written to disk
after the producer is created.
Diagnosing undelivered messages
An unreachable broker does not raise on produce(): librdkafka
queues the message and retries in the background, so the first sign of
trouble is a flush() that returns with messages still queued. The
producer remembers the last error librdkafka reported (DNS failure,
connection refused, TLS handshake failure, and so on) and uses it to
explain the stall.
create_event() always flushes with
strict=True, so a delivery problem raises rather than returning a
count:
KafkaConnectionError: Delivery timed out after 10s with 1 message(s)
still queued for Kafka broker kafkagracedbtest1.igwn.org:9092. Last
error: sasl_ssl://kafkagracedbtest1.igwn.org:9092/bootstrap: Failed
to resolve 'kafkagracedbtest1.igwn.org:9092': nodename nor servname
provided, or not known [_RESOLVE]. Check bootstrap_servers and DNS.
When calling the producer directly, opt in the same way:
producer.produce({"graceid": "G12345"})
producer.flush(timeout=10, strict=True)
With strict=True, flush() raises
KafkaConnectionError when the client
reported a connection-level error,
KafkaTopicError when the topic is the
problem (see below), and
KafkaMessageDeliveryError when the
broker rejected a message or messages remain queued for any other
reason. With the default strict=False, flush() keeps the plain
confluent-kafka behaviour of returning the number of undelivered
messages and logs a warning with that count and the last error
librdkafka reported. Only a strict flush asks the broker about the
queued topics (see below), since that costs a metadata request per
topic.
Message schema
Kafka messages produced by ligo-gracedb conform to JSON Schemas
that ship with the package. The schemas define the required and
optional fields, their types, and allowed values. Every message
carries a type field that says which schema applies:
"upload": an event upload produced bycreate_event"ping": a connectivity test produced byping()
Both are validated against their schema before publishing. Consumers
and downstream services can use the same schemas to validate incoming
messages: validate_message picks the schema from the type
field, so it accepts both kinds of message.
Validating messages
from ligo.gracedb.kafka.schema import validate_message
msg = {
"type": "upload",
"gracedb_server": "https://gracedb-test.ligo.org/api/",
"graceid": "G123456",
"pipeline": "gstlal",
"gracedb_upload": True,
"eventFile": ["coinc.xml", "<base64 data>"],
}
validate_message(msg) # passes silently
# Missing a required field:
del msg["pipeline"]
validate_message(msg)
# raises jsonschema.ValidationError: 'pipeline' is a required property
# A ping is checked against the ping schema, not the upload one:
validate_message({"type": "ping", "timestamp": 1774900000.0,
"client": "ligo-gracedb"}) # passes silently
Accessing the schema directly
The raw schema dicts are available for inspection or integration with other tools:
from ligo.gracedb.kafka.schema import (
UPLOAD_MESSAGE_SCHEMA, PING_MESSAGE_SCHEMA, MESSAGE_SCHEMAS,
)
print(UPLOAD_MESSAGE_SCHEMA["required"])
# ['type', 'gracedb_server', 'pipeline', 'gracedb_upload', 'eventFile']
print(PING_MESSAGE_SCHEMA["required"])
# ['type', 'timestamp', 'client']
print(list(MESSAGE_SCHEMAS)) # keyed by message type
# ['upload', 'ping']
The schema files themselves are at
ligo/gracedb/kafka/schemas/upload_message.json and
ligo/gracedb/kafka/schemas/ping_message.json inside the installed
package.
Required fields
These fields must be present in every upload message:
Field |
Type |
Description |
|---|---|---|
|
string |
Message type, always |
|
string |
GraceDB service URL the event belongs to |
|
string |
Analysis pipeline name (e.g., |
|
boolean |
Whether the server should upload and process the event |
|
array |
Two-element array: |
Optional fields
These fields are validated if present but do not cause failures if absent. This ensures backward compatibility with older clients:
Field |
Type |
Description |
|---|---|---|
|
string or null |
Reserved GraceDB ID (null when |
|
string or null |
Submitter identity from the graceid reservation |
|
boolean |
If true, the supervisor should not process this message |
|
array or null |
Labels to attach to the event upon creation |
|
string |
SHA-256-based UUID derived from the event file contents |
|
number |
Unix timestamp when the message was produced (for telemetry) |
Ping messages
ping()
publishes a small message that carries no event. Consumers should
recognise it by its type and skip it.
Field |
Type |
Description |
|---|---|---|
|
string |
Message type, always |
|
number |
Unix timestamp when the ping was produced (required) |
|
string |
Producing library, |
|
string or null |
Subject claim of the SciToken used to produce the ping |
|
string or null |
Scope claim of that token |
Forward compatibility
Both schemas set additionalProperties: true, which means messages
with extra fields (e.g., from a newer client version) will pass
validation. This allows new fields to be added without breaking
existing consumers.
Using the schema in a consumer service
A downstream service (such as the gracedb-kafka-supervisor) can
validate incoming messages by importing from ligo-gracedb:
from ligo.gracedb.kafka.schema import validate_message
from jsonschema import ValidationError
for msg in consumer.stream():
try:
validate_message(msg["value"])
except ValidationError as exc:
log.error(f"Invalid message: {exc.message}")
continue
# Pings from producer.ping() are valid messages but carry no
# event; the topics see one whenever someone tests connectivity.
if msg["value"]["type"] == "ping":
log.debug(f"Ping from {msg['value'].get('sub')}")
continue
# Process the validated upload message
process(msg["value"])
API reference
Producer class
- class ligo.gracedb.kafka.producer.GraceDbKafkaProducer(bootstrap_servers, topic=None, instance_name=None, service_url=None, use_ssl=True, ca_cert_path=None, skip_hostname_verification=False, token_scope='gracedb.read', token_audience=None, reload_cred=True, reload_buffer=300, flush_timeout=10, guaranteed_delivery=True, validate_token=True, check_for_topics=True, auto_refresh_token=None, **kafka_config)
Kafka producer with SciToken authentication for GraceDB events.
This producer automatically handles SciToken discovery and OAuth authentication for publishing messages to Kafka brokers.
- Parameters:
bootstrap_servers (str) – Kafka broker address (e.g., “localhost:9092”)
topic (str, optional) – Default topic name for messages (pipeline name, e.g.,
"gstlal"). Will be automatically namespaced with the instance name to produce the actual Kafka topic (e.g.,"gracedb-dev.gstlal"). When omitted, a topic must be specified atproduce()time. When using the producer withcreate_event(), the topic is determined automatically from the event’sgroupandpipelinearguments, so setting a default here is not required.instance_name (str, optional) – GraceDB instance name used to namespace topics (e.g.,
"gracedb-dev"). Exactly one of instance_name or service_url must be provided.service_url (str, optional) – GraceDB service URL from which the instance name is derived (e.g.,
"https://gracedb-dev.ligo.org/api/").use_ssl (bool) – Enable SSL/TLS encryption (default: True)
ca_cert_path (str, optional) – Path to CA certificate for SSL verification
skip_hostname_verification (bool) – Skip SSL hostname verification (default: False). Use for development/testing when cert CN doesn’t match broker hostname.
token_scope (str) – SciToken scope to look for (default: “gracedb.read”, the scope GraceDB requires for publishing).
token_audience (str, optional) – Required audience. Defaults to broker hostname
reload_cred (bool) – Proactively reload SciTokens before expiry (default: True). When enabled, a
SciTokenRefreshManagermonitors token expiration and re-reads the token from disk before it expires.reload_buffer (int) – Seconds before token expiry to trigger a reload (default: 300). Only used when
reload_cred=True.flush_timeout (int) – Seconds to wait for message delivery when
create_event()flushes the producer after publishing (default: 10).guaranteed_delivery (bool) – When
True(default), configures the producer for reliable delivery:acks=all, retries on transient failures, and delivery verification via callback. Set toFalsefor higher throughput at the risk of silent message loss.validate_token (bool) – Look up the SciToken once, immediately, before connecting (default: True). A missing or expired token then fails fast with a
KafkaTokenErrorthat says where it looked and how to get a new token, instead of a generic connection error ten seconds later. Set toFalseif the token will only become available after construction.check_for_topics (bool) – Fetch the broker’s topic list at construction (default: True) and refuse to instantiate if the topics this producer needs are not available. With a default topic, that topic must exist and the token must be authorized for it. Without one (the usual
create_event()setup), the broker must have at least one topic for instance_name; the error otherwise names the instances that do have topics, which catches a wronginstance_name/service_urlbefore any graceid is reserved. An unreachable broker is also an error here. Set toFalsefor a producer that must start before its broker is up; topic problems are then reported when a delivery stalls.**kafka_config – Additional confluent_kafka.Producer configuration options
- Raises:
ImportError – If confluent-kafka or jsonschema is not installed.
ValueError – If neither instance_name nor service_url is given.
KafkaTokenError – If validate_token is
Trueand no valid SciToken can be found.KafkaAuthenticationError – If the Kafka client rejects the credentials while the producer is being created.
KafkaTopicError – If check_for_topics is
Trueand the default topic does not exist or is not authorized, or no topic exists for instance_name.KafkaConnectionError – If check_for_topics is
Trueand the broker’s topic list cannot be fetched, or if the Kafka client cannot be created for any other reason.
Example
Using with
create_event(topic determined automatically):producer = GraceDbKafkaProducer( bootstrap_servers="kafkagracedbtest1.igwn.org:9092", service_url="https://gracedb-dev.ligo.org/api/", ca_cert_path="/tmp/cacert.pem", ) gdb.create_event( group="CBC", pipeline="gstlal", filename="coinc.xml", filecontents=data, kafka=producer, ) # Message is published to gracedb-dev.gstlal
Using with a default topic for direct
produce()calls:producer = GraceDbKafkaProducer( bootstrap_servers="kafkagracedbtest1.igwn.org:9092", topic="test", service_url="https://gracedb-dev.ligo.org/api/", ca_cert_path="/tmp/cacert.pem", ) producer.produce(value={"graceid": "G12345"}) # Message is published to gracedb-dev.test
- close(timeout=30)
Close the producer and release resources.
This method flushes any pending messages before closing.
- Parameters:
timeout (float) – Maximum time to wait for flush in seconds
- flush(timeout=-1, strict=False)
Wait for all messages in the producer queue to be delivered.
- Parameters:
timeout (float) – Maximum time to wait in seconds. -1 = wait indefinitely (default) 0 = non-blocking check >0 = wait up to timeout seconds
strict (bool) – When
True, raise instead of returning if any message is still queued when timeout expires, or if any delivery report since the last flush carried an error. The exception message includes the last error librdkafka reported (for example “Failed to resolve ‘kafka.example.org:9092’”) and a hint about what to check. DefaultFalsepreserves the plain confluent-kafka behaviour of returning the count; a stall is then logged as a warning with the count and the last error, without the per-topic broker diagnosis a strict flush performs.
- Returns:
Number of messages still in queue after timeout
- Return type:
int
- Raises:
KafkaAuthenticationError – If the SciToken was rejected.
KafkaConnectionError – With
strict=True, if messages are still queued and the client reported a connection-level error (DNS, connection refused, TLS).KafkaTopicError – With
strict=True, if a queued topic does not exist on the broker or the token is not authorized for it.KafkaMessageDeliveryError – With
strict=True, if the broker rejected a message or messages are still queued for any other reason.KafkaProducerError – If flush itself fails.
- property pending_messages
Number of messages currently pending delivery.
- ping(topic=None, timeout=10)
Send a test message to verify broker connectivity and authentication.
Produces a small JSON message whose body includes the
subandscopeclaims from the SciToken used to authenticate, then waits for delivery confirmation. This exercises the full path: token discovery, OAUTHBEARER handshake, SSL connection, and message delivery.- Parameters:
topic (str, optional) – Topic to ping. Uses default_topic, or
'gracedb-test'if no default is set.timeout (float) – Seconds to wait for delivery confirmation (default: 10).
- Returns:
Delivery details with keys
'topic','partition','offset', and'message'(the test payload that was sent).- Return type:
dict
- Raises:
KafkaTokenError – If no SciToken can be found.
KafkaAuthenticationError – If the broker rejected the token.
KafkaTopicError – If the topic does not exist on the broker or the token is not authorized for it.
KafkaMessageDeliveryError – If the broker rejected the message, or it was not confirmed within timeout for a reason other than connectivity.
KafkaConnectionError – If the broker is unreachable.
- produce(value, topic=None, key=None, headers=None, partition=None, on_delivery=None)
Produce a message to Kafka.
- Parameters:
value – Message value. Can be: - dict: Automatically serialized to JSON - str: Encoded to UTF-8 bytes - bytes: Sent as-is
topic (str, optional) – Pipeline or topic name (e.g.,
"gstlal"). Automatically namespaced with the instance name (e.g.,"gracedb-dev.gstlal"). Falls back to the default topic set at construction if not specified.key (str or bytes, optional) – Message key for partitioning
headers (dict, optional) – Message headers as key-value pairs
partition (int, optional) – Specific partition to send to
on_delivery (callable, optional) – Callback for delivery reports Signature: on_delivery(err, msg)
- Raises:
ValueError – If topic is not specified and no default was set
KafkaProducerError – If message production fails
Example
>>> producer.produce( ... value={"graceid": "G12345"}, ... topic="gstlal", ... key="G12345", ... )
Consumer class
- class ligo.gracedb.kafka.consumer.GraceDbKafkaConsumer(bootstrap_servers, topics=None, instance_name=None, service_url=None, group_id=None, use_ssl=True, ca_cert_path=None, skip_hostname_verification=False, token_scope='kafka.consume', token_audience=None, reload_cred=True, reload_buffer=300, auto_refresh_token=None, auto_offset_reset='latest', **kafka_config)
Kafka consumer with SciToken authentication for GraceDB events.
This consumer automatically handles SciToken discovery and OAuth authentication for subscribing to Kafka topics and reading messages.
By default, the consumer is configured for live streaming: it generates a unique ephemeral group ID, starts reading from the end of the topic (
auto.offset.reset=latest), and automatically commits offsets. This means a fresh consumer only sees messages produced after it connects.To persist read position across restarts, supply a stable
group_id. Kafka will track committed offsets for that group, so subsequent runs resume where the previous one left off.To replay all historical messages, pass
auto_offset_reset="earliest"(useful with a newgroup_idthat has no committed offsets yet).- Parameters:
bootstrap_servers (str) – Kafka broker address (e.g., “localhost:9092”)
topics (str or list, optional) – Topic(s) to subscribe to immediately. These should be pipeline names (e.g.,
"gstlal"); they are automatically namespaced with the instance name.instance_name (str, optional) – GraceDB instance name used to namespace topics (e.g.,
"gracedb-dev"). Exactly one of instance_name or service_url must be provided.service_url (str, optional) – GraceDB service URL from which the instance name is derived (e.g.,
"https://gracedb-dev.ligo.org/api/").group_id (str, optional) – Consumer group ID. When
None(the default), a unique ephemeral ID is generated from the hostname, PID, and current time so that each consumer instance operates independently. Provide a stable string to share offsets across restarts or multiple consumers.use_ssl (bool) – Enable SSL/TLS encryption (default: True)
ca_cert_path (str, optional) – Path to CA certificate for SSL verification
skip_hostname_verification (bool) – Skip SSL hostname verification (default: False). Use for development/testing when cert CN doesn’t match broker hostname.
token_scope (str) – Required SciToken scope (default: “kafka.consume”)
token_audience (str, optional) – Required audience. Defaults to broker hostname
reload_cred (bool) – Proactively reload SciTokens before expiry (default: True). When enabled, a
SciTokenRefreshManagermonitors token expiration and re-reads the token from disk before it expires.reload_buffer (int) – Seconds before token expiry to trigger a reload (default: 300). Only used when
reload_cred=True.auto_offset_reset (str) – Where to start reading when there is no committed offset for the consumer group. Default:
"latest"(only new messages). Set to"earliest"to replay from the beginning of the topic.**kafka_config – Additional confluent_kafka.Consumer configuration options. For example, pass
**{"enable.auto.commit": False}to manage offset commits manually.
Example
Stream only new messages (default behaviour):
with GraceDbKafkaConsumer( bootstrap_servers="kafkagracedbtest1.igwn.org:9092", topics="test", ca_cert_path="/tmp/cacert.pem", ) as consumer: for msg in consumer.stream(): print(msg)
Resume from last committed offset across restarts:
with GraceDbKafkaConsumer( bootstrap_servers="kafkagracedbtest1.igwn.org:9092", topics="test", group_id="my-pipeline-consumer", ca_cert_path="/tmp/cacert.pem", ) as consumer: for msg in consumer.stream(): process(msg)
Replay all messages from the beginning of a topic:
with GraceDbKafkaConsumer( bootstrap_servers="kafkagracedbtest1.igwn.org:9092", topics="test", group_id="my-replay-group", auto_offset_reset="earliest", ca_cert_path="/tmp/cacert.pem", ) as consumer: for msg in consumer.stream(): print(msg)
- close()
Unsubscribe from topics and close the consumer.
- poll(timeout=1.0)
Poll for a single message.
- Parameters:
timeout (float) – Maximum time to wait for a message in seconds (default: 1.0)
- Returns:
Message dict with keys
topic,partition,offset,key,value,timestamp,headers, or None if no message is available.- Return type:
dict or None
- Raises:
KafkaConsumerError – If a consumer error occurs
- stop()
Signal the stream() loop to stop at the next poll iteration.
- stream(timeout=1.0, count=None)
Generator that yields messages from subscribed topics.
Installs signal handlers for SIGINT and SIGTERM so that the first signal causes the loop to exit cleanly (within at most timeout seconds). Original handlers are restored when the generator exits.
- Parameters:
timeout (float) – Poll timeout in seconds per iteration (default: 1.0)
count (int, optional) – Maximum number of messages to yield. None means stream indefinitely.
- Yields:
dict – Message dicts (same format as
poll())
- subscribe(topics)
Subscribe to one or more Kafka topics.
Topic names are automatically namespaced with the instance name. For example, subscribing to
"gstlal"with instance name"gracedb-dev"subscribes to"gracedb-dev.gstlal".- Parameters:
topics (str or list) – Pipeline name or list of pipeline names
- Raises:
KafkaConsumerError – If subscription fails
Authentication helpers
- ligo.gracedb.kafka.auth.get_scitoken_for_kafka(broker_url, scope='gracedb.read', audience=None)
Find and return a valid SciToken for Kafka authentication.
This function discovers a SciToken using igwn-auth-utils and returns the serialized JWT string suitable for Kafka OAUTHBEARER authentication.
- Parameters:
broker_url (str) – Kafka broker URL (e.g., “localhost:9092”)
scope (str) – Required token scope (default: “gracedb.read”)
audience (str, optional) – Required audience. If None, extracted from broker_url
- Returns:
Serialized JWT token string
- Return type:
str
- Raises:
KafkaTokenError – If no token can be found, or if the only token available (read directly from disk) has already expired. The message names every location that was searched and how to obtain a fresh token.
- ligo.gracedb.kafka.auth.create_oauth_callback(broker_url, scope='gracedb.read', audience=None)
Create an OAuth callback function for confluent_kafka Producer.
The callback is called by the Kafka client to obtain the OAuth token before establishing a connection.
- Parameters:
broker_url (str) – Kafka broker URL
scope (str) – Required token scope
audience (str, optional) – Required audience
- Returns:
OAuth callback function that returns (token_str, expiry_time)
- Return type:
callable
- class ligo.gracedb.kafka.auth.SciTokenRefreshManager(broker_url, scope='gracedb.read', audience=None, refresh_buffer=300)
Manages automatic SciToken refresh for long-running Kafka producers.
This class monitors token expiration and triggers callbacks when tokens need to be refreshed.
- create_callback()
Create an OAuth callback that uses this manager.
- Returns:
OAuth callback function
- Return type:
callable
- get_token()
Get current token, refreshing if necessary.
- Returns:
Current valid token string
- Return type:
str
- Raises:
KafkaTokenError – If token cannot be obtained
Topic utilities
- ligo.gracedb.kafka.utils.derive_instance_name(service_url)
Extract the instance name from a GraceDB service URL.
The instance name is the first component of the hostname (i.e., everything before the first dot, or the full hostname if there is no dot).
- Parameters:
service_url (str) – GraceDB service URL (e.g.,
"https://gracedb-dev.ligo.org/api/")- Returns:
Instance name (e.g.,
"gracedb-dev")- Return type:
str
- Raises:
ValueError – If the hostname cannot be parsed from the URL
Examples
>>> derive_instance_name("https://gracedb-dev.ligo.org/api/") 'gracedb-dev' >>> derive_instance_name("https://gracedb.ligo.org/api/") 'gracedb' >>> derive_instance_name("https://localhost:8080/api/") 'localhost'
- ligo.gracedb.kafka.utils.make_topic_name(instance_name, pipeline)
Build an instance-namespaced Kafka topic name.
- Parameters:
instance_name (str) – GraceDB instance name (e.g.,
"gracedb-dev")pipeline (str) – Pipeline name (e.g.,
"gstlal")
- Returns:
Namespaced topic name (e.g.,
"gracedb-dev.gstlal")- Return type:
str
- ligo.gracedb.kafka.utils.parse_topic_name(namespaced_topic)
Split an instance-namespaced topic into its components.
- Parameters:
namespaced_topic (str) – Namespaced topic name (e.g.,
"gracedb-dev.gstlal")- Returns:
(instance_name, pipeline)- Return type:
tuple
- Raises:
ValueError – If the topic does not contain a dot separator
Exception classes
Exceptions for Kafka integration.
- exception ligo.gracedb.kafka.exceptions.KafkaAuthenticationError
The broker rejected the SciToken.
Typically the token has the wrong audience or scope, or it expired while the producer or consumer was running and no fresh token was written to disk in time. Raised by the next
produce(),flush(),ping()orpoll()after librdkafka reports the failure, and by the producer constructor when confluent-kafka reports it while the client is being created.
- exception ligo.gracedb.kafka.exceptions.KafkaConnectionError
The broker cannot be reached.
DNS failure, connection refused, TLS handshake failure, or the topic list could not be fetched at construction (
check_for_topics=True). Also raised when confluent-kafka refuses the producer configuration. The message carries librdkafka’s own description of the last error and a hint about what to check.
- exception ligo.gracedb.kafka.exceptions.KafkaConsumerError
A consumer-side error from
subscribe()orpoll().
- exception ligo.gracedb.kafka.exceptions.KafkaError
Base class for every exception raised by
ligo.gracedb.kafka.Catch this to handle any Kafka problem with one handler.
- exception ligo.gracedb.kafka.exceptions.KafkaMessageDeliveryError
A message was not delivered.
Raised by a strict
flush(),ping()andcreate_eventwhen the broker rejected a message, or when messages were still queued at the timeout and neither a connection error nor a topic error explains it.
- exception ligo.gracedb.kafka.exceptions.KafkaProducerError
confluent-kafka raised while producing or flushing.
For example a message larger than
message.max.bytes. The wrappedKafkaExceptiontext is included in the message.
- exception ligo.gracedb.kafka.exceptions.KafkaTokenError
No usable SciToken could be found.
Raised before any broker contact: by the producer constructor when
validate_token=True(the default), byping(), and inside the OAuth callback when a token is (re)loaded. The message lists every location searched, or the expiry time of the expired token found, and thehtgettokencommand that fixes it.
- exception ligo.gracedb.kafka.exceptions.KafkaTopicError
The target topic does not exist, or the token is not authorized for it.
Automatic topic creation is disabled on the GraceDB brokers, so this usually means a wrong
instance_name/service_urlor pipeline name. Raised by the producer constructor (check_for_topics=True; without a default topic it means the instance has no topics at all on the broker), byping(), and by a strictflush()orcreate_eventonce a stalled delivery has been traced to the topic. The message names the instances that do have topics on the broker where that is known.