Troubleshooting
Running into problems using the client? This page contains some suggestions and pieces of information that will be useful if you need to contact the developers.
Read the error message
If the client raised ligo.gracedb.exceptions.HTTPError, the server has almost
certainly told you what was wrong, but str(e) hides it: it prints only <Response [403]>.
Catch the exception and look at e.status_code, e.reason and e.text:
from ligo.gracedb.exceptions import HTTPError
try:
client.write_label('S250101a', 'ADVREQ')
except HTTPError as e:
print(f'{e.status_code} {e.reason}: {e.text}')
raise
A 4xx code means the request needs to change (input, credentials or permissions); a 5xx
code means the server, or the path to it, is the problem.
Handling errors explains how to interpret the message and which codes are
worth retrying.
Check package version
Is your version of the package out of date? Check the version and find its release date in the Changelog. If it’s older than ~1 year, you may want to upgrade and see if that resolves your issue.
There are a variety of ways to check the package version, listed here in order of ease:
With the command-line tool:
gracedb --version
If you have installed the package with pip:
pip freeze | grep ligo-gracedb
If you have checked out the git repository:
python setup.py --version
Check your connection to the server
Can you connect to the server at all?
If you’re using the Python module, instantiate a ligo.gracedb.rest.GraceDb object as you normally would and do:
response = client.ping()
print(response.status)
If you’re using the command-line tool, do:
gracedb ping
Note that you will need to use the service_url keyword argument to the client constructor, or the --service-url argument for the command-line tool, if you do not want to use the production GraceDB server.
If you don’t get an HTTP response with a 200 status code, there is likely a problem with your connection to the server.
Check your credentials
Is the client using the credentials that you intend it to use?
If you’re using the Python module, instantiate a ligo.gracedb.rest.GraceDb object as you normally would and do:
client.show_credentials()
If you’re using the command-line tool, do:
gracedb credentials client
When a SciToken is in use, the output reports the parameters of the token the client found:
the subject (sub), the expiration time in human-readable form, the scope, and the audience.
For example:
$ gracedb credentials client
{
"scitoken": {
"SciToken subject": "albert.einstein@ligo.org",
"SciToken expiration": "2025-03-07T19:32:43",
"SciToken scope": "gracedb.read",
"SciToken audience": "ANY"
}
}
Check that the expiration is in the future, that the scope includes gracedb.read
(see SciToken Scopes), and that the audience is either ANY or the
server you are talking to. If the token has expired or is missing, run htgettoken -a vault.ligo.org -i igwn.
Does the output match what you expect in terms of authentication type and credentials? If not, check the description of how authentication credentials can be provided to the client class here. For the command-line client, see here.
Check your user account on the server
Is the server identifying you with the user account that you expect it to?
If you’re using the Python module, instantiate a ligo.gracedb.rest.GraceDb object as you normally would and do:
client.get_user_info()
If you’re using the command-line tool, do:
gracedb credentials server
Note that you will need to use the service_url keyword argument to the client constructor, or the --service-url argument for the command-line tool, if you do not want to use the production GraceDB server.
Does the output match the user account you would expect to use on the server? If it does not, and you have confirmed that the client is using the correct credentials and authentication type in the previous step, then there may be a server-side issue.
Kafka producer problems
The Kafka producer checks your SciToken and the broker’s topic list when it is constructed, and every Kafka exception message says what to check and how to fix it. If you see:
KafkaTokenError— runhtgettoken -a vault.ligo.org -i igwn.KafkaTopicError— compareinstance_nameorservice_urlwith the instances listed in the message, and check the pipeline name.KafkaConnectionError— check the broker address, network access andca_cert_path.KafkaAuthenticationError— the broker rejected the token; check its scope and audience, and that it was refreshed before expiry.
See Error handling for the full list, Startup checks for the flags that control the construction-time checks, and GraceDB Kafka brokers for which broker serves which instance. Broker-side problems (a broker that cannot be reached, a missing topic, or a permission you believe you should have) go to the IGWN Computing Help Desk.
SASL errors every time the token is renewed
A long-running producer that logs a burst of librdkafka errors at the moment the SciToken expires, and then carries on normally, looks like this:
12:00:04 INFO Token refreshed, expires at 1790769605
%3|1790769604.964|FAIL|...: SASL OAUTHBEARER mechanism handshake failed: Broker: Request not valid in current SASL state ...
%3|1790769605.229|FAIL|...: SASL authentication error: {"status":"invalid_token", "scope":"Token expired at Wed Sep 30 12:00:05 UTC 2026 (current time: Wed Sep 30 12:00:05 UTC 2026)"}
12:00:05 INFO Refreshing SciToken
This means the token file on disk was not replaced until the old
token had already expired. The client never mints tokens: the refresh
manager (see Token reloading) only re-reads whatever
htgettoken (or your own renewal job) last wrote to disk. When it
re-reads the file reload_buffer seconds before expiry and finds the
same token, it hands librdkafka the same expiry again, librdkafka
schedules the next attempt at 80% of the remaining lifetime, and the
attempts converge on the expiry time. The final one, a second or so
before expiry, still finds the old token; the broker rejects it; the
next reconnect finally picks up the new file. The Token refreshed
line is logged on every re-read, even when the token did not change.
The usual cause is a cron job running htgettoken with a
--minsecs value that is too small. htgettoken only fetches a
new token when fewer than --minsecs seconds of the old one remain
(default 60), so --minsecs must be larger than the cron interval
plus the few seconds the fetch itself takes; otherwise the run before
expiry skips the renewal and the next one lands after expiry. In the
example above, the token was issued at 09:00:05 and the cron job ran
hourly with --minsecs 3500: at 11:00:00 the token had 3605 s left,
so the renewal was skipped, and at 12:00:00 it had 5 s left. --minsecs 3600 fails
the same way. Leave a generous margin, for example on an hourly cron
job:
htgettoken -a vault.ligo.org -i igwn --minsecs 4000
The file then always holds a token with well over an hour left when the refresh manager re-reads it, and the errors disappear. A value at or above the token lifetime simply causes a renewal on every run, which is also fine.
The errors themselves are benign as long as the gap is short:
librdkafka reconnects, re-authenticates and delivers the queued
messages. An upload whose flush_timeout (default 10 s) overlaps
the gap raises KafkaMessageDeliveryError (or falls back to HTTP
from create_event) rather than being lost silently.
Still having problems?
If you are still having problems, try to find a more experienced user who can help you sort out your issue. If that resource is not available to you, or if you still cannot resolve your issue, LIGO-Virgo-KAGRA users can reach out over the GraceDB Mattermost channel or by creating an issue in the ligo-gracedb GitLab repo.
Bug reporting
Did you find a bug? Please create an issue on the project page and include as much information as possible. Some examples of useful information are:
Operating system
Client version
An error traceback, including the HTTP status code and the server’s message (
e.status_codeande.texton theHTTPError)What you were doing when the error occurred
What you expected the result to be
Specific steps to reproduce the issue