User guide
This section provides information about using the client class. Detailed information on the API is available on the API page, and on the command-line interface at Command-line interface.
Creating a client instance
ligo.gracedb.rest.GraceDb is the client class and is used for all API interactions.
To instantiate a client object, do:
from ligo.gracedb.rest import GraceDb
client = GraceDb()
By default, the client uses the API of the production GraceDB server, https://gracedb.ligo.org/api/.
To specify another server, use the service_url keyword argument:
client = GraceDb(service_url='https://gracedb-playground.ligo.org/api/')
More information about the client class and the options which can be provided to the constructor is covered in detail on the API page.
Credentials
Credentials are not required, but users who possess them will likely want to authenticate in order to gain full access to the GraceDB server.
SciTokens
This space is dedicated to collecting information regarding SciTokens and best practices for using them in LVK workflows. This document should be considered a work in progress and will be updated periodically.
SciToken Scopes
The “scope” of a SciToken defines what a user can do with that token against various services.
Currently, the gracedb.read scope is necessary to map a token to an existing LVK
account in GraceDB. A user with that token’s scope will have all the capability that they
would have with an X.509 certificate: viewing/annotating internal events and superevents,
uploading/downloading data products, etc. There is a planned, but not implemented yet,
capability to map event uploading privileges to SciToken scopes (i.e., gracedb.write:gstlal).
In the time being, please contact a GraceDB administrator to request permission to upload
events.
Getting a Token
Individual (i.e., non-robot) LIGO-Virgo-KAGRA members can obtain a SciToken for use with GraceDB by running the following terminal command:
htgettoken -a vault.ligo.org -i igwn --scopes=gracedb.read
A SciToken keytab is required to get a SciToken for a robot account, which can be obtained
by going to robots.ligo.org and selecting option
4. SciToken keytab.
The form at robots.ligo.org asks for the following information:
Robot Service Name:
{robot_service_name}, generally the name of the analysis pipeline. In the case where more than one keytab is needed (e.g., if the pipeline is running multiple computing centers), consider a naming scheme such as{robot_service_name}-cit.FQDN of the host:
{FQDN}, the fully-qualified domain name of the server where the service will run. This could be access point (login/head node) of an analysis pipeline, or the host of a robot web service. Hint: runhostname -fon the node to get the FQDN.Intended Scopes: The complete set of permissions that the robot will have. Note that scopes can be further restricted each time a SciToken is requested. See this page for a a list of services and their scopes. The output of the
htgettokencommand displays the list of scopes of an existing token;Primary, secondary, and tertiary owner’s
@ligo.orgemail addresses: specify multiple redundant owners in case one or more members leaves the collaboration.
When the request is processed and the keytab is created, the requestor will be “@-mentioned”
in a Computing Helpdesk Ticket and
should receive an email. The ticket contains the keytab’s principal, and should be saved for
later. The principal has the form {robot_service_name}-scitoken/robot/{FQDN}@LIGO.ORG.
If this SciToken is intended to be used to upload events, please contact a GraceDB administrator to add the permission, and the pipeline to the robot account. After the request is processed, you will be contacted with instructions on how to obtain the keytab and request a SciToken. An example can be found here.
Getting an Initial Token for a New Robot
Consult the relevant section in the IGWN Computing Guide.
The “OIDC Workflow” (browser authentication) is required the first time a token is retrieved for a new robot. Run the following command, and then follow the link specified in the output in your browser:
htgettoken --vaultserver vault.ligo.org --issuer igwn --role {robot_service_name}-scitoken --credkey {robot_service_name}-scitoken/robot/{FQDN}
Note that the “role” is everything before the “/”in the keytab principal, and
the “credkey” is the principal with “@LIGO.ORG” at the end.
The htdestroytoken command destroys the initial token.
Getting a Token Using Kerberos
Consult the relevant section in the IGWN Computing Guide.
Once the igwn-robot-auth package is available, igwn-roboot-get will replace the
combination of kinit + htgettoken combination for getting a robot SciToken. See
the igwn-robot-auth
documentation for
up-to-date instructions. Meanwhile:
Use
kinitto get Kerberos credentials with the keytab:kinit {robot_service_name}-scitoken/robot/{FQDN}@LIGO.ORG -k -t /path/to/keytab
Get a SciToken using the Kerberos credentials. In this example,
--nooidcis added to ensure the command fails if Kerberos credentials cannot be used:htgettoken --nooidc --vaultserver vault.ligo.org --issuer igwn --role {robot_service_name}-scitoken --credkey {robot_service_name}-scitoken/robot/{FQDN}
Note that limited scopes can be requested by adding the
--scopesoption with a comma- or space-delimited list of scopes, e.g.,--scopes "gwdatafind.read gracedb.read read:/staging read:/ligo"
The htdecodetoken command can be used to inspect the active token.
Using a Token with GraceDB
The GraceDB client code is configured to automatically find tokens that were created using the above
methods. To examine a token using the command line, run the htdecodetoken command:
$ htdecodetoken
{
"aud": "ANY",
"sub": "albert.einstein@ligo.org",
"uid": "albert.einstein",
"ver": "scitoken:2.0",
"nbf": 1741296149,
"scope": "gracedb.read",
"iss": "https://cilogon.org/igwn",
"exp": 1741306954,
"iat": 1741296154,
"jti": "https://cilogon.org/oauth2/2feca395f73d9e6b7844ef07163869ea?type=accessToken&ts=1741296153907&version=v2.0&lifetime=10800000"
}
You can also use the gracedb credentials client command to verify that the client code
has found the token, and to verify the token expiraiton in a human-readable format:
$ gracedb credentials client
{
"scitoken": {
"SciToken subject": "albert.einstein@ligo.org",
"SciToken expiration": "2025-03-07T19:32:43",
"SciToken scope": "gracedb.read",
"SciToken audience": "ANY"
}
}
And then use the gracedb credentials server command to verify that the token’s subject
successfully maps to a user account on GraceDB:
$ gracedb -s https://gracedb-test.ligo.org/api/ credentials server
{
"username": "albert.einstein@ligo.org",
"first_name": "Albert",
"last_name": "Einstein",
"email": "albert.einstein@ligo.org",
"is_internal_user": true
"upload_to_pipeline": [
"gstlal"
]
}
If the output of this command is {"username": "AnonymousUser"}, then likely the client
code cannot find a valid SciToken, and should be refreshed with the htgettoken command.
If the output is otherwise unexpected, open a ticket on the
IGWN Helpdesk.
Tokens for Long-Running GraceDB Jobs
A SciToken has a lifetime of three hours, and so must be periodically refreshed
by a cron job or similar process. An example script to incorporate into long-running
workflows, such as search pipelines can be found here.
Additionally, the REST client must be configured to reload the new token, similar to
workflows that used X.509 certificates and the reload_certificate keyword argument. A
complete listing can be found in the API documentation for ligo.gracedb.rest.GraceDb,
but a brief overview of the relevant options can be found below:
reload_cred: set toTrueto reload a SciToken for a client session. The client code will check before each request if the validity period of the token is withinreload_bufferseconds, and if so will reload the token. A valid token must be in place initially when using this option, or the client code will fail with anIgwnAuthError.reload_buffer: the buffer (in seconds) for reloading a SciToken before it expires.fail_if_noauth: set to True to have the client code fail with anIgwnAuthErrorif no valid token can be found when reloading. If set to False, then the client will proceed with anAnonymousUsersession, which was the behavior with X.509 reloading. Note that if the client code proceeds with an unauthenticated session, workflows will likley fail with 403 (Unauthorized) and 502 errors.
SciToken Usage and Terminology
Use of SciTokens for IGWN services with HTCondor benefits from a more nuanced understanding of SciTokens terminology:
Previous references to a “token” are specificially a “bearer token”, or “access token”. A bearer token is used to authenticate to services and is valid for three hours.
A “vault token” is also stored on the access point, and has a default lifetime of one week.
The
htgettokencommand first attempts to use an existing vault token to get a bearer token. If a valid vault token cannot be found,htgettokenmust authenticate using either Kerberos or the OIDC workflow in order to get a vault token. The vault token can then be reused to get new bearer tokens without further authentication until the vault token expires.HTCondor can run
htgettokenperiodically to give a new bearer token to a job and keep it valid for the runtime of the job, which is accomplished by HTCondor storing its own vault token internally.HTCondor’s internal vault token will need to be renewed periodically in order to continue to request bearer tokens for jobs.
The
condor_vault_storercommand tells HTCondor to runhtgettoken. Options provided in the place of{OPTIONS}in the following command will be passed intohtgettoken:condor_vault_storer "igwn&options={OPTIONS}"
In order to ensure that HTCondor’s internal vault token is always valid, periodically run
condor_vault_storer with the additional --vaulttokenttl and --vaulttokenminttl
flags. The following commands will get Kerberos credentials and then tell HTCondor to run
htgettoken to renew the vault token regardless of whether the existing vault token is
valid:
kinit {robot_service_name}-scitoken/robot/{FQDN}@LIGO.ORG -k -t {path_to_keytab}
condor_vault_storer "igwn&options=--nooidc --vaultserver vault.ligo.org --issuer igwn --role {robot_service_name}-scitoken --credkey {robot_service_name}-scitoken/robot/{FQDN} --vaulttokenttl 1000000s --vaulttokenminttl 999999s"
Note that in the above example, 1000000s (about 11.5 days) is the maximum time allowed for a normal vault token file. In order to run it automatically:
Put the commands in a shell script, and give the script execute permissions (e.g., with
chmod +x).Run the script periodically as part of a
cronjob or similar task. Note that the script should be run more frequently than the interval specified with the--vaulttokenminttl
A copy of HTCondor’s internal vault token is created and stored in /tmp/vt_u${UID}; inspecting
the age of this file tells the age of HTCondor’s vault token.
SciTokens, GraceDB, and HTCondor
HTCondor can only store robot credentials on an access point configured to use the IGWN issuer. Users who intend to use a host currently configured as an AP issuer must create a new Helpdesk ticket to request that it be changed to the IGWN issuer. For more information on SciToken issuers, see the relevant section of the Computing Guide.
The relevant section in the computing guide on using SciTokens in HTCondor jobs is here; the following should be used as a primer for getting started.
Add the following lines to a job’s submit description file in order to use SciTokens:
use_oauth_services = igwn
igwn_oauth_options = --nooidc --vaultserver vault.ligo.org --issuer igwn --role {robot_service_name}-scitoken --credkey {robot_service_name}-scitoken/robot/{FQDN} --outfile /tmp/bt_u${UID}-igwn
HTCondor’s internal vault token must be valid. The --outfile option sets the location
for a temporary file created by HTCondor when the job is submitted. DAGS may fail to submit
jobs for logged-out users if this option is not set; the file is not used by running jobs.
The directory and filenames in this example follow the conventions used for other tokens
created by condor_vault_storer.
Scopes can be limited with the following, where limited has been chosen for the “handle”:
igwn_oauth_permissions_limited = gracedb.read read:/ligo
Adding the above will allow other lines in the file, such as transfer_input_files, to
reference the token.
Add the following environment variable in order to use the token during runtime and interact with GraceDB:
environment = "BEARER_TOKEN_FILE=$$(CondorScratchDir)/.condor_creds/igwn_limited.use"
Note that if igwn_oauth_permissions_* is not used, so that there is no handle, the
filename in the environment variable should just be igwn.use.
X.509 certificates
NOTE: X.509 certificates are being deprecated and will be removed in a future release.
Response contents
Most of the client methods return a requests.models.Response object.
The server API provides a JSON response which is loaded into a Python dict by using the json() method:
response = client.event('G1234')
data = response.json()
The exact structure of the resulting dictionary varies depending on which resource type you are retrieving (event, superevent, log entry, etc.). A variety of examples are available in the server code documentation.
Note that most client methods will raise a ligo.gracedb.exceptions.HTTPError if the response code indicates an error (>= 400); see Handling errors.
Handling errors
When the server answers a request with an error status code (400 or above), the client raises
ligo.gracedb.exceptions.HTTPError rather than returning the response.
This is a subclass of requests.exceptions.HTTPError, so code that already catches the
requests exception keeps working.
GraceDB is careful to say why a request was rejected, and that explanation is in the body of the error response. Reading it is the first step in diagnosing any failure, and it resolves the large majority of problems without any help from the developers.
Read the message
Printing the exception on its own is not enough. str(e) only gives the response object:
>>> client.write_label('S250101a', 'ADVREQ')
Traceback (most recent call last):
...
ligo.gracedb.exceptions.HTTPError: <Response [403]>
The useful information is carried as attributes of the exception:
Attribute |
Contents |
|---|---|
|
HTTP status code, e.g. |
|
Short status phrase from the server, e.g. |
|
Body of the response: the server’s explanation of what went wrong |
|
The full |
So the minimal pattern is: import the exception, wrap the call, and print the status and the body:
from ligo.gracedb.rest import GraceDb
from ligo.gracedb.exceptions import HTTPError
client = GraceDb()
try:
response = client.write_label('S250101a', 'ADVREQ')
except HTTPError as e:
print(f'{e.status_code} {e.reason}: {e.text}')
raise
which turns the traceback above into:
403 Forbidden: {"detail": "You do not have permission to perform this action."}
Re-raising (or exiting) after printing is deliberate: a bare except: that swallows the error
and carries on is the most common way a pipeline ends up silently doing nothing.
What the message looks like
The body is almost always JSON, but its shape depends on which check failed. Some representative examples:
A problem the GraceDB code itself detects is a dictionary with an
errorkey:400 Bad Request: {"error": "Please provide a valid pipeline."}
A permission or lookup problem is a dictionary with a
detailkey:403 Forbidden: {"detail": "You do not have permission to perform this action."} 404 Not Found: {"detail": "Not found."}
A field-level validation error is flattened to the message itself, or to a list of messages when more than one field is wrong:
400 Bad Request: "This field is required."
A
500, or a502/503/504from the load balancer, is usually an HTML page rather than JSON. The HTML carries no useful detail; see the table below for what to do.
Because of this variety, treat e.text as the authoritative, human-readable message and only
parse it as JSON when you need to branch on its content:
def error_message(e):
"""Best-effort extraction of the server's explanation."""
try:
body = e.response.json()
except ValueError:
# Not JSON (e.g. an HTML error page); trim it for logs
return e.text[:200]
if isinstance(body, dict):
return body.get('error') or body.get('detail') or str(body)
return str(body)
Acting on the status code
The status code tells you who has to fix the problem.
Code |
Meaning |
What to do |
|---|---|---|
400 |
Bad request |
The input is wrong. The body says which field or value; fix it and resend. Retrying unchanged will not help. |
401 / 403 |
Not authenticated / Forbidden |
Credentials are missing, expired, or lack the required permission. See Check your credentials and Check your user account on the server. |
404 |
Not found |
The event, superevent, log, or file does not exist, or you
cannot see it. Check the ID and the |
409 |
Conflict |
The resource already exists or is being changed concurrently. The client retries this automatically a few times. |
413 |
Payload too large |
The upload exceeds the server’s size limit. |
429 |
Too many requests |
Slow down; honour |
500 |
Internal server error |
A bug on the server. Report it with the request details. |
502 / 504 |
Bad gateway / timeout |
Transient. The client already retries these; if it still fails, wait and retry. |
503 |
Service unavailable |
With a |
A typical pattern for a pipeline that uploads events separates “my fault” from “try again later”:
import logging
import sys
from ligo.gracedb.rest import GraceDb
from ligo.gracedb.exceptions import HTTPError
logger = logging.getLogger(__name__)
client = GraceDb(service_url='https://gracedb-playground.ligo.org/api/')
try:
response = client.create_event('CBC', 'gstlal', 'coinc.xml')
except HTTPError as e:
logger.error('Event upload rejected: %s %s: %s',
e.status_code, e.reason, e.text)
if e.status_code in (401, 403):
sys.exit('Check your SciToken or X.509 credentials '
'and your permissions on the server.')
elif 400 <= e.status_code < 500:
sys.exit('Fix the request before resending.')
else:
# 5xx after the client's own retries: give up for now,
# let the scheduler try again later
raise
else:
graceid = response.json()['graceid']
logger.info('Created %s', graceid)
Notes
Only HTTP errors become
HTTPError. Problems reaching the server at all (DNS, TLS, refused connection, timeouts) are raised byrequestsasrequests.exceptions.ConnectionErrororrequests.exceptions.Timeout. Catchrequests.exceptions.RequestExceptionas well if you want one handler for both.The client retries
408,409,502,504, and503-with-Retry-Afterbefore the exception reaches you, so an exception with one of those codes means the retries were exhausted. Seeligo.gracedb.client.GraceDbClientfor theretriesandretry_codesarguments.The command-line tool already does all of this:
gracedbprints the status, reason and message of any error and exits with status 1.If the message does not make sense to you, or the response is a
500or an HTML page, then it is time to ask for help. Include the status code, the fulle.text, the URL (e.response.url) and what you were trying to do; see Bug reporting.
Basic usage for events
Create an event
Create an event on the server with the ligo.gracedb.rest.GraceDb.create_event() method:
response = client.create_event('CBC', 'gstlal', '/path/to/event/data/file.xml', search='AllSky')
Any LVK user can create “test” events by specifying the first argument (analysis group) as 'Test'.
Special permissions are required to upload non-test events.
You can get a list of available analysis groups with ligo.gracedb.rest.GraceDb.groups, available analysis pipelines with ligo.gracedb.rest.GraceDb.pipelines, or a list of searches with ligo.gracedb.rest.GraceDb.searches.
The type of data file depends on which pipeline is submitting the event. If you are not a pipeline expert, you may want to download a file from an existing event and use that for creating test events.
Update an event
Update an event’s parameters by uploading a new event data file with the ligo.gracedb.rest.GraceDb.replace_event() method:
response = client.replace_event('G123456', '/path/to/new/event/data/file.xml')
Only the user who originally submitted the event is allowed to update it.
Search for events
Use the ligo.gracedb.rest.GraceDb.events() method to search for events and get an iterator:
event_iterator = client.events('far < 1e-10')
graceids = [event['graceid'] for event in event_iterator]
See the server documentation for help with constructing search queries. Note that test and MDC events are not included in the search results by default.
Retrieve an individual event
The ligo.gracedb.rest.GraceDb.event() method is used to get information about an individual event:
response = client.event('G1234')
Basic usage for superevents
Create a superevent
Use the ligo.gracedb.rest.GraceDb.create_superevent() method to create a superevent:
response = client.create_superevent(1238350677, 1238350680, 1238350682, 'G1234')
The event G1234 must already exist on the server, and the category (production, test, or MDC) of the superevent must match that of the event.
Special permissions are required to create non-test superevents.
Update a superevent
Use the ligo.gracedb.rest.GraceDb.update_superevent() method to update a superevent:
response = client.update_superevent('S190202p', t_0=1238350700, preferred_event='G2345')
Note that this method is used to update the t_start, t_0, t_end, and preferred_event parameters of the superevent only.
To add events to or remove events from a superevent, use ligo.gracedb.rest.GraceDb.add_event_to_superevent() or ligo.gracedb.rest.GraceDb.remove_event_from_superevent() methods.
Search for superevents
Use the ligo.gracedb.rest.GraceDb.superevents() method to search for superevents and get an iterator:
superevent_iterator = client.superevents('public is_gw: True')
superevent_ids = [superevent['superevent_id'] for superevent in superevent_iterator]
See the server documentation for help with constructing search queries. Note that test and MDC superevents are not included in the search results by default.
Retrieve an individual superevent
The ligo.gracedb.rest.GraceDb.superevent() method is used to get information about an individual superevent:
response = client.superevent('S180331d')
Annotations
Both events and superevents can be annotated with log entries, file uploads, labels, and more. There are several methods attached to the client class for creating these annotations and for getting information about existing annotations. Most of these methods work for both events and superevents and automatically determine which one you are interested in based on the format of the ID provided.
Log entries
To create a log entry, use the ligo.gracedb.rest.GraceDb.write_log() method:
response = client.write_log('G1234', 'comment on event')
You can also upload a file with a log entry:
response = client.write_log('S190202b', 'comment on event', filename='/path/to/file/')
A label can be added to a log’s parent object (event/superevent) by specifying the label name
in the label= keyword argument to write_log().
To get a log or list of logs attached to an event or superevent, use the ligo.gracedb.rest.GraceDb.logs() method:
response = client.logs('G1234', 12)
Here the second argument is the index number of the log, which represents its chronological order in the entire set of logs for the event or superevent. If this argument is not provided, the entire set of logs for the event or superevent will be retrieved.
Files
The procedure for uploading files is covered in the previous section on log entries.
To retrieve a list of files attached to an event or superevent, use ligo.gracedb.rest.GraceDb.files():
response = client.files('G1234')
If a specific filename is provided, the server will provide the contents of the file with that name that is associated with the given event or superevent:
response = client.files('G1234', 'skymap.fits.gz')
file_contents = response.read()
Note that files are the server are versioned; i.e., uploading a file with the same name as another file attached to the same event or superevent will generate a new version.
Different versions are specified by appending ,N to the filename, where N is the version number.
Filenames without a version always point to the latest version of the file.
Labels
Labels can be applied to or removed from events or superevents. They are generally used to indicate some sort of state. Examples:
ADVREQ: advocate signoff is requestedPE_READY: parameter estimation results are availableDQV: the event or superevent is vetoed due to poor data quality
New labels cannot be created by users.
To see which labels are available on the server, use ligo.gracedb.rest.GraceDb.allowed_labels.
A list of labels with descriptions is provided here.
To add a label, use ligo.gracedb.rest.GraceDb.write_label():
response = client.write_label('G1234', 'ADVREQ')
To remove a label, use ligo.gracedb.rest.GraceDb.remove_label():
response = client.remove_label('S190214g', 'INJ')
Other methods
Additional methods are provided for managing signoffs, uploading EM observation data, and more. These are not covered in detail here because they are rather specialized options. See API for information about the client methods used for these actions.