The Spark integration uses OpenLineage to collect and send metadata to Grai. Complete documentation for integrating OpenLineage with Spark can be found here (opens in a new tab).

Web App

Spark Integration

Fields

FieldValueExample
sourceThe name of the source, see sourcesSpark
NameName for connectionproduction
NamespaceNamespace for the connection, see namespacesdefault
namespacesOptional

Webhook

ℹī¸

In order to authenticate your webhook, you will need to create an API key in the Grai web app.

Grai will open an endpoint at /api/v1/openlineage/<connection_id>/ which can be used to receive OpenLineage events.

See the OpenLineage docs (opens in a new tab) for how to configure your client to send OpenLineage events. As part of the configuration settings you'll need to to provide configuration settings via SparkConf parameters.

FieldValue
spark.openlineage.transport.typehttp
spark.openlineage.transport.endpointapi/v1/openlineage/<connection_id>
spark.openlineage.transport.urlhttps://api.grai.io (opens in a new tab)
spark.openlineage.transport.auth.type<your_grai_api_key>