Server Lineage

Overview

Gravitino server provides a pluggable lineage framework to receive, process, and sink OpenLineage events. By leveraging this, you could do custom process for the lineage event and sink to your dedicated systems.

Lineage Configuration

Configuration item	Description	Default value	Required
gravitino.lineage.source	The name of lineage event source.	http	No
gravitino.lineage.${sourceName}.sourceClass	The name of the lineage source class which should implement org.apache.gravitino.lineage.source.LineageSource interface.	(none)	No
gravitino.lineage.processorClass	The name of the lineage processor class which should implement org.apache.gravitino.lineage.processor.LineageProcessor interface. The default noop processor do nothing about the run event.	org.apache.gravitino.lineage.processor.NoopProcessor	No
gravitino.lineage.sinks	The Lineage event sink names (support multiple sinks separated by commas).	log	No
gravitino.lineage.${sinkName}.sinkClass	The name of the lineage sink class which should implement org.apache.gravitino.lineage.sink.LineageSink interface.	(none)	No
gravitino.lineage.queueCapacity	The total capacity of lineage event queues. When there are multiple lineage sinks, each sink utilizes an isolated event queue. The capacity of each queue is calculated by dividing the value of gravitino.lineage.queueCapacity by the number of sinks.	10000	No

Lineage HTTP Source

Http source provides an endpoint which follows OpenLineage API spec to receive OpenLineage run event. The following use example:

cat <<EOF >source.json
{
  "eventType": "START",
  "eventTime": "2023-10-28T19:52:00.001+10:00",
  "run": {
    "runId": "0176a8c2-fe01-7439-87e6-56a1a1b4029f"
  },
  "job": {
    "namespace": "gravitino-namespace",
    "name": "gravitino-job1"
  },
  "inputs": [{
    "namespace": "gravitino-namespace",
    "name": "gravitino-table-identifier"
  }],
  "producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
  "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
}
EOF

curl -X POST \
  -i -H 'Content-Type: application/json' \
  -d '@source.json' \
  http://localhost:8090/api/lineage

Lineage Log Sink

Log sink prints the log in a separate log file gravitino_lineage.log, you could change the default behavior in conf/log4j2.properties.

Lineage HTTP Sink

The HTTP sink supports sending the lineage event to an HTTP server that follows the OpenLineage REST specification, like marquez

Property Name	Description	Default Value	Required
gravitino.lineage.sinks	Specifies the lineage sink implementation to use. For http sink http.	log	Yes
gravitino.lineage.http.sinkClass	Fully qualified class name of the http sink lineage sink implementation org.apache.gravitino.lineage.sink.LineageHttpSink)	org.apache.gravitino.lineage.sink.LineageLogSink	Yes
gravitino.lineage.http.url	URL of the http sink server endpoint for lineage collection(e.g., http://localhost:5000)	none	Yes
gravitino.lineage.http.authType	Authentication type for http sink (options: apiKey or none)	none	Yes
gravitino.lineage.http.apiKey	API key for authenticating with http sink (required if authType=apiKey)	none	No

High Watermark Status

When the lineage sink operates slowly, lineage events accumulate in the async queue. Once the queue size exceeds 90% of its capacity (high watermark threshold), the lineage system enters a high watermark status. In this state, the lineage source must implement retry and logging mechanisms for rejected events to prevent system overload. For the HTTP source, it returns the 429 Too Many Requests status code to the client.