Structured Properties - DataHub OpenAPI v2 Guide

This guides walks through the process of creating and using a Structured Property using the v2 version of the DataHub OpenAPI implementation. Note that this refers to DataHub's OpenAPI version and not the version of OpenAPI itself.

Requirements:

• curl
• jq

Structured Property Definition

Before a structured property can be added to an entity it must first be defined. Here is an example structured property being created against a local quickstart instance.

Create Property Definition

Example Request:

curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/propertyDefinition' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "qualifiedName": "my.test.MyProperty01",
    "displayName": "MyProperty01",
    "valueType": "urn:li:dataType:datahub.string",
    "allowedValues": [
        {
            "value": {"string": "foo"},
            "description": "test foo value"
        },
        {
            "value": {"string": "bar"},
            "description": "test bar value"
        }
    ],
    "cardinality": "SINGLE",
    "entityTypes": [
        "urn:li:entityType:datahub.dataset"
    ],
    "description": "test description"
}' | jq

Read Property Definition

Example Request:

curl -X 'GET' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/propertyDefinition' \
  -H 'accept: application/json' | jq

Example Response:

{
  "value": {
    "allowedValues": [
      {
        "value": {
          "string": "foo"
        },
        "description": "test foo value"
      },
      {
        "value": {
          "string": "bar"
        },
        "description": "test bar value"
      }
    ],
    "qualifiedName": "my.test.MyProperty01",
    "displayName": "MyProperty01",
    "valueType": "urn:li:dataType:datahub.string",
    "description": "test description",
    "entityTypes": [
      "urn:li:entityType:datahub.dataset"
    ],
    "cardinality": "SINGLE"
  }
}

Delete Property Definition

⚠ Not Implemented ⚠

Applying Structured Properties

Structured Properties can now be added to entities which have the structuredProperties as aspect. In the following example we'll attach and remove properties to an example dataset entity with urn urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD).

Set Structured Property Values

This will set/replace all structured properties on the entity. See PATCH operations to add/remove a single property.

curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "properties": [
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
      "values": [
        {"string": "foo"}
      ]
    }
  ]
}' | jq

Patch Structured Property Value

For this example, we'll extend create a second structured property and apply both properties to the same dataset used previously. After this your system should include both my.test.MyProperty01 and my.test.MyProperty02.

curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty02/propertyDefinition' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "qualifiedName": "my.test.MyProperty02",
    "displayName": "MyProperty02",
    "valueType": "urn:li:dataType:datahub.string",
    "allowedValues": [
        {
            "value": {"string": "foo2"},
            "description": "test foo2 value"
        },
        {
            "value": {"string": "bar2"},
            "description": "test bar2 value"
        }
    ],
    "cardinality": "SINGLE",
    "entityTypes": [
        "urn:li:entityType:datahub.dataset"
    ]
}' | jq

This command will attach one of each of the two properties to our test dataset urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD).

curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "properties": [
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
      "values": [
        {"string": "foo"}
      ]
    },
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02",
      "values": [
        {"string": "bar2"}
      ]
    }
  ]
}' | jq

Remove Structured Property Value

The expected state of our test dataset include 2 structured properties. We'd like to remove the first one and preserve the second property.

curl -X 'PATCH' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json-patch+json' \
  -d '{
        "patch": [
            {
                "op": "remove",
                "path": "/properties/urn:li:structuredProperty:my.test.MyProperty01"
            }
        ],
        "arrayPrimaryKeys": {
            "properties": [
                "propertyUrn"
            ]
        }
      }' | jq

The response will show that the expected property has been removed.

{
  "urn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
  "aspects": {
    "structuredProperties": {
      "value": {
        "properties": [
          {
            "values": [
              {
                "string": "bar2"
              }
            ],
            "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02"
          }
        ]
      }
    }
  }
}

Add Structured Property Value

In this example, we'll add the property back with a different value, preserving the existing property.

curl -X 'PATCH' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json-patch+json' \
  -d '{
        "patch": [
            {
                "op": "add",
                "path": "/properties/urn:li:structuredProperty:my.test.MyProperty01",
                "value": {
                    "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
                    "values": [
                        {
                            "string": "bar"
                        }
                    ]
                }
            }
        ],
        "arrayPrimaryKeys": {
            "properties": [
                "propertyUrn"
            ]
        }
    }' | jq

The response shows that the property was re-added with the new value bar instead of the previous value foo.

{
    "urn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
    "aspects": {
        "structuredProperties": {
            "value": {
                "properties": [
                    {
                        "values": [
                            {
                                "string": "bar2"
                            }
                        ],
                        "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02"
                    },
                    {
                        "values": [
                            {
                                "string": "bar"
                            }
                        ],
                        "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01"
                    }
                ]
            }
        }
    }
}