Edit data structures with the tdc CLI

Edit data structures with the tdc CLI#

Introduction#

This guide explains how to use the tdc edit command-line interface (CLI) to edit SDMX structural artefacts.

These artefacts (for example, data structure definitions, data flow definitions, code lists, concept schemes, etc.) are usually recorded in the SDMX-ML format (based on XML, or Extensible Markup Language). This format is both machine- and human-readable, but it is not very user-friendly or easy to edit manually.

tdc edit provides one rudimentary, interactive way to generate and store SDMX-ML containing SDMX artefacts. These can then be used with other, TDC-compliant (meta)data. For example, one can define a data structure definition, and then use it to read an SDMX-CSV file that is formatted accordingly.

Create a new SDMX artefact#

This is currently the only feature provided by tdc edit.

  1. Start by ensuring you have transport_data installed.

  2. Run tdc edit.

    A full-screen command-line interface is shown:

    • At the top is some information about what is happening, and the artefact you are editing.

    • Below the line is a prompt. Always read the prompt text carefully.

    • At the bottom is a reminder that you can press Control-C at any time to exit. If you make a mistake in editing, exit and start again from this step. (It is not currently supported to ‘undo’ or go back to previous steps.)

From this point, follow the steps in one of the subsections below.

Create an item scheme (Agency Scheme, Code list, or Concept Scheme)#

  1. Choose the class to create.

    Type one of 1, 2, or 3. Type Enter.

  2. Enter information about the item scheme.

    Like every ‘maintainable’ SDMX artefact, an item scheme has the following attributes. These allow to distinguish the artefact from others, and for a responsible person (you!) to maintain it by publishing different versions.

    Type each one as prompted, followed by Enter.

    • Maintainer ID. This will be TDCI, or if you are maintaining structures for another organization, the ID (not full name) of that organization.

    • ID of the item scheme itself. See the Data standards for information about what to use here.

    • Name of the item scheme. This is optional.

    • Version. 1.0.0 is a safe default or initial value. If you enter a version that is already in use, the existing stored SDMX artefact will be overwritten. You may choose to do this deliberately, or choose a different version to avoid this.

    These three pieces together are used to construct the Uniform Resource Name (URN) of the item scheme. The URN can be used to locate and identify the item scheme later. The full URN looks like:

    urn:sdmx:org.sdmx.infomodel.datastructure.Codelist=TDCI:CL_EXAMPLE(1.0.0)

    transport_data supports using shortened URNs, for example Codelist=TDCI:CL_EXAMPLE(1.0.0) (specific version) or Codelist=TDCI:CL_EXAMPLE (implicitly the latest version).

  3. Add 0 or more items to the scheme, or edit current items.

    A list of the current items in the scheme (if any) is shown.

    Type either n for a new item or the number of an existing item; followed by Enter.

  4. Enter information about the new or existing item.

    Items within item schemes are ‘identifiable’ SDMX artefacts. These have fewer attributes than ‘maintainable’ artefacts (such as the parent item scheme):

    Type each one as prompted, followed by Enter.

    • ID of the item.

    • Name of the item. This is optional.

    Type Enter again to finish the entry of this item.

  5. Repeat steps (5) and (6) as many times as needed. Then, type Enter to finish.

  6. Save the created item scheme.

    Type y followed by Enter. The created SDMX artefact is saved to the local store.

    You can confirm this by using tdc store command and subcommands to query the store, for example:

    # Show all artefacts with "TDCI" as maintainer
tdc store list --maintainer=TDCI

# Show a particular artefact
tdc store show "Codelist=TDCI:CL_EXAMPLE(1.0.0)"

The program exits. To create or edit other structures, run tdc edit again.

Create a data structure definition (DSD)#

  1. Choose to create a new DSD.

    Type 5, followed by Enter.

  2. Enter information about the DSD. This is the same as step (4) in the Item Scheme section, above.

  3. Add 1 or more DSD dimensions.

    Type the ID of each dimension, followed by Enter.

    After the last dimension, type Enter (with no text) to finish the entry of dimensions.

  4. Add the DSD measure.

    The ‘measure’ is answers the question “What is measured by each observation value?” A broader SDMX convention is to use the ID OBS_VALUE and store elsewhere (in metadata) a reference to a concept (vehicle sales; energy consumption; etc.) that describes the actual measure.

    Type OBS_VALUE followed by Enter.

    Type Enter again to finish the entry of measures. [1]

  5. Add 0 or more DSD attributes.

    An attribute stores information about observations, other than their value. For example, information that an observation’s value is estimated is stored as an attribute. Attributes can be attached to individual observations, to groups of observations, or to entire data sets. (tdc edit does not yet support specifying these.)

    Some attributes commonly used in SDMX applications include:

    • OBS_STATUS: Observation status (usually for individual observations).

    • UNIT_MEASURE: Units of measurement (usually for entire data sets/flows).

    • COMMENT

    Type the ID of each attribute, followed by Enter.

    After the last attribute, type Enter (with no text) to finish the entry of attributes.

  6. Save the created DSD. This is the same as step (8) in the Item Scheme section, above.

Create a data flow definition (DFD)#

  1. Choose to create a new DFD.

    Type 4, followed by Enter.

  2. Enter information about the DFD. This is the same as step (4) in the Item Scheme section, above.

  3. Enter the URN for the DSD that gives the structure of data sets in this data flow.

    For example, type DataStructureDefinition=TDCI:EXAMPLE(1.0.0), followed by Enter.

    The referenced URN must already be present in your local store of SDMX artefacts.

  4. Save the created DFD. This is the same as step (8) in the Item Scheme section, above.