IAM Consortium

The “IAMC format” or “IAMC template” refers to a variety of similar data formats with a similar structure, developed by the Integrated Assessment Modeling Consortium (IAMC) and others, for data that is output from or input to integrated assessment models. Similar DSDs are commonly used in related research.

These data structures are characterized by:

• Common dimensions, including:

  • MODEL, SCENARIO —identifying a model and particular configuration of that model.

  • REGION —geography.

  • YEAR —if in ‘long’ format; or also commonly a ‘wide’ format with one column per distinct year.

  • UNIT —in some cases this is effectively an attribute; in other cases, it may be a dimension. See below.

  • VARIABLE —see below.

• Combination of several data flows in the same file:

  • Codes appearing in the VARIABLE dimension are strings with a varying number of parts separated by the pipe (“|”) character.

    • The first (or only) part indicates the measure concept, for instance “Population”.

    • Subsequent parts are codes for additional dimensions.

    For example in “Population|Female|50–59Y”, “Female” may be a code for a gender dimension, and “50–59Y” may be a code for an age dimension. IAMC data usually do not explicitly identify these dimensions, and separate structural information, if provided, is usually text and not in machine-readable formats.

• Specification of “templates” in the form of files in the same format as the data, with no observation values. These provide code lists for the VARIABLE and sometimes other dimensions. These “templates” thus carry implicit structural information on the measures, dimensions, codes, etc. for the various data flows included.

Convert to/from SDMX

transport_data.iamc currently supports two types of conversions.

1. Create the SDMX structures implied by certain data in IAMC formats: structures_for_data().

2. Convert an SDMX data structure definition to a code list for the IAMC VARIABLE concept: variable_cl_for_dsd().

   This method may also be used to merge multiple DSDs, including from different maintainers, to a single list of “VARIABLE” codes.

Code reference

Handle data and structure for IAMC-like formats.

Todo

Add a function to generate distinct DSDs for each data flow in a data set.

Todo

Add function(s) to reshape IAMC-like data.

Submodules

transport_data.iamc.cli

CLI for iamc.

Functions

cl_for_data(data, id, **ma_kwargs)	Make a codelist for the concept id, given data.
common_structures()	Return common metadata for IAMC-like data and structures.
get_agency()
structures_for_data(data[, base_id, maintainer])	Return IAMC-like data structures describing data.
structures_for_measure(measure, parts, ...)	Create a DSD and code lists for a particular measure from variable parts.
structures_for_variable(data, **ma_kwargs)	Make structures for IAMC-like "VARIABLE" data.
variable_cl_for_dsd(dsd[, codelist])	Generate an SDMX codelist with IAMC "VARIABLE" codes corresponding to dsd.

transport_data.iamc.cl_for_data(data: Series, id: str, **ma_kwargs) → Codelist

Make a codelist for the concept id, given data.

Parameters:

ma_kwargs – Keyword arguments for MaintainableArtefact.

transport_data.iamc.common_structures()

Return common metadata for IAMC-like data and structures.

Returns:

with id “IAMC”, containing the concepts for the IAMC dimensions and attribute.

Return type:

ConceptScheme

transport_data.iamc.get_agency()

transport_data.iamc.structures_for_data(data: DataFrame, base_id: str = 'GENERATED', maintainer: Agency | None = None) → StructureMessage

Return IAMC-like data structures describing data.

Parameters:

• data (pandas.DataFrame) –

  Data in IAMC tabular format.

  • Either long (with a “YEAR” column) or wide (no “YEAR” column but one or more columns with int codes for the “YEAR” dimension).

  • Column names in any case. Upper-cased column names must appear in the IAMC concept scheme (get_iamc_structures()).

• maintainer (Agency, optional) – Maintainer to be associated with generated MaintainableArtefact.

Returns:

including:

• The “IAMC” ConceptScheme from get_iamc_structures().

• A data structure definition with ID base_id.

• One code list for each dimension.

• A “MEASURE” concept scheme including the unique measures: that is, parts appearing before the first pipe (“|”) separator in the “VARIABLE” column of data.

• For each measure: 0 or more code lists, each containing codes for a single dimension in the parts of “VARIABLE” column values beyond the first pipe (“|”) separator.

Return type:

StructureMessage

transport_data.iamc.structures_for_measure(measure: Concept, parts: DataFrame, **ma_kwargs) → list

Create a DSD and code lists for a particular measure from variable parts.

Parameters:

ma_kwargs – Keyword arguments for MaintainableArtefact.

transport_data.iamc.structures_for_variable(data: Series, **ma_kwargs) → list

Make structures for IAMC-like “VARIABLE” data.

Parameters:

ma_kwargs – Keyword arguments for MaintainableArtefact.

Returns:

A sequence of SDMX structures.

Return type:

list

transport_data.iamc.variable_cl_for_dsd(dsd: BaseDataStructureDefinition, codelist: Codelist | None = None) → Codelist

Generate an SDMX codelist with IAMC “VARIABLE” codes corresponding to dsd.

The dimensions of dsd are each enumerated by an associated codelist. DataStructureDefinition.iter_keys generates one key for each member of the Cartesian product of these sets.

variable_cl_for_dsd() collapses each of these into a code whose ID is a constructed IAMC “VARIABLE” name like “Primary Measure|Foo|Bar|Baz”. In these:

• The part before the first pipe (“Primary Measure|”) is the name of the concept identity for the primary measure of dsd.

• The remaining parts (“Foo|Bar|Baz”) are the names of the codes for each dimension of dsd.

If the code lists that enumerate dsd contain codes with the special ID “_T”, then variable_cl_for_dsd() also generates corresponding IAMC-style names for aggregates. In the above example, if the third dimension contains a “_T”

Every code also has annotations with the following IDs and text:

iamc-full-dsd

The URN of dsd. This allows an unambiguous association to the full-dimensionality data structure definition.

iamc-full-key

The repr() of a dict corresponding to a full resolution key, mapping dimension ID to code ID; for example “{‘dim_0’: ‘FOO’, ‘dim_1’: ‘BAR’, ‘dim_2’: ‘BAZ’}”. This allows to restore or recover a valid key within dsd, given the IAMC “VARIABLE”.

If codelist is supplied, the new codes are appended.

Parameters:

• dsd – Existing data structure definition.

• codelist (Codelist, optional) – Existing code list.

Returns:

The same object as codelist, if supplied, else a new code list with id “VARIABLE”.

Return type:

Codelist