IAM Consortium#
The “IAMC format” or “IAMC template” refers to a variety of similar data formats with a similar structure, developed by the Integrated Assessment Modeling Consortium (IAMC) and others, for data that is output from or input to integrated assessment models. Similar DSDs are commonly used in related research.
These data structures are characterized by:
Common dimensions, including:
MODEL
,SCENARIO
—identifying a model and particular configuration of that model.REGION
—geography.YEAR
—if in ‘long’ format; or also commonly a ‘wide’ format with one column per distinct year.UNIT
—in some cases this is effectively an attribute; in other cases, it may be a dimension. See below.VARIABLE
—see below.
Combination of several data flows in the same file:
Codes appearing in the
VARIABLE
dimension are strings with a varying number of parts separated by the pipe (“|”) character.The first (or only) part indicates the measure concept, for instance “Population”.
Subsequent parts are codes for additional dimensions.
For example in “Population|Female|50–59Y”, “Female” may be a code for a gender dimension, and “50–59Y” may be a code for an age dimension. IAMC data usually do not explicitly identify these dimensions, and separate structural information, if provided, is usually text and not in machine-readable formats.
Specification of “templates” in the form of files in the same format as the data, with no observation values. These provide code lists for the
VARIABLE
and sometimes other dimensions. These “templates” thus carry implicit structural information on the measures, dimensions, codes, etc. for the various data flows included.
Convert to/from SDMX#
transport_data.iamc
currently supports two types of conversions.
Create the SDMX structures implied by certain data in IAMC formats:
structures_for_data()
.Convert an SDMX data structure definition to a code list for the IAMC
VARIABLE
concept:variable_cl_for_dsd()
.This method may also be used to merge multiple DSDs, including from different maintainers, to a single list of “VARIABLE” codes.
Code reference#
Handle data and structure for IAMC-like formats.
Todo
Add a function to generate distinct DSDs for each data flow in a data set.
Todo
Add function(s) to reshape IAMC-like data.
Submodules
Functions
|
Make a codelist for the concept id, given data. |
Return common metadata for IAMC-like data and structures. |
|
|
|
|
Return IAMC-like data structures describing data. |
|
Create a DSD and code lists for a particular measure from variable parts. |
|
Make structures for IAMC-like "VARIABLE" data. |
|
Generate an SDMX codelist with IAMC "VARIABLE" codes corresponding to dsd. |
- transport_data.iamc.cl_for_data(data: Series, id: str, **ma_kwargs) Codelist [source]#
Make a codelist for the concept id, given data.
- Parameters:
ma_kwargs – Keyword arguments for
MaintainableArtefact
.
- transport_data.iamc.common_structures()[source]#
Return common metadata for IAMC-like data and structures.
- Returns:
with id “IAMC”, containing the concepts for the IAMC dimensions and attribute.
- Return type:
ConceptScheme
- transport_data.iamc.structures_for_data(data: DataFrame, base_id: str = 'GENERATED', maintainer: Agency | None = None) StructureMessage [source]#
Return IAMC-like data structures describing data.
- Parameters:
data (
pandas.DataFrame
) –Data in IAMC tabular format.
Either long (with a “YEAR” column) or wide (no “YEAR” column but one or more columns with
int
codes for the “YEAR” dimension).Column names in any case. Upper-cased column names must appear in the IAMC concept scheme (
get_iamc_structures()
).
maintainer (
Agency
, optional) – Maintainer to be associated with generatedMaintainableArtefact
.
- Returns:
including:
The “IAMC” ConceptScheme from
get_iamc_structures()
.A data structure definition with ID base_id.
One code list for each dimension.
A “MEASURE” concept scheme including the unique measures: that is, parts appearing before the first pipe (“|”) separator in the “VARIABLE” column of data.
For each measure: 0 or more code lists, each containing codes for a single dimension in the parts of “VARIABLE” column values beyond the first pipe (“|”) separator.
- Return type:
StructureMessage
- transport_data.iamc.structures_for_measure(measure: Concept, parts: DataFrame, **ma_kwargs) list [source]#
Create a DSD and code lists for a particular measure from variable parts.
- Parameters:
ma_kwargs – Keyword arguments for
MaintainableArtefact
.
- transport_data.iamc.structures_for_variable(data: Series, **ma_kwargs) list [source]#
Make structures for IAMC-like “VARIABLE” data.
- Parameters:
ma_kwargs – Keyword arguments for
MaintainableArtefact
.- Returns:
A sequence of SDMX structures.
- Return type:
- transport_data.iamc.variable_cl_for_dsd(dsd: BaseDataStructureDefinition, codelist: Codelist | None = None) Codelist [source]#
Generate an SDMX codelist with IAMC “VARIABLE” codes corresponding to dsd.
The dimensions of dsd are each enumerated by an associated codelist.
DataStructureDefinition.iter_keys
generates one key for each member of the Cartesian product of these sets.variable_cl_for_dsd()
collapses each of these into a code whose ID is a constructed IAMC “VARIABLE” name like “Primary Measure|Foo|Bar|Baz”. In these:The part before the first pipe (“Primary Measure|”) is the name of the concept identity for the primary measure of dsd.
The remaining parts (“Foo|Bar|Baz”) are the names of the codes for each dimension of dsd.
If the code lists that enumerate dsd contain codes with the special ID “_T”, then
variable_cl_for_dsd()
also generates corresponding IAMC-style names for aggregates. In the above example, if the third dimension contains a “_T”Every code also has annotations with the following IDs and text:
iamc-full-dsd
The URN of dsd. This allows an unambiguous association to the full-dimensionality data structure definition.
iamc-full-key
The
repr()
of adict
corresponding to a full resolution key, mapping dimension ID to code ID; for example “{‘dim_0’: ‘FOO’, ‘dim_1’: ‘BAR’, ‘dim_2’: ‘BAZ’}”. This allows to restore or recover a valid key within dsd, given the IAMC “VARIABLE”.
If codelist is supplied, the new codes are appended.
- Parameters:
dsd – Existing data structure definition.
codelist (
Codelist
, optional) – Existing code list.
- Returns:
The same object as codelist, if supplied, else a new code list with id “VARIABLE”.
- Return type:
Codelist