Data Ingestion

The current version of our API, allows for datasets to be uploaded into our platform. However, we used a different internal process for ingesting data from public sources, which allows us to perform data transformations, aggregation, and further manipulation. We are now isolating such processes allowing for data to be "Adapted" into the final Dataset. As we do that, we are also storing every raw dataset version, and keeping track of every difference.

This new data ingestion platform also allows working with data that has multiple dimensions. In the past, we would only accept Date, and a general Entity dimension, that were used across different Variables in the Dataset:

Date	Entity	Production	Imports	Exports
2020-01-01	USA	2123	23000	25000
2020-02-01	Canada	3223	24000	27000
2020-03-01	Mexico	5423	22000	21000

Our API now works with "Columns" rather than Variables, and each column can be specified as Entity, allowing for composed keys, or in other words multiple dimensions:

Date	Country	State	Production	Import	Export
2020-01-01	USA	Florida	2123	23000	25000
2020-02-01	USA	California	3223	24000	27000
2020-03-01	USA	Texas	5423	22000	21000

Data Pipelines

Having a better detail of the structure of each dataset, allows for richer data manipulation scenarios, which we will be enabling as Pipelines:

A Pipeline is triggered whenever a participating Dataset gets updated. Then, individual columns of each dataset are extracted as defined by the pipeline, data is transformed, aggregated or combined, and new Datasets can be created. This may sound like complex scenarios, but we are working to make it super simple to combine data from multiple datasets on our web portal.