Ignore input data for elements not defined Topic is solved

[AndrewC]

I have previously used Excel and the utility gdxxrw for reading in our input data. This has the wonderful side effect of ignoring any entries in the data tables that are defined for elements it doesn't recognise.

For example if I have 2 tables, one with the domain and one with data for that domain, it will first read the domain and then next read the data but ignore any entries it doesn't recognise. In the example below it ignores the data entry for BOSTON and just populates the parameter for HOUSTON and NEWYORK
<I typed this all out by hand for demo purposes, it may contain syntax errors>

Table 1 in Excel:
HOUSTON
NEWYORK

Table 2 in Excel
HOUSTON 10
BOSTON 20
NEWYORK 30

set city;
parameter myData(city);
$call GDXXRW.exe "test.xlsx" output=test_gdx.gdx @excel_dataRead.txt
$gdxin test.xlsx
$load city
$load myData
$gdxin

Display city, myData;

-------------------------------------
---- 22 SET city

HOUSTON, NEWYORK

---- 22 PARAMETER myData

HOUSTON 10.000, NEWYORK 30.000
=====================================================

For a new project the data is going to be supplied in .inc files. I would like something similar to happen. If the supplied city.inc file only contains HOUSTON and NEWYORK then when it reads myData.inc, which contains data for other cities as well, it will just drop those entries.

e.g
file city.inc:
HOUSTON
NEWYORK

file myData.inc:
HOUSTON 10
BOSTON 20
NEWYORK 30

set city
/
$include city.inc
/

set myData
/
$include myData.inc
/


Unfortunately when the above code encounters the unknown BOSTON city it gives me a domain error:

12 parameter myData(city)
13 /
INCLUDE myData.inc
15 HOUSTON 10
16 BOSTON 20
**** $170
17 NEWYORK 30
18 /;

170 Domain violation for element


Is there a switch or option that says something like "when reading data and you encounter a domain violation just skip that entry and move to the next one" ??

Thanks in advance
Andy C

[Fred]

Hi,

With a GDX data layer, you have lots of flexibility when it comes to loading the data into GAMS. Besides the filtered read you get with $load, you can for example turn on domain checking when items are loaded with $loadDC or merge data into an already defined symbol with $loadM, etc. For more details see https://www.gams.com/latest/docs/UG_Dol ... DOLLARload

$include just inserts the contents of the specified include file. So that is the same as if the domain violation would be in some hard coded data and complex operations like a filtered read are not possible in that case.

So even if you get the input data as text files, I suggest to introduce a GDX data layer and then load the data from there as needed. The following self-contained example is based on yours and explains how to do that.

Code: Select all

$onecho > city.inc
HOUSTON
NEWYORK
$offecho

$onecho > myData.inc
HOUSTON 10
BOSTON 20
NEWYORK 30
$offecho

$onecho > createGDX.gms
set city /
$include city.inc
/;
parameter myData(*) /
$include myData.inc
/;
$offecho

$call gams createGDX gdx=allData.gdx
$ife errorlevel<>0 $abort Problem running craeteGDX.gms

set city;
parameter mydata(city);
$gdxin allData.gdx
$load city mydata
display city, mydata;

The trick is to have a little GAMS program that reads the include files and creates a gdx file but in that program, symbols are defined over the universe (https://www.gams.com/latest/docs/UG_Set ... iversalSet) such that you do not get domain violations.
In your actual GAMS program, you can then use the actuak domain sets and do filtered read etc.

I hope this helps!

Fred