read gdx through put_utilities gdxin in loop, but only if file exists

[pecenak21]

Hello,

We have a model which needs to read in hundreds of GDX files. However, to limit the overhead, we try to limit the number of files actually read in through only 'sending' it the files it needs. See the following statement:

Code: Select all

$if exist "<filename.csv>" $call csv2gdx "<filename.csv>" output="<filename.gdx>" id=Par_name ...
$if exist '<filename.csv>' Execute_load '<filename.gdx>' Par_name =Par_name ;

Here, we only send a handful of CSV's instead of the hundreds it could possibly ingest. However, as we expand the model, we are finding this is intractable to do for every file. We are trying to take advantage of file naming to make this process easier. Thus we are transitioning to using put_utilities to read the files in a loop. We are able to successfully create the GDX files in a loop with the following code:

Code: Select all

loop(fueltype,
       put_utility 'shell' / 'if exist ".\FuelLimit_' fueltype.tl:0 '_N1.csv" csv2gdx ".\FuelLimit_' fueltype.tl:0 '_N1.csv" output=".\FuelLimit_' fueltype.tl:0 '_N1.gdx" id=FuelLimitIn index=1,2 values=3..lastCol useHeader=y'
);

Code: Select all

loop(fueltype,
      put_utilities dummyFile 'gdxin' / '.\FuelLimit_' fueltype.tl:0 '_N1.gdx'
      execute_load dummyPar=FuelLimitIn;
      par_dgfuellimit_hourly(fueltype)= dummyPar;            
);

However, this code will fail if the GDX file does not exist. We have tried, but it seems there is no way to embed an 'if exists' in the last put_utilities sequence. Do you have any suggested workarounds here?

[pecenak21]

Hi team,

Any thoughts here would be appreciated.

Zack

[Fred]

Hi,

You could for example first create a list of the exitsing fuel type csv files. That list can be handled as subset of all the fuel types. Once you know that subset, you can loop over it to call csv2gdx and load the data from gdx only for the existing files, e.g.

Code: Select all

*create some dummy csv files for this example
$echo l > a.csv
$echo 2 > c.csv
$echo 3 > g.csv

*write little Gams Script that creates a list of exsiting fuel type csv files
$onecho > createListOfExistingFuelTypes.gms
$echo *list of existing fueltypes > fueltype_exist.inc
set fueltype /a,b,c,d,e,f,g/;
loop(fueltype,
   put_utility 'shell' / 'if exist "' fueltype.tl:0 '.csv" echo ' fueltype.tl:0 ' >> fueltype_exist.inc';
);
$offecho
$call.checkerrorlevel gams createListOfExistingFuelTypes.gms

*declare fueltype set and subset of existing fuel types
set fueltype /a,b,c,d,e,f,g/
    fueltype_exist(fueltype) /
$include fueltype_exist.inc
/ ;
display fueltype, fueltype_exist;

loop(fueltype_exist(fueltype),
  put_utility 'shell' / 'csv2gdx ...';
  put_utility 'gdxin' / ...;
  ...
);

I hope this helps!

Best,
Fred