Page 1 of 1

RGDX compress issue

Posted: Tue Feb 06, 2018 5:36 pm
by hermann.westerholt

I am having some trouble with the compress feature of the GDXMRW tool (GAMS 24.9) in Matlab (R2017b). I hope that the following piece of code can explain my problem (Matlab and GDX file also in the attachments):

Code: Select all

%% Introductory text for clarity:
% price.gdx contains parameters of price data, such as the German day-ahead price
% for electricity in 2016 ('DAP'). In the underlying model, a year consists
% of 52 weeks, and a week consists of 168 hours. When displayed in 'full'
% form, the parameter DAP(w,h) is structured accordingly:
% DAP_full.val = [52x168] double

guel = @(s,v) strcat(s,strsplit(num2str(v))); = 'DAP';
s.form = 'full';
s.uels = {guel('',1:52),guel('',1:168)};
DAP_full = rgdx('prices.gdx',s);

% So far, so good. Displaying the very same parameter in 'sparse' and 
% 'uncompressed' form, should yield a matrix with 52x168 = 8736 lines and
% 3 columns (1st col: week, 2nd col: hour, 3rd col: price).

s.compress = 'false';
s.form = 'sparse';
DAP_sparse = rgdx('prices.gdx',s);

% Unfortunately, DAP_sparse.val only contains 8735 entries. As can be
% verified with DAP_full, the price in hour 124 of week 5 is zero. This is
% the very same entry which is missing in DAP_sparse.val - it appears that
% setting the compress field to 'false' does not have the desired effect.
Any help with this issue is highly appreciated.

Best regards,

Re: RGDX compress issue

Posted: Sat Mar 03, 2018 1:06 am
by dirkse

Your example is perfect for illustrating what compress is for. If I look at the domain info for your data, I see there is none:

Code: Select all

C:\support\mrw>gdxdump prices.gdx domaininfo
SyNr  Type  DomInf Symbol
   4   Par    None AlP(*)
   1   Par    None DAP(*, *)
   3   Par    None PRLSP_Margin(*)
   2   Par    None PRLSP_Median(*)
You provided the domain info in the s.uels in your call to get the right-sized matrix on return: 52 x 168. If you had not done this, you would have gotten something square, of size 168 x 168, but all the rows after row 52 are completely zero. This is because your GDX file just contains data: there is no info about any index sets in your example.

Code: Select all

>> = 'dap';
>> f.form = 'full';
>> dap_fat = rgdx('prices',f)

dap_fat = 

    name: 'dap'
    type: 'parameter'
     dim: 2
     val: [168x168 double]
    form: 'full'
    uels: {{1x168 cell}  {1x168 cell}}
Setting compress = true just tells rgdx to squeeze out any zero rows or columns. This is just the thing in your case.

Code: Select all

>> f.compress = 'true';
>> dap_compr = rgdx('prices',f)

dap_compr = 

    name: 'dap'
    type: 'parameter'
     dim: 2
     val: [52x168 double]
    form: 'full'
    uels: {{1x52 cell}  {1x168 cell}}
If you had a data set where a zero row or column was good data and you wanted to preserve it, compress would be too much. Listing the UELs explicitly, as you did originally, would be the way to go then.

The compress option plays no role in the sparse case.

BTW, if you have domain information in the GDX file (i.e. sets weeks and hours, and DAP(weeks,hours)), then this gets easier. But this post is long enough. I can say more later if you're keen to know.


Re: RGDX compress issue

Posted: Mon Mar 05, 2018 12:37 am
by hermann.westerholt

thank you very much for your detailed response! I am indeed keen on learning more about GDXMRW and would be glad if you could answer the following questions:

1. I use the GDX file prices.gdx (see previous post) as input in my GAMS model. It has been created with the wgdx function. Am I right to assume that linking parameters to their index sets requires passing through GAMS and declaring the sets and parameters there? Small example:

Code: Select all

guel = @(s,v) strcat(s,strsplit(num2str(v))); = 'w';
set_w.type = 'set';
set_w.uels = {guel('',1:52)}; = 'h';
set_h.type = 'set';
set_h.uels = {guel('',1:168)}; = 'TwoDimParam';
param.type = 'parameter';
param.form = 'full';
param.dim = 2;
param.val = reshape((1:8736),[52,168]);
param.uels = [set_w.uels,set_h.uels];

wgdx('test.gdx', set_w, set_h, param);

Test.gdx contains both sets and the parameter TwoDimParam, but no indexing information:

Code: Select all

Symbol info of GDX test.gdx
  Index Type       Dim       NRecs  Name
      1 Set          1          52  w(*)        MATLAB data from GDXMRW
      2 Set          1         168  h(*)        MATLAB data from GDXMRW
      3 Parameter    2        8736  TwoDimParam(*,*)  MATLAB data from GDXMRW 

2. As odd as this question may sound: is there any way to combine explicit UEL listings and the sparse format in order to obtain a symbol's values (non-zeros and zeros!) in the given UEL range in sparse form? Personally, I find data in sparse form a lot easier to plot ... but I am afraid that my idea goes against the very definition of "sparse".

Again, thank your for your help!


Re: RGDX compress issue

Posted: Tue Mar 06, 2018 6:39 am
by dirkse

With GDXMRW, it is not possible to write a GDX file with domain information in it. The tools was developed prior to the advent of domain information in GDX files, and it hasn't been updated in this respect.

If we did anything in this direction, we would probably write a file with relaxed domain information. This is what we do with some other GDX utilities, e.g. GDXRRW.

Re: your question about how to get a "dense" output in sparse form (i.e. one row in the val matrix for each possible tuple of the parameter), this is possible, but you have to change the data a little. The sparse form just returns the values stored in GDX. Typically, there are only nonzero values stored in GDX, so there is one row for each nonzero element. But if you store the zeros in GDX, you'll see them in the sparse form. GAMS/Base won't actually store zeros in GDX, but it will store EPS, so if you tack this onto the trnsport example and read the parameters mat and mateps into Matlab from alldata.gdx, you'll see what I mean:

parameters mat(i,j) / 'seattle'.'topeka' 525 /, mateps(i,j);
mateps(i,j) = mat(i,j) + eps;
execute_unload 'alldata';