Removing .dat files for Python GamsModelInstace use

Questions on using the GAMS APIs (Python, .NET, etc.)
Post Reply
nechiforv
User
User
Posts: 3
Joined: 1 year ago

Removing .dat files for Python GamsModelInstace use

Post by nechiforv » 1 year ago

Hello,

I am running a GAMS CNS model using the Python API GamsModelInstance. The model is run over 100k times and, from a certain point, my RAM memory runs out.

The strategy to avoid this is to delete the model instance (using __del__) whenever possible and then recreate it. While the memory issue seems to be solved, I am now generating thousands of folders with .dat files.

I tried finding an option to suppress these folders whenever the model instance is deleted but couldn't find anything.

Thanks in advance!

Best wishes,
Victor

User avatar
bussieck
Moderator
Moderator
Posts: 615
Joined: 4 years ago

Re: Removing .dat files for Python GamsModelInstace use

Post by bussieck » 1 year ago

Hi,

The point of GAMSModelInstance it to create it once and run it 100k times. But without knowing your use case (and the code) it is hard to recommend a change here.

The file creation depends on several things, e.g. if you use a single GAMSWorkspace or multiple, also if this is a single threaded or multithreaded application. It would be best if you could share a dummy example that demonstrates how you use GAMSModelInstance.

-Michael

nechiforv
User
User
Posts: 3
Joined: 1 year ago

Re: Removing .dat files for Python GamsModelInstace use

Post by nechiforv » 1 year ago

Hi Michael,

Thank you for these initial thoughts.

I am posting a dummy version below. The GAMS model is defined as a CNSModel object. The model is thus first initialised with a workspace, checkpoint and options. Then a model instance using GamsModelIinstance is initialised using initialise_instance() with a parameter "a" that will be changed during the multiple simulations and a variable X that is tracked.

The model is then run from outside through the calculate() method by getting values for parameter "a" (or several) from a different non-GAMS model. calculate() is called for a predefined number of iterations (e.g. 50) and the value of X is cumulated over these successive model runs. Once the max number of iterations is reached, the reset() method is called which calls instance_delete() and then a new instance_initialise().
The process goes on for these iterated simulations for 100k times.

This is where I get a bit stuck - whenever the reset() method is called there is a new folder with .dat files that is created. The process is also slowed down given the IO operations.
Indeed, the advantage of using GamsModelInstance is that the GAMS model instance would not need to be deleted and reinitialised. However, if I simplify the code below and do not use the instance_delete() and instance_initialise() tandem, the RAM resource use is adding up and after 80k runs the process crashes as my memory is running out.

Therefore, I'm looking either at freeing up resources along the way when only one instance initialisation for the whole simulation process is used (ideally), or finding an option to remove the multiple .dat files once they are no longer needed.

Thank you.

Victor



class CNSModel(Model):

def __init__(self, *args, **kwargs):

self.filename = kwargs.pop('model_filename', None)
self.fullpath = str(os.getcwd()+"/"+self.filename)
self.iteration = 0

self.workspace = GamsWorkspace(str(os.getcwd()), debug=DebugLevel.Off)

self.model = self.workspace.add_job_from_file(self.fullpath)

self.opt = self.workspace.add_options()
self.opt.all_model_types = "ipopt"

self.cp = self.workspace.add_checkpoint(os.path.basename(str(self.name+"checkpoint")))

### running to obtain some base values
self.model.run(self.opt,self.cp)
self.instance_initialise()

def instance_initialise(self):

self.mi = self.cp.add_modelinstance()

self.a = self.mi.sync_db.add_parameter("A",0,"parameter A")
....

modifier_list = [GamsModifier(self.a),etc.]

self.x = self.mi.sync_db.add_variable("X",0, VarType.Unknown)

self.mi.instantiate("CNSMODEL USING CNS", self.modifier_list,self.opt)

self.iteration = 0


### default value for a
self.a.add_record().value = 0

### initial cumulative value for X
self.cumulative_x = 0


def instance_delete(self):
self.mi.__del__()

def calculate(self):
self.a.clear()
### obtain the value of a - method inherited from superclass Model enabling the connection of the CNSModel to another model generating the value of "a"
self.a.add_record().value = self.obtain_value_a()

### same for other modifiers
...

self.mi.solve()
self.cumulative_x += self.mi.sync_db.get_variable("X")[()].level
self.iteration += 1


def reset(self):
self.instance_delete()
self.instance_initialise()

User avatar
bussieck
Moderator
Moderator
Posts: 615
Joined: 4 years ago

Re: Removing .dat files for Python GamsModelInstace use

Post by bussieck » 1 year ago

Viktor,

The initialize is a costly function. It runs a GAMS process to create scratch file (the dat files) and initializes the model instance from those. Instead of breaking the sequence 5000k solves into 100k initialize/reset it might be worth seeing where the memory leak comes from. I can reproduce a constant increase in memory when just running a modified transport7 (https://www.gams.com/latest/docs/API_PY ... G_MODIFIER) with anything else than CplexD or Gurobi. These use a different method of being called. I uploaded the example python script:
mem.zip
(1.42 KiB) Downloaded 123 times
We (GAMS) need to investigate.

As a workaround: Run without reset/initialize and observe the memory consumption (https://stackoverflow.com/questions/938 ... on-process). If this gets too big, quit and leave enough information where to continue from. Restart the entire process from there.

I'll file an issue on this, but this might not be as simple as one hopes...

-Michael

nechiforv
User
User
Posts: 3
Joined: 1 year ago

Re: Removing .dat files for Python GamsModelInstace use

Post by nechiforv » 1 year ago

Hi Michael,

Thank you for the suggestions.
Indeed, reinitialising the model instance only when the memory gets full significantly reduces the number of .dat folders generated.

Best wishes,
Victor

Post Reply