Dear all
I'm currently solving a, relatively simple, linear model with the cplex solver, but for a huge amount of data. This makes for a very slow and timeconsuming solving of the model. Now, when I enter the task manager (multiple times during the process, so it's not a one time problem), I observe that the CPU utilization of the 2 Intel Xeon processors E52630 v3 (8 C, 2,4 GHz, 20 M, 85 W) (so 16 cores), never exceeds 10%, and often remains at 3 to 4%. Therefore, there's plenty of processing capacity, but I just can't access it some way or another...
I should say, I'm a university student and I'm using a university server, that I share with others. Their activity (see image attached) fluctuates (0%, 40% and sometimes 97% (when mine takes up the remaining 3%)), but no matter how much they use, the CPU utilization on my account never exceeds 10%.
I already tried to give GAMS high priority in the task manager, but no result. Furthermore, as I am using a university server, I do not have any administrator authorisation. If the problem can be fixed as an administrator, however, I can ask my supervisor to take those steps, but for now she seems to have no idea either.
Maybe it's not a GAMS related problem, maybe it is, but I thought I'd just pitch it here anyway!
Thanks a lot for any help you could provide!
Michiel
Low CPU utilization
Re: Low CPU utilization
Maybe some additional info: he's generating the MIP model right now (see image attached). Does it make sense that this takes up so much time?
Re: Low CPU utilization
Hi,
You can check in the lst file which equation is defined in line 94 and takes 2200 seconds to generate.
Without knowing anything about the model (like e.g. #rows, #columns, #nonzeros) and that equation in particular, it is impossible to say how long it should take to generate. If you can share your code, it might be possible to say more.
Note that you need to instruct the solver explicitly to use multiple threads (https://www.gams.com/latest/docs/UG_Gam ... SAOThreads).
I hope this helps!
Fred
You can check in the lst file which equation is defined in line 94 and takes 2200 seconds to generate.
Without knowing anything about the model (like e.g. #rows, #columns, #nonzeros) and that equation in particular, it is impossible to say how long it should take to generate. If you can share your code, it might be possible to say more.
Note that you need to instruct the solver explicitly to use multiple threads (https://www.gams.com/latest/docs/UG_Gam ... SAOThreads).
I hope this helps!
Fred
Re: Low CPU utilization
Hey Fred
Thanks a lot for your quick and accurate response, I tested the threads option on my personal computer and CPU utilization increased from 30% to 100%. However, unfortunately, it did not speed up the solving (tested it with a smaller set of data).
In attachment, you find the investment model for a European electricity system I built. Line 94 contains the fundamental equation for a stable network: energy generation needs to equal energy demand at every moment (8760 hours) and in every region (19 regions) (all nonzero values), so it makes sense this takes the most time to process. In line 114 I require the power flow from country L1 to L2 to be equal to the negative power flow from country L2 to L1.
Thanks for taking the time to help me out. If you need more information, please do not hesitate to ask!
Yours sincerely
Michiel
P.S. By now (I don't know exactly how long after initializing, as I did not continuously observed the logging file) the iteration process has started, but without the threads option. Do you reckon it would pay off to abort the process and restart it with 'option threads = 0'? Or is there a possibility to activate this option on the fly?
Thanks a lot for your quick and accurate response, I tested the threads option on my personal computer and CPU utilization increased from 30% to 100%. However, unfortunately, it did not speed up the solving (tested it with a smaller set of data).
In attachment, you find the investment model for a European electricity system I built. Line 94 contains the fundamental equation for a stable network: energy generation needs to equal energy demand at every moment (8760 hours) and in every region (19 regions) (all nonzero values), so it makes sense this takes the most time to process. In line 114 I require the power flow from country L1 to L2 to be equal to the negative power flow from country L2 to L1.
Thanks for taking the time to help me out. If you need more information, please do not hesitate to ask!
Yours sincerely
Michiel
P.S. By now (I don't know exactly how long after initializing, as I did not continuously observed the logging file) the iteration process has started, but without the threads option. Do you reckon it would pay off to abort the process and restart it with 'option threads = 0'? Or is there a possibility to activate this option on the fly?
 Attachments

 Investment_model.gms
 (8.21 KiB) Downloaded 12 times
Re: Low CPU utilization
Hi Michiel,
It seems that you have a MIP and an LP version of the model. Please note that the threads setting has different impacts on different model types (and solvers).
The following explanations refer to GAMS/CPLEX, since your code shows that this is the solver you are using. For more details, please see the CPLEX solver manual:https://www.gams.com/latest/docs/S_CPLEX.html
For LPs: If Threads is greater than 1 and no primalfeasible basis is available, the concurrent option will be used. [...] The concurrent option runs multiple methods in parallel. The first thread uses dual simplex. The second thread uses barrier. The next thread uses primal simplex. Remaining threads are used by the barrier run.
For MIPs you also solve an LP in the root node but then the number of threads also determines the number of subproblems solved in parallel during branch and bound (i.e. LPs in the nodes of the search tree).
Please note that more threads do not necessarily always lead to a speedup.
For MIPs, the phenomenon is known as "performance variability" and is really unavoidable. If you want to read more about this, here are a two pointers:
 http://www.thequestforoptimality.com/pe ... readings/
 Section 5 in MIPLIB 2010 paper http://mpc.zib.de/index.php/MPC/article/viewFile/56/28
For LPs the question is, whether you know in advance which LP algorithm performs best for your particular problem. If you do know that, it is recommended to choose that LP algorithm explicitly (https://www.gams.com/latest/docs/S_CPLE ... EXlpmethod) instead of using concurrent mode.
The problem you experience with slow model generation is completely independent from the threads option. Model generation is a sequential process.
I suspect that the slow model generation has to do with a nonoptimal order of index sets for variables and equations that makes it difficult for GAMS to efficiently iterate through the data structures when building the model. We would like to investigate this further. You are welcome to send the input data required to run the model to support@gams.com.
Best,
Fred
It seems that you have a MIP and an LP version of the model. Please note that the threads setting has different impacts on different model types (and solvers).
The following explanations refer to GAMS/CPLEX, since your code shows that this is the solver you are using. For more details, please see the CPLEX solver manual:https://www.gams.com/latest/docs/S_CPLEX.html
For LPs: If Threads is greater than 1 and no primalfeasible basis is available, the concurrent option will be used. [...] The concurrent option runs multiple methods in parallel. The first thread uses dual simplex. The second thread uses barrier. The next thread uses primal simplex. Remaining threads are used by the barrier run.
For MIPs you also solve an LP in the root node but then the number of threads also determines the number of subproblems solved in parallel during branch and bound (i.e. LPs in the nodes of the search tree).
Please note that more threads do not necessarily always lead to a speedup.
For MIPs, the phenomenon is known as "performance variability" and is really unavoidable. If you want to read more about this, here are a two pointers:
 http://www.thequestforoptimality.com/pe ... readings/
 Section 5 in MIPLIB 2010 paper http://mpc.zib.de/index.php/MPC/article/viewFile/56/28
For LPs the question is, whether you know in advance which LP algorithm performs best for your particular problem. If you do know that, it is recommended to choose that LP algorithm explicitly (https://www.gams.com/latest/docs/S_CPLE ... EXlpmethod) instead of using concurrent mode.
The problem you experience with slow model generation is completely independent from the threads option. Model generation is a sequential process.
I suspect that the slow model generation has to do with a nonoptimal order of index sets for variables and equations that makes it difficult for GAMS to efficiently iterate through the data structures when building the model. We would like to investigate this further. You are welcome to send the input data required to run the model to support@gams.com.
Best,
Fred