[Opm] MPI Parallel OPM Flow - SPE 10 Model 2 Hangs

Thu Sep 27 08:01:12 UTC 2018

Hi Mohamad,

I just ran the first time steps of this case with Red Had 6 and OpenMPI, both with one and with four MPI processes. It works over here. I am running on a binary compiled from git yesterday, so I have not checked the release version of flow. So at this point, simply sharing that your problem does not happen on my set-up.

Generally speaking, SPE10 model 2 may not be the best model to use. It was created as an upscaling case with exaggerated heterogeneities to deliberately make it difficult to run. We currently do not include it in any of our integration testing, so the probability of having severe convergence issues with default tuning and solvers in flow is high. The SPE9 or Norne model is far better for benchmarking purposes.

If you have particular interest in SPE10 model 2, there are a couple of steps you should take. One is to re-write it into a two-phase case (when it was created we did not provide support for running two-phase cases, so gas was added with zero saturation). Then you should test with the multigrid based solver provided through a command line switch (please check the manual for documentation). Lastly, you may also want to play with tolerances. Currently we have relatively strict tolerances by default, and time steps with convergence failure is not accepted by default. This means that hard-to-converge models may fail to run through. Hence, it may not be what you want for SPE10 model 2.

Cheers,
Alf

From: Opm <opm-bounces at opm-project.org> On Behalf Of sindimo
Sent: 26. september 2018 21:43
To: Markus Blatt <markus at dr-blatt.de>; opm at opm-project.org
Subject: Re: [Opm] MPI Parallel OPM Flow - SPE 10 Model 2 Hangs

Dear Dr. Blatt,

Thank you for your reply, I really appreciate it.

I am running that model as part of my PhD research work at MIT, it's one of the main models I am benchmarking.

I was suspecting this might be an issue with Flow RedHat7/MPICH build but I am able to reproduce the same problem even on an Ubuntu machine with OpenMPI.

Below are step by step instructions on how to reproduce the hang on a freshly installed AWS Ubuntu machine in case this helps with investigating it further. Similarly this is reproducible on a freshly installed AWS RedHat7 machine (but using yum to install flow).

Thanks again.

Sincerely,

Mohamad Sindi

#--------------Steps to reproduce problem-------------------------------------

#Launch a fresh new machine instance on Amazon AWS for Ubuntu Server 16.04 (e.g. m4.2xlarge with 32 GB RAM and 4 cores)

#Install OPM Flow using instructions from website (https://opm-project.org/?page_id=245):
sudo apt-get update
sudo apt-get install software-properties-common
sudo apt-add-repository ppa:opm/ppa
sudo apt-get update
sudo apt-get install mpi-default-bin
sudo apt-get install libopm-simulators-bin

#Get spe10model2 model from https://github.com/OPM/opm-data/tree/master/spe10model2

#These are the md5sum for the files used just to make sure we are running the exact same model:

ubuntu at ip-172-31-42-209:~/spe10model2$ ls
SPE10_MODEL2.DATA  SPE10MODEL2_PERM.INC  SPE10MODEL2_PHI.INC  SPE10MODEL2_TOPS.INC

ubuntu at ip-172-31-42-209:~/spe10model2$ md5sum *
322c2d60c8f6a7982fc03a42630ba001  SPE10_MODEL2.DATA
72b5f0f1c8fe3ef133bd533c784ff479  SPE10MODEL2_PERM.INC
9bcc43dbaa2670b4ab2190fef3664310  SPE10MODEL2_PHI.INC
2a24fda84f073e96af295b8116400fd3  SPE10MODEL2_TOPS.INC

#Run the model which hangs (seems during cell partitioning):

ubuntu at ip-172-31-42-209:~/spe10model2$  mpirun  --mca btl tcp,self  -np 4   /usr/bin/flow    SPE10_MODEL2.DATA    output_dir=out_parallel
**********************************************************************
*                                                                    *
*                        This is flow 2018.04                        *
*                                                                    *
* Flow is a simulator for fully implicit three-phase black-oil flow, *
*             including solvent and polymer capabilities.            *
*          For more information, see https://opm-project.org          *
*                                                                    *
**********************************************************************

After loadbalancing process 0 has 322630 cells.

On Wed, Sep 26, 2018 at 4:10 AM Markus Blatt <markus at dr-blatt.de<mailto:markus at dr-blatt.de>> wrote:
Hi

On Tue, Sep 25, 2018 at 09:30:26PM -0400, sindimo wrote:
> I am running OPM Flow with MPICH MPI on RedHat 7 (installed via yum through
> OPM repo, version 2018.04).
>
> I am able to successfully run in parallel  some of the test models  (norne,
> spe5, spe9), however spe10model2 always hangs when I try to launch it. It
> seems it hangs during the cell partitioning as below. It just partially
> does the partitioning on a subset of the processes only (example below
> where I have 4 processes and it hangs after doing 2 partitions, I've also
> tried with 8 processors and it shows similar behavior). Any help with this
> is much appreciated as I need to run SPE10 for some work I am doing, many
> thanks!

May I ask what work that is?

So you are using a release (even of a target distribution). That is a bit weired.
Unfortunately I do not have access to such a system and cannot be of much help here.
Maybe somebody else can do a quick test?

But I did a quick test with the current master on my system and it works with 4 processes.
So if nobody else can help you, then you might want to checkout master and compile
OPM yourself.

Cheers,

Markus

--
Dr. Markus Blatt - HPC-Simulation-Software & Services http://www.dr-blatt.de
Pedettistr. 38, 85072 Eichstätt, Germany,  USt-Id: DE279960836
Tel.: +49 (0) 160 97590858
_______________________________________________
Opm mailing list
Opm at opm-project.org<mailto:Opm at opm-project.org>
https://opm-project.org/cgi-bin/mailman/listinfo/opm

-------------------------------------------------------------------
The information contained in this message may be CONFIDENTIAL and is
intended for the addressee only. Any unauthorized use, dissemination of the
information or copying of this message is prohibited. If you are not the
addressee, please notify the sender immediately by return e-mail and delete
this message.
Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <//opm-project.org/pipermail/opm/attachments/20180927/c72612f2/attachment.html>