[Opm] MPI Parallel OPM Flow - SPE 10 Model 2 Hangs

[sindimo]

Dear Dr. Blatt,

Thank you for your reply, I really appreciate it.

I am running that model as part of my PhD research work at MIT, it's one of
the main models I am benchmarking.

I was suspecting this might be an issue with Flow RedHat7/MPICH build but I
am able to reproduce the same problem even on an Ubuntu machine with
OpenMPI.

Below are step by step instructions on how to reproduce the hang on a
freshly installed AWS Ubuntu machine in case this helps with investigating
it further. Similarly this is reproducible on a freshly installed AWS
RedHat7 machine (but using yum to install flow).

Thanks again.

Sincerely,

Mohamad Sindi


#--------------Steps to reproduce
problem-------------------------------------

#Launch a fresh new machine instance on Amazon AWS for Ubuntu Server 16.04
(e.g. m4.2xlarge with 32 GB RAM and 4 cores)

#Install OPM Flow using instructions from website (
https://opm-project.org/?page_id=245):
sudo apt-get update
sudo apt-get install software-properties-common
sudo apt-add-repository ppa:opm/ppa
sudo apt-get update
sudo apt-get install mpi-default-bin
sudo apt-get install libopm-simulators-bin


#Get spe10model2 model from
https://github.com/OPM/opm-data/tree/master/spe10model2

#These are the md5sum for the files used just to make sure we are running
the exact same model:

ubuntu at ip-172-31-42-209:~/spe10model2$ ls
SPE10_MODEL2.DATA  SPE10MODEL2_PERM.INC  SPE10MODEL2_PHI.INC
SPE10MODEL2_TOPS.INC

ubuntu at ip-172-31-42-209:~/spe10model2$ md5sum *
322c2d60c8f6a7982fc03a42630ba001  SPE10_MODEL2.DATA
72b5f0f1c8fe3ef133bd533c784ff479  SPE10MODEL2_PERM.INC
9bcc43dbaa2670b4ab2190fef3664310  SPE10MODEL2_PHI.INC
2a24fda84f073e96af295b8116400fd3  SPE10MODEL2_TOPS.INC

#Run the model which hangs (seems during cell partitioning):

ubuntu at ip-172-31-42-209:~/spe10model2$  mpirun  --mca btl tcp,self  -np 4
 /usr/bin/flow    SPE10_MODEL2.DATA    output_dir=out_parallel
**********************************************************************
*                                                                    *
*                        This is flow 2018.04                        *
*                                                                    *
* Flow is a simulator for fully implicit three-phase black-oil flow, *
*             including solvent and polymer capabilities.            *
*          For more information, see https://opm-project.org          *
*                                                                    *
**********************************************************************

After loadbalancing process 0 has 322630 cells.









On Wed, Sep 26, 2018 at 4:10 AM Markus Blatt <markus at dr-blatt.de> wrote:

> Hi
>
> On Tue, Sep 25, 2018 at 09:30:26PM -0400, sindimo wrote:
> > I am running OPM Flow with MPICH MPI on RedHat 7 (installed via yum
> through
> > OPM repo, version 2018.04).
> >
> > I am able to successfully run in parallel  some of the test models
> (norne,
> > spe5, spe9), however spe10model2 always hangs when I try to launch it. It
> > seems it hangs during the cell partitioning as below. It just partially
> > does the partitioning on a subset of the processes only (example below
> > where I have 4 processes and it hangs after doing 2 partitions, I've also
> > tried with 8 processors and it shows similar behavior). Any help with
> this
> > is much appreciated as I need to run SPE10 for some work I am doing, many
> > thanks!
>
> May I ask what work that is?
>
> So you are using a release (even of a target distribution). That is a bit
> weired.
> Unfortunately I do not have access to such a system and cannot be of much
> help here.
> Maybe somebody else can do a quick test?
>
> But I did a quick test with the current master on my system and it works
> with 4 processes.
> So if nobody else can help you, then you might want to checkout master and
> compile
> OPM yourself.
>
> Cheers,
>
> Markus
>
> --
> Dr. Markus Blatt - HPC-Simulation-Software & Services
> http://www.dr-blatt.de
> Pedettistr. 38, 85072 Eichstätt, Germany,  USt-Id: DE279960836
> Tel.: +49 (0) 160 97590858
> _______________________________________________
> Opm mailing list
> Opm at opm-project.org
> https://opm-project.org/cgi-bin/mailman/listinfo/opm
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <//opm-project.org/pipermail/opm/attachments/20180926/dd571018/attachment.html>