[Opm] OPM Flow multi-node simulations stuck at domain decomposition step

Wed Mar 11 14:49:50 UTC 2020

Hi Kai,

Thank you for your reply. I have run the Norne model with 1, 2, 4, 8, 16, 36, 72, and 108 processes successfully. The run gets stuck at 144 processes. As I mentioned, I did try a larger model, but even there I was not able to scale on more than 1 node for a certain configuration. Finally, I have an 11 million cells model with just 1 producer and 1 injector, which runs on 144 processes.

I would still like to figure out why domain decomposition fails for other models? I understand that the performance benefit may be minimal if there are fewer cells per process, still it should run.

Thank you,

Yogi

From: Kai Bao [mailto:Kai.Bao at sintef.no] 
Sent: Wednesday, March 11, 2020 2:54 AM
To: Yogi Pandey <yogi.pandey at oracle.com>; opm at opm-project.org
Subject: Re: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step

Hi, Yogi,

I see you are trying to run Norne with 144 processes.  Did you see the problem with much less processes, for example 4 or 8 processes?

In my opinion, with the current approach for domain decomposition, it can be challenging to run Norne with so many processes, considering the relatively small size of the Norne model and the long wells existing in this model.  I am not totally sure though. 

Best Regards, 

Kai Bao

  _____  

From: Opm <HYPERLINK "mailto:opm-bounces at opm-project.org"opm-bounces at opm-project.org> on behalf of Yogi Pandey <HYPERLINK "mailto:yogi.pandey at oracle.com"yogi.pandey at oracle.com>
Sent: Tuesday, March 10, 2020 10:15 PM
To: HYPERLINK "mailto:opm at opm-project.org"opm at opm-project.org <HYPERLINK "mailto:opm at opm-project.org"opm at opm-project.org>
Subject: [Opm] OPM Flow multi-node simulations stuck at domain decomposition step 

All,

I am trying to run OPM Flow simulations on multiple nodes. I have built OPM Flow from source on Oracle Linux 7 OS (binary compatible with RHEL) with:

.        GCC-8.3.1

.        openmpi-4.0.2 (built from source)

.        boost-1.72.0 (built from source)

.        cmake-3.16.4 (built from source)

.        parmetis-4.0.3 (built from source)

.        dune-2.6.0: dune-common, dune-geometry, dune-grid, dune-istl (built from source)

.        Zoltan-3.83 (built from source)

.        OPM Flow modules are built using following commads:

o   cmake -DCMAKE_BUILD_TYPE=Release -DUSE_MPI=ON -DUSE_OPENMP=ON -DBLAS_LIBRARIES=/usr/lib64 -DCMAKE_INSTALL_PREFIX=/usr/local ..

o   sudo make

For Norne data set, following is the input file (params) content:

ecl-deck-file-name=NORNE_ATW2013.DATA

output-dir=out_parallel

output-mode=none

output-interval=1000000

enable-opm-rst-file=false

threads-per-process=1

Simulation is being run on 4 nodes with 32 processors each using following command:

mpirun --display-map -mca btl self -x UCX_TLS=rc,self,sm -x HCOLL_ENABLE_MCAST_ALL=0 -mca coll_hcoll_enable 0 -x UCX_IB_TRAFFIC_CLASS=105 -x UCX_IB_GID_INDEX=3 --cpu-set 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35 -np 144 --hostfile /etc/opt/rdma/hostfile /mnt/nfs-share/etc/opm-flow/opm-simulators/build/bin/flow --parameter-file=/mnt/nfs-share/data/norne/params

The simulation get stuck indefinitely at the domain decomposition step. I am able to finish a parallel run up to 3 nodes, but always getting stuck at 4 nodes.

I have also created some customized simulation decks with about 11 million cells to rule-out that fewer number of cells in the Norne model may be a reason, but the simulation gets stuck as soon as I scale from 1 node to 2 nodes. Can someone please help me understand, what might be causing it?

Thank you,

Yogi

_______________________________________________
Opm mailing list
HYPERLINK "mailto:Opm at opm-project.org"Opm at opm-project.org
HYPERLINK "https://urldefense.com/v3/__https:/eur03.safelinks.protection.outlook.com/?url=https*3A*2F*2Fopm-project.org*2Fcgi-bin*2Fmailman*2Flistinfo*2Fopm&data=02*7C01*7Ckai.bao*40sintef.no*7C1ac6994dae0842f9a4a008d7c5387102*7Ce1f00f39604145b0b309e0210d8b32af*7C1*7C1*7C637194718508875070&sdata=dk*2BNtwSBcDA9*2F2ubjoNpmQcLmp1tt63ftJkHkBcUsq0*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUl!!GqivPVa7Brio!NXPeS6Wi75BbMuFJj0iyFFCzX4zA9HcUvdNY4YHmJfQwqEq5Rs07e7ZHq4CWvc8S6g$"https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopm-project.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopm&data=02%7C01%7Ckai.bao%40sintef.no%7C1ac6994dae0842f9a4a008d7c5387102%7Ce1f00f39604145b0b309e0210d8b32af%7C1%7C1%7C637194718508875070&sdata=dk%2BNtwSBcDA9%2F2ubjoNpmQcLmp1tt63ftJkHkBcUsq0%3D&reserved=0