Newer
Older
******************************************************************************
GPI-2
http://www.gpi-site.com
Fraunhofer ITWM
******************************************************************************
1. INTRODUCTION
===============
GPI-2 is the second generation of GPI (www.gpi-site.com). GPI-2
implements the GASPI specification (www.gaspi.de), an API
specification which originates from the ideas and concepts of GPI.
GPI-2 is an API for asynchronous communication. It provides a
flexible, scalable and fault tolerant interface for parallel
applications.
2. INSTALLATION
===============
Requirements:
-------------
The current version of GPI-2 has the following requirements.
Software:
- libibverbs v1.1.6 (Verbs library from OFED).
- ssh server running on compute nodes (requiring no password).
- gawk (GNU Awk)
Hardware:
- Infiniband or RoCE devices.
The easiest way to install GPI-2 is by using the install.sh
script. The default settings install GPI-2 under /opt/GPI2/. This
location can be easily modified by passing the location with the -p
option to the install script.
For example,
./install.sh -p /prog/GPI2
installs GPI-2 under /prog/GPI2 instead of under the default location.
GPI-2 requires the libibverbs from the OFED stack. You can pass the
path of your OFED installation to the install script using the option
(-o) for that, in case the install script is not able to find it:
./install.sh -o <full_path_to_ofed>
You can see the install script usage:
./install -h
MPI Interoperability
--------------------
If you plan to use GPI-2 with MPI to, for instance, start an
incremental port of a large application or to use some libraries that
require MPI, you can enable MPI interoperability with the option:
./install.sh --with-mpi<=path_to_mpi_installation>
If you don't provide a path to your MPI installation and mpirun is in
your PATH, the installation script will take that MPI installation.
This will enable you to use GPI-2 with MPI where the only constraint
is that MPI_Init() must be invoked before gaspi_proc_init(). For this
MPI+GPI2 mixed mode, it is assumed that you start the application with
mpirun (or mpiexec, etc.).
If you plan to use GPI-2 with CUDA and to allow direct data transfer
between GPUs, you enable GPU support with
./install.sh --with-cuda<=path_to_cuda_installation>
If you don't provide a path to your CUDA installation and nvcc is in your PATH,
the installation script will take that CUDA installation.
GPI-2 with CUDA support actually requires GPUDirect RDMA support, so
it will be checked if the required kernel-module is installed
Due to the limitations of GPUDirect RDMA, the maximal GPU memory
segment size is limited to 250MB.
To enable GPU support in your application, call gaspi_init_GPUs. The
function gaspi_number_of_GPUs will return the number of supported GPUs
on the actual NUMA-node, the function gaspi_GPU_ids returns the device
Ids of the supported GPUs.
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
3. BUILDING GPI-2
=================
You can build GPI2 on your own. There are the following make targets:
all - Build everything
gpi - Build the GPI-2 library (including debug version)
mic - Build the GPI-2 library for the MIC
tests - Build provided tests
docs - Generate documentation (requires doxygen)
clean - Clean-up
4. BUILDING GPI-2 APPLICATIONS
==============================
GPI-2 provides two libraries: libGPI2.a and libGPI2-dbg.a.
The libGPI2.a aims at high-performance and is to be used in production
whereas the libGPI2-dbg.a provides a debug version, with extra
parameter checking and debug messages and is to be used to debug and
during development.
5. RUNNING GPI-2 APPLICATIONS
=============================
The gaspi_run utility is used to start and run GPI-2
applications. A machine file with the hostnames of nodes where the
application will run, must be provided.
For example, to start 1 process per node (on 4 nodes), the machine
file looks like:
node01
node02
node03
node04
Similarly, to start 2 processes per node (on 4 nodes):
node01
node01
node02
node02
node03
node03
node04
node04
The gaspi_run utility is invoked as follows:
gaspi_run -m <machinefile> [OPTIONS] <path GASPI program>
IMPORTANT: The path to the program must exist on all nodes where the
program should be started.
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
The gaspi_run utility has the following further options [OPTIONS]:
-b <binary file> Use a different binary for first node (master).
The master (first entry in the machine file) is
started with a different application than the rest
of the nodes (workers).
-N Enable NUMA for processes on same node. With this
option it is only possible to start the same number
of processes as NUMA nodes present on the system.
The processes running on same node will be set with
affinity to the proper NUMA node.
-n <procs> Start as many <procs> from machine file.
This option is used to start less processes than
those listed in the machine file.
-d Run with GDB (debugger) on master node. With this
option, GDB is started in the master node, to allow
debugging the application.
-h Show help.
5. THE GASPI_LOGGER
===================
The gaspi_logger utility is used to view the output from all nodes
except the master node (rank 0). The gaspi_logger is started, on
another session, on the master node. The output of the application,
when called with gaspi_printf, will be redirected to the
gaspi_logger. Other I/O routines (e.g. printf) will not.
6. GPI-2 WITH CO-PROCESSOR (INTEL XEON PHI)
==========================================
GPI-2 can be used with the Intel Xeon Phi (MIC) where the MIC is used
as one node (running possibly more than one GPI-2 process).
To use GPI-2 on the MIC, you need to compile it using the Intel
compiler with the -mmic option. The Makefile includes a build target
mic for that end but this build target is not used by the installation
script. After successful compilation, GPI-2 for the MIC can be found
under lib64/mic. If you are having problems with the compilation, make
sure the OFED_PATH in src/Makefile is setup correctly.
Assuming that the MIC(s) is set up properly, GPI-2 requires:
- ssh connectivity (requiring no password)
- grouping of local host and MIC(s) by using the tool micctrl
micctrl --initdefaults
micctrl --addbridge=br0 --type=internal --ip=172.31.1.254
micctrl --network=static --bridge=br0 --ip=172.31.1.1
where for instance, mic0: 172.31.1.1, mic1: 172.31.1.2 etc.
The MIC must be visible to the whole cluster network. For instance,
if your cluster network is 192.168.1.0 on eth0 and you want to map
mic0 to 192.168.1.100 do:
iptables -t nat -A PREROUTING -d 192.168.1.100 -i eth0 -j DNAT --to-destination 172.31.1.1
iptables -t nat -A POSTROUTING -s 172.31.1.1 -o eth0 -j SNAT --to-source 192.168.1.100
After setup, the MIC hostname can simply be added to the machinefile
and be used as another host.
7. GPU/CUDA SUPPORT
===================
To enable GPU support in your application, call gaspi_init_GPUs. This
function will check how many supported GPUs are available.
The function gaspi_number_of_GPUs will return the number of supported
GPUs on the actual NUMA-node, the function gaspi_GPU_ids returns the
device Ids of the supported GPUs.
To create a GPU memory segment, use the flag GASPI_MEM_GPU. The
segment will be allocated on the current active GPU. Note tha the
actice GPU may change by calling gaspi_init_GPUs and therefore should
be set new bevore creating a GPU memory segment.
The segment pointer of a GPU memory segment is a GPU device
pointer. It can be used in CUDA-related functions (eg. cudaMemcpy) and
in CUDA-kernels. However, a direct access in host memory results in a
segmentation fault.
Host memory segment
Of course, GPU memory segments can be used together with host memory
segments. If GPU-support is enabled, a new host memory segments is
also registered for the GPUs. Therefore, a segment pointer of a local
host memory segments can be used with asynchronous CUDA copy
operations for asynchronous local data transfers/
Special functions
--------------------
Due to some issues with the PCIe-Express bus, the Bandwidth of direct
GPU-to-GPU transfers is imitated for larger messages sizes.
Therefore, gaspi_gpu_write and gaspi_gpu_write_notify are optimized
for remote write operations with a GPU memory segment as source. These
functions use one-sided copies in host memory. Therefore,
gaspi_gpu_write is one-sided, but not fully asynchronous with respect
to the host, since the host thread is required to control the data
flow. Small data packages are still transfered direct. However, it can
still be overlapped with computation on the GPU. gaspi_write and
gaspi_write_notify still can be used if a full asynchronous, but
slower data transfer is preferred.
Since the maximal size pinable GPU memory is limited to less then
256MB on most GPUs, the maximal size of a GPU memory segment is also
limited to this. However, this issue can be removed by using a
firmware patch for current K20 and K40 GPUs and will be removed in
future versions.
To support direct data transfer, the Infiniband device and the GPU
must be located on the same NUMA socket.Otherwise, direct data
transder is not supported. However, the application checks this at the
beginning.
==================
If you're having troubles when building GPI-2, make sure you have the
OFED stack correctly installed and running. You can change the OFED
path in the Makefile in the source directory (src) to the correct path
in your system.
=====================
GPI-2 is on-going work and more features are still to come. Here are
some that are in our roadmap:
- include the support for GPU memory (available but not yet integrated)
- support to add spare nodes (fault tolerance)
- better debugging possibilities
9. MORE INFORMATION
===================
For more information, check the GPI-2 website: www.gpi-site.com