Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
# SLURM Inception
Like the movie, this repository allows you to run a SLURM daemon inside a
current SLURM system, so you can launch your own jobs quickly in your
reservation with a controlled SLURM version.
These script use the nix installation in `/gpfs/projects/bsc15/nix` to access
the SLURM binaries, as well as the tools to enter the namespace that sets the
/nix directory.
All the jobs you launch in the inner SLURM system will have the /nix store
available.
## Usage
First, allocate `$N` nodes in the system:
login$ salloc -N $N
Then, run the `start.sh` script in all nodes:
compute$ srun ./start.sh &
Notice the ampersand, so you get back the control on the first compute node.
Now, run the `shell.sh` script to open a shell where you can submit jobs into
the inception SLURM.
compute$ ./shell.sh
[inception] compute$
Notice the `[inception]` mark, to help you remember where you are.
In that shell, all SLURM commands refer to the inception SLURM daemon:
[inception] compute$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
main* up infinite 2 idle s21r2b[47,55]
To run MPI programs, make sure you set the `--mpi=pmi2` option in srun:
[inception] compute$ srun -N 2 --mpi=pmi2 IMB-MPI1 pingpong
#------------------------------------------------------------
# Intel (R) MPI Benchmarks 2017 update 2, MPI-1 part
#------------------------------------------------------------
# Date : Fri Jan 13 15:43:49 2023
# Machine : x86_64
# System : Linux
# Release : 4.4.59-92.20-default
# Version : #1 SMP Wed May 31 14:05:24 UTC 2017 (8cd473d)
# MPI Version : 3.1
# MPI Thread Environment:
...
#---------------------------------------------------
# Benchmarking PingPong
# #processes = 2
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 1.51 0.00
1 1000 1.51 0.66
2 1000 1.51 1.33
4 1000 1.51 2.66
8 1000 1.51 5.30
16 1000 1.70 9.44
32 1000 1.70 18.80
64 1000 1.69 37.78
128 1000 1.72 74.46
256 1000 1.75 146.00
512 1000 1.83 279.86
1024 1000 1.98 518.34
2048 1000 2.24 915.28
4096 1000 2.75 1489.24
8192 1000 3.92 2087.41 <- notice
16384 1000 7.33 2235.03
32768 1000 9.13 3589.84
65536 640 17.65 3714.06
131072 320 23.98 5465.19
262144 160 36.53 7175.80
524288 80 62.30 8415.50
1048576 40 114.09 9190.84
2097152 20 224.97 9321.85
4194304 10 418.15 10030.61
As a bonus, you also have direct access to the /nix store:
[inception] compute$ srun -N2 --mpi=pmi2 /nix/store/lg0xzcfkd6fh09f238djjfc684cy4d9n-osu-micro-benchmarks-5.7/bin/osu_bw
...
# OSU MPI Bandwidth Test v5.7
# Size Bandwidth (MB/s)
1 3.50
2 7.01
4 13.87
8 28.06
16 47.73
32 95.71
64 193.10
128 377.90
256 742.02
512 1410.46
1024 2556.12
2048 4203.39
4096 6072.21
8192 7539.24 <- yay
16384 6160.90
32768 8649.04
65536 8705.41
131072 8862.76
262144 8940.29
524288 8980.80
1048576 9001.11
2097152 9017.25
4194304 9024.85
To exit and terminate the SLURM inception daemon, first exit the inception
shell:
[inception] compute$ exit
Then return to the srun job in the background, and then press `^C` twice
compute$ fg
^Csrun: interrupt (one more within 1 sec to abort)
srun: StepId=27021659.1 tasks 0-1: running
^Csrun: sending Ctrl-C to StepId=27021659.1
srun: forcing job termination
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 27021659.1 ON s21r2b47 CANCELLED AT 2023-01-13T15:53:47 ***
srun: launch/slurm: _step_signal: Terminating StepId=27021659.1
You can also use `scancel` from the `login0` node, to kill the outside job
allocation.
## Configuration
You can tune the configuration of the inner SLURM bu changing the
`slurm.base.conf` file. Notice that some options will be appended.