Difference between revisions of "GPU Cloud VMs"

From KENET Training
Jump to: navigation, search
 
(21 intermediate revisions by the same user not shown)
Line 15: Line 15:
 
The codes can be run on the terminal directly, however, the SLURM job scheduler is also installed on the VM,
 
The codes can be run on the terminal directly, however, the SLURM job scheduler is also installed on the VM,
 
and alternately, the codes can be run via the scheduler.  
 
and alternately, the codes can be run via the scheduler.  
 +
 +
[[File:GROMACS_logo.png|350px]]
  
 
=== Gromacs GPU VM usage ===
 
=== Gromacs GPU VM usage ===
−
In the Gromacs GPU vm, gromacs and mpi are available, to run gromacs, you can use the following:  
+
In the Gromacs GPU vm, gromacs and mpi are available, to run gromacs, you can use the following:
 
<code bash>
 
<code bash>
 
     $ mpirun -np 1  /usr/local/bin/gmx_mpi  
 
     $ mpirun -np 1  /usr/local/bin/gmx_mpi  
 +
 
</code>
 
</code>
 +
We can retreive some examples to work with:
 +
<code bash>
 +
  $ mkdir ~/membrane
 +
  $ cd ~/membrane
 +
  $ wget https://gitlab.com/gromacs/online-tutorials/membrane-protein/-/archive/main/membrane-protein-main.zip
 +
  $ unzip membrane-protein-main.zip
 +
  $ mv  membrane-protein-main/*  .
 +
  $ mkdir run
 +
  $ cd run
 +
  $ cp -rf ../data/input/charmm-gui-1MAL/gromacs/{step5_input.gro,step5_input.pdb,topol.top,index.ndx,toppar}  .
 +
  $ cp ../data/input/mdp/*.mdp .
 +
</code>
 +
 +
and finally run gromacs through an energy minimization:
 +
<code bash>
 +
  $ mpirun -np 1  /usr/local/bin/gmx_mpi grompp -f step6.0_minimization.mdp -o minimization.tpr -c step5_input.gro  -r step5_input.gro -p topol.top
 +
</code>
 +
 
===== Advanced usage with slurm: =====
 
===== Advanced usage with slurm: =====
−
to run gromacs in the GPU vm with slurm, create a submission script with the following contents:  
+
 
 +
To run gromacs in the GPU vm with slurm, create a submission script with the following contents:  
 
<code bash>
 
<code bash>
−
  #!/bin/bash                                                                                                   
+
#!/bin/bash                                                                                                   
−
  ##SBATCH --job-name="example-name"                                                                             
+
##SBATCH --job-name="example-name"                                                                             
−
  ##SBATCH --get-user-env                                                                                       
+
##SBATCH --get-user-env                                                                                       
−
  ##SBATCH --output=_scheduler-stdout.txt                                                                       
+
##SBATCH --output=_scheduler-stdout.txt                                                                       
−
  ##SBATCH --error=_scheduler-stderr.txt                                                                         
+
##SBATCH --error=_scheduler-stderr.txt                                                                         
−
  ##SBATCH --nodes=1                                                                                             
+
##SBATCH --nodes=1                                                                                             
−
  ##SBATCH --ntasks-per-node=1                                                                                   
+
##SBATCH --ntasks-per-node=1                                                                                   
−
  ##SBATCH --cpus-per-task=1                                                                                     
+
##SBATCH --cpus-per-task=1                                                                                     
−
  ##SBATCH --time=23:58:20                                                                                       
+
##SBATCH --time=23:58:20                                                                                       
−
  ##SBATCH --partition=jobs                                                                                     
+
##SBATCH --partition=jobs                                                                                     
 
                                                                                                                  
 
                                                                                                                  
−
  export OMP_NUM_THREADS=2                                                                                    
+
export OMP_NUM_THREADS=2
−
  mpirun -np 1 gmx_mpi  ...
+
cd ~/membrane                                                                               
 +
mpirun -np 1 gmx_mpi grompp -f step6.0_minimization.mdp -o minimization.tpr -c step5_input.gro -r step5_input.gro -p topol.top
 
</code>
 
</code>
−
give the file a name like job.mpi,
+
give the file a name like '''job.mpi''',
 
edit the last line to include your commands to gromacs, and submit with slurm:
 
edit the last line to include your commands to gromacs, and submit with slurm:
 +
 
<code bash>
 
<code bash>
−
  sbatch  test.mpi
+
  sbatch  job.mpi
 
</code>
 
</code>
 +
== [https://asciinema.org/a/bPFAntHZt1xyLcpFkowGpoFZ3 Watch Gromacs Demo] ==
  
 +
[[File:Quantum_ESPRESSO_logo.jpg|350px]]
 
=== Quantum Espresso GPU VM usage ===
 
=== Quantum Espresso GPU VM usage ===
 
In the QE GPU vm, quantum espresso and mpi are available, to run it, you can use the following:  
 
In the QE GPU vm, quantum espresso and mpi are available, to run it, you can use the following:  
 
<code bash>
 
<code bash>
 
     $ mpirun -np 1  /usr/local/bin/pw.x  
 
     $ mpirun -np 1  /usr/local/bin/pw.x  
 +
</code>
 +
We can retreive some examples to work with:
 +
<code bash>
 +
  mkdir ~/examples
 +
  cd ~/examples/
 +
  git clone https://github.com/Materials-Modelling-Group/training-examples.git
 +
  cd  training-examples
 +
 
 
</code>
 
</code>
 
===== Advanced usage with slurm: =====
 
===== Advanced usage with slurm: =====
 
to run gromacs in the GPU vm with slurm, create a submission script with the following contents:  
 
to run gromacs in the GPU vm with slurm, create a submission script with the following contents:  
 
<code bash>
 
<code bash>
−
  #!/bin/bash                                                                                                   
+
#!/bin/bash                                                                                                   
−
  ##SBATCH --job-name="example-name"                                                                             
+
##SBATCH --job-name="example-name"                                                                             
−
  ##SBATCH --get-user-env                                                                                       
+
##SBATCH --get-user-env                                                                                       
−
  ##SBATCH --output=_scheduler-stdout.txt                                                                       
+
##SBATCH --output=_scheduler-stdout.txt                                                                       
−
  ##SBATCH --error=_scheduler-stderr.txt                                                                         
+
##SBATCH --error=_scheduler-stderr.txt                                                                         
−
  ##SBATCH --nodes=1                                                                                             
+
##SBATCH --nodes=1                                                                                             
−
  ##SBATCH --ntasks-per-node=1                                                                                   
+
##SBATCH --ntasks-per-node=1                                                                                   
−
  ##SBATCH --cpus-per-task=1                                                                                     
+
##SBATCH --cpus-per-task=1                                                                                     
−
  ##SBATCH --time=23:58:20                                                                                       
+
##SBATCH --time=23:58:20                                                                                       
−
  ##SBATCH --partition=jobs                                                                                     
+
##SBATCH --partition=jobs                                                                                     
 
                                                                                                                                                                                                        
 
                                                                                                                                                                                                        
−
  mpirun -np 1 pw.x ...
+
cd $HOME/examples/training-examples
 +
mpirun -np 1 pw.x <al.scf.david.in > output.out
 
</code>
 
</code>
−
give the file a name like job.mpi,
+
give the file a name like '''job.mpi''',
 
edit the last line to include your commands to pw.x, and submit with slurm:
 
edit the last line to include your commands to pw.x, and submit with slurm:
 
<code bash>
 
<code bash>
−
   sbatch  test.mpi
+
   sbatch  job.mpi
 
</code>
 
</code>
 +
== [https://asciinema.org/a/F5Q7kQ9cUjxwSY291T6W3b38y Watch Quantum Espresso Demo] ==
  
−
 
+
[[File:yambo_logo_overlay.png|350px]]
 
=== YAMBO GPU VM usage ===
 
=== YAMBO GPU VM usage ===
 
In the YAMBO GPU vm,  yambo and mpi are available, to run yambo, you can use the following:  
 
In the YAMBO GPU vm,  yambo and mpi are available, to run yambo, you can use the following:  
Line 77: Line 113:
 
     $ mpirun -np 1  /usr/local/bin/yambo
 
     $ mpirun -np 1  /usr/local/bin/yambo
 
</code>
 
</code>
 +
We can retreive some examples to work with:
 +
<code bash>
 +
mkdir examples
 +
cd examples
 +
wget https://media.yambo-code.eu/educational/tutorials/files/Silicon.tar.gz
 +
tar -xf Silicon.tar.gz
 +
cd Silicon/YAMBO/4x4x4
 +
</code>
 +
we have some prepared inputs we can use to run a convergence calculation,
 +
we can run it as such
 +
<code bash>
 +
mpirun -np 1 yambo -F Inputs/01HF_corrections -J HF_XXRy
 +
</code>
 +
or
 
===== Advanced usage with slurm: =====
 
===== Advanced usage with slurm: =====
 
to run yambo in the GPU vm with slurm, create a submission script with the following contents:  
 
to run yambo in the GPU vm with slurm, create a submission script with the following contents:  
 
<code bash>
 
<code bash>
−
  #!/bin/bash                                                                                                   
+
#!/bin/bash                                                                                                   
−
  ##SBATCH --job-name="example-name"                                                                             
+
##SBATCH --job-name="example-name"                                                                             
−
  ##SBATCH --get-user-env                                                                                       
+
##SBATCH --get-user-env                                                                                       
−
  ##SBATCH --output=_scheduler-stdout.txt                                                                       
+
##SBATCH --output=_scheduler-stdout.txt                                                                       
−
  ##SBATCH --error=_scheduler-stderr.txt                                                                         
+
##SBATCH --error=_scheduler-stderr.txt                                                                         
−
  ##SBATCH --nodes=1                                                                                             
+
##SBATCH --nodes=1                                                                                             
−
  ##SBATCH --ntasks-per-node=1                                                                                   
+
##SBATCH --ntasks-per-node=1                                                                                   
−
  ##SBATCH --cpus-per-task=1                                                                                     
+
##SBATCH --cpus-per-task=1                                                                                     
−
  ##SBATCH --time=23:58:20                                                                                       
+
##SBATCH --time=23:58:20                                                                                       
−
  ##SBATCH --partition=jobs                                                                                     
+
##SBATCH --partition=jobs                                                                                     
 
                                                                                                                  
 
                                                                                                                  
−
                                                                                   
+
cd $HOME/examples/Silicon/YAMBO/4x4x4                                                                               
−
  mpirun -np 1 yambo ...
+
mpirun -np 1 yambo -F Inputs/01HF_corrections -J HF_XXRy
 
</code>
 
</code>
 
give the file a name like job.mpi,
 
give the file a name like job.mpi,
 
edit the last line to include your commands to yambo, and submit with slurm:
 
edit the last line to include your commands to yambo, and submit with slurm:
 
<code bash>
 
<code bash>
−
   sbatch  test.mpi
+
   sbatch  job.mpi
 
</code>
 
</code>
 +
== [https://asciinema.org/a/RKZ8ll1ngRxWPBNHbAksgFr9h Watch Yambo Demo] ==
  
−
 
+
[[File:TensorFlow_logo.svg.png|350px]]
 
=== Tensorflow GPU VM usage ===
 
=== Tensorflow GPU VM usage ===
 
We will try out the Tensorflow MNIST example from the documentation: [https://www.tensorflow.org/datasets/keras_example]
 
We will try out the Tensorflow MNIST example from the documentation: [https://www.tensorflow.org/datasets/keras_example]
Line 113: Line 164:
 
We can now attempt to run some code, place the following code in a plain text file, call it `example.py`
 
We can now attempt to run some code, place the following code in a plain text file, call it `example.py`
 
<code python>
 
<code python>
−
  import tensorflow as tf
+
import tensorflow as tf
−
  import tensorflow_datasets as tfds
+
import tensorflow_datasets as tfds
−
 
+
−
  (ds_train, ds_test), ds_info = tfds.load(
+
(ds_train, ds_test), ds_info = tfds.load(
−
      'mnist',
+
    'mnist',
−
      split=['train', 'test'],
+
    split=['train', 'test'],
−
      shuffle_files=True,
+
    shuffle_files=True,
−
      as_supervised=True,
+
    as_supervised=True,
−
      with_info=True,
+
    with_info=True,
−
  )
+
)
 +
 +
# training pipeline
 +
def normalize_img(image, label):
 +
  """Normalizes images: `uint8` -> `float32`."""
 +
  return tf.cast(image, tf.float32) / 255., label
 +
 +
ds_train = ds_train.map(
 +
    normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
 +
ds_train = ds_train.cache()
 +
ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)
 +
ds_train = ds_train.batch(128)
 +
ds_train = ds_train.prefetch(tf.data.AUTOTUNE)
 +
 +
# Evaluation pipeline
 +
ds_test = ds_test.map(
 +
    normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
 +
ds_test = ds_test.batch(128)
 +
ds_test = ds_test.cache()
 +
ds_test = ds_test.prefetch(tf.data.AUTOTUNE)
 +
 +
# Create and train the model:
 +
model = tf.keras.models.Sequential([
 +
  tf.keras.layers.Flatten(input_shape=(28, 28)),
 +
  tf.keras.layers.Dense(128, activation='relu'),
 +
  tf.keras.layers.Dense(10)
 +
])
 +
model.compile(
 +
    optimizer=tf.keras.optimizers.Adam(0.001),
 +
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
 +
    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],
 +
)
 
    
 
    
−
  # training pipeline
+
model.fit(
−
  def normalize_img(image, label):
+
    ds_train,
−
    """Normalizes images: `uint8` -> `float32`."""
+
    epochs=6,
−
    return tf.cast(image, tf.float32) / 255., label
+
    validation_data=ds_test,
−
 
+
)
−
  ds_train = ds_train.map(
 
−
      normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
 
−
  ds_train = ds_train.cache()
 
−
  ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)
 
−
  ds_train = ds_train.batch(128)
 
−
  ds_train = ds_train.prefetch(tf.data.AUTOTUNE)
 
−
 
 
−
  # Evaluation pipeline
 
−
  ds_test = ds_test.map(
 
−
      normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
 
−
  ds_test = ds_test.batch(128)
 
−
  ds_test = ds_test.cache()
 
−
  ds_test = ds_test.prefetch(tf.data.AUTOTUNE)
 
−
 
 
−
  # Create and train the model:
 
−
  model = tf.keras.models.Sequential([
 
−
    tf.keras.layers.Flatten(input_shape=(28, 28)),
 
−
    tf.keras.layers.Dense(128, activation='relu'),
 
−
    tf.keras.layers.Dense(10)
 
−
  ])
 
−
  model.compile(
 
−
      optimizer=tf.keras.optimizers.Adam(0.001),
 
−
      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
 
−
      metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],
 
−
  )
 
−
 
 
−
  model.fit(
 
−
      ds_train,
 
−
      epochs=6,
 
−
      validation_data=ds_test,
 
−
  )
 
 
</code>
 
</code>
  
 
And now we can test it:  
 
And now we can test it:  
 +
 
<code bash>
 
<code bash>
−
$ python  example.py  
+
  $ python  example.py
−
...
+
  ...
−
Epoch 1/6
+
  Epoch 1/6  
−
469/469 [==============================] - 3s 2ms/step - loss: 0.3494 - sparse_categorical_accuracy: 0.9040 - val_loss: 0.1970 - val_sparse_categorical_accuracy: 0.9431
+
  469/469 [==============================] - 3s 2ms/step - loss: 0.3494 - sparse_categorical_accuracy: 0.9040 - val_loss: 0.1970 -  
−
Epoch 2/6
+
  val_sparse_categorical_accuracy: 0.9431
−
469/469 [==============================] - 1s 2ms/step - loss: 0.1655 - sparse_categorical_accuracy: 0.9530 - val_loss: 0.1394 - val_sparse_categorical_accuracy: 0.9576
+
  Epoch 2/6
−
Epoch 3/6
+
  469/469 [==============================] - 1s 2ms/step - loss: 0.1655 - sparse_categorical_accuracy: 0.9530 - val_loss: 0.1394 -  
−
469/469 [==============================] - 1s 2ms/step - loss: 0.1189 - sparse_categorical_accuracy: 0.9660 - val_loss: 0.1096 - val_sparse_categorical_accuracy: 0.9666
+
  val_sparse_categorical_accuracy: 0.9576
−
Epoch 4/6
+
  Epoch 3/6
−
469/469 [==============================] - 1s 2ms/step - loss: 0.0915 - sparse_categorical_accuracy: 0.9736 - val_loss: 0.0993 - val_sparse_categorical_accuracy: 0.9695
+
  469/469 [==============================] - 1s 2ms/step - loss: 0.1189 - sparse_categorical_accuracy: 0.9660 - val_loss: 0.1096 -  
−
Epoch 5/6
+
  val_sparse_categorical_accuracy: 0.9666
−
469/469 [==============================] - 1s 2ms/step - loss: 0.0735 - sparse_categorical_accuracy: 0.9786 - val_loss: 0.0870 - val_sparse_categorical_accuracy: 0.9743
+
  ...
−
Epoch 6/6
+
  Epoch 6/6
−
469/469 [==============================] - 1s 2ms/step - loss: 0.0599 - sparse_categorical_accuracy: 0.9827 - val_loss: 0.0775 - val_sparse_categorical_accuracy: 0.9769
+
  469/469 [==============================] - 1s 2ms/step - loss: 0.0599 - sparse_categorical_accuracy: 0.9827 - val_loss: 0.0775 -  
 +
  val_sparse_categorical_accuracy: 0.9769
 
</code>
 
</code>
  
 
We have run tensorflow+Keras on the MNIST dataset, with a final accuract of 98%.  
 
We have run tensorflow+Keras on the MNIST dataset, with a final accuract of 98%.  
 +
== [https://asciinema.org/a/nYgHYjz5n87diRgqBBnIysHlf Watch Tensoflow Demo] ==
  
−
 
+
[[File:Pytorch_logo.png|350px]]
 
=== PyTorch GPU VM Usage ===
 
=== PyTorch GPU VM Usage ===
 
We will run the MNIST example from the PyTorch documentation available here: [https://github.com/pytorch/examples/tree/main/mnist]. Once you have logged in, there are instructions on how to activate the PyTorch Conda based virtualenv environment.
 
We will run the MNIST example from the PyTorch documentation available here: [https://github.com/pytorch/examples/tree/main/mnist]. Once you have logged in, there are instructions on how to activate the PyTorch Conda based virtualenv environment.
Line 214: Line 268:
 
</code>
 
</code>
 
This code downloads some training MNIST data, runs a convolutional neural network based training, and gives a summary of the accuracy at the end 99%).  There is no need to install PyTorch since its aready preconfigured in the `pt` environment.  
 
This code downloads some training MNIST data, runs a convolutional neural network based training, and gives a summary of the accuracy at the end 99%).  There is no need to install PyTorch since its aready preconfigured in the `pt` environment.  
 +
 +
== [https://asciinema.org/a/jzpfFhoguxXpWME0FHHBWwbi4 Watch PyTorch Demo] ==
  
 
Up:
 
Up:
 
[[ HPC_Usage| HPC_Usage]]
 
[[ HPC_Usage| HPC_Usage]]

Latest revision as of 20:11, 8 May 2025

Preconfigured GPU appliances

KENET provides a set of preconfigured Virtual Machine appliances with the following codes:

  1. Quantum Espresso
  2. YAMBO
  3. SIESTA
  4. GROMACS
  5. Tensorflow
  6. PyTorch

To request for access please apply through this form: [1] The appliance requires no user configuration, and the above listed appliances will have the individual code ready with GPU support.

The codes can be run on the terminal directly, however, the SLURM job scheduler is also installed on the VM, and alternately, the codes can be run via the scheduler.

GROMACS logo.png

Gromacs GPU VM usage

In the Gromacs GPU vm, gromacs and mpi are available, to run gromacs, you can use the following: 

   $ mpirun -np 1  /usr/local/bin/gmx_mpi

We can retreive some examples to work with: 

 $ mkdir ~/membrane
 $ cd ~/membrane
 $ wget https://gitlab.com/gromacs/online-tutorials/membrane-protein/-/archive/main/membrane-protein-main.zip 
 $ unzip membrane-protein-main.zip
 $ mv  membrane-protein-main/*  .
 $ mkdir run
 $ cd run 
 $ cp -rf ../data/input/charmm-gui-1MAL/gromacs/{step5_input.gro,step5_input.pdb,topol.top,index.ndx,toppar}  . 
 $ cp ../data/input/mdp/*.mdp .

and finally run gromacs through an energy minimization: 

 $ mpirun -np 1  /usr/local/bin/gmx_mpi grompp -f step6.0_minimization.mdp -o minimization.tpr -c step5_input.gro  -r step5_input.gro -p topol.top

Advanced usage with slurm:

To run gromacs in the GPU vm with slurm, create a submission script with the following contents: 

#!/bin/bash                                                                                                  
##SBATCH --job-name="example-name"                                                                            
##SBATCH --get-user-env                                                                                      
##SBATCH --output=_scheduler-stdout.txt                                                                      
##SBATCH --error=_scheduler-stderr.txt                                                                        
##SBATCH --nodes=1                                                                                            
##SBATCH --ntasks-per-node=1                                                                                  
##SBATCH --cpus-per-task=1                                                                                    
##SBATCH --time=23:58:20                                                                                      
##SBATCH --partition=jobs                                                                                    
                                                                                                                
export OMP_NUM_THREADS=2  
cd ~/membrane                                                                                
mpirun -np 1 gmx_mpi grompp -f step6.0_minimization.mdp -o minimization.tpr -c step5_input.gro  -r step5_input.gro -p topol.top

give the file a name like job.mpi, edit the last line to include your commands to gromacs, and submit with slurm: 

 sbatch  job.mpi

Watch Gromacs Demo

Quantum ESPRESSO logo.jpg

Quantum Espresso GPU VM usage

In the QE GPU vm, quantum espresso and mpi are available, to run it, you can use the following: 

   $ mpirun -np 1  /usr/local/bin/pw.x

We can retreive some examples to work with: 

 mkdir ~/examples
 cd ~/examples/
 git clone https://github.com/Materials-Modelling-Group/training-examples.git
 cd  training-examples

Advanced usage with slurm:

to run gromacs in the GPU vm with slurm, create a submission script with the following contents: 

#!/bin/bash                                                                                                  
##SBATCH --job-name="example-name"                                                                            
##SBATCH --get-user-env                                                                                      
##SBATCH --output=_scheduler-stdout.txt                                                                      
##SBATCH --error=_scheduler-stderr.txt                                                                        
##SBATCH --nodes=1                                                                                            
##SBATCH --ntasks-per-node=1                                                                                  
##SBATCH --cpus-per-task=1                                                                                    
##SBATCH --time=23:58:20                                                                                      
##SBATCH --partition=jobs                                                                                    
                                                                                                                                                                                                     
cd $HOME/examples/training-examples
mpirun -np 1  pw.x <al.scf.david.in > output.out

give the file a name like job.mpi, edit the last line to include your commands to pw.x, and submit with slurm: 

  sbatch  job.mpi

Watch Quantum Espresso Demo

Yambo logo overlay.png

YAMBO GPU VM usage

In the YAMBO GPU vm, yambo and mpi are available, to run yambo, you can use the following: 

   $ mpirun -np 1  /usr/local/bin/yambo

We can retreive some examples to work with: 

mkdir examples
cd examples
wget https://media.yambo-code.eu/educational/tutorials/files/Silicon.tar.gz
tar -xf Silicon.tar.gz
cd Silicon/YAMBO/4x4x4

we have some prepared inputs we can use to run a convergence calculation, we can run it as such 

mpirun -np 1 yambo -F Inputs/01HF_corrections -J HF_XXRy

or

Advanced usage with slurm:

to run yambo in the GPU vm with slurm, create a submission script with the following contents: 

#!/bin/bash                                                                                                  
##SBATCH --job-name="example-name"                                                                            
##SBATCH --get-user-env                                                                                      
##SBATCH --output=_scheduler-stdout.txt                                                                      
##SBATCH --error=_scheduler-stderr.txt                                                                        
##SBATCH --nodes=1                                                                                            
##SBATCH --ntasks-per-node=1                                                                                  
##SBATCH --cpus-per-task=1                                                                                    
##SBATCH --time=23:58:20                                                                                      
##SBATCH --partition=jobs                                                                                    
                                                                                                                
cd $HOME/examples/Silicon/YAMBO/4x4x4                                                                                 
mpirun -np 1 yambo -F Inputs/01HF_corrections -J HF_XXRy

give the file a name like job.mpi, edit the last line to include your commands to yambo, and submit with slurm: 

  sbatch  job.mpi

Watch Yambo Demo

TensorFlow logo.svg.png

Tensorflow GPU VM usage

We will try out the Tensorflow MNIST example from the documentation: [2] After logging in, there are instructions on how to activate the right conda based virtualenv environment: 

   $ conda activate tf

this environment is preconfigured with tensorflow and has CUDA support. Next is we have to get the data and code to run, starting with the tensorflow_dataset package, 

   $ pip3 install tensorflow_datasets

We can now attempt to run some code, place the following code in a plain text file, call it `example.py` 

import tensorflow as tf
import tensorflow_datasets as tfds

(ds_train, ds_test), ds_info = tfds.load(
    'mnist',
    split=['train', 'test'],
    shuffle_files=True,
    as_supervised=True,
    with_info=True,
)

# training pipeline
def normalize_img(image, label):
  """Normalizes images: `uint8` -> `float32`."""
  return tf.cast(image, tf.float32) / 255., label

ds_train = ds_train.map(
    normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
ds_train = ds_train.cache()
ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)
ds_train = ds_train.batch(128)
ds_train = ds_train.prefetch(tf.data.AUTOTUNE)

# Evaluation pipeline
ds_test = ds_test.map(
    normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
ds_test = ds_test.batch(128)
ds_test = ds_test.cache()
ds_test = ds_test.prefetch(tf.data.AUTOTUNE)

# Create and train the model: 
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(10)
])
model.compile(
    optimizer=tf.keras.optimizers.Adam(0.001),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],
)
 
model.fit(
    ds_train,
    epochs=6,
    validation_data=ds_test,
)

And now we can test it: 

 $ python  example.py
  ...
 Epoch 1/6 
 469/469 [==============================] - 3s 2ms/step - loss: 0.3494 - sparse_categorical_accuracy: 0.9040 - val_loss: 0.1970 - 
 val_sparse_categorical_accuracy: 0.9431
 Epoch 2/6
 469/469 [==============================] - 1s 2ms/step - loss: 0.1655 - sparse_categorical_accuracy: 0.9530 - val_loss: 0.1394 - 
 val_sparse_categorical_accuracy: 0.9576
 Epoch 3/6
 469/469 [==============================] - 1s 2ms/step - loss: 0.1189 - sparse_categorical_accuracy: 0.9660 - val_loss: 0.1096 - 
 val_sparse_categorical_accuracy: 0.9666
 ...
 Epoch 6/6
 469/469 [==============================] - 1s 2ms/step - loss: 0.0599 - sparse_categorical_accuracy: 0.9827 - val_loss: 0.0775 - 
 val_sparse_categorical_accuracy: 0.9769

We have run tensorflow+Keras on the MNIST dataset, with a final accuract of 98%.

Watch Tensoflow Demo

Pytorch logo.png

PyTorch GPU VM Usage

We will run the MNIST example from the PyTorch documentation available here: [3]. Once you have logged in, there are instructions on how to activate the PyTorch Conda based virtualenv environment. 

 $ conda activate pt

Once this is activated, we can now retreive the python code for the example, place it in a directory and run it: 

 $ mkdir mnist    # creating a working dir
 $ cd  mnist      # changing directory to the working dir
 $ wget https://raw.githubusercontent.com/pytorch/examples/refs/heads/main/mnist/main.py

and finally, we are ready to run it: 

 $ python  main.py
  Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz
  Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to ../data/MNIST/raw/t10k-labels-idx1-ubyte.gz
  100.0%
  Extracting ../data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ../data/MNIST/raw

  Train Epoch: 1 [0/60000 (0%)]	Loss: 2.277304
  Train Epoch: 1 [640/60000 (1%)]	Loss: 1.823465
  ...
  Train Epoch: 14 [58880/60000 (98%)]	Loss: 0.013244
  Train Epoch: 14 [59520/60000 (99%)]	Loss: 0.000718

  Test set: Average loss: 0.0268, Accuracy: 9918/10000 (99%)

This code downloads some training MNIST data, runs a convolutional neural network based training, and gives a summary of the accuracy at the end 99%). There is no need to install PyTorch since its aready preconfigured in the `pt` environment.

Watch PyTorch Demo

Up: HPC_Usage

Retrieved from "http://training.kenet.or.ke/index.php?title=GPU_Cloud_VMs&oldid=1358"