Intermediate Usage: PyTorch and Tensorflow - Revision history

Atambo at 18:47, 20 May 2026

2026-05-20T18:47:46Z

Atambo at 18:33, 20 May 2026

2026-05-20T18:33:00Z

Atambo at 15:37, 26 March 2026

2026-03-26T15:37:24Z

Atambo at 15:13, 8 May 2025

2025-05-08T15:13:43Z

Atambo at 15:13, 8 May 2025

2025-05-08T15:13:06Z

Atambo at 15:10, 8 May 2025

2025-05-08T15:10:58Z

Atambo at 14:56, 8 May 2025

2025-05-08T14:56:31Z

Atambo at 14:51, 8 May 2025

2025-05-08T14:51:31Z

Atambo at 14:51, 8 May 2025

2025-05-08T14:51:14Z

Atambo at 14:50, 8 May 2025

2025-05-08T14:50:51Z

@@ Line 12: / Line 12: @@
     mpi/openmpi-x86_64
     ----------------------------------------------------------------- /opt/ohpc/pub/modulefiles ------------------------------
-   applications/gpu/gromacs/2024.4        applications/eng/gpu/python/conda-26.1.0-python-3.14
+    applications/eng/gpu/python/conda-26.1.0-python-3.14
-    applications/gpu/python/base-3.9.21    applications/gpu/qespresso/7.3.1
+    applications/gpu/python/base-3.14
     ---------------------------------------------------------- /usr/share/lmod/lmod/modulefiles/Core -------------------------
     lmod    settar
@@ Line 36: / Line 36: @@
 <code bash>
 $ conda env list
 base                     /scratch/lustre/apps/eng/gpu/miniconda3
 octave                   /scratch/lustre/apps/eng/gpu/miniconda3/envs/octave
 python-3.12              /scratch/lustre/apps/eng/gpu/miniconda3/envs/python-3.12
 python-3.14          *   /scratch/lustre/apps/eng/gpu/miniconda3/envs/python-3.14
 qgis                     /scratch/lustre/apps/eng/gpu/miniconda3/envs/qgis
 </code>
 we can safely ignore the base environment, and make use of the *python-3.14* conda environment, this has the two
 machine learning frameworks, Tensorflow and PyTorch.

@@ Line 12: / Line 12: @@
     mpi/openmpi-x86_64
     ----------------------------------------------------------------- /opt/ohpc/pub/modulefiles ------------------------------
-    applications/gpu/gromacs/2024.4        applications/gpu/python/conda-25.1.1-python-3.9.21 (D)
+    applications/gpu/gromacs/2024.4        applications/eng/gpu/python/conda-26.1.0-python-3.14
     applications/gpu/python/base-3.9.21    applications/gpu/qespresso/7.3.1
     ---------------------------------------------------------- /usr/share/lmod/lmod/modulefiles/Core -------------------------
@@ Line 22: / Line 22: @@
 This conda  module that appear in the prior list has both TensorFlow and PyTorch installed:
 <code bash>
-   applications/gpu/python/conda-25.1.1-python-3.9.21
+   applications/eng/gpu/python/conda-26.1.0-python-3.14
 </code>
@@ Line 28: / Line 28: @@
 We can Load the module using this Slurm command:
 <code bash>
-   module load applications/gpu/python/conda-25.1.1-python-3.9.21
+   module load applications/eng/gpu/python/conda-26.1.0-python-3.14
 </code>
@@ Line 36: / Line 36: @@
 <code bash>
 $ conda env list
-  base                   /opt/ohpc/pub/conda/instdir
+base                     /scratch/lustre/apps/eng/gpu/miniconda3
-  python-3.9.21          /opt/ohpc/pub/conda/instdir/envs/python-3.9.21
+octave                   /scratch/lustre/apps/eng/gpu/miniconda3/envs/octave
 </code>
-we can safely ignore the base environment, and make use of the *python-3.9.21* conda environment, this has the two
+we can safely ignore the base environment, and make use of the *python-3.14* conda environment, this has the two
 machine learning frameworks, Tensorflow and PyTorch.
 [[File:Pytorch_logo.png|250px]]
 <code bash>
-   $ conda activate python-3.9.21
+   $ conda activate python-3.14
    (python-3.9.21)$
 </code>
@@ Line 68: / Line 71: @@
   cd ~/localscratch/mnist
-  module load applications/gpu/python/conda-25.1.1-python-3.9.21
+  module load applications/eng/gpu/python/conda-26.1.0-python-3.14
-  conda activate python-3.9.21
+  conda activate python-3.14
   python  main.py
 </code>

← Older revision		Revision as of 15:37, 26 March 2026
Line 79:		Line 79:

	== [https://asciinema.org/a/m8HJLldFQk0SrrpOYOAIQQrYj Watch Demo] ==		== [https://asciinema.org/a/m8HJLldFQk0SrrpOYOAIQQrYj Watch Demo] ==
		+
		+	== Caveat: Downloading Data Ahead of Time ==
		+	Compute nodes will typically be sealed off from the internet, and as such, it is important to have all data aready on disk before a batch job submission as such, we can now refactor the mixed data download and training in https://raw.githubusercontent.com/pytorch/examples/refs/heads/main/mnist/main.py as shown in this repo: https://github.com/Materials-Modelling-Group/training-examples/tree/main/mnist
		+
		+
		+	the first can be run from the login node, the latter can be run from the batch script.

	Next:		Next:

@@ Line 73: / Line 73: @@
 </code>
 Finally we can submit this script to Slurm, which will run the entire process for in the background.
 <code bash>
    $ sbatch torch.job

@@ Line 76: / Line 76: @@
    $ sbatch torch.job
 </code>
 Next:

@@ Line 51: / Line 51: @@
 MNIST example from PyTorch, run these shell commands to create the working directory and retreive the files:
 <code bash>
-   $ mkdir ~/mnist    # creating a working dir
+   $ mkdir -p ~/localscratch/mnist    # creating a working dir
-   $ cd  ~/mnist      # changing directory to the working dir
+   $ cd  ~/localscratch/mnist       # changing directory to the working dir
    $ wget https://raw.githubusercontent.com/pytorch/examples/refs/heads/main/mnist/main.py
 </code>
@@ Line 67: / Line 67: @@
   #SBATCH --time=00:03:00        # Run time (hh:mm:ss) - 1.5 hours
-  cd ~/mnist
+  cd ~/localscratch/mnist
   module load applications/gpu/python/conda-25.1.1-python-3.9.21
   conda activate python-3.9.21

@@ Line 57: / Line 57: @@
 And now we can place the python script in our submission script, place the following in a plain text file called torch.job:
 <code bash>
-  #!/bin/bash
+ #!/bin/bash
-  #SBATCH -J  gputest               # Job name
+ #SBATCH -J  gputest               # Job name
-  #SBATCH -o job.%j.out         # Name of stdout output file (%j expands to jobId)
+ #SBATCH -o job.%j.out         # Name of stdout output file (%j expands to jobId)
-  #SBATCH -e %j.err             # Name of std err
+ #SBATCH -e %j.err             # Name of std err
-  #SBATCH --partition=gpu1    # Queue
+ #SBATCH --partition=gpu1    # Queue
-  #SBATCH --nodes=1             # Total number of nodes requested
+ #SBATCH --nodes=1             # Total number of nodes requested
-  #SBATCH --gres=gpu:1             # Total number of gpus requested
+ #SBATCH --gres=gpu:1             # Total number of gpus requested
-  #SBATCH --cpus-per-task=1     #
+ #SBATCH --cpus-per-task=1     #
-  #SBATCH --time=00:03:00        # Run time (hh:mm:ss) - 1.5 hours
+ #SBATCH --time=00:03:00        # Run time (hh:mm:ss) - 1.5 hours
-  cd ~/mnist
+ cd ~/mnist
-  module load applications/gpu/python/conda-25.1.1-python-3.9.21
+ module load applications/gpu/python/conda-25.1.1-python-3.9.21
-  conda activate python-3.9.21
+ conda activate python-3.9.21
-  python  main.py
+ python  main.py
 </code>
 Finally we can submit this script to Slurm, which will run the entire process for in the background.

@@ Line 33: / Line 33: @@
 == Listing Conda Environments ==
 The loaded module gives us access to a custom conda module, and we can now list the conda environments available
 <code bash>
 $ conda env list