Hyperion Cluster: GPU Programming with CUDA

GPU Programming with CUDA

In October 2010, SNS Computing added 4 CUDA cables servers to the computing environment. Each server has one Tesla C2050 GPU, one 8-core AMD Opteron 6134 processor, and 16 GB of RAM.

Two of the servers, node65.hyperion and node66.hyperion, are part of the Hyperion cluster and are only available for batch jobs submitted to SGE using the qsub command. The use of cuda can be requested by specifying '-l cuda' as part of your qsub command.

The other two nodes, coda01.sns.ias.edu and cuda02.sns.ias.edu, are available for interactive access so that you can develop and test your programs before submitting them to the cluster to run as batch jobs.

On a CUDA-capable node, you need to make sure your PATH and LD_LIBRARY_PATH environment variables include the proper settings for CUDA, This should be done automatically, but the correct commands are shown below for completeness.

bash syntax:

PATH=/usr/local/cuda/bin:$PATH
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib
export PATH LD_LIBRARY_PATH

csh syntax:

setenv PATH /usr/local/cuda/bin:$PATH
setenv LD_LIBRARY_PATH /usr/local/cuda/lib64:/usr/local/cuda/lib

You can check that your environment is setup correctly by checking to see if the cuda compiler, nvcc, is installed in your using the 'which' command:

[prentice@cuda02 ~]$ which nvcc
/usr/local/cuda/bin/nvcc

Once this is done you can compile some sample cuda code, like this code. Be sure to save with the ".cu" extension, or else nvcc won't handle it correctly.

#include <stdio.h>

#define N 10

__global__ void add(int *a, int *b, int *c) {
  int tid = blockIdx.x;
  if (tid < N) 
    c[tid] = a[tid] + b [tid];
}

int main (int argc, char *argv[]) {

  int count;
  int i; 
  cudaError_t err;
  cudaDeviceProp prop;

  int a[N], b[N], c[N];
  int *dev_a, *dev_b, *dev_c;

  // How many CUDA devices are there? 
  err = cudaGetDeviceCount(&count);
  if (err != cudaSuccess) {
    printf("Error getting device count\n");
    return(1);
  }	

  if (count == 0) {
    printf("No CUDA devices found\n");
    return(0);
  }
  printf("\nCUDA Device count = %i\n\n", count);

  // List the properties for each device
  for (i=0; i <count; i++) {
    err = cudaGetDeviceProperties(&prop, i);
    if (err != cudaSuccess) {
      printf("Error getting device properties for device %i\n", i);
      return(1);
    }
    printf("Some properties of CUDA device %i:\n", i);
    printf("=================================\n");
    printf("Name: %s\n", prop.name);
    printf("Compute capability: %i.%i\n", prop.major, prop.minor);
    printf("Number of multiprocessors: %i\n", prop.multiProcessorCount);
    printf("Total global memory: %ld bytes\n", prop.totalGlobalMem);
    printf("Shared Mem/Block: %ld bytes\n", prop.sharedMemPerBlock);
    printf("\n");
  }

  // Do a simple vector addition, a + b =c 

  // allocate memory on the CPU
  err = cudaMalloc( (void**)&dev_a, N*sizeof(int));
  if (err != cudaSuccess) {
    printf ("Error allocating memory on device\n");
    return(1);
  }
  err = cudaMalloc( (void**)&dev_b, N*sizeof(int));
  if (err != cudaSuccess) {
    printf ("Error allocating memory on device\n");
    return(1);
  }
  err = cudaMalloc( (void**)&dev_c, N*sizeof(int));
  if (err != cudaSuccess) {
    printf ("Error allocating memory on device\n");
    return(1);
  }
  
  //fill the vectors with data
  for (i = 0; i < N; i++) {
    a[i] = i;
    b[i] = i;
  }

  //copy data to device
  err = cudaMemcpy(dev_a, a,  N*sizeof(int),cudaMemcpyHostToDevice);
  if (err != cudaSuccess) {
    printf("Error copying data to device\n)");
    return(1);
  }
  err = cudaMemcpy(dev_b, b, N*sizeof(int),cudaMemcpyHostToDevice);
  if (err != cudaSuccess) {
    printf("Error copying data to device\n)");
    return(1);
  }
  
  // Call CUDA kernel to do the actual work
  add<<<N,1>>>(dev_a, dev_b, dev_c);
  
  //copy results from device to host
  err = cudaMemcpy(c, dev_c, N*sizeof(int),cudaMemcpyDeviceToHost);
  if (err != cudaSuccess) {
    printf("Error copying data from device\n)");
    return(1);
  }
  
  printf("Results of vector addition on GPU:\n");
  printf("==================================\n");
  for (i=0; i<N; i++) {
    printf("%i + %i = %i\n", a[i], b[i], c[i]);
  }

  //Free allocated memory on GPU
  cudaFree(dev_a);
  cudaFree(dev_b);
  cudaFree(dev_c);

  printf("\nSample CUDA program completed successfully\n");
  return(0);
  
}

Compile it with the nvcc command:

$ nvcc -o example example.cu

And then run the resulting program "example". If everything goes right, the output should look like that below:

$ ./example 

CUDA Device count = 1

Some properties of CUDA device 0:
=================================
Name: Tesla C2050
Compute capability: 2.0
Number of multiprocessors: 14
Total global memory: 2817982464 bytes
Shared Mem/Block: 49152 bytes

Results of vector addition on GPU:
==================================
0 + 0 = 0
1 + 1 = 2
2 + 2 = 4
3 + 3 = 6
4 + 4 = 8
5 + 5 = 10
6 + 6 = 12
7 + 7 = 14
8 + 8 = 16
9 + 9 = 18

Sample CUDA program completed successfully

Ta-da! You've just successfully run a CUDA program. Now run it in batchmode on the cluster, you just need to wrap it in a qsub script just like the other examples above. The only difference is that we are going to request the resources "cuda and exclusive" using the -l switch in the submission script:

#!/bin/bash

#$ -l cuda
#$ -l exclusive
#$ -V 
#$ -cwd 

./example

If you saved the above shell script as 'cuda.sh', you can then run it on the cluster with the qsub command:

$ qsub cuda.sh
Your job 552495 ("cuda.sh") has been submitted

If all goes well, the standard output file for this job should be identical to the output shown above for the case when the program was run on the command-line.

Natural Sciences

Natural Sciences

Hyperion Cluster: GPU Programming with CUDA

GPU Programming with CUDA