IISc commissions Param Pravega, one of the most powerful supercomputers in India.
Feb 03, 2022, Bengaluru – The Indian Institute of Science (IISc) has deployed and commissioned Param Pravega, one of the country’s most powerful supercomputers and the largest in an Indian academic institution, as part of the National Supercomputing Mission (NSM).
The machine has a total supercomputing capacity of 3.3 petaflops and is planned to power a variety of research and educational endeavors (1 petaflop equals a quadrillion or 1015 operations per second). It was created by the Centre for Advanced Computing Development (C-DAC). In keeping with the Make in India initiative, the majority of the components required to make this system were manufactured and assembled in India, coupled with an indigenous software stack built by C-DAC.
The Department of Science and Technology (DST) and the Ministry of Electronics and Information Technology (MeitY) jointly lead NSM, which is executed by C-DAC and IISc. So far, the Mission has funded the deployment of 10 supercomputer systems with a total processing capability of 17 petaflops at IISc, IITs, IISER Pune, JNCASR, NABI-Mohali, and C-DAC. To present, around 31,00,000 computational jobs have been successfully completed by approximately 2,600 researchers across the country. These technologies have aided teachers and students in completing key R&D projects such as building genomes and drug discovery platforms, analyzing urban environmental concerns, establishing flood warning and prediction systems, and optimizing telecom networks.
The Intel Xeon Cascade Lake processors for the CPU nodes and NVIDIA Tesla V100 cards for the GPU nodes make up the Param Pravega system at IISc. An ATOS BullSequana XH2000 series system with a total peak computing power of 3.3 petaflops powers the system. C-DAC provides and supports the software stack that runs on top of the hardware. The machine is equipped with a variety of program development tools, utilities, and libraries for creating and running HPC programs.
IISc already has a cutting-edge supercomputing facility established several years ago. In 2015, the Institute procured and installed SahasraT, which was at that time the fastest supercomputer in the country. Faculty members and students have been using this facility to carry out research in various impactful and socially-relevant areas. These include research on COVID-19 and other infectious diseases, such as modeling viral entry and binding, studying interactions of proteins in bacterial and viral diseases, and designing new molecules with antibacterial and antiviral properties. Researchers have also used the facility to simulate turbulent flows for green energy technologies, study climate change and associated impacts, analyze aircraft engines and hypersonic flight vehicles, and many other research activities. These efforts are expected to ramp up significantly with Param Pravega.
The Param Pravega supercomputer consists of 11 DCLC racks of compute nodes, 2 Service racks of Master/Service nodes, and 4 Storage racks of DDN storage. The node configuration includes 2 Master nodes, 11 Login nodes, 2 Firewall nodes, 4 Management, 1 NIS Slave and 624 (CPU+GPU) compute nodes. The compute nodes are of three categories, namely, Regular CPU, High-memory CPU, and GPU nodes. All the nodes in the system are connected using Mellanox high-speed HDR-Infiniband interconnection network using a FAT-tree topology with a 1:1 subscription ratio. The system is also augmented with a 4 Petabyte parallel storage from DDN for parallel filesystem access.
Regular CPU nodes: CPU nodes are built using Intel Xeon Cascade Lake 8268 2.9 GHz processors in a 2-socket configuration with 48 cores, 192GB RAM (4 GB per core), and 480GB SSD local storage per node. There are a total of 428 such nodes on PARAM Pravega constituting 20,544 cores for CPU-only nodes for computations resulting in 1.9PF peak capability.
High-Memory CPU nodes: The system also hosts High-memory CPU-only nodes that are similar in configuration to the CPU-only nodes except that these high-memory nodes have higher RAM of 768 GB per node (16 GB per core). There are a total of 156 such nodes on this system yielding a maximum of 7488 cores for high-memory computations giving 0.694PF of peak computing capability.
GPU nodes: PARAM Pravega also hosts 40 GPU nodes. The CPU of each node consists of Intel Xeon G-6248 2.5 GHz processor in a 2-socket configuration with 40 cores, 192GB RAM, and 480GB SSD local storage. The GPU of each node is made up of two Nvidia V100 Tesla 16GB (HBM2 device memory) GPU cards. Thus, the 40 GPU nodes consist of a total of 1600 host CPU cores and 80 Nvidia V100 cards. The accelerator nodes contribute a total of 0.688 PFs (0.128 (host cpus) + 0.560 (GPUs)) computational capability.
High-Speed Parallel Storage: 4 PetaBytes of usable space is provided by a high-speed parallel Lustre filesystem with a throughput of 100 GB per second. The storage subsystem is connected to the machine using Infiniband interconnection.
High-Speed Interconnect: The system is integrated with a BullSequana XH200 Mellanox HDR Infiniband interconnection using FAT-Tree topology for MPI communications. The line speed of the interconnection is 100Gbps. Apart from this the system also has a secondary 10Gbps Ethernet connection for login and storage communication.
Software environment:
The complete system is designed to run on a Linux operating system based on the CentOS 7. x distribution. The machine includes a number of program development tools, utilities, and libraries that make developing and running HPC programs on the system’s heterogeneous hardware a breeze. GNU and Intel compilers for MPI and OpenMP parallel libraries are available on the system. The system includes the CUDA and OpenACC SDKs for use on the GPU nodes. The system also includes popular parallel mathematical, scientific, and application libraries such as Intel-MKL, GNU Scientific Library, HDF5, NetCDF, and a variety of Python-based mathematical and data manipulation libraries. In addition, the machine houses CDAC-developed system monitoring and management tools.