HugePages

What are HugePages?

Modern computer architectures use virtual memory pages. Hence if an array is allocated in memory, the pages containing the data as well as the address of the pages is stored in the page table which also is located in the memory. Thus if you access an array in memory, the system will have to access the page table first to find out where the page is stored and then in a second transaction access the data.

If you use hugepages the size of the actual page is changed from small pages (4k) to a larger size.

For a more detailed description check HugePage@Wiki

Why use HugePages?

By increasing the size of memory pages more physical memory is mapped within one virtual page. This increases the likelyhood that a TLB can be reused instead of recalculated. This can improve the application performance. For more details see HLRS statment

How to use HugePages?

The following modules for HugePages are available on hazelhen:

craype-hugepages2M    craype-hugepages32M   craype-hugepages512M
craype-hugepages4M    craype-hugepages64M     
craype-hugepages8M    craype-hugepages128M 
craype-hugepages16M   craype-hugepages256M 

There are more Huge Pages modules available, but any page size below 1M is not supported on a Cray XC like Hazel Hen.

Self compiled applications

If you want to use HugePages with an application you compile yourself, you have to do both of the following steps if you compile yourself:

• During compilation
  • One of the craype-hugepages<size> (i.e. craype-hugepages16M) modules has to be loaded.
  • This will link the hugepages library to your application (size do not matter for linking).
• During runtime
  • Load the craype-hugepages<size> module with the pagesize you want to use.
  • The size chosen during runtime is completely independent from the size you have chosen during compilation.

Hence, you can compile once and test different sizes during runtime without changing the binary.

precompiled, dynamically linked applications

If you want to use HugePages with your precompiled application, which is dynamically linked (i.e. ISV codes), you can use the LD_PRELOAD environment variable. Therefore, you have to set the following environment variables in your batch script.

 export LD_PRELOAD=/usr/lib64/libhugetlbfs.so
 export HUGETLB_DEFAULT_PAGE_SIZE=8M      # The size can be varied
 export HUGETLB_MORECORE_HEAPBASE=10000000000
 export HUGETLB_MORECORE=yes
 export HUGETLB_ELFMAP=W
 export HUGETLB_FORCE_ELFMAP=yes+

If you want to use HugePages only on a specific binary, i.e. because you are using a lot of system applications like ls, cp, etc. in your script you can set the variable below to use HugePages on specified binary only.

 export HUGETLB_RESTRICT_EXE=a.out:b.out        

If you want to check if your application really uses hugepages you can set

 export HUGETLB_VERBOSE=99
 export HUGETLB_DEBUG=yes

Note: If you are using HugePages your application won't use the first touch policy for memory application, but the memory will be used as soon as you do the allocate. This can cause trouble if you are using threads across numa nodes. If you run into trouble doing this please open a ticket.