HugePages

What are HugePages?

Modern computer architectures use virtual memory pages. Hence if an array is allocated in memory, the data is stored in pages and Thus if you access an array in memory, the system will have to

For a more detailed description check [[ https://en.wikipedia.org/wiki/Page_%28computer_memory%29#Huge_pages |HugePage@Wiki ]]

How to use HugePages?

The following modules for HugePages are available on hazelhen:

craype-hugepages2M    craype-hugepages32M   craype-hugepages512M
craype-hugepages4M    craype-hugepages64M     
craype-hugepages8M    craype-hugepages128M 
craype-hugepages16M   craype-hugepages256M 

There are more modules available on the system, but these will not work for hazelhen.

Self compiled applications

If you want to use HugePages with an application you compile yourself, you have to do both of the following steps if you compile yourself:

• During compilation
  • One of the craype-hugepages<size> (i.e. craype-hugepages16M) modules has to be loaded.
  • This will link the hugepages library to your application (size do not matter for linking).
• During runtime
  • Load the craype-hugepages<size> module with the pagesize you want to use.
  • The size chosen during runtime is completely independent from the size you have chosen during compilation.

Hence, you can compile once and test different sizes during runtime without changing the binary.

precompiled, dynamically linked applications

If you want to use HugePages with your precompiled application, which is dynamically linked (i.e. ISV codes), you can use the LD_PRELOAD environment variable. Therefore, you have to set the following environment variables in your batch script.

 export LD_PRELOAD=/usr/lib64/libhugetlbfs.so
 export HUGETLB_DEFAULT_PAGE_SIZE=8M      # The size can be varied
 export HUGETLB_MORECORE_HEAPBASE=10000000000
 export HUGETLB_MORECORE=yes
 export HUGETLB_ELFMAP=W
 export HUGETLB_FORCE_ELFMAP=yes+

If you want to use HugePages only on a specific binary, i.e. because you are using a lot of system applications like ls, cp, etc. in your script you can set the variable below to use HugePages on specified binary only.

 export HUGETLB_RESTRICT_EXE=a.out:b.out        

If you want to check if your application really uses hugepages you can set

 export HUGETLB_VERBOSE=99
 export HUGETLB_DEBUG=yes

Note: If you are using HugePages your application won't use the first touch policy for memory application, but the memory will be used as soon as you do the allocate. This can cause trouble if you are using threads across numa nodes. If you run into trouble doing this please open a ticket.