- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -
HugePages
What are HugePages?
Modern computer architectures use virtual memory pages. Hence if an array is allocated in memory, the pages containing the data as well as the address of the pages is stored in the page table which also is located in the memory. Thus if you access an array in memory, the system will have to access the page table first to find out where the page is stored and then in a second transaction access the data.
If you use hugepages the size of the actual page is changed from small pages (4k) to a larger size.
For a more detailed description check HugePage@Wikipedia
Why use HugePages?
By increasing the size of memory pages more physical memory is mapped within one virtual page. This increases the likelyhood that a TLB can be reused instead of recalculated. This can improve the application performance. For more details see HLRS statment
How to use HugePages?
The following modules for HugePages are available on hazelhen:
craype-hugepages2M craype-hugepages32M craype-hugepages512M craype-hugepages4M craype-hugepages64M craype-hugepages8M craype-hugepages128M craype-hugepages16M craype-hugepages256M
There are more Huge Pages modules available, but any page size below 1M is not supported on a Cray XC like Hazel Hen.
Self compiled applications
If you want to use HugePages with an application you compile yourself, you have to do both of the following steps if you compile yourself:
- During compilation
- One of the craype-hugepages<size> (i.e. craype-hugepages16M) modules has to be loaded.
- This will link the hugepages library to your application (size do not matter for linking).
- During runtime
- Load the craype-hugepages<size> module with the pagesize you want to use.
- The size chosen during runtime is completely independent from the size you have chosen during compilation.
Hence, you can compile once and test different sizes during runtime without changing the binary.
precompiled, dynamically linked applications
If you want to use HugePages with your precompiled application, which is dynamically linked (i.e. ISV codes), you can use the LD_PRELOAD environment variable. Therefore, you have to set the following environment variables in your batch script.
export LD_PRELOAD=/usr/lib64/libhugetlbfs.so export HUGETLB_DEFAULT_PAGE_SIZE=8M # The size can be varied export HUGETLB_MORECORE_HEAPBASE=10000000000 export HUGETLB_MORECORE=yes export HUGETLB_ELFMAP=W export HUGETLB_FORCE_ELFMAP=yes+
If you want to use HugePages only on a specific binary, i.e. because you are using a lot of system applications like ls, cp, etc. in your script you can set the variable below to use HugePages on specified binary only.
export HUGETLB_RESTRICT_EXE=a.out:b.out
If you want to check if your application really uses hugepages you can set
export HUGETLB_VERBOSE=99 export HUGETLB_DEBUG=yes
Note: If you are using HugePages your application won't use the first touch policy for memory application, but the memory will be used as soon as you do the allocate. This can cause trouble if you are using threads across numa nodes.
Please report issues with compiling, running and performance to the ticket system.