- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

HugePages: Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
 
(17 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== What are HugePages? ==
== What are HugePages? ==
For an introduction for HugePages check [[ https://en.wikipedia.org/wiki/Page_%28computer_memory%29#Huge_pages |HugePage@Wiki ]]
Modern computer architectures use virtual memory pages. Hence if an array is allocated in memory, the pages containing the data as well as the address of the pages is stored in the page table which also is located in the memory. Thus if you access an array in memory, the system will have to access the page table first to find out where the page is stored and then in a second transaction access the data.
 
If you use hugepages the size of the actual page is changed from small pages (4k) to a larger size.
 
For a more detailed description check [https://en.wikipedia.org/wiki/Page_%28computer_memory%29#Huge_pages HugePage@Wikipedia]


== Why use HugePages? ==
== Why use HugePages? ==
By increasing the size of memory pages more physical memory is mapped within one virtual page. This increases the likelyhood that a TLB can be reused instead of recalculated. This can improve the application performance.
For more details see [[ Communication_on_Cray_XC40_Aries_network | HLRS statment ]]


== How to use HugePages? ==
== How to use HugePages? ==
Line 10: Line 16:
  craype-hugepages8M    craype-hugepages128M  
  craype-hugepages8M    craype-hugepages128M  
  craype-hugepages16M  craype-hugepages256M  
  craype-hugepages16M  craype-hugepages256M  
There are more modules available on the system, but these will not work for hazelhen.
There are more Huge Pages modules available, but any page size below 1M is not supported on a Cray XC like Hazel Hen.






=== Self compiled applications ===
=== Self compiled applications ===
If you want to use HugePages with an application you compile yourself, you have to do both of the steps below if you compile yourself:
If you want to use HugePages with an application you compile yourself, you have to do both of the following steps if you compile yourself:
* During Compilation
* During '''compilation'''
**one of the craype-hugepages<size> (i.e. craype-hugepages16M) modules has to be loaded.
**One of the craype-hugepages<size> (i.e. craype-hugepages16M) modules has to be loaded.
**This will link the hugepages library to your application.
**This will link the hugepages library to your application (size do not matter for linking).
* During runtime
* During '''runtime'''
** Load the craype-hugepages<size> module with the pagesize you want to use.
** Load the craype-hugepages<size> module with the pagesize you want to use.
** The size chosen during runtime is completely independent from the size you have chosen during compilation.
** The size chosen during runtime is completely independent from the size you have chosen during compilation.
** Therefore you can compile once and test different sizes during runtime without changing the binary.
Hence, you can compile once and test different sizes during runtime without changing the binary.


=== precompiled, dynamically linked applications ===
=== precompiled, dynamically linked applications ===
If you want to use HugePages with your precompiled application which is dynamically linked (i.e. ISV codes) you can use the LD_PRELOAD environment variable.
If you want to use HugePages with your precompiled application, which is dynamically linked (i.e. ISV codes), you can use the LD_PRELOAD environment variable.
Therefore you have to set the following environment variables in your batch script.
Therefore, you have to set the following environment variables in your batch script.


<pre>
  export LD_PRELOAD=/usr/lib64/libhugetlbfs.so
  export LD_PRELOAD=/usr/lib64/libhugetlbfs.so
  export HUGETLB_DEFAULT_PAGE_SIZE=8M      # The size can bevaried
  export HUGETLB_DEFAULT_PAGE_SIZE=8M      # The size can be varied
  export HUGETLB_MORECORE_HEAPBASE=10000000000
  export HUGETLB_MORECORE_HEAPBASE=10000000000
  export HUGETLB_MORECORE=yes
  export HUGETLB_MORECORE=yes
  export HUGETLB_ELFMAP=W
  export HUGETLB_ELFMAP=W
  export HUGETLB_FORCE_ELFMAP=yes+
  export HUGETLB_FORCE_ELFMAP=yes+
</pre>


If you want to use Hugepages only on a specific binary, i.e. because you are using a lot of system applications like ls, cp, etc. in your script you can set the variable below to use HugePages on specified binary only.
If you want to use HugePages only on a specific binary, i.e. because you are using a lot of system applications like ls, cp, etc. in your script you can set the variable below to use HugePages on specified binary only.
  export HUGETLB_RESTRICT_EXE=a.out         
<pre>
  export HUGETLB_RESTRICT_EXE=a.out:b.out         
</pre>




 
If you want to check if your application really uses hugepages you can set
If you want to check if your application really uses hugepages you can set:
<pre>
  export HUGETLB_VERBOSE=99
  export HUGETLB_VERBOSE=99
  export HUGETLB_DEBUG=yes
  export HUGETLB_DEBUG=yes
</pre>






Note: If you are using HugePages your application won't use the first touch policy for memory application, but the memory will be used as soon as you do the allocate.
This can cause trouble if you are using threads across numa nodes.


Note: If you are using HugePages your application won't use the first touch policy for memory application, but the memory will be used as soon as you do the allocate.
Please report issues with compiling, running and performance to the [http://www.hlrs.de/trouble-ticket-submission-form/ ticket system].
This can cause trouble if you are using threads across numa nodes. If you run into trouble doing this please open a ticket.

Latest revision as of 10:30, 22 February 2016

What are HugePages?

Modern computer architectures use virtual memory pages. Hence if an array is allocated in memory, the pages containing the data as well as the address of the pages is stored in the page table which also is located in the memory. Thus if you access an array in memory, the system will have to access the page table first to find out where the page is stored and then in a second transaction access the data.

If you use hugepages the size of the actual page is changed from small pages (4k) to a larger size.

For a more detailed description check HugePage@Wikipedia

Why use HugePages?

By increasing the size of memory pages more physical memory is mapped within one virtual page. This increases the likelyhood that a TLB can be reused instead of recalculated. This can improve the application performance. For more details see HLRS statment

How to use HugePages?

The following modules for HugePages are available on hazelhen:

craype-hugepages2M    craype-hugepages32M   craype-hugepages512M
craype-hugepages4M    craype-hugepages64M     
craype-hugepages8M    craype-hugepages128M 
craype-hugepages16M   craype-hugepages256M 

There are more Huge Pages modules available, but any page size below 1M is not supported on a Cray XC like Hazel Hen.


Self compiled applications

If you want to use HugePages with an application you compile yourself, you have to do both of the following steps if you compile yourself:

  • During compilation
    • One of the craype-hugepages<size> (i.e. craype-hugepages16M) modules has to be loaded.
    • This will link the hugepages library to your application (size do not matter for linking).
  • During runtime
    • Load the craype-hugepages<size> module with the pagesize you want to use.
    • The size chosen during runtime is completely independent from the size you have chosen during compilation.

Hence, you can compile once and test different sizes during runtime without changing the binary.

precompiled, dynamically linked applications

If you want to use HugePages with your precompiled application, which is dynamically linked (i.e. ISV codes), you can use the LD_PRELOAD environment variable. Therefore, you have to set the following environment variables in your batch script.

 export LD_PRELOAD=/usr/lib64/libhugetlbfs.so
 export HUGETLB_DEFAULT_PAGE_SIZE=8M      # The size can be varied
 export HUGETLB_MORECORE_HEAPBASE=10000000000
 export HUGETLB_MORECORE=yes
 export HUGETLB_ELFMAP=W
 export HUGETLB_FORCE_ELFMAP=yes+

If you want to use HugePages only on a specific binary, i.e. because you are using a lot of system applications like ls, cp, etc. in your script you can set the variable below to use HugePages on specified binary only.

 export HUGETLB_RESTRICT_EXE=a.out:b.out        


If you want to check if your application really uses hugepages you can set

 export HUGETLB_VERBOSE=99
 export HUGETLB_DEBUG=yes


Note: If you are using HugePages your application won't use the first touch policy for memory application, but the memory will be used as soon as you do the allocate. This can cause trouble if you are using threads across numa nodes.

Please report issues with compiling, running and performance to the ticket system.