The Linux feature transparent huge pages should be disabled on all MemSQL clusters.
What are Transparent Huge Pages?
Linux organizes RAM into pages that are ordinarily 4KB. With huge pages, Linux can instead use 2MB pages or larger. But Linux THP transparently re-organizes memory used by a process inside the kernel, merging some small pages to huge pages, and splitting some huge pages to small pages. This background process holds locks on the memory manager, and can prevent your process from accessing memory. Since MemSQL uses a lot of memory, the THP process can potentially block memory use for seconds at a time.
MemSQL, as well as most other databases, recommend THP be completely disabled at boot time on all nodes in the cluster. That means the Master Aggregator, all Child Aggregators, and all leaves.
How to Disable Transparent Huge Pages
To disable THP, add the following lines to the end of
/etc/rc.local before the exit line (if present), and reboot the affected servers:
echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag echo no > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag echo 0 > /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag echo no > /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag
On Red Hat / CentOS distributions, THP will be under redhat_transparent_hugepage, on other distributions of Linux it will just be transparent_hugepage. You can check which your system uses by checking
ls /sys/kernel/mm/, and keep only the 4 relevant settings.
khugepaged/defrag option will be 1 or 0 on newer Linux versions (e.g. CentOS 7+), but yes or no on older versions (e.g. CentOS 6). You can check which your system uses by checking
cat /sys/kernel/mm/*transparent_hugepage/khugepaged/defrag, and keep only the matching setting (i.e. if you see 1 or 0 keep the line with
echo 0, if you see yes or no keep the line with
You should end up with at least 3 lines of settings, for
khugepaged/defrag. For example, the correct configuration for CentOS 7 would be:
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag echo 0 > /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag
However you can just include all 8 settings and you should be fine.
Transparent Huge Page Symptoms
THP lag can be difficult to diagnose, as it can manifest in a number of subtle ways. The simplest issue is sporadic query times. If a query usually takes 100ms, but occasionally takes 5 seconds, it may be stuck waiting for a THP defrag on one of the leaves. Generally this will affect all queries currently active on that leaf, so you would often see a group of queries all have a latency spike.
You may also detect this as high system CPU (sometimes called red CPU). If you have hundreds of gigabytes of memory in a single node, THP defrags can be detectable as seconds long spikes of system CPU.
If you see any of these symptoms, check for THP by running
cat /sys/kernel/mm/*transparent_hugepage/enabled. If the [brackets] are around [always], you should disable THP as described above, on all nodes in your cluster. Fully remedying the issue may require restarting MemSQL, or fully restarting all of the servers. Cluster operators should disable THP before launching in production, and follow the other MemSQL best practices described here: https://docs.memsql.com/docs/installation-best-practices.