Ravikiran G Thirumalai
e073ae1b34
[PATCH] x86-64: Set HASHDIST_DEFAULT to 1 for x86_64 NUMA
...
Enable system hashtable memory to be distributed among nodes on x86_64 NUMA
Forcing the kernel to use node interleaved vmalloc instead of bootmem for
the system hashtable memory (alloc_large_system_hash) reduces the memory
imbalance on node 0 by around 40MB on a 8 node x86_64 NUMA box:
Before the following patch, on bootup of a 8 node box:
Node 0 MemTotal: 3407488 kB
Node 0 MemFree: 3206296 kB
Node 0 MemUsed: 201192 kB
Node 0 Active: 7012 kB
Node 0 Inactive: 512 kB
Node 0 Dirty: 0 kB
Node 0 Writeback: 0 kB
Node 0 FilePages: 1912 kB
Node 0 Mapped: 420 kB
Node 0 AnonPages: 5612 kB
Node 0 PageTables: 468 kB
Node 0 NFS_Unstable: 0 kB
Node 0 Bounce: 0 kB
Node 0 Slab: 5408 kB
Node 0 SReclaimable: 644 kB
Node 0 SUnreclaim: 4764 kB
After the patch (or using hashdist=1 on the kernel command line):
Node 0 MemTotal: 3407488 kB
Node 0 MemFree: 3247608 kB
Node 0 MemUsed: 159880 kB
Node 0 Active: 3012 kB
Node 0 Inactive: 616 kB
Node 0 Dirty: 0 kB
Node 0 Writeback: 0 kB
Node 0 FilePages: 2424 kB
Node 0 Mapped: 380 kB
Node 0 AnonPages: 1200 kB
Node 0 PageTables: 396 kB
Node 0 NFS_Unstable: 0 kB
Node 0 Bounce: 0 kB
Node 0 Slab: 6304 kB
Node 0 SReclaimable: 1596 kB
Node 0 SUnreclaim: 4708 kB
I guess it is a good idea to keep HASHDIST_DEFAULT "on" for x86_64 NUMA
since x86_64 has no dearth of vmalloc space? Or maybe enable hash
distribution for all 64bit NUMA arches? The following patch does it only
for x86_64.
I ran a HPC MPI benchmark -- 'Ansys wingsolid', which takes up quite a bit of
memory and uses up tlb entries. This was on a 4 way, 2 socket
Tyan AMD box (non vsmp), with 8G total memory (4G pernode).
The results with and without hash distribution are:
1. Vanilla - runtime of 1188.000s
2. With hashdist=1 runtime of 1154.000s
Oprofile output for the duration of run is:
1. Vanilla:
PU: AMD64 processors, speed 2411.16 MHz (estimated)
Counted L1_AND_L2_DTLB_MISSES events (L1 and L2 DTLB misses) with a unit
mask of 0x00 (No unit mask) count 500
samples % app name symbol name
163054 6.5513 libansys1.so MultiFront::decompose(int, int,
Elemset *, int *, int, int, int)
162061 6.5114 libansys3.so blockSaxpy6L_fd
162042 6.5107 libansys3.so blockInnerProduct6L_fd
156286 6.2794 libansys3.so maxb33_
87879 3.5309 libansys1.so elmatrixmultpcg_
84857 3.4095 libansys4.so saxpy_pcg
58637 2.3560 libansys4.so .st4560
46612 1.8728 libansys4.so .st4282
43043 1.7294 vmlinux-t copy_user_generic_string
41326 1.6604 libansys3.so blockSaxpyBackSolve6L_fd
41288 1.6589 libansys3.so blockInnerProductBackSolve6L_fd
2. With hashdist=1
CPU: AMD64 processors, speed 2411.13 MHz (estimated)
Counted L1_AND_L2_DTLB_MISSES events (L1 and L2 DTLB misses) with a unit
mask of 0x00 (No unit mask) count 500
samples % app name symbol name
162993 6.9814 libansys1.so MultiFront::decompose(int, int,
Elemset *, int *, int, int, int)
160799 6.8874 libansys3.so blockInnerProduct6L_fd
160459 6.8729 libansys3.so blockSaxpy6L_fd
156018 6.6826 libansys3.so maxb33_
84700 3.6279 libansys4.so saxpy_pcg
83434 3.5737 libansys1.so elmatrixmultpcg_
58074 2.4875 libansys4.so .st4560
46000 1.9703 libansys4.so .st4282
41166 1.7632 libansys3.so blockSaxpyBackSolve6L_fd
41033 1.7575 libansys3.so blockInnerProductBackSolve6L_fd
35762 1.5318 libansys1.so inner_product_sub
35591 1.5245 libansys1.so inner_product_sub2
28259 1.2104 libansys4.so addVectors
Signed-off-by: Pravin B. Shelar <pravin.shelar@calsoftinc.com >
Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org >
Signed-off-by: Shai Fultheim <shai@scalex86.org >
Signed-off-by: Andi Kleen <ak@suse.de >
Acked-by: Christoph Lameter <clameter@engr.sgi.com >
Cc: Andi Kleen <ak@suse.de >
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
2007-05-02 19:27:08 +02:00
..
2007-02-11 10:51:34 -08:00
2007-04-27 15:44:34 -03:00
2007-03-01 14:53:39 -08:00
2006-12-13 09:05:52 -08:00
2007-03-06 13:26:55 +01:00
2007-04-27 19:16:19 +01:00
2007-04-25 22:27:30 -07:00
2007-01-09 14:33:11 -08:00
2007-04-25 22:28:58 -07:00
2007-04-25 22:25:34 -07:00
2007-02-08 12:39:23 -08:00
2007-02-14 08:09:53 -08:00
2007-04-04 21:12:47 -07:00
2007-03-16 19:25:04 -07:00
2007-03-06 09:30:26 -08:00
2007-04-27 13:28:39 -07:00
2006-10-04 07:55:32 -07:00
2006-12-04 02:00:22 -05:00
2007-02-16 08:13:58 -08:00
2007-02-13 02:58:52 -05:00
2006-10-05 15:10:12 +01:00
2007-02-03 17:16:24 -05:00
2006-10-03 23:01:26 +02:00
2006-12-13 09:05:50 -08:00
2007-02-09 17:08:57 +00:00
2006-10-05 15:10:12 +01:00
2007-04-28 14:40:40 -04:00
2007-04-25 22:25:31 -07:00
2007-02-08 16:01:09 -08:00
2006-12-02 21:22:55 -08:00
2007-02-08 15:13:47 +00:00
2006-09-29 17:13:24 -07:00
2006-12-02 21:22:55 -08:00
2007-03-06 09:30:25 -08:00
2007-02-20 17:10:15 -08:00
2006-10-03 23:01:26 +02:00
2007-03-16 19:25:05 -07:00
2007-02-20 09:26:53 +00:00
2007-02-13 13:26:26 +01:00
2007-04-30 09:08:17 +02:00
2006-10-11 11:14:22 -07:00
2007-01-26 13:51:00 -08:00
2006-12-08 08:28:39 -08:00
2007-04-30 09:01:23 +02:00
2006-12-01 10:39:12 +01:00
2007-05-02 19:27:08 +02:00
2006-12-07 08:39:20 -08:00
2007-02-12 09:48:29 -08:00
2006-12-08 08:28:39 -08:00
2006-12-07 08:39:30 -08:00
2007-01-30 08:26:45 -08:00
2007-02-20 17:10:14 -08:00
2007-02-20 17:10:14 -08:00
2007-02-16 08:13:59 -08:00
2007-02-16 08:14:00 -08:00
2007-02-12 09:48:46 -08:00
2006-10-21 13:35:05 -07:00
2007-03-08 07:38:22 -08:00
2007-02-11 10:51:35 -08:00
2007-03-26 14:23:52 -07:00
2006-12-12 20:05:50 +01:00
2006-12-18 01:53:58 -08:00
2007-02-11 10:51:24 -08:00
2007-02-11 10:51:27 -08:00
2006-10-03 08:04:12 -07:00
2007-04-02 10:06:09 -07:00
2007-05-02 19:27:08 +02:00
2007-02-20 17:10:13 -08:00
2007-02-12 09:48:44 -08:00
2006-09-29 09:18:20 -07:00
2006-12-08 08:28:39 -08:00
2007-02-07 09:21:02 +11:00
2006-10-11 11:14:25 -07:00
2007-04-25 22:28:56 -07:00
2006-12-22 08:55:48 -08:00
2007-04-27 10:57:31 -07:00
2006-12-07 08:39:25 -08:00
2006-12-08 08:29:09 -08:00
2007-04-27 10:57:34 -07:00
2006-12-08 08:29:09 -08:00
2007-02-09 17:39:36 -05:00
2007-02-09 17:39:36 -05:00
2006-10-21 18:37:02 +02:00
2007-02-11 11:18:07 -08:00
2007-02-01 16:17:06 -08:00
2007-02-12 09:48:46 -08:00
2007-02-09 16:23:18 +00:00
2006-12-20 11:04:12 +01:00
2006-12-07 08:39:48 -08:00
2007-03-27 09:05:15 -07:00
2006-09-30 20:52:28 +02:00
2007-02-12 09:48:46 -08:00
2006-12-07 08:39:35 -08:00
2007-02-17 19:20:16 +01:00
2006-10-11 11:14:18 -07:00
2006-10-11 11:14:18 -07:00
2007-02-12 09:48:46 -08:00
2006-12-07 08:39:35 -08:00
2006-12-08 08:29:03 -08:00
2007-02-20 08:38:46 +00:00
2007-04-25 22:28:35 -07:00
2006-12-22 08:55:50 -08:00
2006-12-13 09:05:49 -08:00
2006-12-08 08:28:40 -08:00
2006-12-07 08:39:23 -08:00
2007-04-26 15:02:26 -07:00
2007-04-28 11:01:04 -04:00
2006-12-08 08:28:41 -08:00
2006-12-07 08:39:32 -08:00
2006-12-10 09:55:41 -08:00
2006-12-13 09:05:52 -08:00
2006-10-02 07:57:12 -07:00
2006-09-29 09:18:24 -07:00
2006-12-08 08:28:57 -08:00
2006-12-06 18:38:41 -08:00
2007-02-11 11:37:45 -08:00
2006-09-30 01:47:55 +02:00
2007-02-11 10:51:18 -08:00
2006-11-30 10:34:36 -05:00
2007-02-12 09:48:30 -08:00
2007-03-05 07:57:51 -08:00
2007-02-16 08:13:59 -08:00
2006-10-03 23:01:26 +02:00
2007-04-28 11:01:07 -04:00
2007-01-30 08:26:45 -08:00
2007-02-05 10:00:38 +01:00
2007-04-19 14:56:12 +02:00
2006-10-05 15:10:12 +01:00
2007-01-08 19:49:54 +00:00
2007-04-07 10:03:43 -07:00
2006-11-08 18:29:25 -08:00
2007-03-01 17:18:39 -08:00
2006-12-10 21:21:33 +01:00
2006-12-10 21:21:33 +01:00
2006-12-10 21:21:33 +01:00
2006-12-10 21:21:33 +01:00
2007-02-13 22:09:01 +01:00
2006-12-10 21:21:29 +01:00
2006-11-30 12:24:45 +00:00
2007-02-13 22:09:00 +01:00
2007-01-30 08:26:45 -08:00
2006-12-07 08:39:40 -08:00
2007-04-25 22:25:31 -07:00
2007-04-25 22:25:31 -07:00
2007-04-10 22:38:37 +02:00
2007-04-25 22:23:43 -07:00
2007-04-25 22:25:12 -07:00
2007-04-25 22:28:44 -07:00
2007-04-25 22:30:01 -07:00
2006-12-22 08:55:50 -08:00
2007-04-25 22:29:10 -07:00
2007-04-25 22:29:14 -07:00
2007-04-25 22:28:57 -07:00
2007-04-25 22:24:41 -07:00
2007-01-23 22:17:23 -08:00
2007-04-25 22:26:20 -07:00
2006-09-28 18:02:50 -07:00
2007-04-25 22:25:31 -07:00
2007-04-25 22:29:57 -07:00
2007-04-25 22:29:10 -07:00
2006-09-28 18:02:29 -07:00
2006-12-02 22:04:04 -08:00
2006-12-02 21:22:50 -08:00
2007-02-12 09:48:32 -08:00
2007-05-02 19:27:05 +02:00
2006-12-14 13:37:24 +01:00
2007-02-19 14:21:50 -08:00
2007-04-17 16:36:27 -07:00
2006-10-05 15:10:12 +01:00
2006-10-10 15:37:22 -07:00
2007-04-28 14:15:59 -04:00
2006-12-02 21:21:21 -08:00
2007-04-25 22:25:31 -07:00
2007-03-27 09:05:16 -07:00
2006-12-07 08:39:47 -08:00
2006-12-07 08:39:47 -08:00
2006-12-07 08:39:47 -08:00
2006-10-04 00:31:09 -07:00
2007-04-25 22:25:31 -07:00
2006-11-05 14:11:24 -08:00
2007-03-01 14:53:38 -08:00
2006-11-26 16:27:17 -08:00
2006-12-08 08:28:57 -08:00
2006-12-08 08:29:00 -08:00
2006-11-30 05:24:39 +01:00
2006-12-07 08:39:40 -08:00
2006-12-07 08:39:40 -08:00
2007-04-25 22:25:52 -07:00
2007-02-16 08:13:56 -08:00
2007-04-30 16:40:39 -07:00
2007-03-16 19:25:05 -07:00
2007-04-25 22:29:41 -07:00
2007-04-04 21:12:47 -07:00
2007-04-30 16:40:40 -07:00
2006-12-04 12:45:29 -08:00
2007-02-16 14:37:06 +00:00
2007-04-26 15:46:23 -07:00
2007-02-11 10:51:32 -08:00
2007-02-23 14:54:57 -08:00
2007-04-27 10:57:31 -07:00
2006-12-07 08:39:38 -08:00
2007-02-20 17:10:14 -08:00
2007-04-25 22:29:49 -07:00
2007-03-04 11:12:40 +02:00
2007-03-04 11:12:42 +02:00
2006-10-01 00:39:19 -07:00
2007-02-20 09:26:53 +00:00
2006-09-29 09:18:13 -07:00
2007-04-28 14:51:33 -04:00
2007-02-11 10:51:32 -08:00
2007-03-22 19:39:06 -07:00
2007-02-17 19:17:37 +01:00
2006-09-29 09:18:06 -07:00
2007-03-04 11:12:43 +02:00
2007-02-11 10:51:32 -08:00
2006-10-01 00:39:18 -07:00
2006-10-01 00:39:18 -07:00
2006-10-21 13:35:06 -07:00
2007-03-05 07:57:51 -08:00
2007-02-20 17:10:13 -08:00
2006-12-01 14:51:59 -08:00
2007-02-11 10:51:17 -08:00
2007-03-01 14:53:35 -08:00
2007-02-11 10:51:18 -08:00
2006-12-08 08:28:51 -08:00
2007-02-17 01:09:12 -05:00
2007-02-16 15:19:18 -08:00
2007-02-19 13:29:02 -08:00
2007-02-11 10:51:25 -08:00
2006-09-30 20:52:31 +02:00
2006-11-30 04:40:22 +01:00
2006-09-28 18:02:22 -07:00
2007-02-12 09:48:46 -08:00
2006-12-07 08:39:38 -08:00
2007-03-12 16:31:50 -07:00
2007-01-23 00:34:54 -05:00
2007-01-26 13:51:00 -08:00
2007-03-06 06:14:54 -05:00
2006-12-13 09:05:53 -08:00
2007-04-27 10:57:33 -07:00
2006-12-07 08:39:47 -08:00
2007-03-06 13:26:27 +01:00
2007-02-12 09:48:46 -08:00
2006-12-13 09:05:53 -08:00
2007-04-26 15:48:28 -07:00
2007-04-28 21:04:03 -07:00
2007-04-25 22:28:57 -07:00
2006-12-13 16:54:25 -08:00
2006-12-13 16:54:25 -08:00
2007-04-25 22:27:45 -07:00
2007-04-25 22:29:03 -07:00
2006-12-05 19:36:26 +00:00
2007-02-16 08:14:01 -08:00
2007-02-16 08:14:01 -08:00
2007-03-16 19:25:05 -07:00
2007-03-16 19:25:05 -07:00
2007-04-20 22:56:29 -07:00
2007-02-12 22:43:25 -08:00
2007-04-25 22:29:41 -07:00
2007-02-13 13:26:24 +01:00
2007-02-20 17:10:13 -08:00
2006-10-04 07:55:30 -07:00
2007-01-30 13:35:18 -08:00
2006-10-20 10:26:38 -07:00
2007-04-27 16:01:46 +02:00
2007-02-09 08:06:14 -08:00
2006-10-05 15:10:12 +01:00
2006-12-01 22:40:28 -05:00
2006-10-18 11:36:12 -07:00
2007-04-29 10:48:48 -07:00
2007-03-12 16:31:50 -07:00
2007-04-28 14:15:58 -04:00
2006-10-06 08:53:41 -07:00
2006-11-13 07:40:42 -08:00
2007-02-08 15:25:18 -08:00
2007-02-12 09:48:46 -08:00
2007-04-28 11:01:04 -04:00
2007-01-30 13:40:36 -08:00
2007-02-12 09:48:32 -08:00
2007-03-27 08:55:47 +02:00
2006-12-08 08:28:38 -08:00
2006-12-13 15:38:46 -08:00
2007-04-17 16:36:26 -07:00
2007-04-30 16:40:41 -07:00
2007-03-26 12:35:17 +10:00
2007-02-11 10:51:28 -08:00
2006-12-04 02:00:36 -05:00
2006-09-29 09:18:15 -07:00
2006-10-03 23:01:26 +02:00
2007-02-14 08:10:00 -08:00
2007-01-11 18:18:21 -08:00
2006-09-29 09:18:16 -07:00
2007-02-12 09:48:46 -08:00
2007-02-12 09:48:28 -08:00
2006-12-07 08:39:44 -08:00
2006-12-07 08:39:25 -08:00
2006-10-04 13:27:26 -05:00
2006-09-30 20:52:18 +02:00
2007-02-12 09:48:44 -08:00
2006-09-30 20:26:56 +02:00
2006-10-04 07:55:31 -07:00
2006-12-13 09:05:49 -08:00
2006-09-29 09:18:11 -07:00
2007-01-23 07:52:06 -08:00
2006-12-05 14:37:56 +00:00
2007-02-12 09:48:46 -08:00
2007-02-11 11:18:05 -08:00
2007-02-11 10:51:28 -08:00
2007-02-11 11:18:07 -08:00
2006-12-22 08:55:49 -08:00
2006-12-10 09:55:40 -08:00
2007-01-26 13:51:00 -08:00
2007-04-25 22:27:04 -07:00
2006-12-07 08:39:36 -08:00
2007-04-26 15:48:28 -07:00
2007-04-27 10:46:51 -07:00
2006-12-07 02:14:04 +01:00
2007-04-25 22:25:31 -07:00
2006-09-29 09:18:06 -07:00
2006-10-04 07:55:14 -07:00
2007-03-14 15:27:48 -07:00
2006-12-07 08:39:46 -08:00
2007-04-27 10:44:42 -07:00
2006-12-07 08:39:35 -08:00
2007-02-14 08:09:52 -08:00
2007-02-14 08:09:52 -08:00
2007-02-18 01:40:30 -05:00
2007-03-01 17:18:39 -08:00
2006-09-29 09:18:24 -07:00
2006-12-07 08:39:22 -08:00
2007-04-30 00:07:31 -07:00
2007-02-11 10:51:18 -08:00
2006-12-13 09:05:49 -08:00
2007-02-20 17:10:14 -08:00
2007-02-20 17:10:14 -08:00
2006-12-13 09:05:53 -08:00
2006-10-10 16:15:34 -07:00
2006-12-07 08:39:38 -08:00
2007-04-30 00:58:19 -07:00
2007-04-26 15:48:28 -07:00
2006-12-02 21:22:23 -08:00
2007-04-28 22:06:01 -04:00
2006-10-04 07:55:32 -07:00
2006-11-25 13:28:34 -08:00
2007-03-05 07:57:53 -08:00
2006-10-04 07:55:30 -07:00
2006-12-08 08:28:59 -08:00
2006-12-07 02:14:08 +01:00
2007-02-28 09:41:59 -08:00
2006-10-01 00:39:18 -07:00
2007-04-26 01:54:39 -07:00
2006-12-07 08:39:27 -08:00
2007-02-12 09:48:41 -08:00
2007-02-11 10:51:18 -08:00
2007-02-20 17:10:13 -08:00
2006-10-01 00:39:22 -07:00
2006-10-11 11:14:21 -07:00
2007-04-25 22:28:57 -07:00
2007-02-07 14:03:19 +11:00
2007-04-27 10:57:32 -07:00
2006-12-13 09:05:50 -08:00
2006-12-10 09:55:41 -08:00
2006-12-10 09:55:41 -08:00
2006-12-07 08:39:34 -08:00
2007-04-24 08:23:08 -07:00
2007-02-09 16:23:15 +00:00
2007-04-25 22:25:31 -07:00
2006-11-30 05:32:19 +01:00
2006-12-11 14:34:42 -08:00
2007-02-16 08:13:59 -08:00
2007-02-04 20:54:10 +01:00
2007-02-16 08:13:57 -08:00
2007-02-16 08:13:59 -08:00
2007-02-16 08:13:56 -08:00
2006-12-02 21:21:08 -08:00
2006-10-18 19:55:22 -07:00
2007-03-05 07:57:51 -08:00
2007-02-12 09:48:42 -08:00
2006-10-01 00:39:29 -07:00
2006-12-08 08:28:56 -08:00
2006-12-08 08:28:56 -08:00
2007-02-12 09:48:32 -08:00
2006-12-04 19:41:15 -08:00
2006-12-07 08:59:11 -08:00
2007-04-25 22:25:31 -07:00
2007-02-12 09:48:40 -08:00
2007-02-12 09:48:30 -08:00
2007-03-16 19:25:03 -07:00
2006-10-02 07:57:23 -07:00
2006-12-15 08:47:51 -08:00
2007-02-07 15:44:32 -08:00
2007-02-16 15:32:19 -08:00
2007-04-27 13:28:39 -07:00
2007-02-16 15:32:20 -08:00
2006-10-01 00:39:19 -07:00
2007-03-27 09:05:15 -07:00
2006-12-20 01:46:41 -05:00
2007-04-27 15:43:27 -03:00
2006-11-13 07:40:42 -08:00
2007-02-11 10:51:25 -08:00
2007-03-16 19:25:05 -07:00
2006-10-30 12:08:40 -08:00
2007-02-08 12:38:54 -08:00
2007-04-28 11:01:01 -04:00
2007-04-26 15:45:32 -07:00
2007-03-01 14:53:38 -08:00
2007-02-11 11:18:05 -08:00
2007-04-28 21:20:32 -07:00