Peter Zijlstra d16dfc550f
mm: mmu_gather rework ...
Rework the existing mmu_gather infrastructure.
The direct purpose of these patches was to allow preemptible mmu_gather,
but even without that I think these patches provide an improvement to the
status quo.
The first 9 patches rework the mmu_gather infrastructure. For review
purpose I've split them into generic and per-arch patches with the last of
those a generic cleanup.
The next patch provides generic RCU page-table freeing, and the followup
is a patch converting s390 to use this. I've also got 4 patches from
DaveM lined up (not included in this series) that uses this to implement
gup_fast() for sparc64.
Then there is one patch that extends the generic mmu_gather batching.
After that follow the mm preemptibility patches, these make part of the mm
a lot more preemptible. It converts i_mmap_lock and anon_vma->lock to
mutexes which together with the mmu_gather rework makes mmu_gather
preemptible as well.
Making i_mmap_lock a mutex also enables a clean-up of the truncate code.
This also allows for preemptible mmu_notifiers, something that XPMEM I
think wants.
Furthermore, it removes the new and universially detested unmap_mutex.
This patch:
Remove the first obstacle towards a fully preemptible mmu_gather.
The current scheme assumes mmu_gather is always done with preemption
disabled and uses per-cpu storage for the page batches. Change this to
try and allocate a page for batching and in case of failure, use a small
on-stack array to make some progress.
Preemptible mmu_gather is desired in general and usable once i_mmap_lock
becomes a mutex. Doing it before the mutex conversion saves us from
having to rework the code by moving the mmu_gather bits inside the
pte_lock.
Also avoid flushing the tlb batches from under the pte lock, this is
useful even without the i_mmap_lock conversion as it significantly reduces
pte lock hold times.
[akpm@linux-foundation.org: fix comment tpyo]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Miller <davem@davemloft.net>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Russell King <rmk@arm.linux.org.uk>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Richard Weinberger <richard@nod.at>
Cc: Tony Luck <tony.luck@intel.com>
Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Acked-by: Hugh Dickins <hughd@google.com>
Acked-by: Mel Gorman <mel@csn.ul.ie>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Nick Piggin <npiggin@kernel.dk>
Cc: Namhyung Kim <namhyung@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2011-05-25 08:39:12 -07:00
2011-03-31 11:26:23 -03:00
2011-05-11 14:50:41 -04:00
2011-01-30 01:14:14 -08:00
2011-05-04 14:08:36 -07:00
2011-03-29 12:11:16 -07:00
2011-03-31 11:26:23 -03:00
2011-05-20 18:51:09 -03:00
2011-05-24 16:39:23 -07:00
2011-05-19 00:59:55 -07:00
2011-03-31 11:26:23 -03:00
2010-12-16 12:37:26 -05:00
2011-05-24 00:06:26 -07:00
2011-03-23 12:24:23 -07:00
2011-05-16 11:32:26 -04:00
2011-05-24 11:51:26 -07:00
2011-04-19 11:24:06 -07:00
2011-03-31 11:26:23 -03:00
2011-01-20 17:50:17 +01:00
2011-01-20 17:50:17 +01:00
2011-01-13 08:03:19 -08:00
2011-03-31 11:26:23 -03:00
2011-03-16 15:11:04 -07:00
2011-03-01 22:24:19 +03:00
2011-03-31 11:26:23 -03:00
2011-03-26 14:15:06 +00:00
2011-05-12 08:28:56 -07:00
2011-05-20 13:43:21 -07:00
2011-04-24 14:28:45 -04:00
2011-01-13 08:03:08 -08:00
2011-05-23 12:33:02 -07:00
2011-03-31 11:26:23 -03:00
2011-02-24 19:58:42 +01:00
2011-01-20 18:30:17 -08:00
2011-03-21 22:59:08 -04:00
2010-11-23 20:14:46 +00:00
2011-04-28 13:39:17 -07:00
2011-03-15 02:42:32 -04:00
2011-01-31 14:03:00 -08:00
2011-04-25 14:50:08 -04:00
2011-03-30 16:53:38 -07:00
2010-11-12 07:55:30 -08:00
2011-01-19 23:51:37 +01:00
2011-01-15 20:07:45 -05:00
2010-12-06 15:58:43 -05:00
2011-03-10 08:52:07 +01:00
2011-03-22 17:43:59 -07:00
2011-03-11 14:25:50 +00:00
2011-01-10 13:31:14 -08:00
2011-04-09 15:53:56 +02:00
2011-03-08 08:28:01 +01:00
2011-04-25 18:10:58 -07:00
2010-11-15 13:27:04 -05:00
2011-03-10 08:52:27 +01:00
2011-04-19 13:32:46 +02:00
2011-03-03 10:53:20 -05:00
2011-05-11 18:50:44 -07:00
2011-05-19 16:55:27 +09:30
2011-03-10 08:52:07 +01:00
2011-05-24 22:55:24 +10:00
2011-01-13 08:03:17 -08:00
2011-03-31 11:26:23 -03:00
2010-12-16 17:53:38 +01:00
2011-03-31 11:26:23 -03:00
2011-02-16 13:30:48 +01:00
2011-03-31 11:26:23 -03:00
2010-11-26 10:51:04 +00:00
2011-05-19 14:24:16 +02:00
2011-05-23 18:04:51 +01:00
2011-03-31 11:26:23 -03:00
2010-12-20 09:07:35 -08:00
2011-03-22 17:44:00 -07:00
2011-03-22 17:44:10 -07:00
2011-03-22 17:44:10 -07:00
2011-03-22 17:44:10 -07:00
2011-01-05 14:15:50 +01:00
2011-03-31 11:26:23 -03:00
2011-03-30 17:14:33 -07:00
2011-01-26 10:50:06 +10:00
2011-03-31 11:26:23 -03:00
2011-01-24 14:51:56 -08:00
2010-11-10 16:57:11 -08:00
2011-05-04 11:50:57 -04:00
2011-01-12 12:47:34 -05:00
2011-01-13 08:03:22 -08:00
2010-11-30 13:39:55 +00:00
2011-03-22 17:44:15 -07:00
2011-05-13 11:45:33 -07:00
2011-01-13 08:03:13 -08:00
2011-03-22 19:25:05 -07:00
2011-04-24 07:58:46 -07:00
2011-03-14 17:02:43 -07:00
2011-01-20 17:01:09 -08:00
2011-01-10 08:51:44 -08:00
2011-03-08 16:10:38 +01:00
2011-04-18 18:25:41 +10:00
2011-05-19 18:24:11 -07:00
2011-05-23 10:47:06 -05:00
2011-03-24 13:54:30 +00:00
2011-01-13 19:59:52 +00:00
2011-03-31 11:26:23 -03:00
2010-11-26 09:57:36 +01:00
2011-02-25 12:00:34 -08:00
2011-03-31 11:26:23 -03:00
2011-03-10 11:35:17 +01:00
2011-03-10 11:35:15 +01:00
2011-03-31 11:26:23 -03:00
2011-03-07 01:12:28 +05:30
2011-04-04 12:48:08 -04:00
2011-03-14 08:40:51 -07:00
2011-04-05 23:51:37 +02:00
2011-05-14 21:36:55 +01:00
2011-03-22 17:44:11 -07:00
2011-01-13 21:49:56 -08:00
2011-05-13 14:37:28 -04:00
2011-03-31 11:26:23 -03:00
2011-03-31 11:26:23 -03:00
2011-03-23 19:46:17 -07:00
2011-01-12 20:16:43 -05:00
2010-12-15 13:56:33 -05:00
2011-05-12 07:37:51 -07:00
2011-03-14 19:12:20 -04:00
2011-01-09 15:42:55 -08:00
2011-03-15 02:21:45 -04:00
2011-05-22 21:01:22 -04:00
2011-03-31 11:26:23 -03:00
2011-05-10 22:53:46 +02:00
2011-01-10 08:51:44 -08:00
2011-02-03 15:39:17 -08:00
2011-04-28 16:12:47 -04:00
2011-02-16 17:48:59 +01:00
2011-01-07 17:50:27 +11:00
2011-05-15 10:22:10 -07:00
2011-03-31 11:26:23 -03:00
2011-03-31 11:26:23 -03:00
2011-05-02 16:59:38 -07:00
2010-11-15 13:49:51 +09:00
2011-01-07 17:50:23 +11:00
2011-01-07 17:50:23 +11:00
2011-05-06 13:20:59 -07:00
2011-05-18 19:22:52 -04:00
2010-12-07 20:16:56 +01:00
2011-01-07 17:50:29 +11:00
2011-03-22 08:35:35 +01:00
2011-05-25 08:39:11 -07:00
2011-04-11 23:53:19 -07:00
2011-01-10 22:11:23 +01:00
2011-01-14 08:06:39 -08:00
2011-03-05 10:56:00 +01:00
2011-02-03 16:37:28 +01:00
2011-03-31 11:26:23 -03:00
2011-02-11 15:05:49 +01:00
2010-12-17 15:18:04 +01:00
2011-03-31 11:26:23 -03:00
2011-05-23 13:59:54 +02:00
2011-04-28 11:28:20 -07:00
2010-11-12 14:51:55 +01:00
2011-02-17 09:52:03 -08:00
2010-11-09 09:31:10 -08:00
2011-02-23 00:53:26 +00:00
2011-03-20 14:50:53 +01:00
2011-03-31 11:26:23 -03:00
2011-03-12 15:08:54 -08:00
2011-05-08 16:41:45 -07:00
2011-05-11 14:44:48 -04:00
2010-11-19 17:47:57 +08:00
2011-01-11 23:55:51 +01:00
2011-05-09 12:10:49 -07:00
2011-01-19 23:31:09 -08:00
2010-11-16 10:58:30 -08:00
2011-03-18 15:14:45 -07:00
2011-05-09 11:41:41 -07:00
2011-02-27 16:11:51 -08:00
2011-03-10 16:37:26 -08:00
2011-02-10 07:51:44 -05:00
2011-02-18 12:42:28 -08:00
2011-05-24 22:55:24 +10:00
2011-05-16 14:46:30 -04:00
2011-01-31 21:17:41 -08:00
2011-04-18 13:29:03 -07:00
2011-05-05 23:16:54 -07:00
2010-11-11 13:40:11 +01:00
2010-12-17 10:01:09 -08:00
2010-11-25 10:42:59 +09:00
2011-03-23 19:47:08 -07:00
2010-12-14 00:21:51 -05:00
2011-03-31 11:26:23 -03:00
2010-11-24 11:16:42 -08:00
2011-04-23 15:56:24 +02:00
2011-05-18 14:59:08 +02:00
2010-10-28 11:58:39 +02:00
2011-03-31 11:26:23 -03:00
2011-03-20 21:13:43 -04:00
2010-12-09 20:17:07 -08:00
2011-03-31 11:26:23 -03:00
2011-03-20 20:08:48 -04:00
2011-04-04 12:48:08 -04:00
2011-03-24 08:36:10 +01:00
2011-03-22 16:17:32 -07:00
2011-05-20 09:30:46 -03:00
2011-02-17 11:12:40 -08:00
2011-01-13 17:32:31 -08:00
2011-05-18 15:29:51 -04:00
2011-04-01 16:14:30 +11:00
2011-03-08 11:17:18 +11:00
2011-03-17 11:59:32 +11:00
2011-03-08 11:17:22 +11:00
2011-03-18 16:54:31 +00:00
2011-01-13 17:32:47 -08:00
2011-02-13 16:54:24 -08:00
2011-01-24 14:45:11 +10:30
2011-05-24 22:55:24 +10:00
2011-01-13 08:03:07 -08:00
2011-02-18 08:02:10 -08:00
2011-01-07 17:02:58 -08:00
2011-01-07 08:43:37 +01:00
2011-03-22 17:44:01 -07:00
2011-03-31 11:26:23 -03:00
2011-05-23 15:39:34 -07:00
2011-01-12 11:23:08 +02:00
2011-05-22 08:47:53 -04:00
2011-03-31 11:26:23 -03:00
2011-01-13 08:03:06 -08:00
2011-01-13 08:03:05 -08:00
2011-04-13 11:11:43 +02:00
2011-03-25 21:00:29 -07:00
2011-05-19 20:50:53 -04:00
2011-05-24 14:33:35 +02:00
2011-04-25 18:14:10 -07:00
2011-05-19 14:15:29 -07:00
2011-01-21 13:38:57 -08:00
2011-03-31 11:26:23 -03:00
2011-04-25 18:14:07 -04:00
2011-03-18 22:33:38 -07:00
2010-11-22 08:34:23 -08:00
2011-01-10 19:04:08 +01:00
2010-12-06 11:03:46 -08:00
2011-03-22 04:53:56 -03:00
2011-04-14 16:06:54 -07:00
2011-01-15 13:28:17 +02:00
2011-02-03 16:08:57 -08:00
2011-02-14 17:38:30 -08:00
2011-01-13 17:32:34 -08:00
2011-01-13 17:32:46 -08:00
2011-03-22 17:44:03 -07:00
2011-05-25 08:39:12 -07:00
2011-01-13 17:32:46 -08:00
2011-02-15 10:24:31 +01:00
2011-05-10 15:54:54 -04:00
2011-05-19 16:55:27 +09:30
2011-05-19 16:55:25 +09:30
2011-01-16 13:47:07 -05:00
2011-03-31 11:26:23 -03:00
2011-05-04 12:18:54 -07:00
2011-03-02 13:43:15 -08:00
2011-04-14 08:52:33 +02:00
2011-05-09 11:40:38 +10:00
2011-03-18 09:04:20 -04:00
2011-01-12 20:03:43 -05:00
2011-01-12 20:03:43 -05:00
2011-05-05 11:10:14 -07:00
2011-05-20 00:33:18 -04:00
2011-04-04 17:00:54 +02:00
2011-03-03 13:32:07 -08:00
2010-12-16 12:37:23 -05:00
2011-03-31 11:26:23 -03:00
2011-04-24 14:28:18 -04:00
2011-03-23 15:29:04 -04:00
2011-03-11 15:39:26 -05:00
2011-03-11 15:38:44 -05:00
2011-03-27 17:48:07 +02:00
2011-05-11 22:52:13 -04:00
2011-01-25 15:24:47 -05:00
2011-05-10 22:21:45 +09:00
2011-05-16 19:32:19 -04:00
2011-05-23 11:58:59 +02:00
2011-02-03 16:08:57 -08:00
2011-05-22 21:01:19 -04:00
2010-12-23 15:57:48 -07:00
2011-05-18 12:32:23 -06:00
2011-01-15 22:01:58 -07:00
2011-04-20 17:01:19 +10:00
2010-11-01 01:08:14 -04:00
2011-02-04 11:46:50 -07:00
2011-03-23 14:55:56 -06:00
2011-03-16 17:28:10 -07:00
2011-03-31 11:26:23 -03:00
2011-05-25 08:39:10 -07:00
2011-02-15 11:10:20 +01:00
2011-03-31 11:26:23 -03:00
2011-05-23 10:24:31 +02:00
2011-05-25 08:39:08 -07:00
2011-03-14 02:58:03 -04:00
2011-01-16 13:47:07 -05:00
2011-05-23 15:39:34 -07:00
2011-05-11 15:18:53 -07:00
2011-01-16 11:56:26 -08:00
2011-05-21 12:16:44 -07:00
2011-04-11 09:01:41 +02:00
2011-05-24 12:06:40 -07:00
2011-05-04 14:20:20 -07:00
2011-05-06 21:07:38 +02:00
2011-03-09 11:59:33 -08:00
2011-04-18 10:35:30 -07:00
2011-01-10 07:38:54 -08:00
2011-04-11 13:44:25 -07:00
2011-05-16 20:17:47 +02:00
2011-03-11 15:13:26 -05:00
2011-04-30 00:25:44 +02:00
2011-03-15 00:43:14 +01:00
2011-05-17 23:26:21 +02:00
2011-03-31 11:26:23 -03:00
2011-01-25 15:24:47 -05:00
2011-05-10 10:16:21 +02:00
2011-04-26 14:01:46 -07:00
2011-03-01 22:27:26 +03:00
2011-01-13 08:03:21 -08:00
2011-01-13 08:03:21 -08:00
2011-03-31 11:26:23 -03:00
2011-01-13 08:03:10 -08:00
2011-05-18 02:55:24 -07:00
2011-05-16 11:05:00 -07:00
2011-05-13 16:31:00 -07:00
2011-04-25 17:28:24 +02:00
2011-03-22 17:44:00 -07:00
2011-03-31 11:26:23 -03:00
2011-01-12 19:14:55 +01:00
2011-03-24 21:16:31 -04:00
2011-01-13 17:32:48 -08:00
2010-10-29 04:16:31 -04:00
2010-11-29 11:17:22 -08:00
2011-04-26 14:01:42 -07:00
2011-01-14 02:36:43 +00:00
2011-05-19 14:15:29 -07:00
2011-05-05 23:16:59 -07:00
2011-05-05 23:16:59 -07:00
2011-05-05 23:16:59 -07:00
2011-03-31 11:26:23 -03:00
2011-02-01 11:12:29 -05:00
2011-03-23 19:46:23 -07:00
2010-11-12 07:55:30 -08:00
2011-04-19 15:38:02 -04:00
2010-11-15 13:24:06 -05:00
2011-03-09 13:52:27 -05:00
2011-03-23 19:46:41 -07:00
2011-04-14 16:06:56 -07:00
2011-02-25 15:07:37 -08:00
2011-04-14 16:06:56 -07:00
2011-03-22 17:44:03 -07:00
2011-01-13 08:03:23 -08:00
2011-05-12 08:28:47 -07:00
2011-03-29 18:44:05 -07:00
2010-11-15 11:29:30 -08:00
2011-01-27 12:30:38 +01:00
2011-01-27 12:30:39 +01:00
2011-01-27 12:30:40 +01:00
2010-12-22 02:39:57 +03:00
2011-05-25 08:39:10 -07:00
2011-04-25 10:20:32 -04:00
2010-10-30 12:12:50 +02:00
2011-05-12 12:13:43 +02:00
2011-05-19 16:51:01 -07:00
2011-05-19 16:51:02 -07:00
2011-01-19 15:37:31 +09:00
2010-11-09 14:45:52 +01:00
2010-11-15 18:48:25 +09:00
2010-11-09 16:38:20 +09:00
2010-10-31 10:40:39 -04:00
2011-05-19 08:19:36 -07:00
2011-03-22 17:44:15 -07:00
2011-05-20 13:33:21 -07:00
2011-05-23 08:39:24 -07:00
2010-11-28 21:16:28 +02:00
2011-01-23 21:00:05 +02:00
2011-05-21 12:53:53 +03:00
2011-03-22 17:16:34 +09:00
2011-03-31 11:26:23 -03:00
2011-03-23 07:48:42 -07:00
2011-04-13 17:13:00 -07:00
2010-12-08 12:16:33 -08:00
2011-05-05 11:10:14 -07:00
2011-03-09 14:08:09 -08:00
2011-03-28 06:05:24 -04:00
2011-03-31 11:26:23 -03:00
2011-01-27 12:30:38 +01:00
2011-05-20 12:51:07 -07:00
2011-03-31 11:26:23 -03:00
2010-11-18 14:37:34 +01:00
2011-05-03 16:09:40 -07:00
2011-03-31 11:26:23 -03:00
2011-05-19 16:55:28 +09:30
2011-04-11 22:54:42 +02:00
2011-03-22 15:47:22 +09:00
2011-03-24 10:16:26 -07:00
2011-05-05 11:10:14 -07:00
2011-03-15 00:43:46 +01:00
2011-03-09 16:43:24 -08:00
2011-05-11 21:37:15 +02:00
2011-05-13 16:05:51 -07:00
2011-01-24 09:33:36 -08:00
2010-12-22 19:43:34 -08:00
2011-02-28 18:00:31 -08:00
2011-05-23 13:59:54 +02:00
2011-04-22 17:01:09 -07:00
2011-05-23 13:59:53 +02:00
2011-05-23 13:59:53 +02:00
2011-04-26 14:01:42 -07:00
2011-02-02 15:28:18 +01:00
2011-03-13 16:35:18 -04:00
2011-05-10 16:03:56 -04:00
2011-01-13 08:03:08 -08:00
2010-11-29 08:55:25 +11:00
2010-11-29 08:55:22 +11:00
2011-05-20 13:33:21 -07:00
2011-04-04 12:48:08 -04:00
2011-02-17 11:59:56 -08:00
2011-04-22 17:31:53 -07:00
2011-05-13 16:31:00 -07:00
2011-03-22 17:44:09 -07:00
2010-11-28 23:08:04 +01:00
2011-03-31 11:26:23 -03:00
2010-12-20 09:37:33 +01:00
2010-11-10 16:57:11 -08:00
2011-04-29 17:24:29 -07:00
2011-01-13 08:03:18 -08:00
2011-03-23 19:47:03 -07:00
2011-05-20 09:30:46 -03:00
2011-03-31 11:26:23 -03:00
2011-05-20 11:28:49 -03:00
2011-03-22 04:53:33 -03:00
2011-01-06 13:05:04 +10:00
2011-03-31 11:26:23 -03:00
2010-11-10 12:04:41 +09:00
2010-12-11 02:01:35 -05:00
2011-05-20 12:17:35 -03:00
2011-01-24 14:45:11 +10:30
2011-02-09 08:19:55 +10:30
2011-03-28 12:53:29 +01:00
2011-05-25 08:39:09 -07:00
2011-02-17 11:13:19 -08:00
2011-03-31 11:26:23 -03:00
2011-04-19 16:49:20 +03:00
2011-02-21 10:07:23 +01:00
2011-03-24 21:17:51 -04:00
2011-01-17 08:05:27 -08:00
2011-03-13 20:22:28 -07:00
2011-03-31 11:26:23 -03:00
2011-01-13 08:03:24 -08:00
2011-03-22 17:44:17 -07:00
d16dfc550f