mm: vmscan: do not OOM if aborting reclaim to start compaction

commit 7335084d446b83cbcb15da80497d03f0c1dc9e21 upstream.

Stable note: Not tracked in Bugzilla. This patch makes later patches
	easier to apply but otherwise has little to justify it. The
	problem it fixes was never observed but the source of the
	theoretical problem did not exist for very long.

During direct reclaim it is possible that reclaim will be aborted so that
compaction can be attempted to satisfy a high-order allocation.  If this
decision is made before any pages are reclaimed, it is possible that 0 is
returned to the page allocator potentially triggering an OOM.  This has
not been observed but it is a possibility so this patch addresses it.

Signed-off-by: Mel Gorman <>
Reviewed-by: Rik van Riel <>
Cc: Andrea Arcangeli <>
Cc: Minchan Kim <>
Cc: Dave Jones <>
Cc: Jan Kara <>
Cc: Andy Isaacson <>
Cc: Nai Xia <>
Cc: Johannes Weiner <>
Signed-off-by: Andrew Morton <>
Signed-off-by: Linus Torvalds <>
Signed-off-by: Greg Kroah-Hartman <>
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 744ffe6..b026bff 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2247,6 +2247,7 @@
 	struct zoneref *z;
 	struct zone *zone;
 	unsigned long writeback_threshold;
+	bool should_abort_reclaim;
@@ -2258,7 +2259,8 @@
 		sc->nr_scanned = 0;
 		if (!priority)
-		if (shrink_zones(priority, zonelist, sc))
+		should_abort_reclaim = shrink_zones(priority, zonelist, sc);
+		if (should_abort_reclaim)
@@ -2325,6 +2327,10 @@
 	if (oom_killer_disabled)
 		return 0;
+	/* Aborting reclaim to try compaction? don't OOM, then */
+	if (should_abort_reclaim)
+		return 1;
 	/* top priority shrink_zones still had more to do? don't OOM, then */
 	if (scanning_global_lru(sc) && !all_unreclaimable(zonelist, sc))
 		return 1;