Reading large files in PHP without killing your server

A PHP script that processes a file of a few megabytes works fine. The same script on a 70 MB file dies with Allowed memory size exhausted. The cause is nearly always the same: the whole file is loaded into memory when walking through it would have been enough. This article measures the different reading strategies on a real file and shows which one to pick for which job.

The measurement protocol

Test file: a 73.1 MB CSV, 2,000,000 rows, four columns.

<?php

$h = fopen('gros.csv', 'wb');
$tampon = '';

for ($i = 1; $i <= 2_000_000; $i++) {
    $tampon .= $i . ',client-' . str_pad((string) ($i % 999999), 6, '0', STR_PAD_LEFT)
        . ',' . (($i % 977) + 0.5)
        . ',2026-' . str_pad((string) (($i % 12) + 1), 2, '0', STR_PAD_LEFT) . "-15\n";

    if (($i % 50_000) === 0) {
        fwrite($h, $tampon);
        $tampon = '';
    }
}

fwrite($h, $tampon);
fclose($h);

Each strategy runs in its own process, with memory_limit at 512 MB, inside a container capped at 1 core and 1 GB. That part is not optional: memory_get_peak_usage() keeps the highest figure reached since the script started, so measuring several strategies in the same process produces nonsense, the first measurement contaminates every one after it.

<?php

declare(strict_types=1);

$t0 = hrtime(true);
$resultat = /* … the strategy under test … */;

printf(
    "%7.0f ms  pic %7.1f Mo  %s\n",
    (hrtime(true) - $t0) / 1e6,
    memory_get_peak_usage(true) / 1_048_576,
    $resultat,
);

A note on method that cost us an hour: the first measurements were taken on a folder mounted from macOS into Docker. The number of lines read went down on every run, 1,990,083, then 1,989,958, then 1,989,132, while the file size did not move. The mount was truncating long sequential reads. Moving the file into a Docker volume made the measurements reproducible, with exactly 2,000,000 lines on every pass. A measurement that varies for no reason is a wrong measurement, not a noisy one.

Walking through a file line by line

Four ways to count lines, same file, same machine.

// 1. file(): the whole file in an array
$lignes = file('gros.csv', FILE_IGNORE_NEW_LINES);
$n = count($lignes);

// 2. fgets accumulated in an array
$lignes = [];
$h = fopen('gros.csv', 'rb');
while (($ligne = fgets($h)) !== false) {
    $lignes[] = rtrim($ligne, "\r\n");
}
fclose($h);
$n = count($lignes);

// 3. Generator
function lireLignes(string $chemin): Generator
{
    $h = fopen($chemin, 'rb');
    try {
        while (($ligne = fgets($h)) !== false) {
            yield rtrim($ligne, "\r\n");
        }
    } finally {
        fclose($h);
    }
}

$n = 0;
foreach (lireLignes('gros.csv') as $ligne) {
    $n++;
}

// 4. SplFileObject
$f = new SplFileObject('gros.csv', 'rb');
$f->setFlags(SplFileObject::DROP_NEW_LINE | SplFileObject::READ_AHEAD);
$n = 0;
foreach ($f as $ligne) {
    if ($ligne !== false && $ligne !== '') {
        $n++;
    }
}

That is a ratio of 114 to one between file() and the generator. The timings, meanwhile, are all within touching distance: between 490 and 660 ms, with more spread from one pass to the next than between the strategies themselves. In other words, not loading the file into memory costs nothing in speed and divides consumption by a hundred.

The 229 MB figure for a 73 MB file often comes as a surprise. A PHP array of two million short strings does not just store the characters: every entry carries an array structure and every string a header. The overhead is far bigger than the data itself.

What that changes in practice

With memory_limit = 128M, a common value on shared hosting:

file()        -> Fatal error: Allowed memory size of 134217728 bytes exhausted
                 (tried to allocate 16777224 bytes)
générateur    -> 659 ms, pic 2,0 Mo, 2 000 000 lignes

At 192 MB, file() still fails. It needs 256 MB to get through. The generator works with 2 MB, and would work just the same on a 700 MB file: its consumption does not depend on the size of the file, only on the longest line.

Processing the columns of a CSV

The same comparison, this time summing the third column.

// A. Load everything, then split
$texte = file_get_contents('gros.csv');
$somme = 0.0;
foreach (explode("\n", $texte) as $ligne) {
    if ($ligne === '') { continue; }
    $colonnes = explode(',', $ligne);
    $somme += (float) ($colonnes[2] ?? 0);
}

// B. Read line by line and split
$h = fopen('gros.csv', 'rb');
$somme = 0.0;
while (($ligne = fgets($h)) !== false) {
    $colonnes = explode(',', $ligne);
    $somme += (float) ($colonnes[2] ?? 0);
}
fclose($h);

// C. fgetcsv, which handles quotes and escaped separators
$h = fopen('gros.csv', 'rb');
$somme = 0.0;
while (($colonnes = fgetcsv($h, 0, ',', '"', '\\')) !== false) {
    $somme += (float) ($colonnes[2] ?? 0);
}
fclose($h);

All three return the same total, down to the penny. But fgetcsv is seven times slower than explode. That gap is not a flaw: fgetcsv applies the rules of the CSV format, handling quotes, separators inside fields and enclosed line breaks. explode(',') does none of that and breaks as soon as a field contains a comma.

So the choice comes down to the nature of the file, not to performance: fgetcsv for a CSV that comes from outside, explode only for a format you can guarantee has no quotes and no escaped separators.

Since PHP 8.4, the last three parameters of fgetcsv() have to be passed explicitly to avoid a deprecation notice on the escape parameter.

Transforming a file without reading it

A different situation: compressing a file without ever needing to look at its contents. Loading the lot and handing it to gzencode() works, but stream filters do the job block by block.

// In memory
$compresse = gzencode(file_get_contents('gros.csv'), 6);
file_put_contents('gros.csv.gz', $compresse);

// Streaming
$entree = fopen('gros.csv', 'rb');
$sortie = fopen('gros.csv.gz', 'wb');

stream_filter_append($sortie, 'zlib.deflate', STREAM_FILTER_WRITE, [
    'level'  => 6,
    'window' => 31,   // 31 = gzip header; 15 would produce raw zlib
]);

stream_copy_to_stream($entree, $sortie);

fclose($entree);
fclose($sortie);

The memory saving is real, but less spectacular than on reading: stream_copy_to_stream uses a sizeable internal buffer. Both produce a file of the same size. The window parameter set to 31 is what separates a genuine gzip file from a raw zlib stream that gunzip will refuse to open.

Processing in batches

The most common case in practice: walk through a large file and push the rows in batches to a database or a queue. A generator combines nicely with an accumulator.

<?php

declare(strict_types=1);

/**
 * @param  iterable<mixed> $source
 * @return Generator<int, array>
 */
function parLots(iterable $source, int $taille = 1000): Generator
{
    $lot = [];

    foreach ($source as $element) {
        $lot[] = $element;

        if (count($lot) === $taille) {
            yield $lot;
            $lot = [];
        }
    }

    if ($lot !== []) {
        yield $lot;   // the remainder
    }
}

$pdo->beginTransaction();
$insertion = $pdo->prepare('INSERT INTO clients (reference, nom, montant) VALUES (?, ?, ?)');

foreach (parLots(lireLignes('gros.csv'), 1000) as $numero => $lot) {
    foreach ($lot as $ligne) {
        $colonnes = str_getcsv($ligne, ',', '"', '\\');
        $insertion->execute([$colonnes[0], $colonnes[1], $colonnes[2]]);
    }

    if ($numero % 50 === 0) {
        $pdo->commit();
        $pdo->beginTransaction();
    }
}

$pdo->commit();

That final yield after the loop is the line everyone forgets: without it, the last rows of the file, the ones that do not fill a whole batch, are lost in silence.

Grouping the inserts into transactions changes everything over the length of the run. One transaction per row forces a disk write every time, a commit every 50,000 rows makes it one per batch.

Time limits

Memory is not the only wall. On the command line, max_execution_time is 0: the script runs with no limit. Behind a web server it is usually 30 seconds, and the front-end server has a timeout of its own.

// In CLI, check before relying on it
if (PHP_SAPI !== 'cli') {
    throw new RuntimeException('Ce traitement doit être lancé en ligne de commande.');
}

Processing a large file has no place inside an HTTP request. The request records the work to be done, a command-line script or a queue takes care of it.

Key points

• A generator reads a file of any size with 2 MB of memory. file() and file_get_contents() load everything.
• The time penalty is nil, as the measurements show: the choice has no downside.
• fgetcsv is seven times slower than explode, and that is the price of correctness. Worth paying on a file you do not produce yourself.
• Stream filters compress without loading everything.
• Measure each strategy in its own process, or memory_get_peak_usage() will lie to you.

Controlling memory matters even more in a process that never stops: a WebSocket server in PHP measures that cost connection by connection. For processing date columns, see getting and formatting the date and time.

See also connecting to a database in PHP for the insert side of the job, and the web development hub.