Large datasets and query performance

Laravel · Advanced · 14 min

Code that works on 100 rows can exhaust memory or time out on 10 million; these tools prevent that.

Model::all() and get() load every row into memory as full Eloquent models. For batch work over big tables, stream instead:

  • chunkById(1000, function ($users) { ... }) fetches pages ordered by id using where id > last_id, which is fast and stays correct even if you update rows while iterating. Plain chunk() uses offsets, which get slower on deep pages and can skip rows when the loop modifies the filtered column.
  • lazyById(1000) does the same but returns a LazyCollection you can each, filter and map like a normal collection.
  • cursor() runs one query and hydrates models one at a time using a PHP generator. It has the lowest model memory, but the database driver may still buffer the full result set, and eager loading isn't possible.

Do less work per row: select(['id', 'email']) only the columns you need, use pluck() or value() for single fields, and push set-based work to the database with query-builder update(), increment(), insert() and upsert() instead of saving models one by one.

See what Eloquent sends: ->toRawSql() prints SQL with bindings, ->explain() shows the query plan, and DB::listen(), Telescope, Pulse or Debugbar log queries and timings. Look for missing indexes on columns used in where, orderBy and joins, and add them in a migration.

DB::transaction() wraps related writes; long-running batch jobs should commit in reasonably small chunks to avoid long locks.

Example

<?php
use App\Models\User;
use Illuminate\Support\Facades\DB;

// Bad: loads every user into memory at once
// User::all()->each(fn ($u) => ...);

// Keyset pagination: constant memory, safe while updating rows
User::where('active', false)->chunkById(1000, function ($users) {
    User::whereKey($users->modelKeys())->update(['archived_at' => now()]);
});
// LazyCollection pipeline over a big table
$emails = User::select(['id', 'email'])->lazyById(2000)
    ->filter(fn ($u) => str_ends_with($u->email, '@example.com'))
    ->map->email;
foreach ($emails as $email) { /* stream to a CSV, etc. */ }
// Inspect the SQL and plan
echo User::where('email', 'a@b.io')->toRawSql();
// select * from "users" where "email" = 'a@b.io'
dump(User::where('email', 'a@b.io')->explain());
// Set-based write instead of N model saves
DB::table('products')->upsert(
    [['sku' => 'A1', 'price' => 10], ['sku' => 'B2', 'price' => 12]],
    uniqueBy: ['sku'], update: ['price']
);

Remember

  • chunkById and lazyById stream big tables with keyset pagination
  • Select only needed columns and prefer set-based update, insert and upsert
  • Use toRawSql, explain and query logs to find slow queries and missing indexes

Watch out for

  • Using chunk() while updating the column you filter on, which silently skips rows.

Check yourself

  1. Why is chunkById safer than chunk when the callback updates the rows being filtered?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: laravel.com/docs/12.x/eloquent#chunking-results