Spring Batch in Practice: Chunks, Restarts and Making Slow Jobs Fast

Most backend systems eventually need to process a lot of data outside the request path: nightly imports, report generation, data migrations, reconciliation. You can write this as a loop in a @Scheduled method, and it works until the first time it dies halfway through 2 million rows and you have no idea where to resume. Spring Batch exists to solve exactly that class of problem.

This post covers the mental model, a minimal job, and the techniques I reach for when a batch job is too slow. Examples use Spring Boot 3 / Spring Batch 5.

The mental model

A batch application is built from a few pieces:

Job: the whole unit of work (e.g. "import customer file").
Step: one phase of a job. A job is a sequence (or flow) of steps.
JobInstance / JobExecution: a logical run (job + identifying parameters) and each attempt to run it. If an execution fails, you can launch a new execution of the same instance and it resumes.
JobRepository: a database schema where Spring Batch records all of the above. This metadata is what makes restarts possible.

Inside a typical step, work is chunk-oriented:

An ItemReader reads one item at a time.
An ItemProcessor transforms or filters it (optional).
Once chunkSize items are collected, an ItemWriter writes them all at once, inside one transaction.

That last point is the heart of it. The chunk is your transaction boundary and your checkpoint: after each commit, the framework records progress. If the job crashes, completed chunks stay committed and a restart picks up after the last one.

A minimal job

@Configuration
public class ImportJobConfig {

    @Bean
    Job importJob(JobRepository jobRepository, Step importStep) {
        return new JobBuilder("importJob", jobRepository)
                .start(importStep)
                .build();
    }

    @Bean
    Step importStep(JobRepository jobRepository,
                    PlatformTransactionManager txManager,
                    ItemReader<CustomerCsv> reader,
                    ItemProcessor<CustomerCsv, Customer> processor,
                    ItemWriter<Customer> writer) {
        return new StepBuilder("importStep", jobRepository)
                .<CustomerCsv, Customer>chunk(500, txManager)
                .reader(reader)
                .processor(processor)
                .writer(writer)
                .build();
    }

    @Bean
    FlatFileItemReader<CustomerCsv> reader() {
        return new FlatFileItemReaderBuilder<CustomerCsv>()
                .name("customerReader")
                .resource(new FileSystemResource("customers.csv"))
                .delimited()
                .names("id", "name", "email")
                .targetType(CustomerCsv.class)
                .linesToSkip(1)
                .build();
    }

    @Bean
    JdbcBatchItemWriter<Customer> writer(DataSource dataSource) {
        return new JdbcBatchItemWriterBuilder<Customer>()
                .dataSource(dataSource)
                .sql("INSERT INTO customer (id, name, email) VALUES (:id, :name, :email)")
                .beanMapped()
                .build();
    }
}

With Spring Boot, the batch schema can be initialised for you (see spring.batch.jdbc.initialize-schema) and jobs run at startup unless you set spring.batch.job.enabled=false, which you will want if you trigger jobs from a scheduler or an API.

Failure handling: skip, retry, restart

Real data is dirty and real networks are flaky. Decide explicitly how each failure type should behave:

.faultTolerant()
.skip(FlatFileParseException.class)   // bad row: log it and move on
.skipLimit(100)                        // but fail if it's more than a typo
.retry(TransientDataAccessException.class)
.retryLimit(3)

Skip for bad data. Always cap it with skipLimit and attach a SkipListener so skipped rows are written somewhere you can inspect. Silent data loss is worse than a failed job.
Retry for bad luck: deadlocks, timeouts, transient connection errors.
Restart for everything else. Fix the cause, relaunch with the same identifying parameters, and the job continues from the last committed chunk.

One subtlety: with fault tolerance enabled, when a write fails Spring Batch re-processes the chunk item by item to find the offender. If your processor has side effects (calls an API, sends a message), it can run more than once for the same item. Keep processors idempotent.

Making a slow job fast

When a job is slower than it should be, measure before changing anything. Step execution metadata (read count, write count, commit count, duration) is already in the job repository, and Spring Batch publishes Micrometer metrics too. Then, roughly in order of effort:

1. Tune the chunk size and the I/O

Per-item writes are the most common culprit. Use batched writers (JdbcBatchItemWriter, JPA with batch inserts enabled) and pick a chunk size by experiment: too small and commit overhead dominates; too large and you hold big transactions and lots of memory. Hundreds to a few thousand is a typical range, but test with your data.

Also check the reader. Reading with one query per item (an N+1 hidden in the processor) will dwarf everything else. Prefer a single paging or cursor reader, and join the data you need up front.

2. Multi-threaded step

The simplest parallelism is to give the step a TaskExecutor. Chunks are then processed concurrently:

return new StepBuilder("importStep", jobRepository)
        .<CustomerCsv, Customer>chunk(500, txManager)
        .reader(reader)
        .processor(processor)
        .writer(writer)
        .taskExecutor(new SimpleAsyncTaskExecutor("batch-"))
        .build();

This is where correctness bites. The reader is shared between threads, so it must be thread-safe:

JdbcPagingItemReader is thread-safe. JdbcCursorItemReader and FlatFileItemReader are not. Wrap them in a SynchronizedItemStreamReader.
Most readers store their position in the execution context for restarts. That state is meaningless when threads race, so disable it with saveState(false) and accept that restart granularity is weaker.
Processors and writers must not share mutable state. A stray non-thread-safe SimpleDateFormat or a shared ArrayList will cause intermittent, hard-to-reproduce failures.

If a "thread-safe" change makes a job faster but occasionally produces wrong output, it is not an optimisation. Add a test that runs the step with many threads and compares the results against a single-threaded run.

Try it. The simulation below runs 240 records through a chunk-oriented step. Change the thread count and chunk size and watch where the gains stop. With tiny chunks, commit overhead dominates; with huge chunks, some threads have nothing left to do. It is a simplified model, not a benchmark.

Spring Batch in Practice: Chunks, Restarts and Making Slow Jobs Fast

The mental model

A minimal job

Failure handling: skip, retry, restart

Making a slow job fast

1. Tune the chunk size and the I/O

2. Multi-threaded step

3. Partitioning

4. Look at the database

Operational habits worth having

Wrapping up