Paimon + Flink Crash Course

Goal: Build enough broad and precise understanding to follow Paimon + Flink conversations, speed up later code/doc reading, and answer HLD/LLD-style questions with confidence.

Rules for this Crash Course

Keep this separate from the original 12-week learning plan.
Prefer mental models, terminology, and architecture over implementation detail.
Use pseudocode only when it clarifies the design.
Park deep dives when a simpler model is enough for now.

Checkpoints

Running Mental Models

0. Big Picture

Data platform: system that ingests, stores, transforms, serves, governs, and operates data at scale.
Streaming system: continuously processes unbounded event streams instead of waiting for a complete batch.
Flink: distributed compute/runtime engine for stateful stream and batch processing.
Paimon: lakehouse table format/storage layer that stores table data and metadata on object storage/filesystems, with snapshots, manifests, primary-key tables, changelogs, and compaction.
In a Plato-style pipeline, Flink is usually the moving compute plane; Paimon is the durable cold-tier table/storage plane.
Paimon-first is viable, but Paimon correctness in streaming writes depends on understanding Flink checkpoints and sink commits.

1. Paimon Table Layout Mental Model

A Paimon table is a base directory containing data files plus metadata files.
Data files usually store rows in Parquet/ORC/Avro.
Schema files describe columns, types, primary keys, partition keys, and table options.
Snapshots are table versions. The latest snapshot tells readers which files currently make up the table.
Manifests are metadata files that list file additions/deletions and file stats, so readers do not scan all object-storage paths.
Append tables accept inserts as new rows. They are simplest and are close to Hive-style tables plus snapshots.
Primary-key tables accept insert/update/delete/upsert semantics. They use an LSM-tree style layout inside buckets so updates can be written cheaply and merged at read/compaction time.
Commit protocol is the atomic publishing path: write data files first, then publish metadata/snapshot so readers see either the old table version or the new one, not a half-written state.
One snapshot does not mean one data file. One snapshot is a metadata version and may reference many data files through manifests.
A single commit/checkpoint can add many data files across partitions, buckets, and writer tasks.
Paimon data files are not limited to Parquet; depending on version/config, tables may use formats such as Parquet, ORC, or Avro.
In ordinary discussion, "compaction" usually means data-file compaction: merge small/overlapping files into larger cleaner files and publish a new snapshot.
Metadata can also be cleaned/compacted separately: manifest compaction reduces metadata-file overhead, and snapshot expiration removes old snapshots/unreferenced files.

1.1 Flink Checkpoint vs Paimon Commit

Flink checkpoint: runtime consistency point for source offsets and operator state.
Paimon commit: table consistency point that publishes new files as a new table snapshot.
In streaming writes, the sink normally follows a checkpoint-driven two-phase pattern:
- Phase 1: write/prepare data files and committables before/during checkpoint.
- Phase 2: after the checkpoint succeeds, commit/publish the Paimon snapshot.
This is not a classic database XA lock over Kafka and S3. It is a recoverable, checkpoint-coordinated commit protocol.
Exactly-once comes from making source progress and sink visibility advance together by checkpoint ID, with idempotent/recoverable commits.
Important nuance: the external table commit must not become visible before Flink knows the checkpoint has completed successfully. During the checkpoint, Flink records enough sink commit information to recover; after checkpoint success, the sink can safely publish the external commit.
If the sink commits externally before the checkpoint succeeds and the checkpoint later fails, Flink may replay the same Kafka records while Paimon has already exposed their output.
Kafka offsets are not treated as finally consumed just because records were read. In exactly-once pipelines, source progress is captured in Flink checkpoints; after recovery, Flink resumes from the last successful checkpointed offsets.
Paimon commit recovery is based on checkpoint-linked committables and commit identifiers. On recovery, the sink can retry pending commits and detect already-committed work instead of publishing duplicates.
"Paimon commit includes data files" means the commit publishes metadata that references prepared data files. The large data files are usually written before the snapshot commit; the commit makes them visible as part of the table.
A prepared-but-uncommitted data file must not be cleaned before the corresponding checkpoint commit can be recovered. Orphan cleanup jobs therefore need conservative age thresholds; otherwise recovery can fail because the committable references missing files.
A later checkpoint does not conceptually "wait for the previous external commit" in the same way an external commit waits for checkpoint completion. Flink completes checkpoints based on durable runtime state; sink commits happen after completion notifications. Sink implementations may still serialize commits or block/backpressure if previous commits are slow.
If checkpoint C102 completes while commit C100 is still pending, the sink state in C102 must still include unresolved committables for C100/C101 as needed. On recovery from C102, the sink should not skip older pending commits; it resumes with the pending commit queue/checkpoint-linked state.
Orphan-file cleanup is safe only with conservative retention. Prepared files for unresolved checkpoint commits may look unreferenced until their commit publishes a snapshot.
If prepared files needed by a pending commit are deleted, exactly-once recovery is broken for that checkpoint. The usual fix is operational/manual: recover from an older valid checkpoint/savepoint or restart/reprocess from a safe source offset, not "current - 1 commit" in Paimon alone.
More precise recovery rule: restore from the newest Flink checkpoint/savepoint whose source/operator/sink state does not reference the deleted prepared files. If C100 prepared files are gone and C100/C101/C102 all carry C100 pending committables, restore from C99 if available and reprocess from C99's source offsets.
"Consistent restore point" means Flink source offsets/operator state/sink state and the external Paimon snapshot are aligned. Example: restoring Flink to C99 is clean only if Paimon is also committed only up to C99, or if later commits can be safely overwritten/absorbed.
Replay is table-semantics dependent. Primary-key/upsert tables can often converge after replay because repeated updates for the same key collapse to one current row. Append tables can duplicate facts unless there is a stable event id/dedupe mechanism.
Pending prepared-file loss and committed data-file loss are different incidents:
- Pending file loss: file was written but no Paimon snapshot referenced it yet. Restore Flink to the latest clean checkpoint before the missing committable and reprocess.
- Committed file loss: a Paimon snapshot already references the file. Flink rollback alone does not repair the table; restore the object, roll back/repair the table, or replay/backfill.
If committed data must be reconstructed by replay/backfill, duplication risk depends on table semantics. Primary-key/upsert tables may converge if keys/order are correct; append tables need dedupe or carefully chosen replay boundaries.

2. Batch Mode vs Streaming Mode Mental Model

The physical table format does not become a different format.
The read/write interpretation changes.
Batch read: pick a snapshot and scan the files in that snapshot as a normal finite table.
Streaming read: start from a snapshot position and keep watching later snapshots as incremental changes.
Batch write: produce a finite set of file changes and commit once or a few times.
Streaming write: every checkpoint/commit publishes another incremental table version.

3. Why Streaming Systems Exist

A plain Kafka consumer is fine for stateless side effects, but production data pipelines need stronger semantics: parallelism, state, time, recovery, backpressure handling, and coordinated sinks.
The hard parts are not "read message and write output"; the hard parts are:
- keep correct state across crashes and rescaling,
- handle late/out-of-order events,
- commit source progress and sink output consistently,
- diagnose slow downstream systems,
- operate continuously without manual replay chaos.
Flink exists because these concerns need a distributed runtime, not just application code around a Kafka client.
Batch and streaming are often the same business logic over different input bounds: bounded historical input vs unbounded live input.

4. Flink Runtime Mental Model

A Flink job is a distributed dataflow graph: source operators, transformation operators, and sink operators connected by data streams.
Each operator can have parallel subtasks. Parallelism determines how many copies run.
Records move between operators through edges. Edges can be one-to-one, rebalance, broadcast, or keyed shuffle.
keyBy repartitions records by key so all records for the same key go to the same downstream subtask.
Keyed state is local to the subtask responsible for that key, but Flink manages checkpointing and redistribution.
Operator state is state owned by an operator subtask, not by individual keys.
Event time is time from the event payload/domain; processing time is wall-clock time at the Flink worker.
Watermarks are progress signals saying "we believe no earlier event time should arrive, except late events."
Checkpoint barriers flow through the job graph to create a consistent distributed snapshot.
For Paimon pipelines, the practical Flink concepts to master first are source offsets, parallelism, key distribution, checkpoints, sink commits, and backpressure.

5. Paimon Table Format Deeper

Catalog: namespace/service/config that lets engines discover tables and their locations/schemas/options.
Table path: physical object-storage/filesystem location where a table's data and metadata live.
Schema: column names/types plus constraints such as primary keys and partition keys.
Table options: key-value configuration that controls storage/write/read behavior.
Partition: directory-level pruning unit, usually based on low/medium-cardinality fields such as date or region.
Bucket: hash-based distribution unit inside a table/partition. Buckets help organize writes/reads and primary-key lookup/merge behavior.
Append table: insert-only logical table. Best for immutable facts/events.
Primary-key table: upsert/delete logical table. Best for current entity state or CDC-style data.
Partition is for coarse query pruning and lifecycle management; bucket is for distributing data and update/merge work.
Bad partitioning creates too many directories or poor pruning. Bad bucketing creates skew, tiny files, slow lookup/merge, or poor parallelism.

Glossary

Batch: finite input; job can eventually finish.
Stream: potentially infinite input; job is long-running.
Event: one record/fact entering the system.
Source: where data enters a compute job, e.g. Kafka/Kinesis.
Sink: where processed data is written, e.g. Paimon/StarRocks.
Table format: metadata + layout + transaction rules that make files behave like a table.
Lakehouse: object-storage-backed table storage with database-like table semantics.
Snapshot: table state at a point in time.
Checkpoint: Flink runtime snapshot of operator state and source positions.
Manifest: Paimon metadata file describing data-file/changelog-file changes and stats.
Manifest list: file that points to the manifest files for a snapshot.
Table options: key-value configuration controlling format, buckets, compaction, changelog behavior, retention, etc.
Append table: table without a primary key; mainly insert-only data.
Primary-key table: table with unique key semantics; supports insert/update/delete/upsert.
LSM tree: write-optimized structure where new sorted files are added first and later merged/compacted.
Commit protocol: rules for making a write visible atomically and recoverably.
Committable: sink-side object that says "these prepared files are ready to publish for checkpoint X."
Data-file compaction: rewrites multiple small/overlapping data files into fewer larger files.
Manifest compaction: rewrites metadata manifests to reduce metadata overhead.
Snapshot expiration: removes old table versions after retention, eventually allowing unreferenced files to be deleted.
Backpressure: downstream cannot keep up, so upstream operators slow down.
Stateful processing: computation remembers information across records, e.g. dedupe sets, counters, joins, windows.
Out-of-order event: event arrives later than other events with newer event timestamps.
Late event: event arrives after the system's event-time progress has passed the point where it would normally be included.
Job graph: Flink's execution graph of sources, operators, and sinks.
Operator: a processing node in the graph, e.g. map, filter, window, sink.
Subtask: one parallel instance of an operator.
Parallelism: number of subtasks for an operator.
keyBy: repartitions records by key so same-key records are processed together.
Keyed state: state partitioned by key and owned by keyed operator subtasks.
Operator state: state owned by an operator subtask, not scoped per key.
Watermark: event-time progress marker.
Catalog: metadata namespace that maps table names to schemas, options, and storage locations.
Partition: coarse table subdivision, often visible in directory paths, used for pruning and retention.
Bucket: hash/distribution subdivision used to organize data within a table/partition.
Primary key: columns identifying a logical row for upsert/delete semantics.

Parking Lot

Full Chandy-Lamport checkpoint algorithm.
Detailed Flink operator lifecycle.
Paimon source-code internals.
Iceberg/Hudi/Delta comparison.