An Alternative to Stream Processing

People keep comparing Inferal to stream processing.

When someone hears “real-time data, pattern detection, automated reactions,” their mind goes to Flink, Kafka Streams, or Spark Streaming. It’s a natural comparison. But it’s a paradigm difference, not a feature gap.

The Pipeline Is the Program

Stream processing replaced batch. Instead of ETL jobs running overnight and decisions based on yesterday’s numbers, systems like Storm, Kafka Streams, and Flink made it possible to process data as it arrives. Flink brought exactly-once semantics, event-time processing, and sophisticated windowing. But the paradigm carries an assumption: the topology is the program.

Data flows from sources through operators to sinks. The shape of the pipeline is the shape of the program.

But the topology bakes in a constraint: your data must enter the pipeline before the system can reason about it.

You’re building a fraud detection system. Your primary signal is a stream of payment transactions from Kafka. Straightforward: read from the topic, apply rules, flag suspicious transactions.

Now you need to check the customer’s historical risk score, which lives in a Postgres table. You need to verify the merchant’s reputation from a third-party API. You need to cross-reference the transaction amount against the customer’s typical spending pattern, computed from a data warehouse.

Each of these requires integration work. Modern Flink has streamlined much of it: CDC connectors that talk directly to Postgres without a separate Debezium deployment, async lookup joins with built-in caching and retry, temporal table joins against JDBC sources. You’re not building from scratch.

But even in Flink SQL, each source must be explicitly connected, each join must be declared, and the query still names its sources. The fundamental model is the same: everything the system reasons about must flow through the topology.

These enrichment patterns (lookup joins, temporal joins, dimension tables, broadcast state) are some of the most complex parts of any stream processing deployment. When a decision depends on parameters from databases or external APIs, those sources must be plumbed into the processing topology before the logic can reference them.

Now consider what “pattern matching” means in this paradigm. Flink CEP detects sequences of events: event A followed by event B within a time window. SQL’s MATCH_RECOGNIZE clause brings regex-like matching to streaming queries.

Three failed logins, a success, then a high-value transfer. All within ten minutes, all from the same account. CEP handles this well: ordered events over time in a single partitioned stream.

But extend the scenario. You want to detect: a customer with a recently changed email address AND a login from a new device AND a high-value transfer to an account that’s been flagged for fraud. These signals live in different systems. The condition is simple: “these things are true simultaneously for the same customer.”

In Flink SQL (assuming sources are already configured), the query logic looks like this:

SELECT t.customer_id, t.destination_account, t.amount
FROM transfers t
JOIN logins l ON t.customer_id = l.customer_id
  AND l.login_time BETWEEN t.transfer_time - INTERVAL '1' HOUR
                        AND t.transfer_time
JOIN email_changes e ON t.customer_id = e.customer_id
  AND e.changed_at > t.transfer_time - INTERVAL '48' HOUR
JOIN fraud_flags FOR SYSTEM_TIME AS OF t.transfer_time AS f
  ON t.destination_account = f.account_id
WHERE l.device_known = FALSE
  AND t.amount > 10000
  AND f.flagged = TRUE;

Every system needs source configuration, Inferal included, so that’s not the difference. The difference is in the query: it explicitly names and joins its source tables. When source number five arrives, you add another JOIN and redeploy (Flink’s savepoints make this less disruptive than it sounds, but the query still changes).

Flink SQL can combine detection and action in a single job via INSERT INTO sink SELECT ..., and statement sets can chain multiple such statements together. But each hop is still a sink-to-source boundary: one statement writes to a table, the next reads from it. The chaining is explicit, declared in the topology.

MATCH_RECOGNIZE and CEP handle a different shape of problem: sequences on a timeline. When your condition spans multiple data sources and the question is “do these things all hold true right now?”, the query must enumerate every source it touches.

What if the Program Were the Pattern?

Inferal starts from a different question. Not “how should data flow?” but “what patterns matter?”

You write rules. A rule describes conditions: when these facts are true, infer this conclusion. The system evaluates them against available facts, wherever those facts originate.

The fraud detection example from above:

rule Suspected_Account_Takeover
  match Customer,
        Email_Change { Customer } where Age < hours(48),
        Login { Customer } where not Device_Known,
        Transfer { Customer } where Amount > 10000,
        Fraud_Flag { Transfer.Destination_Account }
  infers object Alert {
      Customer,
      Type: "account_takeover",
      Severity: "critical"
  }

Or in plain English:

When a customer’s email was changed in the last 48 hours, they logged in from an unrecognized device, initiated a transfer over $10,000, and the destination account has been flagged for fraud, create a critical account takeover alert.

This doesn’t describe a pipeline. It doesn’t specify which data source the email change comes from, or how the fraud flag gets into the system, or what order these facts should be checked. It describes a pattern: when these conditions align, generate an alert.

This matters: infers object Alert doesn’t produce a row in a result set. It creates a new fact in the system, and that fact immediately participates in further reasoning. An escalation rule watches for multiple alerts on the same customer. A notification rule triggers delivery when severity reaches a threshold. A compliance rule logs the decision chain for audit.

Each rule’s output is another rule’s input, within the same reasoning loop. There is no sink table between detection and response, no second pipeline to wire up, no separate job to deploy. The system reasons forward: one conclusion feeds the next, continuously, as facts arrive and conditions change.

That reasoning chain is source-agnostic. CDC from Postgres brings customer profile changes. A webhook from the authentication service provides login events. A periodic sync from the fraud database supplies account flags. The rule doesn’t know or care about any of that.

This is the key difference. In Flink SQL, the query names its physical sources: FROM transfers t JOIN logins l JOIN email_changes e JOIN fraud_flags f. The detection logic and the source topology are the same artifact. In Inferal, rules reference logical fact types: Email_Change, Login, Transfer, Fraud_Flag. The mapping from physical sources to fact types lives in the ingestion layer, not the rule. Adding a new data source means making its data available as facts. Existing rules just work.

Ingestion still requires work. Connecting to a new database, handling schema changes, managing backpressure: these are real engineering problems regardless of paradigm. Integration is the hardest part of any rule engine (Fresh Data In, Fast Decisions Out).

What Stream Processing Is Stretching Toward

The stream processing ecosystem recognizes these limitations. Pull queries in ksqlDB let you query materialized state directly, bypassing the stream. Flink’s Table API blurs the line between streaming and relational queries. Streaming databases like Materialize and RisingWave make the pipeline invisible: you write SQL views, the system maintains them incrementally.

But notice what stays constant. Pull queries are still backed by a topology that materializes the state. Flink SQL still requires you to enumerate every source the query touches. Materialize still needs its inputs connected as sources. The pipeline paradigm is the foundation, and every extension is layered on top of it.

When to Use What

Use a stream processor when:

• You’re transforming high-volume event streams: filtering, aggregating, enriching, routing.
• Your logic is fundamentally about data transformation: input records become output records through well-defined steps.
• Your data sources are already streams (Kafka topics, event buses, log files) and your detection logic is primarily temporal: sequences, windows, sessionization.
• You need running aggregations: counters, averages, percentiles, sliding windows over time.

Flink, Kafka Streams, and their ecosystem are excellent at these problems. The pipeline model maps naturally onto data transformation workflows.

Use a pattern-centric system when:

• Your decisions depend on the intersection of data from multiple, heterogeneous systems: databases, APIs, event streams, warehouses.
• Adding a new data source to your decision logic shouldn’t require rearchitecting your processing infrastructure.
• Your conditions are things that align, not sequences that unfold. “When these three things are true simultaneously” rather than “when A happens, then B happens, then C happens.”
• Your rules need to be readable and auditable by people who don’t understand distributed systems topology.

The paradigm you choose shapes what’s easy and what’s a workaround.

Two Starting Points

Stream processing asks: how should data flow? The answer is the topology.

Pattern-centric systems ask: what patterns matter? The answer is the rules.

You wouldn’t replace Flink for log aggregation any more than you’d use Flink to evaluate compliance rules spanning five databases and three APIs. These paradigms solve the same class of problems from different starting points. The starting point determines what’s natural and what requires engineering effort.

We started from patterns. Everything else follows from that.