Observe, Inc. is hiring a

Software Engineer: Stream Processing

Zürich, Switzerland
Full-Time
Observe is building an Observability Cloud that allows customers to bring all their machine data (logs, metrics, traces, etc.) together in one place. At the core of the service is our data management platform which enables our users to build sophisticated streaming pipelines and express materialized views and temporal queries all without learning complex database concepts.

As a stream processing engineer, you will extend our existing batch processing platform with a state of the art streaming engine, forming a lambda architecture. You will make sure that streaming queries are compiled to tasks in a correct and efficient manner, and the tasks are scheduled and given appropriate resources on either platform (Snowflake or streaming).

The typical work may involve:
- Extend our batch and stream processing engines with new relational operators, functions, data types, message formats etc.
- Compile standing queries and ad-hoc queries from our proprietary query language (OPAL) to streaming SQL
- Optimize the execution plan for each pipeline, choosing between engines and alternative execution plans
- Determine the optimal scheduling policy for the pipelines to achieve end-to-end freshness goals
- Maintain and propagate metadata throughout the pipeline, and use the metadata to further optimize processing
- Build new capabilities to allow more flexible streaming and materialization policies

The ideal candidate should be:

- Experience with relational database internals
- Some experience with existing stream processing system (e.g., Apache Flink)
- Experience with incremental view maintenance is a plus
- Experience with Go or Java is a plus but not required
- Self-motivated, looking for impact, likes to dive into the code and get stuff done
- Teamplayer who can work independently but cares about context and knows when to reach out

This job is no longer available

Enter your email address below to get notified whenever we find a similar job post.

Unsubscribe at any time.