Daan van Gorkum

Bridging Singapore Power Smart Meters to Home Assistant with Rust and MQTT

2026-10-23T00:00:00Z

One of the best built-in features of Home Assistant is its Energy Dashboard. Feed it cumulative electricity, water, and gas counters, and it gives you long-term hourly and daily consumption curves, cost breakdowns, and baselines against individual smart plugs and air-conditioning circuits.

In Singapore, SP Group (SP Services / SP Digital) has rolled out Advanced Metering Infrastructure (AMI) smart electricity meters across residential households. Instead of waiting for a manual meter reader once every two months, AMI meters record half-hourly electricity consumption (48 intervals per day) alongside monthly water (m³) and piped town gas (kWh) readings.

In a typical Singapore condominium or HDB flat, however, your utility meters sit inside a common riser closet in the lift lobby—meaning clamping a local CT sensor or ESPHome optical pulse reader onto the meter isn't practical. While the official SP app displays this data nicely on your phone, there is no built-in way to feed your household's own utility readings into Home Assistant for local energy automation and historical dashboards.

Fresh off building iplusliving2mqtt for my condo intercom, I wanted the exact same setup for my household utilities: a tiny, polite, self-contained Rust daemon running in Kubernetes that authenticates with my SP account and exposes my home's meter readings to Home Assistant via MQTT Auto-Discovery.

That project is sp2mqtt.

Understanding the Client Protocol: Three Design Challenges

Inspecting how the mobile client communicates with SP Digital's backend (identity.spdigital.sg and b2c.api.spdigital.sg) highlighted three interesting engineering problems to solve for a clean, unattended Kubernetes integration.

1. Standard Auth0 Android SDK Telemetry & Request Metadata

SP Digital authenticates users against https://identity.spdigital.sg using Auth0's Resource Owner Password Realm flow (http://auth0.com/oauth/grant-type/password-realm) with realm Username-Password-Authentication and audience https://profile.up.spdigital.sg/.

Like most modern Auth0 deployments, the identity provider expects requests to include standard Auth0 mobile SDK telemetry and client headers rather than bare HTTP requests with a generic library User-Agent. When a request omits the expected SDK context, Auth0 flags it for additional verification:

{
  "error": "requires_verification",
  "error_description": "Suspicious request requires verification"
}

Looking at how the client's HTTP layer is structured, outgoing requests include standard metadata headers across three areas:

Application User-Agent: Identifies the client version (Infinity/<versionName> (Android)).
Auth0 SDK Telemetry: Includes the standard Auth0-Client header (Base64URL-encoded JSON describing the Auth0.Android SDK version and target API level) along with Accept-Language: en-US.
Installation & Distributed Tracing Headers: Attaches a persistent per-install UUIDv4 (X-Installation-ID) and standard Zipkin B3 tracing headers (X-B3-TraceId and X-B3-SpanId) for request correlation.

Including the standard client headers and OAuth2 scopes (openid email profile offline_access enroll read:authenticators user_metadata me me:uportal me:eva me:rbac) allows the OAuth2 handshake to proceed smoothly into the standard Multi-Factor Authentication flow.

2. Handling SMS OTP Multi-Factor Authentication in Kubernetes

Security on utility accounts matters, and SP Digital enforces Multi-Factor Authentication (MFA). Once password authentication succeeds, Auth0 responds with HTTP 403 and {"error": "mfa_required", "mfa_token": "..."}. Completing the login follows the standard Auth0 Out-of-Band (OOB) sequence:

Querying GET /mfa/authenticators with Bearer <mfa_token> to list the account's active authenticators (such as sms|dev_...).
Calling POST /mfa/challenge (challenge_type: "oob") to request a 6-digit SMS OTP and receive an oob_code.
Submitting the 6-digit binding_code to POST /oauth/token using grant type http://auth0.com/oauth/grant-type/mfa-oob.

Because the initial scope includes offline_access, completing MFA issues a long-lived refresh_token. Subsequent token refreshes (grant_type: "refresh_token") mint fresh 1-hour access_token and id_token JWTs automatically without sending repeat SMS messages.

Still, I wanted the initial onboarding—and any future re-authentication if a session ever expires—to feel native in a headless Kubernetes deployment without needing to kubectl exec into a running pod.

To achieve that, sp2mqtt exposes the MFA workflow directly inside Home Assistant over MQTT:

When sp2mqtt starts with SP_EMAIL and SP_PASSWORD (and no cached refresh_token on its persistent volume), it publishes an SP Services Account device via MQTT Auto-Discovery with sensor.sp_services_auth_status set to MFA Required.
Pressing the 1. Send SP SMS OTP button in Home Assistant triggers the Auth0 OOB challenge and updates the status sensor to OTP Sent (XXXXXXX8632).
Typing the 6-digit code into the 2. Submit SP 6-Digit SMS OTP text box in Home Assistant completes the mfa-oob token exchange, atomically saves the session to /data/state.json, and starts the telemetry poller.

For declarative GitOps setups, sp2mqtt also supports two non-interactive authentication paths:

Pre-provisioned SP_REFRESH_TOKEN: Seeding an initial refresh_token and installation_id via a Kubernetes Secret so the daemon can mint access tokens immediately on first boot.
RFC 6238 TOTP (SP_TOTP_SECRET): If an OTP authenticator is enrolled on the account, sp2mqtt can compute the 6-digit TOTP code in memory during login without waiting for an SMS challenge.

3. Reconciling Monthly Meter Reads with Half-Hourly AMI Intervals

Authenticated requests to SP Digital's B2C API (https://b2c.api.spdigital.sg) include both the OAuth2 Authorization: Bearer <access_token> header and the X-id-token: <id_token> header.

One helpful detail when working with the API is how accounts and premises are indexed: GET /jarvis/v3/me returns both a 12-digit billing account_number (008938562736) and a 10-digit premise_id (2100102993). All /jarvis/* utility endpoints are keyed by the 10-digit premise_id rather than the billing account number.

Telemetry for each premise is served across three endpoints:

GET /jarvis/v3/smrd-uportal/{premise_id}: Returns the official cumulative meter readings (last_actual_value and last_actual_at) for electric (kWh), water (m³), and gas (kWh).
GET /jarvis/v4/charts/{premise_id}: Returns the monthly billed consumption history, reading status (Actual vs. Estimated), and neighborhood averages.
POST /jarvis/v3/ami/charts: Returns Smart Meter (AMI) interval data—either 48 half-hourly slots for a single day (grouped_by: "day") or daily totals across a calendar month (grouped_by: "month").

There is a subtle nuance when feeding /jarvis/v3/smrd-uportal directly into Home Assistant's Energy Dashboard: last_actual_value only updates on the monthly billing read date (for example, 12,186.0 kWh on the 25th of the month).

To give Home Assistant a cumulative total_increasing electricity counter that updates every 30 minutes as smart meter data arrives, sp2mqtt computes a live cumulative kWh reading:

live_cumulative_kwh = meter_reading_at_t0 + sum(ami_daily_kwh for date > t0)

By taking the official cumulative reading anchored at last_actual_at and adding every subsequent day's AMI smart meter intervals (fetching both the previous month's and current month's daily AMI series whenever last_actual_at falls in the prior month), sensor.<prefix>_<premise>_electricity_cumulative_kwh advances smoothly every half hour while staying anchored to the official billing meter read.

System Architecture

Like iplusliving2mqtt, sp2mqtt is written in Rust as a single static binary using tokio, reqwest (rustls), and rumqttc:

Event-Driven Concurrency & State Coordination

Internally, sp2mqtt runs three decoupled asynchronous tasks on the tokio runtime:

Coordinator & Poller Loop: A tokio::select! state machine that owns the SpClient and PersistedState. It wakes either on the 30-minute polling timer, on incoming DaemonCommand messages (SendSmsOtp, SubmitOtp, PollNow) over a bounded tokio::sync::mpsc channel, or on SIGTERM/SIGINT for graceful shutdown.
Reactive MQTT Bridge: Manages the persistent rumqttc connection, publishes retained discovery configurations, and subscribes to command topics (sp2mqtt/cmd/#) and Home Assistant's birth topic (homeassistant/status). Whenever the coordinator updates authentication state or premise telemetry in shared ReadinessState, it signals a tokio::sync::Notify waitlist so the MQTT task immediately flushes updated entity payloads without polling locks.
Observability & Control Server: A lightweight HTTP server on :8080 exposing /healthz, /readyz, /metrics.json, and an on-demand POST /poll trigger.

Polite Polling & Proactive Token Refresh

Since AMI smart meter data settles in 30-minute intervals, sp2mqtt defaults to a polite 30-minute polling cadence (1800 seconds)—making only a handful of lightweight HTTPS requests per half hour. Before each cycle, the token manager inspects the exp claim inside both the access_token and id_token JWTs (refreshing proactively with a 5-minute safety margin) and persists updated credentials atomically via a temporary file (state.json.tmp → rename) so an unexpected pod eviction never corrupts the cached refresh token. Transient upstream 5xx gateway responses are handled with bounded exponential backoff.

Home Assistant Entities

Once connected to the MQTT broker, sp2mqtt registers two devices in Home Assistant with zero manual YAML configuration:

1. Account & Authentication Device (`SP Services Account`)

Entity	Type	Description
Auth Status	`sensor`	`Authenticated`, `MFA Required`, `OTP Sent (...)`, or `Error` with diagnostic attributes.
Authenticated	`binary_sensor`	`ON` when tokens are valid; `OFF` if interactive MFA is needed.
1. Send SP SMS OTP	`button`	Initiates Auth0 login and triggers an SMS OTP challenge to your registered phone.
2. Submit SP 6-Digit SMS OTP	`text`	Masked input box to submit the 6-digit SMS code directly from the Home Assistant UI.
Refresh Utility Usage Now	`button`	Triggers an immediate on-demand telemetry poll across all premises.

2. Per-Premise Utility Device (`SP Premise <premise_id>`)

Entity	Class / State Class	Unit	Description
Electricity Cumulative (Live)	`energy` / `total_increasing`	`kWh`	Energy Dashboard primary sensor: Official meter reading + subsequent AMI intervals.
Electricity Meter Reading	`energy` / `total_increasing`	`kWh`	Official cumulative meter reading from `smrd-uportal`.
Electricity Today (AMI)	`energy` / `total`	`kWh`	Today's smart meter consumption across 48 half-hour slots.
Electricity Yesterday (AMI)	`energy` / `measurement`	`kWh`	Yesterday's smart meter total (with full 48-slot breakdown in attributes).
Electricity Month-to-Date (AMI)	`energy` / `total`	`kWh`	Current month's daily AMI total (with daily bars in attributes).
Electricity Latest 30m Slot	`energy` / `measurement`	`kWh`	Consumption during the most recently completed 30-minute AMI interval.
Electricity Latest 30m Power	`power` / `measurement`	`W`	Average load (`kWh * 2000 W/kWh`) during the latest 30-minute slot.
Electricity Billed Month	`energy` / `measurement`	`kWh`	Latest monthly billed electricity usage (`Actual` / `Estimated` + neighbor average).
Water Meter Reading	`water` / `total_increasing`	`m³`	Energy Dashboard water sensor: Cumulative water meter reading.
Water Billed Month	`water` / `measurement`	`m³`	Latest monthly billed water consumption.
Gas Meter Reading	`energy` / `total_increasing`	`kWh`	Energy Dashboard gas sensor: Cumulative piped town gas meter reading.
Gas Billed Month	`energy` / `measurement`	`kWh`	Latest monthly billed gas consumption.

Kubernetes Runtime Architecture

Running a stateful IoT bridge in Kubernetes that also supports interactive MFA requires careful alignment between container security, volume lifecycle, and probe semantics:

Hardened Distroless Runtime: sp2mqtt is compiled in a multi-stage build and packaged onto gcr.io/distroless/cc-debian13:nonroot. The pod runs as UID/GID 65532 with readOnlyRootFilesystem: true, allowPrivilegeEscalation: false, automountServiceAccountToken: false, and all Linux capabilities dropped—mounting only a read-only ConfigMap at /etc/sp2mqtt and a tiny PersistentVolumeClaim at /data for state.json.
Single-Writer Rollout Strategy (Recreate): Because Auth0 refresh tokens and installation metadata are persisted on a ReadWriteOnce volume, the Deployment uses strategy.type: Recreate so two replicas never race to refresh or overwrite the same OAuth2 session concurrently during rollouts.
Decoupled Readiness vs. Auth State: If /readyz required an active access_token, Kubernetes would mark an unauthenticated pod unready—and if a liveness probe were tied to authentication, the kubelet would enter a CrashLoopBackOff right while the user is waiting for an SMS OTP. Instead, /healthz verifies that the Tokio runtime is responsive, while /readyz checks that the MQTT broker connection is active (mqtt_connected: true). That way, the pod stays healthy and ready so it can expose the SP Services Account MFA controls in Home Assistant while awaiting initial OTP verification.
MQTT Last Will & Birth Synchronization: The MQTT client registers a Last Will and Testament (sp2mqtt/status → offline) and subscribes to homeassistant/status. If Home Assistant restarts while sp2mqtt is already running, receiving the online birth message triggers an immediate re-broadcast of all discovery schemas and cached premise states so sensors never show up as unavailable.

If you're in Singapore and interested in how household smart meter, water, and gas telemetry can be bridged into Home Assistant's Energy Dashboard, feel free to reach out at me@dj.vg.

CVE-2026-109618 & CVE-2026-109619: Mitigating PHP's Critical CVSS 9.2 RCE Vulnerabilities When You Cannot Immediately Upgrade

2026-10-21T00:00:00Z

On 20 October 2026, the PHP Security Team issued an rare advance warning of an upcoming emergency security release addressing two critical vulnerabilities—both rated CVSS v4.0 9.2—while reserving the right to release early if vulnerability details leaked or active exploitation appeared in the wild.

Hours later, ahead of the originally scheduled 14:00 UTC window, the PHP team published emergency releases (PHP 8.5.12, PHP 8.4.27, PHP 8.3.36, and PHP 8.2.35) and official advisories for CVE-2026-109618 (GHSA-w8hw-w9fx-j79w) and CVE-2026-109619 (GHSA-6wmf-4rv5-p3rp).

If you operate PHP in production—including Apache mod_php, PHP-FPM, FrankenPHP, LiteSpeed (lsphp), or php-cgi—you need to take action immediately.

Why You Must Act Right Now (Even If You Cannot Upgrade Today)

The PHP Security Team is explicit: updating to a patched PHP release is the only complete fix. However, millions of production systems cannot be upgraded on zero notice:

Volunteer RPM repositories lag behind: The official PHP advisory explicitly warns that rpms.remirepo.net will not have updated packages available today, urging operators on RHEL/Rocky/AlmaLinux/Fedora to switch to vendor packages, build from source, or apply immediate risk-reduction controls.
End-of-Life PHP branches (PHP 7.4, 8.0, 8.1): Roughly a large fraction of legacy enterprise workloads still run PHP 7.4–8.1. Because the vulnerabilities are so severe, the PHP team published emergency source-only backport branches (PHP 7.4, PHP 8.0, and PHP 8.1), while PHP 8.2 itself reaches End-of-Life on 31 December 2026.
Pre-script execution: The most dangerous attack paths trigger during multipart request startup and early INI/session initialization—before a single line of your application's .php code runs. Application-level authentication, framework middleware, and ini_set() calls will not protect you.

If you cannot roll out patched binaries across every server this hour, you must immediately apply the configuration, web-server (NGINX, Apache, HAProxy), and WAF mitigations detailed below to break the exploit chains while you prepare your upgrade.

Executive Summary & Official Facts

Attribute	Details
CVE Identifiers	CVE-2026-109618 (GHSA-w8hw-w9fx-j79w) & CVE-2026-109619 (GHSA-6wmf-4rv5-p3rp)
Severity	Critical — CVSS v4.0 9.2 (Unauthenticated Remote Code Execution)
At-Risk Runtimes	All web-serving PHP SAPIs: Apache `mod_php` (highest risk), PHP-FPM, FrankenPHP, LiteSpeed (`lsphp`), `php-cgi`
Patched Releases	PHP 8.5.12, PHP 8.4.27, PHP 8.3.36, PHP 8.2.35
EOL Source Backports	Official source branches published for PHP 7.4, PHP 8.0, and PHP 8.1
Remi Repo Warning	`rpms.remirepo.net` does not have updates on release day
Primary Emergency Controls	`session.upload_progress.enabled = 0`, `memory_limit = -1` (paired with cgroup limits & low `post_max_size`), `fatal_error_backtraces = 0` (PHP 8.5), plus strict proxy/WAF multipart filtering

Technical Breakdown of the Two Vulnerabilities

1. CVE-2026-109618: RCE via Out-of-Memory (OOM) Corruption (`GHSA-w8hw-w9fx-j79w`)

Historically, PHP used longjmp() when encountering an Out-of-Memory (OOM) error to bail out of the current execution context and abort the request.

In several internal contexts, request memory becomes reachable before it is fully initialized. If any memory allocation fails at that moment, longjmp() exits early without cleaning up or resetting the partially initialized structures. During request shutdown, PHP treats that uninitialized/corrupted memory as valid state and reads or branches on it—leading directly to Remote Code Execution.

How Attackers Trigger Controllable OOM During Request Startup

An attacker does not need an application memory leak to hit memory_limit. Several quirks in PHP's multipart/form-data parser allow a request to exhaust memory_limit during request startup while appearing to stay within post_max_size:

Missing or Unusable Content-Length: A multipart request without a usable declared Content-Length can bypass the initial post_max_size gate and continue feeding decoded body chunks.
Unbounded Multipart Part Headers: A single multipart part can contain an arbitrary number of headers. Because PHP's internal hash tables and zend_string structures consume far more memory than the raw wire bytes, header amplification can exhaust memory_limit well below post_max_size.
Empty File Controls & Nested Array Amplification: Empty file upload inputs do not count against max_file_uploads, yet still allocate $_FILES metadata structures. Deeply nested field names (a[b][c]...) multiply the internal HashTable allocations.

SAPI Risk Profile for CVE-2026-109618

Apache mod_php & FrankenPHP (Highest Risk): Both SAPIs unconditionally invoke the request shutdown procedure even when request startup fails (php_request_startup()). An unauthenticated attacker can trigger a multipart OOM during startup and achieve RCE during shutdown before any application code executes.
PHP-FPM & LiteSpeed (lsphp): FPM terminates the worker process if php_request_startup() fails, preventing immediate startup shutdown exploitation. However, OOM corruption can still be triggered after request startup completes, and FPM is directly exposed to CVE-2026-109619.

How PHP Patched It: The security release removes longjmp() on OOM, replacing it with a delayed soft OOM checked at the next Zend VM safe point, backed by an immediate hard _exit() if memory consumption crosses a secondary threshold.

2. CVE-2026-109619: RCE via Stale Global State & Session Upload Progress UAF (`GHSA-6wmf-4rv5-p3rp`)

Persistent PHP workers (mod_php, php-fpm, lsphp, FrankenPHP) retain global state structures—spanning error handling, session tracking, object storage, and extensions—that temporarily point to memory owned by the current request's heap.

Exploitation of CVE-2026-109619 follows a two-step Producer → Consumer model across requests handled by the same worker process:

The Producer (Request 1): A PHP operation stores a pointer to request-owned heap memory in a persistent worker global. Before the operation can reset that pointer, the request is interrupted by a fatal error (such as an OOM, uncaught error, failed require, or Maximum execution time exceeded)—or a late shutdown/cleanup warning repopulates the error state after normal shutdown cleared it. PHP frees the request heap, leaving a dangling pointer in the worker's global state.
The Consumer (Request 2 on the same worker): Before normal request initialization (php_request_startup()) resets the worker's globals, an early pre-initialization operation accesses, frees, or branches on the stale pointer, triggering a Use-After-Free (UAF) on a controlled heap layout and yielding unauthenticated Remote Code Execution.

Confirmed Exploitation Paths for CVE-2026-109619

RFC 1867 Session Upload Progress Lifecycle (Confirmed RCE on both PHP-FPM and mod_php):
Because session.upload_progress.enabled is enabled by default (1), sending a multipart/form-data request containing PHP_SESSION_UPLOAD_PROGRESS initializes session and decoder state during SAPI input parsing—before normal module initialization and without any application call to session_start().
In addition to intra-request stale zval* pointer hazards (post_bytes_processed / current_file_bytes_processed) when multipart streams stall across session.upload_progress.min_freq, interrupting an upload or request after session/decoder state is published leaves persistent session globals pointing into freed request memory. A subsequent upload progress request routed to the same worker consumes those stale globals prior to initialization, achieving full RCE on both PHP-FPM and mod_php.
PHP 8.5 fatal_error_backtraces (Confirmed RCE on mod_php):
In PHP 8.5, when fatal_error_backtraces is enabled, fatal error reporting stores request-owned backtrace state in global memory and fails to reset it during shutdown. Any early diagnostic on the next request consumes the freed backtrace structure.
Early INI Diagnostics Before Request Initialization:
Any warning, notice, or deprecation emitted while applying per-request INI overrides (via .htaccess, Apache php_value/php_flag/php_admin_*, or FastCGI PHP_VALUE/PHP_ADMIN_VALUE) accesses global error state before php_request_startup() resets it.
Examples include deprecated settings (auto_detect_line_endings=1, non-default filter.default, PHP 8.3+ assert.*, PHP 8.4+ legacy session directives, PHP 8.5 report_memleaks=0, explicit mbstring.* or iconv.* encodings) or paths rejected by open_basedir.
Crucial detail: PHP accesses the stale global state before checking error_reporting, display_errors, log_errors, or the @ suppression operator. Hiding warnings does not prevent exploitation.

Immediate Mitigations If You Cannot Upgrade PHP Today

Warning from the PHP Security Team: Only upgrading to PHP 8.5.12, 8.4.27, 8.3.36, or 8.2.35 (or applying the official source backports for 7.4/8.0/8.1) completely resolves these vulnerabilities. If you cannot upgrade immediately, you must apply the following layered controls across both PHP and your web servers/proxies and restart all PHP workers.

Step 1: Lock Down PHP Configuration (`php.ini` / FPM Pool)

Do not rely on ini_set() inside PHP scripts or .user.ini files—both run after the vulnerable pre-initialization paths have already executed. Apply these settings in global php.ini or FPM pool definitions and restart every worker:

Create /etc/php/*/fpm/conf.d/99-cve-2026-109618-109619-mitigation.ini (and the equivalent for Apache/LSAPI):

; 1. Block CVE-2026-109619 session upload progress producer & consumer paths
;    Recommended by the PHP Security Team even AFTER patching.
session.upload_progress.enabled = 0

; 2. Block CVE-2026-109619 PHP 8.5 fatal error backtrace producer path
fatal_error_backtraces = 0

; 3. Mitigate CVE-2026-109618 controllable per-request PHP OOM longjmp()
;    IMPORTANT: Pair this with strict post_max_size and cgroup memory.max limits!
memory_limit = -1

; 4. Bound multipart and input amplification vectors (CVE-2026-109618)
post_max_size = 8M          ; Use 8M if uploads aren't needed, or 32M if required
upload_max_filesize = 4M    ; Keep strictly below post_max_size
max_file_uploads = 5
max_input_vars = 500
max_input_nesting_level = 16

In your PHP-FPM pool configuration (/etc/php/*/fpm/pool.d/www.conf), enforce these with php_admin_value / php_admin_flag so no application or directory override can re-enable finite memory_limit or upload_progress:

php_admin_flag[session.upload_progress.enabled] = off
php_admin_flag[fatal_error_backtraces] = off
php_admin_value[memory_limit] = -1
php_admin_value[post_max_size] = 8M
php_admin_value[max_input_nesting_level] = 16

Critical Cgroup vs. RLIMIT Warning: When setting memory_limit = -1 to prevent attackers from triggering a predictable PHP longjmp(), enforce your process memory boundary using a container or systemd/cgroup v2 limit (MemoryMax= / memory.max) that OOM-kills the worker cleanly. NEVER use RLIMIT_AS or RLIMIT_DATA, because when RLIMIT is reached, mmap() / brk() returns NULL to PHP, which immediately triggers the exact vulnerable longjmp() OOM path!

Restart PHP-FPM or Apache immediately after applying:

systemctl restart php*-fpm

Step 2: Web Server & FastCGI Hardening (NGINX, Apache, HAProxy)

audit any PHP_VALUE or PHP_ADMIN_VALUE directives passed from NGINX, Apache, or HAProxy to ensure they never pass deprecated or invalid INI directives (which act as early diagnostic consumers for CVE-2026-109619), while enforcing session.upload_progress.enabled=0 and strict body limits at the proxy boundary.

Option A: NGINX + PHP-FPM

http {
    # Enforce decoded body size at the reverse proxy (including chunked requests)
    client_max_body_size 8m;
    client_body_timeout 10s;

    # Buffer the full client request before forwarding to PHP-FPM
    fastcgi_request_buffering on;
    proxy_request_buffering on;

    server {
        location ~ \.php$ {
            include fastcgi_params;
            fastcgi_pass unix:/run/php/php-fpm.sock;

            # Enforce administrative kill switches (ensure no deprecated INI keys are sent)
            fastcgi_param PHP_ADMIN_VALUE "session.upload_progress.enabled=0\nmemory_limit=-1";
        }
    }
}

Reload NGINX:

nginx -t && systemctl reload nginx

Option B: Apache (`mod_php` and `mod_proxy_fcgi`)

Because Apache mod_php faces the highest risk (unconditional shutdown execution on startup OOM and pre-startup .htaccess INI evaluation):

Disable .htaccess overrides (AllowOverride None) wherever possible so attackers or legacy directories cannot trigger pre-initialization INI warnings via php_value / php_flag.
Enforce request size and rate limits via LimitRequestBody and mod_reqtimeout:

# Reject oversized or un-bounded request bodies at the Apache layer (8 MB)
LimitRequestBody 8388608

<IfModule mod_reqtimeout.c>
    RequestReadTimeout header=10-20,MinRate=500 body=10,MinRate=1000
</IfModule>

<IfModule mod_php.c>
    php_admin_flag session.upload_progress.enabled off
    php_admin_flag fatal_error_backtraces off
    php_admin_value memory_limit -1
    php_admin_value post_max_size 8M
</IfModule>

<IfModule mod_proxy_fcgi.c>
    SetEnv PHP_ADMIN_VALUE "session.upload_progress.enabled=0"
</IfModule>

Restart Apache so all worker processes start with a clean global heap:

apachectl configtest && systemctl restart apache2

Option C: HAProxy (Edge Body Filtering, Buffering & FastCGI)

In HAProxy, you can simultaneously: 1. Block requests missing a valid Content-Length on multipart/form-data POSTs (preventing the CVE-2026-109618 post_max_size bypass), 2. Block PHP_SESSION_UPLOAD_PROGRESS in buffered multipart bodies (preventing the CVE-2026-109619 session lifecycle consumer/producer), and 3. Lock session.upload_progress.enabled=0 if using HAProxy's native fcgi-app.

frontend https_edge
    bind :443 ssl crt /etc/haproxy/certs/
    mode http

    # Buffer client request bodies and enforce strict timeouts
    option http-buffer-request
    timeout http-request 5s
    timeout client-fin 5s

    acl is_post method POST
    acl is_multipart req.hdr(Content-Type) -i -m sub multipart/form-data

    # 1. CVE-2026-109618: Require a valid Content-Length on multipart POSTs and cap size
    acl has_content_length req.hdr_cnt(Content-Length) eq 1
    acl body_too_large req.body_size gt 8388608
    http-request deny deny_status 411 if is_post is_multipart !has_content_length
    http-request deny deny_status 413 if is_post body_too_large

    # 2. CVE-2026-109619: Block multipart payloads containing PHP_SESSION_UPLOAD_PROGRESS
    acl has_upload_progress req.body -m sub PHP_SESSION_UPLOAD_PROGRESS
    http-request deny deny_status 403 if is_post is_multipart has_upload_progress

    default_backend php_fpm_nodes

fcgi-app php-fpm-app
    docroot /var/www/html
    index index.php
    path-info ^(/.+\.php)(/.*)?$
    set-param PHP_ADMIN_VALUE "session.upload_progress.enabled=0"

backend php_fpm_nodes
    mode http
    use-fcgi-app php-fpm-app
    server fpm1 127.0.0.1:9000 proto fcgi check

Validate and reload HAProxy:

haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy

Step 3: WAF & Edge Rules (ModSecurity & OpenResty)

If your application does not use multipart/form-data file uploads at all, the PHP Security Team recommends blocking multipart/form-data requests at the frontend entirely. Otherwise, enforce strict multipart part/header bounds and block PHP_SESSION_UPLOAD_PROGRESS:

ModSecurity (Apache, NGINX, OWASP CRS)

# Block PHP_SESSION_UPLOAD_PROGRESS in parsed POST fields or raw multipart streams
SecRule REQUEST_METHOD "@streq POST" \
    "id:1000101,\
    phase:2,\
    block,\
    status:403,\
    log,\
    msg:'CVE-2026-109619: Blocked PHP_SESSION_UPLOAD_PROGRESS field',\
    chain"
    SecRule ARGS_POST_NAMES "@contains PHP_SESSION_UPLOAD_PROGRESS" ""

SecRule REQUEST_BODY "@rx name=[\"']?PHP_SESSION_UPLOAD_PROGRESS[\"']?" \
    "id:1000102,\
    phase:2,\
    block,\
    status:403,\
    log,\
    msg:'CVE-2026-109619: Blocked raw multipart PHP_SESSION_UPLOAD_PROGRESS'"

# Require Content-Length on multipart/form-data requests (CVE-2026-109618)
SecRule REQUEST_HEADERS:Content-Type "@rx (?i)multipart/form-data" \
    "id:1000103,\
    phase:1,\
    block,\
    status:411,\
    log,\
    msg:'CVE-2026-109618: Multipart request without Content-Length',\
    chain"
    SecRule &REQUEST_HEADERS:Content-Length "@eq 0" ""

OpenResty / NGINX with `lua-nginx-module`

location / {
    access_by_lua_block {
        local headers = ngx.req.get_headers()
        local content_type = headers["content-type"]
        if content_type and string.find(content_type:lower(), "multipart/form-data", 1, true) then
            -- CVE-2026-109618: Reject multipart requests without a declared Content-Length
            if not headers["content-length"] then
                ngx.log(ngx.ERR, "Blocked chunked/unbounded multipart request (CVE-2026-109618)")
                return ngx.exit(ngx.HTTP_LENGTH_REQUIRED)
            end

            ngx.req.read_body()
            local body = ngx.req.get_body_data()
            if not body then
                local body_file = ngx.req.get_body_file()
                if body_file then
                    local f = io.open(body_file, "rb")
                    if f then
                        body = f:read(65536)
                        f:close()
                    end
                end
            end
            -- CVE-2026-109619: Reject PHP_SESSION_UPLOAD_PROGRESS
            if body and string.find(body, "PHP_SESSION_UPLOAD_PROGRESS", 1, true) then
                ngx.log(ngx.ERR, "Blocked PHP_SESSION_UPLOAD_PROGRESS attempt (CVE-2026-109619)")
                return ngx.exit(ngx.HTTP_FORBIDDEN)
            end
        end
    }

    # Standard proxy_pass or fastcgi_pass configuration below
}

Auditing Your Fleet & Logs

The PHP Security Team recommends running a targeted configuration and log audit across all PHP hosts:

1. Check Active Runtime Versions & Vulnerable INI Directives

for bin in $(which php php-fpm php-fpm8.1 php-fpm8.2 php-fpm8.3 php-fpm8.4 php-fpm8.5 lsphp 2>/dev/null | sort -u); do
    echo "=== Auditing: $bin ($("$bin" -v 2>/dev/null | head -n1)) ==="
    "$bin" -i 2>/dev/null | grep -E "^(memory_limit|post_max_size|upload_max_filesize|max_file_uploads|max_input_nesting_level|fatal_error_backtraces|session\.upload_progress\.(enabled|name))"
done

2. Audit Startup & Per-Request INI Overrides for Early Diagnostic Consumers

Search your Apache, NGINX, .htaccess, and FPM configs for deprecated or invalid INI settings that emit warnings before php_request_startup() (which act as consumers for CVE-2026-109619):

grep -rnE "php_(admin_)?(value|flag)|PHP_(ADMIN_)?VALUE|auto_detect_line_endings|filter\.default|mbstring\.(http_|internal_)|iconv\." \
  /etc/nginx /etc/apache2 /etc/httpd /etc/php* /var/www 2>/dev/null

3. Inspect Logs for Producer Triggers

Enable log_errors=On and error_reporting=E_ALL in your startup php.ini and check your PHP-FPM and Apache error logs for evidence of producer interruptions or early diagnostics:

Allowed memory size of ... bytes exhausted
Maximum execution time of ... seconds exceeded
Warnings or deprecations logged without a script filename or at Unknown on line 0 (indicating pre-initialization or post-deactivation diagnostics)

Official Patched Versions & Upgrade Matrix

As soon as your change window or build pipeline allows, upgrade to the patched releases published on 21 October 2026:

PHP Branch	Affected Versions	Patched Release	Status / Notes
PHP 8.5	`< 8.5.12`	8.5.12	Active support — Download
PHP 8.4	`< 8.4.27`	8.4.27	Active support — Download
PHP 8.3	`< 8.3.36`	8.3.36	Security support — Download
PHP 8.2	`< 8.2.35`	8.2.35	Security support (EOL on 31 Dec 2026)
PHP 8.1 (EOL)	All `8.1.x`	`PHP-8.1-CVE-2026-109618-109619`	Official minimal source backport branch or LTS distro vendor
PHP 8.0 (EOL)	All `8.0.x`	`PHP-8.0-CVE-2026-109618-109619`	Official minimal source backport branch or LTS distro vendor
PHP 7.4 (EOL)	All `7.4.x`	`PHP-7.4-CVE-2026-109618-109619`	Official minimal source backport branch or LTS distro vendor

Bridging a Cloud Condo Intercom to Home Assistant with Rust, SIP, and MQTT

2026-10-17T00:00:00Z

In a well-tuned smart home, almost every physical interaction is programmable. Lighting scenes adapt to the time of day, climate control reacts to occupancy, and alerts route cleanly through a single dashboard in Home Assistant.

Yet in many modern condominiums in Singapore, one piece of hardware stubbornly resists integration: the building intercom.

Instead of exposing a standard local LAN doorphone inside the apartment, many residential developments route their lobby call panels through cloud-managed property management apps—such as iPlus Living backed by a Fermax MeetMe SIP infrastructure. When a visitor or delivery driver dials your unit from the lobby or side gate, your phone receives a push notification from a proprietary app, you wait for the app to cold-start and negotiate a video stream, and you tap an icon to buzz the gate open.

From a home automation perspective, this is a dead end:

Siloed notifications: You cannot route doorbell events or camera snapshots into Home Assistant notifications, wall tablets, or TVs.
No programmable access: You cannot trigger an unlock from a Home Assistant dashboard, an NFC tag, or a temporary "delivery window" automation.
Single-platform lock-in: Everything depends on a mobile app staying logged in and surviving aggressive mobile OS background battery killers.

I wanted the condo intercom to behave like any other native device in my cluster. So I built iplusliving2mqtt—a self-contained Rust daemon running in Kubernetes that bridges the entire intercom lifecycle into Home Assistant via MQTT Auto-Discovery.

Why Standard SIP Clients Don't Work

At first glance, a SIP-based intercom sounds trivial: grab the SIP credentials, point baresip, Asterisk, or Home Assistant's VoIP integration at the server, and wait for an incoming INVITE.

In practice, inspecting the protocol flow revealed three layers that prevent a standard SIP client from ever receiving a call:

1. Split-Key Encrypted Provisioning API

Rather than giving residents static SIP credentials, the iPlus Living mobile app dynamically provisions them after login. Every HTTP request and response body is encrypted with AES-256-CBC using a fixed initialization vector, while the 32-byte AES key is hidden inside a 42-byte random alphanumeric prefix prepended to the base64 ciphertext (payload[:16] + payload[22:38]).

Authenticating through the four-step login sequence (user/login → user/logindetail → intercom/meetmelicencedetails → intercom/intercomconfiguration) returns:

Cloud MeetMe SIP credentials (sip.fermax.com)
Building PBX SIP credentials (the development's Grandstream server)
The call panel directory (mapping numeric SIP caller IDs like 14 or 22 to human-readable gate and lobby names)

2. The Push Notification Catch-22 (RFC 8599)

Even with valid SIP credentials and an active TLS connection to Fermax's Flexisip proxy (_sips._tcp.sip.fermax.com), a standard SIP client will sit idle forever when someone presses the doorbell.

Why? Because the Flexisip server is configured for mobile push notifications under RFC 8599 with pn-timeout=0:

Contact: <sip:USER@IP:PORT;pn-prid=FCM_TOKEN;pn-provider=fcm;pn-param=SENDER_ID;pn-silent=1;pn-timeout=0;transport=tls>

With pn-timeout=0, the SIP proxy assumes mobile clients go to sleep immediately after registering. When the lobby panel places a call:

Flexisip pauses the incoming INVITE transaction.
It dispatches a high-priority data push (title: "IC_CALL") through Google Firebase Cloud Messaging (FCM) to the pn-prid token registered in the Contact header.
It waits for the client to wake up and send a fresh SIP REGISTER over TLS.
Only after receiving that REGISTER does Flexisip release the held INVITE to the client.

Without a real FCM token and a persistent listener on Google's Mobile Connection Server (mtalk.google.com:5228), the SIP server never forwards the call.

3. Early-Media H.264 Video & Dual-Mode DTMF

Once the INVITE arrives, the lobby panel streams H.264 video during early media (183 Session Progress) before the call is answered. However, the door relay controller ignores unlock commands during early media—to actually buzz the gate open, the client must transition the call to answered (200 OK with SDP), wait for the ACK, and transmit DTMF unlock tones before hanging up (BYE).

System Architecture

To make this rock-solid in Kubernetes without relying on external headless Android emulators, ffmpeg subprocesses, or Python scripts, I built iplusliving2mqtt as a single static Rust binary:

Zero-Touch Provisioning & FCM Emulation

On startup, if no cached state exists on its persistent volume, iplusliving2mqtt performs a two-stage bootstrap:

FCM Registration: Emulates an Android check-in against android.clients.google.com/checkin and registers with Google Cloud Messaging (/c2dm/register3) using the official app's Firebase project parameters to obtain a valid fcm_token.
iPlus Auto-Provisioning: Logs into the iPlus Living API using the resident's email and password, binds the fcm_token, and caches the resulting SIP accounts and call panel directory to state.json.
MCS Stream: Opens a persistent TLS connection to mtalk.google.com:5228 speaking Google's binary MCS framing + Protobuf protocol. The moment an IC_CALL data message arrives, it signals the SIP worker tasks over an internal tokio::sync::mpsc channel to fire an immediate SIP REGISTER.

In-Memory H.264 Keyframe Extraction

Lobby panels stream H.264 video encapsulated in RTP packets (RFC 6184), including fragmented NAL units (FU-A, type 28) and aggregation packets (STAP-A, type 24).

Rather than spawning ffmpeg on every doorbell ring, iplusliving2mqtt handles the media stream directly in memory:

Sends UDP NAT-punching packets plus an RTCP Picture Loss Indication (PLI) and a SIP INFO (picture_fast_update) request immediately upon answering with 183 Session Progress so the panel transmits an IDR keyframe right away.
Reassembles fragmented H.264 NAL units (SPS, PPS, and IDR slices) and feeds the Annex-B bitstream into Cisco's openh264 decoder in memory.
Converts the decoded YUV420p frame to RGB and encodes a JPEG snapshot directly onto the MQTT camera topic (fermax/intercom/snapshot/image) and an HTTP /snapshot.jpg endpoint.

Dual-Mode DTMF Unlock

Different intercom relays along the call path can be picky about how DTMF digits are delivered. When you press the Unlock button in Home Assistant (or when auto-unlock is enabled), the daemon:

Answers the pending INVITE with 200 OK (if still in 183 Session Progress).
Transmits RFC 2833 (telephone-event/8000) RTP payload packets with the end-of-event marker bit and triple redundancy.
Follows up with synthesized in-band G.711 µ-law (PCMU) dual-tone sine waves (697–941 Hz + 1209–1477 Hz) at 8,000 Hz so both digital SIP gateways and analog door strike controllers reliably trigger.
Sends a clean SIP BYE to release the lobby panel.

A Fun Dual-Stack Debugging Quirk: IPv6 vs. IPv4 in SIP

When testing the daemon on a dual-stack host with native IPv6 connectivity, the initial SIP TLS connection to sip5.fermax.com:5223 completed its TLS handshake and then immediately dropped with an EOF.

Digging into the packet trace uncovered two separate issues:

RFC 3261 IPv6 Formatting: sip5.fermax.com publishes both A and AAAA records. Because the host preferred IPv6, tcp.local_addr() returned an IPv6 address. Formatting Via and Contact headers as ip:port without enclosing the IPv6 literal in square brackets ([2a09:...]:5060) caused Flexisip's parser to reject the REGISTER packet immediately.
IPv4-Only Media Relays: Even with bracketed IPv6 headers, the building's local PBX and the RTP audio/video media relays (c=IN IP4 ... in SDP) are strictly IPv4. Because the SIP engine learns its external NAT address from the SIP server's Via: ...;received=<ip>;rport=<port> response header to advertise in the SDP answer, registering over IPv6 leaves the media engine with an IPv6 address that an IPv4 RTP relay can never reach.

Explicitly resolving and selecting IPv4 socket addresses for the SIP/RTP path immediately brought both the MeetMe and Grandstream registrations to 200 OK.

Home Assistant Integration

Because iplusliving2mqtt publishes Home Assistant MQTT Auto-Discovery payloads under homeassistant/#, zero manual YAML configuration is required in Home Assistant. All entities automatically group under a single Fermax MeetMe Intercom device:

Entity	Type	Description
Doorbell Ringing	`binary_sensor`	Turns `ON` the instant an incoming `INVITE` arrives from any gate or lobby panel.
Call State	`sensor`	Tracks the call lifecycle: `idle` → `ringing` → `answered` → `unlocked`.
Last Caller	`sensor`	Displays the resolved lobby or gate name (e.g., Side Gate or Block Lobby) plus JSON attributes.
Intercom Camera	`camera`	Live JPEG snapshot decoded from the panel's H.264 RTP video stream.
Unlock Door	`button`	Immediately answers the active call and sends the DTMF unlock sequence.
Door Lock	`lock`	Reflects `LOCKED` / `UNLOCKED` state and supports standard Home Assistant lock actions.
Auto-Unlock on Ring	`switch`	Optional toggle that automatically answers and buzzes the door open after a configurable delay.

With everything in Home Assistant, a doorbell ring now triggers an instant push notification to our phones with the live decoded camera snapshot attached and an inline "Unlock Gate" action button—plus a temporary "expecting a delivery" toggle that auto-unlocks the lobby gate for the next ring and then turns itself off.

Kubernetes Deployment

True to the rest of my home infrastructure, iplusliving2mqtt is packaged as a minimal multi-stage container (gcr.io/distroless/cc-debian13:nonroot, running as UID 65532 with a read-only root filesystem and all Linux capabilities dropped) and deployed via a Helm chart with /healthz and /readyz probes that verify both MQTT broker connectivity and active SIP registration state.

Fixing Entra ID / Azure AD AADSTS75011 on Ceph Dashboard SAML

2026-10-17T00:00:00Z

While configuring SAML 2.0 Single Sign-On (SSO) for the Ceph Dashboard against Microsoft Entra ID (formerly Azure AD), standard password authentication works out of the box. However, the moment you authenticate with modern, phishing-resistant credentials—such as FIDO2 security keys, Windows Hello, or passwordless MFA—Entra ID rejects the login with AADSTS75011:

AADSTS75011: Authentication method 'MultiFactor, Fido' by which the user 
authenticated with the service doesn't match requested authentication method 
'Password, ProtectedTransport'. Contact the Ceph Dashboard application owner.

At first glance, the error seems backwards: authenticating with a FIDO2 hardware key and MFA is strictly more secure than a password, yet the Identity Provider blocks the session.

Here is why this happens in Ceph and how to fix it cleanly in the Ceph monitor configuration store without patching Python files inside the ceph-mgr container.

Why It Happens: `python-saml` and `RequestedAuthnContext`

Under the hood, the Ceph Dashboard relies on OneLogin's python-saml library to act as a SAML Service Provider.

By default, python-saml initializes its security configuration with:

'requestedAuthnContext': True

Unless overridden, every outbound AuthnRequest generated by the Ceph Dashboard includes an explicit <samlp:RequestedAuthnContext> block demanding PasswordProtectedTransport with Comparison="exact":

<samlp:RequestedAuthnContext Comparison="exact">
  <saml:AuthnContextClassRef>
    urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
  </saml:AuthnContextClassRef>
</samlp:RequestedAuthnContext>

Because Comparison="exact" is enforced, Entra ID checks the exact authentication method used to satisfy the session. When a user signs in with a passkey/FIDO2 token or MFA, Entra ID records the session context as MultiFactor, Fido. Since that does not literally match Password, ProtectedTransport, Entra ID aborts the SAML flow with AADSTS75011.

To allow FIDO2, passkeys, and MFA, Ceph needs to omit <samlp:RequestedAuthnContext> from its AuthnRequest and let Entra ID's Conditional Access policies govern acceptable authentication methods.

The Fix: Updating `ssodb_v1` in the Monitor Store

The ceph dashboard sso setup saml2 CLI command does not expose a flag to toggle requestedAuthnContext, and inspecting ceph dashboard sso show saml2 will show it enabled by default:

$ ceph dashboard sso show saml2 | jq '.onelogin_settings.security.requestedAuthnContext'
true

Instead of modifying sso.py on the container filesystem—which would be wiped on the next container restart or Ceph upgrade—we can patch the serialized SSO database (mgr/dashboard/ssodb_v1) directly in the Ceph monitor config-key store.

1. Export the Current SSO Configuration

Dump the raw JSON configuration from the monitor store:

ceph config-key get mgr/dashboard/ssodb_v1 > ssodb.json

2. Disable `requestedAuthnContext` Under `.saml2`

Ceph stores SAML 2.0 settings under .saml2.onelogin_settings. Use jq to set both requestedAuthnContext and failOnAuthnContextMismatch to false inside .saml2.onelogin_settings.security:

jq '.saml2.onelogin_settings.security += {
  "requestedAuthnContext": false,
  "failOnAuthnContextMismatch": false
}' ssodb.json > ssodb_updated.json

Verify the updated security block before applying it:

$ jq '.saml2.onelogin_settings.security' ssodb_updated.json
{
  "nameIdEncrypted": false,
  "authnRequestsSigned": false,
  "logoutRequestSigned": false,
  "logoutResponseSigned": false,
  "signMetadata": false,
  "wantMessagesSigned": false,
  "wantAssertionsSigned": false,
  "wantAssertionsEncrypted": false,
  "wantNameIdEncrypted": false,
  "metadataValidUntil": "",
  "wantAttributeStatement": false,
  "requestedAuthnContext": false,
  "failOnAuthnContextMismatch": false
}

3. Write the Configuration Back and Reload the Dashboard

Write the updated JSON back to mgr/dashboard/ssodb_v1:

ceph config-key set mgr/dashboard/ssodb_v1 -i ssodb_updated.json

Reload the dashboard manager module so ceph-mgr picks up the new settings from the monitor store:

ceph mgr module disable dashboard
ceph mgr module enable dashboard

Confirm that the active SAML configuration now reflects false:

$ ceph dashboard sso show saml2 | jq '.onelogin_settings.security.requestedAuthnContext'
false

Persistence and Operational Notes

With requestedAuthnContext disabled, SP-initiated logins to the Ceph Dashboard omit the <samlp:RequestedAuthnContext> constraint, allowing Entra ID to complete authentication using FIDO2 security keys, Windows Hello, or MFA without triggering AADSTS75011.

Because the configuration lives in the Ceph monitor database (config-key), it persists across ceph-mgr failovers, container restarts, and node reboots. The only caveat to keep in mind is that running ceph dashboard sso setup saml2 in the future (for instance, when rotating IdP signing certificates) will regenerate the .saml2 dictionary and reset requestedAuthnContext to true—so keep this jq patch in your cluster runbook or configuration management whenever SAML metadata is re-imported.

Why I Built BCRSTracking: Telemetry for Singapore's Bottle Return Scheme

2026-09-12T00:00:00Z

Few things in modern municipal living are as quietly frustrating as carrying two giant tote bags of rinsed plastic bottles and aluminum cans through Singapore's midday heat to your neighborhood supermarket, only to be greeted by a glowing red screen:

Machine Status: Bin Full. Please try again later.

Singapore’s Beverage Container Return Scheme (BCRS) is a fantastic sustainability initiative. Across the island, Reverse Vending Machines (RVMs) accept beverage containers and return deposits. But like any physical infrastructure distributed across hundreds of residential hubs, MRT stations, and supermarkets, real-world machines fill up, experience mechanical jams, or undergo routine maintenance.

As a Site Reliability Engineer, I don't like guessing whether a distributed system is healthy before interacting with it. If a network node has a status, it should have telemetry.

So I built BCRSTracking.

The Upstream API

When inspecting how the official tracking system works (bts.bcrs.sg), it turns out the platform exposes public API endpoints under /forapi/v2/. These endpoints provide:

Location Listings (/locations): A directory of all deployed RVM stations across the island, including operating hours, physical coordinates, and machine identifiers.
Bin Status & Telemetry (/bin-status): Real-time machine operational status, current bin capacity percentage, and hardware availability.

While the data was publicly accessible, there was no historical trend tracking, no differential event stream, and no way to query capacity curves over time to see when specific machines are emptied or filled.

Engineering Requirements

When architecting the tracker, I wanted it to be lightweight, resilient, and polite:

Polite Upstream Concurrency: Never overwhelm public services. Polling must use strict token-bucket rate limiting (golang.org/x/time/rate), bounded worker pools, and exponential backoff with jitter on HTTP 429 or 5xx responses.
Differential State Engine: Instead of blindly dumping snapshots, compute state transitions in memory (MACHINE_ADDED, STATUS_CHANGED, CAPACITY_THRESHOLD_CROSSED) so downstream systems can alert on state changes.
Fast Columnar Analytics: Store telemetry snapshots in ClickHouse using compressed columnar engines (ReplacingMergeTree for current state, MergeTree with TTL for historical bin levels).
Native Observability: Expose Prometheus metrics for scrapers, queue depth, upstream latency, and error rates, scraped by VictoriaMetrics.

System Architecture

The resulting service is written in Go 1.27 and runs as a lightweight container in Kubernetes:

What the Telemetry Reveals

Having continuous time-series telemetry on reverse vending machines across Singapore reveals fascinating patterns:

The Weekend Surge: Neighborhood machines near residential housing estates typically reach 80–100% capacity on Saturday and Sunday between 2:00 PM and 6:00 PM.
Clearance Cadence: Logistics crews typically clear bins during early morning runs between 7:00 AM and 9:30 AM on weekdays.
Pre-Trip Confirmation: Instead of walking down hoping for the best, I can query my nearest machine via Prometheus/Grafana or a quick API curl to confirm it is operational and has available bin capacity.

Open Source

The entire tracker—including the Go poller, ClickHouse migrations, and Helm charts—is open-source on GitHub:

Repository: github.com/TheDJVG/BCRSTracking

If you're in Singapore and interested in tracking local smart-city infrastructure, feel free to check out the repository, submit PRs, or send me feedback at me@dj.vg.

The 'Datacenter at Home': A Multi-Homed BGP & WireGuard Mesh Architecture

2026-01-31T00:00:00Z

For the past five years, my home network hasn't just been a gateway to the internet; it's been a miniature ISP backbone. While most residential setups rely on a single ISP router doing basic NAT, I wanted something more robust: carrier-grade reliability, deterministic traffic engineering, and seamless multi-homing using my own public Autonomous System Number (AS209792).

By applying hyperscale datacenter design principles—like eBGP-on-the-host and Infrastructure as Code—I’ve built a network that treats residential ISPs as mere transport utilities.

The High-Level Architecture

The core of the network is built on redundancy. I am connected to two primary fiber ISPs in Singapore, MyRepublic and ViewQwest, along with a 5G backup link. Instead of relying on these ISPs for routing, I use them as "underlay" transport to reach my "Upstream Edge Cloud"—a fleet of six VPS nodes (edge1.sin through edge6.sin) hosted across providers.

The edge nodes are the true gateways to the internet. They announce my public prefixes to the global BGP table, while my home Core Router manages the paths between them.

The Connectivity Fabric: WireGuard and Deterministic Addressing

To link my Core Router to the cloud, I use a mesh of WireGuard tunnels. Rather than managing a messy internal IPAM (IP Address Management) database, I use a deterministic addressing logic.

Every tunnel interface is assigned an IPv6 address and a specific UDP listen port generated procedurally based on the IDs of the two nodes. This ensures that the same two peers always get the same IP pair and port, making firewalling and BGP peering definitions completely predictable across the entire mesh. To simplify things further, I utilize RFC 5549, which allows me to carry IPv4 routing information over an IPv6 next-hop, eliminating the need for private IPv4 point-to-point links.

Design Choice: eBGP Everywhere

A common question in BGP design is whether to use internal BGP (iBGP) or external BGP (eBGP). While iBGP is standard for single-ASN networks, it introduces complexity with split-horizon rules and requires Route Reflectors.

I chose the "eBGP in the Datacenter" approach. Every node in my network—the Core Router, the VPS edges, the pfSense firewall, and the Kubernetes nodes—operates under its own unique private ASN (645xx range).

Why this works:

Loop Prevention: BGP’s AS_PATH attribute naturally prevents loops. If a node sees its own ASN in a path, it drops the route.
Observability: I can look at my routing table and instantly see the path a packet took by looking at the sequence of ASNs.
A Clean Public Face: To the internet, I am a single entity. My edge nodes use BIRD (Internet Routing Daemon) with a strict export filter that "scrubs" these internal private ASNs, ensuring the world only sees a clean, single-hop path originating from AS209792.

The Brain: SaltStack & The Salt Mine

Orchestrating this manually would be impossible. I use SaltStack as the network’s "Control Plane." The secret sauce is the Salt Mine, a secure metadata store on the Salt Master.

Metadata Exchange: When a node joins, it generates its own WireGuard keys locally and pushes them, along with its current ISP IP, into the Salt Mine.
Automated Handshakes: All other nodes query the Mine and automatically "wire" themselves to the new peer.
Full System State: Salt doesn't just configure BGP. It manages the entire stack:
- systemd-networkd: Generates .netdev and .network files for the tunnels.
- Firewalling: Automatically opens the procedurally generated UDP ports only for authorized peers.
- BIRD: Templates the BGP sessions and filters.
- Kubernetes: Peering pfSense with the K8s nodes so that Pod/Service IPs are reachable across the entire network without NAT.

Reliability and Convergence

With residential ISPs, "silent failures" (where the link is up but traffic is blackholed) are a reality. To counter this, every BGP session runs BFD (Bidirectional Forwarding Detection). If a link drops even a few packets, BFD detects the failure in milliseconds and BGP reroutes traffic to a healthy ISP.

As a final failover, the 5G link is configured with heavy AS-path prepending. It stays in standby, receiving no traffic, until every other fiber path is exhausted.

Conclusion

This setup has been rock-solid for over five years. By moving the "intelligence" into SaltStack and using BGP for what it does best—path selection—I've created a home network that behaves like a professional datacenter fabric. It’s stateless, modular, and can scale to a new cloud provider with a single state.apply.

Easy and free Dynamic DNS

2025-10-04T00:00:00Z

The nasty need for dynamic DNS.

For a while now I've been maintaining a couple of 4G/5G enabled out of band devices. These devices are used to provide access to serial consoles of network devices. Why you ask? Well, sometimes a configuration error or something else causes the normal access paths to be unavailable. Who hasn't made a L2 loop by accident...

I call these devices "last resort". They provide a last resort access to my network when everything else fails, and they use 4G/5G for connectivity. The downside? They don't have a fixed IP address, quite often the IPs change. For most cases this is fine because I have reverse tunnels, but I still want to access them in case those fail. But for that to work I need to know 1 crucial thing: its IP address on the internet.

I'm sure many people have used a service like "dyn.com" to solve this problem, the device in question sends a HTTP request (we wish those devices support HTTPS...) and a DNS record gets updated with the IP of the requesting device. That way every time the IP changes the DNS record gets updated and we can access it using the DNS name and it makes it easy to monitor its availibility. The main problem I had with this solution is the cost (also the annoying ratelimiting, but I partially blame the devices in question), it was not worth the money to use "dyn.com" for something I can make myself, right?

So I made a service myself.

When I started to think about this had the following requirements: - Dead simple to request a new dynamic hostname - No need for accounts - Compatibility with DynDNS style updates (/nic/update) (more to add?)

When I had the requirements clear I also wanted to make it stateless, who wants to keep track of what hostnames are in use and not? If you lose the credentials, just get a fresh hostname. I quickly wrote an implementation using FastAPI and looked in my unused domain invenstory for a good candidate.

I present to you: XYZ.FRL

Now the initial version of this service is deployed I really like it myself, mainly because how simple it is. You want to know how simple? Open your terminal and come along.

Request a new dynamic hostname

› curl -s https://xyz.frl/generate | jq .
{
  "hostname": "4bd36fd6538b45b3b5449e615afc8e21.xyz.frl",
  "username": "4bd36fd6538b45b3b5449e615afc8e21",
  "password": "+MihUvQ8pc1QwYA6/16A/D/155qHfJKlUqUhNgD6osE="
}

Update the hostname with a new IP

Simple use curl to send an update request and set the ip to the valuse set in the myip query param (you can also leave it out to use the IP the request is coming from).

› curl -s --user '4bd36fd6538b45b3b5449e615afc8e21:+MihUvQ8pc1QwYA6/16A/D/155qHfJKlUqUhNgD6osE=' 'http://xyz.frl/nic/update?myip=192.0.2.0,100::'

Check your new IP!

It might take a brief moment for changes to propagate

› host 4bd36fd6538b45b3b5449e615afc8e21.xyz.frl
4bd36fd6538b45b3b5449e615afc8e21.xyz.frl has address 192.0.2.0
4bd36fd6538b45b3b5449e615afc8e21.xyz.frl has IPv6 address 100::

Profit.

From now simply use your compatible client (or curl) with our without the myip query param to update the IP address in the record. Make your own life easy by simply CNAMEing to this hostname and you should be good to go.

Final thoughts

This was a small fun project that I threw together on a Saturday morning when I got a renewal notice (and now cancellation) for DynDNS. I hope it can benefit others too.

Do you have any ideas to make this service even better? Contact me on me@dj.vg.

Security

While I've applied pretty aggressive rate limiting (1 request per minute on the generate and update endpoints) it's of course possible people might try to abuse this, if that happens I'll deal with it.

Timelapse system: a start

2022-01-25T00:00:00Z

Everyone that’s interested in technology will at some point face the same struggle: how do I apply all these interesting technologies in my own projects.

This is really the start of a system that (at the moment of writing) stores over 750 thousand photos occupying 6.2 terabyte of storage and every single minute this grows.

In the following comparison you can see a picture taken just after I started this project and one that has been taken on the day of publishing this piece.

Just after starting the project

Day of publishing

Times are changing

I’m living in an area that’s actively being developed and for some reason I can’t let go of the past. I want to keep track of how the area is changing and feel nostalgic on “how thing once were”. This is why is decided to put a camera on my balcony, with a Raspberry Pi and a Cronjob to take a picture every single minute using gPhoto2. The pictures would be saved on a simple NFS share backed by 2 times 1TB hard disks in raid 1. That was it. Anyone working in IT will immediately see a couple of problems with this approach, just to name a couple:

Reliability
Monitoring
Observability

Reliability

As I was not actively looking at the pictures it would happen multiple times that the Raspberry Pi was hanging or that for some reason the camera was not responding ending up with thousands of hanging Cronjobs. Also unreliable Wifi from outdoor made NFS very sad. That had to change, it should just work, even if one of the components is not working.

Monitoring

As mentioned in the previous paragraphs I was not aware of these problems. It was just (not) running and that is not great for a project that tries to capture a changing landscape one minute at a time. The moment there’s a problem with the setup I want to know about and preferable it should recover automatically (by for example resetting the power of the camera).

Observability

Taking a photo is one thing, but there’s so much extra information that I was not aware of. Some of the information is obvious, for example the ISO and exposure, which is stored in the Exif portion of the file. Other information that I wanted to store was: Outside temperature and relative humidity, how much time is spend on the camera, how reliable is this process, how long does it take to write the file to storage, and how much energy does all of this consume!

How to get there

In the blogs after this I will write down exactly how I got to a system that I almost never have to touch. How videos are automatically created daily. How the camera and controller is able to recover from almost any failure, how data is stored and replicated to the other side to the world.

So if you’re interested in Kubernetes, Ceph (and its S3 compatible API radosgw), Django, VictoriaMetrics (and metrics in general), Home Assistant, and many more (don’t forget the hardware). Come back soon to see how all of these things are combined to create a system that allows me to take reliable pictures that I plan to store forever!

Daan van Gorkum

Bridging Singapore Power Smart Meters to Home Assistant with Rust and MQTT

Understanding the Client Protocol: Three Design Challenges

1. Standard Auth0 Android SDK Telemetry & Request Metadata

2. Handling SMS OTP Multi-Factor Authentication in Kubernetes

3. Reconciling Monthly Meter Reads with Half-Hourly AMI Intervals

System Architecture

Event-Driven Concurrency & State Coordination

Polite Polling & Proactive Token Refresh

Home Assistant Entities

1. Account & Authentication Device (SP Services Account)

2. Per-Premise Utility Device (SP Premise <premise_id>)

Kubernetes Runtime Architecture

CVE-2026-109618 & CVE-2026-109619: Mitigating PHP's Critical CVSS 9.2 RCE Vulnerabilities When You Cannot Immediately Upgrade

Why You Must Act Right Now (Even If You Cannot Upgrade Today)

Executive Summary & Official Facts

Technical Breakdown of the Two Vulnerabilities

1. CVE-2026-109618: RCE via Out-of-Memory (OOM) Corruption (GHSA-w8hw-w9fx-j79w)

How Attackers Trigger Controllable OOM During Request Startup

SAPI Risk Profile for CVE-2026-109618

2. CVE-2026-109619: RCE via Stale Global State & Session Upload Progress UAF (GHSA-6wmf-4rv5-p3rp)

Confirmed Exploitation Paths for CVE-2026-109619

Immediate Mitigations If You Cannot Upgrade PHP Today

Step 1: Lock Down PHP Configuration (php.ini / FPM Pool)

Step 2: Web Server & FastCGI Hardening (NGINX, Apache, HAProxy)

Option A: NGINX + PHP-FPM

Option B: Apache (mod_php and mod_proxy_fcgi)

Option C: HAProxy (Edge Body Filtering, Buffering & FastCGI)

Step 3: WAF & Edge Rules (ModSecurity & OpenResty)

ModSecurity (Apache, NGINX, OWASP CRS)

OpenResty / NGINX with lua-nginx-module

Auditing Your Fleet & Logs

1. Check Active Runtime Versions & Vulnerable INI Directives

2. Audit Startup & Per-Request INI Overrides for Early Diagnostic Consumers

3. Inspect Logs for Producer Triggers

Official Patched Versions & Upgrade Matrix

Bridging a Cloud Condo Intercom to Home Assistant with Rust, SIP, and MQTT

Why Standard SIP Clients Don't Work

1. Split-Key Encrypted Provisioning API

2. The Push Notification Catch-22 (RFC 8599)

3. Early-Media H.264 Video & Dual-Mode DTMF

System Architecture

Zero-Touch Provisioning & FCM Emulation

In-Memory H.264 Keyframe Extraction

Dual-Mode DTMF Unlock

A Fun Dual-Stack Debugging Quirk: IPv6 vs. IPv4 in SIP

Home Assistant Integration

Kubernetes Deployment

Fixing Entra ID / Azure AD AADSTS75011 on Ceph Dashboard SAML

Why It Happens: python-saml and RequestedAuthnContext

The Fix: Updating ssodb_v1 in the Monitor Store

1. Export the Current SSO Configuration

2. Disable requestedAuthnContext Under .saml2

3. Write the Configuration Back and Reload the Dashboard

Persistence and Operational Notes

Why I Built BCRSTracking: Telemetry for Singapore's Bottle Return Scheme

The Upstream API

Engineering Requirements

System Architecture

What the Telemetry Reveals

Open Source

The 'Datacenter at Home': A Multi-Homed BGP & WireGuard Mesh Architecture

The High-Level Architecture

The Connectivity Fabric: WireGuard and Deterministic Addressing

Design Choice: eBGP Everywhere

Why this works:

The Brain: SaltStack & The Salt Mine

Reliability and Convergence

Conclusion

Easy and free Dynamic DNS

The nasty need for dynamic DNS.

So I made a service myself.

I present to you: XYZ.FRL

Request a new dynamic hostname

Update the hostname with a new IP

Check your new IP!

Profit.

Final thoughts

Security

Timelapse system: a start

1. Account & Authentication Device (`SP Services Account`)

2. Per-Premise Utility Device (`SP Premise <premise_id>`)

1. CVE-2026-109618: RCE via Out-of-Memory (OOM) Corruption (`GHSA-w8hw-w9fx-j79w`)

2. CVE-2026-109619: RCE via Stale Global State & Session Upload Progress UAF (`GHSA-6wmf-4rv5-p3rp`)

Step 1: Lock Down PHP Configuration (`php.ini` / FPM Pool)

Option B: Apache (`mod_php` and `mod_proxy_fcgi`)

OpenResty / NGINX with `lua-nginx-module`

Why It Happens: `python-saml` and `RequestedAuthnContext`

The Fix: Updating `ssodb_v1` in the Monitor Store

2. Disable `requestedAuthnContext` Under `.saml2`