<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Daan van Gorkum</title>
  <subtitle>Senior Site Reliability Engineer specializing in Linux, Infrastructure as Code, Python, and Go.</subtitle>
  <link href="https://dj.vg/feed.xml" rel="self" type="application/atom+xml"/>
  <link href="https://dj.vg/" rel="alternate" type="text/html"/>
  <id>https://dj.vg/</id>
  <updated>2026-10-21T00:00:00Z</updated>
  <author>
    <name>Daan van Gorkum</name>
    <email>me@dj.vg</email>
    <uri>https://dj.vg/</uri>
  </author>
  <entry>
    <title>CVE-2026-109618 &amp; CVE-2026-109619: Mitigating Critical PHP Session Upload Progress Use-After-Free Vulnerabilities Without Upgrading</title>
    <link href="https://dj.vg/posts/cve-2026-109618-cve-2026-109619-php-upload-progress-uaf-mitigation/" rel="alternate" type="text/html"/>
    <link href="https://dj.vg/posts/cve-2026-109618-cve-2026-109619-php-upload-progress-uaf-mitigation/index.md" rel="alternate" type="text/markdown"/>
    <id>https://dj.vg/posts/cve-2026-109618-cve-2026-109619-php-upload-progress-uaf-mitigation/</id>
    <published>2026-10-21T00:00:00Z</published>
    <updated>2026-10-21T00:00:00Z</updated>
    <summary>Urgent request-level and web-server mitigations for CVE-2026-109618 and CVE-2026-109619 in PHP's RFC 1867 session upload progress handler (ext/session/session.c) using NGINX, Apache, HAProxy, ModSecurity, and OpenResty when you cannot immediately upgrade PHP.</summary>
    <content type="html">&lt;p&gt;If you operate PHP workloads in production—whether behind NGINX and &lt;code&gt;php-fpm&lt;/code&gt;, Apache (&lt;code&gt;mod_proxy_fcgi&lt;/code&gt; or &lt;code&gt;mod_php&lt;/code&gt;), HAProxy, or LiteSpeed (&lt;code&gt;lsphp&lt;/code&gt;)—you need to take action &lt;strong&gt;immediately&lt;/strong&gt; to mitigate &lt;strong&gt;CVE-2026-109618&lt;/strong&gt; and &lt;strong&gt;CVE-2026-109619&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Both vulnerabilities are critical &lt;strong&gt;Use-After-Free (UAF)&lt;/strong&gt; flaws in PHP's built-in RFC 1867 session upload progress mechanism (&lt;code&gt;ext/session/session.c&lt;/code&gt;) that allow unauthenticated &lt;strong&gt;Remote Code Execution (RCE)&lt;/strong&gt; over a standard HTTP &lt;code&gt;POST&lt;/code&gt; request.&lt;/p&gt;
&lt;p&gt;Worse still:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;session.upload_progress.enabled&lt;/code&gt; is turned &lt;code&gt;On&lt;/code&gt; (&lt;code&gt;1&lt;/code&gt;) by default&lt;/strong&gt; in upstream PHP and across virtually all major Linux distribution packages (Debian, Ubuntu, RHEL/Rocky, Alpine, and official Docker images).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The vulnerable code executes before your PHP script even starts.&lt;/strong&gt; Because &lt;code&gt;multipart/form-data&lt;/code&gt; parsing happens during the SAPI request startup phase (&lt;code&gt;rfc1867_post_handler&lt;/code&gt;), application-level middleware, authentication checks, and runtime &lt;code&gt;ini_set()&lt;/code&gt; calls &lt;strong&gt;cannot&lt;/strong&gt; protect you.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Any reachable &lt;code&gt;.php&lt;/code&gt; endpoint is a target&lt;/strong&gt;—even if your application never uses &lt;code&gt;$_FILES&lt;/code&gt;, never accepts file uploads, and never calls &lt;code&gt;session_start()&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;In an ideal world, every team would immediately roll out patched PHP binaries across every node. &lt;strong&gt;In reality, not everyone can upgrade PHP right now.&lt;/strong&gt; Legacy dependencies, pinned container base images, distro backport delays, third-party binary extensions (such as ionCube or proprietary APM agents), and change freezes mean millions of production PHP instances cannot be upgraded overnight.&lt;/p&gt;
&lt;p&gt;If you cannot upgrade PHP today, &lt;strong&gt;you do not have to wait for a binary patch to neutralize this threat.&lt;/strong&gt; You can completely sever the attack path right now at the web server, FastCGI, or WAF layer with zero application downtime.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="executive-summary-take-action-now"&gt;Executive Summary: Take Action Now&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style="text-align: left;"&gt;Attribute&lt;/th&gt;
&lt;th style="text-align: left;"&gt;Details&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;CVE Identifiers&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;CVE-2026-109618&lt;/strong&gt;, &lt;strong&gt;CVE-2026-109619&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Vulnerability Class&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Use-After-Free (CWE-416) leading to Pre-Auth Remote Code Execution (RCE)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Affected Component&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;PHP &lt;code&gt;ext/session/session.c&lt;/code&gt; (&lt;code&gt;php_session_rfc1867_callback&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Default Exposure&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Vulnerable by default&lt;/strong&gt; (&lt;code&gt;session.upload_progress.enabled = 1&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Attack Vector&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;code&gt;multipart/form-data&lt;/code&gt; HTTP &lt;code&gt;POST&lt;/code&gt; containing &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; stalled across &lt;code&gt;session.upload_progress.min_freq&lt;/code&gt; (default &lt;code&gt;1s&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Fastest Fix (No Upgrade)&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Force &lt;code&gt;session.upload_progress.enabled=0&lt;/code&gt; via &lt;code&gt;php.ini&lt;/code&gt;, &lt;code&gt;PHP_ADMIN_VALUE&lt;/code&gt; in NGINX/Apache, and block &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; at the edge/WAF&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;hr /&gt;
&lt;h2 id="understanding-cve-2026-109618-and-cve-2026-109619"&gt;Understanding CVE-2026-109618 and CVE-2026-109619&lt;/h2&gt;
&lt;p&gt;To understand why request-level mitigation is mandatory—and why application code cannot block these vulnerabilities—we need to look at how PHP processes &lt;code&gt;multipart/form-data&lt;/code&gt; uploads under the hood.&lt;/p&gt;
&lt;p&gt;When an HTTP client sends a &lt;code&gt;POST&lt;/code&gt; request with &lt;code&gt;Content-Type: multipart/form-data&lt;/code&gt;, PHP's SAPI layer invokes &lt;code&gt;rfc1867_post_handler()&lt;/code&gt; &lt;strong&gt;before&lt;/strong&gt; compiling or executing the target &lt;code&gt;.php&lt;/code&gt; script. If &lt;code&gt;session.upload_progress.enabled&lt;/code&gt; is enabled (the default) and the multipart stream contains a form field named &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; alongside a session identifier (e.g., &lt;code&gt;PHPSESSID&lt;/code&gt;), PHP hooks &lt;code&gt;php_session_rfc1867_callback()&lt;/code&gt; in &lt;code&gt;ext/session/session.c&lt;/code&gt; into the multipart parser events:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;MULTIPART_EVENT_START&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;MULTIPART_EVENT_FORMDATA&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;MULTIPART_EVENT_FILE_START&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;MULTIPART_EVENT_FILE_DATA&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;MULTIPART_EVENT_FILE_END&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;MULTIPART_EVENT_END&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;During the upload, PHP maintains a &lt;code&gt;php_session_rfc1867_progress&lt;/code&gt; structure holding cached &lt;code&gt;zval*&lt;/code&gt; pointers directly into the session's progress &lt;code&gt;HashTable&lt;/code&gt; so it can rapidly update byte counters as chunks arrive:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;progress-&amp;gt;post_bytes_processed&lt;/code&gt; (outer progress counter for the overall request)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;progress-&amp;gt;current_file_bytes_processed&lt;/code&gt; (inner progress counter for the active file part)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Periodically—governed by &lt;code&gt;session.upload_progress.min_freq&lt;/code&gt; (which defaults to &lt;strong&gt;1 second&lt;/strong&gt;)—PHP calls &lt;code&gt;php_session_rfc1867_update()&lt;/code&gt; to flush progress state to the underlying session storage. When a second update occurs after &lt;code&gt;session.upload_progress.min_freq&lt;/code&gt; has elapsed, PHP re-reads and decodes the session data, calling &lt;code&gt;zend_hash_update()&lt;/code&gt; on the session array. That update invokes the hash table destructor (&lt;code&gt;HashTable.pDestructor = zval_ptr_dtor&lt;/code&gt;) on the previously decoded progress tree—&lt;strong&gt;freeing the underlying &lt;code&gt;zval&lt;/code&gt; structures while cached pointers in &lt;code&gt;progress&lt;/code&gt; still reference them&lt;/strong&gt;.&lt;/p&gt;
&lt;h3 id="cve-2026-109618-stale-progress-post_bytes_processed-pointer-across-multiple-files"&gt;CVE-2026-109618: Stale &lt;code&gt;progress-&amp;gt;post_bytes_processed&lt;/code&gt; Pointer Across Multiple Files&lt;/h3&gt;
&lt;p&gt;In &lt;strong&gt;CVE-2026-109618&lt;/strong&gt;, the outer &lt;code&gt;zval*&lt;/code&gt; pointer &lt;code&gt;progress-&amp;gt;post_bytes_processed&lt;/code&gt; (resolved via &lt;code&gt;zend_hash_str_find&lt;/code&gt; on &lt;code&gt;data["bytes_processed"]&lt;/code&gt;) is cached once when upload progress tracking initializes and is &lt;strong&gt;not refreshed&lt;/strong&gt; on a subsequent &lt;code&gt;MULTIPART_EVENT_FILE_START&lt;/code&gt;.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;An attacker initiates a &lt;code&gt;multipart/form-data&lt;/code&gt; request with &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; and begins streaming a first file part, triggering the initial &lt;code&gt;php_session_rfc1867_update()&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The attacker stalls the TCP transmission for longer than &lt;code&gt;session.upload_progress.min_freq&lt;/code&gt; (&lt;code&gt;1&lt;/code&gt; second).&lt;/li&gt;
&lt;li&gt;Upon resuming the stream, a second &lt;code&gt;php_session_rfc1867_update()&lt;/code&gt; runs, decoding the session and destroying the previous progress tree via &lt;code&gt;zval_ptr_dtor&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;When a second file part arrives, &lt;code&gt;MULTIPART_EVENT_FILE_START&lt;/code&gt; refreshes &lt;code&gt;current_file_bytes_processed&lt;/code&gt;, but leaves &lt;code&gt;post_bytes_processed&lt;/code&gt; pointing to freed memory.&lt;/li&gt;
&lt;li&gt;The subsequent &lt;code&gt;MULTIPART_EVENT_FILE_DATA&lt;/code&gt; callback executes &lt;code&gt;Z_LVAL_P(progress-&amp;gt;post_bytes_processed) = data-&amp;gt;post_bytes_processed&lt;/code&gt;, performing a write-after-free into the freed heap bin.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id="cve-2026-109619-stale-progress-current_file_bytes_processed-pointer-within-a-single-file"&gt;CVE-2026-109619: Stale &lt;code&gt;progress-&amp;gt;current_file_bytes_processed&lt;/code&gt; Pointer Within a Single File&lt;/h3&gt;
&lt;p&gt;In &lt;strong&gt;CVE-2026-109619&lt;/strong&gt;, the vulnerability triggers within a &lt;strong&gt;single&lt;/strong&gt; file upload without requiring a second &lt;code&gt;MULTIPART_EVENT_FILE_START&lt;/code&gt;:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;progress-&amp;gt;current_file_bytes_processed&lt;/code&gt; is looked up and cached at &lt;code&gt;MULTIPART_EVENT_FILE_START&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;On each &lt;code&gt;MULTIPART_EVENT_FILE_DATA&lt;/code&gt; chunk, PHP first writes to both cached pointers (&lt;code&gt;Z_LVAL_P(progress-&amp;gt;current_file_bytes_processed) = offset + length&lt;/code&gt; and &lt;code&gt;Z_LVAL_P(progress-&amp;gt;post_bytes_processed) = post_bytes&lt;/code&gt;) and &lt;strong&gt;then&lt;/strong&gt; calls &lt;code&gt;php_session_rfc1867_update()&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;By stalling the upload stream for longer than &lt;code&gt;session.upload_progress.min_freq&lt;/code&gt; (&lt;code&gt;1&lt;/code&gt; second) mid-file, the first post-stall &lt;code&gt;MULTIPART_EVENT_FILE_DATA&lt;/code&gt; event triggers &lt;code&gt;php_session_rfc1867_update()&lt;/code&gt;, which decodes the session and frees the old progress tree via &lt;code&gt;zval_ptr_dtor&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Because no new &lt;code&gt;MULTIPART_EVENT_FILE_START&lt;/code&gt; occurs for the next chunk of the same file, &lt;strong&gt;both&lt;/strong&gt; &lt;code&gt;current_file_bytes_processed&lt;/code&gt; and &lt;code&gt;post_bytes_processed&lt;/code&gt; remain dangling on the immediately following &lt;code&gt;MULTIPART_EVENT_FILE_DATA&lt;/code&gt; event, causing dual use-after-free writes into reclaimed Zend memory allocator (&lt;code&gt;emalloc&lt;/code&gt;) bins.&lt;/li&gt;
&lt;/ol&gt;
&lt;hr /&gt;
&lt;h2 id="why-waiting-to-upgrade-is-not-an-option-and-why-ini_set-fails"&gt;Why Waiting to Upgrade Is Not an Option (And Why &lt;code&gt;ini_set()&lt;/code&gt; Fails)&lt;/h2&gt;
&lt;p&gt;Many engineering teams assume they can mitigate PHP configuration issues by adding &lt;code&gt;ini_set('session.upload_progress.enabled', '0');&lt;/code&gt; at the top of &lt;code&gt;index.php&lt;/code&gt; or in a framework bootstrap file.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;That will not work here.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;In PHP's configuration table, &lt;code&gt;session.upload_progress.enabled&lt;/code&gt; is registered with the &lt;code&gt;PHP_INI_PERDIR&lt;/code&gt; scope—meaning it can only be set in &lt;code&gt;php.ini&lt;/code&gt;, &lt;code&gt;.user.ini&lt;/code&gt;, &lt;code&gt;php-fpm&lt;/code&gt; pool definitions, &lt;code&gt;.htaccess&lt;/code&gt;, or web server FastCGI parameters (&lt;code&gt;PHP_ADMIN_VALUE&lt;/code&gt;). By the time your PHP script executes &lt;code&gt;ini_set()&lt;/code&gt;, the RFC 1867 multipart stream has &lt;strong&gt;already been parsed&lt;/strong&gt; and the use-after-free has already been triggered.&lt;/p&gt;
&lt;p&gt;If you cannot immediately upgrade your PHP runtime due to:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;End-of-Life or LTS Linux distributions&lt;/strong&gt; awaiting backported security packages,&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Legacy PHP applications&lt;/strong&gt; (PHP 7.x / 8.0 / 8.1 / 8.2 / 8.3 / 8.4 / 8.5) locked to specific minor versions,&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Shared hosting or managed appliances&lt;/strong&gt; running precompiled &lt;code&gt;lsphp&lt;/code&gt; or &lt;code&gt;php-fpm&lt;/code&gt; binaries, or&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Containerized fleets&lt;/strong&gt; where rebuilding and validating hundreds of images takes days,&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;you must disable the vulnerable subsystem or block the trigger at the &lt;strong&gt;request / web-server layer&lt;/strong&gt; today.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="mitigation-1-zero-downtime-server-side-kill-switch-recommended"&gt;Mitigation 1: Zero-Downtime Server-Side Kill Switch (Recommended)&lt;/h2&gt;
&lt;p&gt;Virtually no modern web application uses PHP's legacy RFC 1867 session upload progress (modern browsers track file upload progress natively on the client side using &lt;code&gt;XMLHttpRequest.upload.onprogress&lt;/code&gt; or the Fetch API).&lt;/p&gt;
&lt;p&gt;Disabling &lt;code&gt;session.upload_progress.enabled&lt;/code&gt; completely bypasses &lt;code&gt;php_session_rfc1867_callback()&lt;/code&gt; in &lt;code&gt;ext/session/session.c&lt;/code&gt;, neutralizing both &lt;strong&gt;CVE-2026-109618&lt;/strong&gt; and &lt;strong&gt;CVE-2026-109619&lt;/strong&gt; with zero functional impact on file uploads or sessions.&lt;/p&gt;
&lt;h3 id="option-a-nginx-php-fpm-fastcgi_param-php_admin_value"&gt;Option A: NGINX + PHP-FPM (&lt;code&gt;fastcgi_param PHP_ADMIN_VALUE&lt;/code&gt;)&lt;/h3&gt;
&lt;p&gt;You do not even need to touch &lt;code&gt;php.ini&lt;/code&gt; or restart &lt;code&gt;php-fpm&lt;/code&gt; workers if you inject &lt;code&gt;PHP_ADMIN_VALUE&lt;/code&gt; directly from NGINX. Using &lt;code&gt;PHP_ADMIN_VALUE&lt;/code&gt; (instead of &lt;code&gt;PHP_VALUE&lt;/code&gt;) ensures that no &lt;code&gt;.user.ini&lt;/code&gt; file can re-enable the setting:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sr"&gt;\.php$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;fastcgi_params&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;fastcgi_pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;unix:/run/php/php-fpm.sock&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# or your upstream pool&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Neutralize CVE-2026-109618 &amp;amp; CVE-2026-109619 across all requests&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;fastcgi_param&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;PHP_ADMIN_VALUE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;"session.upload_progress.enabled=0"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Test and reload NGINX with zero downtime:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;reload&lt;span class="w"&gt; &lt;/span&gt;nginx
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="option-b-apache-mod_proxy_fcgi-or-mod_php"&gt;Option B: Apache (&lt;code&gt;mod_proxy_fcgi&lt;/code&gt; or &lt;code&gt;mod_php&lt;/code&gt;)&lt;/h3&gt;
&lt;p&gt;If you run Apache fronting PHP-FPM via &lt;code&gt;mod_proxy_fcgi&lt;/code&gt;, pass &lt;code&gt;PHP_ADMIN_VALUE&lt;/code&gt; in the environment:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;IfModule&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;mod_proxy_fcgi.c&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;SetEnv&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;PHP_ADMIN_VALUE&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"session.upload_progress.enabled=0"&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/IfModule&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you run legacy &lt;code&gt;mod_php&lt;/code&gt; (or LiteSpeed reading Apache configuration directives), lock the flag down with &lt;code&gt;php_admin_flag&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;IfModule&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;mod_php.c&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;php_admin_flag&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;session.upload_progress.enabled&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;off&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/IfModule&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Gracefully reload Apache:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;apachectl&lt;span class="w"&gt; &lt;/span&gt;configtest&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;reload&lt;span class="w"&gt; &lt;/span&gt;apache2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="option-c-haproxy-native-fastcgi-fcgi-app"&gt;Option C: HAProxy Native FastCGI (&lt;code&gt;fcgi-app&lt;/code&gt;)&lt;/h3&gt;
&lt;p&gt;If HAProxy communicates directly with &lt;code&gt;php-fpm&lt;/code&gt; backends over FastCGI (&lt;code&gt;proto fcgi&lt;/code&gt;), inject &lt;code&gt;PHP_ADMIN_VALUE&lt;/code&gt; inside your &lt;code&gt;fcgi-app&lt;/code&gt; definition:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;fcgi-app php-fpm-app
    docroot /var/www/html
    index index.php
    path-info ^(/.+\.php)(/.*)?$
    # Disable PHP session upload progress across all FastCGI requests
    set-param PHP_ADMIN_VALUE "session.upload_progress.enabled=0"

backend php_fpm_nodes
    mode http
    use-fcgi-app php-fpm-app
    server fpm1 127.0.0.1:9000 proto fcgi check
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Validate and reload HAProxy cleanly:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;haproxy&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/haproxy/haproxy.cfg&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;reload&lt;span class="w"&gt; &lt;/span&gt;haproxy
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="option-d-global-phpini-and-php-fpm-pool-configuration"&gt;Option D: Global &lt;code&gt;php.ini&lt;/code&gt; and PHP-FPM Pool Configuration&lt;/h3&gt;
&lt;p&gt;To enforce the mitigation permanently across CLI, FPM, and all SAPIs on the host, drop a dedicated configuration snippet into your PHP &lt;code&gt;conf.d&lt;/code&gt; directory (e.g., &lt;code&gt;/etc/php/*/fpm/conf.d/99-disable-upload-progress.ini&lt;/code&gt;):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;; Mitigate CVE-2026-109618 and CVE-2026-109619&lt;/span&gt;
&lt;span class="na"&gt;session.upload_progress.enabled&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Or lock it inside your PHP-FPM pool configurations (&lt;code&gt;/etc/php/*/fpm/pool.d/www.conf&lt;/code&gt;):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;php_admin_flag[session.upload_progress.enabled]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;off&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr /&gt;
&lt;h2 id="mitigation-2-edge-waf-request-filtering-block-php_session_upload_progress"&gt;Mitigation 2: Edge &amp;amp; WAF Request Filtering (Block &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt;)&lt;/h2&gt;
&lt;p&gt;In large multi-tenant environments, reverse-proxy fleets, or appliances where you cannot immediately modify backend FastCGI parameters, you should block any incoming &lt;code&gt;multipart/form-data&lt;/code&gt; request attempting to pass the &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; field (or whatever custom name is configured in &lt;code&gt;session.upload_progress.name&lt;/code&gt;).&lt;/p&gt;
&lt;h3 id="haproxy-l7-request-body-inspection"&gt;HAProxy (L7 Request Body Inspection)&lt;/h3&gt;
&lt;p&gt;When HAProxy fronts your web tier over HTTP/HTTPS, you can enable &lt;code&gt;option http-buffer-request&lt;/code&gt; in your &lt;code&gt;frontend&lt;/code&gt; or &lt;code&gt;backend&lt;/code&gt; and reject &lt;code&gt;multipart/form-data&lt;/code&gt; requests containing &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; before they ever reach an upstream server:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;frontend https_edge
    bind :443 ssl crt /etc/haproxy/certs/
    mode http

    # Buffer the incoming request body so ACLs can inspect multipart fields
    option http-buffer-request

    # Match POST multipart/form-data requests containing PHP_SESSION_UPLOAD_PROGRESS
    acl is_post method POST
    acl is_multipart req.hdr(Content-Type) -i -m sub multipart/form-data
    acl has_upload_progress req.body -m sub PHP_SESSION_UPLOAD_PROGRESS

    http-request deny deny_status 403 if is_post is_multipart has_upload_progress

    default_backend web_backends
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip:&lt;/strong&gt; HAProxy inspects &lt;code&gt;req.body&lt;/code&gt; up to &lt;code&gt;tune.bufsize&lt;/code&gt; (default &lt;code&gt;16384&lt;/code&gt; bytes). Because RFC 1867 requires &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; to appear &lt;strong&gt;before&lt;/strong&gt; the uploaded file part in the multipart stream so PHP knows to initialize tracking at &lt;code&gt;MULTIPART_EVENT_FILE_START&lt;/code&gt; (or if you also increase &lt;code&gt;tune.bufsize 65536&lt;/code&gt; in &lt;code&gt;global&lt;/code&gt;), inspecting the buffered request body reliably catches exploit attempts at the edge.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3 id="modsecurity-apache-nginx-or-owasp-crs"&gt;ModSecurity (Apache, NGINX, or OWASP CRS)&lt;/h3&gt;
&lt;p&gt;Add the following rules to your custom ModSecurity configuration (e.g., &lt;code&gt;REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf&lt;/code&gt; or &lt;code&gt;custom-rules.conf&lt;/code&gt;):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c"&gt;# Rule 1: Block parsed POST form fields named PHP_SESSION_UPLOAD_PROGRESS&lt;/span&gt;
&lt;span class="nb"&gt;SecRule&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;REQUEST_METHOD&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"@streq POST"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;"id:1000101,\&lt;/span&gt;
&lt;span class="s2"&gt;    phase:2,\&lt;/span&gt;
&lt;span class="s2"&gt;    block,\&lt;/span&gt;
&lt;span class="s2"&gt;    status:403,\&lt;/span&gt;
&lt;span class="s2"&gt;    log,\&lt;/span&gt;
&lt;span class="s2"&gt;    msg:'CVE-2026-109618/CVE-2026-109619: Blocked PHP_SESSION_UPLOAD_PROGRESS field',\&lt;/span&gt;
&lt;span class="s2"&gt;    chain"&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;SecRule&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ARGS_POST_NAMES&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"@contains PHP_SESSION_UPLOAD_PROGRESS"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;

&lt;span class="c"&gt;# Rule 2: Inspect raw multipart/form-data body for the Content-Disposition field name&lt;/span&gt;
&lt;span class="nb"&gt;SecRule&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;REQUEST_BODY&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"@rx name=[\"']?PHP_SESSION_UPLOAD_PROGRESS[\"']?"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;"id:1000102,\&lt;/span&gt;
&lt;span class="s2"&gt;    phase:2,\&lt;/span&gt;
&lt;span class="s2"&gt;    block,\&lt;/span&gt;
&lt;span class="s2"&gt;    status:403,\&lt;/span&gt;
&lt;span class="s2"&gt;    log,\&lt;/span&gt;
&lt;span class="s2"&gt;    msg:'CVE-2026-109618/CVE-2026-109619: Blocked raw multipart PHP_SESSION_UPLOAD_PROGRESS'"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="openresty-nginx-with-lua-nginx-module"&gt;OpenResty / NGINX with &lt;code&gt;lua-nginx-module&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;If you terminate traffic at OpenResty or NGINX with Lua enabled, you can inspect &lt;code&gt;multipart/form-data&lt;/code&gt; bodies at the edge and reject matching requests with &lt;code&gt;403 Forbidden&lt;/code&gt; before they ever reach an upstream PHP worker:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;access_by_lua_block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;local&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;content_type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ngx.req.get_headers()["content-type"]&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="s"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;content_type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;string.find(content_type:lower(),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;"multipart/form-data",&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;true)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="s"&gt;ngx.req.read_body()&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="s"&gt;local&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;body&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ngx.req.get_body_data()&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="s"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;body&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s"&gt;local&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;body_file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ngx.req.get_body_file()&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;body_file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;                    &lt;/span&gt;&lt;span class="s"&gt;local&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;io.open(body_file,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;"rb")&lt;/span&gt;
&lt;span class="w"&gt;                    &lt;/span&gt;&lt;span class="s"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="s"&gt;body&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;f:read(65536)&lt;/span&gt;
&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="s"&gt;f:close()&lt;/span&gt;
&lt;span class="w"&gt;                    &lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="s"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;body&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;string.find(body,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;"PHP_SESSION_UPLOAD_PROGRESS",&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;true)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s"&gt;ngx.log(ngx.ERR,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;"Blocked&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;CVE-2026-109618&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;CVE-2026-109619&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;exploit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;attempt")&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ngx.exit(ngx.HTTP_FORBIDDEN)&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="s"&gt;end&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Standard proxy_pass or fastcgi_pass configuration below&lt;/span&gt;
&lt;span class="err"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr /&gt;
&lt;h2 id="mitigation-3-reverse-proxy-request-buffering-timeout-hardening"&gt;Mitigation 3: Reverse Proxy Request Buffering &amp;amp; Timeout Hardening&lt;/h2&gt;
&lt;p&gt;Both &lt;strong&gt;CVE-2026-109618&lt;/strong&gt; and &lt;strong&gt;CVE-2026-109619&lt;/strong&gt; require the multipart upload stream to span longer than &lt;code&gt;session.upload_progress.min_freq&lt;/code&gt; (default &lt;code&gt;1&lt;/code&gt; second) while PHP is actively reading the request body, forcing a mid-stream &lt;code&gt;php_session_rfc1867_update()&lt;/code&gt; cycle.&lt;/p&gt;
&lt;p&gt;When your reverse proxy &lt;strong&gt;fully buffers&lt;/strong&gt; the client request body on disk/memory before opening the FastCGI or HTTP connection to the PHP backend, PHP receives the entire multipart payload in a single local burst in under a millisecond—preventing the 1-second &lt;code&gt;min_freq&lt;/code&gt; window from ever elapsing during &lt;code&gt;rfc1867_post_handler()&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="nginx-request-buffering"&gt;NGINX Request Buffering&lt;/h3&gt;
&lt;p&gt;Ensure that &lt;code&gt;fastcgi_request_buffering&lt;/code&gt; and &lt;code&gt;proxy_request_buffering&lt;/code&gt; are set to &lt;code&gt;on&lt;/code&gt; (the NGINX default, though sometimes disabled by operators for large file upload routes):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;http&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Buffer the entire client request body before forwarding to PHP-FPM&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;fastcgi_request_buffering&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;proxy_request_buffering&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Enforce strict timeouts against intentionally stalled request bodies&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;client_body_timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;10s&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="haproxy-request-buffering-stall-timeouts"&gt;HAProxy Request Buffering &amp;amp; Stall Timeouts&lt;/h3&gt;
&lt;p&gt;In HAProxy, enabling &lt;code&gt;option http-buffer-request&lt;/code&gt; alongside a strict &lt;code&gt;timeout http-request&lt;/code&gt; forces HAProxy to wait for the buffered payload within the timeout window rather than trickling stalled chunks directly into an open PHP worker:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;defaults
    mode http
    option http-buffer-request
    timeout http-request 5s
    timeout client-fin 5s
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="apache-mod_reqtimeout"&gt;Apache &lt;code&gt;mod_reqtimeout&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;If Apache accepts client connections directly without a buffering reverse proxy in front of it, enable &lt;code&gt;mod_reqtimeout&lt;/code&gt; to drop connections that intentionally stall mid-body:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;IfModule&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;mod_reqtimeout.c&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;RequestReadTimeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;header=10-20,MinRate=500&lt;span class="w"&gt; &lt;/span&gt;body=10,MinRate=1000
&lt;span class="nt"&gt;&amp;lt;/IfModule&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Request buffering and timeouts are defense-in-depth measures. Always pair them with &lt;strong&gt;Mitigation 1&lt;/strong&gt; (&lt;code&gt;session.upload_progress.enabled=0&lt;/code&gt;) or &lt;strong&gt;Mitigation 2&lt;/strong&gt; (WAF filtering), as large uploads over high-latency disks could still cross the 1-second threshold if upload progress remains enabled.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;hr /&gt;
&lt;h2 id="auditing-your-fleet-right-now"&gt;Auditing Your Fleet Right Now&lt;/h2&gt;
&lt;p&gt;To quickly check whether any PHP installations or FPM pools on a Linux host still have &lt;code&gt;session.upload_progress.enabled&lt;/code&gt; active, run this one-liner audit across your servers:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;bin&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;which&lt;span class="w"&gt; &lt;/span&gt;php&lt;span class="w"&gt; &lt;/span&gt;php-fpm&lt;span class="w"&gt; &lt;/span&gt;php-fpm8.1&lt;span class="w"&gt; &lt;/span&gt;php-fpm8.2&lt;span class="w"&gt; &lt;/span&gt;php-fpm8.3&lt;span class="w"&gt; &lt;/span&gt;php-fpm8.4&lt;span class="w"&gt; &lt;/span&gt;php-fpm8.5&lt;span class="w"&gt; &lt;/span&gt;lsphp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"=== Auditing: &lt;/span&gt;&lt;span class="nv"&gt;$bin&lt;/span&gt;&lt;span class="s2"&gt; ==="&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$bin&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-E&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"session\.upload_progress\.(enabled|name|min_freq)"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And inspect all &lt;code&gt;php.ini&lt;/code&gt; and FPM pool files on disk:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-rn&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"upload_progress"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/etc/php*&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/php*&lt;span class="w"&gt; &lt;/span&gt;/usr/local/lsws/lsphp*&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="verifying-that-the-mitigation-is-active"&gt;Verifying That the Mitigation Is Active&lt;/h3&gt;
&lt;p&gt;Once you have applied &lt;code&gt;session.upload_progress.enabled=0&lt;/code&gt;, verify via a temporary diagnostic check or &lt;code&gt;php -i&lt;/code&gt; that both the &lt;strong&gt;Local Value&lt;/strong&gt; and &lt;strong&gt;Master Value&lt;/strong&gt; report &lt;code&gt;Off&lt;/code&gt; (&lt;code&gt;0&lt;/code&gt;):&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;session.upload_progress.enabled =&amp;gt; Off =&amp;gt; Off
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you deployed the WAF or OpenResty rule, test against a staging endpoint with a benign &lt;code&gt;multipart/form-data&lt;/code&gt; request containing the &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; field and confirm that your edge immediately returns &lt;code&gt;HTTP/1.1 403 Forbidden&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;POST&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://staging.example.com/index.php"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-F&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PHP_SESSION_UPLOAD_PROGRESS=test"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-F&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"file=@/dev/null;filename=test.txt"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr /&gt;
&lt;h2 id="frequently-asked-questions-faq"&gt;Frequently Asked Questions (FAQ)&lt;/h2&gt;
&lt;h3 id="does-disabling-sessionupload_progressenabled-break-normal-file-uploads-in-php"&gt;Does disabling &lt;code&gt;session.upload_progress.enabled&lt;/code&gt; break normal file uploads in PHP?&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;No.&lt;/strong&gt; Standard file uploads via &lt;code&gt;$_FILES&lt;/code&gt; and &lt;code&gt;move_uploaded_file()&lt;/code&gt; are handled by PHP's core RFC 1867 parser (&lt;code&gt;file_uploads = On&lt;/code&gt;) and are completely independent of &lt;code&gt;session.upload_progress.enabled&lt;/code&gt;. Disabling session upload progress only stops PHP from writing real-time byte counters into &lt;code&gt;$_SESSION["upload_progress_..."]&lt;/code&gt; while the upload is in flight.&lt;/p&gt;
&lt;h3 id="are-applications-that-dont-use-session_start-or-file-uploads-still-vulnerable"&gt;Are applications that don't use &lt;code&gt;session_start()&lt;/code&gt; or file uploads still vulnerable?&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Yes.&lt;/strong&gt; When &lt;code&gt;session.upload_progress.enabled=1&lt;/code&gt; is set (the default) and a &lt;code&gt;POST&lt;/code&gt; request contains &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt;, PHP automatically initializes the session module during &lt;code&gt;rfc1867_post_handler()&lt;/code&gt; before executing the target PHP script—even if that script never calls &lt;code&gt;session_start()&lt;/code&gt; or touches &lt;code&gt;$_FILES&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="can-i-fix-cve-2026-109618-and-cve-2026-109619-using-ini_set-in-my-php-code"&gt;Can I fix CVE-2026-109618 and CVE-2026-109619 using &lt;code&gt;ini_set()&lt;/code&gt; in my PHP code?&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;No.&lt;/strong&gt; Because &lt;code&gt;rfc1867_post_handler()&lt;/code&gt; runs during the SAPI request initialization phase prior to script execution, the memory corruption occurs before the first line of your &lt;code&gt;.php&lt;/code&gt; script is reached. You must disable &lt;code&gt;session.upload_progress.enabled&lt;/code&gt; in &lt;code&gt;php.ini&lt;/code&gt;, &lt;code&gt;php-fpm&lt;/code&gt; pool configs, or via &lt;code&gt;PHP_ADMIN_VALUE&lt;/code&gt; in NGINX/Apache.&lt;/p&gt;</content>
  </entry>
  <entry>
    <title>Bridging a Cloud Condo Intercom to Home Assistant with Rust, SIP, and MQTT</title>
    <link href="https://dj.vg/posts/bridging-condo-intercom-to-home-assistant-rust-sip-mqtt/" rel="alternate" type="text/html"/>
    <link href="https://dj.vg/posts/bridging-condo-intercom-to-home-assistant-rust-sip-mqtt/index.md" rel="alternate" type="text/markdown"/>
    <id>https://dj.vg/posts/bridging-condo-intercom-to-home-assistant-rust-sip-mqtt/</id>
    <published>2026-10-17T00:00:00Z</published>
    <updated>2026-10-17T00:00:00Z</updated>
    <summary>How I replaced a sluggish property-management app with a self-contained Rust daemon that speaks Firebase Cloud Messaging, SIP/TLS, H.264 RTP, and Home Assistant MQTT Auto-Discovery.</summary>
    <content type="html">&lt;p&gt;In a well-tuned smart home, almost every physical interaction is programmable. Lighting scenes adapt to the time of day, climate control reacts to occupancy, and alerts route cleanly through a single dashboard in &lt;a href="https://www.home-assistant.io/"&gt;Home Assistant&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Yet in many modern condominiums in Singapore, one piece of hardware stubbornly resists integration: &lt;strong&gt;the building intercom&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Instead of exposing a standard local LAN doorphone inside the apartment, many residential developments route their lobby call panels through cloud-managed property management apps—such as &lt;strong&gt;iPlus Living&lt;/strong&gt; backed by a &lt;strong&gt;Fermax MeetMe&lt;/strong&gt; SIP infrastructure. When a visitor or delivery driver dials your unit from the lobby or side gate, your phone receives a push notification from a proprietary app, you wait for the app to cold-start and negotiate a video stream, and you tap an icon to buzz the gate open.&lt;/p&gt;
&lt;p&gt;From a home automation perspective, this is a dead end:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Siloed notifications&lt;/strong&gt;: You cannot route doorbell events or camera snapshots into Home Assistant notifications, wall tablets, or TVs.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No programmable access&lt;/strong&gt;: You cannot trigger an unlock from a Home Assistant dashboard, an NFC tag, or a temporary "delivery window" automation.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Single-platform lock-in&lt;/strong&gt;: Everything depends on a mobile app staying logged in and surviving aggressive mobile OS background battery killers.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I wanted the condo intercom to behave like any other native device in my cluster. So I built &lt;strong&gt;&lt;code&gt;iplusliving2mqtt&lt;/code&gt;&lt;/strong&gt;—a self-contained Rust daemon running in Kubernetes that bridges the entire intercom lifecycle into Home Assistant via MQTT Auto-Discovery.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="why-standard-sip-clients-dont-work"&gt;Why Standard SIP Clients Don't Work&lt;/h2&gt;
&lt;p&gt;At first glance, a SIP-based intercom sounds trivial: grab the SIP credentials, point &lt;code&gt;baresip&lt;/code&gt;, Asterisk, or Home Assistant's VoIP integration at the server, and wait for an incoming &lt;code&gt;INVITE&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;In practice, inspecting the protocol flow revealed three layers that prevent a standard SIP client from ever receiving a call:&lt;/p&gt;
&lt;h3 id="1-split-key-encrypted-provisioning-api"&gt;1. Split-Key Encrypted Provisioning API&lt;/h3&gt;
&lt;p&gt;Rather than giving residents static SIP credentials, the iPlus Living mobile app dynamically provisions them after login. Every HTTP request and response body is encrypted with &lt;strong&gt;AES-256-CBC&lt;/strong&gt; using a fixed initialization vector, while the 32-byte AES key is hidden inside a 42-byte random alphanumeric prefix prepended to the base64 ciphertext (&lt;code&gt;payload[:16] + payload[22:38]&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;Authenticating through the four-step login sequence (&lt;code&gt;user/login&lt;/code&gt; → &lt;code&gt;user/logindetail&lt;/code&gt; → &lt;code&gt;intercom/meetmelicencedetails&lt;/code&gt; → &lt;code&gt;intercom/intercomconfiguration&lt;/code&gt;) returns:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Cloud MeetMe SIP credentials&lt;/strong&gt; (&lt;code&gt;sip.fermax.com&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Building PBX SIP credentials&lt;/strong&gt; (the development's Grandstream server)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The call panel directory&lt;/strong&gt; (mapping numeric SIP caller IDs like &lt;code&gt;14&lt;/code&gt; or &lt;code&gt;22&lt;/code&gt; to human-readable gate and lobby names)&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="2-the-push-notification-catch-22-rfc-8599"&gt;2. The Push Notification Catch-22 (RFC 8599)&lt;/h3&gt;
&lt;p&gt;Even with valid SIP credentials and an active TLS connection to Fermax's &lt;code&gt;Flexisip&lt;/code&gt; proxy (&lt;code&gt;_sips._tcp.sip.fermax.com&lt;/code&gt;), a standard SIP client will sit idle forever when someone presses the doorbell.&lt;/p&gt;
&lt;p&gt;Why? Because the &lt;code&gt;Flexisip&lt;/code&gt; server is configured for mobile push notifications under &lt;strong&gt;RFC 8599&lt;/strong&gt; with &lt;code&gt;pn-timeout=0&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Contact: &amp;lt;sip:USER@IP:PORT;pn-prid=FCM_TOKEN;pn-provider=fcm;pn-param=SENDER_ID;pn-silent=1;pn-timeout=0;transport=tls&amp;gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With &lt;code&gt;pn-timeout=0&lt;/code&gt;, the SIP proxy assumes mobile clients go to sleep immediately after registering. When the lobby panel places a call:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;code&gt;Flexisip&lt;/code&gt; &lt;strong&gt;pauses&lt;/strong&gt; the incoming &lt;code&gt;INVITE&lt;/code&gt; transaction.&lt;/li&gt;
&lt;li&gt;It dispatches a high-priority data push (&lt;code&gt;title: "IC_CALL"&lt;/code&gt;) through &lt;strong&gt;Google Firebase Cloud Messaging (FCM)&lt;/strong&gt; to the &lt;code&gt;pn-prid&lt;/code&gt; token registered in the &lt;code&gt;Contact&lt;/code&gt; header.&lt;/li&gt;
&lt;li&gt;It waits for the client to wake up and send a fresh SIP &lt;code&gt;REGISTER&lt;/code&gt; over TLS.&lt;/li&gt;
&lt;li&gt;Only after receiving that &lt;code&gt;REGISTER&lt;/code&gt; does &lt;code&gt;Flexisip&lt;/code&gt; release the held &lt;code&gt;INVITE&lt;/code&gt; to the client.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Without a real FCM token and a persistent listener on Google's Mobile Connection Server (&lt;code&gt;mtalk.google.com:5228&lt;/code&gt;), the SIP server never forwards the call.&lt;/p&gt;
&lt;h3 id="3-early-media-h264-video-dual-mode-dtmf"&gt;3. Early-Media H.264 Video &amp;amp; Dual-Mode DTMF&lt;/h3&gt;
&lt;p&gt;Once the &lt;code&gt;INVITE&lt;/code&gt; arrives, the lobby panel streams H.264 video during &lt;strong&gt;early media&lt;/strong&gt; (&lt;code&gt;183 Session Progress&lt;/code&gt;) before the call is answered. However, the door relay controller ignores unlock commands during early media—to actually buzz the gate open, the client must transition the call to answered (&lt;code&gt;200 OK&lt;/code&gt; with SDP), wait for the &lt;code&gt;ACK&lt;/code&gt;, and transmit DTMF unlock tones before hanging up (&lt;code&gt;BYE&lt;/code&gt;).&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="system-architecture"&gt;System Architecture&lt;/h2&gt;
&lt;p&gt;To make this rock-solid in Kubernetes without relying on external headless Android emulators, &lt;code&gt;ffmpeg&lt;/code&gt; subprocesses, or Python scripts, I built &lt;code&gt;iplusliving2mqtt&lt;/code&gt; as a single static &lt;strong&gt;Rust&lt;/strong&gt; binary:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;  ┌──────────────────────────┐        ┌──────────────────────────┐
  │    iPlus Living API      │        │   Google FCM / MCS       │
  │  (AES-256-CBC Provision) │        │ (mtalk.google.com:5228)  │
  └────────────┬─────────────┘        └────────────┬─────────────┘
               │ 1. Fetch SIP + Panels             │ 2. Push "IC_CALL"
               ▼                                   ▼
  ┌──────────────────────────────────────────────────────────────┐
  │                      iplusliving2mqtt                        │
  │                                                              │
  │  ┌──────────────────────┐         ┌───────────────────────┐  │
  │  │  Provisioning State  │         │  Native FCM Listener  │  │
  │  │     (state.json)     │         │  (Protobuf over TLS)  │  │
  │  └──────────┬───────────┘         └───────────┬───────────┘  │
  │             │                                 │ Wake Signal  │
  │             ▼                                 ▼              │
  │  ┌────────────────────────────────────────────────────────┐  │
  │  │            Dual-Account SIP/TLS Engine                 │  │
  │  │        (MeetMe Cloud + Building Grandstream)           │  │
  │  └──────────┬─────────────────────────────────┬───────────┘  │
  │             │ RTP H.264 + RTCP PLI            │ SIP Events   │
  │             ▼                                 ▼              │
  │  ┌──────────────────────┐         ┌───────────────────────┐  │
  │  │   OpenH264 Decoder   │────────▶│  MQTT Auto-Discovery  │  │
  │  │   &amp;amp; JPEG Encoder     │         │   &amp;amp; Command Bridge    │  │
  │  └──────────────────────┘         └───────────┬───────────┘  │
  └───────────────────────────────────────────────┼──────────────┘
                                                  │
                                                  ▼
                                      ┌───────────────────────┐
                                      │    Home Assistant     │
                                      │ (Camera, Lock, Ring)  │
                                      └───────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="zero-touch-provisioning-fcm-emulation"&gt;Zero-Touch Provisioning &amp;amp; FCM Emulation&lt;/h3&gt;
&lt;p&gt;On startup, if no cached state exists on its persistent volume, &lt;code&gt;iplusliving2mqtt&lt;/code&gt; performs a two-stage bootstrap:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;FCM Registration&lt;/strong&gt;: Emulates an Android check-in against &lt;code&gt;android.clients.google.com/checkin&lt;/code&gt; and registers with Google Cloud Messaging (&lt;code&gt;/c2dm/register3&lt;/code&gt;) using the official app's Firebase project parameters to obtain a valid &lt;code&gt;fcm_token&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;iPlus Auto-Provisioning&lt;/strong&gt;: Logs into the iPlus Living API using the resident's email and password, binds the &lt;code&gt;fcm_token&lt;/code&gt;, and caches the resulting SIP accounts and call panel directory to &lt;code&gt;state.json&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;MCS Stream&lt;/strong&gt;: Opens a persistent TLS connection to &lt;code&gt;mtalk.google.com:5228&lt;/code&gt; speaking Google's binary MCS framing + Protobuf protocol. The moment an &lt;code&gt;IC_CALL&lt;/code&gt; data message arrives, it signals the SIP worker tasks over an internal &lt;code&gt;tokio::sync::mpsc&lt;/code&gt; channel to fire an immediate SIP &lt;code&gt;REGISTER&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id="in-memory-h264-keyframe-extraction"&gt;In-Memory H.264 Keyframe Extraction&lt;/h3&gt;
&lt;p&gt;Lobby panels stream H.264 video encapsulated in RTP packets (RFC 6184), including fragmented NAL units (&lt;code&gt;FU-A&lt;/code&gt;, type 28) and aggregation packets (&lt;code&gt;STAP-A&lt;/code&gt;, type 24).&lt;/p&gt;
&lt;p&gt;Rather than spawning &lt;code&gt;ffmpeg&lt;/code&gt; on every doorbell ring, &lt;code&gt;iplusliving2mqtt&lt;/code&gt; handles the media stream directly in memory:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Sends UDP NAT-punching packets plus an &lt;strong&gt;RTCP Picture Loss Indication (PLI)&lt;/strong&gt; and a &lt;strong&gt;SIP &lt;code&gt;INFO&lt;/code&gt; (&lt;code&gt;picture_fast_update&lt;/code&gt;)&lt;/strong&gt; request immediately upon answering with &lt;code&gt;183 Session Progress&lt;/code&gt; so the panel transmits an IDR keyframe right away.&lt;/li&gt;
&lt;li&gt;Reassembles fragmented H.264 NAL units (&lt;code&gt;SPS&lt;/code&gt;, &lt;code&gt;PPS&lt;/code&gt;, and &lt;code&gt;IDR&lt;/code&gt; slices) and feeds the Annex-B bitstream into Cisco's &lt;strong&gt;&lt;code&gt;openh264&lt;/code&gt;&lt;/strong&gt; decoder in memory.&lt;/li&gt;
&lt;li&gt;Converts the decoded YUV420p frame to RGB and encodes a JPEG snapshot directly onto the MQTT camera topic (&lt;code&gt;fermax/intercom/snapshot/image&lt;/code&gt;) and an HTTP &lt;code&gt;/snapshot.jpg&lt;/code&gt; endpoint.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="dual-mode-dtmf-unlock"&gt;Dual-Mode DTMF Unlock&lt;/h3&gt;
&lt;p&gt;Different intercom relays along the call path can be picky about how DTMF digits are delivered. When you press the Unlock button in Home Assistant (or when auto-unlock is enabled), the daemon:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Answers the pending &lt;code&gt;INVITE&lt;/code&gt; with &lt;code&gt;200 OK&lt;/code&gt; (if still in &lt;code&gt;183 Session Progress&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Transmits &lt;strong&gt;RFC 2833 (&lt;code&gt;telephone-event/8000&lt;/code&gt;)&lt;/strong&gt; RTP payload packets with the end-of-event marker bit and triple redundancy.&lt;/li&gt;
&lt;li&gt;Follows up with synthesized &lt;strong&gt;in-band G.711 µ-law (PCMU)&lt;/strong&gt; dual-tone sine waves (&lt;code&gt;697–941 Hz&lt;/code&gt; + &lt;code&gt;1209–1477 Hz&lt;/code&gt;) at 8,000 Hz so both digital SIP gateways and analog door strike controllers reliably trigger.&lt;/li&gt;
&lt;li&gt;Sends a clean SIP &lt;code&gt;BYE&lt;/code&gt; to release the lobby panel.&lt;/li&gt;
&lt;/ol&gt;
&lt;hr /&gt;
&lt;h2 id="a-fun-dual-stack-debugging-quirk-ipv6-vs-ipv4-in-sip"&gt;A Fun Dual-Stack Debugging Quirk: IPv6 vs. IPv4 in SIP&lt;/h2&gt;
&lt;p&gt;When testing the daemon on a dual-stack host with native IPv6 connectivity, the initial SIP TLS connection to &lt;code&gt;sip5.fermax.com:5223&lt;/code&gt; completed its TLS handshake and then immediately dropped with an &lt;code&gt;EOF&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Digging into the packet trace uncovered two separate issues:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;RFC 3261 IPv6 Formatting&lt;/strong&gt;: &lt;code&gt;sip5.fermax.com&lt;/code&gt; publishes both &lt;code&gt;A&lt;/code&gt; and &lt;code&gt;AAAA&lt;/code&gt; records. Because the host preferred IPv6, &lt;code&gt;tcp.local_addr()&lt;/code&gt; returned an IPv6 address. Formatting &lt;code&gt;Via&lt;/code&gt; and &lt;code&gt;Contact&lt;/code&gt; headers as &lt;code&gt;ip:port&lt;/code&gt; without enclosing the IPv6 literal in square brackets (&lt;code&gt;[2a09:...]:5060&lt;/code&gt;) caused &lt;code&gt;Flexisip&lt;/code&gt;'s parser to reject the &lt;code&gt;REGISTER&lt;/code&gt; packet immediately.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;IPv4-Only Media Relays&lt;/strong&gt;: Even with bracketed IPv6 headers, the building's local PBX and the RTP audio/video media relays (&lt;code&gt;c=IN IP4 ...&lt;/code&gt; in SDP) are strictly IPv4. Because the SIP engine learns its external NAT address from the SIP server's &lt;code&gt;Via: ...;received=&amp;lt;ip&amp;gt;;rport=&amp;lt;port&amp;gt;&lt;/code&gt; response header to advertise in the SDP answer, registering over IPv6 leaves the media engine with an IPv6 address that an IPv4 RTP relay can never reach.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Explicitly resolving and selecting IPv4 socket addresses for the SIP/RTP path immediately brought both the &lt;code&gt;MeetMe&lt;/code&gt; and &lt;code&gt;Grandstream&lt;/code&gt; registrations to &lt;code&gt;200 OK&lt;/code&gt;.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="home-assistant-integration"&gt;Home Assistant Integration&lt;/h2&gt;
&lt;p&gt;Because &lt;code&gt;iplusliving2mqtt&lt;/code&gt; publishes Home Assistant MQTT Auto-Discovery payloads under &lt;code&gt;homeassistant/#&lt;/code&gt;, zero manual YAML configuration is required in Home Assistant. All entities automatically group under a single &lt;strong&gt;Fermax MeetMe Intercom&lt;/strong&gt; device:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th style="text-align: left;"&gt;Entity&lt;/th&gt;
&lt;th style="text-align: left;"&gt;Type&lt;/th&gt;
&lt;th style="text-align: left;"&gt;Description&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Doorbell Ringing&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;code&gt;binary_sensor&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Turns &lt;code&gt;ON&lt;/code&gt; the instant an incoming &lt;code&gt;INVITE&lt;/code&gt; arrives from any gate or lobby panel.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Call State&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;code&gt;sensor&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Tracks the call lifecycle: &lt;code&gt;idle&lt;/code&gt; → &lt;code&gt;ringing&lt;/code&gt; → &lt;code&gt;answered&lt;/code&gt; → &lt;code&gt;unlocked&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Last Caller&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;code&gt;sensor&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Displays the resolved lobby or gate name (e.g., &lt;em&gt;Side Gate&lt;/em&gt; or &lt;em&gt;Block Lobby&lt;/em&gt;) plus JSON attributes.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Intercom Camera&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;code&gt;camera&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Live JPEG snapshot decoded from the panel's H.264 RTP video stream.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Unlock Door&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;code&gt;button&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Immediately answers the active call and sends the DTMF unlock sequence.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Door Lock&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;code&gt;lock&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Reflects &lt;code&gt;LOCKED&lt;/code&gt; / &lt;code&gt;UNLOCKED&lt;/code&gt; state and supports standard Home Assistant lock actions.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align: left;"&gt;&lt;strong&gt;Auto-Unlock on Ring&lt;/strong&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;&lt;code&gt;switch&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: left;"&gt;Optional toggle that automatically answers and buzzes the door open after a configurable delay.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;With everything in Home Assistant, a doorbell ring now triggers an instant push notification to our phones with the live decoded camera snapshot attached and an inline &lt;strong&gt;"Unlock Gate"&lt;/strong&gt; action button—plus a temporary "expecting a delivery" toggle that auto-unlocks the lobby gate for the next ring and then turns itself off.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="kubernetes-deployment"&gt;Kubernetes Deployment&lt;/h2&gt;
&lt;p&gt;True to the rest of my home infrastructure, &lt;code&gt;iplusliving2mqtt&lt;/code&gt; is packaged as a minimal multi-stage container (&lt;code&gt;gcr.io/distroless/cc-debian13:nonroot&lt;/code&gt;, running as UID &lt;code&gt;65532&lt;/code&gt; with a read-only root filesystem and all Linux capabilities dropped) and deployed via a Helm chart with &lt;code&gt;/healthz&lt;/code&gt; and &lt;code&gt;/readyz&lt;/code&gt; probes that verify both MQTT broker connectivity and active SIP registration state.&lt;/p&gt;</content>
  </entry>
  <entry>
    <title>Fixing Entra ID / Azure AD AADSTS75011 on Ceph Dashboard SAML</title>
    <link href="https://dj.vg/posts/fixing-ceph-dashboard-saml-aadsts75011/" rel="alternate" type="text/html"/>
    <link href="https://dj.vg/posts/fixing-ceph-dashboard-saml-aadsts75011/index.md" rel="alternate" type="text/markdown"/>
    <id>https://dj.vg/posts/fixing-ceph-dashboard-saml-aadsts75011/</id>
    <published>2026-10-17T00:00:00Z</published>
    <updated>2026-10-17T00:00:00Z</updated>
    <summary>How to resolve Microsoft Entra ID error AADSTS75011 when authenticating to the Ceph Dashboard with FIDO2 security keys or MFA by disabling requestedAuthnContext in the Ceph monitor store.</summary>
    <content type="html">&lt;p&gt;While configuring SAML 2.0 Single Sign-On (SSO) for the &lt;a href="https://ceph.io/"&gt;Ceph&lt;/a&gt; Dashboard against Microsoft Entra ID (formerly Azure AD), standard password authentication works out of the box. However, the moment you authenticate with modern, phishing-resistant credentials—such as FIDO2 security keys, Windows Hello, or passwordless MFA—Entra ID rejects the login with &lt;code&gt;AADSTS75011&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;AADSTS75011: Authentication method 'MultiFactor, Fido' by which the user 
authenticated with the service doesn't match requested authentication method 
'Password, ProtectedTransport'. Contact the Ceph Dashboard application owner.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;At first glance, the error seems backwards: authenticating with a FIDO2 hardware key and MFA is strictly more secure than a password, yet the Identity Provider blocks the session.&lt;/p&gt;
&lt;p&gt;Here is why this happens in Ceph and how to fix it cleanly in the Ceph monitor configuration store without patching Python files inside the &lt;code&gt;ceph-mgr&lt;/code&gt; container.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="why-it-happens-python-saml-and-requestedauthncontext"&gt;Why It Happens: &lt;code&gt;python-saml&lt;/code&gt; and &lt;code&gt;RequestedAuthnContext&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;Under the hood, the Ceph Dashboard relies on OneLogin's &lt;a href="https://github.com/SAML-Toolkits/python-saml"&gt;&lt;code&gt;python-saml&lt;/code&gt;&lt;/a&gt; library to act as a SAML Service Provider.&lt;/p&gt;
&lt;p&gt;By default, &lt;code&gt;python-saml&lt;/code&gt; initializes its security configuration with:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="s1"&gt;'requestedAuthnContext'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Unless overridden, every outbound &lt;code&gt;AuthnRequest&lt;/code&gt; generated by the Ceph Dashboard includes an explicit &lt;code&gt;&amp;lt;samlp:RequestedAuthnContext&amp;gt;&lt;/code&gt; block demanding &lt;code&gt;PasswordProtectedTransport&lt;/code&gt; with &lt;code&gt;Comparison="exact"&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;samlp:RequestedAuthnContext&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;Comparison=&lt;/span&gt;&lt;span class="s"&gt;"exact"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;lt;saml:AuthnContextClassRef&amp;gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;lt;/saml:AuthnContextClassRef&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/samlp:RequestedAuthnContext&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Because &lt;code&gt;Comparison="exact"&lt;/code&gt; is enforced, Entra ID checks the exact authentication method used to satisfy the session. When a user signs in with a passkey/FIDO2 token or MFA, Entra ID records the session context as &lt;code&gt;MultiFactor, Fido&lt;/code&gt;. Since that does not literally match &lt;code&gt;Password, ProtectedTransport&lt;/code&gt;, Entra ID aborts the SAML flow with &lt;code&gt;AADSTS75011&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;To allow FIDO2, passkeys, and MFA, Ceph needs to omit &lt;code&gt;&amp;lt;samlp:RequestedAuthnContext&amp;gt;&lt;/code&gt; from its &lt;code&gt;AuthnRequest&lt;/code&gt; and let Entra ID's Conditional Access policies govern acceptable authentication methods.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="the-fix-updating-ssodb_v1-in-the-monitor-store"&gt;The Fix: Updating &lt;code&gt;ssodb_v1&lt;/code&gt; in the Monitor Store&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;ceph dashboard sso setup saml2&lt;/code&gt; CLI command does not expose a flag to toggle &lt;code&gt;requestedAuthnContext&lt;/code&gt;, and inspecting &lt;code&gt;ceph dashboard sso show saml2&lt;/code&gt; will show it enabled by default:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;ceph&lt;span class="w"&gt; &lt;/span&gt;dashboard&lt;span class="w"&gt; &lt;/span&gt;sso&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt; &lt;/span&gt;saml2&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'.onelogin_settings.security.requestedAuthnContext'&lt;/span&gt;
&lt;span class="nb"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Instead of modifying &lt;code&gt;sso.py&lt;/code&gt; on the container filesystem—which would be wiped on the next container restart or Ceph upgrade—we can patch the serialized SSO database (&lt;code&gt;mgr/dashboard/ssodb_v1&lt;/code&gt;) directly in the Ceph monitor &lt;code&gt;config-key&lt;/code&gt; store.&lt;/p&gt;
&lt;h3 id="1-export-the-current-sso-configuration"&gt;1. Export the Current SSO Configuration&lt;/h3&gt;
&lt;p&gt;Dump the raw JSON configuration from the monitor store:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ceph&lt;span class="w"&gt; &lt;/span&gt;config-key&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;mgr/dashboard/ssodb_v1&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;ssodb.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="2-disable-requestedauthncontext-under-saml2"&gt;2. Disable &lt;code&gt;requestedAuthnContext&lt;/code&gt; Under &lt;code&gt;.saml2&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;Ceph stores SAML 2.0 settings under &lt;code&gt;.saml2.onelogin_settings&lt;/code&gt;. Use &lt;code&gt;jq&lt;/code&gt; to set both &lt;code&gt;requestedAuthnContext&lt;/code&gt; and &lt;code&gt;failOnAuthnContextMismatch&lt;/code&gt; to &lt;code&gt;false&lt;/code&gt; inside &lt;code&gt;.saml2.onelogin_settings.security&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;jq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'.saml2.onelogin_settings.security += {&lt;/span&gt;
&lt;span class="s1"&gt;  "requestedAuthnContext": false,&lt;/span&gt;
&lt;span class="s1"&gt;  "failOnAuthnContextMismatch": false&lt;/span&gt;
&lt;span class="s1"&gt;}'&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ssodb.json&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;ssodb_updated.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Verify the updated security block before applying it:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'.saml2.onelogin_settings.security'&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ssodb_updated.json
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"nameIdEncrypted"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"authnRequestsSigned"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"logoutRequestSigned"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"logoutResponseSigned"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"signMetadata"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"wantMessagesSigned"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"wantAssertionsSigned"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"wantAssertionsEncrypted"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"wantNameIdEncrypted"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"metadataValidUntil"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"wantAttributeStatement"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"requestedAuthnContext"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"failOnAuthnContextMismatch"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="3-write-the-configuration-back-and-reload-the-dashboard"&gt;3. Write the Configuration Back and Reload the Dashboard&lt;/h3&gt;
&lt;p&gt;Write the updated JSON back to &lt;code&gt;mgr/dashboard/ssodb_v1&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ceph&lt;span class="w"&gt; &lt;/span&gt;config-key&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;mgr/dashboard/ssodb_v1&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;ssodb_updated.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Reload the &lt;code&gt;dashboard&lt;/code&gt; manager module so &lt;code&gt;ceph-mgr&lt;/code&gt; picks up the new settings from the monitor store:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ceph&lt;span class="w"&gt; &lt;/span&gt;mgr&lt;span class="w"&gt; &lt;/span&gt;module&lt;span class="w"&gt; &lt;/span&gt;disable&lt;span class="w"&gt; &lt;/span&gt;dashboard
ceph&lt;span class="w"&gt; &lt;/span&gt;mgr&lt;span class="w"&gt; &lt;/span&gt;module&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;dashboard
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Confirm that the active SAML configuration now reflects &lt;code&gt;false&lt;/code&gt;:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;ceph&lt;span class="w"&gt; &lt;/span&gt;dashboard&lt;span class="w"&gt; &lt;/span&gt;sso&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt; &lt;/span&gt;saml2&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'.onelogin_settings.security.requestedAuthnContext'&lt;/span&gt;
&lt;span class="nb"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr /&gt;
&lt;h2 id="persistence-and-operational-notes"&gt;Persistence and Operational Notes&lt;/h2&gt;
&lt;p&gt;With &lt;code&gt;requestedAuthnContext&lt;/code&gt; disabled, SP-initiated logins to the Ceph Dashboard omit the &lt;code&gt;&amp;lt;samlp:RequestedAuthnContext&amp;gt;&lt;/code&gt; constraint, allowing Entra ID to complete authentication using FIDO2 security keys, Windows Hello, or MFA without triggering &lt;code&gt;AADSTS75011&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Because the configuration lives in the Ceph monitor database (&lt;code&gt;config-key&lt;/code&gt;), it persists across &lt;code&gt;ceph-mgr&lt;/code&gt; failovers, container restarts, and node reboots. The only caveat to keep in mind is that running &lt;code&gt;ceph dashboard sso setup saml2&lt;/code&gt; in the future (for instance, when rotating IdP signing certificates) will regenerate the &lt;code&gt;.saml2&lt;/code&gt; dictionary and reset &lt;code&gt;requestedAuthnContext&lt;/code&gt; to &lt;code&gt;true&lt;/code&gt;—so keep this &lt;code&gt;jq&lt;/code&gt; patch in your cluster runbook or configuration management whenever SAML metadata is re-imported.&lt;/p&gt;</content>
  </entry>
  <entry>
    <title>Why I Built BCRSTracking: Telemetry for Singapore's Bottle Return Scheme</title>
    <link href="https://dj.vg/posts/why-i-built-bcrs-tracking/" rel="alternate" type="text/html"/>
    <link href="https://dj.vg/posts/why-i-built-bcrs-tracking/index.md" rel="alternate" type="text/markdown"/>
    <id>https://dj.vg/posts/why-i-built-bcrs-tracking/</id>
    <published>2026-09-12T00:00:00Z</published>
    <updated>2026-09-12T00:00:00Z</updated>
    <summary>Tired of walking to full reverse vending machines with bags of recyclables, I built a Go and ClickHouse service to monitor Singapore's nationwide container return scheme.</summary>
    <content type="html">&lt;p&gt;Few things in modern municipal living are as quietly frustrating as carrying two giant tote bags of rinsed plastic bottles and aluminum cans through Singapore's midday heat to your neighborhood supermarket, only to be greeted by a glowing red screen:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Machine Status: Bin Full. Please try again later.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Singapore’s Beverage Container Return Scheme (BCRS) is a fantastic sustainability initiative. Across the island, Reverse Vending Machines (RVMs) accept beverage containers and return deposits. But like any physical infrastructure distributed across hundreds of residential hubs, MRT stations, and supermarkets, real-world machines fill up, experience mechanical jams, or undergo routine maintenance.&lt;/p&gt;
&lt;p&gt;As a Site Reliability Engineer, I don't like guessing whether a distributed system is healthy before interacting with it. If a network node has a status, it should have telemetry. &lt;/p&gt;
&lt;p&gt;So I built &lt;a href="https://l.dj.vg/bcrs"&gt;&lt;strong&gt;BCRSTracking&lt;/strong&gt;&lt;/a&gt;.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="the-upstream-api"&gt;The Upstream API&lt;/h2&gt;
&lt;p&gt;When inspecting how the official tracking system works (&lt;code&gt;bts.bcrs.sg&lt;/code&gt;), it turns out the platform exposes public API endpoints under &lt;code&gt;/forapi/v2/&lt;/code&gt;. These endpoints provide:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Location Listings&lt;/strong&gt; (&lt;code&gt;/locations&lt;/code&gt;): A directory of all deployed RVM stations across the island, including operating hours, physical coordinates, and machine identifiers.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Bin Status &amp;amp; Telemetry&lt;/strong&gt; (&lt;code&gt;/bin-status&lt;/code&gt;): Real-time machine operational status, current bin capacity percentage, and hardware availability.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;While the data was publicly accessible, there was no historical trend tracking, no differential event stream, and no way to query capacity curves over time to see when specific machines are emptied or filled.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2 id="engineering-requirements"&gt;Engineering Requirements&lt;/h2&gt;
&lt;p&gt;When architecting the tracker, I wanted it to be lightweight, resilient, and polite:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Polite Upstream Concurrency&lt;/strong&gt;: Never overwhelm public services. Polling must use strict token-bucket rate limiting (&lt;code&gt;golang.org/x/time/rate&lt;/code&gt;), bounded worker pools, and exponential backoff with jitter on HTTP 429 or 5xx responses.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Differential State Engine&lt;/strong&gt;: Instead of blindly dumping snapshots, compute state transitions in memory (&lt;code&gt;MACHINE_ADDED&lt;/code&gt;, &lt;code&gt;STATUS_CHANGED&lt;/code&gt;, &lt;code&gt;CAPACITY_THRESHOLD_CROSSED&lt;/code&gt;) so downstream systems can alert on state changes.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Fast Columnar Analytics&lt;/strong&gt;: Store telemetry snapshots in &lt;a href="https://clickhouse.com/"&gt;ClickHouse&lt;/a&gt; using compressed columnar engines (&lt;code&gt;ReplacingMergeTree&lt;/code&gt; for current state, &lt;code&gt;MergeTree&lt;/code&gt; with TTL for historical bin levels).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Native Observability&lt;/strong&gt;: Expose Prometheus metrics for scrapers, queue depth, upstream latency, and error rates, scraped by VictoriaMetrics.&lt;/li&gt;
&lt;/ul&gt;
&lt;hr /&gt;
&lt;h2 id="system-architecture"&gt;System Architecture&lt;/h2&gt;
&lt;p&gt;The resulting service is written in &lt;strong&gt;Go 1.27&lt;/strong&gt; and runs as a lightweight container in Kubernetes:&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;               ┌───────────────────────────────────────────────┐
               │           Upstream BCRS Platform              │
               │            (bts.bcrs.sg/forapi/v2)            │
               └──────────────────────┬────────────────────────┘
                                      │
                                      │ Rate-Limited HTTP/2 (Token Bucket)
                                      │ Exponential Jittered Backoff
                                      ▼
               ┌───────────────────────────────────────────────┐
               │               bcrs-tracker                    │
               │                                               │
               │  ┌─────────────────────────────────────────┐  │
               │  │      Location Poller &amp;amp; Diff Engine      │  │
               │  └────────────────────┬────────────────────┘  │
               │                       │ Active RVM IDs        │
               │                       ▼                       │
               │  ┌─────────────────────────────────────────┐  │
               │  │   Bin Worker Pool (Bounded Concurrency) │  │
               │  └────────────────────┬────────────────────┘  │
               │                       │                       │
               │                       ▼                       │
               │  ┌─────────────────────────────────────────┐  │
               │  │    Batch Writer (500 rows / 5s flush)   │  │
               │  └────────────────────┬────────────────────┘  │
               │                       │                       │
               └───────────────────────┼───────────────────────┘
                                       │ Native Batch Insert (LZ4)
                                       ▼
               ┌───────────────────────────────────────────────┐
               │             ClickHouse Database               │
               │                                               │
               │   • rvm_current        (ReplacingMergeTree)   │
               │   • rvm_events         (MergeTree)            │
               │   • rvm_bin_snapshots  (MergeTree + 2yr TTL)  │
               └───────────────────────────────────────────────┘
                                       ▲
                                       │ SQL / PromQL
                                       │
               ┌───────────────────────┴───────────────────────┐
               │      VictoriaMetrics &amp;amp; Grafana Dashboards     │
               └───────────────────────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr /&gt;
&lt;h2 id="what-the-telemetry-reveals"&gt;What the Telemetry Reveals&lt;/h2&gt;
&lt;p&gt;Having continuous time-series telemetry on reverse vending machines across Singapore reveals fascinating patterns:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;The Weekend Surge&lt;/strong&gt;: Neighborhood machines near residential housing estates typically reach 80–100% capacity on Saturday and Sunday between 2:00 PM and 6:00 PM.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Clearance Cadence&lt;/strong&gt;: Logistics crews typically clear bins during early morning runs between 7:00 AM and 9:30 AM on weekdays.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Pre-Trip Confirmation&lt;/strong&gt;: Instead of walking down hoping for the best, I can query my nearest machine via Prometheus/Grafana or a quick API curl to confirm it is operational and has available bin capacity.&lt;/li&gt;
&lt;/ol&gt;
&lt;hr /&gt;
&lt;h2 id="open-source"&gt;Open Source&lt;/h2&gt;
&lt;p&gt;The entire tracker—including the Go poller, ClickHouse migrations, and Helm charts—is open-source on GitHub:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Repository&lt;/strong&gt;: &lt;a href="https://github.com/TheDJVG/BCRSTracking"&gt;github.com/TheDJVG/BCRSTracking&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you're in Singapore and interested in tracking local smart-city infrastructure, feel free to check out the repository, submit PRs, or send me feedback at &lt;a href="mailto:me@dj.vg"&gt;me@dj.vg&lt;/a&gt;.&lt;/p&gt;</content>
  </entry>
  <entry>
    <title>The 'Datacenter at Home': A Multi-Homed BGP &amp; WireGuard Mesh Architecture</title>
    <link href="https://dj.vg/posts/datacenter-at-home-bgp-wireguard-saltstack/" rel="alternate" type="text/html"/>
    <link href="https://dj.vg/posts/datacenter-at-home-bgp-wireguard-saltstack/index.md" rel="alternate" type="text/markdown"/>
    <id>https://dj.vg/posts/datacenter-at-home-bgp-wireguard-saltstack/</id>
    <published>2026-01-31T00:00:00Z</published>
    <updated>2026-01-31T00:00:00Z</updated>
    <summary>How to apply hyperscale datacenter design principles to a home network using eBGP, WireGuard, and SaltStack orchestration.</summary>
    <content type="html">&lt;p&gt;For the past five years, my home network hasn't just been a gateway to the internet; it's been a miniature ISP backbone. While most residential setups rely on a single ISP router doing basic NAT, I wanted something more robust: carrier-grade reliability, deterministic traffic engineering, and seamless multi-homing using my own public Autonomous System Number (&lt;strong&gt;AS209792&lt;/strong&gt;).&lt;/p&gt;
&lt;p&gt;By applying hyperscale datacenter design principles—like &lt;strong&gt;eBGP-on-the-host&lt;/strong&gt; and &lt;strong&gt;Infrastructure as Code&lt;/strong&gt;—I’ve built a network that treats residential ISPs as mere transport utilities.&lt;/p&gt;
&lt;h2 id="the-high-level-architecture"&gt;The High-Level Architecture&lt;/h2&gt;
&lt;p&gt;The core of the network is built on redundancy. I am connected to two primary fiber ISPs in Singapore, &lt;strong&gt;MyRepublic&lt;/strong&gt; and &lt;strong&gt;ViewQwest&lt;/strong&gt;, along with a &lt;strong&gt;5G backup&lt;/strong&gt; link. Instead of relying on these ISPs for routing, I use them as "underlay" transport to reach my "Upstream Edge Cloud"—a fleet of six VPS nodes (edge1.sin through edge6.sin) hosted across providers.&lt;/p&gt;
&lt;p&gt;The edge nodes are the true gateways to the internet. They announce my public prefixes to the global BGP table, while my home Core Router manages the paths between them.&lt;/p&gt;
&lt;h2 id="the-connectivity-fabric-wireguard-and-deterministic-addressing"&gt;The Connectivity Fabric: WireGuard and Deterministic Addressing&lt;/h2&gt;
&lt;p&gt;To link my Core Router to the cloud, I use a mesh of &lt;strong&gt;WireGuard&lt;/strong&gt; tunnels. Rather than managing a messy internal IPAM (IP Address Management) database, I use a &lt;strong&gt;deterministic addressing logic&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Every tunnel interface is assigned an IPv6 address and a specific UDP listen port generated procedurally based on the IDs of the two nodes. This ensures that the same two peers always get the same IP pair and port, making firewalling and BGP peering definitions completely predictable across the entire mesh. To simplify things further, I utilize &lt;strong&gt;RFC 5549&lt;/strong&gt;, which allows me to carry IPv4 routing information over an IPv6 next-hop, eliminating the need for private IPv4 point-to-point links.&lt;/p&gt;
&lt;h2 id="design-choice-ebgp-everywhere"&gt;Design Choice: eBGP Everywhere&lt;/h2&gt;
&lt;p&gt;A common question in BGP design is whether to use internal BGP (iBGP) or external BGP (eBGP). While iBGP is standard for single-ASN networks, it introduces complexity with split-horizon rules and requires Route Reflectors.&lt;/p&gt;
&lt;p&gt;I chose the &lt;strong&gt;"eBGP in the Datacenter"&lt;/strong&gt; approach. Every node in my network—the Core Router, the VPS edges, the pfSense firewall, and the Kubernetes nodes—operates under its own &lt;strong&gt;unique private ASN&lt;/strong&gt; (645xx range).&lt;/p&gt;
&lt;h3 id="why-this-works"&gt;Why this works:&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Loop Prevention:&lt;/strong&gt; BGP’s &lt;code&gt;AS_PATH&lt;/code&gt; attribute naturally prevents loops. If a node sees its own ASN in a path, it drops the route. &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Observability:&lt;/strong&gt; I can look at my routing table and instantly see the path a packet took by looking at the sequence of ASNs.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A Clean Public Face:&lt;/strong&gt; To the internet, I am a single entity. My edge nodes use BIRD (Internet Routing Daemon) with a strict export filter that "scrubs" these internal private ASNs, ensuring the world only sees a clean, single-hop path originating from &lt;strong&gt;AS209792&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="the-brain-saltstack-the-salt-mine"&gt;The Brain: SaltStack &amp;amp; The Salt Mine&lt;/h2&gt;
&lt;p&gt;Orchestrating this manually would be impossible. I use &lt;strong&gt;SaltStack&lt;/strong&gt; as the network’s "Control Plane." The secret sauce is the &lt;strong&gt;Salt Mine&lt;/strong&gt;, a secure metadata store on the Salt Master.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Metadata Exchange:&lt;/strong&gt; When a node joins, it generates its own WireGuard keys locally and pushes them, along with its current ISP IP, into the Salt Mine.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automated Handshakes:&lt;/strong&gt; All other nodes query the Mine and automatically "wire" themselves to the new peer.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Full System State:&lt;/strong&gt; Salt doesn't just configure BGP. It manages the entire stack:&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;systemd-networkd:&lt;/strong&gt; Generates &lt;code&gt;.netdev&lt;/code&gt; and &lt;code&gt;.network&lt;/code&gt; files for the tunnels.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Firewalling:&lt;/strong&gt; Automatically opens the procedurally generated UDP ports only for authorized peers.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;BIRD:&lt;/strong&gt; Templates the BGP sessions and filters.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Kubernetes:&lt;/strong&gt; Peering pfSense with the K8s nodes so that Pod/Service IPs are reachable across the entire network without NAT.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="reliability-and-convergence"&gt;Reliability and Convergence&lt;/h2&gt;
&lt;p&gt;With residential ISPs, "silent failures" (where the link is up but traffic is blackholed) are a reality. To counter this, every BGP session runs &lt;strong&gt;BFD (Bidirectional Forwarding Detection)&lt;/strong&gt;. If a link drops even a few packets, BFD detects the failure in milliseconds and BGP reroutes traffic to a healthy ISP.&lt;/p&gt;
&lt;p&gt;As a final failover, the &lt;strong&gt;5G link&lt;/strong&gt; is configured with heavy AS-path prepending. It stays in standby, receiving no traffic, until every other fiber path is exhausted.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;This setup has been rock-solid for over five years. By moving the "intelligence" into SaltStack and using BGP for what it does best—path selection—I've created a home network that behaves like a professional datacenter fabric. It’s stateless, modular, and can scale to a new cloud provider with a single &lt;code&gt;state.apply&lt;/code&gt;.&lt;/p&gt;</content>
  </entry>
  <entry>
    <title>Easy and free Dynamic DNS</title>
    <link href="https://dj.vg/posts/free-dynamic-dns/" rel="alternate" type="text/html"/>
    <link href="https://dj.vg/posts/free-dynamic-dns/index.md" rel="alternate" type="text/markdown"/>
    <id>https://dj.vg/posts/free-dynamic-dns/</id>
    <published>2025-10-04T00:00:00Z</published>
    <updated>2025-10-04T00:00:00Z</updated>
    <summary>A lightweight, stateless Dynamic DNS service built with FastAPI.</summary>
    <content type="html">&lt;h2 id="the-nasty-need-for-dynamic-dns"&gt;The nasty need for dynamic DNS.&lt;/h2&gt;
&lt;p&gt;For a while now I've been maintaining a couple of 4G/5G enabled out of band devices. These devices are used to provide access to serial consoles of network devices. Why you ask? Well, sometimes a configuration error or something else causes the normal access paths to be unavailable. Who hasn't made a L2 loop by accident...&lt;/p&gt;
&lt;p&gt;I call these devices "last resort". They provide a last resort access to my network when everything else fails, and they use 4G/5G for connectivity. The downside? They don't have a fixed IP address, quite often the IPs change. For most cases this is fine because I have reverse tunnels, but I still want to access them in case those fail. But for that to work I need to know 1 crucial thing: its IP address on the internet.&lt;/p&gt;
&lt;p&gt;I'm sure many people have used a service like "dyn.com" to solve this problem, the device in question sends a HTTP request (we wish those devices support HTTPS...) and a DNS record gets updated with the IP of the requesting device. That way every time the IP changes the DNS record gets updated and we can access it using the DNS name and it makes it easy to monitor its availibility. The main problem I had with this solution is the cost (also the annoying ratelimiting, but I partially blame the devices in question), it was not worth the money to use "dyn.com" for something I can make myself, right?&lt;/p&gt;
&lt;h2 id="so-i-made-a-service-myself"&gt;So I made a service myself.&lt;/h2&gt;
&lt;p&gt;When I started to think about this had the following requirements:
- Dead simple to request a new dynamic hostname
- No need for accounts
- Compatibility with DynDNS style updates (&lt;code&gt;/nic/update&lt;/code&gt;) (more to add?)&lt;/p&gt;
&lt;p&gt;When I had the requirements clear I also wanted to make it stateless, who wants to keep track of what hostnames are in use and not? If you lose the credentials, just get a fresh hostname. I quickly wrote an implementation using &lt;a href="https://fastapi.tiangolo.com/"&gt;FastAPI&lt;/a&gt; and looked in my unused domain invenstory for a good candidate.&lt;/p&gt;
&lt;h2 id="i-present-to-you-xyzfrl"&gt;I present to you: &lt;a href="https://l.dj.vg/xyz.frl"&gt;XYZ.FRL&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Now the initial version of this service is deployed I really like it myself, mainly because how simple it is. You want to know how simple? Open your terminal and come along.&lt;/p&gt;
&lt;h3 id="request-a-new-dynamic-hostname"&gt;Request a new dynamic hostname&lt;/h3&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;›&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;https://xyz.frl/generate&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;.
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"hostname"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"4bd36fd6538b45b3b5449e615afc8e21.xyz.frl"&lt;/span&gt;,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"username"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"4bd36fd6538b45b3b5449e615afc8e21"&lt;/span&gt;,
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;"password"&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"+MihUvQ8pc1QwYA6/16A/D/155qHfJKlUqUhNgD6osE="&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="update-the-hostname-with-a-new-ip"&gt;Update the hostname with a new IP&lt;/h3&gt;
&lt;p&gt;Simple use &lt;code&gt;curl&lt;/code&gt; to send an update request and set the ip to the valuse set in the &lt;code&gt;myip&lt;/code&gt; query param (you can also leave it out to use the IP the request is coming from).&lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;›&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'4bd36fd6538b45b3b5449e615afc8e21:+MihUvQ8pc1QwYA6/16A/D/155qHfJKlUqUhNgD6osE='&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'http://xyz.frl/nic/update?myip=192.0.2.0,100::'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="check-your-new-ip"&gt;Check your new IP!&lt;/h3&gt;
&lt;p&gt;It might take a brief moment for changes to propagate &lt;/p&gt;
&lt;div class="codehilite"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;›&lt;span class="w"&gt; &lt;/span&gt;host&lt;span class="w"&gt; &lt;/span&gt;4bd36fd6538b45b3b5449e615afc8e21.xyz.frl
4bd36fd6538b45b3b5449e615afc8e21.xyz.frl&lt;span class="w"&gt; &lt;/span&gt;has&lt;span class="w"&gt; &lt;/span&gt;address&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.0
4bd36fd6538b45b3b5449e615afc8e21.xyz.frl&lt;span class="w"&gt; &lt;/span&gt;has&lt;span class="w"&gt; &lt;/span&gt;IPv6&lt;span class="w"&gt; &lt;/span&gt;address&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;::
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="profit"&gt;Profit.&lt;/h3&gt;
&lt;p&gt;From now simply use your compatible client (or curl) with our without the &lt;code&gt;myip&lt;/code&gt; query param to update the IP address in the record. Make your own life easy by simply CNAMEing to this hostname and you should be good to go.&lt;/p&gt;
&lt;h2 id="final-thoughts"&gt;Final thoughts&lt;/h2&gt;
&lt;p&gt;This was a small fun project that I threw together on a Saturday morning when I got a renewal notice (and now cancellation) for DynDNS. I hope it can benefit others too.&lt;/p&gt;
&lt;p&gt;Do you have any ideas to make this service even better? Contact me on me@dj.vg.&lt;/p&gt;
&lt;h3 id="security"&gt;Security&lt;/h3&gt;
&lt;p&gt;While I've applied pretty aggressive rate limiting (1 request per minute on the generate and update endpoints) it's of course possible people might try to abuse this, if that happens I'll deal with it.&lt;/p&gt;</content>
  </entry>
  <entry>
    <title>Timelapse system: a start</title>
    <link href="https://dj.vg/posts/timelapse-system-a-start/" rel="alternate" type="text/html"/>
    <link href="https://dj.vg/posts/timelapse-system-a-start/index.md" rel="alternate" type="text/markdown"/>
    <id>https://dj.vg/posts/timelapse-system-a-start/</id>
    <published>2022-01-25T00:00:00Z</published>
    <updated>2022-01-25T00:00:00Z</updated>
    <summary>How I built a system to take a photo every minute, eventually storing over 750k photos and 6.2TB of data.</summary>
    <content type="html">&lt;p&gt;Everyone that’s interested in technology will at some point face the same struggle: how do I apply all these interesting technologies in my own projects.&lt;/p&gt;
&lt;p&gt;This is really the start of a system that (at the moment of writing) stores over 750 thousand photos occupying 6.2 terabyte of storage and every single minute this grows.&lt;/p&gt;
&lt;p&gt;In the following comparison you can see a picture taken just after I started this project and one that has been taken on the day of publishing this piece.&lt;/p&gt;
&lt;div class="image-comparison-grid"&gt;
    &lt;figure&gt;
        &lt;img src="/images/timelapse/first-scaled.jpg" alt="First Timelapse Image" loading="lazy" decoding="async"&gt;
        &lt;figcaption&gt;Just after starting the project&lt;/figcaption&gt;
    &lt;/figure&gt;
    &lt;figure&gt;
        &lt;img src="/images/timelapse/last-scaled.jpg" alt="Latest Timelapse Image" loading="lazy" decoding="async"&gt;
        &lt;figcaption&gt;Day of publishing&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;

&lt;h2 id="times-are-changing"&gt;Times are changing&lt;/h2&gt;
&lt;p&gt;I’m living in an area that’s actively being developed and for some reason I can’t let go of the past. I want to keep track of how the area is changing and feel nostalgic on “how thing once were”. This is why is decided to put a &lt;a href="https://en.wikipedia.org/wiki/Canon_EOS_2000D"&gt;camera&lt;/a&gt; on my balcony, with a &lt;a href="https://www.raspberrypi.org/"&gt;Raspberry Pi&lt;/a&gt; and a &lt;a href="https://en.wikipedia.org/wiki/Cron"&gt;Cronjob&lt;/a&gt; to take a picture every single minute using &lt;a href="http://www.gphoto.org/"&gt;gPhoto2&lt;/a&gt;. The pictures would be saved on a simple &lt;a href="https://en.wikipedia.org/wiki/Network_File_System"&gt;NFS&lt;/a&gt; share backed by 2 times 1TB hard disks in &lt;a href="https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1"&gt;raid 1&lt;/a&gt;. That was it. Anyone working in IT will immediately see a couple of problems with this approach, just to name a couple:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Reliability&lt;/li&gt;
&lt;li&gt;Monitoring&lt;/li&gt;
&lt;li&gt;Observability&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="reliability"&gt;Reliability&lt;/h3&gt;
&lt;p&gt;As I was not actively looking at the pictures it would happen multiple times that the Raspberry Pi was hanging or that for some reason the camera was not responding ending up with thousands of hanging Cronjobs. Also unreliable Wifi from outdoor made NFS very sad. That had to change, it should just work, even if one of the components is not working.&lt;/p&gt;
&lt;h3 id="monitoring"&gt;Monitoring&lt;/h3&gt;
&lt;p&gt;As mentioned in the previous paragraphs I was not aware of these problems. It was just (not) running and that is not great for a project that tries to capture a changing landscape one minute at a time. The moment there’s a problem with the setup I want to know about and preferable it should recover automatically (by for example resetting the power of the camera).&lt;/p&gt;
&lt;h3 id="observability"&gt;Observability&lt;/h3&gt;
&lt;p&gt;Taking a photo is one thing, but there’s so much extra information that I was not aware of. Some of the information is obvious, for example the &lt;a href="https://en.wikipedia.org/wiki/Film_speed#ISO"&gt;ISO&lt;/a&gt; and &lt;a href="https://en.wikipedia.org/wiki/Exposure_(photography)"&gt;exposure&lt;/a&gt;, which is stored in the &lt;a href="https://en.wikipedia.org/wiki/Exif"&gt;Exif&lt;/a&gt; portion of the file. Other information that I wanted to store was: Outside temperature and relative humidity, how much time is spend on the camera, how reliable is this process, how long does it take to write the file to storage, and how much energy does all of this consume!&lt;/p&gt;
&lt;h2 id="how-to-get-there"&gt;How to get there&lt;/h2&gt;
&lt;p&gt;In the blogs after this I will write down exactly how I got to a system that I almost never have to touch. How videos are automatically created daily. How the camera and controller is able to recover from almost any failure, how data is stored and replicated to the other side to the world.&lt;/p&gt;
&lt;p&gt;So if you’re interested in &lt;a href="https://kubernetes.io/"&gt;Kubernetes&lt;/a&gt;, &lt;a href="https://docs.ceph.com/en/pacific/"&gt;Ceph&lt;/a&gt; (and its S3 compatible API &lt;a href="https://docs.ceph.com/en/pacific/radosgw/index.html"&gt;radosgw&lt;/a&gt;), &lt;a href="https://www.djangoproject.com/"&gt;Django&lt;/a&gt;, &lt;a href="https://github.com/VictoriaMetrics/VictoriaMetrics"&gt;VictoriaMetrics&lt;/a&gt; (and &lt;a href="https://openmetrics.io/"&gt;metrics&lt;/a&gt; in general), &lt;a href="https://www.home-assistant.io/"&gt;Home Assistant&lt;/a&gt;, and many more (don’t forget the hardware). Come back soon to see how all of these things are combined to create a system that allows me to take reliable pictures that I plan to store forever!&lt;/p&gt;</content>
  </entry>
</feed>
