Nginx Mail Proxy (`ngx_mail_core_module`)

Cheatsheet#nginx-mail-proxy

Nginx Mail Proxy (`ngx_mail_core_module`)

Nginx includes a high-performance TCP proxy for mail protocols (IMAP, POP3, and SMTP). It offloads SSL/TLS termination, performs authentication routing via an HTTP auth service, and balances incoming mail client connections to backend mail servers (such as Mailcow, Dovecot, or Postfix).

`server_name` Directive (Official Nginx Spec)

Reference: Nginx ngx_mail_core_module Documentation

Specification

Syntax: server_name name;
Default: server_name hostname; (System OS hostname)
Context: mail, server

Official Functions

SMTP: Sets the hostname in the initial 220 greeting banner (220 <server_name> ESMTP ready) and in EHLO responses.
IMAP: Sets the hostname in the initial IMAP server capability banner (* OK IMAP4rev1 <server_name> ready).
POP3: Sets the hostname in the initial POP3 banner (+OK POP3 <server_name> ready).
TLS SNI (v1.21.0+): Matches the incoming TLS Server Name Indication (SNI) extension sent by the mail client to select the corresponding SSL certificate and server block.

Complete `nginx.conf` Mail Block Example

Unlike HTTP proxying, Nginx mail proxying requires top-level mail { ... } blocks and an external HTTP Authentication Service (auth_http).

# Top-level mail context
mail {
    # Official server_name set at the mail block level or per-server
    server_name mail.cloud.id;
 
    # External HTTP authentication script URL
    auth_http http://127.0.0.1:9000/auth;
 
    # Global Mail SSL Settings
    ssl_certificate     /etc/letsencrypt/live/mail.cloud.id/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mail.cloud.id/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    ssl_session_cache   shared:SSLMAIL:10m;
    ssl_session_timeout 1h;
 
    # STARTTLS setting: on | off | starttls (only)
    ssl_starttls on;
 
    # -------------------------------------------------------------
    # SMTP Proxy (Ports 25, 465, 587)
    # -------------------------------------------------------------
    server {
        listen 25;
        listen 587;
        protocol smtp;
        server_name mail.cloud.id;
        
        # SMTP specific Capabilities
        smtp_auth login plain;
        xclient off;
    }
 
    server {
        listen 465 ssl;
        protocol smtp;
        server_name mail.cloud.id;
        
        smtp_auth login plain;
    }
 
    # -------------------------------------------------------------
    # IMAP Proxy (Ports 143, 993)
    # -------------------------------------------------------------
    server {
        listen 143;
        protocol imap;
        server_name mail.cloud.id;
        
        imap_capabilities "IMAP4rev1" "LITERAL+" "SASL-IR" "LOGIN-REFERRALS" "ID" "ENABLE" "IDLE" "STARTTLS" "AUTH=PLAIN";
    }
 
    server {
        listen 993 ssl;
        protocol imap;
        server_name mail.cloud.id;
        
        imap_capabilities "IMAP4rev1" "LITERAL+" "SASL-IR" "LOGIN-REFERRALS" "ID" "ENABLE" "IDLE" "AUTH=PLAIN";
    }
 
    # -------------------------------------------------------------
    # POP3 Proxy (Ports 110, 995)
    # -------------------------------------------------------------
    server {
        listen 110;
        protocol pop3;
        server_name mail.cloud.id;
        
        pop3_capabilities "TOP" "USER" "UIDL" "STLS";
    }
 
    server {
        listen 995 ssl;
        protocol pop3;
        server_name mail.cloud.id;
        
        pop3_capabilities "TOP" "USER" "UIDL";
    }
}

Multiple Domains with TLS SNI (v1.21.0+)

Starting from Nginx version 1.21.0, Nginx supports SNI inside mail server blocks. You can host multiple mail domains with separate SSL certificates on a single IP address:

mail {
    auth_http http://127.0.0.1:9000/auth;
 
    # Server block 1: Primary Domain
    server {
        listen 993 ssl;
        protocol imap;
        server_name mail.primary.com;
 
        ssl_certificate     /etc/letsencrypt/live/mail.primary.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/mail.primary.com/privkey.pem;
    }
 
    # Server block 2: Secondary Domain (SNI Match)
    server {
        listen 993 ssl;
        protocol imap;
        server_name mail.secondary.com;
 
        ssl_certificate     /etc/letsencrypt/live/mail.secondary.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/mail.secondary.com/privkey.pem;
    }
}

HTTP Auth Protocol (`auth_http`) Requirements

Nginx does NOT validate mail passwords directly against a database. It delegates auth to an HTTP API server by sending HTTP headers (Auth-User, Auth-Pass, Auth-Protocol, Auth-Login-Attempt).

Required Response Headers from Auth Server

To grant access and proxy the client connection to the backend mail server, your HTTP auth script MUST return HTTP Status 200 with these custom headers:

HTTP/1.1 200 OK
Auth-Status: OK
Auth-Server: 127.0.0.1
Auth-Port: 143

To reject authentication:

HTTP/1.1 200 OK
Auth-Status: Invalid login or password
Auth-Wait: 3

Testing & Verifying Banners

# Test SMTP Greeting Banner (Check 220 <server_name>)
nc -v 127.0.0.1 25
# Output: 220 mail.cloud.id ESMTP ready
 
# Test IMAP Banner (Check * OK IMAP4rev1 <server_name> ready)
nc -v 127.0.0.1 143
# Output: * OK IMAP4rev1 mail.cloud.id ready
 
# Test POP3 Banner (Check +OK POP3 <server_name> ready)
nc -v 127.0.0.1 110
# Output: +OK POP3 mail.cloud.id ready
 
# Verify TLS SNI Resolution on Mail Port 993
openssl s_client -connect 127.0.0.1:993 -servername mail.cloud.id

Cheatsheets