/munetakaJupyterHub/Computer/2024/MailServer/Haraka/17-MongoDB_plugin_for

§2024-09-25

MongoDB plugin for Haraka

This plugin will store incoming emails (including bounces, etc.) in MongoDB and store all attachments on the disk. Additionally, it can also be used to store all results for outgoing emails.

As of version 1.6.5, we've added the option to limit incoming emails, also. There are many other options. Please make sure to read this README.

This plugin has been tested with over 500,000 incoming and outgoing emails a day.

Installation

Depending on your operating system, you might first have to install python, make, and compiler, e.g.:

apt install python2 make cmake g++ build-essential --> pyuthon2 not available

The Winmail.dat file is an attachment format used by Microsoft Outlook when it sends emails formatted in Rich Text Format (RTF). This file format encapsulates formatting elements such as text styles and embedded objects that are not natively supported by other email clients.

In order to store winmail.dat files (yes some people are still using those) you need to also install tnef with:

apt install tnef

After you have created the Haraka configuration directory you can cd into the haraka directory and do...

npm install haraka-plugin-mongodb

This will install everything that is needed for you to store incoming emails to MongoDB and also store results from outgoing emails.

Alternatively you can also do a git clone into the Haraka node_modules directory. The installation directory depends if you installed Haraka globally or not.

haraka-plugin-mongodb/config/mongodb.ini

$ haraka -i hc4NobleMailServerMongodb
create: /home/haraka/build/hc4NobleMailServerMongodb
create: /home/haraka/build/hc4NobleMailServerMongodb/plugins
create: /home/haraka/build/hc4NobleMailServerMongodb/docs
create: /home/haraka/build/hc4NobleMailServerMongodb/config
create: /home/haraka/build/hc4NobleMailServerMongodb/config/smtp.ini
create: /home/haraka/build/hc4NobleMailServerMongodb/config/log.ini
create: /home/haraka/build/hc4NobleMailServerMongodb/config/plugins
$ cd hc4NobleMailServerMongodb
$ npm install haraka-plugin-mongodb
$ cat package.json 
{
  "name": "haraka_local",
  "description": "An SMTP Server",
  "version": "0.0.1",
  "dependencies": {
    "haraka-plugin-mongodb": "^1.8.6"
  },
  "repository": "",
  "license": "MIT"
}

config

; This file must be placed in "config" directory of your Haraka server.

;
; MongoDB Credentials
;

[mongodb]
; user
user=
; password
pass=
; host
host=127.0.0.1
; string (full mongodb connection string)
string=
; port
port=27017
; database name
db=myApp
; restart = throw error on insert error
restart=yes

; collection name
[collections]
queue=email_incoming_haraka
delivery=email_delivery_results

; Absolute path to store attachments
[attachments]
path=/opt/attachments
; The below path has to exists or else we stop Haraka from starting. Leave empty if you don't want to execute a check.
; Great if you have to make sure that a value is mounted or keeps being mounted. If we loose connection we exit Haraka.
path_check=
; Attachments that should always be rejected. The values below are the ones that Gmail and Outlook also reject
reject=['application/x-freearc','application/x-bzip','application/x-bzip2','application/x-csh','application/gzip','application/java-archive','text/javascript','application/vnd.apple.installer+xml','application/x-httpd-php','application/x-sh','application/xhtml+xml','application/vnd.microsoft.portable-executable','application/x-msdownload','application/exe','application/x-exe','application/dos-exe','vms/exe','application/x-winexe','application/msdos-windows','application/x-msdos-program']
custom_content_type={ 'application/imed' : ['imed'] }
; You can also reject attachments by filename. Enter the complete filename or only part of it
reject_by_name=['~WRD']

; Convert inline images or not
; cid = leave value as cid:(number) - this is useful if you want to process the images later on
; base64 = convert inline images to base64 - will convert inline images to base64 and remove it from the attachment array
; path = convert the cid:(number) value to the path given, the filename will be appended to your path, e.g., "(path)/image.png".
; Do NOT append a "/" at the end of your path
cid = cid

; Enable the section you want
[enable]
queue=yes
delivery=yes

; Message settings
; MongoDB has a limit of 16 MB per document. Hence you might want to check the size of the message (https://docs.mongodb.com/manual/reference/limits/)
[message]
limit=16777216
; Store the whole raw email and the parsed haraka body (Enabling this will create large documents and should only be used for debugging)
store_raw=no

; SMTP settings for error messages
; Define your custom smtp server settings to send message when a message is to large or there is an error with inserting the message
[smtp]
user=
pass=
host=
port=587
tls=yes
ssl=no
msg_limit=Your message could not be accepted as the message body is over 16 MB. Please remove parts of the email and send again.
msg_error_insert=An error occurred accepting your message. Please send again.
msg_error_parsing=We are not able to parse this message properly. Please make sure this email is RFC body conform.
from=mailer-daemon@domain.com
; CC and BCC can be a comma separated list of email addresses
cc=
bcc=

; Limit settings
; Limit incoming messages. Please see the README for instructions
[limits]
incoming=no

; Limit in seconds
incoming_seconds=30

; Strings that should be excluded from the limits check
exclude[] = helpmonks
exclude[] = sendmonks
exclude[] = razuna

; String that should be included in the limits check
include[] = monitor
include[] = notification
include[] = java
include[] = noreply
include[] = notreply
include[] = no-reply
include[] = not-reply
include[] = deployment
include[] = notifier
include[] = root
include[] = alert
include[] = uptime
include[] = robot
include[] = opsgenie
include[] = arubanetworks

; Set the database to be used. Either "mongodb" or "redis"
db=redis

; Set the collection to be used if you use mongodb
incoming_collection=limit_incoming

; Redis Credentials
; Used for the limit incoming option above
[redis]
; user
user=
; password
pass=
; host
host=127.0.0.1
; string (full redis connection string)
string=
; port
port=6379

test run

haraka@hc4Noble:~/build/hc4NobleMailServerMongodb$ sudo haraka -c ./
loaded TLD files:
  1=1448
  2=8320
  3=3571
loaded 9672 Public Suffixes
loglevel: INFO
log format: DEFAULT
[WARN] [-] [server] smtp.ini.nodes unset, using 1, see https://github.com/haraka/Haraka/wiki/Performance-Tuning
Starting up Haraka version 3.0.4
[INFO] [-] [plugins] loading mail_from.is_resolvable
[INFO] [-] [plugins] loading rcpt_to.in_host_list
[INFO] [-] [plugins] loading queue/smtp_forward
[NOTICE] [-] [server] worker started worker=1 pid=39807
loaded TLD files:
  1=1448
  2=8320
  3=3571
loaded 9672 Public Suffixes
loglevel: INFO
log format: DEFAULT
[WARN] [-] [server] smtp.ini.nodes unset, using 1, see https://github.com/haraka/Haraka/wiki/Performance-Tuning
Starting up Haraka version 3.0.4
[INFO] [-] [plugins] loading mail_from.is_resolvable
[INFO] [-] [plugins] loading rcpt_to.in_host_list
[INFO] [-] [plugins] loading queue/smtp_forward
[INFO] [-] [core] loading tls.ini
[ERROR] [-] [core] tls key /home/haraka/build/hc4NobleMailServerMongodb/config/tls_key.pem could not be loaded.
[ERROR] [-] [core] tls cert /home/haraka/build/hc4NobleMailServerMongodb/config/tls_cert.pem could not be loaded.
[NOTICE] [-] [server] Listening on [::0]:25
[NOTICE] [-] [server] worker 1 listening on [::0]:25
[NOTICE] [428F4177-DFC9-4401-9D9A-FC0B1566D028] [core] connect ip=::1 port=43890 local_ip=::1 local_port=25
[INFO] [428F4177-DFC9-4401-9D9A-FC0B1566D028.1] [mail_from.is_resolvable] pass:has_fwd_dns
[NOTICE] [428F4177-DFC9-4401-9D9A-FC0B1566D028.1] [core] sender <alexlai@yushei.net> code=CONT msg=""
[NOTICE] [428F4177-DFC9-4401-9D9A-FC0B1566D028.1] [core] recipient <alexlai@hcnoble> code=OK msg="" sender=alexlai@yushei.net
[INFO] [428F4177-DFC9-4401-9D9A-FC0B1566D028.1] [core]  hook=rcpt plugin=rcpt_to.in_host_list function=hook_rcpt params=<alexlai@hc4Noble> retval=OK msg=""
[NOTICE] [428F4177-DFC9-4401-9D9A-FC0B1566D028.1] [core] recipient <alexlai@hc4Noble> code=OK msg="" sender=alexlai@yushei.net
[NOTICE] [428F4177-DFC9-4401-9D9A-FC0B1566D028.1] [core] message mid="" size=46 rcpts=1/0/1 delay=0.001 code=CONT msg=""
[INFO] [-] [core] [smtp_client] uuid=5BC12F6A-2D7B-467B-BD72-AA24A7751858 host=localhost port=2555 created
[INFO] [428F4177-DFC9-4401-9D9A-FC0B1566D028.1] [queue/smtp_forward] forwarding to localhost:2555

$ telnet  localhost 25
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 hc4Noble ESMTP Haraka/3.0.4 ready
EHLO yushei.net
250-hc4Noble Hello ip6-localhost [::1], Haraka is at your service.
250-PIPELINING
250-8BITMIME
250-SMTPUTF8
250 SIZE 0
MAIL FROM: alexlai@yushei.net
250 sender <alexlai@yushei.net> OK
RCPT TO: alexlai@hcnoble
550 I cannot deliver mail for <alexlai@hcnoble>
RCPT TO: alexlai@hc4Noble
250 recipient <alexlai@hc4Noble> OK
DATA
354 go ahead, make my day
Subject: 2024-09-25 08:08
This is a test
.
quit
450 plugin timeout (428F4177-DFC9-4401-9D9A-FC0B1566D028.1)
221 hc4Noble closing connection. Have a jolly good day.
Connection closed by foreign host.