§2024-06-02 機器: redisMongo05.yushei.com.tw, 192.168.2.235 1. modify /etc/syslog-ng/syslog-ng.conf 1.01. backup, ``` root@yushei-dtk-server:/home/alexlai# cp -v /etc/syslog-ng/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf.2024-06-02 '/etc/syslog-ng/syslog-ng.conf' -> '/etc/syslog-ng/syslog-ng.conf.2024-06-02' root@yushei-dtk-server:/home/alexlai# cat /etc/syslog-ng/syslog-ng.conf @include "scl.conf" options { keep-hostname(yes); }; # Sources source s_mongo_log { file("/opt/package/mongoDB/log/mongod.log" flags(no-parse)); }; # Destination destination d_mongoDB { syslog("hc4nas02.yushei.net" transport("udp") port(514)); }; # Log Statements log { source(s_mongo_log); # parser { # json-parser(prefix("json.")); # }; destination(d_mongoDB); }; ``` 1.03. /etc/syslog-ng/syslog-ng.conf ``` @version: 3.25 @include "scl.conf" options { keep-hostname(yes); }; # Sources source s_mongo_log { file("/opt/package/mongoDB/log/mongod.log" flags(no-parse)); }; parser p_syslog { syslog-parser(); }; # Destination destination d_mongoDB { syslog("hc4nas02.yushei.net" transport("udp") port(514)); }; destination d_mongodb_replication { mongodb( # using alexlai:alex1765 is not working # uri("mongodb+srv://alexlai:alex1765@ys20220318.yushei.com.tw/syslog_ng_db?replicaSet=ys20220318&authSource=syslog_ng_db&tls=false") uri("mongodb+srv://siteRootAdmin:b23258585@ys20220318.yushei.com.tw/syslog_ng_db?replicaSet=ys20220318&authSource=admin&tls=false") collection("ys20220318Replication") # template("$(format-json --scope rfc5424 --scope nv-pairs)") ); }; log { source(s_mongo_log); parser(p_syslog); destination(d_mongoDB); }; # Send mongod.log from redismongo03.yushei.com.tw to hc4nas02.yushei.net log { source(s_mongo_log); parser(p_syslog); destination(d_mongodb_replication); }; ``` 1.05. test ``` root@yushei-dtk-server:/home/alexlai# syslog-ng -s -f /etc/syslog-ng/syslog-ng.conf root@yushei-dtk-server:/home/alexlai# systemctl restart syslog-ng root@yushei-dtk-server:/home/alexlai# systemctl status syslog-ng ``` - check mongodb ``` $ mongosh "mongodb+srv://siteRootAdmin:b23258585@ys20220318.yushei.com.tw/syslog_ng_db?replicaSet=ys20220318&authSource=admin&tls=false" Current Mongosh Log ID: 665c0b994995e65a2f8db5fa Connecting to: mongodb+srv://@ys20220318.yushei.com.tw/syslog_ng_db?replicaSet=ys20220318&authSource=admin&tls=false&appName=mongosh+2.2.6 Using MongoDB: 5.0.8 Using Mongosh: 2.2.6 For mongosh info see: https://docs.mongodb.com/mongodb-shell/ ------ The server generated these startup warnings when booting 2024-05-09T17:37:42.790+08:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem ------ ys20220318 [primary] syslog_ng_db> show collections ys20220318Replication ys20220318 [primary] syslog_ng_db> db.ys20220318Replication.count() DeprecationWarning: Collection.count() is deprecated. Use countDocuments or estimatedDocumentCount. 29 ys20220318 [primary] syslog_ng_db> db.ys20220318Replication.count() 29 ys20220318 [primary] syslog_ng_db> db.ys20220318Replication.count() 30 <-- see if anything write into it ys20220318 [primary] syslog_ng_db> ``` --- 以下: kept for records only. - 2024-06-01 ``` @version: 3.38 @include "scl.conf" @module afmongodb source s_mongo_log { file("/var/log/mongodb/mongod.log" flags(no-parse)); }; parser p_syslog { syslog-parser(); }; destination d_mongodb { mongodb( # using alexlai:alex1765 is not working # uri("mongodb+srv://alexlai:alex1765@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=syslog_ng_db&tls=false") uri("mongodb+srv://siteRootAdmin:b23258585@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=admin&tls=false") collection("Odroid01Replication") # template("$(format-json --scope rfc5424 --scope nv-pairs)") ); }; log { source(s_mongo_log); parser(p_syslog); destination(d_mongodb); }; ``` > testing with ``` root@n2Bookworm:/home/alexlai# /sbin/syslog-ng -s -f /etc/syslog-ng/syslog-ng.conf root@n2Bookworm:/home/alexlai# systemctl restart syslog-ng root@n2Bookworm:/home/alexlai# systemctl status syslog-ng ● syslog-ng.service - System Logger Daemon Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; preset: enabled) Active: active (running) since Sat 2024-06-01 08:08:49 CST; 6s ago Docs: man:syslog-ng(8) Main PID: 9401 (syslog-ng) Tasks: 12 (limit: 3414) Memory: 3.9M CPU: 164ms CGroup: /system.slice/syslog-ng.service └─9401 /usr/sbin/syslog-ng -F Jun 01 08:08:48 n2Bookworm systemd[1]: Starting syslog-ng.service - System Logger Daemon... Jun 01 08:08:49 n2Bookworm systemd[1]: Started syslog-ng.service - System Logger Daemon. ``` - old ``` @version: 3.27 @include "scl.conf" source s_mongo_log { file("/var/log/mongodb/mongod.log" flags(no-parse)); }; # Destinations destination d_hc4Jammy { syslog("hc4Jammy.yushei.com.tw" transport("udp") port(514)); }; destination d_mongodb { mongodb( database("Syslog-NG") collection("odroid-01-replication") server("mongodb://alexlai:alexlai@inLaneCatch.yushei.com.tw:27017/?replicaSet=odroid01&authSource=admin&tls=false") template("$(format-json --scope rfc5424 --scope nv-pairs)") ); }; parser p_syslog { syslog-parser(); }; filter f_mongodb_replication { match("PRIMARY" value("MESSAGE") flags("icase")) or match("SECONDARY" value("MESSAGE") flags("icase")) or match("RECOVERING" value("MESSAGE") flags("icase")) or match("STARTUP" value("MESSAGE") flags("icase")) or match("ROLLBACK" value("MESSAGE") flags("icase")) or match("RESYNCING" value("MESSAGE") flags("icase")) or match("heartbeat" value("MESSAGE") flags("icase")) or match("election" value("MESSAGE") flags("icase")) or match("sync source" value("MESSAGE") flags("icase")) or match("initial sync" value("MESSAGE") flags("icase")) or match("vote" value("MESSAGE") flags("icase")) or match("Term changed" value("MESSAGE") flags("icase")) or match("cannot connect" value("MESSAGE") flags("icase")) or match("failed to connect" value("MESSAGE") flags("icase")) or match("Stepping down" value("MESSAGE") flags("icase")) or match("replSet" value("MESSAGE") flags("icase")) or match("replication lag" value("MESSAGE") flags("icase")) or match("slow" value("MESSAGE") flags("icase")) or match("error" value("MESSAGE") flags("icase")) or match("exceeded" value("MESSAGE") flags("icase")); }; # Filters # filter f_hc4Jammy { # host("hc4Jammyyushei.com.tw"); # }; # Bind Sources, Destinations, and Filters # Send mongod.log from redismongo03.yushei.com.tw to hc4nas02.yushei.net log { source(s_mongo_log); filter(f_mongodb_replication); # destination(d_hc4Jammy); parser(p_syslog); destination(d_mongodb); }; ``` - new ``` @version: 3.38 @include "scl.conf" source s_mongo_log { file("/var/log/mongodb/mongod.log" flags(no-parse)); }; parser p_syslog { syslog-parser(); }; # Destinations destination d_hc4Jammy { syslog("hc4Jammy.yushei.com.tw" transport("udp") port(514)); }; destination d_mongodb { mongodb( database("Syslog-NG") collection("odroid-01-replication") server("mongodb://alexlai:alexlai@inLaneCatch.yushei.com.tw:27017/?replicaSet=odroid01&authSource=admin&tls=false") template("$(format-json --scope rfc5424 --scope nv-pairs)") ); }; filter f_mongodb_replication { match("PRIMARY" value("MESSAGE") flags("icase")) or match("SECONDARY" value("MESSAGE") flags("icase")) or match("RECOVERING" value("MESSAGE") flags("icase")) or match("STARTUP" value("MESSAGE") flags("icase")) or match("ROLLBACK" value("MESSAGE") flags("icase")) or match("RESYNCING" value("MESSAGE") flags("icase")) or match("heartbeat" value("MESSAGE") flags("icase")) or match("election" value("MESSAGE") flags("icase")) or match("sync source" value("MESSAGE") flags("icase")) or match("initial sync" value("MESSAGE") flags("icase")) or match("vote" value("MESSAGE") flags("icase")) or match("Term changed" value("MESSAGE") flags("icase")) or match("cannot connect" value("MESSAGE") flags("icase")) or match("failed to connect" value("MESSAGE") flags("icase")) or match("Stepping down" value("MESSAGE") flags("icase")) or match("replSet" value("MESSAGE") flags("icase")) or match("replication lag" value("MESSAGE") flags("icase")) or match("slow" value("MESSAGE") flags("icase")) or match("error" value("MESSAGE") flags("icase")) or match("exceeded" value("MESSAGE") flags("icase")); }; # Filters # filter f_hc4Jammy { # host("hc4Jammyyushei.com.tw"); # }; # Bind Sources, Destinations, and Filters # Send mongod.log log { source(s_mongo_log); filter(f_mongodb_replication); # destination(d_hc4Jammy); parser(p_syslog); destination(d_mongodb); }; ``` ``` root@n2Bookworm:/home/alexlai# cat /etc/syslog-ng/syslog-ng.conf @version: 3.38 @include "scl.conf" @module afmongodb source s_mongo_log { file("/var/log/mongodb/mongod.log" flags(no-parse)); }; parser p_syslog { syslog-parser(); }; destination d_mongodb { mongodb( # uri("mongodb+srv://alexlai:alex1765@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=syslog_ng_db&tls=false") uri("mongodb+srv://siteRootAdmin:b23258585@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=admin&tls=false") collection("Odroid01Replication") # template("$(format-json --scope rfc5424 --scope nv-pairs)") ); }; log { source(s_mongo_log); parser(p_syslog); destination(d_mongodb); }; ``` --- GPT, I had a mongodb replicationSet that could be connected with "mongodb+srv://siteRootAdmin:b23258585@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=admin&tls=false". I would like to write a javascript using MongoClient to watch a capped collection of "Odroid01Replication" inside the database "syslog_ng_db" and evertime a document was inserted into this capped collection, it will output with console.log. ``` const { MongoClient } = require('mongodb'); const uri = "mongodb+srv://siteRootAdmin:b23258585@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=admin&tls=false"; const client = new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true }); async function run() { try { await client.connect(); console.log("Connected to the database!"); const db = client.db('syslog_ng_db'); const collection = db.collection('Odroid01Replication'); console.log("Watching for changes in the 'Odroid01Replication' collection..."); // Open a change stream on the collection const changeStream = collection.watch([], { fullDocument: 'updateLookup' }); // Listen for changes in the stream changeStream.on('change', (change) => { if (change.operationType === 'insert') { console.log('New document inserted:'); console.log(change.fullDocument); } }); } catch (err) { console.error(err); } } run().catch(console.dir); ``` - watch for insert ``` const { MongoClient } = require('mongodb'); const uri = "mongodb+srv://siteRootAdmin:b23258585@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=admin&tls=false"; const client = new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true }); async function run() { try { await client.connect(); console.log("Connected to the database!"); const db = client.db('syslog_ng_db'); const collection = db.collection('Odroid01Replication'); console.log("Watching for insertions in the 'Odroid01Replication' collection..."); // Open a change stream on the collection with a filter for insert operations const changeStream = collection.watch( [{ $match: { 'operationType': 'insert' } }], { fullDocument: 'updateLookup' } ); // Listen for changes in the stream changeStream.on('change', (change) => { console.log('New document inserted:'); console.log(change.fullDocument); }); } catch (err) { console.error(err); } } run().catch(console.dir); ``` - out put to web page - server.js ``` const express = require('express'); const http = require('http'); const socketIo = require('socket.io'); const { MongoClient } = require('mongodb'); const uri = "mongodb+srv://siteRootAdmin:b23258585@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=admin&tls=false"; const client = new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true }); const app = express(); const server = http.createServer(app); const io = socketIo(server); const port = process.env.PORT || 48597; app.get('/', (req, res) => { res.sendFile(__dirname + '/index.html'); }); async function run() { try { await client.connect(); console.log("Connected to the database!"); const db = client.db('syslog_ng_db'); const collection = db.collection('Odroid01Replication'); console.log("Watching for insertions in the 'Odroid01Replication' collection..."); // Open a change stream on the collection with a filter for insert operations const changeStream = collection.watch( [{ $match: { 'operationType': 'insert' } }], { fullDocument: 'updateLookup' } ); // Listen for changes in the stream changeStream.on('change', (change) => { console.log('New document inserted:'); console.log(change.fullDocument); io.emit('newDocument', change.fullDocument); }); } catch (err) { console.error(err); } } run().catch(console.dir); server.listen(port, () => { console.log(`Server is running on port ${port}`); }); ``` - index.html ``` MongoDB Insert Monitor

MongoDB Insert Monitor

``` - index.html, limits to 180 lines ``` MongoDB Insert Monitor

MongoDB Insert Monitor

``` - index.html circulate from line 0 ``` MongoDB Insert Monitor

MongoDB Insert Monitor

``` --- To modify the provided index.html file so that it only displays MongoDB documents when 'Election', 'error', and 'slow' events occur, you will need to update the server-side logic to filter the documents based on these criteria and send only the relevant events to the client. Server-Side (Node.js) Update First, update your server.js to filter the relevant events: javascript Copy code const express = require('express'); const http = require('http'); const socketIo = require('socket.io'); const { MongoClient } = require('mongodb'); const uri = "mongodb+srv://siteRootAdmin:b23258585@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=admin&tls=false"; const client = new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true }); const app = express(); const server = http.createServer(app); const io = socketIo(server); const port = process.env.PORT || 3000; app.get('/', (req, res) => { res.sendFile(__dirname + '/index.html'); }); async function run() { try { await client.connect(); console.log("Connected to the database!"); const db = client.db('syslog_ng_db'); const collection = db.collection('Odroid01Replication'); console.log("Watching for relevant events in the 'Odroid01Replication' collection..."); // Open a change stream on the collection with a filter for relevant events const changeStream = collection.watch([ { $match: { $or: [ { 'fullDocument.message': /Election/i }, { 'fullDocument.message': /error/i }, { 'fullDocument.message': /slow/i } ] } } ], { fullDocument: 'updateLookup' }); // Listen for changes in the stream changeStream.on('change', (change) => { console.log('Relevant event detected:'); console.log(change.fullDocument); io.emit('newDocument', change.fullDocument); }); } catch (err) { console.error(err); } } run().catch(console.dir); server.listen(port, () => { console.log(`Server is running on port ${port}`); }); Client-Side (HTML and JavaScript) Update Next, update your index.html to handle the filtered events from the server: html Copy code MongoDB Insert Monitor

MongoDB Insert Monitor

``` --- >resolved: # uri("mongodb+srv://alexlai:alex1765@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=syslog_ng_db&tls=false") failed, but uri("mongodb+srv://siteRootAdmin:b23258585@inLaneCatch.yushei.com.tw/syslog_ng_db?replicaSet=odroid01&authSource=admin&tls=false") works?? ¶ syslog-ng write into a mongoDB capped collection To configure syslog-ng to write log messages into a MongoDB capped collection, you need to follow these general steps: - Install syslog-ng: Make sure syslog-ng is installed on your system. You can typically install it using your system's package manager. - Install MongoDB driver for syslog-ng: You'll need a MongoDB driver for syslog-ng to be able to interact with MongoDB. One popular - option is using the "mongodb" destination available in syslog-ng. - Configure syslog-ng: Configure syslog-ng to read log messages from your sources (e.g., files, network), filter and parse them if needed, and then send them to MongoDB. ¶Configure MongoDB: Set up a capped collection in MongoDB where you want to store the log messages. Here's a basic example configuration for syslog-ng to write log messages into a MongoDB capped collection: ``` @version: 3.28 @include "scl.conf" source s_network { tcp(ip(0.0.0.0) port(514)); }; parser p_syslog { syslog-parser(); }; destination d_mongodb { mongodb( database("your_database_name") collection("your_capped_collection_name") server("mongodb://localhost:27017") template("$(format-json --scope rfc5424 --scope nv-pairs)") ); }; log { source(s_network); parser(p_syslog); destination(d_mongodb); }; ``` - Explanation: - source s_network: This defines a source where syslog-ng will collect log messages from. In this example, it's set to collect messages over TCP on port 514. - parser p_syslog: This defines a parser to parse the incoming log messages. The syslog-parser() function is a built-in parser for syslog messages. - destination d_mongodb: This defines the destination where syslog-ng will send the parsed log messages. It's configured to send messages to MongoDB. You need to specify the database name, the capped collection name, and the MongoDB server URL. - log { ... }: This is the main logging configuration block where you define which source to use, which parser to use, and which destination to send the parsed messages to. - Make sure to replace "your_database_name", "your_capped_collection_name", and "mongodb://localhost:27017" with the appropriate values for your MongoDB setup. ¶ see of module for mongodb is compiled in the ubuntu packages? ``` alexlai@n2Bookworm:~$ /usr/sbin/syslog-ng --version syslog-ng 3 (3.38.1) Config version: 3.35 Installer-Version: 3.38.1 Revision: 3.38.1-5 Compile-Date: Jan 30 2023 18:18:56 Module-Directory: /usr/lib/syslog-ng/3.38 Module-Path: /usr/lib/syslog-ng/3.38 Include-Path: /usr/share/syslog-ng/include Available-Modules: azure-auth-header,mod-python.cpython-311-aarch64-linux-gnu,riemann,secure-logging,afsmtp,xml,affile,afprog,afsocket,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,dbparser,disk-buffer,hook-commands,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pacctformat,pseudofile,rate-limit-filter,regexp-parser,sdjournal,syslogformat,system-source,tags-parser,tfgetent,timestamp,afsql,afmongodb,add-contextual-data,afamqp,examples,geoip2-plugin,graphite,http,kafka,redis,afsnmp,afstomp,stardate Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on ``` - the output of the syslog-ng --version command indicates that syslog-ng has been compiled with MongoDB support (afmongodb module is listed under Available-Modules). This means you can proceed to configure syslog-ng to write logs to a MongoDB database.