Skip to content

MongoDB

Here, you find possible solutions to problems that can occur with MongoDB when installing or running PLOSSYS Output Engine.

Reduce Memory Usage of MongoDB

MongoDB uses per default (RAM / 2) - 1 GB memory. On computers with limited RAM, MongoDB cache can be restricted.

Follow these steps to reduce the MongoDB cache size:

  1. On the Output Engine server, open the configuration file of MongoDB in an editor:

    C:\ProgramData\SEAL Systems\config\mongod.conf

  2. Add the following lines to storage to reduce the cache size. Adapt the value as needed.

        wiredTiger:
        engineConfig:
            cacheSizeGB: 2

    Example of mongod.conf

    storage:
    dbPath: /opt/seal/data/seal-mongodb
    wiredTiger:
        engineConfig:
            cacheSizeGB: 2

  3. Additionally, increasing the swap ensures that there is enough buffer. You can do this retroactively via script. Ideally this should have been done during the setup of the machine with an appropriate swap partition. On Linux, the swap should be at least 50% of RAM.

    Example - increasing the swap retroactively for a Red Hat machine

    #!/usr/bin/env bash

# Create 8G swap file
sudo fallocate -l 8G /opt/seal/swapfile
sudo chmod 600 /opt/seal/swapfile
sudo mkswap /opt/seal/swapfile

# Disable old swap, enable new one
sudo swapoff /dev/sysvg/lv_swap
sudo swapon /opt/seal/swapfile

sudo sed -i 's|^\(.*swap.*\)$|#\1\n/opt/seal/swapfile  none  swap  sw  0 0|' /etc/fstab

Reduce Size of MongoDB

If you need to reduce the size of MongoDB, use the compact command of MongoDB. For how to use it, refer to the Manual Reference of MongoDB.

Check Certificate Validity for MongoDB

If your SSL certificate is not valid anymore, MongoDB can crash with a fatal assertion 28652 and cause your complete system to stand still.

If this happens, apply this temporary workaround:

  1. On the Output Engine server, open the configuration file of MongoDB in an editor:

    C:\ProgramData\SEAL Systems\config\mongod.conf

  2. Replace the following line:

    CAFile: C:\ProgramData\SEAL Systems\config\tls\ca.pem

    by this line:

    allowInvalidCertificates: true

  3. Save the configuration file.

  4. Restart MongoDB:

    • Windows: restart-service seal-mongodb
    • Linux: sudo systemctl restart mongod

If the problem has been verified this way, continue with the following:

  1. Check the system time. Is the date set correctly?

  2. Check the validity of the PEM file.

    Hint - how to check PEM files

    You can use the openssl commands to explore the details of a certificate. For example:

    openssl x509 -in mycert.pem -text -noout

    If you do not have openssl installed, you can use a web tool to check the certificate.

  3. If needed, replace the the invalid certificate file with a new PEM file.

  4. Undo the previous temporary changes in the configuration file of MongoDB.
  5. Restart MongoDB.

Other MongoDB TLS Certificate Errors

Check the MongoDB log file for the exact error message:

  • Linux: /var/log/seal/mongod.log
  • Windows: C:\ProgramData\SEAL Systems\log\mongod.log

The following error messages point to these causes:

  • The use of TLS without specifying a chain of trust is no longer supported ...: The root certificate is missing. Refer to Specifying a CA Certificate for Windows or Linux.
  • The use of both a CA File and the System Certificate store is not supported.: CAFile and tlsUseSystemCA are both set in mongod.conf. Remove tlsUseSystemCA from the setParameter section.
  • Can not set up PEM key file.: The file referenced by certificateKeyFile is invalid. It must contain both the certificate and the private key. Refer to TLS Encryption for MongoDB Windows or Linux.

Database Lock Timeout

If an error message containing Unable to acquire lock ... within a max lock request timeout of '5ms' milliseconds. is shown in the seal-controller log, the system is temporarily overloaded and the transaction lock timeout needs to be increased.

Add the following line in the setParameter: section in mongod.conf:

setParameter:
   ...
   maxTransactionLockRequestTimeoutMillis: 30000
   ...

ReplicaSet Not Initialized

If a call to rs.status() returns the message MongoServerError: no replset config has been received, then rs.initiate() was omitted.

Hint - background

A replica set has to be configured in MongoDB and the database initialized. For more information refer to Configuring SEAL MongoDB for Windows or Linux.

This is how you check the ReplicaSet:

  • on Windows:

    & "C:\Program Files\mongosh\mongosh.exe" --tls --tlsAllowInvalidCertificates --eval "rs.status()"

  • on Linux:

    mongosh --tls --tlsAllowInvalidCertificates --eval "rs.status()"

Disk Full on a MongoDB Cluster Member

If a MongoDB cluster member runs out of disk space, restart the cluster in this order to avoid synchronization problems:

  1. Shut down PLOSSYS Output Engine and MongoDB on every server in the cluster.
  2. Increase the disk space.
  3. Start MongoDB on every server in the cluster.

  4. Wait until the following command shows that all servers are synchronized:

    • Windows: & "C:\Program Files\mongosh\mongosh.exe" --tls --tlsAllowInvalidCertificates --eval "rs.status()"
    • Linux: mongosh --tls --tlsAllowInvalidCertificates --eval "rs.status()"

  5. Start PLOSSYS Output Engine on every server in the cluster.

MongoDB Cluster Member Stuck in Recovering

A MongoDB cluster member that permanently shows the status RECOVERING in rs.status() cannot synchronize anymore. This can happen after a long outage of that server.

Follow these steps to force a full resynchronization:

  1. Shut down PLOSSYS Output Engine on every server in the cluster.
  2. Shut down MongoDB on the affected server.

  3. Delete all files in the MongoDB data directory of the affected server:

    • Linux: /opt/seal/data/seal-mongodb
    • Windows: C:\ProgramData\SEAL Systems\data\seal-mongodb

    Caution - keep the directory

    Delete only the files inside the data directory. Do not delete the directory itself.

  4. Start MongoDB on the affected server.

  5. Wait until rs.status() no longer shows RECOVERING for this server. Depending on the amount of data and the network latency between the cluster members, this can take 10 to 15 minutes or longer.
  6. Start PLOSSYS Output Engine on every server in the cluster.

Useful MongoDB Commands

Use these commands to inspect the Output Engine databases in MongoDB.

Open the MongoDB shell:

  • Windows: & "C:\Program Files\mongosh\mongosh.exe" --tls --tlsAllowInvalidCertificates
  • Linux: mongosh --tls --tlsAllowInvalidCertificates

In the shell, use the following commands:

  • show dbs: Lists all databases.
  • use <database>: Switches to database <database>, for example use spooler-jobs.
  • show collections: Lists all collections in the current database.
  • db.<collection>.countDocuments(): Counts the documents in collection <collection> of the current database, for example db.outbox.countDocuments().

Caution - system databases

Do not delete or change the databases admin, config, and local. MongoDB needs them to operate.

Back to top