My Journey with Large Scale Data Gathering

Create additional indexes on the DB

Setting up authenticated Mongo

mongo.conf


# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: mongod.log

# Where and how to store data.
storage:
  dbPath: ~/mongo/data

# how the process runs
processManagement:
  fork: false  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile

# network interfaces
net:
  port: 27017
  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
  bindIp: 127.0.0.1, outside_access.domain.com

security:
  authorization: enabled

Mongo restart comand with auth

mongod --bind_ip_all  --auth --port 27017 --dbpath ~/mongo/data/

To add users + roles


    // main admin user
    use admin
    db.createUser({ user: "***", pwd: passwordPrompt(), roles: [{ role: "root", db: "admin" }, { role: "dbOwner", db: "work_db" }] })

    // specific DB user
    use work_db
    db.createUser({ user: "***", pwd: passwordPrompt(), roles: [{ role: "dbOwner", db: "work_db" }] })

Connection string

connectionURI = "mongodb://user:pass@localhost:27017/?serverSelectionTimeoutMS=5000&connectTimeoutMS=10000&authSource=WORK_DB&authMechanism=SCRAM-SHA-256"

Port Forwarding