MongoDB is a NoSQL document database system that scales well horizontally and implements data storage through a key-value system. A popular choice for web applications and websites, MongoDB is easy to implement and access programmatically.
MongoDB Sharding Topology
Sharding is implemented through three separate components. Each part performs a specific function:
Config Server: Each production sharding implementation must contain exactly three configuration servers. This is to ensure redundancy and high availability.
Config servers are used to store the metadata that links requested data with the shard that contains it. It organizes the data so that information can be retrieved reliably and consistently.
Query Routers: The query routers are the machines that your application actually connects to. These machines are responsible for communicating to the config servers to figure out where the requested data is stored. It then accesses and returns the data from the appropriate shard(s).
Each query router runs the "mongos" command.
Shard Servers: Shards are responsible for the actual data storage operations. In production environments, a single shard is usually composed of a replica set instead of a single machine. This is to ensure that data will still be accessible in the event that a primary shard server goes offline.
Initial Set Up
If you were paying attention above, you probably noticed that this configuration requires quite a few machines. In this tutorial, we will configure an example sharding cluster that contains:
3 Config Servers (Required in production environments)
1 Query Routers (Minimum of 1 necessary)
3 Shard Servers (Minimum of 2 necessary)
192.168.1.1 config1
192.168.1.2 config2
192.168.1.3 config3
192.168.1.4 Query
192.168.1.5 shard1
192.168.1.6 shard2
192.168.1.7 shard3
Login into all server and change hostname as well as mention above hosts in /etc/hosts
Config1, Config2 & Config3:
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y mongodb-org=3.2.7 mongodb-org-server=3.2.7 mongodb-org-shell=3.2.7 mongodb-org-mongos=3.2.7 mongodb-org-tools=3.2.7
Below configuration should be smae between all config server
root@config1:/home/ubuntu# vim /etc/mongod.conf
net:
port: 27017
bindIp: 0.0.0.0
sharding:
clusterRole: configsvr
replication:
replSetName: configReplSet
Go to config1
root@config1:~# mongo
rs.initiate( {
_id: "configReplSet",
configsvr: true,
members: [
{ _id: 0, host: "config1:27017" },
{ _id: 1, host: "config2:27017" },
{ _id: 2, host: "config3:27017" }
]
} )
NOTE: If you want to add one more config server in replica run below command on Primery config server:
rs.add({_id: 1, host: "config4:27017", priority: 0, hidden: true})
Shard1:
service mongod stop
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y mongodb-org=3.2.7 mongodb-org-server=3.2.7 mongodb-org-shell=3.2.7 mongodb-org-mongos=3.2.7 mongodb-org-tools=3.2.7
Do on all shard node
mkdir -p /data/db1
mkdir /data/db2
mkdir /data/db3
mongod --port 27017 --dbpath /data/db1 --smallfiles --fork --logpath /var/log/mongodb/mongod.log --replSet "rs1"
mongod --port 27018 --dbpath /data/db2 --smallfiles --fork --logpath /var/log/mongodb/mongod.log --replSet "rs2"
mongod --port 27019 --dbpath /data/db3 --smallfiles --fork --logpath /var/log/mongodb/mongod.log --replSet "rs3"
Run below commands on every primary server
mongo
config = { _id: "rs1", members:[ { _id : 0, host : "shard1:27017" }, { _id : 1, host : "shard2:27017" },{ _id : 2, host : "shard3:27017" } ]};
rs.initiate(config)
mongo --port 27018
config = { _id: "rs2", members:[ { _id : 0, host : "shard1:27018" }, { _id : 1, host : "shard2:27018" },{ _id : 2, host : "shard3:27018" } ]};
rs.initiate(config)
mongo --port 27019
config = { _id: "rs3", members:[ { _id : 0, host : "shard1:27019" }, { _id : 1, host : "shard2:27019" },{ _id : 2, host : "shard3:27019" } ]};
rs.initiate(config)
Query Server:
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y mongodb-org=3.2.7 mongodb-org-server=3.2.7 mongodb-org-shell=3.2.7 mongodb-org-mongos=3.2.7 mongodb-org-tools=3.2.7
root@query:~# service mongod stop
root@query:~# mongos --logpath /var/log/mongo/mongo.log --configdb configReplSet/config1:27017,config2:27017,config3:27017 &
mongos> db.adminCommand( { listShards: 1 } )
{ "shards" : [ ], "ok" : 1 }
mongos> db.adminCommand({addShard : "rs1/shard1:27017,shard2:27017,shard3:27017", name : "rs1"});
{ "shardAdded" : "rs1", "ok" : 1 }
mongos> db.adminCommand({addShard : "rs2/shard1:27018,shard2:27018,shard3:27018", name : "rs2"});
{ "shardAdded" : "rs2", "ok" : 1 }
mongos> db.adminCommand({addShard : "rs3/shard1:27019,shard2:27019,shard3:27019", name : "rs3"});
{ "shardAdded" : "rs3", "ok" : 1 }
mongos> db.runCommand("getShardMap")
mongos> use raj
switched to db raj
mongos> sh.enableSharding("raj")
{ "ok" : 1 }
mongos> sh.shardCollection("raj.collections",{ "_id":"hashed" } ) ====> RUN this command for all collections
{ "collectionsharded" : "raj.collections", "ok" : 1 }
For data insert
for (var i = 1; i <= 100000; i++) { db.collections.insert( { _id : i } ) } ====. _id should be change in prod choose date
================================================================================================================================================
Basic Commands
mongos> rs.status()
mongos> db.stats()
mongos> use database
mongos> db.stats()
mongos> sh.getBalancerState()
mongos> use admin
mongos> db.runCommand( { listshards : 1 } );
db.runCommand("getShardMap")
To check Cluster status and shrading
mongos> db.printShardingStatus();
mongos> sh.status()
