Skip to content

mongodb 磁盘空间重用测试对比 (2.6.12 v.s. 3.2.11)

Conclusion

从测试日志可以看出： mongodb3默认的WiredTiger存储引擎，与mongodb2的MMAP相比，在空间利用率上有着质的飞越； mongodb3在删除集合后，对应的磁盘空间会立即回收，这极大的减轻了运维工作量。

以下是测试日志 (mongodb_reuse_diskspace_test_log.txt).

Mongodb 2.6.12 test log

$ mongo --version
MongoDB shell version: 2.6.12
=====================================================
测试集合名称：person2
准备
db.person2.drop();
db.repairDatabase();

$ du -h -d 1 data/master
80M data/master/admin
0B data/master/journal
10G data/master/local
80M data/master/logevents
10G data/master

写入727.2万
727.2万 person2.dataSize=1.6254186630249023G 每条数据 240bytes
$ du -h -d 1 data/master
0B data/master/_tmp
80M data/master/admin
289M data/master/journal
10G data/master/local
4.0G data/master/logevents
14G data/master

再写入500万
[2016-12-15 14:08:33.460 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:09:39.644 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-15 14:09:40.471 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:10:48.308 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-15 14:10:48.969 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:11:58.608 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-15 14:11:59.592 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:13:09.636 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-15 14:13:10.622 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:14:35.282 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功

1227.2万 person2.dataSize=2.7430057525634766G
$ du -h -d 1 data/master
0B data/master/_tmp
80M data/master/admin
491M data/master/journal
10G data/master/local
6.0G data/master/logevents
17G data/master

删除集合 db.person2.drop();
$ du -h -d 1 data/master
0B data/master/_tmp
80M data/master/admin
492M data/master/journal
10G data/master/local
6.0G data/master/logevents
17G data/master

再写入500万
[2016-12-15 14:17:39.876 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:18:29.688 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-15 14:18:30.380 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:19:18.009 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-15 14:19:18.591 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:20:07.668 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-15 14:20:08.188 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:20:58.533 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-15 14:20:59.098 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-15 14:21:56.991 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功

500万 person2.dataSize=1.1175870895385742G
$ du -h -d 1 data/master
0B data/master/_tmp
80M data/master/admin
325M data/master/journal
10G data/master/local
6.0G data/master/logevents
16G data/master

Mongodb 3.2.11 test log

$ mongo --version
MongoDB shell version: 3.2.11
=====================================================
[echo@EchoMa data]$ du -h -d 1 master
20K master/diagnostic.data
300M master/journal
64K master/local
300M master

首先向logevents.person2写入800万

[2016-12-20 11:14:57.037 CST] main INFO context.TestContextManager: Could not instantiate TestExecutionListener [org.springframework.test.context.web.ServletTestExecutionListener]. Specify custom listener classes or make the default listener classes (and their required dependencies) available. Offending class: [javax/servlet/ServletContext]
[2016-12-20 11:14:57.212 CST] main INFO xml.XmlBeanDefinitionReader: Loading XML bean definitions from URL [file:/Users/echo/workjava/ctower-v3.0/mongodb-demo/target/classes/applicationContext-mongo2.xml]
[2016-12-20 11:14:57.533 CST] main INFO support.GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@321e4dd0: startup date [Tue Dec 20 11:14:57 CST 2016]; root of context hierarchy
[2016-12-20 11:14:57.644 CST] main INFO config.PropertyPlaceholderConfigurer: Loading properties file from URL [file:/Users/echo/workjava/ctower-v3.0/mongodb-demo/target/classes/conf/conf-mongo2.properties]
[2016-12-20 11:14:57.678 CST] main INFO support.DefaultListableBeanFactory: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@a78f12e: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,mongo,org.springframework.beans.factory.config.CustomEditorConfigurer#0,org.springframework.beans.factory.config.CustomEditorConfigurer#1,org.springframework.beans.factory.config.CustomEditorConfigurer#2,mongoDbFactory,mongoTemplate,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy
[2016-12-20 11:14:59.535 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:16:11.159 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:16:11.201 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 1000010
[2016-12-20 11:16:12.463 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:17:13.819 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:17:13.820 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 2000010
[2016-12-20 11:17:14.273 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:18:15.463 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:18:15.479 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 3000010
[2016-12-20 11:18:17.394 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:19:13.793 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:19:13.794 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 4000010
[2016-12-20 11:19:14.279 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:20:46.799 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:20:46.806 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 5000010
[2016-12-20 11:20:47.462 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:22:59.124 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:22:59.127 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 6000010
[2016-12-20 11:23:00.799 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:24:26.280 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:24:26.283 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 7000010
[2016-12-20 11:24:27.261 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:26:03.717 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:26:03.723 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 8000010
[2016-12-20 11:26:03.805 CST] Thread-2 INFO support.GenericApplicationContext: Closing org.springframework.context.support.GenericApplicationContext@321e4dd0: startup date [Tue Dec 20 11:14:57 CST 2016]; root of context hierarchy
[2016-12-20 11:26:03.811 CST] Thread-2 INFO support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@a78f12e: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,mongo,org.springframework.beans.factory.config.CustomEditorConfigurer#0,org.springframework.beans.factory.config.CustomEditorConfigurer#1,org.springframework.beans.factory.config.CustomEditorConfigurer#2,mongoDbFactory,mongoTemplate,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy

查看磁盘文件大小
[echo@EchoMa mongodb3]$ du -h -d 1 data/master
140K data/master/admin
1.1M data/master/diagnostic.data
400M data/master/journal
478M data/master/local
703M data/master/logevents
1.5G data/master

删除person2集合
db.person2.drop();

[echo@EchoMa mongodb3]$ du -h -d 1 data/master
140K data/master/admin
1.1M data/master/diagnostic.data
400M data/master/journal
478M data/master/local
8.0K data/master/logevents
879M data/master

再写入800万
[2016-12-20 11:43:27.632 CST] main INFO context.TestContextManager: Could not instantiate TestExecutionListener [org.springframework.test.context.web.ServletTestExecutionListener]. Specify custom listener classes or make the default listener classes (and their required dependencies) available. Offending class: [javax/servlet/ServletContext]
[2016-12-20 11:43:27.814 CST] main INFO xml.XmlBeanDefinitionReader: Loading XML bean definitions from URL [file:/Users/echo/workjava/ctower-v3.0/mongodb-demo/target/classes/applicationContext-mongo.xml]
[2016-12-20 11:43:28.159 CST] main INFO support.GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@401a37b1: startup date [Tue Dec 20 11:43:28 CST 2016]; root of context hierarchy
[2016-12-20 11:43:28.296 CST] main INFO config.PropertyPlaceholderConfigurer: Loading properties file from URL [file:/Users/echo/workjava/ctower-v3.0/mongodb-demo/target/classes/conf/conf-mongo.properties]
[2016-12-20 11:43:28.329 CST] main INFO support.DefaultListableBeanFactory: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@54891f43: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,mongo,org.springframework.beans.factory.config.CustomEditorConfigurer#0,org.springframework.beans.factory.config.CustomEditorConfigurer#1,org.springframework.beans.factory.config.CustomEditorConfigurer#2,mongoDbFactory,mongoTemplate,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy
[2016-12-20 11:43:30.177 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:44:55.370 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:44:55.435 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 1000000
[2016-12-20 11:44:56.017 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:46:36.982 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:46:36.987 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 2000000
[2016-12-20 11:46:37.680 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:47:39.489 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:47:39.492 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 3000000
[2016-12-20 11:47:40.754 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:49:25.819 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:49:25.828 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 4000000
[2016-12-20 11:49:26.283 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:50:47.129 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:50:47.135 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 5000000
[2016-12-20 11:50:48.615 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:52:36.344 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:52:36.348 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 6000000
[2016-12-20 11:52:37.756 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:54:06.097 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:54:06.100 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 7000000
[2016-12-20 11:54:06.721 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 1000000条记录已就绪
[2016-12-20 11:55:36.462 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: 写入1000000条记录至Person2成功
[2016-12-20 11:55:36.462 CST] main INFO mongodb.MongoDbReuseDiskspaceTest: Person2 记录数: 8000000
[2016-12-20 11:55:36.486 CST] Thread-2 INFO support.GenericApplicationContext: Closing org.springframework.context.support.GenericApplicationContext@401a37b1: startup date [Tue Dec 20 11:43:28 CST 2016]; root of context hierarchy
[2016-12-20 11:55:36.489 CST] Thread-2 INFO support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@54891f43: defining beans [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,mongo,org.springframework.beans.factory.config.CustomEditorConfigurer#0,org.springframework.beans.factory.config.CustomEditorConfigurer#1,org.springframework.beans.factory.config.CustomEditorConfigurer#2,mongoDbFactory,mongoTemplate,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy

查看磁盘文件大小
[echo@EchoMa mongodb3]$ du -h -d 1 data/master
140K data/master/admin
1.3M data/master/diagnostic.data
400M data/master/journal
956M data/master/local
694M data/master/logevents
2.0G data/master

Test codes

MongoDbReuseDiskspaceTest.java 
import java.util.ArrayList;
import java.util.Collection;

import javax.annotation.Resource;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

/**
* MongoDB 重用磁盘空间测试
*/
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath*:applicationContext-mongo.xml")
public class MongoDbReuseDiskspaceTest {
  static Log LOG = LogFactory.getLog(MongoDbReuseDiskspaceTest.class);
  @Resource
  MongoTemplate mongoTemplate;

  @Test
  public void testMultiInsert() {
    for (int i = 0; i < 8; i++) {// 800万记录
      // 100万记录
      this.testInsert(100 * 10000);
    }
  }

  private void testInsert(int count) {
    Collection<Person2> list = new ArrayList<Person2>();
    String suffix = String.valueOf(Math.random());
    for (int i = 0; i < count; i++) {
      Person2 p = new Person2();
      p.setId(i + suffix);
      // p.setId("1"); 重复会报错
      p.setAge((int) (Math.random() * 80));
      p.setName("Tom" + (int) (Math.random() * 1000));
      list.add(p);
    }

    LOG.info(list.size() + "条记录已就绪");
    mongoTemplate.insertAll(list);
    LOG.info("写入" + list.size() + "条记录至Person2成功");

    long records = mongoTemplate.count(null, Person2.class);
    LOG.info("Person2 记录数: " + records);
  }
}