mc-crusher/README at master · dormando/mc-crusher · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
An extremely rough but extremely damaging benchmark utility for memcached.

Does not use a client library, does not use threads (though it will). Uses
event based sockets via libevent.

Building
--------

$ (install libevent + headers)
$ ./compile

Usage
-----

Look in the conf/ directory for example configuration files.

$ ./mc-crusher ./conf/loadconf [host] [port]
defaults:
host: 127.0.0.1
port: 11211

Running
-------

This tool is extremely rough.

First, run it for a few seconds with a config like "conf/loadconf", this will
seed the instance with some values to fetch later. Now, kill it.

You can also use the bench-warmer.pl tool to preload some keys.

Pick a config file, and start it as above. You should start a fresh memcached,
and run the "sample" script that comes with mc-crusher. mc-crusher makes no
attempt to care about what it does. It flails tiredlessly against the fanged
defenses of the cache daemon.

Modify bench-sample to print what you are most interested in examining.

You can change the "default" destination ip/port by adding extra arguments:
$ ./mc-crusher ./conf/loadconf 192.168.1.1 11211

Configuration
-------------

mc-crusher reads a configuration file then executes. These configurations
describe a "type" of connection, one per line.

Each "type" can spawn N connections. This allows you to mix setters and
getters, getters of different sizes, binprot + asciiprot. It's very limited
right now, but I wanted to build this in as a base.

Config Options
--------------

send : defines what function to use to send requests to memcached
 - ascii_get : one get per request via asciiprot
 - ascii_set : one set per request via asciiprot
 - ascii_delete : one delete per request via asciiprot
 - ascii_mget : multiget test via asciiprot
 - bin_get : one get per req via binprot
 - bin_getq : endless streaming multiget from hell
 - bin_set : one set per req via binprot
 - bin_setq : unleash cthulu upon the cache_lock

recv : same, but for received data
 - blind_recv : mindlessly slurp any responses without inspecting them

mget_count : set this to the number of keys to fetch per get in ascii_mget

key_prefix : a string to prefix before each key's number (default 'foo')

value_size : size of the value to set

expire : the expiration value (default 0)

flags : client flags (raw number, default 0)

value : define a value by hand. Must be shortish and a string.

host : define a host to connect to

port : define a port to connect to

key_prealloc : pre-create key strings in memory to reduce bench cpu usage

key_count : number of keys to iterate across. use with key_prealloc=0 to
iterate over large numbers of keys

key_randomize : if using prealloc'ed keys, shuffle the list after generating

usleep : fire one write event (up to pipeline count) with a microsecond sleep
after. the sleep is per-connection (so conns=2 will write twice per usleep)

stop_after : stop all conns for this test after N writes have been made.

Caveats
-------

- I bundled the protocol_binary.h header. It will need to be copied from the
  primary repo for updates.

- Does not make any attempt to safely parse the config file. If you don't type
  it exactly right you will end up with bizarre failures.

- It only works well with small values

- It makes no attempt to reconcile with errors in the protocol, and can break
  if memcached throws errors. It will also stop if the connections are closed.

- It's a ton of fun!

Future Features
---------------

A short list of things I intend to change or add:

- Fix the command generators to have a *little* error handling

- Switch to writev and use iovectors + pregenerated keys for almost everything

- Add back some routines which are sprintf only, for doing tests with
  extremely wide keyspaces where performance is less of an issue.

- Add a "run_every" option which fires a conn every N microseconds instead of
  endlessly.

- Add a "timer" option which times the commands it runs and periodically dumps
  a histogram of response times. Mainly to be used with "run_every"

- Add missing protocol commands. *_incr, *_decr, *_delete, etc.

- Make the iteration loop per-connection, and randomizeable.

- Add commands to iterate over different value sizes

- Add multithreading to ease management.

- Allow running timer conns in their own thread (or just define if a conn
  template should use a thread, then define conns=1?)

- Bundle a better perl util for printing stats

- Write/bundle a better util for running iterative benchmarks.