Fritzy's Daily Gists

A code pattern or snippet per day.

Using Postgres as a Document Store

Lately, I’ve been working on pgDOWN, which is a Postgres backend for levelup, so that I can support Postgres in Dulcimer. My motivation is to have a CP backend for Duclimer. Currently leveldb (levelDOWN) works great for an embedded backend, and Riak for an AP backend, but I we have needs at &yet for a CP backend.

Postgres has some handy features, which can make it very nice as a document/key-value store; Namely the JSON and HSTORE data types.

Generally key-stores have GET, PUT, and RANGE queries. Let’s make some functions for those.

Here’s a put function based on Postgres’s lack of a REPLACE query.

We don’t really need a function for GET by key, except it’s nice to be injection safe. but let’s do one anyway for consistency.

The RANGE query is very similar.

But what if we wanted to query by values in the JSON itself? Well, that’d be pretty slow, UNLESS we made an index for the values we wanted to query by.

If you want to prevent duplicate values, you could use the UNIQUE restriction on the index.

Now we can query by the lastname field of the JSON without it having to scan the entire table! value->>'lastname' generates an index for the literal, unescaped TEXT value of the field. You should do this for every field that you plan on escaping. It does incur a small cost on writes (as any atomic index system does).

And again, a field RANGE function would be handy.

You probably want to set up different tables for different object types to keep indexes from being full of junk. As such, you probably want to add a tablename field into the above functions and escape it with quote_ident in your EXECUTE statements.

Using these functions are as easy as SELECTing them.

SELECT * FROM documentstore_get_by_field('lastname', 'myusers', 'Fritz', 10);

Let me know on @fritzy Twitter if you have any questions, fixes, or comments. I currently have availability for consulting work through &yet.

Generating Reverse Sort Indexes for Riak in Node

While developing the prototype for what became @quitlahok’s riakdown, I quickly realized that while Riak can give keys back in lexical order of indexes, it cannot do the same in reverse. We needed reverse support in order to support the levelup interface.

Our solution was to generate extra indexes that would be the lexical mirror of each byte for each index given, as well as the key. This creates double the indexes, but for our purpose was important.

In order to mirror the byte, I did a NOT operation on it, which is not straight forward in JS. For each byte, we need to get the ascii value, and NOT that, but the ascii value is returned as a JS Number. When we perform a NOT on a Number, we get an unexpected result, due to its signed nature. Really, what we want is the last byte, so we mask it with AND 0xFF.

If there is a more direct way to do this in JavaScript, please let me know.


@brycebaril points out that things get weird with unicode characters, and that we can reduce the number of operations by doing 255 - value, rather than NOT+AND.


@quitlahok noted that riakdown doesn’t actually produce these extra indexes (although my prototype did). Instead he uses a map-reduce query to accomplish this.

Create A Levelup Proxy to Riak

Riak is a clustered key-value database that applies “AP” from the CAP Theorem. It has HTTP and Protocol Buffers APIs, but can be a little bit cumbersome to browse keys for debugging. I’ve been getting used to hij1nx’s lev for debugging levelup and thought it would be handy to use the same tool for Riak.

Since Nathan LaFreniere wrote riakdown, a Riak backend for level, it wasn’t too hard. Essentially just mix multilevel (for a network enabled levelup) with riakdown, and away you go!

$ level2riak --bucket somebucket
Connected to riak://localhost:8087/somebucket
Listening for multilevel connection on: 8091

Now just load lev and save connection details to match the proxy.

$ lev

lev connection config

You can also use lev’s other modes: cli and repl.

But wait, there’s more!

I turned this into an npm package.


npm install --global level2riak

You can then run it with:

level2riak --bucket somebucket


As of this writing, I’ve made an update to lev, but I’m waiting on hij1nx to publish to npm. The manifest is no longer required, repl can handle remote connections, and I added —conns and —use for listing and using preconfigured connections.


hij1nx just publish lev@2.1.2.

Archive Redis to Riak

Redis is nice for dealing with data as it is relevant and changing, but once it goes a little stale, you probably want to archive it to free up resources. Wouldn’t it be nice to be able to archive it and still be able to query it?

Using my new redisscan module, this script scrapes Redis, value by value, diving into the structures, and preserves it in Riak in a normalized way. You could then query the same keys in Riak or restore it back to Redis at any time.

Redisscan does most of the hard work of diving into each value of Redis keys. The biggest trick here is preserving List ordering when restoring back to Redis.

This approach is not atomic, so keys can change out from under it. A better approach might be an rdb file analyzer, but this allows us to scan production data without an rdb file using the new SCAN commands. Your use case may not be appropriate for this approach.

When I’ve added advanced pattern matching and Riak indexing, I’ll turn this into a standalone project. For now this is just a proof-of-concept to share.

Replacing Loading Images with Spinners Using Mutation Observers

Sometimes you’d rather images loading into your page don’t show up until they’re finished. Sometimes you don’t have direct control over when images get added to your page.

The MutationObserver is a handy API for reacting when DOM changes within an element. It’s commonly used in plugins, but sometimes it is handy for dealing with a central place for DOM changes, without having to mess with the script that actually changed the DOM.

In this case, it works pretty well for putting image placeholders in the place of loading images.

When I wrote my StayDown library, it became apparent that images needed to handled as a special case when added to an overflow element that we’re trying to keep scrolled down. There was no event that fires when a loading image element changes size, so the library didn’t realize it needed to scroll down until the image finished loading. Loading a cached image in its place until loaded simplified the problem.

Redis Lua: Updating a Key and Lookup Hash

Yesterday I mentioned that you should maintain a lookup hash as you update keys.

So today, I decided to put that together. The only real gotcha is deleting the lookup to the old attribute value before updating the lookup hash.

Largely the same caviates apply as yesterday. We could flesh this out to handle many attributes and attributes deeper than the root. Again, this assumes that each attribute will only exist in one key per lookup hash.

Redis: Generating a Hash Lookup with SCAN

Using a keystore efficiently is all about anticipating how the data will be looked up. If you can’t do that, you may be better using a relational database.

Generating lookup tables and indexes can be helpful for finding the key you’re looking for. For example, if you store users as user:id, but you’d also like to be able to look them up by a username, you can generate a lookup table.

The SCAN command allows us to iterate through keys without getting them all at once and really slowing down the server. It does this by breaking up the hash of keys into equal spaces and iterating them. Unfortunately, this approach will give the keys in different iterated results on different servers, because the hash is seeded randomly, making SCAN a non-deterministic command.

This example will fail, only because HSET (any writing command) isn’t allowed after using a non-deterministic command (in this case, SCAN).

[Error: ERR Error running script (call to f_a707d1274859064311a626bf76f47b58f3764c62): @user_script:12: @user_script: 12: Write commands not allowed after non deterministic commands ]

This means that we’ll have to do this from the client. Here is an example of doing this in Node.

Basically, in both cases, we call SCAN [iter] starting with zero, until iter is 0 again. For each key, we take the designated attribute, and we set that as the lookup key, and the scanned key as the lookup value. Now we can use the lookup hash to find the key associated with that attribute.

A few notes:

  • This does not handle cases where multiple keys might have the same attribute.
  • Using SCAN in this way from the client means that we may miss keys that have been added while scanning. A subsequent update will have to find them.
  • Ideally you’ll maintain the lookup hash when you set keys, but we’ll do that gist another time.

Redis Lua: Updating a Key and Publishing a Logged Notification

One of the cool things about Redis Lua scripting is the ability to publish notifications of your changes atomically with those changes made. There’s actually a new Redis feature that can do this for you if configured: keyspace notifications. But maybe you want to send out a single notification from a script that does several things, and this keeps a log of notifications.

You could also use an iterator id so that clients can make sure they’ve received every notification.

P.S. I’ve been busy today, but I’d like these to be happening every work day, regardless of whether I go to work or not, so here’s a late-day entry.

Redis Lua: Collate Redis Key/Value Results

Any Redis command that returns key/value pairs returns the results as a single list of alternating keys and values. If you need to work the results in a Lua script, you’ll likely want to collate them in a Lua table.

This script takes a Hash key and returns a collated JSON object of key/values.

> HMSET some-hash greeting1 hi
> HMSET some-hash greeting2 hello
> HMSET some-hash signoff1 bye

> EVAL "this script" 1 some-hash
{"greeting1": "hi", "greeting2": "hello", "signoff1": "bye"}

Keep in mind that there are actually a lot of Redis commands that return key/values this way, not just HGETALL, and that I’m returning JSON just for the eexample’s sake.

Update: Pierre Chapuis @pchapuis suggested using a for-step loop rather than a for-each loop.

Redis Lua: Storing and Checking Hashed Passwords

See the update at the bottom: this approach isn’t secure (but I still think it’s interesting).

Storing passwords in plain text is bad. Time and time again, databases get hacked and passwords get leaked. Hashing a password creates a consistent string from a password that you can store, but isn’t the password, and isn’t easily reversed. Reversing a hash is difficult, and typically done with brute force if the hashing algorthim is fairly secure.

People end up creating giant lookup tables called “rainbow tables” of every possible string combination up to a certain number of characters in order to quickly reverse hashes. The best defense against this is to hash something larger than a typical password because it takes exponential time to create a rainbow table for each additional character. We do this with dynamic and static salts. Salts are used to add length to the password string before hashing. A static salt is global across the application, and a dynamic salt is unique for each user.

Awhile back ago, @antirez accepted a pull request for redis.sha1hex() in Redis’s EVAL scripts. This gives us a great way to store and check hashed passwords.

> EVAL "setpassword script" 1 testpass bob "hi"
"Password must be at least 8 characters."

> EVAL "setpassword script" 1 testpass bob "hi there"

> EVAL "checkpassword script" 1 testpass bob "hi there"
1) (nil)
2) (integer) 1

> EVAL "checkpassword script" 1 testpass bob "wrong password"
1) (nil)
2) (nil)

There are a few things to note here.

  • The schema for creating your pre-hashed string isn’t important except that you need to be consistent.
  • We could use anything as the dynamic salt, like the user’s phone number, but you’d have to get the user’s password and re-generate the hash everytime this information is changed.
  • I’m using an error-first return pattern for these scripts. The first argument is falsey if there is no error, and a string if there is an error. This is a common JavaScript pattern, and useful in Redis Lua scripts.
  • The password length check is actually counting bytes.


Again, Pierre Chapuis @pchapuis has some good feedback. He references which argues against using typical hashing algorithms for hashing. Since someone who has your redis db also has your Lua scripts, the salts are rather available to the cracker, even the dynamic ones.

We can easily argue that bcrypt/scrypt is the better way to go. Unfortunately, it’s not available in Redis, but that’s not a terribly valid reason to discount it.

This approach is much better than nothing, but perhaps outdated.

As I’m writing this, Alex Gaynor @alex_gaynor just gave the same advice.