The DES encryption used by VNC servers

by Vidar Holen, http://www.vidarholen.net

VNC, Virtual Network Computing, is a way of remote controlling other computers, using the Remote FrameBuffer protocol, RFB. It's not platform specific, so you can access any computer from any other (assuming software is written for them). I wanted to implement a viewer, but I ran into trouble as early as the authentication.

My DES (Data Encryption Standard) implementation, which worked fine with all the calibration data I could find, didn't seem to match what the VNC software did. It took me a while to figure out why, since I didn't find the difference documented anywhere.

What VNC actually does

The RFB specification says that VNC authentication is done by receiving a 16 byte challenge, encrypting it with DES using the user specified password, and sending back the resulting 16 bytes.

The actual software encrypts the challenge with all the bit fields in each byte of the password mirrored.

Actual password ("COW"):
01000011 01001111 01010111 (plus five bytes of zero)
Key used as encryption:
11000010 11110010 11101010 (plus five bytes of zero)

The actual flipping is, in effect, done in rfb/d3des.c in realvnc.com's unix server. The bytebit array has had its entries mirrored, as the comments mention.

This is no doubt because the most significant bit in 7-bit ascii is always 0. When the DES algorithm creates a 56bit key based on the password, it makes sense to trick it into removing this 0 instead of the the least significant bit which is part of the actual password (otherwise B and C would for authentication be the same character).

Here's a rather stupid DES implementation, by yours truly, with this taken into account. GPL.