Since version 2.0 of WordPress, it has stopped supporting custom file type uploading. This means only files with certain extensions can be uploaded, and there is currently no work-around. As frustrating as it may be, if you host your own WordPress script and have access to editting the scripts yourself, you are in luck.

In /wp-admin/admin-functions.php, around line 1668 (as of WordPress 2.02), there should be something like this. Search within the file if you can’t find it.

$mimes = apply_filters('upload_mimes', array (
    'jpg|jpeg|jpe' => 'image/jpeg',
    'gif' => 'image/gif',
    'png' => 'image/png',
    'bmp' => 'image/bmp',
    'tif|tiff' => 'image/tiff',
    'ico' => 'image/x-icon',
    'asf|asx|wax|wmv|wmx' => 'video/asf',
    'avi' => 'video/avi',
    'mov|qt' => 'video/quicktime',
    'mpeg|mpg|mpe' => 'video/mpeg',
    'txt|c|cc|h' => 'text/plain',
    'rtx' => 'text/richtext',
    'css' => 'text/css',
    'htm|html' => 'text/html',
    'mp3|mp4' => 'audio/mpeg',
    'ra|ram' => 'audio/x-realaudio',
    'wav' => 'audio/wav',
    'ogg' => 'audio/ogg',
    'mid|midi' => 'audio/midi',
    'wma' => 'audio/wma',
    'rtf' => 'application/rtf',
    'js' => 'application/javascript',
    'pdf' => 'application/pdf',
    'doc' => 'application/msword',
    'pot|pps|ppt' => 'application/vnd.ms-powerpoint',
    'wri' => 'application/vnd.ms-write',
    'xla|xls|xlt|xlw' => 'application/vnd.ms-excel',
    'mdb' => 'application/vnd.ms-access',
    'mpp' => 'application/vnd.ms-project',
    'swf' => 'application/x-shockwave-flash',
    'class' => 'application/java',
    'tar' => 'application/x-tar',
    'zip' => 'application/zip',
    'gz|gzip' => 'application/x-gzip',
    'exe' => 'application/x-msdownload'
));

These are the default extensions and MIME-types for uploads. Simply add in what you want and you can upload your own extension. For specific MIME-types to file extensions — http://filext.com/. This is to serve the right MIME when WordPress serves the file (since WordPress handles all requests first-hand with .htaccess as of version 2).

I’m guessing a plugin could be created as a fix for this problem, since it does use filters. If someone finds a suitable solution, please tell. :)