Home > Daynotes Home > Week of 21 December 2009

Photograph of Robert Bruce Thompson
Daynotes Journal

Week of 21 December 2009


Latest Update: Sunday, 27 December 2009 13:40 -0500

Paying for this Place
Visit Barbara's Journal Page

Monday, 21 December 2009
[Daynotes Forums]    [Last Week]   [Mon]  [Tue]  [Wed]  [Thu]  [Fri]  [Sat]  [Sun]   [Next Week]    [HardwareGuys Forums]

09:45 - The Winter Solstice occurs today at 12:47 p.m. EST. That means it's time again for Top Ten Lists. My favorite so far is The Top Ten Stories of the Last 4.5 Billion Years, and my two favorites on that list are Sumerians Look On In Confusion As God Creates World and Woman Domesticated.

With the approach of Christmas Day, things will be winding down for the year at Maker Shed after the last-minute push today through noon PST tomorrow. I just noticed this on the Maker Shed front page:

Spend $175 or more and receive...
FREE shipping in time for Christmas!!!
Use coupon code "HURRY" at checkout
Offer expires at Noon Tuesday (PST) so "HURRY"!
Not to be combined with any other offer, continental US only

So if you haven't finished your last-minute shopping, there's still time.


[top]

Tuesday, 22 December 2009
[Daynotes Forums]    [Last Week]   [Mon]  [Tue]  [Wed]  [Thu]  [Fri]  [Sat]  [Sun]   [Next Week]    [HardwareGuys Forums]

08:47 - Throttled by Netflix! Well, not really. Throttled by the post office, more like. I should have received three discs from Netflix yesterday, but only two arrived. I'm 99% certain that Netflix actually shipped all three for arrival yesterday, but the post office delivered only two of them, no doubt because of the flood of Christmas traffic. Netflix itself hasn't throttled me at all for the entire eight months of my current membership. Not so much as a single disc for a single day.

Since I rejoined Netflix on 26 April, they've charged me $135.92 plus tax, for a total of $145.63. They've shipped me 199 discs since then, and by the time this month's subscription runs out on 12/25, they'll have shipped me 202 discs. That means Netflix has received about $0.70 per disc from me, which surely makes me their least profitable customer.

I always return discs two per envelope when possible, which means they've paid postage to send me 202 discs and to receive 134 return envelopes, for a total of 336 envelopes outbound and inbound. I'm sure they get a volume discount from the USPS, but even so that comes to total revenue of only about $0.40 per envelope. In other words, counting the BRE charge, they're probably not even breaking even on postage costs. For Netflix, I'm the Customer from Hell. And yet they continue to honor their promise.

In fact, I admit to feeling a bit guilty for holding them to the letter of their promise. Perhaps I'll make a New Year's resolution to rent fewer discs per month from them and let them make at least a small profit from me.


[top]

Wednesday, 23 December 2009
[Daynotes Forums]    [Last Week]   [Mon]  [Tue]  [Wed]  [Thu]  [Fri]  [Sat]  [Sun]   [Next Week]    [HardwareGuys Forums]

07:44 - I'm not usually a fan of short stories, but here's one I enjoyed. The characterizations, plot, and dialog are all excellent. You might want to save a local copy for when you have time to read it.

#! /usr/bin/python

# unswindle.pyw, version 5-rc1

# To run this program install a 32-bit version of Python 2.6 from
# <http://www.python.org/download/>.  Save this script file as unswindle.pyw.
# Find and save in the same directory a copy of mobidedrm.py.  Double-click on
# unswindle.pyw.  It will run Kindle For PC.  Open the book you want to
# decrypt.  Close Kindle For PC.  A dialog will open allowing you to select the
# output file.  And you're done!

# Revision history:
#   1 - Initial release
#   2 - Fixes to work properly on Windows versions >XP
#   3 - Fix minor bug in path extraction
#   4 - Fix error opening threads; detect Topaz books;
#       detect unsupported versions of K4PC
#   5 - Work on new (20091222) version of K4PC

"""
Decrypt Kindle For PC encrypted Mobipocket books.
"""

__license__ = 'GPL v3'

import sys
import os
import re
import tempfile
import shutil
import subprocess
import struct
import hashlib
import ctypes
from ctypes import *
from ctypes.wintypes import *
import binascii
import _winreg as winreg
import Tkinter
import Tkconstants
import tkMessageBox
import tkFileDialog
import traceback

#
# _extrawintypes.py

UBYTE = c_ubyte
ULONG_PTR = POINTER(ULONG)
PULONG = ULONG_PTR
PVOID = LPVOID
LPCTSTR = LPTSTR = c_wchar_p
LPBYTE = c_char_p
SIZE_T = c_uint
SIZE_T_p = POINTER(SIZE_T)

#
# _ntdll.py

NTSTATUS = DWORD

ntdll = windll.ntdll

class PROCESS_BASIC_INFORMATION(Structure):
    _fields_ = [('Reserved1', PVOID),
                ('PebBaseAddress', PVOID),
                ('Reserved2', PVOID * 2),
                ('UniqueProcessId', ULONG_PTR),
                ('Reserved3', PVOID)]

# NTSTATUS WINAPI NtQueryInformationProcess(
#   __in       HANDLE ProcessHandle,
#   __in       PROCESSINFOCLASS ProcessInformationClass,
#   __out      PVOID ProcessInformation,
#   __in       ULONG ProcessInformationLength,
#   __out_opt  PULONG ReturnLength
# );
NtQueryInformationProcess = ntdll.NtQueryInformationProcess
NtQueryInformationProcess.argtypes = [HANDLE, DWORD, PVOID, ULONG, PULONG]
NtQueryInformationProcess.restype = NTSTATUS

#
# _kernel32.py

INFINITE = 0xffffffff

CREATE_UNICODE_ENVIRONMENT = 0x00000400
DEBUG_ONLY_THIS_PROCESS = 0x00000002
DEBUG_PROCESS = 0x00000001

THREAD_GET_CONTEXT = 0x0008
THREAD_QUERY_INFORMATION = 0x0040
THREAD_SET_CONTEXT = 0x0010
THREAD_SET_INFORMATION = 0x0020

EXCEPTION_BREAKPOINT = 0x80000003
EXCEPTION_SINGLE_STEP = 0x80000004
EXCEPTION_ACCESS_VIOLATION = 0xC0000005

DBG_CONTINUE = 0x00010002L
DBG_EXCEPTION_NOT_HANDLED = 0x80010001L

EXCEPTION_DEBUG_EVENT = 1
CREATE_THREAD_DEBUG_EVENT = 2
CREATE_PROCESS_DEBUG_EVENT = 3
EXIT_THREAD_DEBUG_EVENT = 4
EXIT_PROCESS_DEBUG_EVENT = 5
LOAD_DLL_DEBUG_EVENT = 6
UNLOAD_DLL_DEBUG_EVENT = 7
OUTPUT_DEBUG_STRING_EVENT = 8
RIP_EVENT = 9

class DataBlob(Structure):
    _fields_ = [('cbData', c_uint),
                ('pbData', c_void_p)]
DataBlob_p = POINTER(DataBlob)

class SECURITY_ATTRIBUTES(Structure):
    _fields_ = [('nLength', DWORD),
                ('lpSecurityDescriptor', LPVOID),
                ('bInheritHandle', BOOL)]
LPSECURITY_ATTRIBUTES = POINTER(SECURITY_ATTRIBUTES)

class STARTUPINFO(Structure):
    _fields_ = [('cb', DWORD),
                ('lpReserved', LPTSTR),
                ('lpDesktop', LPTSTR),
                ('lpTitle', LPTSTR),
                ('dwX', DWORD),
                ('dwY', DWORD),
                ('dwXSize', DWORD),
                ('dwYSize', DWORD),
                ('dwXCountChars', DWORD),
                ('dwYCountChars', DWORD),
                ('dwFillAttribute', DWORD),
                ('dwFlags', DWORD),
                ('wShowWindow', WORD),
                ('cbReserved2', WORD),
                ('lpReserved2', LPBYTE),
                ('hStdInput', HANDLE),
                ('hStdOutput', HANDLE),
                ('hStdError', HANDLE)]
LPSTARTUPINFO = POINTER(STARTUPINFO)

class PROCESS_INFORMATION(Structure):
    _fields_ = [('hProcess', HANDLE),
                ('hThread', HANDLE),
                ('dwProcessId', DWORD),
                ('dwThreadId', DWORD)]
LPPROCESS_INFORMATION = POINTER(PROCESS_INFORMATION)

EXCEPTION_MAXIMUM_PARAMETERS = 15
class EXCEPTION_RECORD(Structure):
    pass
EXCEPTION_RECORD._fields_ = [
    ('ExceptionCode', DWORD),
    ('ExceptionFlags', DWORD),
    ('ExceptionRecord', POINTER(EXCEPTION_RECORD)),
    ('ExceptionAddress', LPVOID),
    ('NumberParameters', DWORD),
    ('ExceptionInformation', ULONG_PTR * EXCEPTION_MAXIMUM_PARAMETERS)]

class EXCEPTION_DEBUG_INFO(Structure):
    _fields_ = [('ExceptionRecord', EXCEPTION_RECORD),
                ('dwFirstChance', DWORD)]

class CREATE_THREAD_DEBUG_INFO(Structure):
    _fields_ = [('hThread', HANDLE),
                ('lpThreadLocalBase', LPVOID),
                ('lpStartAddress', LPVOID)]

class CREATE_PROCESS_DEBUG_INFO(Structure):
    _fields_ = [('hFile', HANDLE),
                ('hProcess', HANDLE),
                ('hThread', HANDLE),
                ('dwDebugInfoFileOffset', DWORD),
                ('nDebugInfoSize', DWORD),
                ('lpThreadLocalBase', LPVOID),
                ('lpStartAddress', LPVOID),
                ('lpImageName', LPVOID),
                ('fUnicode', WORD)]

class EXIT_THREAD_DEBUG_INFO(Structure):
    _fields_ = [('dwExitCode', DWORD)]

class EXIT_PROCESS_DEBUG_INFO(Structure):
    _fields_ = [('dwExitCode', DWORD)]

class LOAD_DLL_DEBUG_INFO(Structure):
    _fields_ = [('hFile', HANDLE),
                ('lpBaseOfDll', LPVOID),
                ('dwDebugInfoFileOffset', DWORD),
                ('nDebugInfoSize', DWORD),
                ('lpImageName', LPVOID),
                ('fUnicode', WORD)]

class UNLOAD_DLL_DEBUG_INFO(Structure):
    _fields_ = [('lpBaseOfDll', LPVOID)]

class OUTPUT_DEBUG_STRING_INFO(Structure):
    _fields_ = [('lpDebugStringData', LPSTR),
                ('fUnicode', WORD),
                ('nDebugStringLength', WORD)]

class RIP_INFO(Structure):
    _fields_ = [('dwError', DWORD),
                ('dwType', DWORD)]

class _U(Union):
    _fields_ = [('Exception', EXCEPTION_DEBUG_INFO),
                ('CreateThread', CREATE_THREAD_DEBUG_INFO),
                ('CreateProcessInfo', CREATE_PROCESS_DEBUG_INFO),
                ('ExitThread', EXIT_THREAD_DEBUG_INFO),
                ('ExitProcess', EXIT_PROCESS_DEBUG_INFO),
                ('LoadDll', LOAD_DLL_DEBUG_INFO),
                ('UnloadDll', UNLOAD_DLL_DEBUG_INFO),
                ('DebugString', OUTPUT_DEBUG_STRING_INFO),
                ('RipInfo', RIP_INFO)]

class DEBUG_EVENT(Structure):
    _anonymous_ = ('u',)
    _fields_ = [('dwDebugEventCode', DWORD),
                ('dwProcessId', DWORD),
                ('dwThreadId', DWORD),
                ('u', _U)]
LPDEBUG_EVENT = POINTER(DEBUG_EVENT)

CONTEXT_X86 = 0x00010000
CONTEXT_i386 = CONTEXT_X86
CONTEXT_i486 = CONTEXT_X86

CONTEXT_CONTROL = (CONTEXT_i386 | 0x0001) # SS:SP, CS:IP, FLAGS, BP
CONTEXT_INTEGER = (CONTEXT_i386 | 0x0002) # AX, BX, CX, DX, SI, DI
CONTEXT_SEGMENTS = (CONTEXT_i386 | 0x0004) # DS, ES, FS, GS
CONTEXT_FLOATING_POINT = (CONTEXT_i386 | 0x0008L) # 387 state
CONTEXT_DEBUG_REGISTERS =  (CONTEXT_i386 | 0x0010L) # DB 0-3,6,7
CONTEXT_EXTENDED_REGISTERS =  (CONTEXT_i386 | 0x0020L)
CONTEXT_FULL = (CONTEXT_CONTROL | CONTEXT_INTEGER | CONTEXT_SEGMENTS)
CONTEXT_ALL = (CONTEXT_CONTROL | CONTEXT_INTEGER | CONTEXT_SEGMENTS |
               CONTEXT_FLOATING_POINT | CONTEXT_DEBUG_REGISTERS |
               CONTEXT_EXTENDED_REGISTERS)

SIZE_OF_80387_REGISTERS = 80
class FLOATING_SAVE_AREA(Structure):
    _fields_ = [('ControlWord', DWORD),
                ('StatusWord', DWORD),
                ('TagWord', DWORD),
                ('ErrorOffset', DWORD),
                ('ErrorSelector', DWORD),
                ('DataOffset', DWORD),
                ('DataSelector', DWORD),
                ('RegisterArea', BYTE * SIZE_OF_80387_REGISTERS),
                ('Cr0NpxState', DWORD)]

MAXIMUM_SUPPORTED_EXTENSION = 512
class CONTEXT(Structure):
    _fields_ = [('ContextFlags', DWORD),
                ('Dr0', DWORD),
                ('Dr1', DWORD),
                ('Dr2', DWORD),
                ('Dr3', DWORD),
                ('Dr6', DWORD),
                ('Dr7', DWORD),
                ('FloatSave', FLOATING_SAVE_AREA),
                ('SegGs', DWORD),
                ('SegFs', DWORD),
                ('SegEs', DWORD),
                ('SegDs', DWORD),
                ('Edi', DWORD),
                ('Esi', DWORD),
                ('Ebx', DWORD),
                ('Edx', DWORD),
                ('Ecx', DWORD),
                ('Eax', DWORD),
                ('Ebp', DWORD),
                ('Eip', DWORD),
                ('SegCs', DWORD),
                ('EFlags', DWORD),
                ('Esp', DWORD),
                ('SegSs', DWORD),
                ('ExtendedRegisters', BYTE * MAXIMUM_SUPPORTED_EXTENSION)]
LPCONTEXT = POINTER(CONTEXT)

class LDT_ENTRY(Structure):
    _fields_ = [('LimitLow', WORD),
                ('BaseLow',  WORD),
                ('BaseMid', UBYTE),
                ('Flags1', UBYTE),
                ('Flags2', UBYTE),
                ('BaseHi', UBYTE)]
LPLDT_ENTRY = POINTER(LDT_ENTRY)

kernel32 = windll.kernel32

# BOOL WINAPI CloseHandle(
#   __in  HANDLE hObject
# );
CloseHandle = kernel32.CloseHandle
CloseHandle.argtypes = [HANDLE]
CloseHandle.restype = BOOL

# BOOL WINAPI CreateProcess(
#   __in_opt     LPCTSTR lpApplicationName,
#   __inout_opt  LPTSTR lpCommandLine,
#   __in_opt     LPSECURITY_ATTRIBUTES lpProcessAttributes,
#   __in_opt     LPSECURITY_ATTRIBUTES lpThreadAttributes,
#   __in         BOOL bInheritHandles,
#   __in         DWORD dwCreationFlags,
#   __in_opt     LPVOID lpEnvironment,
#   __in_opt     LPCTSTR lpCurrentDirectory,
#   __in         LPSTARTUPINFO lpStartupInfo,
#   __out        LPPROCESS_INFORMATION lpProcessInformation
# );
CreateProcess = kernel32.CreateProcessW
CreateProcess.argtypes = [LPCTSTR, LPTSTR, LPSECURITY_ATTRIBUTES,
                          LPSECURITY_ATTRIBUTES, BOOL, DWORD, LPVOID, LPCTSTR,
                          LPSTARTUPINFO, LPPROCESS_INFORMATION]
CreateProcess.restype = BOOL

# HANDLE WINAPI OpenThread(
#   __in  DWORD dwDesiredAccess,
#   __in  BOOL bInheritHandle,
#   __in  DWORD dwThreadId
# );
OpenThread = kernel32.OpenThread
OpenThread.argtypes = [DWORD, BOOL, DWORD]
OpenThread.restype = HANDLE

# BOOL WINAPI ContinueDebugEvent(
#   __in  DWORD dwProcessId,
#   __in  DWORD dwThreadId,
#   __in  DWORD dwContinueStatus
# );
ContinueDebugEvent = kernel32.ContinueDebugEvent
ContinueDebugEvent.argtypes = [DWORD, DWORD, DWORD]
ContinueDebugEvent.restype = BOOL

# BOOL WINAPI DebugActiveProcess(
#   __in  DWORD dwProcessId
# );
DebugActiveProcess = kernel32.DebugActiveProcess
DebugActiveProcess.argtypes = [DWORD]
DebugActiveProcess.restype = BOOL

# BOOL WINAPI GetThreadContext(
#   __in     HANDLE hThread,
#   __inout  LPCONTEXT lpContext
# );
GetThreadContext = kernel32.GetThreadContext
GetThreadContext.argtypes = [HANDLE, LPCONTEXT]
GetThreadContext.restype = BOOL

# BOOL WINAPI GetThreadSelectorEntry(
#   __in   HANDLE hThread,
#   __in   DWORD dwSelector,
#   __out  LPLDT_ENTRY lpSelectorEntry
# );
GetThreadSelectorEntry = kernel32.GetThreadSelectorEntry
GetThreadSelectorEntry.argtypes = [HANDLE, DWORD, LPLDT_ENTRY]
GetThreadSelectorEntry.restype = BOOL

# BOOL WINAPI ReadProcessMemory(
#   __in   HANDLE hProcess,
#   __in   LPCVOID lpBaseAddress,
#   __out  LPVOID lpBuffer,
#   __in   SIZE_T nSize,
#   __out  SIZE_T *lpNumberOfBytesRead
# );
ReadProcessMemory = kernel32.ReadProcessMemory
ReadProcessMemory.argtypes = [HANDLE, LPCVOID, LPVOID, SIZE_T, SIZE_T_p]
ReadProcessMemory.restype = BOOL

# BOOL WINAPI SetThreadContext(
#   __in  HANDLE hThread,
#   __in  const CONTEXT *lpContext
# );
SetThreadContext = kernel32.SetThreadContext
SetThreadContext.argtypes = [HANDLE, LPCONTEXT]
SetThreadContext.restype = BOOL

# BOOL WINAPI WaitForDebugEvent(
#   __out  LPDEBUG_EVENT lpDebugEvent,
#   __in   DWORD dwMilliseconds
# );
WaitForDebugEvent = kernel32.WaitForDebugEvent
WaitForDebugEvent.argtypes = [LPDEBUG_EVENT, DWORD]
WaitForDebugEvent.restype = BOOL

# BOOL WINAPI WriteProcessMemory(
#   __in   HANDLE hProcess,
#   __in   LPVOID lpBaseAddress,
#   __in   LPCVOID lpBuffer,
#   __in   SIZE_T nSize,
#   __out  SIZE_T *lpNumberOfBytesWritten
# );
WriteProcessMemory = kernel32.WriteProcessMemory
WriteProcessMemory.argtypes = [HANDLE, LPVOID, LPCVOID, SIZE_T, SIZE_T_p]
WriteProcessMemory.restype = BOOL

# BOOL WINAPI FlushInstructionCache(
#   __in  HANDLE hProcess,
#   __in  LPCVOID lpBaseAddress,
#   __in  SIZE_T dwSize
# );
FlushInstructionCache = kernel32.FlushInstructionCache
FlushInstructionCache.argtypes = [HANDLE, LPCVOID, SIZE_T]
FlushInstructionCache.restype = BOOL


#
# debugger.py

FLAG_TRACE_BIT = 0x100

class DebuggerError(Exception):
    pass

class Debugger(object):
    def __init__(self, process_info):
        self.process_info = process_info
        self.pid = process_info.dwProcessId
        self.tid = process_info.dwThreadId
        self.hprocess = process_info.hProcess
        self.hthread = process_info.hThread
        self._threads = {self.tid: self.hthread}
        self._processes = {self.pid: self.hprocess}
        self._bps = {}
        self._inactive = {}

    def read_process_memory(self, addr, size=None, type=str):
        if issubclass(type, basestring):
            buf = ctypes.create_string_buffer(size)
            ref = buf
        else:
            size = ctypes.sizeof(type)
            buf = type()
            ref = byref(buf)
        copied = SIZE_T(0)
        rv = ReadProcessMemory(self.hprocess, addr, ref, size, byref(copied))
        if not rv:
            addr = getattr(addr, 'value', addr)
            raise DebuggerError("could not read memory @ 0x%08x" % (addr,))
        if copied.value != size:
            raise DebuggerError("insufficient memory read")
        if issubclass(type, basestring):
            return buf.raw
        return buf

    def set_bp(self, addr, callback, bytev=None):
        hprocess = self.hprocess
        if bytev is None:
            byte = self.read_process_memory(addr, type=ctypes.c_byte)
            bytev = byte.value
        else:
            byte = ctypes.c_byte(0)
        self._bps[addr] = (bytev, callback)
        byte.value = 0xcc
        copied = SIZE_T(0)
        rv = WriteProcessMemory(hprocess, addr, byref(byte), 1, byref(copied))
        if not rv:
            addr = getattr(addr, 'value', addr)
            raise DebuggerError("could not write memory @ 0x%08x" % (addr,))
        if copied.value != 1:
            raise DebuggerError("insufficient memory written")
        rv = FlushInstructionCache(hprocess, None, 0)
        if not rv:
            raise DebuggerError("could not flush instruction cache")
        return

    def _restore_bps(self):
        for addr, (bytev, callback) in self._inactive.items():
            self.set_bp(addr, callback, bytev=bytev)
        self._inactive.clear()

    def _handle_bp(self, addr):
        hprocess = self.hprocess
        hthread = self.hthread
        bytev, callback = self._inactive[addr] = self._bps.pop(addr)
        byte = ctypes.c_byte(bytev)
        copied = SIZE_T(0)
        rv = WriteProcessMemory(hprocess, addr, byref(byte), 1, byref(copied))
        if not rv:
            raise DebuggerError("could not write memory")
        if copied.value != 1:
            raise DebuggerError("insufficient memory written")
        rv = FlushInstructionCache(hprocess, None, 0)
        if not rv:
            raise DebuggerError("could not flush instruction cache")
        context = CONTEXT(ContextFlags=CONTEXT_FULL)
        rv = GetThreadContext(hthread, byref(context))
        if not rv:
            raise DebuggerError("could not get thread context")
        context.Eip = addr
        callback(self, context)
        context.EFlags |= FLAG_TRACE_BIT
        rv = SetThreadContext(hthread, byref(context))
        if not rv:
            raise DebuggerError("could not set thread context")
        return

    def _get_peb_address(self):
        hthread = self.hthread
        hprocess = self.hprocess
        try:
            pbi = PROCESS_BASIC_INFORMATION()
            rv = NtQueryInformationProcess(hprocess, 0, byref(pbi),
                                           sizeof(pbi), None)
            if rv != 0:
                raise DebuggerError("could not query process information")
            return pbi.PebBaseAddress
        except DebuggerError:
            pass
        try:
            context = CONTEXT(ContextFlags=CONTEXT_FULL)
            rv = GetThreadContext(hthread, byref(context))
            if not rv:
                raise DebuggerError("could not get thread context")
            entry = LDT_ENTRY()
            rv = GetThreadSelectorEntry(hthread, context.SegFs, byref(entry))
            if not rv:
                raise DebuggerError("could not get selector entry")
            low, mid, high = entry.BaseLow, entry.BaseMid, entry.BaseHi
            fsbase = low | (mid << 16) | (high << 24)
            pebaddr = self.read_process_memory(fsbase + 0x30, type=c_voidp)
            return pebaddr.value
        except DebuggerError:
            pass
        return 0x7ffdf000

    def get_base_address(self):
        addr = self._get_peb_address() + (2 * 4)
        baseaddr = self.read_process_memory(addr, type=c_voidp)
        return baseaddr.value

    def main_loop(self):
        event = DEBUG_EVENT()
        finished = False
        while not finished:
            rv = WaitForDebugEvent(byref(event), INFINITE)
            if not rv:
                raise DebuggerError("could not get debug event")
            self.pid = pid = event.dwProcessId
            self.tid = tid = event.dwThreadId
            self.hprocess = self._processes.get(pid, None)
            self.hthread = self._threads.get(tid, None)
            status = DBG_CONTINUE
            evid = event.dwDebugEventCode
            if evid == EXCEPTION_DEBUG_EVENT:
                first = event.Exception.dwFirstChance
                record = event.Exception.ExceptionRecord
                exid = record.ExceptionCode
                flags = record.ExceptionFlags
                addr = record.ExceptionAddress
                if exid == EXCEPTION_BREAKPOINT:
                    if addr in self._bps:
                        self._handle_bp(addr)
                elif exid == EXCEPTION_SINGLE_STEP:
                    self._restore_bps()
                else:
                    status = DBG_EXCEPTION_NOT_HANDLED
            elif evid == LOAD_DLL_DEBUG_EVENT:
                hfile = event.LoadDll.hFile
                if hfile is not None:
                    rv = CloseHandle(hfile)
                    if not rv:
                        raise DebuggerError("error closing file handle")
            elif evid == CREATE_THREAD_DEBUG_EVENT:
                info = event.CreateThread
                self.hthread = info.hThread
                self._threads[tid] = self.hthread
            elif evid == EXIT_THREAD_DEBUG_EVENT:
                hthread = self._threads.pop(tid, None)
                if hthread is not None:
                    rv = CloseHandle(hthread)
                    if not rv:
                        raise DebuggerError("error closing thread handle")
            elif evid == CREATE_PROCESS_DEBUG_EVENT:
                info = event.CreateProcessInfo
                self.hprocess = info.hProcess
                self._processes[pid] = self.hprocess
            elif evid == EXIT_PROCESS_DEBUG_EVENT:
                hprocess = self._processes.pop(pid, None)
                if hprocess is not None:
                    rv = CloseHandle(hprocess)
                    if not rv:
                        raise DebuggerError("error closing process handle")
                if pid == self.process_info.dwProcessId:
                    finished = True
            rv = ContinueDebugEvent(pid, tid, status)
            if not rv:
                raise DebuggerError("could not continue debug")
        return True


#
# unswindle.py

KINDLE_REG_KEY = \
    r'Software\Classes\Amazon.KindleForPC.content\shell\open\command'

class UnswindleError(Exception):
    pass

class PC1KeyGrabber(object):
    HOOKS = {
        'b9f7e422094b8c8966a0e881e6358116e03e5b7b': {
            0x004a719d: '_no_debugger_here',
            0x005a795b: '_no_debugger_here',
            0x0054f7e0: '_get_pc1_pid',
            0x004f9c79: '_get_book_path',
        },
        'd5124ee20dab10e44b41a039363f6143725a5417': {
            0x0041150d: '_i_like_wine',
            0x004a681d: '_no_debugger_here',
            0x005a438b: '_no_debugger_here',
            0x0054c9e0: '_get_pc1_pid',
            0x004f8ac9: '_get_book_path',
        },
    }

    @classmethod
    def supported_version(cls, hexdigest):
        return (hexdigest in cls.HOOKS)

    def _taddr(self, addr):
        return (addr - 0x00400000) + self.baseaddr

    def __init__(self, debugger, hexdigest):
        self.book_path = None
        self.book_pid = None
        self.baseaddr = debugger.get_base_address()
        hooks = self.HOOKS[hexdigest]
        for addr, mname in hooks.items():
            debugger.set_bp(self._taddr(addr), getattr(self, mname))

    def _i_like_wine(self, debugger, context):
        context.Eax = 1
        return

    def _no_debugger_here(self, debugger, context):
        context.Eip += 2
        context.Eax = 0
        return

    def _get_book_path(self, debugger, context):
        addr = debugger.read_process_memory(context.Esp, type=ctypes.c_voidp)
        try:
            path = debugger.read_process_memory(addr, 4096)
        except DebuggerError:
            pgrest = 0x1000 - (addr.value & 0xfff)
            path = debugger.read_process_memory(addr, pgrest)
        path = path.decode('utf-16', 'ignore')
        if u'\0' in path:
            path = path[:path.index(u'\0')]
        if path[-4:].lower() not in ('.prc', '.pdb', '.mobi'):
            return
        self.book_path = path

    def _get_pc1_pid(self, debugger, context):
        addr = context.Esp + ctypes.sizeof(ctypes.c_voidp)
        addr = debugger.read_process_memory(addr, type=ctypes.c_char_p)
        pid = debugger.read_process_memory(addr, 8)
        pid = self._checksum_pid(pid)
        self.book_pid = pid

    def _checksum_pid(self, s):
        letters = "ABCDEFGHIJKLMNPQRSTUVWXYZ123456789"
        crc = (~binascii.crc32(s,-1))&0xFFFFFFFF
        crc = crc ^ (crc >> 16)
        res = s
        l = len(letters)
        for i in (0,1):
            b = crc & 0xff
            pos = (b // l) ^ (b % l)
            res += letters[pos%l]
            crc >>= 8
        return res

class Unswindler(object):
    def __init__(self):
        self._exepath = self._get_exe_path()
        self._hexdigest = self._get_hexdigest()
        self._exedir = os.path.dirname(self._exepath)
        self._mobidedrmpath = self._get_mobidedrm_path()

    def _get_mobidedrm_path(self):
        basedir = sys.modules[self.__module__].__file__
        basedir = os.path.dirname(basedir)
        for basename in ('mobidedrm', 'mobidedrm.py', 'mobidedrm.pyw'):
            path = os.path.join(basedir, basename)
            if os.path.isfile(path):
                return path
        raise UnswindleError("could not locate MobiDeDRM script")

    def _get_exe_path(self):
        path = None
        for root in (winreg.HKEY_CURRENT_USER, winreg.HKEY_LOCAL_MACHINE):
            try:
                regkey = winreg.OpenKey(root, KINDLE_REG_KEY)
                path = winreg.QueryValue(regkey, None)
                break
            except WindowsError:
                pass
        else:
            raise UnswindleError("Kindle For PC installation not found")
        if '"' in path:
            path = re.search(r'"(.*?)"', path).group(1)
        return path

    def _get_hexdigest(self):
        path = self._exepath
        sha1 = hashlib.sha1()
        with open(path, 'rb') as f:
            data = f.read(4096)
            while data:
                sha1.update(data)
                data = f.read(4096)
        hexdigest = sha1.hexdigest()
        if not PC1KeyGrabber.supported_version(hexdigest):
            raise UnswindleError("Unsupported version of Kindle For PC")
        return hexdigest

    def _is_topaz(self, path):
        with open(path, 'rb') as f:
            magic = f.read(4)
        if magic == 'TPZ0':
            return True
        return False

    def get_book(self):
        creation_flags = (CREATE_UNICODE_ENVIRONMENT |
                          DEBUG_PROCESS |
                          DEBUG_ONLY_THIS_PROCESS)
        startup_info = STARTUPINFO()
        process_info = PROCESS_INFORMATION()
        path = pid = None
        try:
            rv = CreateProcess(self._exepath, None, None, None, False,
                               creation_flags, None, self._exedir,
                               byref(startup_info), byref(process_info))
            if not rv:
                raise UnswindleError("failed to launch Kindle For PC")
            debugger = Debugger(process_info)
            grabber = PC1KeyGrabber(debugger, self._hexdigest)
            debugger.main_loop()
            path = grabber.book_path
            pid = grabber.book_pid
        finally:
            if process_info.hThread is not None:
                CloseHandle(process_info.hThread)
            if process_info.hProcess is not None:
                CloseHandle(process_info.hProcess)
        if path is None:
            raise UnswindleError("failed to determine book path")
        if self._is_topaz(path):
            raise UnswindleError("cannot decrypt Topaz format book")
        if pid is None:
            raise UnswindleError("failed to determine book PID")
        return (path, pid)

    def decrypt_book(self, inpath, outpath, pid):
        # darkreverser didn't protect mobidedrm's script execution to allow
        # importing, so we have to just run it in a subprocess
        with tempfile.NamedTemporaryFile(delete=False) as tmpf:
            tmppath = tmpf.name
        args = [sys.executable, self._mobidedrmpath, inpath, tmppath, pid]
        mobidedrm = subprocess.Popen(args, stderr=subprocess.STDOUT,
                                     stdout=subprocess.PIPE,
                                     universal_newlines=True)
        output = mobidedrm.communicate()[0]
        if not output.endswith("done\n"):
            try:
                os.remove(tmppath)
            except OSError:
                pass
            raise UnswindleError("problem running MobiDeDRM:\n" + output)
        shutil.move(tmppath, outpath)
        return

class ExceptionDialog(Tkinter.Frame):
    def __init__(self, root, text):
        Tkinter.Frame.__init__(self, root, border=5)
        label = Tkinter.Label(self, text="Unexpected error:",
                              anchor=Tkconstants.W, justify=Tkconstants.LEFT)
        label.pack(fill=Tkconstants.X, expand=0)
        self.text = Tkinter.Text(self)
        self.text.pack(fill=Tkconstants.BOTH, expand=1)
        self.text.insert(Tkconstants.END, text)

def gui_main(argv=sys.argv):
    root = Tkinter.Tk()
    root.withdraw()
    progname = os.path.basename(argv[0])
    try:
        unswindler = Unswindler()
        inpath, pid = unswindler.get_book()
        outpath = tkFileDialog.asksaveasfilename(
            parent=None, title='Select unencrypted Mobipocket file to produce',
            defaultextension='.mobi', filetypes=[('MOBI files', '.mobi'),
                                                 ('All files', '.*')])
        if not outpath:
            return 0
        unswindler.decrypt_book(inpath, outpath, pid)
    except UnswindleError, e:
        tkMessageBox.showerror("Unswindle For PC", "Error: " + str(e))
        return 1
    except Exception:
        root.wm_state('normal')
        root.title('Unswindle For PC')
        text = traceback.format_exc()
        ExceptionDialog(root, text).pack(fill=Tkconstants.BOTH, expand=1)
        root.mainloop()
        return 1

def cli_main(argv=sys.argv):
    progname = os.path.basename(argv[0])
    args = argv[1:]
    if len(args) != 1:
        sys.stderr.write("usage: %s OUTFILE\n" % (progname,))
        return 1
    outpath = args[0]
    unswindler = Unswindler()
    inpath, pid = unswindler.get_book()
    unswindler.decrypt_book(inpath, outpath, pid)
    return 0

if __name__ == '__main__':
    sys.exit(gui_main())



[top]

Thursday, 24 December 2009
[Daynotes Forums]    [Last Week]   [Mon]  [Tue]  [Wed]  [Thu]  [Fri]  [Sat]  [Sun]   [Next Week]    [HardwareGuys Forums]

00:00 -



[top]

Friday, 25 December 2009
[Daynotes Forums]    [Last Week]   [Mon]  [Tue]  [Wed]  [Thu]  [Fri]  [Sat]  [Sun]   [Next Week]    [HardwareGuys Forums]

00:00 -



[top]

Saturday, 26 December 2009
[Daynotes Forums]    [Last Week]   [Mon]  [Tue]  [Wed]  [Thu]  [Fri]  [Sat]  [Sun]   [Next Week]    [HardwareGuys Forums]

00:00 -



[top]

Sunday, 27 December 2009
[Daynotes Forums]    [Last Week]   [Mon]  [Tue]  [Wed]  [Thu]  [Fri]  [Sat]  [Sun]   [Next Week]    [HardwareGuys Forums]

13:40 - We're winding down after the Christmas holiday. Yesterday and today, I've been laundry and general cleanup, year-end computer stuff, and so on, while Barbara deep cleans room by room. I've done nine loads of laundry so far this weekend, including a couple of rug loads, with a couple more left to do on Tuesday. (Barbara's only workday next week is tomorrow, so I figured I'd wait.) I'm also spending a lot of time with supplier catalogs, deciding what new items I'd like to carry next year in the Science Room.

I came across an interesting Washington Post article this morning, about the future of e-books. The author got some stuff right, notably the unsustainability of a $9.99 price for e-books. As she notes, the majority of the Kindle Top 100 titles are priced below $9.99 (usually well below...) and the most popular price point is $0.00. What she ignores or is unaware of is the huge impact of DRM on pricing. People, at least in large numbers, simply won't pay $9.99 for an ebook that can't be moved freely between devices, can't be resold or lent/given to a friend, and may disappear at any moment without warning.

For current mass-market fiction, the price of a used paperback, call it $2, provides a ceiling for e-book prices, assuming that full-color ebook readers that support all common formats are available for $100 or less. For backlist fiction titles, call the ceiling $1. For "useful" mainstream non-fiction titles (such as my own science books, reference books, and so on), the ceiling may be $5 to $10. For limited-run titles, such as specialized science books that commonly sell for $200 to $1,000+, the ceiling may be $20 to $100. Call it 10% of the retail price of a hardback edition. Same thing for text books, where a $100 or $150 hardback title should sell for $10 to $15 as an ebook. Blame that all on DRM.

So long as DRM persists, book prices will trend lower and lower. And how that revenue will be split is an interesting question. Of course, without the author there is no book, so one would expect the authors to get most of the revenue. Of course, publishers are fully aware of the sea change that e-books portend, and for nearly 20 years they've been doing everything they can to lock authors into contracts that give the authors very little and reserve most of the revenue for the publishers. That's going to change, and in fact is already changing.

The one big thing that publishers have historically provided that authors could not realistically do for themselves is production/distribution, which is not a factor for e-books. Of course, publishers have also provided editorial/layout/design services, but those can be easily outsourced by individual authors to individual specialists. What about marketing and publicity, you might ask. The simple truth is that in mainstream fiction, publishers spend almost nothing to market and promote mid-list and lower authors. Those resources are devoted almost exclusively to books by top-selling authors. And, once again, there are individual companies that provide marketing and promotion services as good or better than anything publishers provide to the vast majority of their stables.

Amazon itself will be in trouble as the shift to e-books accelerates, which is why they so tightly lock the Kindle to their own e-book offerings. But the Kindle and its DRM are just flashes in the pan. The future of e-book formats is, as Tim O'Reilly has pointed out, in unprotected books that use open formats that support a wide variety of devices. As e-books come to dominate book sales, it'll be because those books are inexpensive, and easily transferred from device to device without any DRM in the way.

The overall pie will shrink, I'm sure, offset somewhat by increased sales of most titles, albeit for a much lower price. In the short term, over the next few years, I expect we'll see a growing trend toward a 50/25/25 split to the author, publisher, and reseller, respectively. Eventually, things will trend towards authors retaining 100% of revenue, cutting out both publishers and resellers. In the long term, in the next 10 to 15 years, I expect to see more companies specializing in author support services, including editing/layout/design and building storefronts for the authors to sell their own wares. In effect, those companies will allow authors to outsource all of the necessary work that's currently done by publishers and resellers.

In short, over the coming decade, I think things are going to get a lot tougher for authors, especially those who don't yet have an established track record and audience. I think we're also in the last days of the era of author superstars getting $1,000,000 advances. The best fiction authors, assuming they adapt to the new reality, will still be able to earn a middle-class income with their writing, although I expect tip-jars to feature heavily in that. But if authors are in for a long, tough haul, they should consider themselves lucky. Publishers and booksellers are in for a much worse ride, with no good outcome at the end. And they aren't going to go down quietly. I expect a real bloodbath as authors, publishers, and resellers compete each for a larger share of a shrinking pie. Ultimately, the authors will win, simply because there's no book without the author. But about the best I expect most authors to be able to say at the end of this process is "you should see the other guy".

Not that fiction authors are essential, which is the real problem for anyone who's a fiction author. Fiction authors compete not just against other current fiction authors and their backlists, but against legions of dead authors and their backlists. If no more mysteries were ever written, for example, we could all still read 10 high-quality mystery novels every day for the rest of our lives without running out. I read many more books than most people so, and yet I could easily list 1,000 or even 10,000 older books that I've never read but would like to. And, of course, the competition is not just books that are in the public domain, but books that are still in copyright but freely available on the Internet.



[top]

Copyright 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 by Robert Bruce Thompson. All Rights Reserved.