Aha! A pin vice drill! That's a different animal, yet!
Usually, speed is your friend when drilling holes, especially in wood. Fast is good, but that's not going to happen with a 'pin-vice'. Not to fear, all is not lost.
Firstly, make sure that you are deeply marking your holes with an awl or something sharp. This will keep your bit from wandering too awfully bad, however. With soft woods, like basswood often found in kits, the grain of the wood can easily steer a thin, small diameter bit off center and also tear out wood grain even if deeply punched prior to drilling. If drill bit drifting and 'tear-out' are your issues either A) place masking tape or thin wood or cardboard over the part to be drilled to help guide the bit, or B) use a 'down-cutting' end mill as a drill bit. Down cutting end mill bits will pull the wood fibers 'down' into the hole rather than pulling them 'up' causing tear-out and sloppy holes. Yes, you can drill with an end-mill bit. Best bet for clean holes is with a rotary tool with speed, like a Dremel.