I find that when my tiny bits start to get a little dull or begin to wobble I embed them a little deeper into the chuck to the level of the depth of hole required. This really steadies them up, and when that no longer works it's time for a new bit.
By the time you have your setup as you wish for a "less costly drill press" you will have spent a near amount of the Proxxon Milling machine MF70 price.
I have the Proxxon Milling MF70 and the Proxon drill press. Guess what, I use the MF70 for Milling and press drilling most of the times.
For ~$400 and Amazon free shipping you have 2 amazing tools (Milling and press drill).
Just my thoughts based on my experience. Others could see it differently.