From your post it looks like this might be the first time you've tried sanding primer on your model. I'd back into the best grit. Start with 600 or 800 and see what happens. If you can safely be more aggressive back out to 400, 240 etc until you find the grit that works efficiently but leaves the primer and wood below in good shape.