I’d go table saw as well but mostly because I picked up an inexpensive disc sander (from Menards online) that works for me for now - though I’d love to upgrade to the Byrnes. The table saw was useful cutting replacement parts/planks but do consider if you’ll really need it.
What bits are you using now? I've had a lot of great success with HSS micro bits from McMaster-Carr: https://www.mcmaster.com/catalog/129/2750/30585A94. I had tried some carbide cheap end mill/PCB bits and they were nothing but frustration in a handheld pin-vice. I switched to Uncoated High-Speed Steel Drill Bit, 76 Gauge Size for drilling out the treenail holes on my cross section and never broke a bit (even though I bought 5 just in case).
Currently my big issue is that I don't really drill straight holes by hand, but that's not the bit's fault :-).