Putting a larger inverter in will probably need the DC cables increasing in size if you stayed at 12V, but in my view, the 3000 units is the max you should run at 12V, if you go to the 5000 unit then I would recommend 24V. If the boat is 12V now, then going 24V is going to probably need a dual voltage system plus some method of charging at 24V. The AC system will also need upgrading to handle higher current, new breakers, possibly new circuits.
Regarding kit, I used to have an old boat with a mix of cheaper inverter, battery charger and solar control, it all worked OK. I had a new boat built a few years ago, all Victron, all works together seamlessly via the Cerbo GX with good monitoring and I have not regretted the extra expense of the Victron gear.