To make things cheap, get your tank second hand, and get the biggest tank you have space for. 15-20 gallons is a good starter size, not too huge ( would fit on top of a solid chest of drawers or strong solid desk or worktop if you haven't space for a stand ) and the bigger your tank, the EASIER is will be to maintain.
Ebay, car boot sales, Freecycle, newspaper classifieds, asking around the neighbors if anyone has an old spare tank lurking in the garage/loft that they no longer want... all good places to get a cheap or free tank.
Heaters and filters can be bought cheaply on ebay, and you can't go far wrong with a basic fluval 2 or 3 ( for the tank size I mentioned) internal cannister filter. Exceptionally easy to run and set up, a child could do it. A 50 watt heater will be fine.
As stated, your heater and filter will not cost much to run the tank at all so have no worries about that.
By being careful with your feeding regime ( in other words don't feed too much or even every single day ) and by doing 50% water changes once a week on a filtered tank, your fish should stay healthy and happy. Changing 50% once a week on a smallish tank will take you less than half an hour. It only takes me 45 mins to do 3 tanks!!!
Not sure if you've seen the pics but if you like goldfish then I would honestly reccomend Gold barbs very highly. Similar body shape, they don't grow massive and you can have a shoal of 6 in a 15-20 gal tank . They are very active fish, I have 6 myself and they've livened the tank up no end. They are also very simple to care for ( most basic tropicals are really )
Other easy fish to try, are Male Guppies, Male Endlers, Danios and Platies (males only, like guppies they are livebearing fish and breed like crazy, which is not something you need if you are busy ) Cories and Khulie loaches are easy bottom feeders to try too.
I would advise to avoid tetras completely since the tank will be freshly set up. They need mature tanks that have been running for at least 6 months.
You will need to read the stickied topics on cycling. This is important for setting your tank up properly.
to TFF!!!!
/en.wikipedia.org/wiki/File:White_Cloud_Mountain_Minnow_1.jpg Clicky!! 
the best thing about mountain minnows,in my oppinion,is that they are compatible with chinese fire bellied newts.I think you could put a pair of newtts in there with some mountain minnows.
Poor newty! (XD at the name) 
