The most natural ways of making caves are probably by stacking various pieces of driftwood or rocks, on top of, and beside, each other. A combination of the two would look great! Slate rock and Mopani driftwood would probably be best for this, because mopani comes in arching shapes, already creating a 'from' of cave. Stacking them and adding rocks would make a really nice display!
A Flower pot laying down on it's side is also very good, but i'd prefer the above method because with a flowerpot, you only get 1 cave, and compared to how much area the flowerpot takes up and in the same amount of area how many caves you'd get with driftwood, the driftwood would ideally provide more caves for your fish.
Be careful with driftwoods and rocks though, because some may alter your PH.
There are also ways to make caves out of styrofoam. You should check out the DIY section for some cave ideas. The good thing about making them from styrofoam is that you'll get the shape you want, but it's a bit more time consuming
I've seen PVC piping used as longer, slender caves for fish like loaches and bichirs.