My opinion - I wouldn't use it. It'll be dirty, may have oil on it from cars & trucks, etc. that may be very difficult or impossible to get washed out.
Don't put anything from the ground in your tank except for possibly rocks that you have boiled. That gravel from beside a road will almost certainly have oil in it from the cars that have passed by and could have any number of pesticides and other contaminants that have been sprayed in it.