The what, where, when and how of ballast changed over times.
I know that ships when empty of cargo would carry more temporary ballast aka rocks or gravel, which was dumped overboard often when heavy cargo was loaded to keep the ship at proper depth in water.
Most ships carried some permanent type ballast which could be adjust from fore to aft, or port to starboard as winds and seas dictated,
This type of ballast was know as trim ballast. What they were made from varied with age and country of the ships origin.