Tim,
Most likely your father is just rattling your cage about the poison in Castor oil.
...
That's probably what he is referring to as the "poison".
http://en.wikipedia.org/wiki/Castor_oil
From the article:
The castor seed contains ricin, a toxic protein. Harvesting castor beans is not without risk. Allergenic compounds found on the plant surface can cause permanent nerve damage, making the harvest of castor beans a human health risk. India, Brazil, and China are the major crop producers, and the workers suffer harmful side effects from working with these plants. These health issues, in addition to concerns about the toxic byproduct (ricin) from castor oil production, have encouraged the quest for alternative sources for hydroxy fatty acids.
From the article on Ricin:
The LD50 (50% fatalities from an exposed population - Asa) of ricin is around 22 micrograms per kilogram (1.76 mg for an average adult, around 1/228 of a standard aspirin tablet (0.4g gross)) in humans if exposure is from injection or inhalation. Oral exposure to ricin is far less toxic and lethal dose can be up to 20–30 mg/kg.
And
Despite ricin's extreme toxicity and utility as an agent of chemical/biological warfare, it is extremely difficult to limit the production of the toxin. Under both the 1972 Biological Weapons Convention and the 1997 Chemical Weapons Convention, ricin is listed as a schedule 1 controlled substance. Despite this, more than 1 million tonnes of castor beans are processed each year, and approximately 5% of the total is rendered into a waste containing high concentrations of ricin toxin.
To put ricin used as a weapon into perspective, it is worth noting that as a biological weapon or chemical weapon, ricin may not be considered very powerful in comparison with other agents such as botulinum or anthrax...
The major reason it is dangerous is that it is very easy to obtain (the castor bean plant is a common ornamental, and can be grown at home without any special care). Ricin is actually several orders of magnitude less toxic than botulinum or tetanus toxin, but those are more difficult to obtain.