Robert Fitz-Roy was an aristocrat, the grandson of the Duke of Grafton and the nephew of Lord Castlereagh.

He was born on July 5th in 1805 (the year of the battle of Trafalgar) on Hampton Road, Suffolk. His mother died when he was six years old, and he was sent to boarding school near Brighton. At the age of twelve, he entered the Royal Naval College in Portsmouth where he received outstanding grades, the mathematics prize and the first ranking in his class. He sailed at the age of fourteen, as a volunteer apprentice, on the Owen Glendower, to South America and was promoted to midshipman one year later.

After having served in the Mediterranean, the Canal of the Mancha and South America, he was promoted to lieutenant at the age of nineteen, the first among 27 in his class.