In January 1807, the family of Hnery Lee was living at Stratford Hall on the Potomac River. The home had been built in the 1730s. It was the hereditary home of the Lees but, it was not Henry Lee’s home. The home had belonged to his cousin Thomas Lee. Henry Lee had married Thomas Lee's daughter, Matilda. Henry and Matilda Lee had lived there. Matilda bore Henry four children, only two of whom, Lucy Grymes Lee and Henry Lee IV, survived. When Matilda died in 1790, the home passed to Henry IV, their surviving son, still a minor.
Henry and his children continued to live in Stratford Hall. In 1793, Henry married Ann Hill Carter, an heiress of the immensely wealthy Carter family. Ann Carter Lee bore 5 children to Henry Lee. The first, Algernon Sydney Lee, died when he was ten . On a cold January morning in 1807, the 19th, herself sick with a cold, 33 year old Ann Carter Lee bore her fourth child and last son. She named him after two of her brothers, Robert and Edward Carter. She named him Robert Edward Lee.
Henry Lee, "Light Horse Harry Lee", had been a hero of the Revolutionary War, the most famous cavalry officer in the Continental Army. He rose to the rank of major general. He was three times Governor of Virginia. But he couldn't handle money. He lost everything he had and everything that Ann Carter Lee brought into the family. In April 1809, when Robert was barely two, Henry Lee was jailed for debt. He was released but then jailed again. While imprisoned, he wrote his "Memoirs of the War in the Southern Department". He hoped that its sales would repair his shattered fortunes. It did not.
By 1810, when he had been released a second time, Henry IV, the son of his first marriage, had come of age and become the master of Stratford Hall. Henry IV, by then 24, could not be expected to support his father's second family indefinitely. The general's family had to move. That summer, they settled in a small house in Alexandria where they lived off the meager income from a trust set up by Ann Lee's father. The family woes were far from over. In 1812, Henry Lee got involved in a controversy involving a Baltimore newspaper editor. His martial spirit aroused, he went to Baltimore to assist his friend. There, he was severely beaten by an irate mob and left for dead. His strong constitution enabled him to survive but he was a broken man, physically and financially. His creditors were unrelenting and he fled to the British West Indies in 1813, even though we were in the midst of the War of 1812 with Britain. Robert was six years old when his father left. He never saw him again for Henry died in 1818 while trying to make it back to Virginia. Henry was buried on Cumberland Island on the Georgia coast on the property of his old commander, Gen. Nathaniel Greene.
What future could 11 year old Robert have? Impoverished, son of a broken horne, son of a disgraced and fugitive father and an invalid mother? It says something about the man that he survived such a background and thrived. In 1820, when Lee was 13, he had to assume the duties of de facto head of the household. He "carried the keys" in the old Virginia phrase. His oldest surviving brother, Charles Carter Lee, had finished Harvard and was practicing law in Washington. His middle brother, Sidney Smith Lee, had been appointed a midshipman at Annapolis. His mother was a virtual invalid. His sister Ann always had been sickly. There simply was no one else.