James Cathcart Johnston was known as a bachelor. Research published in 2013 by Mary Maillard ("'Faithfully Drawn from Real Life': Autobiographical Elements in Frank J. Webb's The Garies and Their Friends," Pennsylvania Magazine of History and Biography) reveals that although Johnston never married, he was the father of four daughters by his manumitted mistress, Edith "Edy" Wood, of nearby Hertford, N.C. Two of his girls died at the age of eight and nine in 1836, and his eldest daughter, Mary Virginia Wood Forten (daughter-in-law of wealthy African American abolitionist, James Forten), died in Philadelphia of tuberculosis in 1840, leaving behind her three-year-old daughter, the future diarist, poet, and equal rights activist Charlotte Forten Grimke. Johnston's youngest daughter, Annie Wood (1831–1879), was just six years older than her niece Charlotte, and the two girls were raised by Edy Wood until Edy's death in 1846. The girls continued to be raised with the Forten-Purvis clan while Annie Wood was adopted by Amy Matilda Cassey, daughter of African American Episcopal priest, Peter Williams, Jr. and wife of wealthy African American financier and benefactor, Joseph Cassey. In 1850, after Joseph Cassey's death, Amy Matilda Cassey married antislavery orator, Charles Lenox Remond, and moved from Philadelphia to his home in Salem, Massachusetts, taking Annie Wood with her. While in Salem, Annie Wood married her childhood sweetheart, John G. Webb, grandson of Vice President Aaron Burr and brother of Frank J. Webb, author of the second African American novel, The Garies and Their Friends. James Cathcart Johnston paid for Annie Wood's education, made generous payments to her as she grew up, and promised her an "independence."
James C. Johnston lived on the plantation that his Samuel had built and left to him, Hayes Plantation. "Samuel Johnston died at his home, Hayes Plantation, near Edenton in Chowan County, in 1816 and is buried in the Johnston Burial Ground there. The plantation house is privately owned, but was designated a National Historic Landmark in 1973. It is now within Edenton. However the current house was completed by his son, James Cathcart Johnston, a year after Samuel's death." James died at the end of the Civil War