CORNWALL—Charlotte Bronson Hunnewell Martin once wrote that “one of the aristocrat’s attributes needed now more than ever is his endeavor to preserve the culture of the golden gifts of the past. Men have always lived with symbols, and culture is the outward form of the spirit within.”

During her time, Mrs. Martin—who is buried on Cemetery Hill in Cornwall—gave the New England town she loved a symbol that still stands today. It is Hidden Valley Castle.

Built between 1921 and 1924, the castle near Cornwall center is privately owned and closed to the public. But resident and historical society member Jeff Jacobson plans to bring the woman for whom it was built—with her romantic liaisons, New York high society dramas, philanthropy and unique obsession with aristocracy—back to life with a talk at the historical society on Nov. 13.

Charlotte Bronson Hunnewell came from an old Massachusetts family that had made its fortune in banking and railroading. After her parents’ early death, by her 20s she was living with her brother and sister-in-law, and decided she wanted out of her situation.

After one unsuccessful marriage of convenience to a Victor Sorchan—whom Mr. Jacobson says his research has described as a “complete zero,” and for whom she eventually bought a seat on the New York Stock exchange, just to give him something to do—she eloped with a Dr. Walton Martin, successful New York surgeon, whom she married in 1920 and loved up until the day he died in 1949. It was with Dr. Martin with whom she built the castle.

“He was the real deal,” said Mr. Jacobson of the second husband. “Chief surgeon of St. Luke’s, multilingual, cultured, interesting, and in later years, he was one of the key people in developing the local NAACP. He was instrumental in setting up a major hospital in France during World War I, which treated battlefield wounded. People in Cornwall still remember him with great fondness, as he was a real warm and good person.”

Mrs. Martin’s first foray into architecture was Turtle Bay Gardens, a series of 19 brownstones in New York that she bought in 1918 to turn into an enclave for New York’s artistic and literary set. Her creation still stands today along East 49th street, and her vision of societal splendor was fulfilled; E.B. White wrote his famous book “Charlotte’s Web” there, and the brownstone Audrey Hepburn owned for 60 years is available for rent (at $29,000 a month). But this was only the beginning, as she always believed that true nobility lived out in their country chateaux.

“Her focus was on the nobility in Europe around the French revolutionary period. She felt she was an aristocrat, and the whole thing she had done in Turtle Bay Gardens was a stage setting to be surrounded by the people she wanted to be surrounded by,” said Mr. Jacobson. “The second stage of this was up here in Cornwall.”

The castle was built by Dr. Martin and his wife as their country retreat in 1921, long before Litchfield County was a hotspot for country retreats. Local stone was used, and the couple used Torrington builders Marola Construction—now a heavy duty truck dealership—to build it. Whether they were inspired during their many trips to Europe, Mr. Jacobson was never able to discover even though he interviewed the grandson of the architect. Still, the building is emblematic of the last-ever spate of castles that the rich built for fun, rather than as protection from invaders.

“This is not the sort of castle you read about in medieval history,” he said. “This was a style called storybook architecture, similar to Mad King Ludwig’s Neuschwanstein.”

Dr. Martin and his wife were together a strange sight in Cornwall, which was then still a rural farming town in the 1920s and not the eclectic community of artists and expat New Yorkers it is now. The residents who remember her before she died in 1961—one of whom will be speaking at the event—say she always wore formal, tailored clothing and was chauffeured from Manhattan to Cornwall in her husband’s 1929 Lincoln.

To fulfill the dream world, Dr. Martin would even occasionally ride a white horse through the valley that surrounds the castle, wearing a white tailored cape with red lining. His wife’s fixation on aristocracy came to a head in the 1950s, when she wrote “The World and the Aristocrat,” which Mr. Jacobson calls a “remarkably well researched” account of French aristocracy at the time of the revolution.

The castle’s story has a personal significance for Mr. Jacobson, as 10 years ago, he became friends with a new neighbor named Nick Rondinone. Mr. Rondinone’s father Vincenzo—an Italian immigrant and skilled potter who emigrated to New York—was chosen from an entire ensemble of potters to be the castle’s resident artisan in the 1920s, just as medieval royalty patronized artists like Da Vinci. Mrs. Martin built a home and workshop for him and his family.

“It was common for aristocracy in Europe to have craftsmen at that time, and if you were well bred, well educated and well taught, you recognize the value of artisans, and sponsor these people,” he said. “The general feeling was that the manor lord would make himself better in the eyes of his associates and contemporaries if they saw he was doing this.”

Branded “Narrow Valley Pottery,” Vincenzo Rondinone’s work was for a decade a fixture at the castle, decorating the interior, sold through an agent in New York and given as gifts to guests, until he died of a stroke in the 1930s.

Nick Rondinone bought the pottery workshop near the castle where he had spent his childhood in 1999, intending to turn his into his retirement home. Six months afterward, though, he and his wife both died in a tragic car accident in Kent.

“I’ve never gotten that out of my mind,” Mr. Jacobson said of his late friend. “I have been digging for the past two years about the castle, from Massachusetts to Manhattan, about them, about the pottery, and about the potter and his wife. And mainly about the amazing lady who was responsible for all of this.”

While Mrs. Martin was not beloved by all—in a Leona Helmsley-like move, she cut her only child out of her will but left $1,000 to the castle caretaker—and some of her descendants still paint her in an unflattering light, Mr. Jacobson said he has a nuanced view of the woman who was once Cornwall’s wealthiest resident.

“She was very definitive, knew who she was and what she wanted to be, the people she wanted to be with,” he said. “The people who do things, who create things, are different. You have to be a bit outside the ordinary, and this woman was something that you read about in novels.”

Mr. Jacobson’s talk, “The Cornwall Castle Aristocrat,” takes place Nov. 13, at 2 p.m. at town hall. More information can be found at www.cornwallhistoricalsociety.org.