Mistress of Misfortune by Hazel Hunter

Rumors swirl around Dredthorne Hall like snowflakes in a blizzard: the handsome war hero who lives there never sleeps; he never allows women to enter; worst of all, the Hall is haunted. But Meredith Starling will hear none of it. Instead she finds it enchanting.
Though prone to accidents and mishaps, Meredith rides by the Hall at every opportunity. For her the rundown manor evokes a grander time. She looks past the broken gate and moss covered statues and sees the imposing facade’s elegant lines and graceful charm. In fact, she is determined to see inside.
But Colonel Alistair Thorne rebuffs all callers without exception—and not for the sake of being a recluse. He avoids women because of his family curse. The master of Dredthorne is doomed to fall in love with a lady who spends a night at the Hall. Though love would be welcome, its consequences would not. All of the mistresses of Dredthorne have vanished, gone mad, or suddenly died.