Brown, alias Ben Bailey Smith — his sister is novelist Zadie Smith — is used to a buzz. Before he became a comedian he was a rising star of the UK rap scene, working with Mark Ronson, Amy Winehouse and Lily Allen. He performed with them in front of 40,000 fans at festivals around the UK but then decided that that world was not for him. His show tells the story of how he fell out of love with the music business, had a family and changed his priorities.

"I was in a dark place. I just didn't like who I was, or how the people around me were changing."

The soft-spoken, polite 31-year-old saw first-hand the descent of Amy Winehouse: "When I first met Amy she seemed quite cool. But she'd just broken up with a guy and I think it hit her confidence. We hit it off and had a laugh, but then I saw that disappear."

He mocks the shallow trappings of success, from bling to groupies, in Unfamous, but does not joke about his early rap career, when violence dogged the scene. "You did spots called battles to get your name known but your credentials were questioned. I was never into streetlife, I never touched a weapon, but when I won a battle against some hard guys there were calls, death threats in the middle of the night. Crazy stuff. Eventually I carried a can of CS gas as self-protection. When I look back, it's nuts. I can't imagine ever feeling the need to carry a weapon now." He looks down at the table and shudders. "I can't tell you what people in my crew were using." It is no surprise he prefers comedy.

Brown's life sounds very different from that of his more famous sibling, but he and Zadie are incredibly close. He even gives her most famous book, White Teeth, a plug during his gigs, asking his fans to buy it — "in hardback". It was Zadie, who is two years older, who ignited Brown's interest in stand-up. "She and my dad were the real comedy fans, I only really knew Peter Kay and Jimmy Carr. Zadie used to go to clubs and tell me who she had seen." She is a fan of his comedy and he is a fan of her books. Most of them. "I loved the first and third books. The second one, On Beauty, went a bit over my head!"

There was little rivalry growing up in Kilburn. Brown only ever felt he was living in his sister's shadow once. They both attended Hampstead Comprehensive School and while Zadie was a star pupil, going on to Cambridge, Brown was a geeky tearaway. "I'd been in trouble for smoking behind the drama studios and being late and I was going to be suspended. I was called into the head's office and as she was laying into me I saw a framed photo of Zadie with the Speaker of the House of Commons, Betty Boothroyd, on the wall. But on the whole, it's an amazing thing to watch someone touch the world with their imagination. She was always someone who inspired me and someone I always looked up to."

The only real problem was the dent to his credibility. "I was so proud of her but I was so heavily into the street side I almost didn't want people to know I had a sister who had been to Cambridge. I laugh about it now, but at the time they were going, You've got a posh sister, how can you be a genuine rapper?'"

His rise through the stand-up ranks has been meteoric. He first got into comedy by writing humorous songs for Radio 4, before landing small parts on the radio sitcom Rudy's Rare Records, starring Lenny Henry. "Lenny was really supportive. In one show I had a scene with him and when I started getting laughs I thought, I like this '." He began to gig and quickly attracted attention for his spoof raps. "After one set I was talking to some friends and this guy offered me his card. I'd had loads of people like that when I was a rapper, so I just shrugged him off. After he went, my friend said, That was Ricky Gervais's agent'." The same man, Duncan Hayes, now handles Brown.

The passionate Crystal Palace supporter represents a new kind of black comedy for multicultural Britain that doesn't dwell on race. "I don't see why I need to talk about it. It's always under the surface because of my colour. Not referencing it directly is the strongest statement there is." He thinks for a moment. "I don't know if there is a race divide in Britain today. It's more about class. Look at the way our politicians all come from the same backgrounds. That's why Obama is great. Not because he's black but because he had a different background."

Having a white father, Harvey, who was 30 years older than his black mother, Yvonne, Brown's childhood wasn't all laughs. "Being mixed race is a blessing now but as a kid it was a curse. I didn't know who I was. You look at your parents and you don't look like either of them. Combine that with racists shouting at you and it can really hit your self-confidence. Now, it's no big deal, England is more tolerant than any other country in the world. I'm such a patriot, it's ridiculous."

He loves London, even though all he could afford to buy with his wife, who is a teacher, was an ex-council flat in Newington Green. They have two young daughters — "the finer things", as he calls them in his show. Does he miss the roar of 40,000 music fans? "Sometimes, no question. With music you can live off one great song for 20 years. If you tried that in comedy with one joke, you'd last a week. I miss the ease of music. Comedy is much harder!"