Have you ever wondered how the entire Griffin family has been able to get into all sorts of hijinks, ranging from the paranormal to societal, without there being any repercussions? Well, the answer is very simple - the entire plot is within Peter's mind.
It all started with Meg - rather than being the big, unattractive, unpleasant girl that everyone in the show makes her out to be, she was a very beautiful and very popular teenage girl, at the prime of her life in high school. And she was also very caring for her poor younger brother, Chris. Chris didn't get enough oxygen when he was born, thus resulting in his mental retardation throughout his life.
Meg had a special bond and love for Chris, and could not stand the fact that Chris could not receive the attention and love that she had received from everyone, including their parents. As a result, one night Meg took her younger brother to a party. However, after heavy drinking on Meg's part and a panic attack on Chris' part, Meg decided to take her brother home.
As a result of her inebriated state, Meg totaled the car - resulting in the death of her and her brother. Unable to grieve with the death of her two children, Lois committed suicide shortly after the accident, while she was pregnant with a son that Peter posthumously decided to name "Stewie". Unable to cope with the death of everyone that he had ever loved, Peter slowly slipped into insanity, creating a world where his three children and his wife were still alive. That is why everyone in the show is so mean and distasteful towards Meg - Peter still resents her for ultimately being responsible for the death of everyone around him.
However, there is still a minuscule part of Peter that loves Meg for the beautiful and wonderful girl she was, and the love that she held for Chris, which is why, every once in a while, you see Peter showing some sort of compassion towards his daughter.
This would also explain the hyper intelligence of Stewie. Unsure of how Stewie's future would have become, Peter imagined his son with great intelligence and great potential, creating any sort of gadget that he would please. However, his resentment towards Lois also builds up in his image of Stewie, angry at her for taking away their still unborn child, which would explain why Stewie constantly plans for his mother's death, which would explain why Stewie never seems to age. As for Peter's extreme stupidity, he wasn't like that in real life.
Peter was actually a very intelligent man. However, feeling like he had not met the standards of his role as a father and a husband, unable to protect his children and his wife, Peter in himself has turned himself into the complete idiot that he perceives himself to be.
However, he has created a different personality to express the intelligent man that he had always been, in the form of the family dog, Brian. As for Cleveland, Quagmire, and Joe, they all are real people involved in Peter's life. Cleveland and Quaqmire did live in the same neighborhood as Peter. Their appearances in the show represent the times that they visit their old friend, Peter, in the mental hospital, and attempt to humor him in his insanity-induced world.
Cleveland and Quagmire also told Peter about Joe, the man who moved into his new house with his wife, his son, and a newborn baby on the way. Having never met Joe, Peter decided to include him into this new world, as the man that he never could have been - a strong, heroic, devoted father and husband, who unconditionally loved his family, despite his crippling disability.
Cleveland, having visited his friend Peter more in the hospital, found that he himself was slowly starting to lose his mind when he was helping Peter indulge in his imagination. As a result, Cleveland decided to stop his visitations to Peter, resulting in the "spin-off show" that Peter created. In his new world, Peter constantly is the one who brings troubles onto himself and his family. However, in the end of it all, he knows that his whole family is safe, and that he loved each and every member unconditionally. And that is really all that Peter Griffin ever wanted.