mmhh, thats entirely up to your own interpretation. there are certainly moments that i purposefully left ambiguous, like when toji says naobito poisoned the sweet memories of his father (maybe he didn't, maybe he was just cruelly reminding toji of what actually happened and toji was in denial) or the scene where, after hitting jinichi, their father picks toji up and brings him alone to another room and locks the door.
the reason i didnt make it explicit is bc i dont think it matters tbh. the abuse toji endured from his father and naobito are two very distinct creatures in tojis mind. i dont think he ever really blamed his father, even though he was deeply hurt by him. he kinda justified it with the fact that he had "killed" his mother. with naobito though, he never understood why he did it and so he had a much harder time protecting himself from it