There already are diesel drift cars, especially mercedes. HP and torque sort of go hand in hand, but for drifting (especially low HP) torque will be more important.
In the end you are only limited by budget. Just today I have seen a 4000HP diesel quad turbo truck. So it is possibl to get serious with a diesel engine. It will however be much harder for you to start with it because there will be barely any info on it as opposed to a much more common petrol engine. If I were to build a diesel car, I would try to find an early 330d bmw. Straight 6, 3 litre, turbo, relatively simple (compared to todays) and the chassis if very much driftable.