Basically the diesel process is this : Piston comes up and compresses the air, in the process the air gets hot*. At (hopefully) the right point, the injection system sprays fuel into this hot air - and as long as the air is hot enough*, the fuel will spontaneously combust. The fuel is broken up into tiny droplets, each one burning a bit like a candle flame - the liquid gets hot, evaporates and forms a "shell" of vapour, the vapour mixes with the surrounding air, and in the combustible mixture zone it burns, with the heat from the flame providing heat to keep the evaporation process going.
In the presence of lots of excess air (ie engine at low power), and given sufficient time (ie engine running slowly), the droplets can more or less completely combust.
Change either of these, or both when doing your "drag start" at high power and high revs, then there is little (if any) spare air and limited time for combustion. In this situation, a small "core" of mostly carbon gets left over from each droplet of fuel when the air (and time) runs out - add lots of droplet cores together and you get the grey/black smoke that's so familiar when following a diesel that's just "put the foot down".
So much development work by the engine manufacturers for decades has been on two fronts: getting a finer and finer spray (hence the very high pressure systems common these days) because smaller droplets will burn fully much better; getting a better mix of the fuel mist and cylinder air because the better the mix, the less variation and so less opportunities for fuel to have insufficient air for complete combustion.
It should also be fairly obvious why setting of maximum fuelling is critical. Add too much fuel and there's not enough air to burn it all - so you get lots of smoke. Set it too low and you don't get full power. And if you're struggling to pass the MoT smoke test, a quick fix can often be to turn the fuelling down a bit - which can mask that the injectors are getting a bit worn and not creating as fine/even a spray as they did when new
* This is why starting a diesel can be so hard in cold weather. Compression ratio is critical (many engines have a choice of cylinder head gasket thickness to accommodate manufacturing variations), as is cranking speed (the faster you crank, the less time available for the heat to get absorbed into the cold metal of the engine). And also why most small diesels have heater plugs - basically to provide a hot spot that can initiate combustion when cranking alone doesn't cut it.