Ok, this is going to be a long post but I feel there are a few things I want to help clear out, if at all possible.
I'm aware that there's a height map that is used to determine the terrain mesh geometry, and the terrain textures themselves do not modify the terrain geometry, only lighting. No worries on that regard, misunderstandings happen.
I'm using the term "elevation" because bump mapping uses a height map, and I use height synonymous to elevation, so no problem there. Also, in the case of water textures, BumpH maps actually do modify the water surface's geometry, but obviously not for terrain textures. The texture format is the same in both cases anyway - greyscale height map.
The reason I pointed out that BumpH maps are not normal maps is that bump mapping is a completely different technique for creating the impression of surface geometry than normal mapping is. The difference is mainly in that bump mapping uses a height map to interpolate surface normals on-the-fly, while normal map has all that already built into it so you have a more direct way of adjusting the surface normals - and not just that, but because bump mapping relies on interpolating the
gradient of the height map, you get lower effective resolution as well.
And the reason why this is relevant is that a process to create a height map should never, ever involve converting it into a normal map and then back to height map. That's just going to throw all sorts of wrecking balls to the information actually present in the texture, and while it might produce some kind of results that look acceptable.
Let's see if I can make sense of this...
Here's an example height map. It's not a terrain texture, but it will do for the demonstration. It has a few recesses (darker areas) and elevated areas (brighter areas, like the pyramid):
And here's how you would expect it to effect the surface, assuming light is coming from above and left:
Now if you were to convert this into a normal map, you would get something like this:
And now we get to the problematic thing about the process you described.
If you now take this normal map, and convert it to greyscale, you get
this:
...and now there's a problem, because what is bright on the normal map does not correspond AT ALL to what is bright (elevated) on the actual height map! In fact, you get a situation where you have lost all the gradient information from the original height map, and instead of a naturally changing elevation you have these flat areas of about equal brightness, but with very sharp jumps from one "plateau" to another.
If you use this in bump mapping, the end result will not in any way correspond to the original texture's possible height variances - which, by the way, are usually not directly related to the brightness of the texture, so at best the method of converting a texture directly into height map by desaturation is a bit iffy, and at worst produces completely bogus height maps - for example if there is a dark rock in the middle of some white sand, it will come out as a hole in the bump-mapped surface. But I'm sure you are aware of this. The bigger problem is that by including a normal map in this process you essentially destroy a lot of information and then treat the output as the same thing as the input, which isn't exactly a good idea.
For demonstration's sake, here's what you get when you use the desaturated normal map for bumpmapping:
...and a side-by-side comparison with the original height map bumpmapped:
normal map result
proper height map result
If I had to make a height map of your sample texture with the rocks in it, I would do it like this:
desaturate by Average intensity:
Level so that the "cracks" between the rocks become dark and the stones themselves bright:
Applied gaussian blur to the desaturated texture and added a really small amount of RGB noise:
Add the black "cracks" between the rocks, from the earlier step:
Finally, I applied levels to reduce the range from "lowest" to "highest" elevation, instead of running the full gamut from 0 to 255 I made it run from 94 to 160:
This produces the following bump mapping results (in GIMP):
...and now all that's left is to do a little test to see whether the bump mapping results
make sense, in comparison with the original texture:
Since the bump mapping seems to "blend" in a sensible way with the original source texture, it's probably going to work more or less well in-game as well. Yes, it's rounded - that comes from the gaussian blur applied to the thing, but to be honest I don't really see any obvious way to recreate the angular shapes of this kind of rocks just from the source texture alone. This result is what I would call "good enough"... but even better would be to make some randomized rocky ground mesh in 3d application, then make a height map based on that. Or make a laser probe that can minutely track the elevation differences of a surface, and use that to record some data from some actual rocks on the ground... and use that to make a height map.
But just for making a height map for rocky terrain, these include so much more effort that it's probably going to be better to just use the source texture and accept that the result isn't going to be perfect.
Now, I realize that the levels (range from darkest to brightest colour, as well as gamma curve) need to be edited so that the end result looks good in specific program use; in this case, IL-2 rendering system. However, the actual geometry of the height map should be retained as much as possible, and definitely not subjected to normal mapping in the process of editing.
Stuff for the mathematically inclined:
Height map can be consider a scalar field. Scalar fields have a property called gradient, which is essentially the partial derivatives of the field; for a two-dimensional scalar field, gradient H (height) has two components: dH/dx component (derivative by X axis) and dH/dy. When the gradient is zero, that means the RATE OF CHANGE in elevation is zero in either X or Y direction, so that will be interpreted as "flat" or "even" area. If the elevation is changing when you move along X or Y axis, then you have a non-zero gradient.
Normal mapping basically just interpolates these derivatives from the height map, and saves them as pixel values in two channels, ranging from negative derivative (values [0,127] on the channel information) to zero derivative (127 or 128 depending or both depending on implementation) to positive derivative (values [128,255]). So what you're actually getting is that a very high SLOPE on the height map will get either dark or bright values, regardless of their actual elevation. This is actually a classic case of losing information while performing derivative operation, and you cannot get that information back without knowing the integration constant... and actually it's nearly impossible to get height map information back from normal map once that step is done.
If all that just sounds like a lot of words, don't worry, this stuff is basically university level maths and is usually not handled in high school besides possibly cursory overview.
Finally...
I don't say this because I'm trying to step on anyone's toes or to insult anyone's intelligence. I made this post to help everyone get the best possible results from their modding, and to perhaps help clarify any misconceptions that might be going on in the theoretical side of all this modding, which might end up affecting the end results. If you have any questions, I'll be more than glad to answer them...